Hi,

I am trying to create a DS2 large configuration on a system Linux system (mysql backend).

After creating the prod.csv and inv.csv I am trying to create the database using the mysqlds2_create_all.sh script.

This script has been running for last 2 days and now has created a data file ~103GB.
--
ls -lh /mnt/disk2/mysql/ibdata1
-rw-rw---- 1 mysql mysql 103G 2009-12-22 08:01 /mnt/disk2/mysql/ibdata1
--

As far as I understand the large configuration should create a data set of 100GB.

Now that this has gone beyond 100GB, I am not sure if it is functioning correctly or should I stop it ???

I have ran the smaller configurations successfully before on this system.

Please advice.

Thx
-sk

mysqlds2_create_all.sh is used to create and load the database after the datafiles have been created. You should run the command first using the default small datafiles to make sure everything works, then create and load the large datafiles.

You mention that you created the large prod.csv and inv.csv. What about the files in the /ds2/data_files/cust and orders directories? You need to create large versions of those too. (The large datafiles take up about 70GB altogether). Remember, you have to put the large datafiles in the same directory structure or modify the load scripts.

Dave

Hi Dave,

Thanks for the reply.

I have actually tried with the small/medium configurations successfully. This was the first time I was creating the large configuration.

The files in darta_files/cust and and data_files/orders were allso created.

But... I just re-checked the data_files/prod directory and it does not have inv.csv any more. Does that file gets deleted in process? Or may be there was a creation issue?

The steps I used to create data files were [in /ds2/datafiles/]:

1. cd cust; gcc -o ds2_create_cust ds2_create_cust.c -lm ; sh ds2_create_cust_large.sh

2. cd orders; gcc -o ds2_create_orders ds2_create_orders.c -lm ; sh ds2_create_orders_large.sh

3. cd orders; gcc -o ds2_create_inv ds2_create_inv.c -lm; ./ds2_create_inv 1000000 > ../prod/inv.csv

4. cd prod; gcc -o ds2_create_prod ds2_create_prod.c -lm ; ./ds2_create_prod 1000000 > prod.csv

And then to create the the database:

1. cd mysqlds2/ ; sh mysqlds2_create_all.sh

Is there something wrong with the steps?

Thx
-sk

No, the inv.csv should not get deleted. That may be your problem.
I haven't used the MySQL version in several years but the large SQL and Oracle databases are loading in a few hours on new systems (dual quadcore procs).

Dave

Retried again. But no luck .

This time I verified than the inv.csv exist
--
[root@xyz data_files]# cd orders/

[root@xyz orders]# ls
apr_cust_hist.csv ds2_create_orders jan_orders.csv may_orderlines.csv
apr_orderlines.csv ds2_create_orders.c jul_cust_hist.csv may_orders.csv
apr_orders.csv ds2_create_orders_large.sh jul_orderlines.csv nov_cust_hist.csv
aug_cust_hist.csv ds2_create_orders_med.sh jul_orders.csv nov_orderlines.csv
aug_orderlines.csv ds2_create_orders_readme.txt jun_cust_hist.csv nov_orders.csv
aug_orders.csv ds2_create_orders_small.sh jun_orderlines.csv oct_cust_hist.csv
dec_cust_hist.csv feb_cust_hist.csv jun_orders.csv oct_orderlines.csv
dec_orderlines.csv feb_orderlines.csv mar_cust_hist.csv oct_orders.csv
dec_orders.csv feb_orders.csv mar_orderlines.csv sep_cust_hist.csv
ds2_create_inv jan_cust_hist.csv mar_orders.csv sep_orderlines.csv
ds2_create_inv.c jan_orderlines.csv may_cust_hist.csv sep_orders.csv

[root@XYZ orders]# cd ../prod/

[root@XYZ prod]# ls
ds2_create_prod ds2_create_prod.c ds2_create_prod_readme.txt ds2_data.h inv.csv prod.csv

[root@XYZ prod]# cd ../cust/

[root@XYZ cust]# ls
ds2_create_cust ds2_create_cust_large.sh ds2_create_cust_readme.txt row_cust.csv
ds2_create_cust.c ds2_create_cust_med.sh ds2_create_cust_small.sh us_cust.csv

[root@XYZ cust]#
---
"sh mysqlds2_create_all.sh" just keeps on running for over 7/8 hours ....after which I killed it

-sk

Try the steps in mysqlds2_create_all.sh individually to see where it hangs.