faster data migration

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

faster data migration

greg jocher
Hi all

We are working on a data migration to migrate multiple stores.
Each store uses all or a part of the ~150 virtual products.
Each virtual has between 10 and 1k variants (combination of 2 to 6
features...).
This migration has to be done in less than 5 hours.

What do you think is the fastest way to migrate these products and its
300k customers? pure sql-scripts? postgres datafiles? ofbiz-xml-data?

Btw: The production installation will not be inhouse - we currently use
xmlrpc for the migration, but it is just to slow:
I have written the product part and it is running  since more than 8
hours. DB, Ofbiz, Migration all on the same dev-machine (16G mem, amd
phenom x6, SSD, Postgres on Raid10) - yes, xmlrpc is really to slow for
bulk migration..

Regards
Greg
Reply | Threaded
Open this post in threaded view
|

Re: faster data migration

Malin Nicolas
Hi,

Prepare ofbiz xml-data with ofbiz system or ftl or xslt or other etl.
Desactivate your eeca on spot server and load your files.

Nicolas

Le 08/08/2012 10:52, greg jocher a écrit :

> Hi all
>
> We are working on a data migration to migrate multiple stores.
> Each store uses all or a part of the ~150 virtual products.
> Each virtual has between 10 and 1k variants (combination of 2 to 6
> features...).
> This migration has to be done in less than 5 hours.
>
> What do you think is the fastest way to migrate these products and its
> 300k customers? pure sql-scripts? postgres datafiles? ofbiz-xml-data?
>
> Btw: The production installation will not be inhouse - we currently
> use xmlrpc for the migration, but it is just to slow:
> I have written the product part and it is running  since more than 8
> hours. DB, Ofbiz, Migration all on the same dev-machine (16G mem, amd
> phenom x6, SSD, Postgres on Raid10) - yes, xmlrpc is really to slow
> for bulk migration..
>
> Regards
> Greg


--
Nicolas MALIN
Consultant
Tél : 06.17.66.40.06
Site projet : http://www.neogia.org/
-------
Société LibrenBerry
Tél : 02.48.02.56.12
Site : http://www.librenberry.net/

Reply | Threaded
Open this post in threaded view
|

Re: faster data migration

greg jocher
Hi,

Thanks for your input - i have done some tests now:

1. I have created all products, features and variants through xmlrpc (3M
records, 12h..)
2. Exported ofbiz-xml-data
3. pg_dumped
4. imported xml-data on a clean install: ~40m
5. imported pg_dump of the same data: ~7m

I think we will choose ofbiz-xml-data, its fast enough!

Regards
Greg

On 08/08/2012 01:45 PM, Nicolas Malin wrote:

> Hi,
>
> Prepare ofbiz xml-data with ofbiz system or ftl or xslt or other etl.
> Desactivate your eeca on spot server and load your files.
>
> Nicolas
>
> Le 08/08/2012 10:52, greg jocher a écrit :
>> Hi all
>>
>> We are working on a data migration to migrate multiple stores.
>> Each store uses all or a part of the ~150 virtual products.
>> Each virtual has between 10 and 1k variants (combination of 2 to 6
>> features...).
>> This migration has to be done in less than 5 hours.
>>
>> What do you think is the fastest way to migrate these products and
>> its 300k customers? pure sql-scripts? postgres datafiles?
>> ofbiz-xml-data?
>>
>> Btw: The production installation will not be inhouse - we currently
>> use xmlrpc for the migration, but it is just to slow:
>> I have written the product part and it is running  since more than 8
>> hours. DB, Ofbiz, Migration all on the same dev-machine (16G mem, amd
>> phenom x6, SSD, Postgres on Raid10) - yes, xmlrpc is really to slow
>> for bulk migration..
>>
>> Regards
>> Greg
>
>