Re: [OFBiz] Users - Exporting large tables with webtools

Posted by David E Jones-2 on Jul 07, 2006; 7:49pm
URL: http://ofbiz.116.s1.nabble.com/OFBiz-Users-Exporting-large-tables-with-webtools-tp140549p140550.html


It's been a while since I've done any major work with these and I know some changes have been made since then, but it should not be reading everything into memory nor should it be creating the output XML in memory. The code that I wrote way back when used the EntityListIterator to pull from the database, and streamed small chunks of text for each entity and field out to a file as it iterates over the ELI.

If it's not doing that then some investigation would be necessary to find out why...

-David


Brett Palmer wrote:

> Has anyone used the webtools export utilities to export tables with
> millions of rows?  I have used it on smaller tables, but when I try to
> export on a large table I get an out of memory error.  I believe the
> webtools utility tries to read the entire table in memory before
> writing it to the file.
>
> We need this functionality to do a major migration from Oracle to
> postgres.  If the functionality doesn't exist we will implement it
> ourself and contribute it back if anyone is interested.  If you have
> any suggestions please post them to the group.
>
> Thanks,
>
>
> Brett
>  
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.ofbiz.org/mailman/listinfo/users