Re: [OFBiz] Users - Exporting large tables with webtools

Posted by Brett on Jul 07, 2006; 8:05pm
URL: http://ofbiz.116.s1.nabble.com/OFBiz-Users-Exporting-large-tables-with-webtools-tp140549p140555.html

I'll dive in a little deeper and see where the error is coming from.
I did notice that the behavior seems to be different if I select a
single table for export versus all the tables.  Maybe they are calling
different services.

Brett

On 7/7/06, David E. Jones <[hidden email]> wrote:

>
> It's been a while since I've done any major work with these and I know some changes have been made since then, but it should not be reading everything into memory nor should it be creating the output XML in memory. The code that I wrote way back when used the EntityListIterator to pull from the database, and streamed small chunks of text for each entity and field out to a file as it iterates over the ELI.
>
> If it's not doing that then some investigation would be necessary to find out why...
>
> -David
>
>
> Brett Palmer wrote:
> > Has anyone used the webtools export utilities to export tables with
> > millions of rows?  I have used it on smaller tables, but when I try to
> > export on a large table I get an out of memory error.  I believe the
> > webtools utility tries to read the entire table in memory before
> > writing it to the file.
> >
> > We need this functionality to do a major migration from Oracle to
> > postgres.  If the functionality doesn't exist we will implement it
> > ourself and contribute it back if anyone is interested.  If you have
> > any suggestions please post them to the group.
> >
> > Thanks,
> >
> >
> > Brett
> >
> > _______________________________________________
> > Users mailing list
> > [hidden email]
> > http://lists.ofbiz.org/mailman/listinfo/users
>
>
>