Re: /framework/datafile caching.

Posted by David E. Jones-2 on Mar 16, 2010; 2:51am
URL: http://ofbiz.116.s1.nabble.com/framework-datafile-caching-tp1594248p1594371.html


On Mar 15, 2010, at 8:47 PM, Adam Heath wrote:

> David E Jones wrote:
>> On Mar 15, 2010, at 6:58 PM, Adam Heath wrote:
>>
>>> BJ Freeman wrote:
>>>> one of my smaller import files (8mb) is taking forever to be read in and
>>>> there is no output.
>>>> I am seeing the memory rail against the max setting.
>>>> got this error.
>>> The datafile set of classes is very broken for large files.  It has a
>>> List<Record>, which means it will copy the entire file into memory
>>> before doing anything with it.  The datafile code is not designed to
>>> handle large files.
>>>
>>> I consider this a bug that needs to be fixed.  Checking...
>>
>> This should be possible, but may require API changes. The problem is that, like XML, data files can be hierarchical and a "node" can have header and footer lines in the file.
>>
>> I wouldn't consider this a bug, just like XML DOM parsing is not a "bug". Of course, you're certainly entitled to your opinion.
>
> Well, then, what would you suggest be done in this situation?  Saying
> "don't do that" is not acceptable, as BJ has a real-world situation
> he is trying to solve.
>
> The datafile subsystem is close to providing this feature.  It has a
> RecordIterator class, which is used by DataFile itself; modifying the
> calling code to use the former should be possible.

Yep, exactly, it's a "feature" as you say here. As I said it could certainly be implemented, but might require some API changes.

-David