/framework/datafile caching.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

/framework/datafile caching.

BJ Freeman
one of my smaller import files (8mb) is taking forever to be read in and
there is no output.
I am seeing the memory rail against the max setting.
got this error.

javax.servlet.ServletException: Servlet execution threw an exception
org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:259)

root cause
java.lang.OutOfMemoryError: Java heap space
        java.util.Arrays.copyOfRange(Unknown Source)
        java.lang.String.<init>(Unknown Source)
        java.lang.StringBuffer.toString(Unknown Source)
        java.io.StringWriter.toString(Unknown Source)
        org.apache.log4j.spi.LocationInfo.<init>(LocationInfo.java:114)
org.apache.log4j.spi.LoggingEvent.getLocationInformation(LoggingEvent.java:247)
        org.apache.log4j.AsyncAppender.append(AsyncAppender.java:160)
org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
        org.apache.log4j.Category.callAppenders(Category.java:206)
        org.apache.log4j.Category.forcedLog(Category.java:391)
        org.apache.log4j.Category.log(Category.java:856)
        org.ofbiz.base.util.Debug.log(Debug.java:166)
        org.ofbiz.base.util.Debug.log(Debug.java:149)
        org.ofbiz.base.util.Debug.logInfo(Debug.java:258)
        org.ofbiz.datafile.RecordIterator.getNextLine(RecordIterator.java:117)
        org.ofbiz.datafile.RecordIterator.next(RecordIterator.java:163)
        org.ofbiz.datafile.DataFile.readDataFile(DataFile.java:126)
        org.ofbiz.datafile.DataFile.readFile(DataFile.java:63)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        java.lang.reflect.Method.invoke(Unknown Source)
        org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:86)
        groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:230)
        groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1105)
        org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:749)
        org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:170)
        viewdatafile.run(viewdatafile.groovy:65)
        org.ofbiz.base.util.GroovyUtil.runScriptAtLocation(GroovyUtil.java:117)
        org.ofbiz.widget.screen.ModelScreenAction$Script.runAction(ModelScreenAction.java:408)
org.ofbiz.widget.screen.ModelScreenAction.runSubActions(ModelScreenAction.java:118)


some of my files are 331 megs or larger.
they are imported from a URL, so there is no chance to chop them up.

My suggested solution is redesign to read/write and not store the
records. once a record is created it is then parse and removed from memory.


does anyone have another solution?


=========================
BJ Freeman
http://bjfreeman.elance.com
Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
Specialtymarket.com <http://www.specialtymarket.com/>

Systems Integrator-- Glad to Assist

Chat  Y! messenger: bjfr33man
Linkedin
<http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>



Reply | Threaded
Open this post in threaded view
|

Re: /framework/datafile caching.

Adam Heath-2
BJ Freeman wrote:
> one of my smaller import files (8mb) is taking forever to be read in and
> there is no output.
> I am seeing the memory rail against the max setting.
> got this error.

The datafile set of classes is very broken for large files.  It has a
List<Record>, which means it will copy the entire file into memory
before doing anything with it.  The datafile code is not designed to
handle large files.

I consider this a bug that needs to be fixed.  Checking...
Reply | Threaded
Open this post in threaded view
|

Re: /framework/datafile caching.

David E. Jones-2

On Mar 15, 2010, at 6:58 PM, Adam Heath wrote:

> BJ Freeman wrote:
>> one of my smaller import files (8mb) is taking forever to be read in and
>> there is no output.
>> I am seeing the memory rail against the max setting.
>> got this error.
>
> The datafile set of classes is very broken for large files.  It has a
> List<Record>, which means it will copy the entire file into memory
> before doing anything with it.  The datafile code is not designed to
> handle large files.
>
> I consider this a bug that needs to be fixed.  Checking...

This should be possible, but may require API changes. The problem is that, like XML, data files can be hierarchical and a "node" can have header and footer lines in the file.

I wouldn't consider this a bug, just like XML DOM parsing is not a "bug". Of course, you're certainly entitled to your opinion.

-David

Reply | Threaded
Open this post in threaded view
|

Re: /framework/datafile caching.

BJ Freeman
In reply to this post by BJ Freeman

yes thanks for verifying.
are you planning to fix?
appreciate pointers and will fix if you are not.

=========================
BJ Freeman
http://bjfreeman.elance.com
Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
Specialtymarket.com <http://www.specialtymarket.com/>

Systems Integrator-- Glad to Assist

Chat  Y! messenger: bjfr33man
Linkedin
<http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>


Adam Heath sent the following on 3/15/2010 5:58 PM:

> BJ Freeman wrote:
>> one of my smaller import files (8mb) is taking forever to be read in and
>> there is no output.
>> I am seeing the memory rail against the max setting.
>> got this error.
>
> The datafile set of classes is very broken for large files.  It has a
> List<Record>, which means it will copy the entire file into memory
> before doing anything with it.  The datafile code is not designed to
> handle large files.
>
> I consider this a bug that needs to be fixed.  Checking...
>


Reply | Threaded
Open this post in threaded view
|

Re: /framework/datafile caching.

BJ Freeman
In reply to this post by Adam Heath-2
when code will not handle real world data, it is broken.
you and I discussed this when importing xml file and I choose to parse
them manually instead of using DOM.
So what term is there besides bug for broken design.

=========================
BJ Freeman
http://bjfreeman.elance.com
Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
Specialtymarket.com <http://www.specialtymarket.com/>

Systems Integrator-- Glad to Assist

Chat  Y! messenger: bjfr33man
Linkedin
<http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>


David E Jones sent the following on 3/15/2010 6:05 PM:

> On Mar 15, 2010, at 6:58 PM, Adam Heath wrote:
>
>> BJ Freeman wrote:
>>> one of my smaller import files (8mb) is taking forever to be read in and
>>> there is no output.
>>> I am seeing the memory rail against the max setting.
>>> got this error.
>> The datafile set of classes is very broken for large files.  It has a
>> List<Record>, which means it will copy the entire file into memory
>> before doing anything with it.  The datafile code is not designed to
>> handle large files.
>>
>> I consider this a bug that needs to be fixed.  Checking...
>
> This should be possible, but may require API changes. The problem is that, like XML, data files can be hierarchical and a "node" can have header and footer lines in the file.
>
> I wouldn't consider this a bug, just like XML DOM parsing is not a "bug". Of course, you're certainly entitled to your opinion.
>
> -David
>
>


Reply | Threaded
Open this post in threaded view
|

Re: /framework/datafile caching.

David E. Jones-2

Sorry, that's silly. A bug can't exist without a requirement. Where was this requirement ever established? What in the design implies that this was a requirement? The API very clearly represents a process that reads the entire file into memory.

It sounds like it doesn't meet a requirement that you came up with. That isn't the definition of a "bug", well except for the purposes of trolling on mailing lists I suppose.

-David


On Mar 15, 2010, at 7:16 PM, BJ Freeman wrote:

> when code will not handle real world data, it is broken.
> you and I discussed this when importing xml file and I choose to parse
> them manually instead of using DOM.
> So what term is there besides bug for broken design.
>
> =========================
> BJ Freeman
> http://bjfreeman.elance.com
> Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
> Specialtymarket.com <http://www.specialtymarket.com/>
>
> Systems Integrator-- Glad to Assist
>
> Chat  Y! messenger: bjfr33man
> Linkedin
> <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>
>
>
> David E Jones sent the following on 3/15/2010 6:05 PM:
>> On Mar 15, 2010, at 6:58 PM, Adam Heath wrote:
>>
>>> BJ Freeman wrote:
>>>> one of my smaller import files (8mb) is taking forever to be read in and
>>>> there is no output.
>>>> I am seeing the memory rail against the max setting.
>>>> got this error.
>>> The datafile set of classes is very broken for large files.  It has a
>>> List<Record>, which means it will copy the entire file into memory
>>> before doing anything with it.  The datafile code is not designed to
>>> handle large files.
>>>
>>> I consider this a bug that needs to be fixed.  Checking...
>>
>> This should be possible, but may require API changes. The problem is that, like XML, data files can be hierarchical and a "node" can have header and footer lines in the file.
>>
>> I wouldn't consider this a bug, just like XML DOM parsing is not a "bug". Of course, you're certainly entitled to your opinion.
>>
>> -David
>>
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: /framework/datafile caching.

Adrian Crum-2
In reply to this post by BJ Freeman
--- On Mon, 3/15/10, BJ Freeman <[hidden email]> wrote:
> when code will not handle real world
> data, it is broken.
> you and I discussed this when importing xml file and I
> choose to parse
> them manually instead of using DOM.
> So what term is there besides bug for broken design.

"Feature"

;)



     
Reply | Threaded
Open this post in threaded view
|

Re: /framework/datafile caching.

BJ Freeman
In reply to this post by Adam Heath-2

here is my solution feel free to comment.

check the files size if in acceptable limits (TBD)
use the code as is.
if above the limits send records to a datafile on disk, instead of the list.
then add a new routine in the datafile2entityxml.java that reads that
disk file to create the xml.


========================

BJ Freeman
http://bjfreeman.elance.com
Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
Specialtymarket.com <http://www.specialtymarket.com/>

Systems Integrator-- Glad to Assist

Chat  Y! messenger: bjfr33man
Linkedin
<http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>


BJ Freeman sent the following on 3/15/2010 6:16 PM:

> when code will not handle real world data, it is broken.
> you and I discussed this when importing xml file and I choose to parse
> them manually instead of using DOM.
> So what term is there besides bug for broken design.
>
> =========================
> BJ Freeman
> http://bjfreeman.elance.com
> Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
> Specialtymarket.com <http://www.specialtymarket.com/>
>
> Systems Integrator-- Glad to Assist
>
> Chat  Y! messenger: bjfr33man
> Linkedin
> <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>
>
>
> David E Jones sent the following on 3/15/2010 6:05 PM:
>> On Mar 15, 2010, at 6:58 PM, Adam Heath wrote:
>>
>>> BJ Freeman wrote:
>>>> one of my smaller import files (8mb) is taking forever to be read in and
>>>> there is no output.
>>>> I am seeing the memory rail against the max setting.
>>>> got this error.
>>> The datafile set of classes is very broken for large files.  It has a
>>> List<Record>, which means it will copy the entire file into memory
>>> before doing anything with it.  The datafile code is not designed to
>>> handle large files.
>>>
>>> I consider this a bug that needs to be fixed.  Checking...
>> This should be possible, but may require API changes. The problem is that, like XML, data files can be hierarchical and a "node" can have header and footer lines in the file.
>>
>> I wouldn't consider this a bug, just like XML DOM parsing is not a "bug". Of course, you're certainly entitled to your opinion.
>>
>> -David
>>
>>
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: /framework/datafile caching.

Adam Heath-2
In reply to this post by David E. Jones-2
David E Jones wrote:

> On Mar 15, 2010, at 6:58 PM, Adam Heath wrote:
>
>> BJ Freeman wrote:
>>> one of my smaller import files (8mb) is taking forever to be read in and
>>> there is no output.
>>> I am seeing the memory rail against the max setting.
>>> got this error.
>> The datafile set of classes is very broken for large files.  It has a
>> List<Record>, which means it will copy the entire file into memory
>> before doing anything with it.  The datafile code is not designed to
>> handle large files.
>>
>> I consider this a bug that needs to be fixed.  Checking...
>
> This should be possible, but may require API changes. The problem is that, like XML, data files can be hierarchical and a "node" can have header and footer lines in the file.
>
> I wouldn't consider this a bug, just like XML DOM parsing is not a "bug". Of course, you're certainly entitled to your opinion.

Well, then, what would you suggest be done in this situation?  Saying
 "don't do that" is not acceptable, as BJ has a real-world situation
he is trying to solve.

The datafile subsystem is close to providing this feature.  It has a
RecordIterator class, which is used by DataFile itself; modifying the
calling code to use the former should be possible.
Reply | Threaded
Open this post in threaded view
|

Re: /framework/datafile caching.

David E. Jones-2

On Mar 15, 2010, at 8:47 PM, Adam Heath wrote:

> David E Jones wrote:
>> On Mar 15, 2010, at 6:58 PM, Adam Heath wrote:
>>
>>> BJ Freeman wrote:
>>>> one of my smaller import files (8mb) is taking forever to be read in and
>>>> there is no output.
>>>> I am seeing the memory rail against the max setting.
>>>> got this error.
>>> The datafile set of classes is very broken for large files.  It has a
>>> List<Record>, which means it will copy the entire file into memory
>>> before doing anything with it.  The datafile code is not designed to
>>> handle large files.
>>>
>>> I consider this a bug that needs to be fixed.  Checking...
>>
>> This should be possible, but may require API changes. The problem is that, like XML, data files can be hierarchical and a "node" can have header and footer lines in the file.
>>
>> I wouldn't consider this a bug, just like XML DOM parsing is not a "bug". Of course, you're certainly entitled to your opinion.
>
> Well, then, what would you suggest be done in this situation?  Saying
> "don't do that" is not acceptable, as BJ has a real-world situation
> he is trying to solve.
>
> The datafile subsystem is close to providing this feature.  It has a
> RecordIterator class, which is used by DataFile itself; modifying the
> calling code to use the former should be possible.

Yep, exactly, it's a "feature" as you say here. As I said it could certainly be implemented, but might require some API changes.

-David

Reply | Threaded
Open this post in threaded view
|

Re: /framework/datafile caching.

BJ Freeman
In reply to this post by BJ Freeman
that reminds me of the wolf and the three pigs.
if you build something that has unrealistic requirements, then it will
not function in the real world.
the person making the requirements should be aware of what they are and
the limits or usage.
but to build something just to say you built is not what I call good design.

Just to note, while I was documenting my way of doing things It was
suggested I look at the datafile way.

I have and have attempted to use it in good faith.
I have realworld files that I handled long before coming to ofbiz that
are gigs in size, so the requirements have  been around since the early 90's


======================

BJ Freeman
http://bjfreeman.elance.com
Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
Specialtymarket.com <http://www.specialtymarket.com/>

Systems Integrator-- Glad to Assist

Chat  Y! messenger: bjfr33man
Linkedin
<http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>


David E Jones sent the following on 3/15/2010 6:48 PM:

> Sorry, that's silly. A bug can't exist without a requirement. Where was this requirement ever established? What in the design implies that this was a requirement? The API very clearly represents a process that reads the entire file into memory.
>
> It sounds like it doesn't meet a requirement that you came up with. That isn't the definition of a "bug", well except for the purposes of trolling on mailing lists I suppose.
>
> -David
>
>
> On Mar 15, 2010, at 7:16 PM, BJ Freeman wrote:
>
>> when code will not handle real world data, it is broken.
>> you and I discussed this when importing xml file and I choose to parse
>> them manually instead of using DOM.
>> So what term is there besides bug for broken design.
>>
>> =========================
>> BJ Freeman
>> http://bjfreeman.elance.com
>> Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
>> Specialtymarket.com <http://www.specialtymarket.com/>
>>
>> Systems Integrator-- Glad to Assist
>>
>> Chat  Y! messenger: bjfr33man
>> Linkedin
>> <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>
>>
>>
>> David E Jones sent the following on 3/15/2010 6:05 PM:
>>> On Mar 15, 2010, at 6:58 PM, Adam Heath wrote:
>>>
>>>> BJ Freeman wrote:
>>>>> one of my smaller import files (8mb) is taking forever to be read in and
>>>>> there is no output.
>>>>> I am seeing the memory rail against the max setting.
>>>>> got this error.
>>>> The datafile set of classes is very broken for large files.  It has a
>>>> List<Record>, which means it will copy the entire file into memory
>>>> before doing anything with it.  The datafile code is not designed to
>>>> handle large files.
>>>>
>>>> I consider this a bug that needs to be fixed.  Checking...
>>> This should be possible, but may require API changes. The problem is that, like XML, data files can be hierarchical and a "node" can have header and footer lines in the file.
>>>
>>> I wouldn't consider this a bug, just like XML DOM parsing is not a "bug". Of course, you're certainly entitled to your opinion.
>>>
>>> -David
>>>
>>>
>>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: /framework/datafile caching.

David E. Jones-2

I'm sorry, was there a point in there somewhere?

-David


On Mar 15, 2010, at 9:00 PM, BJ Freeman wrote:

> that reminds me of the wolf and the three pigs.
> if you build something that has unrealistic requirements, then it will
> not function in the real world.
> the person making the requirements should be aware of what they are and
> the limits or usage.
> but to build something just to say you built is not what I call good design.
>
> Just to note, while I was documenting my way of doing things It was
> suggested I look at the datafile way.
>
> I have and have attempted to use it in good faith.
> I have realworld files that I handled long before coming to ofbiz that
> are gigs in size, so the requirements have  been around since the early 90's
>
>
> ======================
>
> BJ Freeman
> http://bjfreeman.elance.com
> Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
> Specialtymarket.com <http://www.specialtymarket.com/>
>
> Systems Integrator-- Glad to Assist
>
> Chat  Y! messenger: bjfr33man
> Linkedin
> <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>
>
>
> David E Jones sent the following on 3/15/2010 6:48 PM:
>> Sorry, that's silly. A bug can't exist without a requirement. Where was this requirement ever established? What in the design implies that this was a requirement? The API very clearly represents a process that reads the entire file into memory.
>>
>> It sounds like it doesn't meet a requirement that you came up with. That isn't the definition of a "bug", well except for the purposes of trolling on mailing lists I suppose.
>>
>> -David
>>
>>
>> On Mar 15, 2010, at 7:16 PM, BJ Freeman wrote:
>>
>>> when code will not handle real world data, it is broken.
>>> you and I discussed this when importing xml file and I choose to parse
>>> them manually instead of using DOM.
>>> So what term is there besides bug for broken design.
>>>
>>> =========================
>>> BJ Freeman
>>> http://bjfreeman.elance.com
>>> Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
>>> Specialtymarket.com <http://www.specialtymarket.com/>
>>>
>>> Systems Integrator-- Glad to Assist
>>>
>>> Chat  Y! messenger: bjfr33man
>>> Linkedin
>>> <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>
>>>
>>>
>>> David E Jones sent the following on 3/15/2010 6:05 PM:
>>>> On Mar 15, 2010, at 6:58 PM, Adam Heath wrote:
>>>>
>>>>> BJ Freeman wrote:
>>>>>> one of my smaller import files (8mb) is taking forever to be read in and
>>>>>> there is no output.
>>>>>> I am seeing the memory rail against the max setting.
>>>>>> got this error.
>>>>> The datafile set of classes is very broken for large files.  It has a
>>>>> List<Record>, which means it will copy the entire file into memory
>>>>> before doing anything with it.  The datafile code is not designed to
>>>>> handle large files.
>>>>>
>>>>> I consider this a bug that needs to be fixed.  Checking...
>>>> This should be possible, but may require API changes. The problem is that, like XML, data files can be hierarchical and a "node" can have header and footer lines in the file.
>>>>
>>>> I wouldn't consider this a bug, just like XML DOM parsing is not a "bug". Of course, you're certainly entitled to your opinion.
>>>>
>>>> -David
>>>>
>>>>
>>>
>>
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: /framework/datafile caching.

BJ Freeman
In reply to this post by BJ Freeman
yup

=========================
BJ Freeman
http://bjfreeman.elance.com
Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
Specialtymarket.com <http://www.specialtymarket.com/>

Systems Integrator-- Glad to Assist

Chat  Y! messenger: bjfr33man
Linkedin
<http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>


David E Jones sent the following on 3/15/2010 8:06 PM:

> I'm sorry, was there a point in there somewhere?
>
> -David
>
>
> On Mar 15, 2010, at 9:00 PM, BJ Freeman wrote:
>
>> that reminds me of the wolf and the three pigs.
>> if you build something that has unrealistic requirements, then it will
>> not function in the real world.
>> the person making the requirements should be aware of what they are and
>> the limits or usage.
>> but to build something just to say you built is not what I call good design.
>>
>> Just to note, while I was documenting my way of doing things It was
>> suggested I look at the datafile way.
>>
>> I have and have attempted to use it in good faith.
>> I have realworld files that I handled long before coming to ofbiz that
>> are gigs in size, so the requirements have  been around since the early 90's
>>
>>
>> ======================
>>
>> BJ Freeman
>> http://bjfreeman.elance.com
>> Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
>> Specialtymarket.com <http://www.specialtymarket.com/>
>>
>> Systems Integrator-- Glad to Assist
>>
>> Chat  Y! messenger: bjfr33man
>> Linkedin
>> <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>
>>
>>
>> David E Jones sent the following on 3/15/2010 6:48 PM:
>>> Sorry, that's silly. A bug can't exist without a requirement. Where was this requirement ever established? What in the design implies that this was a requirement? The API very clearly represents a process that reads the entire file into memory.
>>>
>>> It sounds like it doesn't meet a requirement that you came up with. That isn't the definition of a "bug", well except for the purposes of trolling on mailing lists I suppose.
>>>
>>> -David
>>>
>>>
>>> On Mar 15, 2010, at 7:16 PM, BJ Freeman wrote:
>>>
>>>> when code will not handle real world data, it is broken.
>>>> you and I discussed this when importing xml file and I choose to parse
>>>> them manually instead of using DOM.
>>>> So what term is there besides bug for broken design.
>>>>
>>>> =========================
>>>> BJ Freeman
>>>> http://bjfreeman.elance.com
>>>> Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
>>>> Specialtymarket.com <http://www.specialtymarket.com/>
>>>>
>>>> Systems Integrator-- Glad to Assist
>>>>
>>>> Chat  Y! messenger: bjfr33man
>>>> Linkedin
>>>> <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>
>>>>
>>>>
>>>> David E Jones sent the following on 3/15/2010 6:05 PM:
>>>>> On Mar 15, 2010, at 6:58 PM, Adam Heath wrote:
>>>>>
>>>>>> BJ Freeman wrote:
>>>>>>> one of my smaller import files (8mb) is taking forever to be read in and
>>>>>>> there is no output.
>>>>>>> I am seeing the memory rail against the max setting.
>>>>>>> got this error.
>>>>>> The datafile set of classes is very broken for large files.  It has a
>>>>>> List<Record>, which means it will copy the entire file into memory
>>>>>> before doing anything with it.  The datafile code is not designed to
>>>>>> handle large files.
>>>>>>
>>>>>> I consider this a bug that needs to be fixed.  Checking...
>>>>> This should be possible, but may require API changes. The problem is that, like XML, data files can be hierarchical and a "node" can have header and footer lines in the file.
>>>>>
>>>>> I wouldn't consider this a bug, just like XML DOM parsing is not a "bug". Of course, you're certainly entitled to your opinion.
>>>>>
>>>>> -David
>>>>>
>>>>>
>>>
>>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: /framework/datafile caching.

Adam Heath-2
BJ Freeman wrote:
> yup

(see my reply to David below too)

David didn't see the point, so you post a worthless response; helpful.
 Why couldn't you have just explained it better?

> =========================
> BJ Freeman
> http://bjfreeman.elance.com
> Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
> Specialtymarket.com <http://www.specialtymarket.com/>
>
> Systems Integrator-- Glad to Assist
>
> Chat  Y! messenger: bjfr33man
> Linkedin
> <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>
>
>
> David E Jones sent the following on 3/15/2010 8:06 PM:
>> I'm sorry, was there a point in there somewhere?

I had a hard time finding one too.

Reply | Threaded
Open this post in threaded view
|

Re: /framework/datafile caching.

BJ Freeman
In reply to this post by BJ Freeman
bottom line is my explaination don't seem to ht the mark
if you see my response for a solution, there was not comment.
here is my solution feel free to comment.

check the files size if in acceptable limits (TBD)
use the code as is.
if above the limits send records to a datafile on disk, instead of the list.
then add a new routine in the datafile2entityxml.java that reads that
disk file to create the xml.


btw it is almost completed

=========================
BJ Freeman
http://bjfreeman.elance.com
Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
Specialtymarket.com <http://www.specialtymarket.com/>

Systems Integrator-- Glad to Assist

Chat  Y! messenger: bjfr33man
Linkedin
<http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>


Adam Heath sent the following on 3/15/2010 8:33 PM:

> BJ Freeman wrote:
>> yup
>
> (see my reply to David below too)
>
> David didn't see the point, so you post a worthless response; helpful.
>  Why couldn't you have just explained it better?
>
>> =========================
>> BJ Freeman
>> http://bjfreeman.elance.com
>> Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
>> Specialtymarket.com <http://www.specialtymarket.com/>
>>
>> Systems Integrator-- Glad to Assist
>>
>> Chat  Y! messenger: bjfr33man
>> Linkedin
>> <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>
>>
>>
>> David E Jones sent the following on 3/15/2010 8:06 PM:
>>> I'm sorry, was there a point in there somewhere?
>
> I had a hard time finding one too.
>
>