Users - StoreAll crash on big import files ?

[oceatoon]

Hi everyone
we have managed to import dynamically small xml files coming from a CSV then transformed to a valid entity engine XML format. That worked great. We then tested with big files and unfortunetly it crashed on us ? has anobody noticed this ?
The crash seems to be on the StoreAll.
is there a limit to the size of files that can be imported ?


regards
Tibor

 
_______________________________________________
Users mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/users

[Jacopo Cappellato]

maybe the transaction timeout is too short... are you importing the data
from Webtools? does the crash happens after a fixed amount of time (i.e.
two minutes)?

Jacopo

tibor katelbach wrote:

> Hi everyone
> we have managed to import dynamically small xml files coming from a CSV
> then transformed to a valid entity engine XML format. That worked great.
> We then tested with big files and unfortunetly it crashed on us ? has
> anobody noticed this ?
> The crash seems to be on the StoreAll.
> is there a limit to the size of files that can be imported ?
>
>
> regards
> Tibor
>
>
> ------------------------------------------------------------------------
>
>  
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.ofbiz.org/mailman/listinfo/users

 
_______________________________________________
Users mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/users

[BJ Freeman]

not sure what version you have, or if they have changed the import since
5,000 reves, but the import is a stream, therefore memory.
you can try increasing the memory, or cutting up the import.


tibor katelbach sent the following on 3/30/06 5:48 AM:

> Hi everyone
> we have managed to import dynamically small xml files coming from a CSV then
> transformed to a valid entity engine XML format. That worked great. We then
> tested with big files and unfortunetly it crashed on us ? has anobody
> noticed this ?
> The crash seems to be on the StoreAll.
> is there a limit to the size of files that can be imported ?
>
>
> regards
> Tibor
>
>
>
> ------------------------------------------------------------------------
>
>  
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.ofbiz.org/mailman/listinfo/users

 
_______________________________________________
Users mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/users

[Fred Forester-2]

I do it like this

java -jar ofbiz.jar -install -timeout=21600 -file=/full/path/to/file

I have used this to import a files as large as 496766076


BJ Freeman wrote:

not sure what version you have, or if they have changed the import since 
5,000 reves, but the import is a stream, therefore memory.
you can try increasing the memory, or cutting up the import.


tibor katelbach sent the following on 3/30/06 5:48 AM:
  

Hi everyone
we have managed to import dynamically small xml files coming from a CSV then
transformed to a valid entity engine XML format. That worked great. We then
tested with big files and unfortunetly it crashed on us ? has anobody
noticed this ?
The crash seems to be on the StoreAll.
is there a limit to the size of files that can be imported ?


regards
Tibor



------------------------------------------------------------------------

 
_______________________________________________
Users mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/users
    

 
_______________________________________________
Users mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/users


  

 
_______________________________________________
Users mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/users

[oceatoon]

thanks for the quick answer

we are not using directly the webtools but the same methods as in the import api.
we are on Sequoia 0.8,
it's quite hard to say if the time before failure is constant.
BJ how would you increase the streams memory ? this seems be a good solution.
is there a limit to this stream  memory size, cause ultematly we will have an xml file with 100000 insert tags. Right now we are still testing with 10000 ?

offfcourse the cut up will be our last way to go.
Thanks a lot

On 3/30/06, BJ Freeman <[hidden email]> wrote:

not sure what version you have, or if they have changed the import since
5,000 reves, but the import is a stream, therefore memory.
you can try increasing the memory, or cutting up the import.


tibor katelbach sent the following on 3/30/06 5:48 AM:

> Hi everyone
> we have managed to import dynamically small xml files coming from a CSV then
> transformed to a valid entity engine XML format. That worked great. We then
> tested with big files and unfortunetly it crashed on us ? has anobody
> noticed this ?
> The crash seems to be on the StoreAll.
> is there a limit to the size of files that can be imported ?
>
>
> regards
> Tibor
>
>
>
> ------------------------------------------------------------------------
>
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.ofbiz.org/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/users

 
_______________________________________________
Users mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/users

[oceatoon]

interesting Fred
but you have to stop the server for this , would it bve possible to have a similar hot plugged solution ?

 
_______________________________________________
Users mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/users

[Jacopo Cappellato]

Hi Tibor,

I don't think that the problem is the size of your data (100000 are not
so many...) so my guess is that there are some problem in the data itself...

Jacopo


tibor katelbach wrote:

> thanks for the quick answer
>
> we are not using directly the webtools but the same methods as in the
> import api.
> we are on Sequoia 0.8,
> it's quite hard to say if the time before failure is constant.
> BJ how would you increase the streams memory ? this seems be a good
> solution.
> is there a limit to this stream  memory size, cause ultematly we will
> have an xml file with 100000 insert tags. Right now we are still testing
> with 10000 ?
>
> offfcourse the cut up will be our last way to go.
> Thanks a lot
>
>
> On 3/30/06, *BJ Freeman* <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     not sure what version you have, or if they have changed the import since
>     5,000 reves, but the import is a stream, therefore memory.
>     you can try increasing the memory, or cutting up the import.
>
>
>     tibor katelbach sent the following on 3/30/06 5:48 AM:
>      > Hi everyone
>      > we have managed to import dynamically small xml files coming from
>     a CSV then
>      > transformed to a valid entity engine XML format. That worked
>     great. We then
>      > tested with big files and unfortunetly it crashed on us ? has
>     anobody
>      > noticed this ?
>      > The crash seems to be on the StoreAll.
>      > is there a limit to the size of files that can be imported ?
>      >
>      >
>      > regards
>      > Tibor
>      >
>      >
>      >
>      >
>     ------------------------------------------------------------------------
>
>      >
>      >
>      > _______________________________________________
>      > Users mailing list
>      > [hidden email] <mailto:[hidden email]>
>      > http://lists.ofbiz.org/mailman/listinfo/users
>
>     _______________________________________________
>     Users mailing list
>     [hidden email] <mailto:[hidden email]>
>     http://lists.ofbiz.org/mailman/listinfo/users
>
>
>
> ------------------------------------------------------------------------
>
>  
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.ofbiz.org/mailman/listinfo/users

 
_______________________________________________
Users mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/users

[Fred Forester-2]

I seem to remember having to stop the server if using derby but not if using postgres.


tibor katelbach wrote:

interesting Fred
but you have to stop the server for this , would it bve possible to have a similar hot plugged solution ?

 
_______________________________________________
Users mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/users

 
_______________________________________________
Users mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/users

[oceatoon]

ok thx for all the good info guys.
we'll have to do some testing on the generated xml content.
will send in the results
Best Regards
Tibor

On 3/30/06, Fred Forester <[hidden email]> wrote:

I seem to remember having to stop the server if using derby but not if using postgres.


tibor katelbach wrote:

interesting Fred
but you have to stop the server for this , would it bve possible to have a similar hot plugged solution ?

_______________________________________________ Users mailing list [hidden email] <a href="http://lists.ofbiz.org/mailman/listinfo/users" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://lists.ofbiz.org/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
<a onclick="return top.js.OpenExtLink(window,event,this)" href="http://lists.ofbiz.org/mailman/listinfo/users" target="_blank"> http://lists.ofbiz.org/mailman/listinfo/users

 
_______________________________________________
Users mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/users

[oceatoon]

Btw Fred
can you make your call from inside a java file ?

On 3/30/06, tibor katelbach <[hidden email]> wrote:

ok thx for all the good info guys.
we'll have to do some testing on the generated xml content.
will send in the results
Best Regards

Tibor

On 3/30/06, Fred Forester <[hidden email]> wrote:

I seem to remember having to stop the server if using derby but not if using postgres.


tibor katelbach wrote:

interesting Fred
but you have to stop the server for this , would it bve possible to have a similar hot plugged solution ?

_______________________________________________ Users mailing list [hidden email] <a href="http://lists.ofbiz.org/mailman/listinfo/users" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://lists.ofbiz.org/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
<a href="http://lists.ofbiz.org/mailman/listinfo/users" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> http://lists.ofbiz.org/mailman/listinfo/users

 
_______________________________________________
Users mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/users

[Fred Forester-2]

I have seperate programs that generate the xml and just use the ofbiz jar in batch. (this was a one shot import scenario) however, you should be able to drive
ModelDataFileReader directly.

tibor katelbach wrote:

Btw Fred
can you make your call from inside a java file ?

On 3/30/06, tibor katelbach <[hidden email]> wrote:

ok thx for all the good info guys.
we'll have to do some testing on the generated xml content.
will send in the results
Best Regards

Tibor

On 3/30/06, Fred Forester <[hidden email]> wrote:

I seem to remember having to stop the server if using derby but not if using postgres.


tibor katelbach wrote:

interesting Fred
but you have to stop the server for this , would it bve possible to have a similar hot plugged solution ?

_______________________________________________ Users mailing list [hidden email] <a href="http://lists.ofbiz.org/mailman/listinfo/users" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://lists.ofbiz.org/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
<a href="http://lists.ofbiz.org/mailman/listinfo/users" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://lists.ofbiz.org/mailman/listinfo/users

 
_______________________________________________
Users mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/users

 
_______________________________________________
Users mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/users

[oceatoon]

ok thx
we have that we were using the old methods(simpler) and that our generated xml work fine in the import webtools.
so we'll be refactoring.
Thx a  lot
Tibor


 
_______________________________________________
Users mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/users