Groovy call a service within a while loop (wait, notify?)...

[Robert Gan]

hey guys, I have the following question (maybe groovy basics)...:

I do read a csv file with opencsv, so I get it line by line within a while loop. Within this loop I call a simple method service to store selected data from each line in database. With that I thought not to overload my memory, because only one line of Csv should be in memory. But now I realized, that I get a transaction timeout error from my service. It also does not store anything in database (with just 100 entries it works fine, but not wiht 10.000 entries and more). The transaction is rolled back. So I believe that it tries to use just one transaction for all data? Could that maybe happen because the while loop does not wait for finishing the transaction and just push's more data to it? So if is so, how could I wait for the end of the service before taking the next line from my csv (next while loop).

my loop is like that:


...some code....

while ((nextLine = reader.readNext()) != null) {
                dispatcher.runSync("exampleService", [userLogin: userLogin, var1:var1, var2:var2]);
}
...some code....

any help I appreciate!

greetings, robert

[Deepak Agarwal-2]

In my experience there is a reason why most of the erp provide such long
tasks as jobs. Because, if it is a long task then it is better to do in
separate async jobs (else you might get a browser timeout ).
My suggestion is to make this as a service with a long timeout and run it
as a job.

On Tue, May 21, 2013 at 9:49 PM, Robert G. <[hidden email]> wrote:

> hey guys, I have the following question (maybe groovy basics)...:
>
> I do read a csv file with opencsv, so I get it line by line within a while
> loop. Within this loop I call a simple method service to store selected
> data
> from each line in database. With that I thought not to overload my memory,
> because only one line of Csv should be in memory. But now I realized, that
> I
> get a transaction timeout error from my service. It also does not store
> anything in database (with just 100 entries it works fine, but not wiht
> 10.000 entries and more). The transaction is rolled back. So I believe that
> it tries to use just one transaction for all data? Could that maybe happen
> because the while loop does not wait for finishing the transaction and just
> push's more data to it? So if is so, how could I wait for the end of the
> service before taking the next line from my csv (next while loop).
>
> my loop is like that:
>
>
> /...some code....
>
> while ((nextLine = reader.readNext()) != null) {
>                 dispatcher.runSync("exampleService", [userLogin:
> userLogin, var1:var1,
> var2:var2]);
> }
> ...some code..../
>
> any help I appreciate!
>
> greetings, robert
>
>
>
> --
> View this message in context:
> http://ofbiz.135035.n4.nabble.com/Groovy-call-a-service-within-a-while-loop-wait-notify-tp4641435.html
> Sent from the OFBiz - User mailing list archive at Nabble.com.
>

--
Thanks,
Deepak Agarwal,

Mobile: [phone]

[Adrian Crum-3]

Set use-transaction="false" in the service that contains the loop, and
make sure the service to store the CSV record uses a transaction.

-Adrian

On 5/21/2013 5:19 PM, Robert G. wrote:

> hey guys, I have the following question (maybe groovy basics)...:
>
> I do read a csv file with opencsv, so I get it line by line within a while
> loop. Within this loop I call a simple method service to store selected data
> from each line in database. With that I thought not to overload my memory,
> because only one line of Csv should be in memory. But now I realized, that I
> get a transaction timeout error from my service. It also does not store
> anything in database (with just 100 entries it works fine, but not wiht
> 10.000 entries and more). The transaction is rolled back. So I believe that
> it tries to use just one transaction for all data? Could that maybe happen
> because the while loop does not wait for finishing the transaction and just
> push's more data to it? So if is so, how could I wait for the end of the
> service before taking the next line from my csv (next while loop).
>
> my loop is like that:
>
>
> /...some code....
>
> while ((nextLine = reader.readNext()) != null) {
> dispatcher.runSync("exampleService", [userLogin: userLogin, var1:var1,
> var2:var2]);
> }
> ...some code..../
>
> any help I appreciate!
>
> greetings, robert
>
>
>
> --
> View this message in context: http://ofbiz.135035.n4.nabble.com/Groovy-call-a-service-within-a-while-loop-wait-notify-tp4641435.html
> Sent from the OFBiz - User mailing list archive at Nabble.com.

[Robert Gan]

ha, that was fast :)

I see and this sounds reasonable. But on the otherhand I can change the timeout for the browser, and 10.000 lines is not that much in my opinion. The transaction stops after 2.5 minutes.

Or do you have a hint, how I can change the groovy code to a job? btw. what is the difference between a job and a service, is the job a scheduled service?

I integrated that import into the webtools and start it from there, so I did it with groovy and minilang for the database service (Create...).

[Deepak Agarwal-2]

Yes, you are right, scheduled service is a job. Considering Adrian's
suggestion you should keep a low transaction timeout for exampleService and
use-transaction="false" for the groovy running this service.

Groovy files can be used as ofbiz service. You need to prepare a map and
put your OUT param in it to return, that's it (you can also use
ServiceUtil.returnSuccess/error methods).

On Tue, May 21, 2013 at 10:09 PM, Robert G. <[hidden email]> wrote:

> ha, that was fast :)
>
> I see and this sounds reasonable. But on the otherhand I can change the
> timeout for the browser, and 10.000 lines is not that much in my opinion.
> The transaction stops after 2.5 minutes.
>
> Or do you have a hint, how I can change the groovy code to a job? btw. what
> is the difference between a job and a service, is the job a scheduled
> service?
>
> I integrated that import into the webtools and start it from there, so I
> did
> it with groovy and minilang for the database service (Create...).
>
>
>
>
>
> --
> View this message in context:
> http://ofbiz.135035.n4.nabble.com/Groovy-call-a-service-within-a-while-loop-wait-notify-tp4641435p4641438.html
> Sent from the OFBiz - User mailing list archive at Nabble.com.
>

--
Thanks,
Deepak Agarwal,

Mobile: [phone]

[Robert Gan]

Hey david (Edit: I mean Adrian), also thank you for the fast answer.

How can I handle that with my situation?

I heard from you about --> use-transaction="false"
and from another thread sth. like: --> require-new-transaction=true.

so I understand it like my situation uses one single transaction what in my eyes is not that good concerning memory.

my situation is:

a screen includes a submit button to call a ftl with groovy logic. The groovy file calls the opencsv classes in java and gives back an iterator for the array (each line is one array, ittereated with the while loop). within this loop (groovy) I call the service to store the data in database (dispatcher.runSync("createPlSummary", [someData:someData])), service is written in xml. so I think here i sould say sth like require-new-transaction, but where do i put that? And is this the right way?

btw. the transaction stops after 180seconds (measured again :) )…

[Robert Gan]

ok now I used require-new-transaction="true" within the service declaration in services.xml. This seems to work. It imports all data (all rows are proceeded).

"Attribute : require-new-transaction
If set to true and there is a transaction already in place the Service Engine will suspend that transaction,
 begin a new one just for this service, commit or rollback the local transaction when the service is
 complete, and will resume the original transaction. If set to true and there is no transaction already in
 place it will just begin a transaction and manage it as would be done for a normal user-transaction=true.
 If use-transaction=false this setting is ignored."

I understand it that it pauses an active transaction, when a new one arrives, and resume it after the new one is ready, right?

I think this is fine so, what do you think?

[Jacques Le Roux]

From: "Robert G." <[hidden email]>

> ok now I used require-new-transaction="true" within the service declaration
> in services.xml. This seems to work. It imports all data (all rows are
> proceeded).
>
> "Attribute : require-new-transaction
> If set to true and there is a transaction already in place the Service
> Engine will suspend that transaction,
> begin a new one just for this service, commit or rollback the local
> transaction when the service is
> complete, and will resume the original transaction. If set to true and
> there is no transaction already in
> place it will just begin a transaction and manage it as would be done for a
> normal user-transaction=true.
> If use-transaction=false this setting is ignored."
>
> I understand it that it pauses an active transaction, when a new one
> arrives, and resume it after the new one is ready, right?

Yes, by using require-new-transaction="true" you suspend the (surrouding) transaction in which your Groovy file is (I guess you have one "automatically", normally you should), and create a new one for your called service. Else the current (surrouding) transaction, in which your groovy file was called, will be used by the called service. By using require-new-transaction="true", the suspended (surrouding) transaction is then resumed when the called service finishes. So it seems from your result (I never tested that) the suspended transaction see also its timeout "suspended". Which is why it would work for you. Interesting if true (I did not check the code), I see only this explanation

Note that by default the time out of a transaction is 60 seconds (defined in the Transaction Manager). By using require-new-transaction="true" you can set a higher (or different at large) time out for a called service if necesssary.
require-new-transaction="true" is only necessary for a service timeout to work if you are inside a previous transaction.

Another solution would have been to set a very high timeout on the action which is the origin (a request event I guess). But AFAIK this would work only if the surrounding call is a service. Because events (w/o service call) in requests use a transaction but there are no ways to set a timeout. So the default 60 sec would apply. If your event use a service, then another solution would have been to set a high timeout without using require-new-transaction="true". Because this is only needed for services called inside a service. This near what suggested Adrian,  to set the surrounding call in which you groovy file is, to not use a transaction, hence no timeout. But again this would only work if the event which eventually calls the service in the loop is itself a service. And, by and large,it's always better to set a timeout purposedly thantnot using a transaction.

Last but not least there is a specific trap, see https://issues.apache.org/jira/browse/OFBIZ-5183, which remembers me to do the point 1!

HTH

Jacques
 
> I think this is fine so, what do you think?
>
>
>
> --
> View this message in context: http://ofbiz.135035.n4.nabble.com/Groovy-call-a-service-within-a-while-loop-wait-notify-tp4641435p4641441.html
> Sent from the OFBiz - User mailing list archive at Nabble.com.

[Robert Gan]

yes exactly what I thought, the timout needs to be suspended too. So it works good. What I am wondering why its not possible to have multiple transactions paralell. If my hardware recources and maybe my internet connection are good enough I could handle thousands of transactions, so the time all together would shrink to a minimum (with high ressource ussage).

With the require-new-transaction="true" I get the results at the end, but all transactions will be handled in a row, what in my opinion is ressource friendly, but takes long.

btw. also thanks Deepak, your hints helped me a lot to use Groovy things as jobs, great!

[Jacques Le Roux]

From: "Robert G." <[hidden email]>
> yes exactly what I thought, the timout needs to be suspended too. So it works
> good. What I am wondering why its not possible to have multiple transactions
> paralell. If my hardware recources and maybe my internet connection are good
> enough I could handle thousands of transactions, so the time all together
> would shrink to a minimum (with high ressource ussage).
>
> With the require-new-transaction="true" I get the results at the end, but
> all transactions will be handled in a row, what in my opinion is ressource
> friendly, but takes long.

Then, if loop steps don't depend on each other, call the service asynchronously rather than synchronously.
Depending of the security you want for each step to be completed, you would want to make these async calls persistent (jobs) or not.
To follow jobs use /webtools/control/FindJob

HTH

Jacques

 
> btw. also thanks Deepak, your hints helped me a lot to use Groovy things as
> jobs, great!
>
>
>
> --
> View this message in context: http://ofbiz.135035.n4.nabble.com/Groovy-call-a-service-within-a-while-loop-wait-notify-tp4641435p4641451.html
> Sent from the OFBiz - User mailing list archive at Nabble.com.