Service engine timeouts

[Brett]

We are running into a problem with the service engine when trying to run an
archiving batch service.  Here is how the service runs.

1. We run a service that collects a bunch of records for data warehousing.

2. The data warehouse service calls an async service call to process each
individual record.

3. The service engine them starts working on each individual record.

We like the above process because it allows us to multi-process the large
batch job.

Here is the problem:

There are over 10K records that are submitted as async service calls in step
2 above.  I'm seeing a transaction time out when the service engine tries to
process all of these records as it calls an "UPDATE JOB_SANDBOX for
job_id=XXX".  I believe it is trying to update all 10k of these jobs
individually in a single transaction and times out.  The jobs are then left
in a pending status and never changed but the service engine keeps timing
out.

Here are my questions:

1. Is there a way to increase the transaction time out for the service
engine to update these jobs in the JobSandbox so the service engine threads
can then process them?

or

2.  Is there another way to break these jobs up so the service engine
doesn't have to process 10k jobs in a single transaction.


Note: the above process works very well when the number of records to
process are around 1K.



Brett

[Rishi Solanki]

To increase the transaction time out use  transaction-timeout="{value}"
where value is any valid number.
Note that this needs to be done in your service definition.

Rishi Solanki
Manager, Enterprise Software Development
HotWax Media Pvt. Ltd.
Direct: [phone]
http://www.hotwaxmedia.com


On Sun, Jun 27, 2010 at 2:17 AM, Brett Palmer <[hidden email]>wrote:

> We are running into a problem with the service engine when trying to run an
> archiving batch service.  Here is how the service runs.
>
> 1. We run a service that collects a bunch of records for data warehousing.
>
> 2. The data warehouse service calls an async service call to process each
> individual record.
>
> 3. The service engine them starts working on each individual record.
>
> We like the above process because it allows us to multi-process the large
> batch job.
>
> Here is the problem:
>
> There are over 10K records that are submitted as async service calls in
> step
> 2 above.  I'm seeing a transaction time out when the service engine tries
> to
> process all of these records as it calls an "UPDATE JOB_SANDBOX for
> job_id=XXX".  I believe it is trying to update all 10k of these jobs
> individually in a single transaction and times out.  The jobs are then left
> in a pending status and never changed but the service engine keeps timing
> out.
>
> Here are my questions:
>
> 1. Is there a way to increase the transaction time out for the service
> engine to update these jobs in the JobSandbox so the service engine threads
> can then process them?
>
> or
>
> 2.  Is there another way to break these jobs up so the service engine
> doesn't have to process 10k jobs in a single transaction.
>
>
> Note: the above process works very well when the number of records to
> process are around 1K.
>
>
>
> Brett
>

[Brett]

Rishi,

Thanks for the quick response.

The transaction timeout is not with my defined service.  The problem occurs
in the JobManager.poll() method of the service engine. The JobManager is
going through all the currently pending jobs and changing them to running
but it timesout before it can finish.

I can't see a way without modifying the JobManager code to change this
transaction time in the "poll" method.  I don't think my service definition
transaction time modifies the transaction time in the JobManager.

Let me know if I am missing something here.


Brett

On Mon, Jun 28, 2010 at 2:10 AM, Rishi Solanki <[hidden email]>wrote:

> To increase the transaction time out use  transaction-timeout="{value}"
> where value is any valid number.
> Note that this needs to be done in your service definition.
>
> Rishi Solanki
> Manager, Enterprise Software Development
> HotWax Media Pvt. Ltd.
> Direct: [phone]
> http://www.hotwaxmedia.com
>
>
> On Sun, Jun 27, 2010 at 2:17 AM, Brett Palmer <[hidden email]
> >wrote:
>
> > We are running into a problem with the service engine when trying to run
> an
> > archiving batch service.  Here is how the service runs.
> >
> > 1. We run a service that collects a bunch of records for data
> warehousing.
> >
> > 2. The data warehouse service calls an async service call to process each
> > individual record.
> >
> > 3. The service engine them starts working on each individual record.
> >
> > We like the above process because it allows us to multi-process the large
> > batch job.
> >
> > Here is the problem:
> >
> > There are over 10K records that are submitted as async service calls in
> > step
> > 2 above.  I'm seeing a transaction time out when the service engine tries
> > to
> > process all of these records as it calls an "UPDATE JOB_SANDBOX for
> > job_id=XXX".  I believe it is trying to update all 10k of these jobs
> > individually in a single transaction and times out.  The jobs are then
> left
> > in a pending status and never changed but the service engine keeps timing
> > out.
> >
> > Here are my questions:
> >
> > 1. Is there a way to increase the transaction time out for the service
> > engine to update these jobs in the JobSandbox so the service engine
> threads
> > can then process them?
> >
> > or
> >
> > 2.  Is there another way to break these jobs up so the service engine
> > doesn't have to process 10k jobs in a single transaction.
> >
> >
> > Note: the above process works very well when the number of records to
> > process are around 1K.
> >
> >
> >
> > Brett
> >
>

[Scott Gray-2]

Hi Brett,

The JobManager probably wasn't intended (or at least initially written) to handle that many pending jobs, as a work around you could just add a timeout value to the TransactionUtil.begin() call in poll() and see if that does the trick.  We could always look at making this configurable if you wanted to provide a patch for such a thing.

Regards
Scott

HotWax Media
http://www.hotwaxmedia.com

On 29/06/2010, at 2:28 AM, Brett Palmer wrote:

> Rishi,
>
> Thanks for the quick response.
>
> The transaction timeout is not with my defined service.  The problem occurs
> in the JobManager.poll() method of the service engine. The JobManager is
> going through all the currently pending jobs and changing them to running
> but it timesout before it can finish.
>
> I can't see a way without modifying the JobManager code to change this
> transaction time in the "poll" method.  I don't think my service definition
> transaction time modifies the transaction time in the JobManager.
>
> Let me know if I am missing something here.
>
>
> Brett
>
> On Mon, Jun 28, 2010 at 2:10 AM, Rishi Solanki <[hidden email]>wrote:
>
>> To increase the transaction time out use  transaction-timeout="{value}"
>> where value is any valid number.
>> Note that this needs to be done in your service definition.
>>
>> Rishi Solanki
>> Manager, Enterprise Software Development
>> HotWax Media Pvt. Ltd.
>> Direct: [phone]
>> http://www.hotwaxmedia.com
>>
>>
>> On Sun, Jun 27, 2010 at 2:17 AM, Brett Palmer <[hidden email]
>>> wrote:
>>
>>> We are running into a problem with the service engine when trying to run
>> an
>>> archiving batch service.  Here is how the service runs.
>>>
>>> 1. We run a service that collects a bunch of records for data
>> warehousing.
>>>
>>> 2. The data warehouse service calls an async service call to process each
>>> individual record.
>>>
>>> 3. The service engine them starts working on each individual record.
>>>
>>> We like the above process because it allows us to multi-process the large
>>> batch job.
>>>
>>> Here is the problem:
>>>
>>> There are over 10K records that are submitted as async service calls in
>>> step
>>> 2 above.  I'm seeing a transaction time out when the service engine tries
>>> to
>>> process all of these records as it calls an "UPDATE JOB_SANDBOX for
>>> job_id=XXX".  I believe it is trying to update all 10k of these jobs
>>> individually in a single transaction and times out.  The jobs are then
>> left
>>> in a pending status and never changed but the service engine keeps timing
>>> out.
>>>
>>> Here are my questions:
>>>
>>> 1. Is there a way to increase the transaction time out for the service
>>> engine to update these jobs in the JobSandbox so the service engine
>> threads
>>> can then process them?
>>>
>>> or
>>>
>>> 2.  Is there another way to break these jobs up so the service engine
>>> doesn't have to process 10k jobs in a single transaction.
>>>
>>>
>>> Note: the above process works very well when the number of records to
>>> process are around 1K.
>>>
>>>
>>>
>>> Brett
>>>
>>

[David E. Jones-2]

That sounds like the way to go Scott.

-David


On Jul 1, 2010, at 3:01 AM, Scott Gray wrote:

> Hi Brett,
>
> The JobManager probably wasn't intended (or at least initially written) to handle that many pending jobs, as a work around you could just add a timeout value to the TransactionUtil.begin() call in poll() and see if that does the trick.  We could always look at making this configurable if you wanted to provide a patch for such a thing.
>
> Regards
> Scott
>
> HotWax Media
> http://www.hotwaxmedia.com
>
> On 29/06/2010, at 2:28 AM, Brett Palmer wrote:
>
>> Rishi,
>>
>> Thanks for the quick response.
>>
>> The transaction timeout is not with my defined service.  The problem occurs
>> in the JobManager.poll() method of the service engine. The JobManager is
>> going through all the currently pending jobs and changing them to running
>> but it timesout before it can finish.
>>
>> I can't see a way without modifying the JobManager code to change this
>> transaction time in the "poll" method.  I don't think my service definition
>> transaction time modifies the transaction time in the JobManager.
>>
>> Let me know if I am missing something here.
>>
>>
>> Brett
>>
>> On Mon, Jun 28, 2010 at 2:10 AM, Rishi Solanki <[hidden email]>wrote:
>>
>>> To increase the transaction time out use  transaction-timeout="{value}"
>>> where value is any valid number.
>>> Note that this needs to be done in your service definition.
>>>
>>> Rishi Solanki
>>> Manager, Enterprise Software Development
>>> HotWax Media Pvt. Ltd.
>>> Direct: [phone]
>>> http://www.hotwaxmedia.com
>>>
>>>
>>> On Sun, Jun 27, 2010 at 2:17 AM, Brett Palmer <[hidden email]
>>>> wrote:
>>>
>>>> We are running into a problem with the service engine when trying to run
>>> an
>>>> archiving batch service.  Here is how the service runs.
>>>>
>>>> 1. We run a service that collects a bunch of records for data
>>> warehousing.
>>>>
>>>> 2. The data warehouse service calls an async service call to process each
>>>> individual record.
>>>>
>>>> 3. The service engine them starts working on each individual record.
>>>>
>>>> We like the above process because it allows us to multi-process the large
>>>> batch job.
>>>>
>>>> Here is the problem:
>>>>
>>>> There are over 10K records that are submitted as async service calls in
>>>> step
>>>> 2 above.  I'm seeing a transaction time out when the service engine tries
>>>> to
>>>> process all of these records as it calls an "UPDATE JOB_SANDBOX for
>>>> job_id=XXX".  I believe it is trying to update all 10k of these jobs
>>>> individually in a single transaction and times out.  The jobs are then
>>> left
>>>> in a pending status and never changed but the service engine keeps timing
>>>> out.
>>>>
>>>> Here are my questions:
>>>>
>>>> 1. Is there a way to increase the transaction time out for the service
>>>> engine to update these jobs in the JobSandbox so the service engine
>>> threads
>>>> can then process them?
>>>>
>>>> or
>>>>
>>>> 2.  Is there another way to break these jobs up so the service engine
>>>> doesn't have to process 10k jobs in a single transaction.
>>>>
>>>>
>>>> Note: the above process works very well when the number of records to
>>>> process are around 1K.
>>>>
>>>>
>>>>
>>>> Brett
>>>>
>>>
>

[Brett]

Okay, I opened JIRA ticket OFBIZ-3855 for this issue.  I have tested the
patch in our version of ofbiz.  I will check it with the current version of
ofbiz and attach the patch to the JIRA ticket.

The files affected by this patch will be the following:

- serviceengine.xml
-- Added new attribute:  poll-transaction-timeout="120"  to the thread-pool
element

- service-config.xsd   -- to allow the new attribute in the
serviceengine.xml

And the following source files:

- JobManager.java
- JobPoller.java

Thanks,


Brett

On Thu, Jul 1, 2010 at 3:03 AM, David E Jones <[hidden email]> wrote:

>
> That sounds like the way to go Scott.
>
> -David
>
>
> On Jul 1, 2010, at 3:01 AM, Scott Gray wrote:
>
> > Hi Brett,
> >
> > The JobManager probably wasn't intended (or at least initially written)
> to handle that many pending jobs, as a work around you could just add a
> timeout value to the TransactionUtil.begin() call in poll() and see if that
> does the trick.  We could always look at making this configurable if you
> wanted to provide a patch for such a thing.
> >
> > Regards
> > Scott
> >
> > HotWax Media
> > http://www.hotwaxmedia.com
> >
> > On 29/06/2010, at 2:28 AM, Brett Palmer wrote:
> >
> >> Rishi,
> >>
> >> Thanks for the quick response.
> >>
> >> The transaction timeout is not with my defined service.  The problem
> occurs
> >> in the JobManager.poll() method of the service engine. The JobManager is
> >> going through all the currently pending jobs and changing them to
> running
> >> but it timesout before it can finish.
> >>
> >> I can't see a way without modifying the JobManager code to change this
> >> transaction time in the "poll" method.  I don't think my service
> definition
> >> transaction time modifies the transaction time in the JobManager.
> >>
> >> Let me know if I am missing something here.
> >>
> >>
> >> Brett
> >>
> >> On Mon, Jun 28, 2010 at 2:10 AM, Rishi Solanki <[hidden email]
> >wrote:
> >>
> >>> To increase the transaction time out use  transaction-timeout="{value}"
> >>> where value is any valid number.
> >>> Note that this needs to be done in your service definition.
> >>>
> >>> Rishi Solanki
> >>> Manager, Enterprise Software Development
> >>> HotWax Media Pvt. Ltd.
> >>> Direct: [phone]
> >>> http://www.hotwaxmedia.com
> >>>
> >>>
> >>> On Sun, Jun 27, 2010 at 2:17 AM, Brett Palmer <[hidden email]
> >>>> wrote:
> >>>
> >>>> We are running into a problem with the service engine when trying to
> run
> >>> an
> >>>> archiving batch service.  Here is how the service runs.
> >>>>
> >>>> 1. We run a service that collects a bunch of records for data
> >>> warehousing.
> >>>>
> >>>> 2. The data warehouse service calls an async service call to process
> each
> >>>> individual record.
> >>>>
> >>>> 3. The service engine them starts working on each individual record.
> >>>>
> >>>> We like the above process because it allows us to multi-process the
> large
> >>>> batch job.
> >>>>
> >>>> Here is the problem:
> >>>>
> >>>> There are over 10K records that are submitted as async service calls
> in
> >>>> step
> >>>> 2 above.  I'm seeing a transaction time out when the service engine
> tries
> >>>> to
> >>>> process all of these records as it calls an "UPDATE JOB_SANDBOX for
> >>>> job_id=XXX".  I believe it is trying to update all 10k of these jobs
> >>>> individually in a single transaction and times out.  The jobs are then
> >>> left
> >>>> in a pending status and never changed but the service engine keeps
> timing
> >>>> out.
> >>>>
> >>>> Here are my questions:
> >>>>
> >>>> 1. Is there a way to increase the transaction time out for the service
> >>>> engine to update these jobs in the JobSandbox so the service engine
> >>> threads
> >>>> can then process them?
> >>>>
> >>>> or
> >>>>
> >>>> 2.  Is there another way to break these jobs up so the service engine
> >>>> doesn't have to process 10k jobs in a single transaction.
> >>>>
> >>>>
> >>>> Note: the above process works very well when the number of records to
> >>>> process are around 1K.
> >>>>
> >>>>
> >>>>
> >>>> Brett
> >>>>
> >>>
> >
>
>