We are running into a problem with the service engine when trying to run an
archiving batch service. Here is how the service runs. 1. We run a service that collects a bunch of records for data warehousing. 2. The data warehouse service calls an async service call to process each individual record. 3. The service engine them starts working on each individual record. We like the above process because it allows us to multi-process the large batch job. Here is the problem: There are over 10K records that are submitted as async service calls in step 2 above. I'm seeing a transaction time out when the service engine tries to process all of these records as it calls an "UPDATE JOB_SANDBOX for job_id=XXX". I believe it is trying to update all 10k of these jobs individually in a single transaction and times out. The jobs are then left in a pending status and never changed but the service engine keeps timing out. Here are my questions: 1. Is there a way to increase the transaction time out for the service engine to update these jobs in the JobSandbox so the service engine threads can then process them? or 2. Is there another way to break these jobs up so the service engine doesn't have to process 10k jobs in a single transaction. Note: the above process works very well when the number of records to process are around 1K. Brett |
To increase the transaction time out use transaction-timeout="{value}"
where value is any valid number. Note that this needs to be done in your service definition. Rishi Solanki Manager, Enterprise Software Development HotWax Media Pvt. Ltd. Direct: +91-9893287847 http://www.hotwaxmedia.com On Sun, Jun 27, 2010 at 2:17 AM, Brett Palmer <[hidden email]>wrote: > We are running into a problem with the service engine when trying to run an > archiving batch service. Here is how the service runs. > > 1. We run a service that collects a bunch of records for data warehousing. > > 2. The data warehouse service calls an async service call to process each > individual record. > > 3. The service engine them starts working on each individual record. > > We like the above process because it allows us to multi-process the large > batch job. > > Here is the problem: > > There are over 10K records that are submitted as async service calls in > step > 2 above. I'm seeing a transaction time out when the service engine tries > to > process all of these records as it calls an "UPDATE JOB_SANDBOX for > job_id=XXX". I believe it is trying to update all 10k of these jobs > individually in a single transaction and times out. The jobs are then left > in a pending status and never changed but the service engine keeps timing > out. > > Here are my questions: > > 1. Is there a way to increase the transaction time out for the service > engine to update these jobs in the JobSandbox so the service engine threads > can then process them? > > or > > 2. Is there another way to break these jobs up so the service engine > doesn't have to process 10k jobs in a single transaction. > > > Note: the above process works very well when the number of records to > process are around 1K. > > > > Brett > |
Rishi,
Thanks for the quick response. The transaction timeout is not with my defined service. The problem occurs in the JobManager.poll() method of the service engine. The JobManager is going through all the currently pending jobs and changing them to running but it timesout before it can finish. I can't see a way without modifying the JobManager code to change this transaction time in the "poll" method. I don't think my service definition transaction time modifies the transaction time in the JobManager. Let me know if I am missing something here. Brett On Mon, Jun 28, 2010 at 2:10 AM, Rishi Solanki <[hidden email]>wrote: > To increase the transaction time out use transaction-timeout="{value}" > where value is any valid number. > Note that this needs to be done in your service definition. > > Rishi Solanki > Manager, Enterprise Software Development > HotWax Media Pvt. Ltd. > Direct: +91-9893287847 > http://www.hotwaxmedia.com > > > On Sun, Jun 27, 2010 at 2:17 AM, Brett Palmer <[hidden email] > >wrote: > > > We are running into a problem with the service engine when trying to run > an > > archiving batch service. Here is how the service runs. > > > > 1. We run a service that collects a bunch of records for data > warehousing. > > > > 2. The data warehouse service calls an async service call to process each > > individual record. > > > > 3. The service engine them starts working on each individual record. > > > > We like the above process because it allows us to multi-process the large > > batch job. > > > > Here is the problem: > > > > There are over 10K records that are submitted as async service calls in > > step > > 2 above. I'm seeing a transaction time out when the service engine tries > > to > > process all of these records as it calls an "UPDATE JOB_SANDBOX for > > job_id=XXX". I believe it is trying to update all 10k of these jobs > > individually in a single transaction and times out. The jobs are then > left > > in a pending status and never changed but the service engine keeps timing > > out. > > > > Here are my questions: > > > > 1. Is there a way to increase the transaction time out for the service > > engine to update these jobs in the JobSandbox so the service engine > threads > > can then process them? > > > > or > > > > 2. Is there another way to break these jobs up so the service engine > > doesn't have to process 10k jobs in a single transaction. > > > > > > Note: the above process works very well when the number of records to > > process are around 1K. > > > > > > > > Brett > > > |
Hi Brett,
The JobManager probably wasn't intended (or at least initially written) to handle that many pending jobs, as a work around you could just add a timeout value to the TransactionUtil.begin() call in poll() and see if that does the trick. We could always look at making this configurable if you wanted to provide a patch for such a thing. Regards Scott HotWax Media http://www.hotwaxmedia.com On 29/06/2010, at 2:28 AM, Brett Palmer wrote: > Rishi, > > Thanks for the quick response. > > The transaction timeout is not with my defined service. The problem occurs > in the JobManager.poll() method of the service engine. The JobManager is > going through all the currently pending jobs and changing them to running > but it timesout before it can finish. > > I can't see a way without modifying the JobManager code to change this > transaction time in the "poll" method. I don't think my service definition > transaction time modifies the transaction time in the JobManager. > > Let me know if I am missing something here. > > > Brett > > On Mon, Jun 28, 2010 at 2:10 AM, Rishi Solanki <[hidden email]>wrote: > >> To increase the transaction time out use transaction-timeout="{value}" >> where value is any valid number. >> Note that this needs to be done in your service definition. >> >> Rishi Solanki >> Manager, Enterprise Software Development >> HotWax Media Pvt. Ltd. >> Direct: +91-9893287847 >> http://www.hotwaxmedia.com >> >> >> On Sun, Jun 27, 2010 at 2:17 AM, Brett Palmer <[hidden email] >>> wrote: >> >>> We are running into a problem with the service engine when trying to run >> an >>> archiving batch service. Here is how the service runs. >>> >>> 1. We run a service that collects a bunch of records for data >> warehousing. >>> >>> 2. The data warehouse service calls an async service call to process each >>> individual record. >>> >>> 3. The service engine them starts working on each individual record. >>> >>> We like the above process because it allows us to multi-process the large >>> batch job. >>> >>> Here is the problem: >>> >>> There are over 10K records that are submitted as async service calls in >>> step >>> 2 above. I'm seeing a transaction time out when the service engine tries >>> to >>> process all of these records as it calls an "UPDATE JOB_SANDBOX for >>> job_id=XXX". I believe it is trying to update all 10k of these jobs >>> individually in a single transaction and times out. The jobs are then >> left >>> in a pending status and never changed but the service engine keeps timing >>> out. >>> >>> Here are my questions: >>> >>> 1. Is there a way to increase the transaction time out for the service >>> engine to update these jobs in the JobSandbox so the service engine >> threads >>> can then process them? >>> >>> or >>> >>> 2. Is there another way to break these jobs up so the service engine >>> doesn't have to process 10k jobs in a single transaction. >>> >>> >>> Note: the above process works very well when the number of records to >>> process are around 1K. >>> >>> >>> >>> Brett >>> >> smime.p7s (3K) Download Attachment |
That sounds like the way to go Scott. -David On Jul 1, 2010, at 3:01 AM, Scott Gray wrote: > Hi Brett, > > The JobManager probably wasn't intended (or at least initially written) to handle that many pending jobs, as a work around you could just add a timeout value to the TransactionUtil.begin() call in poll() and see if that does the trick. We could always look at making this configurable if you wanted to provide a patch for such a thing. > > Regards > Scott > > HotWax Media > http://www.hotwaxmedia.com > > On 29/06/2010, at 2:28 AM, Brett Palmer wrote: > >> Rishi, >> >> Thanks for the quick response. >> >> The transaction timeout is not with my defined service. The problem occurs >> in the JobManager.poll() method of the service engine. The JobManager is >> going through all the currently pending jobs and changing them to running >> but it timesout before it can finish. >> >> I can't see a way without modifying the JobManager code to change this >> transaction time in the "poll" method. I don't think my service definition >> transaction time modifies the transaction time in the JobManager. >> >> Let me know if I am missing something here. >> >> >> Brett >> >> On Mon, Jun 28, 2010 at 2:10 AM, Rishi Solanki <[hidden email]>wrote: >> >>> To increase the transaction time out use transaction-timeout="{value}" >>> where value is any valid number. >>> Note that this needs to be done in your service definition. >>> >>> Rishi Solanki >>> Manager, Enterprise Software Development >>> HotWax Media Pvt. Ltd. >>> Direct: +91-9893287847 >>> http://www.hotwaxmedia.com >>> >>> >>> On Sun, Jun 27, 2010 at 2:17 AM, Brett Palmer <[hidden email] >>>> wrote: >>> >>>> We are running into a problem with the service engine when trying to run >>> an >>>> archiving batch service. Here is how the service runs. >>>> >>>> 1. We run a service that collects a bunch of records for data >>> warehousing. >>>> >>>> 2. The data warehouse service calls an async service call to process each >>>> individual record. >>>> >>>> 3. The service engine them starts working on each individual record. >>>> >>>> We like the above process because it allows us to multi-process the large >>>> batch job. >>>> >>>> Here is the problem: >>>> >>>> There are over 10K records that are submitted as async service calls in >>>> step >>>> 2 above. I'm seeing a transaction time out when the service engine tries >>>> to >>>> process all of these records as it calls an "UPDATE JOB_SANDBOX for >>>> job_id=XXX". I believe it is trying to update all 10k of these jobs >>>> individually in a single transaction and times out. The jobs are then >>> left >>>> in a pending status and never changed but the service engine keeps timing >>>> out. >>>> >>>> Here are my questions: >>>> >>>> 1. Is there a way to increase the transaction time out for the service >>>> engine to update these jobs in the JobSandbox so the service engine >>> threads >>>> can then process them? >>>> >>>> or >>>> >>>> 2. Is there another way to break these jobs up so the service engine >>>> doesn't have to process 10k jobs in a single transaction. >>>> >>>> >>>> Note: the above process works very well when the number of records to >>>> process are around 1K. >>>> >>>> >>>> >>>> Brett >>>> >>> > |
Okay, I opened JIRA ticket OFBIZ-3855 for this issue. I have tested the
patch in our version of ofbiz. I will check it with the current version of ofbiz and attach the patch to the JIRA ticket. The files affected by this patch will be the following: - serviceengine.xml -- Added new attribute: poll-transaction-timeout="120" to the thread-pool element - service-config.xsd -- to allow the new attribute in the serviceengine.xml And the following source files: - JobManager.java - JobPoller.java Thanks, Brett On Thu, Jul 1, 2010 at 3:03 AM, David E Jones <[hidden email]> wrote: > > That sounds like the way to go Scott. > > -David > > > On Jul 1, 2010, at 3:01 AM, Scott Gray wrote: > > > Hi Brett, > > > > The JobManager probably wasn't intended (or at least initially written) > to handle that many pending jobs, as a work around you could just add a > timeout value to the TransactionUtil.begin() call in poll() and see if that > does the trick. We could always look at making this configurable if you > wanted to provide a patch for such a thing. > > > > Regards > > Scott > > > > HotWax Media > > http://www.hotwaxmedia.com > > > > On 29/06/2010, at 2:28 AM, Brett Palmer wrote: > > > >> Rishi, > >> > >> Thanks for the quick response. > >> > >> The transaction timeout is not with my defined service. The problem > occurs > >> in the JobManager.poll() method of the service engine. The JobManager is > >> going through all the currently pending jobs and changing them to > running > >> but it timesout before it can finish. > >> > >> I can't see a way without modifying the JobManager code to change this > >> transaction time in the "poll" method. I don't think my service > definition > >> transaction time modifies the transaction time in the JobManager. > >> > >> Let me know if I am missing something here. > >> > >> > >> Brett > >> > >> On Mon, Jun 28, 2010 at 2:10 AM, Rishi Solanki <[hidden email] > >wrote: > >> > >>> To increase the transaction time out use transaction-timeout="{value}" > >>> where value is any valid number. > >>> Note that this needs to be done in your service definition. > >>> > >>> Rishi Solanki > >>> Manager, Enterprise Software Development > >>> HotWax Media Pvt. Ltd. > >>> Direct: +91-9893287847 > >>> http://www.hotwaxmedia.com > >>> > >>> > >>> On Sun, Jun 27, 2010 at 2:17 AM, Brett Palmer <[hidden email] > >>>> wrote: > >>> > >>>> We are running into a problem with the service engine when trying to > run > >>> an > >>>> archiving batch service. Here is how the service runs. > >>>> > >>>> 1. We run a service that collects a bunch of records for data > >>> warehousing. > >>>> > >>>> 2. The data warehouse service calls an async service call to process > each > >>>> individual record. > >>>> > >>>> 3. The service engine them starts working on each individual record. > >>>> > >>>> We like the above process because it allows us to multi-process the > large > >>>> batch job. > >>>> > >>>> Here is the problem: > >>>> > >>>> There are over 10K records that are submitted as async service calls > in > >>>> step > >>>> 2 above. I'm seeing a transaction time out when the service engine > tries > >>>> to > >>>> process all of these records as it calls an "UPDATE JOB_SANDBOX for > >>>> job_id=XXX". I believe it is trying to update all 10k of these jobs > >>>> individually in a single transaction and times out. The jobs are then > >>> left > >>>> in a pending status and never changed but the service engine keeps > timing > >>>> out. > >>>> > >>>> Here are my questions: > >>>> > >>>> 1. Is there a way to increase the transaction time out for the service > >>>> engine to update these jobs in the JobSandbox so the service engine > >>> threads > >>>> can then process them? > >>>> > >>>> or > >>>> > >>>> 2. Is there another way to break these jobs up so the service engine > >>>> doesn't have to process 10k jobs in a single transaction. > >>>> > >>>> > >>>> Note: the above process works very well when the number of records to > >>>> process are around 1K. > >>>> > >>>> > >>>> > >>>> Brett > >>>> > >>> > > > > |
Free forum by Nabble | Edit this page |