Job Manager
Locked
12
12
 
|
Brett,
Any results yet? -Adrian On 8/8/2012 12:57 AM, Brett Palmer wrote: > Adrian, > > Thanks I'll take an update and try it out. > > Brett > On Aug 7, 2012 4:23 PM, "Adrian Crum" <[hidden email]> > wrote: > >> Brett, >> >> I think I solved your problems with my recent commits, ending with rev >> 1370566. Let me know if it helps. >> >> -Adrian >> >> On 8/5/2012 4:53 PM, Brett Palmer wrote: >> >>> Adrian, >>> >>> Thanks for the update. Here are some feedback points on your listed >>> items: >>> >>> 1. JobPoller get out-of-memor error. We've seen this a lot in production >>> servers when the JobSandbox table is not constantly pruned of old records. >>> It would be nice if the poller restricted its search for only active >>> records it could process. >>> >>> 2. Queue for capturing missing records would be good. From item 1 above >>> we >>> have had locks on table when the poller is busy doing a scan and new jobs >>> cannot be added or time out. >>> >>> Other wish items: >>> >>> - Ability to assign different service engines to process specific job >>> types. We often multiple application servers but want to limit how many >>> concurrent jobs are run. For example, if I had 4 app servers connected to >>> the same DB I may only want one app server to service particular jobs. I >>> thought this feature was possible but when I tried to implement it by >>> changing some of the configuration files it never worked correctly. >>> >>> - JMS support for the service engine. It would be nice if there was a JMS >>> interface for those that want to use JMS as their queuing mechanism for >>> jobs. >>> >>> >>> Brett >>> >>> On Sun, Aug 5, 2012 at 6:21 AM, Adrian Crum < >>> adrian.crum@sandglass-**software.com <[hidden email]>> >>> wrote: >>> >>> On 8/5/2012 11:02 AM, Adrian Crum wrote: >>>> I just committed a bunch of changes to the Job Manager group of classes. >>>>> The changes help simplify the code and hopefully make the Job Manager >>>>> more >>>>> robust. On the other hand, I might have broken something. ;) I will >>>>> monitor >>>>> the mailing list for problems. >>>>> >>>>> I believe the JobPoller settings in serviceengine.xml (the <thread-pool> >>>>> element) should be changed. I think min-threads should be set to "2" and >>>>> max-threads should be set to "5". Creating lots of threads can hurt >>>>> throughput because the JVM spends more time managing them. I would be >>>>> interested in hearing what others think. >>>>> >>>>> Thinking about this more, there are some other things that need to be >>>> fixed: >>>> >>>> 1. The JobPoller uses an unbounded queue. In a busy server, there is the >>>> potential the queue will grow in size until it causes an out-of-memory >>>> condition. >>>> 2. There is no accommodation for when a job cannot be added to the queue >>>> - >>>> it is just lost. We could add a dequeue method to the Job interface that >>>> will allow implementations to recover or reschedule the job when it can't >>>> be added to the queue. >>>> 3. There is a JobPoller instance per delegator, and each instance >>>> contains >>>> the number of threads configured in serviceengine.xml. With the current >>>> max-threads setting of 15, a multi-tenant installation with 100 tenants >>>> will create up to 1500 threads. (!!!) A smarter strategy might be to >>>> have a >>>> single JobPoller instance that services multiple JobManagers. >>>> >>>> -Adrian >>>> >>>> >>>> >>>> |
|
|
Adrian,
I need to update our application to work with the newest code from the trunk. We did an update a couple months ago so I don't think it will take me long. I'll try to provide some feedback early next week. Brett On Sat, Aug 18, 2012 at 12:31 AM, Adrian Crum < [hidden email]> wrote: > Brett, > > Any results yet? > > -Adrian > > > On 8/8/2012 12:57 AM, Brett Palmer wrote: > >> Adrian, >> >> Thanks I'll take an update and try it out. >> >> Brett >> On Aug 7, 2012 4:23 PM, "Adrian Crum" <adrian.crum@sandglass-** >> software.com <[hidden email]>> >> wrote: >> >> Brett, >>> >>> I think I solved your problems with my recent commits, ending with rev >>> 1370566. Let me know if it helps. >>> >>> -Adrian >>> >>> On 8/5/2012 4:53 PM, Brett Palmer wrote: >>> >>> Adrian, >>>> >>>> Thanks for the update. Here are some feedback points on your listed >>>> items: >>>> >>>> 1. JobPoller get out-of-memor error. We've seen this a lot in >>>> production >>>> servers when the JobSandbox table is not constantly pruned of old >>>> records. >>>> It would be nice if the poller restricted its search for only active >>>> records it could process. >>>> >>>> 2. Queue for capturing missing records would be good. From item 1 above >>>> we >>>> have had locks on table when the poller is busy doing a scan and new >>>> jobs >>>> cannot be added or time out. >>>> >>>> Other wish items: >>>> >>>> - Ability to assign different service engines to process specific job >>>> types. We often multiple application servers but want to limit how many >>>> concurrent jobs are run. For example, if I had 4 app servers connected >>>> to >>>> the same DB I may only want one app server to service particular jobs. >>>> I >>>> thought this feature was possible but when I tried to implement it by >>>> changing some of the configuration files it never worked correctly. >>>> >>>> - JMS support for the service engine. It would be nice if there was a >>>> JMS >>>> interface for those that want to use JMS as their queuing mechanism for >>>> jobs. >>>> >>>> >>>> Brett >>>> >>>> On Sun, Aug 5, 2012 at 6:21 AM, Adrian Crum < >>>> adrian.crum@sandglass-**softwa**re.com <http://software.com> < >>>> adrian.crum@sandglass-**software.com<[hidden email]> >>>> >> >>>> >>>> wrote: >>>> >>>> On 8/5/2012 11:02 AM, Adrian Crum wrote: >>>> >>>>> I just committed a bunch of changes to the Job Manager group of >>>>> classes. >>>>> >>>>>> The changes help simplify the code and hopefully make the Job Manager >>>>>> more >>>>>> robust. On the other hand, I might have broken something. ;) I will >>>>>> monitor >>>>>> the mailing list for problems. >>>>>> >>>>>> I believe the JobPoller settings in serviceengine.xml (the >>>>>> <thread-pool> >>>>>> element) should be changed. I think min-threads should be set to "2" >>>>>> and >>>>>> max-threads should be set to "5". Creating lots of threads can hurt >>>>>> throughput because the JVM spends more time managing them. I would be >>>>>> interested in hearing what others think. >>>>>> >>>>>> Thinking about this more, there are some other things that need to >>>>>> be >>>>>> >>>>> fixed: >>>>> >>>>> 1. The JobPoller uses an unbounded queue. In a busy server, there is >>>>> the >>>>> potential the queue will grow in size until it causes an out-of-memory >>>>> condition. >>>>> 2. There is no accommodation for when a job cannot be added to the >>>>> queue >>>>> - >>>>> it is just lost. We could add a dequeue method to the Job interface >>>>> that >>>>> will allow implementations to recover or reschedule the job when it >>>>> can't >>>>> be added to the queue. >>>>> 3. There is a JobPoller instance per delegator, and each instance >>>>> contains >>>>> the number of threads configured in serviceengine.xml. With the current >>>>> max-threads setting of 15, a multi-tenant installation with 100 tenants >>>>> will create up to 1500 threads. (!!!) A smarter strategy might be to >>>>> have a >>>>> single JobPoller instance that services multiple JobManagers. >>>>> >>>>> -Adrian >>>>> >>>>> >>>>> >>>>> >>>>> > |
|
|
Cool - thanks!
-Adrian On 8/18/2012 4:11 PM, Brett Palmer wrote: > Adrian, > > I need to update our application to work with the newest code from the > trunk. We did an update a couple months ago so I don't think it will take > me long. I'll try to provide some feedback early next week. > > > Brett > > On Sat, Aug 18, 2012 at 12:31 AM, Adrian Crum < > [hidden email]> wrote: > >> Brett, >> >> Any results yet? >> >> -Adrian >> >> >> On 8/8/2012 12:57 AM, Brett Palmer wrote: >> >>> Adrian, >>> >>> Thanks I'll take an update and try it out. >>> >>> Brett >>> On Aug 7, 2012 4:23 PM, "Adrian Crum" <adrian.crum@sandglass-** >>> software.com <[hidden email]>> >>> wrote: >>> >>> Brett, >>>> I think I solved your problems with my recent commits, ending with rev >>>> 1370566. Let me know if it helps. >>>> >>>> -Adrian >>>> >>>> On 8/5/2012 4:53 PM, Brett Palmer wrote: >>>> >>>> Adrian, >>>>> Thanks for the update. Here are some feedback points on your listed >>>>> items: >>>>> >>>>> 1. JobPoller get out-of-memor error. We've seen this a lot in >>>>> production >>>>> servers when the JobSandbox table is not constantly pruned of old >>>>> records. >>>>> It would be nice if the poller restricted its search for only active >>>>> records it could process. >>>>> >>>>> 2. Queue for capturing missing records would be good. From item 1 above >>>>> we >>>>> have had locks on table when the poller is busy doing a scan and new >>>>> jobs >>>>> cannot be added or time out. >>>>> >>>>> Other wish items: >>>>> >>>>> - Ability to assign different service engines to process specific job >>>>> types. We often multiple application servers but want to limit how many >>>>> concurrent jobs are run. For example, if I had 4 app servers connected >>>>> to >>>>> the same DB I may only want one app server to service particular jobs. >>>>> I >>>>> thought this feature was possible but when I tried to implement it by >>>>> changing some of the configuration files it never worked correctly. >>>>> >>>>> - JMS support for the service engine. It would be nice if there was a >>>>> JMS >>>>> interface for those that want to use JMS as their queuing mechanism for >>>>> jobs. >>>>> >>>>> >>>>> Brett >>>>> >>>>> On Sun, Aug 5, 2012 at 6:21 AM, Adrian Crum < >>>>> adrian.crum@sandglass-**softwa**re.com <http://software.com> < >>>>> adrian.crum@sandglass-**software.com<[hidden email]> >>>>> wrote: >>>>> >>>>> On 8/5/2012 11:02 AM, Adrian Crum wrote: >>>>> >>>>>> I just committed a bunch of changes to the Job Manager group of >>>>>> classes. >>>>>> >>>>>>> The changes help simplify the code and hopefully make the Job Manager >>>>>>> more >>>>>>> robust. On the other hand, I might have broken something. ;) I will >>>>>>> monitor >>>>>>> the mailing list for problems. >>>>>>> >>>>>>> I believe the JobPoller settings in serviceengine.xml (the >>>>>>> <thread-pool> >>>>>>> element) should be changed. I think min-threads should be set to "2" >>>>>>> and >>>>>>> max-threads should be set to "5". Creating lots of threads can hurt >>>>>>> throughput because the JVM spends more time managing them. I would be >>>>>>> interested in hearing what others think. >>>>>>> >>>>>>> Thinking about this more, there are some other things that need to >>>>>>> be >>>>>>> >>>>>> fixed: >>>>>> >>>>>> 1. The JobPoller uses an unbounded queue. In a busy server, there is >>>>>> the >>>>>> potential the queue will grow in size until it causes an out-of-memory >>>>>> condition. >>>>>> 2. There is no accommodation for when a job cannot be added to the >>>>>> queue >>>>>> - >>>>>> it is just lost. We could add a dequeue method to the Job interface >>>>>> that >>>>>> will allow implementations to recover or reschedule the job when it >>>>>> can't >>>>>> be added to the queue. >>>>>> 3. There is a JobPoller instance per delegator, and each instance >>>>>> contains >>>>>> the number of threads configured in serviceengine.xml. With the current >>>>>> max-threads setting of 15, a multi-tenant installation with 100 tenants >>>>>> will create up to 1500 threads. (!!!) A smarter strategy might be to >>>>>> have a >>>>>> single JobPoller instance that services multiple JobManagers. >>>>>> >>>>>> -Adrian >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> |
|
|
In reply to this post by Brett
This change was made in rev 1374596.
The main thing to watch out for is the queue size. The default is 100, but that might be too small in a multi-delegator or multi-tenant installation. Follow the queue size advice in service-config.xsd. -Adrian On 8/13/2012 2:42 AM, Brett Palmer wrote: > *Adrian, > > I think the single JobPoller is a good idea and reduces the chance of too > many JobPoller’s running on a machine. > > We often setup multiple delegators to communicate with different databases. > For example, our data warehouse is hosted on a separate server. These > databases usually have a full ofbiz schema on them (jobsandbox table, etc). > > Here is how our data warehouse process works: > > The application has several ofbiz servers talking to a primary database. > These servers contain all the user information for our application. When > a person logs into the application they are redirected to a secondary ofbiz > server that is used for running the application under heavy loads. The > data is captured on the secondary server. > > A data warehouse process is scheduled to run every 5 mins on these > secondary servers. The secondary servers have a delegator that talks to > its local database and a delegator to talk to the data warehouse. > > With the new job poller changes would the poller pick up jobs from the data > warehouse database since it has a delegator that points to that instance? > > For this example, we would need to make sure the job poller on the > secondary server only serviced jobs from its local database (default > delegator) and not the our configured olap delegator. > > Let me know if this will be possible with the new job poller or if you have > any questions from my scenario. I realize this scenario is not the typical > ofbiz e-commerce type of setup everyone is use to, but we use ofbiz to > create lots of different types applications and have found it very flexible > for creating just about any type of ERP application. > > > Thanks for your work on the job poller. > > > Brett * > > On Sun, Aug 12, 2012 at 5:36 AM, Adrian Crum < > [hidden email]> wrote: > >> On 8/5/2012 1:21 PM, Adrian Crum wrote: >> >>> On 8/5/2012 11:02 AM, Adrian Crum wrote: >>> >>>> I just committed a bunch of changes to the Job Manager group of classes. >>>> The changes help simplify the code and hopefully make the Job Manager more >>>> robust. On the other hand, I might have broken something. ;) I will monitor >>>> the mailing list for problems. >>>> >>>> I believe the JobPoller settings in serviceengine.xml (the <thread-pool> >>>> element) should be changed. I think min-threads should be set to "2" and >>>> max-threads should be set to "5". Creating lots of threads can hurt >>>> throughput because the JVM spends more time managing them. I would be >>>> interested in hearing what others think. >>>> >>> Thinking about this more, there are some other things that need to be >>> fixed: >>> >>> 1. The JobPoller uses an unbounded queue. In a busy server, there is the >>> potential the queue will grow in size until it causes an out-of-memory >>> condition. >>> 2. There is no accommodation for when a job cannot be added to the queue >>> - it is just lost. We could add a dequeue method to the Job interface that >>> will allow implementations to recover or reschedule the job when it can't >>> be added to the queue. >>> 3. There is a JobPoller instance per delegator, and each instance >>> contains the number of threads configured in serviceengine.xml. With the >>> current max-threads setting of 15, a multi-tenant installation with 100 >>> tenants will create up to 1500 threads. (!!!) A smarter strategy might be >>> to have a single JobPoller instance that services multiple JobManagers. >>> >> I fixed #1 and #2. I am considering working on #3, but I want some >> feedback first. >> >> A JobPoller instance is created for each delegator. So, in a multi-tenant >> or multi-delegator scenario, multiple JobPollers will be created - which >> means one job queue per delegator and (threads per queue) threads per >> delegator. In a multi-server installation, things are multiplied: (# of >> servers * # of delegators) job queues. Fortunately, in that scenario we can >> disable the JobPoller on all but one server. >> >> So, we are left with the potential problem of too many queues/threads >> being created on a multi-delegator or multi-tenant server. So, I think >> there should be one JobPoller instance that services all delegators. At >> each polling interval, the JobPoller gets a list of jobs from each >> delegator (JobManager) - creating a list of lists. Then the JobPoller >> creates a queue candidate list from the list of lists - using a round-robin >> approach so each delegator gets an equal opportunity to queue jobs. The >> JobPoller queues the candidate list, and any candidates that don't fit in >> the queue are rescheduled. With this approach the JobPoller can service any >> number of delegators without saturating the server. >> >> What do you think? >> >> -Adrian >> >> |
|
|
In reply to this post by Adrian Crum-3
Yeah, that's not going to fly. Its interesting to note that David Jones replaced the JobPoller with Quartz in Moqui. ----- "Adrian Crum" wrote: > 3. There is a JobPoller instance per delegator, and each instance > contains the number of threads configured in serviceengine.xml. With the > current max-threads setting of 15, a multi-tenant installation with 100 > tenants will create up to 1500 threads. (!!!) A smarter strategy might > be to have a single JobPoller instance that services multiple JobManagers. -- Ean Schuessler, CTO [hidden email] 214-720-0700 x 315 Brainfood, Inc. http://www.brainfood.com |
|
|
I considered integrating Quartz, but the problems with the JobManager
and JobPoller were easy to identify and fix. Time will tell though... -Adrian On 8/18/2012 4:59 PM, Ean Schuessler wrote: > Yeah, that's not going to fly. Its interesting to note that David Jones replaced the JobPoller with Quartz in Moqui. > ----- "Adrian Crum" wrote: >> 3. There is a JobPoller instance per delegator, and each instance >> contains the number of threads configured in serviceengine.xml. With the >> current max-threads setting of 15, a multi-tenant installation with 100 >> tenants will create up to 1500 threads. (!!!) A smarter strategy might >> be to have a single JobPoller instance that services multiple JobManagers. |
|
|
In reply to this post by Brett
Brett,
Any news on this? -Adrian On 8/18/2012 4:11 PM, Brett Palmer wrote: > Adrian, > > I need to update our application to work with the newest code from the > trunk. We did an update a couple months ago so I don't think it will take > me long. I'll try to provide some feedback early next week. > > > Brett > > On Sat, Aug 18, 2012 at 12:31 AM, Adrian Crum < > [hidden email]> wrote: > >> Brett, >> >> Any results yet? >> >> -Adrian >> >> >> On 8/8/2012 12:57 AM, Brett Palmer wrote: >> >>> Adrian, >>> >>> Thanks I'll take an update and try it out. >>> >>> Brett >>> On Aug 7, 2012 4:23 PM, "Adrian Crum" <adrian.crum@sandglass-** >>> software.com <[hidden email]>> >>> wrote: >>> >>> Brett, >>>> I think I solved your problems with my recent commits, ending with rev >>>> 1370566. Let me know if it helps. >>>> >>>> -Adrian >>>> >>>> On 8/5/2012 4:53 PM, Brett Palmer wrote: >>>> >>>> Adrian, >>>>> Thanks for the update. Here are some feedback points on your listed >>>>> items: >>>>> >>>>> 1. JobPoller get out-of-memor error. We've seen this a lot in >>>>> production >>>>> servers when the JobSandbox table is not constantly pruned of old >>>>> records. >>>>> It would be nice if the poller restricted its search for only active >>>>> records it could process. >>>>> >>>>> 2. Queue for capturing missing records would be good. From item 1 above >>>>> we >>>>> have had locks on table when the poller is busy doing a scan and new >>>>> jobs >>>>> cannot be added or time out. >>>>> >>>>> Other wish items: >>>>> >>>>> - Ability to assign different service engines to process specific job >>>>> types. We often multiple application servers but want to limit how many >>>>> concurrent jobs are run. For example, if I had 4 app servers connected >>>>> to >>>>> the same DB I may only want one app server to service particular jobs. >>>>> I >>>>> thought this feature was possible but when I tried to implement it by >>>>> changing some of the configuration files it never worked correctly. >>>>> >>>>> - JMS support for the service engine. It would be nice if there was a >>>>> JMS >>>>> interface for those that want to use JMS as their queuing mechanism for >>>>> jobs. >>>>> >>>>> >>>>> Brett >>>>> >>>>> On Sun, Aug 5, 2012 at 6:21 AM, Adrian Crum < >>>>> adrian.crum@sandglass-**softwa**re.com <http://software.com> < >>>>> adrian.crum@sandglass-**software.com<[hidden email]> >>>>> wrote: >>>>> >>>>> On 8/5/2012 11:02 AM, Adrian Crum wrote: >>>>> >>>>>> I just committed a bunch of changes to the Job Manager group of >>>>>> classes. >>>>>> >>>>>>> The changes help simplify the code and hopefully make the Job Manager >>>>>>> more >>>>>>> robust. On the other hand, I might have broken something. ;) I will >>>>>>> monitor >>>>>>> the mailing list for problems. >>>>>>> >>>>>>> I believe the JobPoller settings in serviceengine.xml (the >>>>>>> <thread-pool> >>>>>>> element) should be changed. I think min-threads should be set to "2" >>>>>>> and >>>>>>> max-threads should be set to "5". Creating lots of threads can hurt >>>>>>> throughput because the JVM spends more time managing them. I would be >>>>>>> interested in hearing what others think. >>>>>>> >>>>>>> Thinking about this more, there are some other things that need to >>>>>>> be >>>>>>> >>>>>> fixed: >>>>>> >>>>>> 1. The JobPoller uses an unbounded queue. In a busy server, there is >>>>>> the >>>>>> potential the queue will grow in size until it causes an out-of-memory >>>>>> condition. >>>>>> 2. There is no accommodation for when a job cannot be added to the >>>>>> queue >>>>>> - >>>>>> it is just lost. We could add a dequeue method to the Job interface >>>>>> that >>>>>> will allow implementations to recover or reschedule the job when it >>>>>> can't >>>>>> be added to the queue. >>>>>> 3. There is a JobPoller instance per delegator, and each instance >>>>>> contains >>>>>> the number of threads configured in serviceengine.xml. With the current >>>>>> max-threads setting of 15, a multi-tenant installation with 100 tenants >>>>>> will create up to 1500 threads. (!!!) A smarter strategy might be to >>>>>> have a >>>>>> single JobPoller instance that services multiple JobManagers. >>>>>> >>>>>> -Adrian >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> |
|
|
*Adrian,
I ported our code over to the newest ofbiz trunk and tried out the changes for the new service engine. The changes that you made are working very well. I configured our server to use multiple pools and then scheduled various jobs to those pools. Servers that were not configured to service the pools left the jobs dormant and servers that were configured to service the new pools did so as expected. These new changes will work for us in our production environment. Thanks for implementing them. I now need to work on tuning the configuration settings based on the particular jobs and the capacity of our servers. I did have another question on the thread-pool configuration. Here is a sample configuration that I was using to do some of the testing. <thread-pool send-to-pool="pool" purge-job-days="4" failed-retry-min="3" ttl="120000" jobs="100" min-threads="2" max-threads="5" wait-millis="1000" poll-enabled="true" poll-db-millis="30000"> <run-from-pool name="pool"/> <run-from-pool name="testPool"/> <run-from-pool name="pool2"/> <run-from-pool name="pool3"/> <run-from-pool name="pool4"/> </thread-pool> Is the “send-to-pool” attribute the default pool that is used by the service engine for any sync and async requests through the service engine API? Is there a relationship between the “send-to-pool” attribute and the “run-from-pool” names or are they independent of each other? For example, if I don't have a run-from-pool element with a name="pool" will the default "pool" still work? Thanks again for you work on the service engine. We really appreciate it. Let me know if you need more feedback on the new changes. Brett* On Mon, Sep 17, 2012 at 3:50 PM, Adrian Crum < [hidden email]> wrote: > Brett, > > Any news on this? > > -Adrian > > > On 8/18/2012 4:11 PM, Brett Palmer wrote: > >> Adrian, >> >> I need to update our application to work with the newest code from the >> trunk. We did an update a couple months ago so I don't think it will take >> me long. I'll try to provide some feedback early next week. >> >> >> Brett >> >> On Sat, Aug 18, 2012 at 12:31 AM, Adrian Crum < >> adrian.crum@sandglass-**software.com <[hidden email]>> >> wrote: >> >> Brett, >>> >>> Any results yet? >>> >>> -Adrian >>> >>> >>> On 8/8/2012 12:57 AM, Brett Palmer wrote: >>> >>> Adrian, >>>> >>>> Thanks I'll take an update and try it out. >>>> >>>> Brett >>>> On Aug 7, 2012 4:23 PM, "Adrian Crum" <adrian.crum@sandglass-** >>>> software.com <adrian.crum@sandglass-**software.com<[hidden email]> >>>> >> >>>> wrote: >>>> >>>> Brett, >>>> >>>>> I think I solved your problems with my recent commits, ending with rev >>>>> 1370566. Let me know if it helps. >>>>> >>>>> -Adrian >>>>> >>>>> On 8/5/2012 4:53 PM, Brett Palmer wrote: >>>>> >>>>> Adrian, >>>>> >>>>>> Thanks for the update. Here are some feedback points on your listed >>>>>> items: >>>>>> >>>>>> 1. JobPoller get out-of-memor error. We've seen this a lot in >>>>>> production >>>>>> servers when the JobSandbox table is not constantly pruned of old >>>>>> records. >>>>>> It would be nice if the poller restricted its search for only >>>>>> active >>>>>> records it could process. >>>>>> >>>>>> 2. Queue for capturing missing records would be good. From item 1 >>>>>> above >>>>>> we >>>>>> have had locks on table when the poller is busy doing a scan and new >>>>>> jobs >>>>>> cannot be added or time out. >>>>>> >>>>>> Other wish items: >>>>>> >>>>>> - Ability to assign different service engines to process specific job >>>>>> types. We often multiple application servers but want to limit how >>>>>> many >>>>>> concurrent jobs are run. For example, if I had 4 app servers >>>>>> connected >>>>>> to >>>>>> the same DB I may only want one app server to service particular jobs. >>>>>> I >>>>>> thought this feature was possible but when I tried to implement it by >>>>>> changing some of the configuration files it never worked correctly. >>>>>> >>>>>> - JMS support for the service engine. It would be nice if there was a >>>>>> JMS >>>>>> interface for those that want to use JMS as their queuing mechanism >>>>>> for >>>>>> jobs. >>>>>> >>>>>> >>>>>> Brett >>>>>> >>>>>> On Sun, Aug 5, 2012 at 6:21 AM, Adrian Crum < >>>>>> adrian.crum@sandglass-****softwa**re.com <http://software.com> < >>>>>> >>>>>> adrian.crum@sandglass-**softwa**re.com <http://software.com>< >>>>>> adrian.crum@sandglass-**software.com<[hidden email]> >>>>>> > >>>>>> wrote: >>>>>> >>>>>> On 8/5/2012 11:02 AM, Adrian Crum wrote: >>>>>> >>>>>> I just committed a bunch of changes to the Job Manager group of >>>>>>> classes. >>>>>>> >>>>>>> The changes help simplify the code and hopefully make the Job >>>>>>>> Manager >>>>>>>> more >>>>>>>> robust. On the other hand, I might have broken something. ;) I will >>>>>>>> monitor >>>>>>>> the mailing list for problems. >>>>>>>> >>>>>>>> I believe the JobPoller settings in serviceengine.xml (the >>>>>>>> <thread-pool> >>>>>>>> element) should be changed. I think min-threads should be set to "2" >>>>>>>> and >>>>>>>> max-threads should be set to "5". Creating lots of threads can hurt >>>>>>>> throughput because the JVM spends more time managing them. I would >>>>>>>> be >>>>>>>> interested in hearing what others think. >>>>>>>> >>>>>>>> Thinking about this more, there are some other things that need >>>>>>>> to >>>>>>>> be >>>>>>>> >>>>>>>> fixed: >>>>>>> >>>>>>> 1. The JobPoller uses an unbounded queue. In a busy server, there is >>>>>>> the >>>>>>> potential the queue will grow in size until it causes an >>>>>>> out-of-memory >>>>>>> condition. >>>>>>> 2. There is no accommodation for when a job cannot be added to the >>>>>>> queue >>>>>>> - >>>>>>> it is just lost. We could add a dequeue method to the Job interface >>>>>>> that >>>>>>> will allow implementations to recover or reschedule the job when it >>>>>>> can't >>>>>>> be added to the queue. >>>>>>> 3. There is a JobPoller instance per delegator, and each instance >>>>>>> contains >>>>>>> the number of threads configured in serviceengine.xml. With the >>>>>>> current >>>>>>> max-threads setting of 15, a multi-tenant installation with 100 >>>>>>> tenants >>>>>>> will create up to 1500 threads. (!!!) A smarter strategy might be to >>>>>>> have a >>>>>>> single JobPoller instance that services multiple JobManagers. >>>>>>> >>>>>>> -Adrian >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> > |
|
|
Quoting Brett Palmer <[hidden email]>:
> *Adrian, > > I ported our code over to the newest ofbiz trunk and tried out the changes > for the new service engine. The changes that you made are working very > well. I configured our server to use multiple pools and then scheduled > various jobs to those pools. Servers that were not configured to service > the pools left the jobs dormant and servers that were configured to service > the new pools did so as expected. To be clear, that has always been the behavior of the Job Scheduler. I did not change that. > > These new changes will work for us in our production environment. Thanks > for implementing them. I now need to work on tuning the configuration > settings based on the particular jobs and the capacity of our servers. The changes were intended to prevent the Job Scheduler from saturating the server under heavy load. So, those are the results I would be interested in hearing about. > > I did have another question on the thread-pool configuration. Here is a > sample configuration that I was using to do some of the testing. > > <thread-pool send-to-pool="pool" > purge-job-days="4" > failed-retry-min="3" > ttl="120000" > jobs="100" > min-threads="2" > max-threads="5" > wait-millis="1000" > poll-enabled="true" > poll-db-millis="30000"> > <run-from-pool name="pool"/> > <run-from-pool name="testPool"/> > <run-from-pool name="pool2"/> > <run-from-pool name="pool3"/> > <run-from-pool name="pool4"/> > </thread-pool> > > Is the “send-to-pool” attribute the default pool that is used by the > service engine for any sync and async requests through the service engine > API? Correct. > > Is there a relationship between the “send-to-pool” attribute and the > “run-from-pool” names or are they independent of each other? For example, > if I don't have a run-from-pool element with a name="pool" will the default > "pool" still work? They are not related. If you don't have a <run-from-pool name="pool"/> element somewhere, then jobs sent to the "pool" pool will not be serviced. > > Thanks again for you work on the service engine. We really appreciate it. > > Let me know if you need more feedback on the new changes. > > > > Brett* > > On Mon, Sep 17, 2012 at 3:50 PM, Adrian Crum < > [hidden email]> wrote: > >> Brett, >> >> Any news on this? >> >> -Adrian >> >> >> On 8/18/2012 4:11 PM, Brett Palmer wrote: >> >>> Adrian, >>> >>> I need to update our application to work with the newest code from the >>> trunk. We did an update a couple months ago so I don't think it will take >>> me long. I'll try to provide some feedback early next week. >>> >>> >>> Brett >>> >>> On Sat, Aug 18, 2012 at 12:31 AM, Adrian Crum < >>> adrian.crum@sandglass-**software.com <[hidden email]>> >>> wrote: >>> >>> Brett, >>>> >>>> Any results yet? >>>> >>>> -Adrian >>>> >>>> >>>> On 8/8/2012 12:57 AM, Brett Palmer wrote: >>>> >>>> Adrian, >>>>> >>>>> Thanks I'll take an update and try it out. >>>>> >>>>> Brett >>>>> On Aug 7, 2012 4:23 PM, "Adrian Crum" <adrian.crum@sandglass-** >>>>> software.com >>>>> <adrian.crum@sandglass-**software.com<[hidden email]> >>>>> >> >>>>> wrote: >>>>> >>>>> Brett, >>>>> >>>>>> I think I solved your problems with my recent commits, ending with rev >>>>>> 1370566. Let me know if it helps. >>>>>> >>>>>> -Adrian >>>>>> >>>>>> On 8/5/2012 4:53 PM, Brett Palmer wrote: >>>>>> >>>>>> Adrian, >>>>>> >>>>>>> Thanks for the update. Here are some feedback points on your listed >>>>>>> items: >>>>>>> >>>>>>> 1. JobPoller get out-of-memor error. We've seen this a lot in >>>>>>> production >>>>>>> servers when the JobSandbox table is not constantly pruned of old >>>>>>> records. >>>>>>> It would be nice if the poller restricted its search for only >>>>>>> active >>>>>>> records it could process. >>>>>>> >>>>>>> 2. Queue for capturing missing records would be good. From item 1 >>>>>>> above >>>>>>> we >>>>>>> have had locks on table when the poller is busy doing a scan and new >>>>>>> jobs >>>>>>> cannot be added or time out. >>>>>>> >>>>>>> Other wish items: >>>>>>> >>>>>>> - Ability to assign different service engines to process specific job >>>>>>> types. We often multiple application servers but want to limit how >>>>>>> many >>>>>>> concurrent jobs are run. For example, if I had 4 app servers >>>>>>> connected >>>>>>> to >>>>>>> the same DB I may only want one app server to service particular jobs. >>>>>>> I >>>>>>> thought this feature was possible but when I tried to implement it by >>>>>>> changing some of the configuration files it never worked correctly. >>>>>>> >>>>>>> - JMS support for the service engine. It would be nice if there was a >>>>>>> JMS >>>>>>> interface for those that want to use JMS as their queuing mechanism >>>>>>> for >>>>>>> jobs. >>>>>>> >>>>>>> >>>>>>> Brett >>>>>>> >>>>>>> On Sun, Aug 5, 2012 at 6:21 AM, Adrian Crum < >>>>>>> adrian.crum@sandglass-****softwa**re.com <http://software.com> < >>>>>>> >>>>>>> adrian.crum@sandglass-**softwa**re.com <http://software.com>< >>>>>>> adrian.crum@sandglass-**software.com<[hidden email]> >>>>>>> > >>>>>>> wrote: >>>>>>> >>>>>>> On 8/5/2012 11:02 AM, Adrian Crum wrote: >>>>>>> >>>>>>> I just committed a bunch of changes to the Job Manager group of >>>>>>>> classes. >>>>>>>> >>>>>>>> The changes help simplify the code and hopefully make the Job >>>>>>>>> Manager >>>>>>>>> more >>>>>>>>> robust. On the other hand, I might have broken something. ;) I will >>>>>>>>> monitor >>>>>>>>> the mailing list for problems. >>>>>>>>> >>>>>>>>> I believe the JobPoller settings in serviceengine.xml (the >>>>>>>>> <thread-pool> >>>>>>>>> element) should be changed. I think min-threads should be set to "2" >>>>>>>>> and >>>>>>>>> max-threads should be set to "5". Creating lots of threads can hurt >>>>>>>>> throughput because the JVM spends more time managing them. I would >>>>>>>>> be >>>>>>>>> interested in hearing what others think. >>>>>>>>> >>>>>>>>> Thinking about this more, there are some other things that need >>>>>>>>> to >>>>>>>>> be >>>>>>>>> >>>>>>>>> fixed: >>>>>>>> >>>>>>>> 1. The JobPoller uses an unbounded queue. In a busy server, there is >>>>>>>> the >>>>>>>> potential the queue will grow in size until it causes an >>>>>>>> out-of-memory >>>>>>>> condition. >>>>>>>> 2. There is no accommodation for when a job cannot be added to the >>>>>>>> queue >>>>>>>> - >>>>>>>> it is just lost. We could add a dequeue method to the Job interface >>>>>>>> that >>>>>>>> will allow implementations to recover or reschedule the job when it >>>>>>>> can't >>>>>>>> be added to the queue. >>>>>>>> 3. There is a JobPoller instance per delegator, and each instance >>>>>>>> contains >>>>>>>> the number of threads configured in serviceengine.xml. With the >>>>>>>> current >>>>>>>> max-threads setting of 15, a multi-tenant installation with 100 >>>>>>>> tenants >>>>>>>> will create up to 1500 threads. (!!!) A smarter strategy might be to >>>>>>>> have a >>>>>>>> single JobPoller instance that services multiple JobManagers. >>>>>>>> >>>>>>>> -Adrian >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >> > |
|
|
Adrian,
I'll have to setup a load test to verify that a large number of pending jobs in the JobSandbox do not saturate the server. I'll also work with the thread counts to see what works for us. Before I read your reply I had configured the service engine to use a "send-to-pool=poolX" but did not configure any "run-from-pool" names that matched "poolX". I then use the webtools interface to run a service. The default pool displayed was "poolX" in the form. I entered the service name and executed the service. The service ran but I didn't expect it to run because there wasn't a "send-to-pool" with a matching name. This functionality seems to disagree with what you said about "If you don't have a <run-from-pool name="pool"/> element somewhere, then jobs sent to the "pool" pool will not be serviced.", but maybe I am missing something. <thread-pool send-to-pool="poolX" purge-job-days="4" failed-retry-min="3" ttl="120000" jobs="100" min-threads="2" max-threads="5" wait-millis="1000" poll-enabled="true" poll-db-millis="30000"> <run-from-pool name="pool"/> <run-from-pool name="testPool"/> <run-from-pool name="pool2"/> <run-from-pool name="pool3"/> <run-from-pool name="pool4"/> </thread-pool> I'll post the results of my load test as soon as it is completed. Brett On Tue, Sep 18, 2012 at 2:38 AM, <[hidden email]> wrote: > Quoting Brett Palmer <[hidden email]>: > > *Adrian, >> >> >> I ported our code over to the newest ofbiz trunk and tried out the changes >> for the new service engine. The changes that you made are working very >> well. I configured our server to use multiple pools and then scheduled >> various jobs to those pools. Servers that were not configured to service >> the pools left the jobs dormant and servers that were configured to >> service >> the new pools did so as expected. >> > > To be clear, that has always been the behavior of the Job Scheduler. I did > not change that. > > > >> These new changes will work for us in our production environment. Thanks >> for implementing them. I now need to work on tuning the configuration >> settings based on the particular jobs and the capacity of our servers. >> > > The changes were intended to prevent the Job Scheduler from saturating the > server under heavy load. So, those are the results I would be interested in > hearing about. > > > >> I did have another question on the thread-pool configuration. Here is a >> sample configuration that I was using to do some of the testing. >> >> <thread-pool send-to-pool="pool" >> purge-job-days="4" >> failed-retry-min="3" >> ttl="120000" >> jobs="100" >> min-threads="2" >> max-threads="5" >> wait-millis="1000" >> poll-enabled="true" >> poll-db-millis="30000"> >> <run-from-pool name="pool"/> >> <run-from-pool name="testPool"/> >> <run-from-pool name="pool2"/> >> <run-from-pool name="pool3"/> >> <run-from-pool name="pool4"/> >> </thread-pool> >> >> Is the “send-to-pool” attribute the default pool that is used by the >> service engine for any sync and async requests through the service engine >> API? >> > > Correct. > > > >> Is there a relationship between the “send-to-pool” attribute and the >> “run-from-pool” names or are they independent of each other? For example, >> if I don't have a run-from-pool element with a name="pool" will the >> default >> "pool" still work? >> > > They are not related. If you don't have a > > <run-from-pool name="pool"/> > > element somewhere, then jobs sent to the "pool" pool will not be serviced. > > >> Thanks again for you work on the service engine. We really appreciate it. >> >> Let me know if you need more feedback on the new changes. >> >> >> >> Brett* >> >> >> |
|
|
Brett,
Thank you for the update. I will look into this. -Adrian On 9/19/2012 6:02 AM, Brett Palmer wrote: > Adrian, > > I'll have to setup a load test to verify that a large number of pending > jobs in the JobSandbox do not saturate the server. I'll also work with the > thread counts to see what works for us. > > Before I read your reply I had configured the service engine to use a > "send-to-pool=poolX" but did not configure any "run-from-pool" names that > matched "poolX". I then use the webtools interface to run a service. The > default pool displayed was "poolX" in the form. I entered the service name > and executed the service. The service ran but I didn't expect it to run > because there wasn't a "send-to-pool" with a matching name. This > functionality seems to disagree with what you said about > > "If you don't have a > <run-from-pool name="pool"/> > element somewhere, then jobs sent to the "pool" pool will not be > serviced.", but maybe I am missing something. > > > <thread-pool send-to-pool="poolX" > purge-job-days="4" > failed-retry-min="3" > ttl="120000" > jobs="100" > min-threads="2" > max-threads="5" > wait-millis="1000" > poll-enabled="true" > poll-db-millis="30000"> > <run-from-pool name="pool"/> > <run-from-pool name="testPool"/> > <run-from-pool name="pool2"/> > <run-from-pool name="pool3"/> > <run-from-pool name="pool4"/> > </thread-pool> > > I'll post the results of my load test as soon as it is completed. > > > Brett > > > On Tue, Sep 18, 2012 at 2:38 AM, <[hidden email]> wrote: > >> Quoting Brett Palmer <[hidden email]>: >> >> *Adrian, >>> >>> I ported our code over to the newest ofbiz trunk and tried out the changes >>> for the new service engine. The changes that you made are working very >>> well. I configured our server to use multiple pools and then scheduled >>> various jobs to those pools. Servers that were not configured to service >>> the pools left the jobs dormant and servers that were configured to >>> service >>> the new pools did so as expected. >>> >> To be clear, that has always been the behavior of the Job Scheduler. I did >> not change that. >> >> >> >>> These new changes will work for us in our production environment. Thanks >>> for implementing them. I now need to work on tuning the configuration >>> settings based on the particular jobs and the capacity of our servers. >>> >> The changes were intended to prevent the Job Scheduler from saturating the >> server under heavy load. So, those are the results I would be interested in >> hearing about. >> >> >> >>> I did have another question on the thread-pool configuration. Here is a >>> sample configuration that I was using to do some of the testing. >>> >>> <thread-pool send-to-pool="pool" >>> purge-job-days="4" >>> failed-retry-min="3" >>> ttl="120000" >>> jobs="100" >>> min-threads="2" >>> max-threads="5" >>> wait-millis="1000" >>> poll-enabled="true" >>> poll-db-millis="30000"> >>> <run-from-pool name="pool"/> >>> <run-from-pool name="testPool"/> >>> <run-from-pool name="pool2"/> >>> <run-from-pool name="pool3"/> >>> <run-from-pool name="pool4"/> >>> </thread-pool> >>> >>> Is the “send-to-pool” attribute the default pool that is used by the >>> service engine for any sync and async requests through the service engine >>> API? >>> >> Correct. >> >> >> >>> Is there a relationship between the “send-to-pool” attribute and the >>> “run-from-pool” names or are they independent of each other? For example, >>> if I don't have a run-from-pool element with a name="pool" will the >>> default >>> "pool" still work? >>> >> They are not related. If you don't have a >> >> <run-from-pool name="pool"/> >> >> element somewhere, then jobs sent to the "pool" pool will not be serviced. >> >> >>> Thanks again for you work on the service engine. We really appreciate it. >>> >>> Let me know if you need more feedback on the new changes. >>> >>> >>> >>> Brett* >>> >>> >>> > snip > |
Administrator
|
In reply to this post by Adrian Crum-3
From: <[hidden email]>
> Quoting Brett Palmer <[hidden email]>: >> Is there a relationship between the “send-to-pool” attribute and the >> “run-from-pool” names or are they independent of each other? For example, >> if I don't have a run-from-pool element with a name="pool" will the default >> "pool" still work? > > They are not related. If you don't have a > > <run-from-pool name="pool"/> > > element somewhere, then jobs sent to the "pool" pool will not be serviced. And you must have poll-enabled="true" for this/these to work Jacques |
«
Return to OFBiz - Dev
|
1 view|%1 views
| Free forum by Nabble | Edit this page |