Async Service Thread Locking

[Ryan Moriarty]

I'm having difficulty with asynchronous jobs not completing. Many jobs
finish normally, but most show "Running" into perpetuity. I see no pattern
in services that finish and services that continue running indefinitely. I
have five invoker threads that appear to always be sleeping. We're using a
single-server config. This is my thread pool config:

 <thread-pool send-to-pool="pool"
                     purge-job-days="4"
                     failed-retry-min="3"
                     ttl="18000000"
                     wait-millis="750"
                     jobs="10"
                     min-threads="5"
                     max-threads="15"
                     poll-enabled="true"
                     poll-db-millis="20000">
            <run-from-pool name="pool"/>

I saw a message that suggested I would be out of heap space, but my max
heap size is 5280M, and we have only ~100 concurrent users, so that doesn't
seem like an issue. Could it be a deadlock? How would I resolve that? Any
suggestions are welcome.

Ryan Moriarty
Software Developer
Able Aerospace Services

[Jacques Le Roux]

Le 31/01/2017 à 21:24, Ryan Moriarty a écrit :

> I'm having difficulty with asynchronous jobs not completing. Many jobs
> finish normally, but most show "Running" into perpetuity. I see no pattern
> in services that finish and services that continue running indefinitely. I
> have five invoker threads that appear to always be sleeping. We're using a
> single-server config. This is my thread pool config:
>
>   <thread-pool send-to-pool="pool"
>                       purge-job-days="4"
>                       failed-retry-min="3"
>                       ttl="18000000"
>                       wait-millis="750"
>                       jobs="10"
>                       min-threads="5"
>                       max-threads="15"
>                       poll-enabled="true"
>                       poll-db-millis="20000">
>              <run-from-pool name="pool"/>
>
> I saw a message that suggested I would be out of heap space, but my max
> heap size is 5280M, and we have only ~100 concurrent users, so that doesn't
> seem like an issue. Could it be a deadlock? How would I resolve that? Any
> suggestions are welcome.
>
> Ryan Moriarty
> Software Developer
> Able Aerospace Services
>

Hi Ryan,

Which OFBiz version are you using?

With jobs=10, don't you ever get "Unable to queue job" exceptions?

And why did you change the default setting at large?

Jacques

[Ryan Moriarty]

This is OFBiz 10.04. Quite old, I know, but I'm stuck for the moment.

I haven't seen any errors about being able to queue jobs, but I may be
searching for the wrong text

The settings are as they are when I inherited the project, I can't speak to
why they are as they are.

On Tue, Jan 31, 2017 at 10:25 PM, Jacques Le Roux <
[hidden email]> wrote:

> Le 31/01/2017 à 21:24, Ryan Moriarty a écrit :
>
>> I'm having difficulty with asynchronous jobs not completing. Many jobs
>> finish normally, but most show "Running" into perpetuity. I see no pattern
>> in services that finish and services that continue running indefinitely. I
>> have five invoker threads that appear to always be sleeping. We're using a
>> single-server config. This is my thread pool config:
>>
>>   <thread-pool send-to-pool="pool"
>>                       purge-job-days="4"
>>                       failed-retry-min="3"
>>                       ttl="18000000"
>>                       wait-millis="750"
>>                       jobs="10"
>>                       min-threads="5"
>>                       max-threads="15"
>>                       poll-enabled="true"
>>                       poll-db-millis="20000">
>>              <run-from-pool name="pool"/>
>>
>> I saw a message that suggested I would be out of heap space, but my max
>> heap size is 5280M, and we have only ~100 concurrent users, so that
>> doesn't
>> seem like an issue. Could it be a deadlock? How would I resolve that? Any
>> suggestions are welcome.
>>
>> Ryan Moriarty
>> Software Developer
>> Able Aerospace Services
>>
>>
> Hi Ryan,
>
> Which OFBiz version are you using?
>
> With jobs=10, don't you ever get "Unable to queue job" exceptions?
>
> And why did you change the default setting at large?
>
> Jacques
>
>

[Jacques Le Roux]

I'd try 1st to increase the number of jobs possibly running at the same time, ie change
jobs="10"
to
jobs="100"

And even compare with the current setting in trunk:

<thread-pool send-to-pool="pool"
purge-job-days="4"
failed-retry-min="3"
ttl="120000"
jobs="100"
min-threads="2"
max-threads="5"
poll-enabled="true"
poll-db-millis="30000">
<run-from-pool name="pool"/>
</thread-pool>

HTH

Jacques

Le 01/02/2017 à 18:39, Ryan Moriarty a écrit :

> This is OFBiz 10.04. Quite old, I know, but I'm stuck for the moment.
>
> I haven't seen any errors about being able to queue jobs, but I may be
> searching for the wrong text
>
> The settings are as they are when I inherited the project, I can't speak to
> why they are as they are.
>
> On Tue, Jan 31, 2017 at 10:25 PM, Jacques Le Roux <
> [hidden email]> wrote:
>
>> Le 31/01/2017 à 21:24, Ryan Moriarty a écrit :
>>
>>> I'm having difficulty with asynchronous jobs not completing. Many jobs
>>> finish normally, but most show "Running" into perpetuity. I see no pattern
>>> in services that finish and services that continue running indefinitely. I
>>> have five invoker threads that appear to always be sleeping. We're using a
>>> single-server config. This is my thread pool config:
>>>
>>>    <thread-pool send-to-pool="pool"
>>>                        purge-job-days="4"
>>>                        failed-retry-min="3"
>>>                        ttl="18000000"
>>>                        wait-millis="750"
>>>                        jobs="10"
>>>                        min-threads="5"
>>>                        max-threads="15"
>>>                        poll-enabled="true"
>>>                        poll-db-millis="20000">
>>>               <run-from-pool name="pool"/>
>>>
>>> I saw a message that suggested I would be out of heap space, but my max
>>> heap size is 5280M, and we have only ~100 concurrent users, so that
>>> doesn't
>>> seem like an issue. Could it be a deadlock? How would I resolve that? Any
>>> suggestions are welcome.
>>>
>>> Ryan Moriarty
>>> Software Developer
>>> Able Aerospace Services
>>>
>>>
>> Hi Ryan,
>>
>> Which OFBiz version are you using?
>>
>> With jobs=10, don't you ever get "Unable to queue job" exceptions?
>>
>> And why did you change the default setting at large?
>>
>> Jacques
>>
>>
>

[Nicolas Malin-2]

Hello Ryan;

Yes 10.04 it's old.

 From my experience :

* check you crashed service : when you restart your OFBiz, if you have
some scheduled service, the "crash"  duplicate them

* check you java heap space with jmx because when a thread overload
ressources or something like that you have an unexpected result on job
manager

* prefer to use more than one pool if you have too many async service.

* Warning on huge async service. On a big project I down the jobs
authorize on running at only 2. I prefer load the queue instead of
overload  the server. And if you service queue is too high, deploy a new
ofbiz to help this pool ;)

Nicolas


Le 01/02/2017 à 18:39, Ryan Moriarty a écrit :

> This is OFBiz 10.04. Quite old, I know, but I'm stuck for the moment.
>
> I haven't seen any errors about being able to queue jobs, but I may be
> searching for the wrong text
>
> The settings are as they are when I inherited the project, I can't speak to
> why they are as they are.
>
> On Tue, Jan 31, 2017 at 10:25 PM, Jacques Le Roux <
> [hidden email]> wrote:
>
>> Le 31/01/2017 à 21:24, Ryan Moriarty a écrit :
>>
>>> I'm having difficulty with asynchronous jobs not completing. Many jobs
>>> finish normally, but most show "Running" into perpetuity. I see no pattern
>>> in services that finish and services that continue running indefinitely. I
>>> have five invoker threads that appear to always be sleeping. We're using a
>>> single-server config. This is my thread pool config:
>>>
>>>    <thread-pool send-to-pool="pool"
>>>                        purge-job-days="4"
>>>                        failed-retry-min="3"
>>>                        ttl="18000000"
>>>                        wait-millis="750"
>>>                        jobs="10"
>>>                        min-threads="5"
>>>                        max-threads="15"
>>>                        poll-enabled="true"
>>>                        poll-db-millis="20000">
>>>               <run-from-pool name="pool"/>
>>>
>>> I saw a message that suggested I would be out of heap space, but my max
>>> heap size is 5280M, and we have only ~100 concurrent users, so that
>>> doesn't
>>> seem like an issue. Could it be a deadlock? How would I resolve that? Any
>>> suggestions are welcome.
>>>
>>> Ryan Moriarty
>>> Software Developer
>>> Able Aerospace Services
>>>
>>>
>> Hi Ryan,
>>
>> Which OFBiz version are you using?
>>
>> With jobs=10, don't you ever get "Unable to queue job" exceptions?
>>
>> And why did you change the default setting at large?
>>
>> Jacques
>>
>>

[Jacopo Cappellato-5]

Hi Ryan,

On Tue, Jan 31, 2017 at 9:24 PM, Ryan Moriarty <
[hidden email]> wrote:

> I'm having difficulty with asynchronous jobs not completing. Many jobs
> finish normally, but most show "Running" into perpetuity. I see no pattern
> in services that finish and services that continue running indefinitely.


Does it happen with different services? Or all the jobs stuck in the
"Running" state are executions of the same service?
It may be really useful to extract and analyze the logs of a job execution
that is stuck.

Jacopo