Async persist service on error restart indefinitely by default

[Nicolas Malin-2]

Hi,

On a customer site, we have huge services that call different rest api
to collect information
To increase the velocity we run all them by persistence asynchrone then
the job pooler manage them with available resources.

The problem is, when a call failed and the service threw an error, the
service engine reschedule it, ... and it failed, rescheduled, failed,
rescheduled, failed ... with beautiful result to overload your pool with
zombie services.

The solution is easy, set on your service definition attribute max-retry
to 0 (or 1, if you want one retry) but I didn't understand why we have
this configuration to reschedule indefinitely a service if it is in error.

This configuration exists before apache migration so I'd happy to have
your vision about this.
 From my view, I'm in favor to set max retry to 0 by default and left
the developer set him self when he wants that a service restart after a
failure.

Easy change  :
Index:
framework/service/src/main/java/org/apache/ofbiz/service/job/PersistedServiceJob.java

@@ -80,7 +80,7 @@
          this.jobValue = jobValue;
          Timestamp storedDate = jobValue.getTimestamp("runTime");
          this.startTime = storedDate.getTime();
-        this.maxRetry = jobValue.get("maxRetry") != null ?
jobValue.getLong("maxRetry") : -1;
+        this.maxRetry = jobValue.get("maxRetry") != null ?
jobValue.getLong("maxRetry") : 0;

Nicolas

--
logoNrd <https://nereide.fr/>
        Nicolas Malin
The apache way <http://theapacheway.com/> : *Charity* Apache’s mission
is providing software for the public good.
[hidden email]
8 rue des Déportés 37000 TOURS, 02 47 50 30 54

Apache OFBiz <http://ofbiz.apache.org/>|The Apache Way
<http://theapacheway.com/>|réseau LE <http://www.libre-entreprise.org/>

[taher]

I could be wrong, but, wouldn't it make sense that if your service is
failing continuously then perhaps something is wrong with the service?
I would imagine that perhaps resilience in the design of the service
might be the better route?
On Fri, Sep 7, 2018 at 6:22 PM Nicolas Malin <[hidden email]> wrote:

>
> Hi,
>
> On a customer site, we have huge services that call different rest api
> to collect information
> To increase the velocity we run all them by persistence asynchrone then
> the job pooler manage them with available resources.
>
> The problem is, when a call failed and the service threw an error, the
> service engine reschedule it, ... and it failed, rescheduled, failed,
> rescheduled, failed ... with beautiful result to overload your pool with
> zombie services.
>
> The solution is easy, set on your service definition attribute max-retry
> to 0 (or 1, if you want one retry) but I didn't understand why we have
> this configuration to reschedule indefinitely a service if it is in error.
>
> This configuration exists before apache migration so I'd happy to have
> your vision about this.
>  From my view, I'm in favor to set max retry to 0 by default and left
> the developer set him self when he wants that a service restart after a
> failure.
>
> Easy change  :
> Index:
> framework/service/src/main/java/org/apache/ofbiz/service/job/PersistedServiceJob.java
>
> @@ -80,7 +80,7 @@
>           this.jobValue = jobValue;
>           Timestamp storedDate = jobValue.getTimestamp("runTime");
>           this.startTime = storedDate.getTime();
> -        this.maxRetry = jobValue.get("maxRetry") != null ?
> jobValue.getLong("maxRetry") : -1;
> +        this.maxRetry = jobValue.get("maxRetry") != null ?
> jobValue.getLong("maxRetry") : 0;
>
> Nicolas
>
> --
> logoNrd <https://nereide.fr/>
>         Nicolas Malin
> The apache way <http://theapacheway.com/> : *Charity* Apache’s mission
> is providing software for the public good.
> [hidden email]
> 8 rue des Déportés 37000 TOURS, 02 47 50 30 54
>
> Apache OFBiz <http://ofbiz.apache.org/>|The Apache Way
> <http://theapacheway.com/>|réseau LE <http://www.libre-entreprise.org/>

[Nicolas Malin-2]

On 08/09/2018 18:43, Taher Alkhateeb wrote:
> I could be wrong, but, wouldn't it make sense that if your service is
> failing continuously then perhaps something is wrong with the service?
> I would imagine that perhaps resilience in the design of the service
> might be the better route?
Yeah sure Taher, reanalyze each reason to improve these services is also
a solution :) but not exactly ma question.
At the beginning theses services has been called on sync to keep the
rest api error contacted and work fine. But when we moved it on async
and use parallelism process through the job manager the service error
has been translate as restart the service.
So yes I can redefine them but *by default* is it logical to restart
indefinitely a async service that failed ?

Nicolas

> On Fri, Sep 7, 2018 at 6:22 PM Nicolas Malin <[hidden email]> wrote:
>> Hi,
>>
>> On a customer site, we have huge services that call different rest api
>> to collect information
>> To increase the velocity we run all them by persistence asynchrone then
>> the job pooler manage them with available resources.
>>
>> The problem is, when a call failed and the service threw an error, the
>> service engine reschedule it, ... and it failed, rescheduled, failed,
>> rescheduled, failed ... with beautiful result to overload your pool with
>> zombie services.
>>
>> The solution is easy, set on your service definition attribute max-retry
>> to 0 (or 1, if you want one retry) but I didn't understand why we have
>> this configuration to reschedule indefinitely a service if it is in error.
>>
>> This configuration exists before apache migration so I'd happy to have
>> your vision about this.
>>   From my view, I'm in favor to set max retry to 0 by default and left
>> the developer set him self when he wants that a service restart after a
>> failure.
>>
>> Easy change  :
>> Index:
>> framework/service/src/main/java/org/apache/ofbiz/service/job/PersistedServiceJob.java
>>
>> @@ -80,7 +80,7 @@
>>            this.jobValue = jobValue;
>>            Timestamp storedDate = jobValue.getTimestamp("runTime");
>>            this.startTime = storedDate.getTime();
>> -        this.maxRetry = jobValue.get("maxRetry") != null ?
>> jobValue.getLong("maxRetry") : -1;
>> +        this.maxRetry = jobValue.get("maxRetry") != null ?
>> jobValue.getLong("maxRetry") : 0;
>>
>> Nicolas
>>
>> --
>> logoNrd <https://nereide.fr/>
>>          Nicolas Malin
>> The apache way <http://theapacheway.com/> : *Charity* Apache’s mission
>> is providing software for the public good.
>> [hidden email]
>> 8 rue des Déportés 37000 TOURS, 02 47 50 30 54
>>
>> Apache OFBiz <http://ofbiz.apache.org/>|The Apache Way
>> <http://theapacheway.com/>|réseau LE <http://www.libre-entreprise.org/>

[Jacques Le Roux]

Maybe we could consider if it's a sync or async service and have 2 different default settings?

Just an idea from the top of my head, no more thinking ;)

Jacques


Le 10/09/2018 à 09:17, Nicolas Malin a écrit :

> On 08/09/2018 18:43, Taher Alkhateeb wrote:
>> I could be wrong, but, wouldn't it make sense that if your service is
>> failing continuously then perhaps something is wrong with the service?
>> I would imagine that perhaps resilience in the design of the service
>> might be the better route?
> Yeah sure Taher, reanalyze each reason to improve these services is also a solution :) but not exactly ma question.
> At the beginning theses services has been called on sync to keep the rest api error contacted and work fine. But when we moved it on async and use
> parallelism process through the job manager the service error has been translate as restart the service.
> So yes I can redefine them but *by default* is it logical to restart indefinitely a async service that failed ?
>
> Nicolas
>> On Fri, Sep 7, 2018 at 6:22 PM Nicolas Malin <[hidden email]> wrote:
>>> Hi,
>>>
>>> On a customer site, we have huge services that call different rest api
>>> to collect information
>>> To increase the velocity we run all them by persistence asynchrone then
>>> the job pooler manage them with available resources.
>>>
>>> The problem is, when a call failed and the service threw an error, the
>>> service engine reschedule it, ... and it failed, rescheduled, failed,
>>> rescheduled, failed ... with beautiful result to overload your pool with
>>> zombie services.
>>>
>>> The solution is easy, set on your service definition attribute max-retry
>>> to 0 (or 1, if you want one retry) but I didn't understand why we have
>>> this configuration to reschedule indefinitely a service if it is in error.
>>>
>>> This configuration exists before apache migration so I'd happy to have
>>> your vision about this.
>>>   From my view, I'm in favor to set max retry to 0 by default and left
>>> the developer set him self when he wants that a service restart after a
>>> failure.
>>>
>>> Easy change  :
>>> Index:
>>> framework/service/src/main/java/org/apache/ofbiz/service/job/PersistedServiceJob.java
>>>
>>> @@ -80,7 +80,7 @@
>>>            this.jobValue = jobValue;
>>>            Timestamp storedDate = jobValue.getTimestamp("runTime");
>>>            this.startTime = storedDate.getTime();
>>> -        this.maxRetry = jobValue.get("maxRetry") != null ?
>>> jobValue.getLong("maxRetry") : -1;
>>> +        this.maxRetry = jobValue.get("maxRetry") != null ?
>>> jobValue.getLong("maxRetry") : 0;
>>>
>>> Nicolas
>>>
>>> --
>>> logoNrd <https://nereide.fr/>
>>>          Nicolas Malin
>>> The apache way <http://theapacheway.com/> : *Charity* Apache’s mission
>>> is providing software for the public good.
>>> [hidden email]
>>> 8 rue des Déportés 37000 TOURS, 02 47 50 30 54
>>>
>>> Apache OFBiz <http://ofbiz.apache.org/>|The Apache Way
>>> <http://theapacheway.com/>|réseau LE <http://www.libre-entreprise.org/>
>
>

[Suraj Khurana]

Hi Nicolas,

+1 for this proposal to set 0 as default value irrespective of its mode
(sync or async).

--
Best Regards,
Suraj Khurana
Omnichannel OMS Technical Expert
HotWax Systems
m: [phone]



On Mon, Sep 10, 2018 at 2:33 PM, Jacques Le Roux <
[hidden email]> wrote:

> Maybe we could consider if it's a sync or async service and have 2
> different default settings?
>
> Just an idea from the top of my head, no more thinking ;)
>
> Jacques
>
>
>
> Le 10/09/2018 à 09:17, Nicolas Malin a écrit :
>
>> On 08/09/2018 18:43, Taher Alkhateeb wrote:
>>
>>> I could be wrong, but, wouldn't it make sense that if your service is
>>> failing continuously then perhaps something is wrong with the service?
>>> I would imagine that perhaps resilience in the design of the service
>>> might be the better route?
>>>
>> Yeah sure Taher, reanalyze each reason to improve these services is also
>> a solution :) but not exactly ma question.
>> At the beginning theses services has been called on sync to keep the rest
>> api error contacted and work fine. But when we moved it on async and use
>> parallelism process through the job manager the service error has been
>> translate as restart the service.
>> So yes I can redefine them but *by default* is it logical to restart
>> indefinitely a async service that failed ?
>>
>> Nicolas
>>
>>> On Fri, Sep 7, 2018 at 6:22 PM Nicolas Malin <[hidden email]>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> On a customer site, we have huge services that call different rest api
>>>> to collect information
>>>> To increase the velocity we run all them by persistence asynchrone then
>>>> the job pooler manage them with available resources.
>>>>
>>>> The problem is, when a call failed and the service threw an error, the
>>>> service engine reschedule it, ... and it failed, rescheduled, failed,
>>>> rescheduled, failed ... with beautiful result to overload your pool with
>>>> zombie services.
>>>>
>>>> The solution is easy, set on your service definition attribute max-retry
>>>> to 0 (or 1, if you want one retry) but I didn't understand why we have
>>>> this configuration to reschedule indefinitely a service if it is in
>>>> error.
>>>>
>>>> This configuration exists before apache migration so I'd happy to have
>>>> your vision about this.
>>>>   From my view, I'm in favor to set max retry to 0 by default and left
>>>> the developer set him self when he wants that a service restart after a
>>>> failure.
>>>>
>>>> Easy change  :
>>>> Index:
>>>> framework/service/src/main/java/org/apache/ofbiz/service/job
>>>> /PersistedServiceJob.java
>>>>
>>>> @@ -80,7 +80,7 @@
>>>>            this.jobValue = jobValue;
>>>>            Timestamp storedDate = jobValue.getTimestamp("runTime");
>>>>            this.startTime = storedDate.getTime();
>>>> -        this.maxRetry = jobValue.get("maxRetry") != null ?
>>>> jobValue.getLong("maxRetry") : -1;
>>>> +        this.maxRetry = jobValue.get("maxRetry") != null ?
>>>> jobValue.getLong("maxRetry") : 0;
>>>>
>>>> Nicolas
>>>>
>>>> --
>>>> logoNrd <https://nereide.fr/>
>>>>          Nicolas Malin
>>>> The apache way <http://theapacheway.com/> : *Charity* Apache’s mission
>>>> is providing software for the public good.
>>>> [hidden email]
>>>> 8 rue des Déportés 37000 TOURS, 02 47 50 30 54
>>>>
>>>> Apache OFBiz <http://ofbiz.apache.org/>|The Apache Way
>>>> <http://theapacheway.com/>|réseau LE <http://www.libre-entreprise.org/>
>>>>
>>>
>>
>>
>

[Rishi Solanki]

+1 Nicolas, let the developer decide how she would like to manage it.

--
Rishi Solanki
Sr Manager, Enterprise Software Development
HotWax Systems Pvt. Ltd.
Direct: [phone]
http://www.hotwaxsystems.com
www.hotwax.co


On Mon, Sep 10, 2018 at 2:55 PM Suraj Khurana <
[hidden email]> wrote:

> Hi Nicolas,
>
> +1 for this proposal to set 0 as default value irrespective of its mode
> (sync or async).
>
> --
> Best Regards,
> Suraj Khurana
> Omnichannel OMS Technical Expert
> HotWax Systems
> m: [phone]
>
>
>
> On Mon, Sep 10, 2018 at 2:33 PM, Jacques Le Roux <
> [hidden email]> wrote:
>
> > Maybe we could consider if it's a sync or async service and have 2
> > different default settings?
> >
> > Just an idea from the top of my head, no more thinking ;)
> >
> > Jacques
> >
> >
> >
> > Le 10/09/2018 à 09:17, Nicolas Malin a écrit :
> >
> >> On 08/09/2018 18:43, Taher Alkhateeb wrote:
> >>
> >>> I could be wrong, but, wouldn't it make sense that if your service is
> >>> failing continuously then perhaps something is wrong with the service?
> >>> I would imagine that perhaps resilience in the design of the service
> >>> might be the better route?
> >>>
> >> Yeah sure Taher, reanalyze each reason to improve these services is also
> >> a solution :) but not exactly ma question.
> >> At the beginning theses services has been called on sync to keep the
> rest
> >> api error contacted and work fine. But when we moved it on async and use
> >> parallelism process through the job manager the service error has been
> >> translate as restart the service.
> >> So yes I can redefine them but *by default* is it logical to restart
> >> indefinitely a async service that failed ?
> >>
> >> Nicolas
> >>
> >>> On Fri, Sep 7, 2018 at 6:22 PM Nicolas Malin <[hidden email]
> >
> >>> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> On a customer site, we have huge services that call different rest api
> >>>> to collect information
> >>>> To increase the velocity we run all them by persistence asynchrone
> then
> >>>> the job pooler manage them with available resources.
> >>>>
> >>>> The problem is, when a call failed and the service threw an error, the
> >>>> service engine reschedule it, ... and it failed, rescheduled, failed,
> >>>> rescheduled, failed ... with beautiful result to overload your pool
> with
> >>>> zombie services.
> >>>>
> >>>> The solution is easy, set on your service definition attribute
> max-retry
> >>>> to 0 (or 1, if you want one retry) but I didn't understand why we have
> >>>> this configuration to reschedule indefinitely a service if it is in
> >>>> error.
> >>>>
> >>>> This configuration exists before apache migration so I'd happy to have
> >>>> your vision about this.
> >>>>   From my view, I'm in favor to set max retry to 0 by default and left
> >>>> the developer set him self when he wants that a service restart after
> a
> >>>> failure.
> >>>>
> >>>> Easy change  :
> >>>> Index:
> >>>> framework/service/src/main/java/org/apache/ofbiz/service/job
> >>>> /PersistedServiceJob.java
> >>>>
> >>>> @@ -80,7 +80,7 @@
> >>>>            this.jobValue = jobValue;
> >>>>            Timestamp storedDate = jobValue.getTimestamp("runTime");
> >>>>            this.startTime = storedDate.getTime();
> >>>> -        this.maxRetry = jobValue.get("maxRetry") != null ?
> >>>> jobValue.getLong("maxRetry") : -1;
> >>>> +        this.maxRetry = jobValue.get("maxRetry") != null ?
> >>>> jobValue.getLong("maxRetry") : 0;
> >>>>
> >>>> Nicolas
> >>>>
> >>>> --
> >>>> logoNrd <https://nereide.fr/>
> >>>>          Nicolas Malin
> >>>> The apache way <http://theapacheway.com/> : *Charity* Apache’s
> mission
> >>>> is providing software for the public good.
> >>>> [hidden email]
> >>>> 8 rue des Déportés 37000 TOURS, 02 47 50 30 54
> >>>>
> >>>> Apache OFBiz <http://ofbiz.apache.org/>|The Apache Way
> >>>> <http://theapacheway.com/>|réseau LE <
> http://www.libre-entreprise.org/>
> >>>>
> >>>
> >>
> >>
> >
>

[Pierre Smits-3]

+1 on enabling the adopter to dictate the setting.


Best regards,

Pierre Smits

*Apache Trafodion <https://trafodion.apache.org>, Vice President*
*Apache Directory <https://directory.apache.org>, PMC Member*
Apache Incubator <https://incubator.apache.org>, committer
*Apache OFBiz <https://ofbiz.apache.org>, contributor (without privileges)
since 2008*
Apache Steve <https://steve.apache.org>, committer

On Mon, Sep 10, 2018 at 9:17 AM, Nicolas Malin <[hidden email]>
wrote:

> On 08/09/2018 18:43, Taher Alkhateeb wrote:
>
>> I could be wrong, but, wouldn't it make sense that if your service is
>> failing continuously then perhaps something is wrong with the service?
>> I would imagine that perhaps resilience in the design of the service
>> might be the better route?
>>
> Yeah sure Taher, reanalyze each reason to improve these services is also a
> solution :) but not exactly ma question.
> At the beginning theses services has been called on sync to keep the rest
> api error contacted and work fine. But when we moved it on async and use
> parallelism process through the job manager the service error has been
> translate as restart the service.
> So yes I can redefine them but *by default* is it logical to restart
> indefinitely a async service that failed ?
>
> Nicolas
>
> On Fri, Sep 7, 2018 at 6:22 PM Nicolas Malin <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> On a customer site, we have huge services that call different rest api
>>> to collect information
>>> To increase the velocity we run all them by persistence asynchrone then
>>> the job pooler manage them with available resources.
>>>
>>> The problem is, when a call failed and the service threw an error, the
>>> service engine reschedule it, ... and it failed, rescheduled, failed,
>>> rescheduled, failed ... with beautiful result to overload your pool with
>>> zombie services.
>>>
>>> The solution is easy, set on your service definition attribute max-retry
>>> to 0 (or 1, if you want one retry) but I didn't understand why we have
>>> this configuration to reschedule indefinitely a service if it is in
>>> error.
>>>
>>> This configuration exists before apache migration so I'd happy to have
>>> your vision about this.
>>>   From my view, I'm in favor to set max retry to 0 by default and left
>>> the developer set him self when he wants that a service restart after a
>>> failure.
>>>
>>> Easy change  :
>>> Index:
>>> framework/service/src/main/java/org/apache/ofbiz/service/job
>>> /PersistedServiceJob.java
>>>
>>> @@ -80,7 +80,7 @@
>>>            this.jobValue = jobValue;
>>>            Timestamp storedDate = jobValue.getTimestamp("runTime");
>>>            this.startTime = storedDate.getTime();
>>> -        this.maxRetry = jobValue.get("maxRetry") != null ?
>>> jobValue.getLong("maxRetry") : -1;
>>> +        this.maxRetry = jobValue.get("maxRetry") != null ?
>>> jobValue.getLong("maxRetry") : 0;
>>>
>>> Nicolas
>>>
>>> --
>>> logoNrd <https://nereide.fr/>
>>>          Nicolas Malin
>>> The apache way <http://theapacheway.com/> : *Charity* Apache’s mission
>>> is providing software for the public good.
>>> [hidden email]
>>> 8 rue des Déportés 37000 TOURS, 02 47 50 30 54
>>>
>>> Apache OFBiz <http://ofbiz.apache.org/>|The Apache Way
>>> <http://theapacheway.com/>|réseau LE <http://www.libre-entreprise.org/>
>>>
>>
>

[Nicolas Malin-2]

Thanks for your return, much appreciated :)

I load the modification on trunk at 1840526

Cheers,

Nicolas


On 07/09/2018 17:22, Nicolas Malin wrote:

> Hi,
>
> On a customer site, we have huge services that call different rest api
> to collect information
> To increase the velocity we run all them by persistence asynchrone
> then the job pooler manage them with available resources.
>
> The problem is, when a call failed and the service threw an error, the
> service engine reschedule it, ... and it failed, rescheduled, failed,
> rescheduled, failed ... with beautiful result to overload your pool
> with zombie services.
>
> The solution is easy, set on your service definition attribute
> max-retry to 0 (or 1, if you want one retry) but I didn't understand
> why we have this configuration to reschedule indefinitely a service if
> it is in error.
>
> This configuration exists before apache migration so I'd happy to have
> your vision about this.
> From my view, I'm in favor to set max retry to 0 by default and left
> the developer set him self when he wants that a service restart after
> a failure.
>
> Easy change  :
> Index:
> framework/service/src/main/java/org/apache/ofbiz/service/job/PersistedServiceJob.java
>
> @@ -80,7 +80,7 @@
>          this.jobValue = jobValue;
>          Timestamp storedDate = jobValue.getTimestamp("runTime");
>          this.startTime = storedDate.getTime();
> -        this.maxRetry = jobValue.get("maxRetry") != null ?
> jobValue.getLong("maxRetry") : -1;
> +        this.maxRetry = jobValue.get("maxRetry") != null ?
> jobValue.getLong("maxRetry") : 0;
>
> Nicolas
>

[Swapnil Mane]

Thanks Nicolas, indeed this was good change :)


- Best Regards,
Swapnil M Mane

On Tue, Sep 11, 2018 at 1:15 PM Nicolas Malin <[hidden email]>
wrote:

> Thanks for your return, much appreciated :)
>
> I load the modification on trunk at 1840526
>
> Cheers,
>
> Nicolas
>
>
> On 07/09/2018 17:22, Nicolas Malin wrote:
> > Hi,
> >
> > On a customer site, we have huge services that call different rest api
> > to collect information
> > To increase the velocity we run all them by persistence asynchrone
> > then the job pooler manage them with available resources.
> >
> > The problem is, when a call failed and the service threw an error, the
> > service engine reschedule it, ... and it failed, rescheduled, failed,
> > rescheduled, failed ... with beautiful result to overload your pool
> > with zombie services.
> >
> > The solution is easy, set on your service definition attribute
> > max-retry to 0 (or 1, if you want one retry) but I didn't understand
> > why we have this configuration to reschedule indefinitely a service if
> > it is in error.
> >
> > This configuration exists before apache migration so I'd happy to have
> > your vision about this.
> > From my view, I'm in favor to set max retry to 0 by default and left
> > the developer set him self when he wants that a service restart after
> > a failure.
> >
> > Easy change  :
> > Index:
> >
> framework/service/src/main/java/org/apache/ofbiz/service/job/PersistedServiceJob.java
> >
> > @@ -80,7 +80,7 @@
> >          this.jobValue = jobValue;
> >          Timestamp storedDate = jobValue.getTimestamp("runTime");
> >          this.startTime = storedDate.getTime();
> > -        this.maxRetry = jobValue.get("maxRetry") != null ?
> > jobValue.getLong("maxRetry") : -1;
> > +        this.maxRetry = jobValue.get("maxRetry") != null ?
> > jobValue.getLong("maxRetry") : 0;
> >
> > Nicolas
> >
>
>