Multiple instances and the JobSandbox

[Craig Ambrose]

I have a multiple instance deployment backed by a single database. I have a user-triggered event that creates a JobSandbox entry that is to execute a week in the future. Is it (reasonably) guaranteed that the job will only be executed by a single instance? It would be bad for our customers if the job is executed by more than one of the instances.

Can anyone shed some light on what mechanism is responsible for ensuring that only one runtime instance will execute a certain job?

Thanks

-craig

[Scott Gray-2]

Give each instance it's own instanceId in general.properties and everything should be fine.  The JobManager periodically polls looking for jobs that aren't assigned, queued or cancelled and then assigns them to its instance using the instanceId and queues them.

It would probably take me as long to explain fully as it would for you to just go to the JobManager.poll() method and have a look at it :-)

Regards
Scott

HotWax Media
http://www.hotwaxmedia.com

On 21/07/2010, at 12:27 PM, Craig Ambrose wrote:

> I have a multiple instance deployment backed by a single database. I have a user-triggered event that creates a JobSandbox entry that is to execute a week in the future. Is it (reasonably) guaranteed that the job will only be executed by a single instance? It would be bad for our customers if the job is executed by more than one of the instances.
>
> Can anyone shed some light on what mechanism is responsible for ensuring that only one runtime instance will execute a certain job?
>
> Thanks
>
> -craig
>

[Craig Ambrose]

Thanks Scott.

Is there a way to make sure EVERY instance in my deployment execute a job? There is a cleanup task that I would like each instance to do at midnight.

Thanks again

-craig

[Brett]

Scott,

Thanks for this information.  I think the instanceId solves the problem for
the Service engine and the JobSandbox table.

Do you know if OFBiz has a similar solution for other tables that might have
problems with concurrency?  We ran into some problems with the order and
inventory tables when we ran with multiple application servers.  The problem
was that ofbiz doesn't appear to support a "select for update" type of
query.  This makes it difficult to guarantee that when one application
server reads a record (e.g. an inventory record) and then updates it that
another application server doesn't read the same record and update it as
well.  We had to work around this issue by writing custom SQL to do the
"select for update" call.

Are there any plans to implement this type of feature in the entity engine?
 Another approach I have seen implemented called "optimistic locking".  This
is where the entity engine checks the timestamps on updates to make sure the
timestamp has not been modified since the record was first read.  This type
of feature is needed any many ERP like applications.  Is there a way in
ofbiz to do this now.

Thanks,


Brett



On Tue, Jul 20, 2010 at 6:55 PM, Scott Gray <[hidden email]>wrote:

> Give each instance it's own instanceId in general.properties and everything
> should be fine.  The JobManager periodically polls looking for jobs that
> aren't assigned, queued or cancelled and then assigns them to its instance
> using the instanceId and queues them.
>
> It would probably take me as long to explain fully as it would for you to
> just go to the JobManager.poll() method and have a look at it :-)
>
> Regards
> Scott
>
> HotWax Media
> http://www.hotwaxmedia.com
>
> On 21/07/2010, at 12:27 PM, Craig Ambrose wrote:
>
> > I have a multiple instance deployment backed by a single database. I have
> a user-triggered event that creates a JobSandbox entry that is to execute a
> week in the future. Is it (reasonably) guaranteed that the job will only be
> executed by a single instance? It would be bad for our customers if the job
> is executed by more than one of the instances.
> >
> > Can anyone shed some light on what mechanism is responsible for ensuring
> that only one runtime instance will execute a certain job?
> >
> > Thanks
> >
> > -craig
> >
>
>

[Scott Gray-2]

There isn't anything off the top of my head, you could look at using a unique pool per instance (see serviceengine.xml) and then creating recurring jobs with the poolId field set so that only that instance would be able to pick the job up for execution.

Regards
Scott

HotWax Media
http://www.hotwaxmedia.com

On 23/07/2010, at 6:41 AM, Craig Ambrose wrote:

>
> Thanks Scott.
>
> Is there a way to make sure EVERY instance in my deployment execute a job?
> There is a cleanup task that I would like each instance to do at midnight.
>
> Thanks again
>
> -craig
> --
> View this message in context: http://ofbiz.135035.n4.nabble.com/Multiple-instances-and-the-JobSandbox-tp2296467p2299148.html
> Sent from the OFBiz - User mailing list archive at Nabble.com.

[Scott Gray-2]

Hi Brett,

I'm no expert on this stuff but a few comments inline.

Regards
Scott

On 23/07/2010, at 1:04 PM, Brett Palmer wrote:

> Scott,
>
> Thanks for this information.  I think the instanceId solves the problem for
> the Service engine and the JobSandbox table.
>
> Do you know if OFBiz has a similar solution for other tables that might have
> problems with concurrency?  We ran into some problems with the order and
> inventory tables when we ran with multiple application servers.  The problem
> was that ofbiz doesn't appear to support a "select for update" type of
> query.  This makes it difficult to guarantee that when one application
> server reads a record (e.g. an inventory record) and then updates it that
> another application server doesn't read the same record and update it as
> well.  We had to work around this issue by writing custom SQL to do the
> "select for update" call.

I think we may have discussed this before but my understanding is that selecting a row within a transaction essentially locks that row for the duration of the transaction.  

I've experienced that first hand in a situation where I was trying to update a row by suspending the current transaction, starting a new one, performing the update, committing and resuming the original transaction.  It didn't work because the original transaction had performed a select on that row which locked it, preventing the new transaction from performing the update.

> Are there any plans to implement this type of feature in the entity engine?
> Another approach I have seen implemented called "optimistic locking".  This
> is where the entity engine checks the timestamps on updates to make sure the
> timestamp has not been modified since the record was first read.  This type
> of feature is needed any many ERP like applications.  Is there a way in
> ofbiz to do this now.

Search the mailing lists for this, I'm pretty sure David described (within the last few months) how OFBiz could (or does) support this with the use of those timestamp fields.

>
> Thanks,
>
>
> Brett
>
>
>
> On Tue, Jul 20, 2010 at 6:55 PM, Scott Gray <[hidden email]>wrote:
>
>> Give each instance it's own instanceId in general.properties and everything
>> should be fine.  The JobManager periodically polls looking for jobs that
>> aren't assigned, queued or cancelled and then assigns them to its instance
>> using the instanceId and queues them.
>>
>> It would probably take me as long to explain fully as it would for you to
>> just go to the JobManager.poll() method and have a look at it :-)
>>
>> Regards
>> Scott
>>
>> HotWax Media
>> http://www.hotwaxmedia.com
>>
>> On 21/07/2010, at 12:27 PM, Craig Ambrose wrote:
>>
>>> I have a multiple instance deployment backed by a single database. I have
>> a user-triggered event that creates a JobSandbox entry that is to execute a
>> week in the future. Is it (reasonably) guaranteed that the job will only be
>> executed by a single instance? It would be bad for our customers if the job
>> is executed by more than one of the instances.
>>>
>>> Can anyone shed some light on what mechanism is responsible for ensuring
>> that only one runtime instance will execute a certain job?
>>>
>>> Thanks
>>>
>>> -craig
>>>
>>
>>

[David E. Jones-2]

On Jul 22, 2010, at 9:26 PM, Scott Gray wrote:

> Hi Brett,
>
> I'm no expert on this stuff but a few comments inline.
>
> Regards
> Scott
>
> On 23/07/2010, at 1:04 PM, Brett Palmer wrote:
>
>> Scott,
>>
>> Thanks for this information.  I think the instanceId solves the problem for
>> the Service engine and the JobSandbox table.
>>
>> Do you know if OFBiz has a similar solution for other tables that might have
>> problems with concurrency?  We ran into some problems with the order and
>> inventory tables when we ran with multiple application servers.  The problem
>> was that ofbiz doesn't appear to support a "select for update" type of
>> query.  This makes it difficult to guarantee that when one application
>> server reads a record (e.g. an inventory record) and then updates it that
>> another application server doesn't read the same record and update it as
>> well.  We had to work around this issue by writing custom SQL to do the
>> "select for update" call.
>
> I think we may have discussed this before but my understanding is that selecting a row within a transaction essentially locks that row for the duration of the transaction.  
>
> I've experienced that first hand in a situation where I was trying to update a row by suspending the current transaction, starting a new one, performing the update, committing and resuming the original transaction.  It didn't work because the original transaction had performed a select on that row which locked it, preventing the new transaction from performing the update.

This depends on the transaction isolation and how the db implements it. If you select for update, it will be locked regardless of the tx isolation and db particulars. This would be nice to have in the entity engine, and was discussed not too long ago, but I don't think it has been added yet by anyone. I did add this to the Moqui design (of course it's easy to add to a design... a little harder to add to an impl... ;) ).

>> Are there any plans to implement this type of feature in the entity engine?
>> Another approach I have seen implemented called "optimistic locking".  This
>> is where the entity engine checks the timestamps on updates to make sure the
>> timestamp has not been modified since the record was first read.  This type
>> of feature is needed any many ERP like applications.  Is there a way in
>> ofbiz to do this now.
>
> Search the mailing lists for this, I'm pretty sure David described (within the last few months) how OFBiz could (or does) support this with the use of those timestamp fields.

There is an attribute for this on the entity definition.

-David

>> On Tue, Jul 20, 2010 at 6:55 PM, Scott Gray <[hidden email]>wrote:
>>
>>> Give each instance it's own instanceId in general.properties and everything
>>> should be fine.  The JobManager periodically polls looking for jobs that
>>> aren't assigned, queued or cancelled and then assigns them to its instance
>>> using the instanceId and queues them.
>>>
>>> It would probably take me as long to explain fully as it would for you to
>>> just go to the JobManager.poll() method and have a look at it :-)
>>>
>>> Regards
>>> Scott
>>>
>>> HotWax Media
>>> http://www.hotwaxmedia.com
>>>
>>> On 21/07/2010, at 12:27 PM, Craig Ambrose wrote:
>>>
>>>> I have a multiple instance deployment backed by a single database. I have
>>> a user-triggered event that creates a JobSandbox entry that is to execute a
>>> week in the future. Is it (reasonably) guaranteed that the job will only be
>>> executed by a single instance? It would be bad for our customers if the job
>>> is executed by more than one of the instances.
>>>>
>>>> Can anyone shed some light on what mechanism is responsible for ensuring
>>> that only one runtime instance will execute a certain job?
>>>>
>>>> Thanks
>>>>
>>>> -craig
>>>>
>>>
>>>
>