Optimistic locking based on timestamps

[Mike Z]

This has been a very useful thread.  I now know that I need to dump
MySQL asap.   I planned on running multiple ofbiz instances for
ecommerce and had no idea that this may cause issues.  Thanks for the
input.

On Fri, Aug 13, 2010 at 5:31 PM, Brett Palmer <[hidden email]> wrote:

> James,
>
> We have run into this same problem on MySQL and ofbiz.  We worked around the
> problem by creating a custom method that got a direction connection from the
> transaction manager.  Then we wrote a custom SELECT for UPDATE on that
> connection.  We needed this functionality because we had multiple
> application servers hitting the same database and ran into concurrency
> problems without it.
>
> I would like to see the optimistic locking feature enhanced in ofbiz.  Maybe
> we could move away from timestamps and use an increasing unique ID as a
> replacement.  This is definitely a problem with MySQL.  We may move away
> from MySQL if we can find a good replication solution from Postgres.
>
>
> Brett
>
> On Thu, Aug 12, 2010 at 2:15 PM, James McGill <
> [hidden email]> wrote:
>
>> We are having problems with the optimistic locking.   With "enable-lock"
>> set
>> on an Entity, updates in GenericDAO use a timestamp to do locking.
>> There are a number of issues with this.  The biggest one is that it's not a
>> synchronized operation, so there's potential for a race condition within
>> customUpdate, which we are actually seeing in production.
>> I added code to introduce the "FOR UPDATE" expression when reading the
>> timestamp.  This brings up another issue, that the timestamp field in MySQL
>> has resolution only to the second.  So even if you don't have contention on
>> the optimistic lock SELECT, you still have to be lucky that your
>> transactions are more than one second apart.
>>
>> I realize this is a fairly difficult problem to address, in general, and
>> that "fixing" many concurrency issues leads to risks of deadlock.  But we
>> are seeing errors in data where the "last update wins."
>>
>> Has anyone else had concurrency problems when multiple threads are updating
>> entities?  Are there any locking provisions in the Delegator that would
>> allow us to prevent this kind of problem?
>>
>> --
>> James McGill
>> Phoenix AZ
>>
>

[Matt Warnock]

I'm still a bit confused.  I think I understand the issues, but not why
so many people are apparently having trouble with them.  Or maybe I
misunderstand them completely.

Optimistic locking (as I understand it) is used primarily when editing
an existing record by hand, since record creation and programmed updates
can just use transactions, which are better for most operations anyway.
Most common business cases I can imagine would not usually involve 2
people editing (not just viewing) the same record at the same time.
What business scenario causes these apparently common collisions?

Most high-volume business uses don't edit other people's records.  If I
enter an e-commerce order for example, I create the header record,
several line item records, perhaps some other stuff.  Eventually I
commit the whole order at once, when it is assigned an order number and
becomes part of the main database, which can all be done in a single
transaction.  

Others may be entering similar orders, but they are creating different
header records with different associated line items.  These records
should all be accumulated into memory-only or temporary tables (I would
assume) until they are committed to the database, and optimistic locking
should never really enter into it, as these records are private to the
user and current session (like an e-commerce shopping cart) until they
are committed.  If they are abandoned before they commit, they should
never leave a trace in the main database, as I see it.  Any code that
updates the record (to total it, apply taxes, figure shipping, or
whatever) can work in-memory, or in a single transaction on the
temporary records, until the whole thing is committed.

If I then go back and edit an order, it is usually one I just recently
entered, and in most cases, no one else should be using it.  When I do
that, the optimistic lock code should read the record data and note the
time that the record was last modified (or the data itself). I then edit
that data on-screen, and when I commit, it first checks to see that the
data was not modified in the meantime.  In most cases, it wasn't
modified, and the new data is written, again within the scope of a
single transaction.

If the last-modified date (or the original data) has changed, then a
collision has occurred, and the system should cancel my commit, because
I was editing data which has changed while I was editing it, and is now
stale.  In most cases, any manual edit takes much more than a second, so
the chance of a time granularity collision on an actual record edit
seems miniscule. If there is a collision, the system re-reads the
recently updated data, tells me about the collision, probably discards
the previous edits, and I can then edit again if necessary.

It's a poor substitute for an update transaction, but you don't want to
lock a database up for several minutes while a user edits a record by
hand, and most transactions will timeout long before the user finishes
the edit.

Programmatic data updates like Mike Z describes are much more common,
but they can usually be managed in a single transaction too.  I don't
need a lock to calculate a total, enter a timestamp, or similar updates,
as these can all be done inside an ACID transaction, thereby protected
from other threads, users, application servers, or whatever.  We can
even suspend one transaction to run an unrelated one, then resume the
first, as David suggested earlier in this thread.

Can you give me an example of the kind of update that leads to the kind
of concurrency issues you describe?  Is OFBiz using optimistic locks
where transactions are really required?  Or what about James' inventory
count scenario prevents using a transaction instead of an optimistic
lock?  What am I missing?  Just want to know where the big bear traps
might be.  Thanks in advance.

--
Matt Warnock <[hidden email]>
RidgeCrest Herbals, Inc.


On Fri, 2010-08-13 at 19:52 -0700, Mike Z wrote:

> This has been a very useful thread.  I now know that I need to dump
> MySQL asap.   I planned on running multiple ofbiz instances for
> ecommerce and had no idea that this may cause issues.  Thanks for the
> input.
>
> On Fri, Aug 13, 2010 at 5:31 PM, Brett Palmer <[hidden email]> wrote:
> > James,
> >
> > We have run into this same problem on MySQL and ofbiz.  We worked around the
> > problem by creating a custom method that got a direction connection from the
> > transaction manager.  Then we wrote a custom SELECT for UPDATE on that
> > connection.  We needed this functionality because we had multiple
> > application servers hitting the same database and ran into concurrency
> > problems without it.
> >
> > I would like to see the optimistic locking feature enhanced in ofbiz.  Maybe
> > we could move away from timestamps and use an increasing unique ID as a
> > replacement.  This is definitely a problem with MySQL.  We may move away
> > from MySQL if we can find a good replication solution from Postgres.
> >
> >
> > Brett
> >
> > On Thu, Aug 12, 2010 at 2:15 PM, James McGill <
> > [hidden email]> wrote:
> >
> >> We are having problems with the optimistic locking.   With "enable-lock"
> >> set
> >> on an Entity, updates in GenericDAO use a timestamp to do locking.
> >> There are a number of issues with this.  The biggest one is that it's not a
> >> synchronized operation, so there's potential for a race condition within
> >> customUpdate, which we are actually seeing in production.
> >> I added code to introduce the "FOR UPDATE" expression when reading the
> >> timestamp.  This brings up another issue, that the timestamp field in MySQL
> >> has resolution only to the second.  So even if you don't have contention on
> >> the optimistic lock SELECT, you still have to be lucky that your
> >> transactions are more than one second apart.
> >>
> >> I realize this is a fairly difficult problem to address, in general, and
> >> that "fixing" many concurrency issues leads to risks of deadlock.  But we
> >> are seeing errors in data where the "last update wins."
> >>
> >> Has anyone else had concurrency problems when multiple threads are updating
> >> entities?  Are there any locking provisions in the Delegator that would
> >> allow us to prevent this kind of problem?
> >>
> >> --
> >> James McGill
> >> Phoenix AZ
> >>
> >

[BJ Freeman]

Matt:
read Davids responses.
in short Optimistic locking as a database Function.
Since ofbiz does it own managing of the data, the database itself does
not have all the info to manage the locking effectively.


Matt Warnock sent the following on 8/13/2010 11:45 PM:

> I'm still a bit confused.  I think I understand the issues, but not why
> so many people are apparently having trouble with them.  Or maybe I
> misunderstand them completely.
>
> Optimistic locking (as I understand it) is used primarily when editing
> an existing record by hand, since record creation and programmed updates
> can just use transactions, which are better for most operations anyway.
> Most common business cases I can imagine would not usually involve 2
> people editing (not just viewing) the same record at the same time.
> What business scenario causes these apparently common collisions?
>
> Most high-volume business uses don't edit other people's records.  If I
> enter an e-commerce order for example, I create the header record,
> several line item records, perhaps some other stuff.  Eventually I
> commit the whole order at once, when it is assigned an order number and
> becomes part of the main database, which can all be done in a single
> transaction.
>
> Others may be entering similar orders, but they are creating different
> header records with different associated line items.  These records
> should all be accumulated into memory-only or temporary tables (I would
> assume) until they are committed to the database, and optimistic locking
> should never really enter into it, as these records are private to the
> user and current session (like an e-commerce shopping cart) until they
> are committed.  If they are abandoned before they commit, they should
> never leave a trace in the main database, as I see it.  Any code that
> updates the record (to total it, apply taxes, figure shipping, or
> whatever) can work in-memory, or in a single transaction on the
> temporary records, until the whole thing is committed.
>
> If I then go back and edit an order, it is usually one I just recently
> entered, and in most cases, no one else should be using it.  When I do
> that, the optimistic lock code should read the record data and note the
> time that the record was last modified (or the data itself). I then edit
> that data on-screen, and when I commit, it first checks to see that the
> data was not modified in the meantime.  In most cases, it wasn't
> modified, and the new data is written, again within the scope of a
> single transaction.
>
> If the last-modified date (or the original data) has changed, then a
> collision has occurred, and the system should cancel my commit, because
> I was editing data which has changed while I was editing it, and is now
> stale.  In most cases, any manual edit takes much more than a second, so
> the chance of a time granularity collision on an actual record edit
> seems miniscule. If there is a collision, the system re-reads the
> recently updated data, tells me about the collision, probably discards
> the previous edits, and I can then edit again if necessary.
>
> It's a poor substitute for an update transaction, but you don't want to
> lock a database up for several minutes while a user edits a record by
> hand, and most transactions will timeout long before the user finishes
> the edit.
>
> Programmatic data updates like Mike Z describes are much more common,
> but they can usually be managed in a single transaction too.  I don't
> need a lock to calculate a total, enter a timestamp, or similar updates,
> as these can all be done inside an ACID transaction, thereby protected
> from other threads, users, application servers, or whatever.  We can
> even suspend one transaction to run an unrelated one, then resume the
> first, as David suggested earlier in this thread.
>
> Can you give me an example of the kind of update that leads to the kind
> of concurrency issues you describe?  Is OFBiz using optimistic locks
> where transactions are really required?  Or what about James' inventory
> count scenario prevents using a transaction instead of an optimistic
> lock?  What am I missing?  Just want to know where the big bear traps
> might be.  Thanks in advance.
>

[David E. Jones-2]

BJ,

I don't remember commenting about optimistic locking yet, only about "nesting" of transaction (or lack thereof).

Am I forgetting something?

-David


On Aug 14, 2010, at 2:36 AM, BJ Freeman wrote:

> Matt:
> read Davids responses.
> in short Optimistic locking as a database Function.
> Since ofbiz does it own managing of the data, the database itself does not have all the info to manage the locking effectively.
>
>
> Matt Warnock sent the following on 8/13/2010 11:45 PM:
>> I'm still a bit confused.  I think I understand the issues, but not why
>> so many people are apparently having trouble with them.  Or maybe I
>> misunderstand them completely.
>>
>> Optimistic locking (as I understand it) is used primarily when editing
>> an existing record by hand, since record creation and programmed updates
>> can just use transactions, which are better for most operations anyway.
>> Most common business cases I can imagine would not usually involve 2
>> people editing (not just viewing) the same record at the same time.
>> What business scenario causes these apparently common collisions?
>>
>> Most high-volume business uses don't edit other people's records.  If I
>> enter an e-commerce order for example, I create the header record,
>> several line item records, perhaps some other stuff.  Eventually I
>> commit the whole order at once, when it is assigned an order number and
>> becomes part of the main database, which can all be done in a single
>> transaction.
>>
>> Others may be entering similar orders, but they are creating different
>> header records with different associated line items.  These records
>> should all be accumulated into memory-only or temporary tables (I would
>> assume) until they are committed to the database, and optimistic locking
>> should never really enter into it, as these records are private to the
>> user and current session (like an e-commerce shopping cart) until they
>> are committed.  If they are abandoned before they commit, they should
>> never leave a trace in the main database, as I see it.  Any code that
>> updates the record (to total it, apply taxes, figure shipping, or
>> whatever) can work in-memory, or in a single transaction on the
>> temporary records, until the whole thing is committed.
>>
>> If I then go back and edit an order, it is usually one I just recently
>> entered, and in most cases, no one else should be using it.  When I do
>> that, the optimistic lock code should read the record data and note the
>> time that the record was last modified (or the data itself). I then edit
>> that data on-screen, and when I commit, it first checks to see that the
>> data was not modified in the meantime.  In most cases, it wasn't
>> modified, and the new data is written, again within the scope of a
>> single transaction.
>>
>> If the last-modified date (or the original data) has changed, then a
>> collision has occurred, and the system should cancel my commit, because
>> I was editing data which has changed while I was editing it, and is now
>> stale.  In most cases, any manual edit takes much more than a second, so
>> the chance of a time granularity collision on an actual record edit
>> seems miniscule. If there is a collision, the system re-reads the
>> recently updated data, tells me about the collision, probably discards
>> the previous edits, and I can then edit again if necessary.
>>
>> It's a poor substitute for an update transaction, but you don't want to
>> lock a database up for several minutes while a user edits a record by
>> hand, and most transactions will timeout long before the user finishes
>> the edit.
>>
>> Programmatic data updates like Mike Z describes are much more common,
>> but they can usually be managed in a single transaction too.  I don't
>> need a lock to calculate a total, enter a timestamp, or similar updates,
>> as these can all be done inside an ACID transaction, thereby protected
>> from other threads, users, application servers, or whatever.  We can
>> even suspend one transaction to run an unrelated one, then resume the
>> first, as David suggested earlier in this thread.
>>
>> Can you give me an example of the kind of update that leads to the kind
>> of concurrency issues you describe?  Is OFBiz using optimistic locks
>> where transactions are really required?  Or what about James' inventory
>> count scenario prevents using a transaction instead of an optimistic
>> lock?  What am I missing?  Just want to know where the big bear traps
>> might be.  Thanks in advance.
>>

[David E. Jones-2]

Timestamp-based optimistic locking is a feature of the Entity Engine, but it is not used very much in OFBiz. In fact, I'm not sure if it's used at all. The way it came into this discussion was, I suppose, as a possible solution to the synchronization problems people were having with race conditions.

As you mentioned here, which is correct, optimistic locking is really only helpful if two people are possibly editing the same data at the same time and you want to notify a user if another user has changed the data they are working on between the time they got the data from the database, and the time they saved their changes to the database. With most manual editing, as you mentioned, the reading and writing are done in two separate transactions, so that is a case where a SELECT FOR UPDATE would not help. As you said, in order for that to be helpful in the common case where optimistic locks are used the transaction would have to live for many minutes and lock resources for that entire time (ie a pessimistic lock).

There are certainly cases where optimistic locks might be useful, and they would be things mostly done manually like editing product information or any content that lives in the database. Two people could accidentally be working on the same product or content at the same time, and without optimistic locking the person who saved second would wipe out the changes of the person who saved first, but neither would know it unless they manually review the data at a later time. If pessimistic locking were used in these scenarios it would be like those REALLY annoying old source repositories where if you check out a file it is "locked" and no one else can change it until you check it back in and release the lock (ie they didn't bother to implement any sort of merging). With the Entity Engine optimistic lock it won't try to do any merging, the purpose is to notify the user that someone else changed the data they were working on between the time they read the data to edit and the time they tried to save it (the separate read and write transactions).

For many race conditions that cause bigger problems the scenario is very different. In your example of order data that is likely to be very low conflict, but there are many data structures that tend to be higher conflict, like inventory data. In order for there to be conflict in inventory data all it takes is for two customers to order the same product at roughly the same time (ie within the span of the time it takes the order transaction to execute, which can be tens of seconds sometimes). For a popular item on a busy site this isn't just possible, it's really likely. In this case optimistic locking wouldn't be that helpful, ie you don't want the behavior where the system essentially says "someone else is ordering that product right now, please try again later". What you would want is for the database to lock certain records so that the second user waits until the first user makes any changes. And, what you want them to wait on is being able to READ the data, not waiting to WRITE it. The common scenario is that two different threads read the current inventory value, then both are working on things including decrementing the inventory value, then both write it. In the end the result will be wrong because they both started with the same value and subtracted from it, and basically whoever writes first will have their value ignored and the total at the end will just be the original value minus whatever the second thread to write subtracted.

That is a case where pessimistic locking is necessary, and a case where things aren't as simple as they may seem.

To understand some of the complexity check out the concept of "transaction isolation". The big trick is that for performance and concurrency reasons databases do NOT totally isolate transactions and update conflicts can easily occur:

http://en.wikipedia.org/wiki/Isolation_(database_systems)

Many databases don't even support the more strict transaction isolation levels, and even if they do they are not commonly used except for special purposes. With things like SERIALIZABLE the problem is that you end up locking, in many cases, entire tables and not just rows within those tables and you have HUGE concurrency and deadlock problems that result.

The most common level you'll see used is READ COMMITTED, and sometimes READ UNCOMMITTED if the database doesn't support READ COMMITTED. You can see these settings in the entityengine.xml file.

That is where SELECT FOR UPDATE is useful. You don't want to use the SERIALIZABLE transaction isolation, but you want this certain record locked even though it hasn't been changed so that other transactions don't read the incorrect value.

-David


On Aug 14, 2010, at 12:45 AM, Matt Warnock wrote:

> I'm still a bit confused.  I think I understand the issues, but not why
> so many people are apparently having trouble with them.  Or maybe I
> misunderstand them completely.
>
> Optimistic locking (as I understand it) is used primarily when editing
> an existing record by hand, since record creation and programmed updates
> can just use transactions, which are better for most operations anyway.
> Most common business cases I can imagine would not usually involve 2
> people editing (not just viewing) the same record at the same time.
> What business scenario causes these apparently common collisions?
>
> Most high-volume business uses don't edit other people's records.  If I
> enter an e-commerce order for example, I create the header record,
> several line item records, perhaps some other stuff.  Eventually I
> commit the whole order at once, when it is assigned an order number and
> becomes part of the main database, which can all be done in a single
> transaction.  
>
> Others may be entering similar orders, but they are creating different
> header records with different associated line items.  These records
> should all be accumulated into memory-only or temporary tables (I would
> assume) until they are committed to the database, and optimistic locking
> should never really enter into it, as these records are private to the
> user and current session (like an e-commerce shopping cart) until they
> are committed.  If they are abandoned before they commit, they should
> never leave a trace in the main database, as I see it.  Any code that
> updates the record (to total it, apply taxes, figure shipping, or
> whatever) can work in-memory, or in a single transaction on the
> temporary records, until the whole thing is committed.
>
> If I then go back and edit an order, it is usually one I just recently
> entered, and in most cases, no one else should be using it.  When I do
> that, the optimistic lock code should read the record data and note the
> time that the record was last modified (or the data itself). I then edit
> that data on-screen, and when I commit, it first checks to see that the
> data was not modified in the meantime.  In most cases, it wasn't
> modified, and the new data is written, again within the scope of a
> single transaction.
>
> If the last-modified date (or the original data) has changed, then a
> collision has occurred, and the system should cancel my commit, because
> I was editing data which has changed while I was editing it, and is now
> stale.  In most cases, any manual edit takes much more than a second, so
> the chance of a time granularity collision on an actual record edit
> seems miniscule. If there is a collision, the system re-reads the
> recently updated data, tells me about the collision, probably discards
> the previous edits, and I can then edit again if necessary.
>
> It's a poor substitute for an update transaction, but you don't want to
> lock a database up for several minutes while a user edits a record by
> hand, and most transactions will timeout long before the user finishes
> the edit.
>
> Programmatic data updates like Mike Z describes are much more common,
> but they can usually be managed in a single transaction too.  I don't
> need a lock to calculate a total, enter a timestamp, or similar updates,
> as these can all be done inside an ACID transaction, thereby protected
> from other threads, users, application servers, or whatever.  We can
> even suspend one transaction to run an unrelated one, then resume the
> first, as David suggested earlier in this thread.
>
> Can you give me an example of the kind of update that leads to the kind
> of concurrency issues you describe?  Is OFBiz using optimistic locks
> where transactions are really required?  Or what about James' inventory
> count scenario prevents using a transaction instead of an optimistic
> lock?  What am I missing?  Just want to know where the big bear traps
> might be.  Thanks in advance.
>
> --
> Matt Warnock <[hidden email]>
> RidgeCrest Herbals, Inc.
>
>
> On Fri, 2010-08-13 at 19:52 -0700, Mike Z wrote:
>> This has been a very useful thread.  I now know that I need to dump
>> MySQL asap.   I planned on running multiple ofbiz instances for
>> ecommerce and had no idea that this may cause issues.  Thanks for the
>> input.
>>
>> On Fri, Aug 13, 2010 at 5:31 PM, Brett Palmer <[hidden email]> wrote:
>>> James,
>>>
>>> We have run into this same problem on MySQL and ofbiz.  We worked around the
>>> problem by creating a custom method that got a direction connection from the
>>> transaction manager.  Then we wrote a custom SELECT for UPDATE on that
>>> connection.  We needed this functionality because we had multiple
>>> application servers hitting the same database and ran into concurrency
>>> problems without it.
>>>
>>> I would like to see the optimistic locking feature enhanced in ofbiz.  Maybe
>>> we could move away from timestamps and use an increasing unique ID as a
>>> replacement.  This is definitely a problem with MySQL.  We may move away
>>> from MySQL if we can find a good replication solution from Postgres.
>>>
>>>
>>> Brett
>>>
>>> On Thu, Aug 12, 2010 at 2:15 PM, James McGill <
>>> [hidden email]> wrote:
>>>
>>>> We are having problems with the optimistic locking.   With "enable-lock"
>>>> set
>>>> on an Entity, updates in GenericDAO use a timestamp to do locking.
>>>> There are a number of issues with this.  The biggest one is that it's not a
>>>> synchronized operation, so there's potential for a race condition within
>>>> customUpdate, which we are actually seeing in production.
>>>> I added code to introduce the "FOR UPDATE" expression when reading the
>>>> timestamp.  This brings up another issue, that the timestamp field in MySQL
>>>> has resolution only to the second.  So even if you don't have contention on
>>>> the optimistic lock SELECT, you still have to be lucky that your
>>>> transactions are more than one second apart.
>>>>
>>>> I realize this is a fairly difficult problem to address, in general, and
>>>> that "fixing" many concurrency issues leads to risks of deadlock.  But we
>>>> are seeing errors in data where the "last update wins."
>>>>
>>>> Has anyone else had concurrency problems when multiple threads are updating
>>>> entities?  Are there any locking provisions in the Delegator that would
>>>> allow us to prevent this kind of problem?
>>>>
>>>> --
>>>> James McGill
>>>> Phoenix AZ
>>>>
>>>
>

[Jacques Le Roux]

Checking entityengine.xml I found that, by default, only HSQL uses isolation-level="ReadUncommitted" while all others use
isolation-level="ReadCommitted". Since its version 2.0, HSQL now supports ReadCommitted. Should we not switch it also to
ReadCommitted?

I remember liking much HSQL when I began on OFBiz. Then switching on something more robust (mostly Derby/Postgres for me) after some
unpleasant surprises.
But I guess if you want something really quick and prepared/checked (demo) it may still be useful...

http://en.wikipedia.org/wiki/HSQLDB

Jacques

From: "David E Jones" <[hidden email]>

> Timestamp-based optimistic locking is a feature of the Entity Engine, but it is not used very much in OFBiz. In fact, I'm not sure
> if it's used at all. The way it came into this discussion was, I suppose, as a possible solution to the synchronization problems
> people were having with race conditions.
>
> As you mentioned here, which is correct, optimistic locking is really only helpful if two people are possibly editing the same
> data at the same time and you want to notify a user if another user has changed the data they are working on between the time they
> got the data from the database, and the time they saved their changes to the database. With most manual editing, as you mentioned,
> the reading and writing are done in two separate transactions, so that is a case where a SELECT FOR UPDATE would not help. As you
> said, in order for that to be helpful in the common case where optimistic locks are used the transaction would have to live for
> many minutes and lock resources for that entire time (ie a pessimistic lock).
>
> There are certainly cases where optimistic locks might be useful, and they would be things mostly done manually like editing
> product information or any content that lives in the database. Two people could accidentally be working on the same product or
> content at the same time, and without optimistic locking the person who saved second would wipe out the changes of the person who
> saved first, but neither would know it unless they manually review the data at a later time. If pessimistic locking were used in
> these scenarios it would be like those REALLY annoying old source repositories where if you check out a file it is "locked" and no
> one else can change it until you check it back in and release the lock (ie they didn't bother to implement any sort of merging).
> With the Entity Engine optimistic lock it won't try to do any merging, the purpose is to notify the user that someone else changed
> the data they were working on between the time they read the data to edit and the time !
> they tried to save it (the separate read and write transactions).
>
> For many race conditions that cause bigger problems the scenario is very different. In your example of order data that is likely
> to be very low conflict, but there are many data structures that tend to be higher conflict, like inventory data. In order for
> there to be conflict in inventory data all it takes is for two customers to order the same product at roughly the same time (ie
> within the span of the time it takes the order transaction to execute, which can be tens of seconds sometimes). For a popular item
> on a busy site this isn't just possible, it's really likely. In this case optimistic locking wouldn't be that helpful, ie you
> don't want the behavior where the system essentially says "someone else is ordering that product right now, please try again
> later". What you would want is for the database to lock certain records so that the second user waits until the first user makes
> any changes. And, what you want them to wait on is being able to READ the data, not waiting to!
>  WRITE it. The common scenario is that two different threads read the current inventory value, then both are working on things
> including decrementing the inventory value, then both write it. In the end the result will be wrong because they both started with
> the same value and subtracted from it, and basically whoever writes first will have their value ignored and the total at the end
> will just be the original value minus whatever the second thread to write subtracted.
>
> That is a case where pessimistic locking is necessary, and a case where things aren't as simple as they may seem.
>
> To understand some of the complexity check out the concept of "transaction isolation". The big trick is that for performance and
> concurrency reasons databases do NOT totally isolate transactions and update conflicts can easily occur:
>
> http://en.wikipedia.org/wiki/Isolation_(database_systems)
>
> Many databases don't even support the more strict transaction isolation levels, and even if they do they are not commonly used
> except for special purposes. With things like SERIALIZABLE the problem is that you end up locking, in many cases, entire tables
> and not just rows within those tables and you have HUGE concurrency and deadlock problems that result.
>
> The most common level you'll see used is READ COMMITTED, and sometimes READ UNCOMMITTED if the database doesn't support READ
> COMMITTED. You can see these settings in the entityengine.xml file.
>
> That is where SELECT FOR UPDATE is useful. You don't want to use the SERIALIZABLE transaction isolation, but you want this certain
> record locked even though it hasn't been changed so that other transactions don't read the incorrect value.
>
> -David
>
>
> On Aug 14, 2010, at 12:45 AM, Matt Warnock wrote:
>
>> I'm still a bit confused.  I think I understand the issues, but not why
>> so many people are apparently having trouble with them.  Or maybe I
>> misunderstand them completely.
>>
>> Optimistic locking (as I understand it) is used primarily when editing
>> an existing record by hand, since record creation and programmed updates
>> can just use transactions, which are better for most operations anyway.
>> Most common business cases I can imagine would not usually involve 2
>> people editing (not just viewing) the same record at the same time.
>> What business scenario causes these apparently common collisions?
>>
>> Most high-volume business uses don't edit other people's records.  If I
>> enter an e-commerce order for example, I create the header record,
>> several line item records, perhaps some other stuff.  Eventually I
>> commit the whole order at once, when it is assigned an order number and
>> becomes part of the main database, which can all be done in a single
>> transaction.
>>
>> Others may be entering similar orders, but they are creating different
>> header records with different associated line items.  These records
>> should all be accumulated into memory-only or temporary tables (I would
>> assume) until they are committed to the database, and optimistic locking
>> should never really enter into it, as these records are private to the
>> user and current session (like an e-commerce shopping cart) until they
>> are committed.  If they are abandoned before they commit, they should
>> never leave a trace in the main database, as I see it.  Any code that
>> updates the record (to total it, apply taxes, figure shipping, or
>> whatever) can work in-memory, or in a single transaction on the
>> temporary records, until the whole thing is committed.
>>
>> If I then go back and edit an order, it is usually one I just recently
>> entered, and in most cases, no one else should be using it.  When I do
>> that, the optimistic lock code should read the record data and note the
>> time that the record was last modified (or the data itself). I then edit
>> that data on-screen, and when I commit, it first checks to see that the
>> data was not modified in the meantime.  In most cases, it wasn't
>> modified, and the new data is written, again within the scope of a
>> single transaction.
>>
>> If the last-modified date (or the original data) has changed, then a
>> collision has occurred, and the system should cancel my commit, because
>> I was editing data which has changed while I was editing it, and is now
>> stale.  In most cases, any manual edit takes much more than a second, so
>> the chance of a time granularity collision on an actual record edit
>> seems miniscule. If there is a collision, the system re-reads the
>> recently updated data, tells me about the collision, probably discards
>> the previous edits, and I can then edit again if necessary.
>>
>> It's a poor substitute for an update transaction, but you don't want to
>> lock a database up for several minutes while a user edits a record by
>> hand, and most transactions will timeout long before the user finishes
>> the edit.
>>
>> Programmatic data updates like Mike Z describes are much more common,
>> but they can usually be managed in a single transaction too.  I don't
>> need a lock to calculate a total, enter a timestamp, or similar updates,
>> as these can all be done inside an ACID transaction, thereby protected
>> from other threads, users, application servers, or whatever.  We can
>> even suspend one transaction to run an unrelated one, then resume the
>> first, as David suggested earlier in this thread.
>>
>> Can you give me an example of the kind of update that leads to the kind
>> of concurrency issues you describe?  Is OFBiz using optimistic locks
>> where transactions are really required?  Or what about James' inventory
>> count scenario prevents using a transaction instead of an optimistic
>> lock?  What am I missing?  Just want to know where the big bear traps
>> might be.  Thanks in advance.
>>
>> --
>> Matt Warnock <[hidden email]>
>> RidgeCrest Herbals, Inc.
>>
>>
>> On Fri, 2010-08-13 at 19:52 -0700, Mike Z wrote:
>>> This has been a very useful thread.  I now know that I need to dump
>>> MySQL asap.   I planned on running multiple ofbiz instances for
>>> ecommerce and had no idea that this may cause issues.  Thanks for the
>>> input.
>>>
>>> On Fri, Aug 13, 2010 at 5:31 PM, Brett Palmer <[hidden email]> wrote:
>>>> James,
>>>>
>>>> We have run into this same problem on MySQL and ofbiz.  We worked around the
>>>> problem by creating a custom method that got a direction connection from the
>>>> transaction manager.  Then we wrote a custom SELECT for UPDATE on that
>>>> connection.  We needed this functionality because we had multiple
>>>> application servers hitting the same database and ran into concurrency
>>>> problems without it.
>>>>
>>>> I would like to see the optimistic locking feature enhanced in ofbiz.  Maybe
>>>> we could move away from timestamps and use an increasing unique ID as a
>>>> replacement.  This is definitely a problem with MySQL.  We may move away
>>>> from MySQL if we can find a good replication solution from Postgres.
>>>>
>>>>
>>>> Brett
>>>>
>>>> On Thu, Aug 12, 2010 at 2:15 PM, James McGill <
>>>> [hidden email]> wrote:
>>>>
>>>>> We are having problems with the optimistic locking.   With "enable-lock"
>>>>> set
>>>>> on an Entity, updates in GenericDAO use a timestamp to do locking.
>>>>> There are a number of issues with this.  The biggest one is that it's not a
>>>>> synchronized operation, so there's potential for a race condition within
>>>>> customUpdate, which we are actually seeing in production.
>>>>> I added code to introduce the "FOR UPDATE" expression when reading the
>>>>> timestamp.  This brings up another issue, that the timestamp field in MySQL
>>>>> has resolution only to the second.  So even if you don't have contention on
>>>>> the optimistic lock SELECT, you still have to be lucky that your
>>>>> transactions are more than one second apart.
>>>>>
>>>>> I realize this is a fairly difficult problem to address, in general, and
>>>>> that "fixing" many concurrency issues leads to risks of deadlock.  But we
>>>>> are seeing errors in data where the "last update wins."
>>>>>
>>>>> Has anyone else had concurrency problems when multiple threads are updating
>>>>> entities?  Are there any locking provisions in the Delegator that would
>>>>> allow us to prevent this kind of problem?
>>>>>
>>>>> --
>>>>> James McGill
>>>>> Phoenix AZ
>>>>>
>>>>
>>
>
>

[Matt Warnock]

Thank you David for the detailed and thoughtful explanation.  I think I
see the issue now.

However, I still see it as largely a business process issue, not a
programming issue.  You're right, you don't want to tell a customer that
a product is available, then later tell them it isn't, because someone
beat them to it.  But the real fact of life is that shopping carts are
abandoned all the time, so you really shouldn't commit inventory to any
order until the order is placed and approved.  There are lots of
potential slips between order placement and shipping (when the product
REALLY moves out of inventory)-- abandoned carts, declined credit, bad
shipping address, no shipping service to that location, import/export
restrictions, management fiat, whatever.  I exercised management fiat
just yesterday. :)

Near-simultaneous inventory transactions *per se* shouldn't be a
problem.  If I order 5 items, then when I commit, my code sets "qty =
qty - 5" in a single sql statement and is further enclosed within a
transaction that checks that the resulting (or beginning) qty is not
less than zero or some other minimum number, or from mixed lot numbers,
or whatever else your business logic may require.  Very busy sites won't
collide unless they try to commit inventory to "pending" orders with an
optimistic lock, which I think is a mistake.  

If inventory levels are high and goods are fungible, you don't need to
worry about it.  Tens of seconds, or even days, isn't going to make a
difference, and even if the item is serialized, you allocate the serial
number when the order is filled.  

If inventory is low, or unique, then you have a scarcity problem, and
you may want to warn the user of that fact, both to encourage the quick
order, and to manage expectations, so that they are not disappointed--
but locking the inventory pending an order that may or may not happen
seems like a wrong approach to me.  If people are doing that, then I
certainly understand why they are seeing concurrency issues on busy
sites.

Another possible approach would be to enter "pending" orders in a
separate table (or flagged as such in the same table), and only deduct
from the actual inventory if and when the sale is finally approved.
That way you could see whether inventory is scarce (orders pending equal
or exceed some ratio of inventory on hand) and also impose a "first
come, first served" approach on pending orders, while still imposing
time limits on completing the order, and possibly even enabling a
"waiting list".  Kind of like an appointment queue-- "we've had a
cancellation, so we can take you today at 3pm."  

It just seems to me that optimistic locking is not the right solution to
a scarce inventory problem.

In my view, any database that relaxes the ACID requirements to achieve
concurrency is barely worthy of the name.  ERP (especially financial
accounting) requires ACID, period.  You can't run an enterprise on less.
But it does mean you need to break your business processes down into
small, atomic steps so that-- like using a hammer to drive a screw.
isolation requirements do not become a concurrency nightmare.  And some
longer steps (like internally consistent dumps of the entire database)
may need to be timed carefully.

Thanks again for your thoughtful explanation.

--
Matt Warnock <[hidden email]>
RidgeCrest Herbals, Inc.

On Sat, 2010-08-14 at 09:46 -0600, David E Jones wrote:

> Timestamp-based optimistic locking is a feature of the Entity Engine,
>  but it is not used very much in OFBiz. In fact, I'm not sure if it's
>  used at all. The way it came into this discussion was, I suppose, as a
>  possible solution to the synchronization problems people were having
>  with race conditions.
>
> As you mentioned here, which is correct, optimistic locking is really
>  only helpful if two people are possibly editing the same data at the
>  same time and you want to notify a user if another user has changed
>  the data they are working on between the time they got the data from
>  the database, and the time they saved their changes to the database.
>  With most manual editing, as you mentioned, the reading and writing
>  are done in two separate transactions, so that is a case where a
>  SELECT FOR UPDATE would not help. As you said, in order for that to be
>  helpful in the common case where optimistic locks are used the
>  transaction would have to live for many minutes and lock resources for
>  that entire time (ie a pessimistic lock).
>
> There are certainly cases where optimistic locks might be useful, and
>  they would be things mostly done manually like editing product
>  information or any content that lives in the database. Two people
>  could accidentally be working on the same product or content at the
>  same time, and without optimistic locking the person who saved second
>  would wipe out the changes of the person who saved first, but neither
>  would know it unless they manually review the data at a later time. If
>  pessimistic locking were used in these scenarios it would be like
>  those REALLY annoying old source repositories where if you check out a
>  file it is "locked" and no one else can change it until you check it
>  back in and release the lock (ie they didn't bother to implement any
>  sort of merging). With the Entity Engine optimistic lock it won't try
>  to do any merging, the purpose is to notify the user that someone else
>  changed the data they were working on between the time they read the
>  data to edit and the time they tried to save it (the separate read and
>  write transactions).
>
> For many race conditions that cause bigger problems the scenario is
>  very different. In your example of order data that is likely to be
>  very low conflict, but there are many data structures that tend to be
>  higher conflict, like inventory data. In order for there to be
>  conflict in inventory data all it takes is for two customers to order
>  the same product at roughly the same time (ie within the span of the
>  time it takes the order transaction to execute, which can be tens of
>  seconds sometimes). For a popular item on a busy site this isn't just
>  possible, it's really likely. In this case optimistic locking wouldn't
>  be that helpful, ie you don't want the behavior where the system
>  essentially says "someone else is ordering that product right now,
>  please try again later". What you would want is for the database to
>  lock certain records so that the second user waits until the first
>  user makes any changes. And, what you want them to wait on is being
>  able to READ the data, not waiting to WRITE it. The common scenario is
>  that two different threads read the current inventory value, then both
>  are working on things including decrementing the inventory value, then
>  both write it. In the end the result will be wrong because they both
>  started with the same value and subtracted from it, and basically
>  whoever writes first will have their value ignored and the total at
>  the end will just be the original value minus whatever the second
>  thread to write subtracted.
>
> That is a case where pessimistic locking is necessary, and a case where
>  things aren't as simple as they may seem.
>
> To understand some of the complexity check out the concept of
>  "transaction isolation". The big trick is that for performance and
>  concurrency reasons databases do NOT totally isolate transactions and
>  update conflicts can easily occur:
>
> http://en.wikipedia.org/wiki/Isolation_(database_systems)
>
> Many databases don't even support the more strict transaction isolation
>  levels, and even if they do they are not commonly used except for
>  special purposes. With things like SERIALIZABLE the problem is that
>  you end up locking, in many cases, entire tables and not just rows
>  within those tables and you have HUGE concurrency and deadlock
>  problems that result.
>
> The most common level you'll see used is READ COMMITTED, and sometimes
>  READ UNCOMMITTED if the database doesn't support READ COMMITTED. You
>  can see these settings in the entityengine.xml file.
>
> That is where SELECT FOR UPDATE is useful. You don't want to use the
>  SERIALIZABLE transaction isolation, but you want this certain record
>  locked even though it hasn't been changed so that other transactions
>  don't read the incorrect value.
>
> -David
>
>
> On Aug 14, 2010, at 12:45 AM, Matt Warnock wrote:
>
> > I'm still a bit confused.  I think I understand the issues, but not why
> > so many people are apparently having trouble with them.  Or maybe I
> > misunderstand them completely.
> >
> > Optimistic locking (as I understand it) is used primarily when editing
> > an existing record by hand, since record creation and programmed updates
> > can just use transactions, which are better for most operations anyway.
> > Most common business cases I can imagine would not usually involve 2
> > people editing (not just viewing) the same record at the same time.
> > What business scenario causes these apparently common collisions?
> >
> > Most high-volume business uses don't edit other people's records.  If I
> > enter an e-commerce order for example, I create the header record,
> > several line item records, perhaps some other stuff.  Eventually I
> > commit the whole order at once, when it is assigned an order number and
> > becomes part of the main database, which can all be done in a single
> > transaction.  
> >
> > Others may be entering similar orders, but they are creating different
> > header records with different associated line items.  These records
> > should all be accumulated into memory-only or temporary tables (I would
> > assume) until they are committed to the database, and optimistic locking
> > should never really enter into it, as these records are private to the
> > user and current session (like an e-commerce shopping cart) until they
> > are committed.  If they are abandoned before they commit, they should
> > never leave a trace in the main database, as I see it.  Any code that
> > updates the record (to total it, apply taxes, figure shipping, or
> > whatever) can work in-memory, or in a single transaction on the
> > temporary records, until the whole thing is committed.
> >
> > If I then go back and edit an order, it is usually one I just recently
> > entered, and in most cases, no one else should be using it.  When I do
> > that, the optimistic lock code should read the record data and note the
> > time that the record was last modified (or the data itself). I then edit
> > that data on-screen, and when I commit, it first checks to see that the
> > data was not modified in the meantime.  In most cases, it wasn't
> > modified, and the new data is written, again within the scope of a
> > single transaction.
> >
> > If the last-modified date (or the original data) has changed, then a
> > collision has occurred, and the system should cancel my commit, because
> > I was editing data which has changed while I was editing it, and is now
> > stale.  In most cases, any manual edit takes much more than a second, so
> > the chance of a time granularity collision on an actual record edit
> > seems miniscule. If there is a collision, the system re-reads the
> > recently updated data, tells me about the collision, probably discards
> > the previous edits, and I can then edit again if necessary.
> >
> > It's a poor substitute for an update transaction, but you don't want to
> > lock a database up for several minutes while a user edits a record by
> > hand, and most transactions will timeout long before the user finishes
> > the edit.
> >
> > Programmatic data updates like Mike Z describes are much more common,
> > but they can usually be managed in a single transaction too.  I don't
> > need a lock to calculate a total, enter a timestamp, or similar updates,
> > as these can all be done inside an ACID transaction, thereby protected
> > from other threads, users, application servers, or whatever.  We can
> > even suspend one transaction to run an unrelated one, then resume the
> > first, as David suggested earlier in this thread.
> >
> > Can you give me an example of the kind of update that leads to the kind
> > of concurrency issues you describe?  Is OFBiz using optimistic locks
> > where transactions are really required?  Or what about James' inventory
> > count scenario prevents using a transaction instead of an optimistic
> > lock?  What am I missing?  Just want to know where the big bear traps
> > might be.  Thanks in advance.
> >
> > --
> > Matt Warnock <[hidden email]>
> > RidgeCrest Herbals, Inc.
> >
> >
> > On Fri, 2010-08-13 at 19:52 -0700, Mike Z wrote:
> >> This has been a very useful thread.  I now know that I need to dump
> >> MySQL asap.   I planned on running multiple ofbiz instances for
> >> ecommerce and had no idea that this may cause issues.  Thanks for the
> >> input.
> >>
> >> On Fri, Aug 13, 2010 at 5:31 PM, Brett Palmer <[hidden email]> wrote:
> >>> James,
> >>>
> >>> We have run into this same problem on MySQL and ofbiz.  We worked around the
> >>> problem by creating a custom method that got a direction connection from the
> >>> transaction manager.  Then we wrote a custom SELECT for UPDATE on that
> >>> connection.  We needed this functionality because we had multiple
> >>> application servers hitting the same database and ran into concurrency
> >>> problems without it.
> >>>
> >>> I would like to see the optimistic locking feature enhanced in ofbiz.  Maybe
> >>> we could move away from timestamps and use an increasing unique ID as a
> >>> replacement.  This is definitely a problem with MySQL.  We may move away
> >>> from MySQL if we can find a good replication solution from Postgres.
> >>>
> >>>
> >>> Brett
> >>>
> >>> On Thu, Aug 12, 2010 at 2:15 PM, James McGill <
> >>> [hidden email]> wrote:
> >>>
> >>>> We are having problems with the optimistic locking.   With "enable-lock"
> >>>> set
> >>>> on an Entity, updates in GenericDAO use a timestamp to do locking.
> >>>> There are a number of issues with this.  The biggest one is that it's not a
> >>>> synchronized operation, so there's potential for a race condition within
> >>>> customUpdate, which we are actually seeing in production.
> >>>> I added code to introduce the "FOR UPDATE" expression when reading the
> >>>> timestamp.  This brings up another issue, that the timestamp field in MySQL
> >>>> has resolution only to the second.  So even if you don't have contention on
> >>>> the optimistic lock SELECT, you still have to be lucky that your
> >>>> transactions are more than one second apart.
> >>>>
> >>>> I realize this is a fairly difficult problem to address, in general, and
> >>>> that "fixing" many concurrency issues leads to risks of deadlock.  But we
> >>>> are seeing errors in data where the "last update wins."
> >>>>
> >>>> Has anyone else had concurrency problems when multiple threads are updating
> >>>> entities?  Are there any locking provisions in the Delegator that would
> >>>> allow us to prevent this kind of problem?
> >>>>
> >>>> --
> >>>> James McGill
> >>>> Phoenix AZ
> >>>>
> >>>
> >

[David E. Jones-2]

On Aug 14, 2010, at 2:02 PM, Matt Warnock wrote:

> Thank you David for the detailed and thoughtful explanation.  I think I
> see the issue now.
>
> However, I still see it as largely a business process issue, not a
> programming issue.  You're right, you don't want to tell a customer that
> a product is available, then later tell them it isn't, because someone
> beat them to it.  But the real fact of life is that shopping carts are
> abandoned all the time, so you really shouldn't commit inventory to any
> order until the order is placed and approved.  There are lots of
> potential slips between order placement and shipping (when the product
> REALLY moves out of inventory)-- abandoned carts, declined credit, bad
> shipping address, no shipping service to that location, import/export
> restrictions, management fiat, whatever.  I exercised management fiat
> just yesterday. :)
>
> Near-simultaneous inventory transactions *per se* shouldn't be a
> problem.  If I order 5 items, then when I commit, my code sets "qty =
> qty - 5" in a single sql statement and is further enclosed within a
> transaction that checks that the resulting (or beginning) qty is not
> less than zero or some other minimum number, or from mixed lot numbers,
> or whatever else your business logic may require.  Very busy sites won't
> collide unless they try to commit inventory to "pending" orders with an
> optimistic lock, which I think is a mistake.  
>
> If inventory levels are high and goods are fungible, you don't need to
> worry about it.  Tens of seconds, or even days, isn't going to make a
> difference, and even if the item is serialized, you allocate the serial
> number when the order is filled.  
>
> If inventory is low, or unique, then you have a scarcity problem, and
> you may want to warn the user of that fact, both to encourage the quick
> order, and to manage expectations, so that they are not disappointed--
> but locking the inventory pending an order that may or may not happen
> seems like a wrong approach to me.  If people are doing that, then I
> certainly understand why they are seeing concurrency issues on busy
> sites.
>
> Another possible approach would be to enter "pending" orders in a
> separate table (or flagged as such in the same table), and only deduct
> from the actual inventory if and when the sale is finally approved.
> That way you could see whether inventory is scarce (orders pending equal
> or exceed some ratio of inventory on hand) and also impose a "first
> come, first served" approach on pending orders, while still imposing
> time limits on completing the order, and possibly even enabling a
> "waiting list".  Kind of like an appointment queue-- "we've had a
> cancellation, so we can take you today at 3pm."  
>
> It just seems to me that optimistic locking is not the right solution to
> a scarce inventory problem.

The scenario I was talking about was for inventory reservation when an order is placed, ie decrementing ATP (Available To Promise). The scenario could happen for QOH on shipping in a busy warehouse with many packers (or at least more than one packer). A really common scenario discussed to demonstrate this point is bank accounts and the possibility of getting an incorrect total because of multiple transactions happening at close enough to the same time.

There are many scenarios, you don't have to do something like reservations on add-to-cart in order for this problem to appear. All you need, in theory, is two users changing the same data at close enough to the same time. In practice it usually happens when there are many users as it increases the chances that two will have a conflict.

BTW, I guess I didn't explain it very well, because I did say the solution is not optimistic locking, but pessimistic locking and more specifically a lock on read (as opposed to the more common lock on write in the ReadCommitted transaction isolation).

> In my view, any database that relaxes the ACID requirements to achieve
> concurrency is barely worthy of the name.  ERP (especially financial
> accounting) requires ACID, period.  You can't run an enterprise on less.
> But it does mean you need to break your business processes down into
> small, atomic steps so that-- like using a hammer to drive a screw.
> isolation requirements do not become a concurrency nightmare.  And some
> longer steps (like internally consistent dumps of the entire database)
> may need to be timed carefully.

You might be surprised to find out that roughly 99.99999% (okay, maybe it's only 5 nines, or maybe only 3, but certainly at least two) of business transactions are run on databases that are not strictly ACID, or at least the I (isolation) part of it. That's just the reality of things. The remaining transactions usually use alternative techniques for isolation, such as SELECT FOR UPDATE, instead of making the whole database always use strictly isolated transactions. Databases running in RepeatableRead or Serializable modes are EXTREMELY rare, and many "enterprise" databases don't even support these isolation levels.

-David

> Thanks again for your thoughtful explanation.
>
> --
> Matt Warnock <[hidden email]>
> RidgeCrest Herbals, Inc.
>
> On Sat, 2010-08-14 at 09:46 -0600, David E Jones wrote:
>> Timestamp-based optimistic locking is a feature of the Entity Engine,
>> but it is not used very much in OFBiz. In fact, I'm not sure if it's
>> used at all. The way it came into this discussion was, I suppose, as a
>> possible solution to the synchronization problems people were having
>> with race conditions.
>>
>> As you mentioned here, which is correct, optimistic locking is really
>> only helpful if two people are possibly editing the same data at the
>> same time and you want to notify a user if another user has changed
>> the data they are working on between the time they got the data from
>> the database, and the time they saved their changes to the database.
>> With most manual editing, as you mentioned, the reading and writing
>> are done in two separate transactions, so that is a case where a
>> SELECT FOR UPDATE would not help. As you said, in order for that to be
>> helpful in the common case where optimistic locks are used the
>> transaction would have to live for many minutes and lock resources for
>> that entire time (ie a pessimistic lock).
>>
>> There are certainly cases where optimistic locks might be useful, and
>> they would be things mostly done manually like editing product
>> information or any content that lives in the database. Two people
>> could accidentally be working on the same product or content at the
>> same time, and without optimistic locking the person who saved second
>> would wipe out the changes of the person who saved first, but neither
>> would know it unless they manually review the data at a later time. If
>> pessimistic locking were used in these scenarios it would be like
>> those REALLY annoying old source repositories where if you check out a
>> file it is "locked" and no one else can change it until you check it
>> back in and release the lock (ie they didn't bother to implement any
>> sort of merging). With the Entity Engine optimistic lock it won't try
>> to do any merging, the purpose is to notify the user that someone else
>> changed the data they were working on between the time they read the
>> data to edit and the time they tried to save it (the separate read and
>> write transactions).
>>
>> For many race conditions that cause bigger problems the scenario is
>> very different. In your example of order data that is likely to be
>> very low conflict, but there are many data structures that tend to be
>> higher conflict, like inventory data. In order for there to be
>> conflict in inventory data all it takes is for two customers to order
>> the same product at roughly the same time (ie within the span of the
>> time it takes the order transaction to execute, which can be tens of
>> seconds sometimes). For a popular item on a busy site this isn't just
>> possible, it's really likely. In this case optimistic locking wouldn't
>> be that helpful, ie you don't want the behavior where the system
>> essentially says "someone else is ordering that product right now,
>> please try again later". What you would want is for the database to
>> lock certain records so that the second user waits until the first
>> user makes any changes. And, what you want them to wait on is being
>> able to READ the data, not waiting to WRITE it. The common scenario is
>> that two different threads read the current inventory value, then both
>> are working on things including decrementing the inventory value, then
>> both write it. In the end the result will be wrong because they both
>> started with the same value and subtracted from it, and basically
>> whoever writes first will have their value ignored and the total at
>> the end will just be the original value minus whatever the second
>> thread to write subtracted.
>>
>> That is a case where pessimistic locking is necessary, and a case where
>> things aren't as simple as they may seem.
>>
>> To understand some of the complexity check out the concept of
>> "transaction isolation". The big trick is that for performance and
>> concurrency reasons databases do NOT totally isolate transactions and
>> update conflicts can easily occur:
>>
>> http://en.wikipedia.org/wiki/Isolation_(database_systems)
>>
>> Many databases don't even support the more strict transaction isolation
>> levels, and even if they do they are not commonly used except for
>> special purposes. With things like SERIALIZABLE the problem is that
>> you end up locking, in many cases, entire tables and not just rows
>> within those tables and you have HUGE concurrency and deadlock
>> problems that result.
>>
>> The most common level you'll see used is READ COMMITTED, and sometimes
>> READ UNCOMMITTED if the database doesn't support READ COMMITTED. You
>> can see these settings in the entityengine.xml file.
>>
>> That is where SELECT FOR UPDATE is useful. You don't want to use the
>> SERIALIZABLE transaction isolation, but you want this certain record
>> locked even though it hasn't been changed so that other transactions
>> don't read the incorrect value.
>>
>> -David
>>
>>
>> On Aug 14, 2010, at 12:45 AM, Matt Warnock wrote:
>>
>>> I'm still a bit confused.  I think I understand the issues, but not why
>>> so many people are apparently having trouble with them.  Or maybe I
>>> misunderstand them completely.
>>>
>>> Optimistic locking (as I understand it) is used primarily when editing
>>> an existing record by hand, since record creation and programmed updates
>>> can just use transactions, which are better for most operations anyway.
>>> Most common business cases I can imagine would not usually involve 2
>>> people editing (not just viewing) the same record at the same time.
>>> What business scenario causes these apparently common collisions?
>>>
>>> Most high-volume business uses don't edit other people's records.  If I
>>> enter an e-commerce order for example, I create the header record,
>>> several line item records, perhaps some other stuff.  Eventually I
>>> commit the whole order at once, when it is assigned an order number and
>>> becomes part of the main database, which can all be done in a single
>>> transaction.  
>>>
>>> Others may be entering similar orders, but they are creating different
>>> header records with different associated line items.  These records
>>> should all be accumulated into memory-only or temporary tables (I would
>>> assume) until they are committed to the database, and optimistic locking
>>> should never really enter into it, as these records are private to the
>>> user and current session (like an e-commerce shopping cart) until they
>>> are committed.  If they are abandoned before they commit, they should
>>> never leave a trace in the main database, as I see it.  Any code that
>>> updates the record (to total it, apply taxes, figure shipping, or
>>> whatever) can work in-memory, or in a single transaction on the
>>> temporary records, until the whole thing is committed.
>>>
>>> If I then go back and edit an order, it is usually one I just recently
>>> entered, and in most cases, no one else should be using it.  When I do
>>> that, the optimistic lock code should read the record data and note the
>>> time that the record was last modified (or the data itself). I then edit
>>> that data on-screen, and when I commit, it first checks to see that the
>>> data was not modified in the meantime.  In most cases, it wasn't
>>> modified, and the new data is written, again within the scope of a
>>> single transaction.
>>>
>>> If the last-modified date (or the original data) has changed, then a
>>> collision has occurred, and the system should cancel my commit, because
>>> I was editing data which has changed while I was editing it, and is now
>>> stale.  In most cases, any manual edit takes much more than a second, so
>>> the chance of a time granularity collision on an actual record edit
>>> seems miniscule. If there is a collision, the system re-reads the
>>> recently updated data, tells me about the collision, probably discards
>>> the previous edits, and I can then edit again if necessary.
>>>
>>> It's a poor substitute for an update transaction, but you don't want to
>>> lock a database up for several minutes while a user edits a record by
>>> hand, and most transactions will timeout long before the user finishes
>>> the edit.
>>>
>>> Programmatic data updates like Mike Z describes are much more common,
>>> but they can usually be managed in a single transaction too.  I don't
>>> need a lock to calculate a total, enter a timestamp, or similar updates,
>>> as these can all be done inside an ACID transaction, thereby protected
>>> from other threads, users, application servers, or whatever.  We can
>>> even suspend one transaction to run an unrelated one, then resume the
>>> first, as David suggested earlier in this thread.
>>>
>>> Can you give me an example of the kind of update that leads to the kind
>>> of concurrency issues you describe?  Is OFBiz using optimistic locks
>>> where transactions are really required?  Or what about James' inventory
>>> count scenario prevents using a transaction instead of an optimistic
>>> lock?  What am I missing?  Just want to know where the big bear traps
>>> might be.  Thanks in advance.
>>>
>>> --
>>> Matt Warnock <[hidden email]>
>>> RidgeCrest Herbals, Inc.
>>>
>>>
>>> On Fri, 2010-08-13 at 19:52 -0700, Mike Z wrote:
>>>> This has been a very useful thread.  I now know that I need to dump
>>>> MySQL asap.   I planned on running multiple ofbiz instances for
>>>> ecommerce and had no idea that this may cause issues.  Thanks for the
>>>> input.
>>>>
>>>> On Fri, Aug 13, 2010 at 5:31 PM, Brett Palmer <[hidden email]> wrote:
>>>>> James,
>>>>>
>>>>> We have run into this same problem on MySQL and ofbiz.  We worked around the
>>>>> problem by creating a custom method that got a direction connection from the
>>>>> transaction manager.  Then we wrote a custom SELECT for UPDATE on that
>>>>> connection.  We needed this functionality because we had multiple
>>>>> application servers hitting the same database and ran into concurrency
>>>>> problems without it.
>>>>>
>>>>> I would like to see the optimistic locking feature enhanced in ofbiz.  Maybe
>>>>> we could move away from timestamps and use an increasing unique ID as a
>>>>> replacement.  This is definitely a problem with MySQL.  We may move away
>>>>> from MySQL if we can find a good replication solution from Postgres.
>>>>>
>>>>>
>>>>> Brett
>>>>>
>>>>> On Thu, Aug 12, 2010 at 2:15 PM, James McGill <
>>>>> [hidden email]> wrote:
>>>>>
>>>>>> We are having problems with the optimistic locking.   With "enable-lock"
>>>>>> set
>>>>>> on an Entity, updates in GenericDAO use a timestamp to do locking.
>>>>>> There are a number of issues with this.  The biggest one is that it's not a
>>>>>> synchronized operation, so there's potential for a race condition within
>>>>>> customUpdate, which we are actually seeing in production.
>>>>>> I added code to introduce the "FOR UPDATE" expression when reading the
>>>>>> timestamp.  This brings up another issue, that the timestamp field in MySQL
>>>>>> has resolution only to the second.  So even if you don't have contention on
>>>>>> the optimistic lock SELECT, you still have to be lucky that your
>>>>>> transactions are more than one second apart.
>>>>>>
>>>>>> I realize this is a fairly difficult problem to address, in general, and
>>>>>> that "fixing" many concurrency issues leads to risks of deadlock.  But we
>>>>>> are seeing errors in data where the "last update wins."
>>>>>>
>>>>>> Has anyone else had concurrency problems when multiple threads are updating
>>>>>> entities?  Are there any locking provisions in the Delegator that would
>>>>>> allow us to prevent this kind of problem?
>>>>>>
>>>>>> --
>>>>>> James McGill
>>>>>> Phoenix AZ
>>>>>>
>>>>>
>>>
>