I am trying to understand the database connection pooling in ofbiz 14,
which uses dbcp2. In the class org.ofbiz.entity.connection.DBCPConnectionFactory there is a static ConcurrentHashMap which provides a ManagedDataSource for each tenant. But when it comes to obtaining a transaction manager reference, it looks like there is one global instance for all tenants. getTransactionManager in org.ofbiz.geronimo.GeronimoTransactionFactory returns the static reference to a single GeronimoTransactionManager. In multi tenant testing overlapping from different tenants seem like they interact with eachother. Can you explain how the implementation is supposed to work in with multi tenancy ? |
Hi Justin,
Straight and simple answer (I trust): The tenandId used by the user loggin into the system is used by the OFBiz system as the delegatorId. This delegatorId is then used (amongst others) by the underlying services regarding the entity engine as the differentiator for executing CRUD functions to the right database. Best regards, Pierre Smits ORRTIZ.COM <http://www.orrtiz.com> OFBiz based solutions & services OFBiz Extensions Marketplace http://oem.ofbizci.net/oci-2/ On Wed, Jul 20, 2016 at 6:38 AM, Justin Robinson < [hidden email]> wrote: > I am trying to understand the database connection pooling in ofbiz 14, > which uses dbcp2. > > In the class org.ofbiz.entity.connection.DBCPConnectionFactory there is a > static ConcurrentHashMap which provides a ManagedDataSource for each > tenant. > > But when it comes to obtaining a transaction manager reference, it looks > like there is one global instance for all tenants. > > getTransactionManager in org.ofbiz.geronimo.GeronimoTransactionFactory > returns the static reference to a single GeronimoTransactionManager. > > In multi tenant testing overlapping from different tenants seem like they > interact with eachother. > > Can you explain how the implementation is supposed to work in with multi > tenancy ? > |
In reply to this post by Justin Robinson-2
Hi Justin,
your understanding of the class implementation is correct and there may be a chance that some code may need to be improved, but I am not sure as I could only give a cursory look at the code. By the way, I see that the code in GeronimoTransactionManager calls a "tenant-aware" method in DBCPConnectionFactory and this should provide the correct connection to every thread. Could you please elaborate more on your concern? If you could share a usage/failure scenario to recreate the issue it would be great too. Thanks, Jacopo On Wed, Jul 20, 2016 at 6:38 AM, Justin Robinson < [hidden email]> wrote: > I am trying to understand the database connection pooling in ofbiz 14, > which uses dbcp2. > > In the class org.ofbiz.entity.connection.DBCPConnectionFactory there is a > static ConcurrentHashMap which provides a ManagedDataSource for each > tenant. > > But when it comes to obtaining a transaction manager reference, it looks > like there is one global instance for all tenants. > > getTransactionManager in org.ofbiz.geronimo.GeronimoTransactionFactory > returns the static reference to a single GeronimoTransactionManager. > > In multi tenant testing overlapping from different tenants seem like they > interact with eachother. > > Can you explain how the implementation is supposed to work in with multi > tenancy ? > |
I am really just interested in the architecture, it's no so much an issue
with any version of ofbiz. (In fact I am working on a system based on ofbiz, it would be better to replace it with pure ofbiz, but that is not my decision to make). I hope you will still be willing to discuss your implementation regardless of that fact. Yes org.ofbiz.geronimo.GeronimoTransactionFactory calls ConnectionFactoryLoader.getInstance().getConnection(helperInfo, datasourceInfo.getInlineJdbc()); which is implemented through org.ofbiz.entity.connection.DBCPConnectionFactory.getConnection(GenericHelperInfo, Element) which uses a ConcurrentHashMap to store a connection pool per tenant datasource and it seems to work very well, the correct pool is provided to the appropriate tenant threads. This is an intermittent issue when testing with 50-100 tenants, transactions across tenants seem to collide occasionally. At least that seems to be the case, I have stepped through the code in a debugger. It seem to happen in SqlProcessor finalize method, the gc calls finalize and when the connection is closed (returned to the pool) it calls commit first and then rolls back because a transaction is in progress as a result the connection is leaked and doesn't return to the pool. This is clearly seen on a massive scale in jprofiler on the jdbc probe page. Is it a limitation of the architecture? Is it possible to provide a TransactionManger per tenant in ThreadLocal or would that not work? Or am I missing something here? I realise it's a complex issue, but I am trying to stay as close to ofbiz as I can I hope one day we can replace the framework with a pure ofbiz implementation. I'd rather not make a change like this which diverges from ofbiz architecture. How would you solve this issue? On Wed, Jul 20, 2016 at 1:17 PM, Jacopo Cappellato < [hidden email]> wrote: > Hi Justin, > > your understanding of the class implementation is correct and there may be > a chance that some code may need to be improved, but I am not sure as I > could only give a cursory look at the code. > By the way, I see that the code in GeronimoTransactionManager calls a > "tenant-aware" method in DBCPConnectionFactory and this should provide the > correct connection to every thread. > > Could you please elaborate more on your concern? If you could share a > usage/failure scenario to recreate the issue it would be great too. > > Thanks, > > Jacopo > > On Wed, Jul 20, 2016 at 6:38 AM, Justin Robinson < > [hidden email]> wrote: > > > I am trying to understand the database connection pooling in ofbiz 14, > > which uses dbcp2. > > > > In the class org.ofbiz.entity.connection.DBCPConnectionFactory there is a > > static ConcurrentHashMap which provides a ManagedDataSource for each > > tenant. > > > > But when it comes to obtaining a transaction manager reference, it looks > > like there is one global instance for all tenants. > > > > getTransactionManager in org.ofbiz.geronimo.GeronimoTransactionFactory > > returns the static reference to a single GeronimoTransactionManager. > > > > In multi tenant testing overlapping from different tenants seem like they > > interact with eachother. > > > > Can you explain how the implementation is supposed to work in with multi > > tenancy ? > > > |
Hi Justin,
On Wed, Jul 20, 2016 at 12:37 PM, Justin Robinson < [hidden email]> wrote: > I am really just interested in the architecture, it's no so much an issue > with any version of ofbiz. (In fact I am working on a system based on > ofbiz, it would be better to replace it with pure ofbiz, but that is not my > decision to make). I hope you will still be willing to discuss your > implementation regardless of that fact. > Definitely! > > Yes org.ofbiz.geronimo.GeronimoTransactionFactory > calls ConnectionFactoryLoader.getInstance().getConnection(helperInfo, > datasourceInfo.getInlineJdbc()); > which is implemented > through > org.ofbiz.entity.connection.DBCPConnectionFactory.getConnection(GenericHelperInfo, > Element) which uses a > ConcurrentHashMap to store a connection pool per tenant datasource and it > seems to work very well, the correct pool is provided to the appropriate > tenant threads. > > This is an intermittent issue when testing with 50-100 tenants, > transactions across tenants seem to collide occasionally. At least that > seems to be the case, I have stepped through the code in a debugger. > > It seem to happen in SqlProcessor finalize method, the gc calls finalize > and when the connection is closed (returned to the pool) it calls commit > first and then rolls back because a transaction is in progress as a result > the connection is leaked and doesn't return to the pool. This is clearly > seen on a massive scale in jprofiler on the jdbc probe page. > Hmmm... this is an interesting detail, there could be an issue in the SqlProcessor class or maybe in the way that is used in your framework. I will dig into this more and let you know if I find something interesting. Jacopo > > Is it a limitation of the architecture? Is it possible to provide a > TransactionManger per tenant in ThreadLocal or would that not work? > > Or am I missing something here? I realise it's a complex issue, but I am > trying to stay as close to ofbiz as I can I hope one day we can replace the > framework with a pure ofbiz implementation. I'd rather not make a change > like this which diverges from ofbiz architecture. How would you solve this > issue? > > > On Wed, Jul 20, 2016 at 1:17 PM, Jacopo Cappellato < > [hidden email]> wrote: > > > Hi Justin, > > > > your understanding of the class implementation is correct and there may > be > > a chance that some code may need to be improved, but I am not sure as I > > could only give a cursory look at the code. > > By the way, I see that the code in GeronimoTransactionManager calls a > > "tenant-aware" method in DBCPConnectionFactory and this should provide > the > > correct connection to every thread. > > > > Could you please elaborate more on your concern? If you could share a > > usage/failure scenario to recreate the issue it would be great too. > > > > Thanks, > > > > Jacopo > > > > On Wed, Jul 20, 2016 at 6:38 AM, Justin Robinson < > > [hidden email]> wrote: > > > > > I am trying to understand the database connection pooling in ofbiz 14, > > > which uses dbcp2. > > > > > > In the class org.ofbiz.entity.connection.DBCPConnectionFactory there > is a > > > static ConcurrentHashMap which provides a ManagedDataSource for each > > > tenant. > > > > > > But when it comes to obtaining a transaction manager reference, it > looks > > > like there is one global instance for all tenants. > > > > > > getTransactionManager in org.ofbiz.geronimo.GeronimoTransactionFactory > > > returns the static reference to a single GeronimoTransactionManager. > > > > > > In multi tenant testing overlapping from different tenants seem like > they > > > interact with eachother. > > > > > > Can you explain how the implementation is supposed to work in with > multi > > > tenancy ? > > > > > > |
Is it possible that your code is creating EntityListIterator objects and
not properly closing them? This may cause connections to stay open until the SqlProcessor objects are finalized. My guess is that the connection leaks are happening before the finalization, and not as a consequence of it, because EntityListIterator are not properly closed, and a db transaction is active when the finalize method attempts to close the connection. Based on the above assumptions, that could be wrong, I don't think that there is an interference of transactions for different tenants, but rather data that you would expect to see is rolled back when the connection is closed before a commit. Please try to investigate in this direction and let me know if you do any progress. Good luck! Jacopo On Wed, Jul 20, 2016 at 1:51 PM, Jacopo Cappellato < [hidden email]> wrote: > Hi Justin, > > On Wed, Jul 20, 2016 at 12:37 PM, Justin Robinson < > [hidden email]> wrote: > >> I am really just interested in the architecture, it's no so much an issue >> with any version of ofbiz. (In fact I am working on a system based on >> ofbiz, it would be better to replace it with pure ofbiz, but that is not >> my >> decision to make). I hope you will still be willing to discuss your >> implementation regardless of that fact. >> > > Definitely! > > >> >> Yes org.ofbiz.geronimo.GeronimoTransactionFactory >> calls ConnectionFactoryLoader.getInstance().getConnection(helperInfo, >> datasourceInfo.getInlineJdbc()); >> which is implemented >> through >> org.ofbiz.entity.connection.DBCPConnectionFactory.getConnection(GenericHelperInfo, >> Element) which uses a >> ConcurrentHashMap to store a connection pool per tenant datasource and it >> seems to work very well, the correct pool is provided to the appropriate >> tenant threads. >> >> This is an intermittent issue when testing with 50-100 tenants, >> transactions across tenants seem to collide occasionally. At least that >> seems to be the case, I have stepped through the code in a debugger. >> >> It seem to happen in SqlProcessor finalize method, the gc calls finalize >> and when the connection is closed (returned to the pool) it calls commit >> first and then rolls back because a transaction is in progress as a result >> the connection is leaked and doesn't return to the pool. This is clearly >> seen on a massive scale in jprofiler on the jdbc probe page. >> > > Hmmm... this is an interesting detail, there could be an issue in the > SqlProcessor class or maybe in the way that is used in your framework. > > I will dig into this more and let you know if I find something interesting. > > Jacopo > > >> >> Is it a limitation of the architecture? Is it possible to provide a >> TransactionManger per tenant in ThreadLocal or would that not work? >> >> Or am I missing something here? I realise it's a complex issue, but I am >> trying to stay as close to ofbiz as I can I hope one day we can replace >> the >> framework with a pure ofbiz implementation. I'd rather not make a change >> like this which diverges from ofbiz architecture. How would you solve this >> issue? >> >> >> On Wed, Jul 20, 2016 at 1:17 PM, Jacopo Cappellato < >> [hidden email]> wrote: >> >> > Hi Justin, >> > >> > your understanding of the class implementation is correct and there may >> be >> > a chance that some code may need to be improved, but I am not sure as I >> > could only give a cursory look at the code. >> > By the way, I see that the code in GeronimoTransactionManager calls a >> > "tenant-aware" method in DBCPConnectionFactory and this should provide >> the >> > correct connection to every thread. >> > >> > Could you please elaborate more on your concern? If you could share a >> > usage/failure scenario to recreate the issue it would be great too. >> > >> > Thanks, >> > >> > Jacopo >> > >> > On Wed, Jul 20, 2016 at 6:38 AM, Justin Robinson < >> > [hidden email]> wrote: >> > >> > > I am trying to understand the database connection pooling in ofbiz 14, >> > > which uses dbcp2. >> > > >> > > In the class org.ofbiz.entity.connection.DBCPConnectionFactory there >> is a >> > > static ConcurrentHashMap which provides a ManagedDataSource for each >> > > tenant. >> > > >> > > But when it comes to obtaining a transaction manager reference, it >> looks >> > > like there is one global instance for all tenants. >> > > >> > > getTransactionManager in org.ofbiz.geronimo.GeronimoTransactionFactory >> > > returns the static reference to a single GeronimoTransactionManager. >> > > >> > > In multi tenant testing overlapping from different tenants seem like >> they >> > > interact with eachother. >> > > >> > > Can you explain how the implementation is supposed to work in with >> multi >> > > tenancy ? >> > > >> > >> > > |
It was a chicken egg case. The transaction manger wasn't the problem. But I
am getting time outs now with tomcat nio exec threads being blocked in UtilCache and CachedClassloader those are major synchronization bottle necks but I have already updated then both to ofbiz 14. Anyway I suppose I should search the mail archive and then try a new thread. On Wed, Jul 20, 2016 at 6:32 PM, Jacopo Cappellato < [hidden email]> wrote: > Is it possible that your code is creating EntityListIterator objects and > not properly closing them? This may cause connections to stay open until > the SqlProcessor objects are finalized. > My guess is that the connection leaks are happening before the > finalization, and not as a consequence of it, because EntityListIterator > are not properly closed, and a db transaction is active when the finalize > method attempts to close the connection. > Based on the above assumptions, that could be wrong, I don't think that > there is an interference of transactions for different tenants, but rather > data that you would expect to see is rolled back when the connection is > closed before a commit. > Please try to investigate in this direction and let me know if you do any > progress. > > Good luck! > > Jacopo > > On Wed, Jul 20, 2016 at 1:51 PM, Jacopo Cappellato < > [hidden email]> wrote: > > > Hi Justin, > > > > On Wed, Jul 20, 2016 at 12:37 PM, Justin Robinson < > > [hidden email]> wrote: > > > >> I am really just interested in the architecture, it's no so much an > issue > >> with any version of ofbiz. (In fact I am working on a system based on > >> ofbiz, it would be better to replace it with pure ofbiz, but that is not > >> my > >> decision to make). I hope you will still be willing to discuss your > >> implementation regardless of that fact. > >> > > > > Definitely! > > > > > >> > >> Yes org.ofbiz.geronimo.GeronimoTransactionFactory > >> calls ConnectionFactoryLoader.getInstance().getConnection(helperInfo, > >> datasourceInfo.getInlineJdbc()); > >> which is implemented > >> through > >> > org.ofbiz.entity.connection.DBCPConnectionFactory.getConnection(GenericHelperInfo, > >> Element) which uses a > >> ConcurrentHashMap to store a connection pool per tenant datasource and > it > >> seems to work very well, the correct pool is provided to the appropriate > >> tenant threads. > >> > >> This is an intermittent issue when testing with 50-100 tenants, > >> transactions across tenants seem to collide occasionally. At least that > >> seems to be the case, I have stepped through the code in a debugger. > >> > >> It seem to happen in SqlProcessor finalize method, the gc calls finalize > >> and when the connection is closed (returned to the pool) it calls commit > >> first and then rolls back because a transaction is in progress as a > result > >> the connection is leaked and doesn't return to the pool. This is clearly > >> seen on a massive scale in jprofiler on the jdbc probe page. > >> > > > > Hmmm... this is an interesting detail, there could be an issue in the > > SqlProcessor class or maybe in the way that is used in your framework. > > > > I will dig into this more and let you know if I find something > interesting. > > > > Jacopo > > > > > >> > >> Is it a limitation of the architecture? Is it possible to provide a > >> TransactionManger per tenant in ThreadLocal or would that not work? > >> > >> Or am I missing something here? I realise it's a complex issue, but I am > >> trying to stay as close to ofbiz as I can I hope one day we can replace > >> the > >> framework with a pure ofbiz implementation. I'd rather not make a change > >> like this which diverges from ofbiz architecture. How would you solve > this > >> issue? > >> > >> > >> On Wed, Jul 20, 2016 at 1:17 PM, Jacopo Cappellato < > >> [hidden email]> wrote: > >> > >> > Hi Justin, > >> > > >> > your understanding of the class implementation is correct and there > may > >> be > >> > a chance that some code may need to be improved, but I am not sure as > I > >> > could only give a cursory look at the code. > >> > By the way, I see that the code in GeronimoTransactionManager calls a > >> > "tenant-aware" method in DBCPConnectionFactory and this should provide > >> the > >> > correct connection to every thread. > >> > > >> > Could you please elaborate more on your concern? If you could share a > >> > usage/failure scenario to recreate the issue it would be great too. > >> > > >> > Thanks, > >> > > >> > Jacopo > >> > > >> > On Wed, Jul 20, 2016 at 6:38 AM, Justin Robinson < > >> > [hidden email]> wrote: > >> > > >> > > I am trying to understand the database connection pooling in ofbiz > 14, > >> > > which uses dbcp2. > >> > > > >> > > In the class org.ofbiz.entity.connection.DBCPConnectionFactory there > >> is a > >> > > static ConcurrentHashMap which provides a ManagedDataSource for each > >> > > tenant. > >> > > > >> > > But when it comes to obtaining a transaction manager reference, it > >> looks > >> > > like there is one global instance for all tenants. > >> > > > >> > > getTransactionManager in > org.ofbiz.geronimo.GeronimoTransactionFactory > >> > > returns the static reference to a single GeronimoTransactionManager. > >> > > > >> > > In multi tenant testing overlapping from different tenants seem like > >> they > >> > > interact with eachother. > >> > > > >> > > Can you explain how the implementation is supposed to work in with > >> multi > >> > > tenancy ? > >> > > > >> > > >> > > > > > |
I am glad to hear you have at least a plan of action and you can move on.
Good luck Jacopo On Thu, Jul 21, 2016 at 9:16 AM, Justin Robinson < [hidden email]> wrote: > It was a chicken egg case. The transaction manger wasn't the problem. But I > am getting time outs now with tomcat nio exec threads being blocked in > UtilCache and CachedClassloader those are major synchronization bottle > necks but I have already updated then both to ofbiz 14. Anyway I suppose I > should search the mail archive and then try a new thread. > > > > On Wed, Jul 20, 2016 at 6:32 PM, Jacopo Cappellato < > [hidden email]> wrote: > > > Is it possible that your code is creating EntityListIterator objects and > > not properly closing them? This may cause connections to stay open until > > the SqlProcessor objects are finalized. > > My guess is that the connection leaks are happening before the > > finalization, and not as a consequence of it, because EntityListIterator > > are not properly closed, and a db transaction is active when the finalize > > method attempts to close the connection. > > Based on the above assumptions, that could be wrong, I don't think that > > there is an interference of transactions for different tenants, but > rather > > data that you would expect to see is rolled back when the connection is > > closed before a commit. > > Please try to investigate in this direction and let me know if you do any > > progress. > > > > Good luck! > > > > Jacopo > > > > On Wed, Jul 20, 2016 at 1:51 PM, Jacopo Cappellato < > > [hidden email]> wrote: > > > > > Hi Justin, > > > > > > On Wed, Jul 20, 2016 at 12:37 PM, Justin Robinson < > > > [hidden email]> wrote: > > > > > >> I am really just interested in the architecture, it's no so much an > > issue > > >> with any version of ofbiz. (In fact I am working on a system based on > > >> ofbiz, it would be better to replace it with pure ofbiz, but that is > not > > >> my > > >> decision to make). I hope you will still be willing to discuss your > > >> implementation regardless of that fact. > > >> > > > > > > Definitely! > > > > > > > > >> > > >> Yes org.ofbiz.geronimo.GeronimoTransactionFactory > > >> calls ConnectionFactoryLoader.getInstance().getConnection(helperInfo, > > >> datasourceInfo.getInlineJdbc()); > > >> which is implemented > > >> through > > >> > > > org.ofbiz.entity.connection.DBCPConnectionFactory.getConnection(GenericHelperInfo, > > >> Element) which uses a > > >> ConcurrentHashMap to store a connection pool per tenant datasource and > > it > > >> seems to work very well, the correct pool is provided to the > appropriate > > >> tenant threads. > > >> > > >> This is an intermittent issue when testing with 50-100 tenants, > > >> transactions across tenants seem to collide occasionally. At least > that > > >> seems to be the case, I have stepped through the code in a debugger. > > >> > > >> It seem to happen in SqlProcessor finalize method, the gc calls > finalize > > >> and when the connection is closed (returned to the pool) it calls > commit > > >> first and then rolls back because a transaction is in progress as a > > result > > >> the connection is leaked and doesn't return to the pool. This is > clearly > > >> seen on a massive scale in jprofiler on the jdbc probe page. > > >> > > > > > > Hmmm... this is an interesting detail, there could be an issue in the > > > SqlProcessor class or maybe in the way that is used in your framework. > > > > > > I will dig into this more and let you know if I find something > > interesting. > > > > > > Jacopo > > > > > > > > >> > > >> Is it a limitation of the architecture? Is it possible to provide a > > >> TransactionManger per tenant in ThreadLocal or would that not work? > > >> > > >> Or am I missing something here? I realise it's a complex issue, but I > am > > >> trying to stay as close to ofbiz as I can I hope one day we can > replace > > >> the > > >> framework with a pure ofbiz implementation. I'd rather not make a > change > > >> like this which diverges from ofbiz architecture. How would you solve > > this > > >> issue? > > >> > > >> > > >> On Wed, Jul 20, 2016 at 1:17 PM, Jacopo Cappellato < > > >> [hidden email]> wrote: > > >> > > >> > Hi Justin, > > >> > > > >> > your understanding of the class implementation is correct and there > > may > > >> be > > >> > a chance that some code may need to be improved, but I am not sure > as > > I > > >> > could only give a cursory look at the code. > > >> > By the way, I see that the code in GeronimoTransactionManager calls > a > > >> > "tenant-aware" method in DBCPConnectionFactory and this should > provide > > >> the > > >> > correct connection to every thread. > > >> > > > >> > Could you please elaborate more on your concern? If you could share > a > > >> > usage/failure scenario to recreate the issue it would be great too. > > >> > > > >> > Thanks, > > >> > > > >> > Jacopo > > >> > > > >> > On Wed, Jul 20, 2016 at 6:38 AM, Justin Robinson < > > >> > [hidden email]> wrote: > > >> > > > >> > > I am trying to understand the database connection pooling in ofbiz > > 14, > > >> > > which uses dbcp2. > > >> > > > > >> > > In the class org.ofbiz.entity.connection.DBCPConnectionFactory > there > > >> is a > > >> > > static ConcurrentHashMap which provides a ManagedDataSource for > each > > >> > > tenant. > > >> > > > > >> > > But when it comes to obtaining a transaction manager reference, it > > >> looks > > >> > > like there is one global instance for all tenants. > > >> > > > > >> > > getTransactionManager in > > org.ofbiz.geronimo.GeronimoTransactionFactory > > >> > > returns the static reference to a single > GeronimoTransactionManager. > > >> > > > > >> > > In multi tenant testing overlapping from different tenants seem > like > > >> they > > >> > > interact with eachother. > > >> > > > > >> > > Can you explain how the implementation is supposed to work in with > > >> multi > > >> > > tenancy ? > > >> > > > > >> > > > >> > > > > > > > > > |
Free forum by Nabble | Edit this page |