Memcached / central caching

[Leonard Lin-2]

Hi,
 
I would like to ask if someone has already investigated or implemented an adapter for OFBiz to use a central caching system like memcached, redis or the like.
 
I know for OFBiz cluster setup there is the distributed cache clear mechanism.
 
I did use that in the past but have found that with more than 2 instances it impacts performance of OFBiz adversely.
Symptoms we started to see in production:
- We started to see long running transaction locks, General Entity operations slower (might be a combination of load and clustering)
- Other idle hosts were 5%-15% busy only to process the cache-clear events without doing anything else
- Especially with higher load, the whole thing becomes even slower
 
More load -> More cluster instances -> more cache clear events -> more cycles used to process cache mgmt. -> general requests/operations slower -> so we need even more instances -> you see where it’s going
 That’s why I was thinking if someone has tried to integrate with a central cache.
 
For single host installation the local in-memory cache is surely faster, even perhaps with a 2host cluster setup,
but with 3 or more hosts, I think a central cache might scale better.
 
Also interested to hear about any thoughts on the idea and architecture.
 
Otherwise, any pointers where in the code to look at would be appreciated.
I'm thinking to implement it as an adaptor/driver you can switch on if you want it.
 
Thanks
Best
Leo

[Nick Rosser-2]

Leonard,

We have clients using distributed cache -- we used ActiveMQ to synchronize any cache modifications from one server to the other(s). Worked very well -- and we saw a significant performance uptick when adding additional "slave" instances.

Our implementations are typically for eCommerce -- with minimal cache-clear events.

Which entities are subject to high volume cache-clear events for you? Perhaps you could consider removing caching for those entities?

Nick

-----Original Message-----
From: Leonard Lin [mailto:[hidden email]]
Sent: Monday, June 27, 2016 6:24 AM
To: [hidden email]
Subject: Memcached / central caching

Hi,
 
I would like to ask if someone has already investigated or implemented an adapter for OFBiz to use a central caching system like memcached, redis or the like.
 
I know for OFBiz cluster setup there is the distributed cache clear mechanism.
 
I did use that in the past but have found that with more than 2 instances it impacts performance of OFBiz adversely.
Symptoms we started to see in production:
- We started to see long running transaction locks, General Entity operations slower (might be a combination of load and clustering)
- Other idle hosts were 5%-15% busy only to process the cache-clear events without doing anything else
- Especially with higher load, the whole thing becomes even slower
 
More load -> More cluster instances -> more cache clear events -> more cycles used to process cache mgmt. -> general requests/operations slower -> so we need even more instances -> you see where it’s going
 That’s why I was thinking if someone has tried to integrate with a central cache.
 
For single host installation the local in-memory cache is surely faster, even perhaps with a 2host cluster setup, but with 3 or more hosts, I think a central cache might scale better.
 
Also interested to hear about any thoughts on the idea and architecture.
 
Otherwise, any pointers where in the code to look at would be appreciated.
I'm thinking to implement it as an adaptor/driver you can switch on if you want it.
 
Thanks
Best
Leo

[Jacques Le Roux]

I can confirm this behaviour (working well and better with more instances to support the growing load) for a configuration of 5 OFBiz machines against
a cluster of 3 Postgres machines (using PgPool II).

It was also for an ecommerce application for a major Internet provider. I then I wrote the wiki page about DCC. As say Nick, you may indeed have a
specific issue we did not cross...

Jacques


Le 27/06/2016 à 15:11, Nick Rosser a écrit :

> Leonard,
>
> We have clients using distributed cache -- we used ActiveMQ to synchronize any cache modifications from one server to the other(s). Worked very well -- and we saw a significant performance uptick when adding additional "slave" instances.
>
> Our implementations are typically for eCommerce -- with minimal cache-clear events.
>
> Which entities are subject to high volume cache-clear events for you? Perhaps you could consider removing caching for those entities?
>
> Nick
>
> -----Original Message-----
> From: Leonard Lin [mailto:[hidden email]]
> Sent: Monday, June 27, 2016 6:24 AM
> To: [hidden email]
> Subject: Memcached / central caching
>
> Hi,
>  
> I would like to ask if someone has already investigated or implemented an adapter for OFBiz to use a central caching system like memcached, redis or the like.
>  
> I know for OFBiz cluster setup there is the distributed cache clear mechanism.
>  
> I did use that in the past but have found that with more than 2 instances it impacts performance of OFBiz adversely.
> Symptoms we started to see in production:
> - We started to see long running transaction locks, General Entity operations slower (might be a combination of load and clustering)
> - Other idle hosts were 5%-15% busy only to process the cache-clear events without doing anything else
> - Especially with higher load, the whole thing becomes even slower
>  
> More load -> More cluster instances -> more cache clear events -> more cycles used to process cache mgmt. -> general requests/operations slower -> so we need even more instances -> you see where it’s going
>   That’s why I was thinking if someone has tried to integrate with a central cache.
>  
> For single host installation the local in-memory cache is surely faster, even perhaps with a 2host cluster setup, but with 3 or more hosts, I think a central cache might scale better.
>  
> Also interested to hear about any thoughts on the idea and architecture.
>  
> Otherwise, any pointers where in the code to look at would be appreciated.
> I'm thinking to implement it as an adaptor/driver you can switch on if you want it.
>  
> Thanks
> Best
> Leo
>
>

[Leonard Lin-2]

Thanks for the input.
I have to try to look at logs to see which entities/processes are the high cache clear flow.
 
We are not using our ofbiz as an ecommerce system but as the manufacturing/fulfillment system.
We do process a lot of orders and have also custom modules.
 
 
 

Gesendet: Montag, 27. Juni 2016 um 15:46 Uhr
Von: "Jacques Le Roux" <[hidden email]>
An: [hidden email]
Betreff: Re: Memcached / central caching
I can confirm this behaviour (working well and better with more instances to support the growing load) for a configuration of 5 OFBiz machines against
a cluster of 3 Postgres machines (using PgPool II).

It was also for an ecommerce application for a major Internet provider. I then I wrote the wiki page about DCC. As say Nick, you may indeed have a
specific issue we did not cross...

Jacques


Le 27/06/2016 à 15:11, Nick Rosser a écrit :

> Leonard,
>
> We have clients using distributed cache -- we used ActiveMQ to synchronize any cache modifications from one server to the other(s). Worked very well -- and we saw a significant performance uptick when adding additional "slave" instances.
>
> Our implementations are typically for eCommerce -- with minimal cache-clear events.
>
> Which entities are subject to high volume cache-clear events for you? Perhaps you could consider removing caching for those entities?
>
> Nick
>
> -----Original Message-----
> From: Leonard Lin [mailto:[hidden email]]
> Sent: Monday, June 27, 2016 6:24 AM
> To: [hidden email]
> Subject: Memcached / central caching
>
> Hi,
>
> I would like to ask if someone has already investigated or implemented an adapter for OFBiz to use a central caching system like memcached, redis or the like.
>
> I know for OFBiz cluster setup there is the distributed cache clear mechanism.
>
> I did use that in the past but have found that with more than 2 instances it impacts performance of OFBiz adversely.
> Symptoms we started to see in production:
> - We started to see long running transaction locks, General Entity operations slower (might be a combination of load and clustering)
> - Other idle hosts were 5%-15% busy only to process the cache-clear events without doing anything else
> - Especially with higher load, the whole thing becomes even slower
>
> More load -> More cluster instances -> more cache clear events -> more cycles used to process cache mgmt. -> general requests/operations slower -> so we need even more instances -> you see where it’s going
> That’s why I was thinking if someone has tried to integrate with a central cache.
>
> For single host installation the local in-memory cache is surely faster, even perhaps with a 2host cluster setup, but with 3 or more hosts, I think a central cache might scale better.
>
> Also interested to hear about any thoughts on the idea and architecture.
>
> Otherwise, any pointers where in the code to look at would be appreciated.
> I'm thinking to implement it as an adaptor/driver you can switch on if you want it.
>
> Thanks
> Best
> Leo
>
>

 

[Jacques Le Roux]

Thanks Leonard,

So it could be indeed that DCC does not support this case well. It would be interesting to know why. Of course using something like memcached is
certainly a good way and a contribution would be quite welcome!

Jacques


Le 29/06/2016 à 08:33, Leonard Lin a écrit :

> Thanks for the input.
> I have to try to look at logs to see which entities/processes are the high cache clear flow.
>  
> We are not using our ofbiz as an ecommerce system but as the manufacturing/fulfillment system.
> We do process a lot of orders and have also custom modules.
>    
>  
>  
>
> Gesendet: Montag, 27. Juni 2016 um 15:46 Uhr
> Von: "Jacques Le Roux" <[hidden email]>
> An: [hidden email]
> Betreff: Re: Memcached / central caching
> I can confirm this behaviour (working well and better with more instances to support the growing load) for a configuration of 5 OFBiz machines against
> a cluster of 3 Postgres machines (using PgPool II).
>
> It was also for an ecommerce application for a major Internet provider. I then I wrote the wiki page about DCC. As say Nick, you may indeed have a
> specific issue we did not cross...
>
> Jacques
>
>
> Le 27/06/2016 à 15:11, Nick Rosser a écrit :
>> Leonard,
>>
>> We have clients using distributed cache -- we used ActiveMQ to synchronize any cache modifications from one server to the other(s). Worked very well -- and we saw a significant performance uptick when adding additional "slave" instances.
>>
>> Our implementations are typically for eCommerce -- with minimal cache-clear events.
>>
>> Which entities are subject to high volume cache-clear events for you? Perhaps you could consider removing caching for those entities?
>>
>> Nick
>>
>> -----Original Message-----
>> From: Leonard Lin [mailto:[hidden email]]
>> Sent: Monday, June 27, 2016 6:24 AM
>> To: [hidden email]
>> Subject: Memcached / central caching
>>
>> Hi,
>>
>> I would like to ask if someone has already investigated or implemented an adapter for OFBiz to use a central caching system like memcached, redis or the like.
>>
>> I know for OFBiz cluster setup there is the distributed cache clear mechanism.
>>
>> I did use that in the past but have found that with more than 2 instances it impacts performance of OFBiz adversely.
>> Symptoms we started to see in production:
>> - We started to see long running transaction locks, General Entity operations slower (might be a combination of load and clustering)
>> - Other idle hosts were 5%-15% busy only to process the cache-clear events without doing anything else
>> - Especially with higher load, the whole thing becomes even slower
>>
>> More load -> More cluster instances -> more cache clear events -> more cycles used to process cache mgmt. -> general requests/operations slower -> so we need even more instances -> you see where it’s going
>> That’s why I was thinking if someone has tried to integrate with a central cache.
>>
>> For single host installation the local in-memory cache is surely faster, even perhaps with a 2host cluster setup, but with 3 or more hosts, I think a central cache might scale better.
>>
>> Also interested to hear about any thoughts on the idea and architecture.
>>
>> Otherwise, any pointers where in the code to look at would be appreciated.
>> I'm thinking to implement it as an adaptor/driver you can switch on if you want it.
>>
>> Thanks
>> Best
>> Leo
>>
>>
>  
>
>

[Jacques Le Roux]

Ah forgot, Moqui is using http://www.ehcache.org/


Le 29/06/2016 à 08:55, Jacques Le Roux a écrit :

> Thanks Leonard,
>
> So it could be indeed that DCC does not support this case well. It would be interesting to know why. Of course using something like memcached is
> certainly a good way and a contribution would be quite welcome!
>
> Jacques
>
>
> Le 29/06/2016 à 08:33, Leonard Lin a écrit :
>> Thanks for the input.
>> I have to try to look at logs to see which entities/processes are the high cache clear flow.
>>   We are not using our ofbiz as an ecommerce system but as the manufacturing/fulfillment system.
>> We do process a lot of orders and have also custom modules.
>>
>> Gesendet: Montag, 27. Juni 2016 um 15:46 Uhr
>> Von: "Jacques Le Roux" <[hidden email]>
>> An: [hidden email]
>> Betreff: Re: Memcached / central caching
>> I can confirm this behaviour (working well and better with more instances to support the growing load) for a configuration of 5 OFBiz machines against
>> a cluster of 3 Postgres machines (using PgPool II).
>>
>> It was also for an ecommerce application for a major Internet provider. I then I wrote the wiki page about DCC. As say Nick, you may indeed have a
>> specific issue we did not cross...
>>
>> Jacques
>>
>>
>> Le 27/06/2016 à 15:11, Nick Rosser a écrit :
>>> Leonard,
>>>
>>> We have clients using distributed cache -- we used ActiveMQ to synchronize any cache modifications from one server to the other(s). Worked very
>>> well -- and we saw a significant performance uptick when adding additional "slave" instances.
>>>
>>> Our implementations are typically for eCommerce -- with minimal cache-clear events.
>>>
>>> Which entities are subject to high volume cache-clear events for you? Perhaps you could consider removing caching for those entities?
>>>
>>> Nick
>>>
>>> -----Original Message-----
>>> From: Leonard Lin [mailto:[hidden email]]
>>> Sent: Monday, June 27, 2016 6:24 AM
>>> To: [hidden email]
>>> Subject: Memcached / central caching
>>>
>>> Hi,
>>>
>>> I would like to ask if someone has already investigated or implemented an adapter for OFBiz to use a central caching system like memcached, redis
>>> or the like.
>>>
>>> I know for OFBiz cluster setup there is the distributed cache clear mechanism.
>>>
>>> I did use that in the past but have found that with more than 2 instances it impacts performance of OFBiz adversely.
>>> Symptoms we started to see in production:
>>> - We started to see long running transaction locks, General Entity operations slower (might be a combination of load and clustering)
>>> - Other idle hosts were 5%-15% busy only to process the cache-clear events without doing anything else
>>> - Especially with higher load, the whole thing becomes even slower
>>>
>>> More load -> More cluster instances -> more cache clear events -> more cycles used to process cache mgmt. -> general requests/operations slower ->
>>> so we need even more instances -> you see where it’s going
>>> That’s why I was thinking if someone has tried to integrate with a central cache.
>>>
>>> For single host installation the local in-memory cache is surely faster, even perhaps with a 2host cluster setup, but with 3 or more hosts, I
>>> think a central cache might scale better.
>>>
>>> Also interested to hear about any thoughts on the idea and architecture.
>>>
>>> Otherwise, any pointers where in the code to look at would be appreciated.
>>> I'm thinking to implement it as an adaptor/driver you can switch on if you want it.
>>>
>>> Thanks
>>> Best
>>> Leo
>>>
>>>
>>
>>
>

[Nick Rosser-2]

It's certainly possible that with a system that has a high volume of updates, on certain entities, that caching will degrade performance on a single-server or clustered environment.

I would certainly suggest that for constantly changing that caching be turned off, or have a quick expiration.

For you that could be Orders and related entities as they move through the manufacturing / fulfillment process? I'm sure you probably have a good sense as to the system usage, where high-volume updates take place, and how that impacts specific DB entities. Even in our eCommerce implementations we do not cache Order related data. We focus on the heavy usage / read-only aspects (Product catalog and related).

It may take some experimentation to really get to the solution. Interested to hear about your experience.

Nick

-----Original Message-----
From: Jacques Le Roux [mailto:[hidden email]]
Sent: Wednesday, June 29, 2016 3:09 AM
To: [hidden email]
Subject: Re: Aw: Re: Memcached / central caching

Ah forgot, Moqui is using http://www.ehcache.org/


Le 29/06/2016 à 08:55, Jacques Le Roux a écrit :

> Thanks Leonard,
>
> So it could be indeed that DCC does not support this case well. It
> would be interesting to know why. Of course using something like memcached is certainly a good way and a contribution would be quite welcome!
>
> Jacques
>
>
> Le 29/06/2016 à 08:33, Leonard Lin a écrit :
>> Thanks for the input.
>> I have to try to look at logs to see which entities/processes are the high cache clear flow.
>>   We are not using our ofbiz as an ecommerce system but as the manufacturing/fulfillment system.
>> We do process a lot of orders and have also custom modules.
>>
>> Gesendet: Montag, 27. Juni 2016 um 15:46 Uhr
>> Von: "Jacques Le Roux" <[hidden email]>
>> An: [hidden email]
>> Betreff: Re: Memcached / central caching I can confirm this behaviour
>> (working well and better with more instances to support the growing
>> load) for a configuration of 5 OFBiz machines against a cluster of 3 Postgres machines (using PgPool II).
>>
>> It was also for an ecommerce application for a major Internet
>> provider. I then I wrote the wiki page about DCC. As say Nick, you may indeed have a specific issue we did not cross...
>>
>> Jacques
>>
>>
>> Le 27/06/2016 à 15:11, Nick Rosser a écrit :
>>> Leonard,
>>>
>>> We have clients using distributed cache -- we used ActiveMQ to
>>> synchronize any cache modifications from one server to the other(s). Worked very well -- and we saw a significant performance uptick when adding additional "slave" instances.
>>>
>>> Our implementations are typically for eCommerce -- with minimal cache-clear events.
>>>
>>> Which entities are subject to high volume cache-clear events for you? Perhaps you could consider removing caching for those entities?
>>>
>>> Nick
>>>
>>> -----Original Message-----
>>> From: Leonard Lin [mailto:[hidden email]]
>>> Sent: Monday, June 27, 2016 6:24 AM
>>> To: [hidden email]
>>> Subject: Memcached / central caching
>>>
>>> Hi,
>>>
>>> I would like to ask if someone has already investigated or
>>> implemented an adapter for OFBiz to use a central caching system like memcached, redis or the like.
>>>
>>> I know for OFBiz cluster setup there is the distributed cache clear mechanism.
>>>
>>> I did use that in the past but have found that with more than 2 instances it impacts performance of OFBiz adversely.
>>> Symptoms we started to see in production:
>>> - We started to see long running transaction locks, General Entity
>>> operations slower (might be a combination of load and clustering)
>>> - Other idle hosts were 5%-15% busy only to process the cache-clear
>>> events without doing anything else
>>> - Especially with higher load, the whole thing becomes even slower
>>>
>>> More load -> More cluster instances -> more cache clear events ->
>>> more cycles used to process cache mgmt. -> general
>>> requests/operations slower -> so we need even more instances -> you see where it’s going That’s why I was thinking if someone has tried to integrate with a central cache.
>>>
>>> For single host installation the local in-memory cache is surely
>>> faster, even perhaps with a 2host cluster setup, but with 3 or more hosts, I think a central cache might scale better.
>>>
>>> Also interested to hear about any thoughts on the idea and architecture.
>>>
>>> Otherwise, any pointers where in the code to look at would be appreciated.
>>> I'm thinking to implement it as an adaptor/driver you can switch on if you want it.
>>>
>>> Thanks
>>> Best
>>> Leo
>>>
>>>
>>
>>
>

[Pierre Smits]

I wonder whether a 3rd party solution (e.g. memcached, ehcache, etc) would
be an improvement over current caching implementation.

What is your take on this? What would be the pro's and con's?

Best regards,

Pierre Smits

ORRTIZ.COM <http://www.orrtiz.com>
OFBiz based solutions & services

OFBiz Extensions Marketplace
http://oem.ofbizci.net/oci-2/

On Wed, Jun 29, 2016 at 4:58 PM, Nick Rosser <[hidden email]> wrote:

> It's certainly possible that with a system that has a high volume of
> updates, on certain entities, that caching will degrade performance on a
> single-server or clustered environment.
>
> I would certainly suggest that for constantly changing that caching be
> turned off, or have a quick expiration.
>
> For you that could be Orders and related entities as they move through the
> manufacturing / fulfillment process? I'm sure you probably have a good
> sense as to the system usage, where high-volume updates take place, and how
> that impacts specific DB entities. Even in our eCommerce implementations we
> do not cache Order related data. We focus on the heavy usage / read-only
> aspects (Product catalog and related).
>
> It may take some experimentation to really get to the solution. Interested
> to hear about your experience.
>
> Nick
>
> -----Original Message-----
> From: Jacques Le Roux [mailto:[hidden email]]
> Sent: Wednesday, June 29, 2016 3:09 AM
> To: [hidden email]
> Subject: Re: Aw: Re: Memcached / central caching
>
> Ah forgot, Moqui is using http://www.ehcache.org/
>
>
> Le 29/06/2016 à 08:55, Jacques Le Roux a écrit :
> > Thanks Leonard,
> >
> > So it could be indeed that DCC does not support this case well. It
> > would be interesting to know why. Of course using something like
> memcached is certainly a good way and a contribution would be quite welcome!
> >
> > Jacques
> >
> >
> > Le 29/06/2016 à 08:33, Leonard Lin a écrit :
> >> Thanks for the input.
> >> I have to try to look at logs to see which entities/processes are the
> high cache clear flow.
> >>   We are not using our ofbiz as an ecommerce system but as the
> manufacturing/fulfillment system.
> >> We do process a lot of orders and have also custom modules.
> >>
> >> Gesendet: Montag, 27. Juni 2016 um 15:46 Uhr
> >> Von: "Jacques Le Roux" <[hidden email]>
> >> An: [hidden email]
> >> Betreff: Re: Memcached / central caching I can confirm this behaviour
> >> (working well and better with more instances to support the growing
> >> load) for a configuration of 5 OFBiz machines against a cluster of 3
> Postgres machines (using PgPool II).
> >>
> >> It was also for an ecommerce application for a major Internet
> >> provider. I then I wrote the wiki page about DCC. As say Nick, you may
> indeed have a specific issue we did not cross...
> >>
> >> Jacques
> >>
> >>
> >> Le 27/06/2016 à 15:11, Nick Rosser a écrit :
> >>> Leonard,
> >>>
> >>> We have clients using distributed cache -- we used ActiveMQ to
> >>> synchronize any cache modifications from one server to the other(s).
> Worked very well -- and we saw a significant performance uptick when adding
> additional "slave" instances.
> >>>
> >>> Our implementations are typically for eCommerce -- with minimal
> cache-clear events.
> >>>
> >>> Which entities are subject to high volume cache-clear events for you?
> Perhaps you could consider removing caching for those entities?
> >>>
> >>> Nick
> >>>
> >>> -----Original Message-----
> >>> From: Leonard Lin [mailto:[hidden email]]
> >>> Sent: Monday, June 27, 2016 6:24 AM
> >>> To: [hidden email]
> >>> Subject: Memcached / central caching
> >>>
> >>> Hi,
> >>>
> >>> I would like to ask if someone has already investigated or
> >>> implemented an adapter for OFBiz to use a central caching system like
> memcached, redis or the like.
> >>>
> >>> I know for OFBiz cluster setup there is the distributed cache clear
> mechanism.
> >>>
> >>> I did use that in the past but have found that with more than 2
> instances it impacts performance of OFBiz adversely.
> >>> Symptoms we started to see in production:
> >>> - We started to see long running transaction locks, General Entity
> >>> operations slower (might be a combination of load and clustering)
> >>> - Other idle hosts were 5%-15% busy only to process the cache-clear
> >>> events without doing anything else
> >>> - Especially with higher load, the whole thing becomes even slower
> >>>
> >>> More load -> More cluster instances -> more cache clear events ->
> >>> more cycles used to process cache mgmt. -> general
> >>> requests/operations slower -> so we need even more instances -> you
> see where it’s going That’s why I was thinking if someone has tried to
> integrate with a central cache.
> >>>
> >>> For single host installation the local in-memory cache is surely
> >>> faster, even perhaps with a 2host cluster setup, but with 3 or more
> hosts, I think a central cache might scale better.
> >>>
> >>> Also interested to hear about any thoughts on the idea and
> architecture.
> >>>
> >>> Otherwise, any pointers where in the code to look at would be
> appreciated.
> >>> I'm thinking to implement it as an adaptor/driver you can switch on if
> you want it.
> >>>
> >>> Thanks
> >>> Best
> >>> Leo
> >>>
> >>>
> >>
> >>
> >
>
>

[Nick Rosser-2]

3rd party -- if considered "best of breed" is always preferable, imo. It's the only thing they do so they normally become experts at it. Don't have an opinion as to which one would be considered the best-of-breed.

Nick


-----Original Message-----
From: Pierre Smits [mailto:[hidden email]]
Sent: Wednesday, June 29, 2016 11:10 AM
To: [hidden email]
Subject: Re: Aw: Re: Memcached / central caching

I wonder whether a 3rd party solution (e.g. memcached, ehcache, etc) would be an improvement over current caching implementation.

What is your take on this? What would be the pro's and con's?

Best regards,

Pierre Smits

ORRTIZ.COM <http://www.orrtiz.com>
OFBiz based solutions & services

OFBiz Extensions Marketplace
http://oem.ofbizci.net/oci-2/

On Wed, Jun 29, 2016 at 4:58 PM, Nick Rosser <[hidden email]> wrote:

> It's certainly possible that with a system that has a high volume of
> updates, on certain entities, that caching will degrade performance on
> a single-server or clustered environment.
>
> I would certainly suggest that for constantly changing that caching be
> turned off, or have a quick expiration.
>
> For you that could be Orders and related entities as they move through
> the manufacturing / fulfillment process? I'm sure you probably have a
> good sense as to the system usage, where high-volume updates take
> place, and how that impacts specific DB entities. Even in our
> eCommerce implementations we do not cache Order related data. We focus
> on the heavy usage / read-only aspects (Product catalog and related).
>
> It may take some experimentation to really get to the solution.
> Interested to hear about your experience.
>
> Nick
>
> -----Original Message-----
> From: Jacques Le Roux [mailto:[hidden email]]
> Sent: Wednesday, June 29, 2016 3:09 AM
> To: [hidden email]
> Subject: Re: Aw: Re: Memcached / central caching
>
> Ah forgot, Moqui is using http://www.ehcache.org/
>
>
> Le 29/06/2016 à 08:55, Jacques Le Roux a écrit :
> > Thanks Leonard,
> >
> > So it could be indeed that DCC does not support this case well. It
> > would be interesting to know why. Of course using something like
> memcached is certainly a good way and a contribution would be quite welcome!
> >
> > Jacques
> >
> >
> > Le 29/06/2016 à 08:33, Leonard Lin a écrit :
> >> Thanks for the input.
> >> I have to try to look at logs to see which entities/processes are
> >> the
> high cache clear flow.
> >>   We are not using our ofbiz as an ecommerce system but as the
> manufacturing/fulfillment system.
> >> We do process a lot of orders and have also custom modules.
> >>
> >> Gesendet: Montag, 27. Juni 2016 um 15:46 Uhr
> >> Von: "Jacques Le Roux" <[hidden email]>
> >> An: [hidden email]
> >> Betreff: Re: Memcached / central caching I can confirm this
> >> behaviour (working well and better with more instances to support
> >> the growing
> >> load) for a configuration of 5 OFBiz machines against a cluster of
> >> 3
> Postgres machines (using PgPool II).
> >>
> >> It was also for an ecommerce application for a major Internet
> >> provider. I then I wrote the wiki page about DCC. As say Nick, you
> >> may
> indeed have a specific issue we did not cross...
> >>
> >> Jacques
> >>
> >>
> >> Le 27/06/2016 à 15:11, Nick Rosser a écrit :
> >>> Leonard,
> >>>
> >>> We have clients using distributed cache -- we used ActiveMQ to
> >>> synchronize any cache modifications from one server to the other(s).
> Worked very well -- and we saw a significant performance uptick when
> adding additional "slave" instances.
> >>>
> >>> Our implementations are typically for eCommerce -- with minimal
> cache-clear events.
> >>>
> >>> Which entities are subject to high volume cache-clear events for you?
> Perhaps you could consider removing caching for those entities?
> >>>
> >>> Nick
> >>>
> >>> -----Original Message-----
> >>> From: Leonard Lin [mailto:[hidden email]]
> >>> Sent: Monday, June 27, 2016 6:24 AM
> >>> To: [hidden email]
> >>> Subject: Memcached / central caching
> >>>
> >>> Hi,
> >>>
> >>> I would like to ask if someone has already investigated or
> >>> implemented an adapter for OFBiz to use a central caching system
> >>> like
> memcached, redis or the like.
> >>>
> >>> I know for OFBiz cluster setup there is the distributed cache
> >>> clear
> mechanism.
> >>>
> >>> I did use that in the past but have found that with more than 2
> instances it impacts performance of OFBiz adversely.
> >>> Symptoms we started to see in production:
> >>> - We started to see long running transaction locks, General Entity
> >>> operations slower (might be a combination of load and clustering)
> >>> - Other idle hosts were 5%-15% busy only to process the
> >>> cache-clear events without doing anything else
> >>> - Especially with higher load, the whole thing becomes even slower
> >>>
> >>> More load -> More cluster instances -> more cache clear events ->
> >>> more cycles used to process cache mgmt. -> general
> >>> requests/operations slower -> so we need even more instances ->
> >>> you
> see where it’s going That’s why I was thinking if someone has tried to
> integrate with a central cache.
> >>>
> >>> For single host installation the local in-memory cache is surely
> >>> faster, even perhaps with a 2host cluster setup, but with 3 or
> >>> more
> hosts, I think a central cache might scale better.
> >>>
> >>> Also interested to hear about any thoughts on the idea and
> architecture.
> >>>
> >>> Otherwise, any pointers where in the code to look at would be
> appreciated.
> >>> I'm thinking to implement it as an adaptor/driver you can switch
> >>> on if
> you want it.
> >>>
> >>> Thanks
> >>> Best
> >>> Leo
> >>>
> >>>
> >>
> >>
> >
>
>

[David E. Jones-2]

The entity value caches in Moqui and OFBiz are similar in concept, and I’ve looked into using a distributed cache for them but it is not a good fit.

The reason is that these are not simple key/value caches where invalidation is a simple matter (for that you would have to restrict to ONLY single record by primary key). Support for caching single records by non primary key fields makes them more list a list (multiple) result cache.

To invalidate when creating a new record, and even in certain other scenarios, you can’t even use a simple reverse-association map to know which cached lists to clear. For view entities where a record for any entity in the view/join could be updated, this is even more complex.

Part of the reason why these don’t work with distributed caches is that we need additional information, like the reverse-association map, to be maintained along with the main cache and if values are lost there needed invalidations may not happen and errors occur (stale cache data).

Distributed caches involve HUGE overhead compared to local only caches for serialization and deserialization. Near caches can help with this but add their own complexities and on top of the cache invalidation complexities for non-PK entity value (or value list) caches I don’t think these would conceptually work, or even if so would be very difficult to get working correctly.

The main solution is to use the most efficient distributed cache clear possible. There are two parts to this:

1. avoid distributed cache clear messages by marking entities as never cached (and in the Entity Engine also look at the never cache flag on an entity and just don’t do a cached query, maybe log a warning or info message but more importantly don’t use the entity cache); for example when an OrderItem is created or updated (which should never be cached) there is no reason to send a cache invalidation message… I don’t know if this is still an issue with OFBiz, but it was 3 years ago in the last big OFBiz-based project I worked on (it was bad enough that we just turned off entity cache altogether and used a MySQL box with a bunch of memory and had it do all the caching… which, BTW, isn’t a bad approach architecturally… it puts more load on the DB server, but it performs very well compared to distributed caching attempts we tried)

2. use a distributed topic for the cache clear messages; ActiveMQ is a good step in the right direction, but more modern distributed data and computing tools are better (Moqui supports Hazelcast for this; Apache Ignite and JBoss Infinispan are other options); Hazelcast is also great for web session replication, distributing background/async service calls over a cluster, etc… and doing all those things with a single cluster configuration with auto-discovery of nodes and such which makes deployment a LOT easier!

Moqui actually doesn’t use Ehcache any more because it is actually pretty slow compared to alternatives and the distributed caching is a nightmare (requires commercial Terracotta server, on top of the algorithmic issues with a distributed cache mentioned above).

Right now Moqui does all caching through the JCache (javax.cache, JSR-107) interfaces, and has its own implementation of a local-only, in-memory cache that is used for configuration (entities, services, screens, compiled templates and scripts, etc) and by default for the entity value/list caches as well. The Moqui JCache implementation is basically a wrapper around a ConcurrentHashMap with support for expiration and a separate thread that monitors cache sizes and if it is larger than the max settings removes old entries in the background.

The only JCache implementation I’ve run into that might compare in performance, and has various nice features and a really fancy size limit clearing approach, is Caffeine cache from Ben Manes (see https://github.com/ben-manes/caffeine). I haven’t tried this yet or done extensive tests, partly because I didn’t find it until after already implementing the JCache interfaces in Moqui’s MCache, but of all the options for a fast local cache it looks like the best.

BTW, a lot of this stuff is very new in Moqui and is part of a bigger effort to handle some pretty significant load and scale, ie getting to something like 50k active users on a small-ish cluster of servers, and placing/shipping 500k orders per hour with a dozen screen renders in the load test scenario, including an inventory report for sales people with a grid of typically around 200 highly active products (overall projection in the table with a hover popup for more detailed information). This even includes real time GL posting so the books are continuously up to date. To do this required some tweaking of how inventory is handled with record level locks in the database and such (on reservation and on issuance).

That inventory report is interesting too: it uses ElasticSearch aggregations instead of a relational database and with millions of inventory, reservation, incoming shipment, active production run, and unplaced order (no inventory reservation yet) records the performance difference is incredible… hundreds of times faster changing it from uselessly slow to smooth running. Deployment is easy too because every server can use the same configuration with the Hazelcast cluster, and no need for a separate MQ server that all servers know about (and yes, Hazelcast and other similar tools like Ignite and Infinispan have auto discovery plugins for more limited hosting options like AWS).

Anyway, scalability is a fairly complex topic but there are some great open source tools these days for getting there, even for transactional systems where scalability is FAR more difficult than with non-transctional that involve no record locking or sensitive incremental updates… hence the target of 50k active transactional users (could handle many more ecommerce users who browse a lot more than place orders, and browsing products is a MUCH easier to scale!) as opposed to 50 million or something… not that OFBiz or Moqui are likely to ever be used for something that big, ie Facebook or Google scale; even big ecommerce like Amazon is not likely though Amazon is transactional for various things and I imagine they have to get pretty fancy with things like inventory reservations and issuance on shipping (likely including partitioning inventory data across multiple data stores with some sort of verify and increment datastore for immediate/deterministic consistency).

-David

> On 29 Jun 2016, at 00:08, Jacques Le Roux <[hidden email]> wrote:
>
> Ah forgot, Moqui is using http://www.ehcache.org/
>
>
> Le 29/06/2016 à 08:55, Jacques Le Roux a écrit :
>> Thanks Leonard,
>>
>> So it could be indeed that DCC does not support this case well. It would be interesting to know why. Of course using something like memcached is certainly a good way and a contribution would be quite welcome!
>>
>> Jacques
>>
>>
>> Le 29/06/2016 à 08:33, Leonard Lin a écrit :
>>> Thanks for the input.
>>> I have to try to look at logs to see which entities/processes are the high cache clear flow.
>>>  We are not using our ofbiz as an ecommerce system but as the manufacturing/fulfillment system.
>>> We do process a lot of orders and have also custom modules.
>>>
>>> Gesendet: Montag, 27. Juni 2016 um 15:46 Uhr
>>> Von: "Jacques Le Roux" <[hidden email]>
>>> An: [hidden email]
>>> Betreff: Re: Memcached / central caching
>>> I can confirm this behaviour (working well and better with more instances to support the growing load) for a configuration of 5 OFBiz machines against
>>> a cluster of 3 Postgres machines (using PgPool II).
>>>
>>> It was also for an ecommerce application for a major Internet provider. I then I wrote the wiki page about DCC. As say Nick, you may indeed have a
>>> specific issue we did not cross...
>>>
>>> Jacques
>>>
>>>
>>> Le 27/06/2016 à 15:11, Nick Rosser a écrit :
>>>> Leonard,
>>>>
>>>> We have clients using distributed cache -- we used ActiveMQ to synchronize any cache modifications from one server to the other(s). Worked very well -- and we saw a significant performance uptick when adding additional "slave" instances.
>>>>
>>>> Our implementations are typically for eCommerce -- with minimal cache-clear events.
>>>>
>>>> Which entities are subject to high volume cache-clear events for you? Perhaps you could consider removing caching for those entities?
>>>>
>>>> Nick
>>>>
>>>> -----Original Message-----
>>>> From: Leonard Lin [mailto:[hidden email]]
>>>> Sent: Monday, June 27, 2016 6:24 AM
>>>> To: [hidden email]
>>>> Subject: Memcached / central caching
>>>>
>>>> Hi,
>>>>
>>>> I would like to ask if someone has already investigated or implemented an adapter for OFBiz to use a central caching system like memcached, redis or the like.
>>>>
>>>> I know for OFBiz cluster setup there is the distributed cache clear mechanism.
>>>>
>>>> I did use that in the past but have found that with more than 2 instances it impacts performance of OFBiz adversely.
>>>> Symptoms we started to see in production:
>>>> - We started to see long running transaction locks, General Entity operations slower (might be a combination of load and clustering)
>>>> - Other idle hosts were 5%-15% busy only to process the cache-clear events without doing anything else
>>>> - Especially with higher load, the whole thing becomes even slower
>>>>
>>>> More load -> More cluster instances -> more cache clear events -> more cycles used to process cache mgmt. -> general requests/operations slower -> so we need even more instances -> you see where it’s going
>>>> That’s why I was thinking if someone has tried to integrate with a central cache.
>>>>
>>>> For single host installation the local in-memory cache is surely faster, even perhaps with a 2host cluster setup, but with 3 or more hosts, I think a central cache might scale better.
>>>>
>>>> Also interested to hear about any thoughts on the idea and architecture.
>>>>
>>>> Otherwise, any pointers where in the code to look at would be appreciated.
>>>> I'm thinking to implement it as an adaptor/driver you can switch on if you want it.
>>>>
>>>> Thanks
>>>> Best
>>>> Leo
>>>>
>>>>
>>>
>>>
>>
>