Low utilization of OFBiz on multi core servers

[Wang, Weikun]

Hi all,

I'm using OFBiz to study the performance of web service in the cloud.
I found a problem when I deploy OFBiz on a 2 core server, which has
4G memory. The OFBiz I used is the 12.04.03 version. The OS is ubuntu
12.04.

I was using OFBench to send requests to OFBiz. The OFBench is able
to send different kinds of requests, like login, logout, visit main page.
I was testing how many users I set in OFBench will saturate the server.
At some point I found that the server was overloaded when I set 20
users because the mean response time for the "main" request is over
10 seconds. However, the CPU utilization is only 60%. I also did a test
for one core server. When it's overloaded, the utilization is almost 100%.

I was using the default derby database. I monitor the memory of the
2 core server, it was like 2GB used. So I guess memory is not a problem.
The other parameters are default. This is not the first time I notice this
problem. I also found this on other multi-core servers.

Could anyone help me on this please?

Thanks a lot in advance!!!!

Best wishes,
Weikun

[Adrian Crum-3]

Your are running your tests on a development deployment. Try setting up
your deployment for production:

https://cwiki.apache.org/confluence/display/OFBTECH/Apache+OFBiz+Technical+Production+Setup+Guide

Adrian Crum
Sandglass Software
www.sandglass-software.com

On 8/13/2014 1:12 PM, Wang, Weikun wrote:

> Hi all,
>
> I'm using OFBiz to study the performance of web service in the cloud.
> I found a problem when I deploy OFBiz on a 2 core server, which has
> 4G memory. The OFBiz I used is the 12.04.03 version. The OS is ubuntu
> 12.04.
>
> I was using OFBench to send requests to OFBiz. The OFBench is able
> to send different kinds of requests, like login, logout, visit main page.
> I was testing how many users I set in OFBench will saturate the server.
> At some point I found that the server was overloaded when I set 20
> users because the mean response time for the "main" request is over
> 10 seconds. However, the CPU utilization is only 60%. I also did a test
> for one core server. When it's overloaded, the utilization is almost 100%.
>
> I was using the default derby database. I monitor the memory of the
> 2 core server, it was like 2GB used. So I guess memory is not a problem.
> The other parameters are default. This is not the first time I notice this
> problem. I also found this on other multi-core servers.
>
> Could anyone help me on this please?
>
> Thanks a lot in advance!!!!
>
> Best wishes,
> Weikun
>

[Wang, Weikun]

Hi Adrian,

Thanks for your quick reply. It seems that this didn't resolve the problem.
I also notice that once it gets overloaded the utilization drops from 80%
to 60%. This is what puzzles me because I thought when it gets overloaded,
the utilization should increase.

While I was testing an 8 core server, the overloaded behaviour happens
when the utilization is only 40%, which is quite low.

Best wishes,
Weikun

________________________________________
From: Adrian Crum [[hidden email]]
Sent: 13 August 2014 14:04
To: [hidden email]
Subject: Re: Low utilization of OFBiz on multi core servers

Your are running your tests on a development deployment. Try setting up
your deployment for production:

https://cwiki.apache.org/confluence/display/OFBTECH/Apache+OFBiz+Technical+Production+Setup+Guide

Adrian Crum
Sandglass Software
www.sandglass-software.com

On 8/13/2014 1:12 PM, Wang, Weikun wrote:

> Hi all,
>
> I'm using OFBiz to study the performance of web service in the cloud.
> I found a problem when I deploy OFBiz on a 2 core server, which has
> 4G memory. The OFBiz I used is the 12.04.03 version. The OS is ubuntu
> 12.04.
>
> I was using OFBench to send requests to OFBiz. The OFBench is able
> to send different kinds of requests, like login, logout, visit main page.
> I was testing how many users I set in OFBench will saturate the server.
> At some point I found that the server was overloaded when I set 20
> users because the mean response time for the "main" request is over
> 10 seconds. However, the CPU utilization is only 60%. I also did a test
> for one core server. When it's overloaded, the utilization is almost 100%.
>
> I was using the default derby database. I monitor the memory of the
> 2 core server, it was like 2GB used. So I guess memory is not a problem.
> The other parameters are default. This is not the first time I notice this
> problem. I also found this on other multi-core servers.
>
> Could anyone help me on this please?
>
> Thanks a lot in advance!!!!
>
> Best wishes,
> Weikun
>

[damon henry]

Hi Weikun,
I have no practical experience with OFBiz, other than lurking on this list for a few months and playing with the demo's.  I do have lots of experience supporting J2EE production applications.  I'm not sure what you are referring to as an overload condition, but it seems that you are watching CPU utilization on your server which appears fine, but some other indicator is telling you that OFBiz is overloaded, perhaps just running very slow???  Driving any J2EE application is a java virtual machine.  Are you monitoring the jvm in any way?  It is possible for the jvm to run out of resources while the physical platform still has plenty available.  For instance you may not have assigned the jvm enough physical memory so it gets stuck swapping objects to a swap file or executing excessive garbage collections.  Other common choke points include database calls and thread pools.  Do you have a way to monitor these?  Of course, it could be poor code, but if that were the case there would likely be lots of people complaining about the same issue.  I would take a close look at the jvm environment you are spinning up.  In my experience adding more memory for the jvm to utilize is the most likely way to fix performance related problems.  You can control this with the -Xms -Xmx -MaxPermSize parameters being passed during startup.  You can use the jstat utility to watch what the jvm memory is doing realtime.
Damon

> From: [hidden email]
> To: [hidden email]
> Subject: RE: Low utilization of OFBiz on multi core servers
> Date: Wed, 13 Aug 2014 15:47:59 +0000
>
> Hi Adrian,
>
> Thanks for your quick reply. It seems that this didn't resolve the problem.
> I also notice that once it gets overloaded the utilization drops from 80%
> to 60%. This is what puzzles me because I thought when it gets overloaded,
> the utilization should increase.
>
> While I was testing an 8 core server, the overloaded behaviour happens
> when the utilization is only 40%, which is quite low.
>
> Best wishes,
> Weikun
>
> ________________________________________
> From: Adrian Crum [[hidden email]]
> Sent: 13 August 2014 14:04
> To: [hidden email]
> Subject: Re: Low utilization of OFBiz on multi core servers
>
> Your are running your tests on a development deployment. Try setting up
> your deployment for production:
>
> https://cwiki.apache.org/confluence/display/OFBTECH/Apache+OFBiz+Technical+Production+Setup+Guide
>
> Adrian Crum
> Sandglass Software
> www.sandglass-software.com
>
> On 8/13/2014 1:12 PM, Wang, Weikun wrote:
> > Hi all,
> >
> > I'm using OFBiz to study the performance of web service in the cloud.
> > I found a problem when I deploy OFBiz on a 2 core server, which has
> > 4G memory. The OFBiz I used is the 12.04.03 version. The OS is ubuntu
> > 12.04.
> >
> > I was using OFBench to send requests to OFBiz. The OFBench is able
> > to send different kinds of requests, like login, logout, visit main page.
> > I was testing how many users I set in OFBench will saturate the server.
> > At some point I found that the server was overloaded when I set 20
> > users because the mean response time for the "main" request is over
> > 10 seconds. However, the CPU utilization is only 60%. I also did a test
> > for one core server. When it's overloaded, the utilization is almost 100%.
> >
> > I was using the default derby database. I monitor the memory of the
> > 2 core server, it was like 2GB used. So I guess memory is not a problem.
> > The other parameters are default. This is not the first time I notice this
> > problem. I also found this on other multi-core servers.
> >
> > Could anyone help me on this please?
> >
> > Thanks a lot in advance!!!!
> >
> > Best wishes,
> > Weikun
> >

     

I agree that looking at just CPU is not a good guide.
IO is usually the issue before cpu. If it was bad code the cpu might go 100% before the IO.
But bus, memory etc all play a part.
Did you switch to Postgres? Not sure if Derby is the best for load testing.
I know with postgres each time a request is used it will use a new thread and can make use of multiple CPU.
I am not sure if the JVM uses multiple core, but I imagine it has some form of using multiple threads.

Any how I appreciate your sharing you found 20 concurrent users seems to be a max.

Not 100% sure, but I know I had to support 250 at my last job, but I used many techniques.
For one I had three web servers. My Database server was very high end with fiber SAN. I used different app servers for different parts of my app. I was using liferay as my portal engine, and I even had a separate server or some of our larger clients, while smaller clients used a shared instance.
I guess my point is there are many ways to get an application to handle the load.

[Nick Rosser-2]

FWIW -- we went through a lot of performance pains implementing some
high-volume eComm apps. Ended up consulting directly with David Jones
for pointers / advise / strategy etc.

As a result we put together this doc, may be of some help / interest:

http://bigfish.solveda.com/help/guideCachingAndMemory.htm

First section is all about the caching aspects of OFBiz, all the good DB
caching stuff in the framework.

Scroll down to the "memory" section of the doc ... for a discussion
about JVM settings etc.

Nick

On 8/13/2014 12:53 PM, [hidden email] wrote:

> I agree that looking at just CPU is not a good guide.
> IO is usually the issue before cpu. If it was bad code the cpu might go 100%
> before the IO.
> But bus, memory etc all play a part.
> Did you switch to Postgres? Not sure if Derby is the best for load testing.
> I know with postgres each time a request is used it will use a new thread
> and can make use of multiple CPU.
> I am not sure if the JVM uses multiple core, but I imagine it has some form
> of using multiple threads.
>
> Any how I appreciate your sharing you found 20 concurrent users seems to be
> a max.
>
> Not 100% sure, but I know I had to support 250 at my last job, but I used
> many techniques.
> For one I had three web servers. My Database server was very high end with
> fiber SAN. I used different app servers for different parts of my app. I was
> using liferay as my portal engine, and I even had a separate server or some
> of our larger clients, while smaller clients used a shared instance.
> I guess my point is there are many ways to get an application to handle the
> load.
>
>
>
> -----
> Joel Fradkin
> --
> View this message in context: http://ofbiz.135035.n4.nabble.com/Low-utilization-of-OFBiz-on-multi-core-servers-tp4653440p4653445.html
> Sent from the OFBiz - User mailing list archive at Nabble.com.

[Wang, Weikun]

Hi Damon,

Thanks a lot for the answer! I have tested with setting the jvm
parameters, it works!! This is great!

Best wishes,
Weikun

________________________________________
From: damon henry [[hidden email]]
Sent: 13 August 2014 17:14
To: [hidden email]
Subject: RE: Low utilization of OFBiz on multi core servers

Hi Weikun,
I have no practical experience with OFBiz, other than lurking on this list for a few months and playing with the demo's.  I do have lots of experience supporting J2EE production applications.  I'm not sure what you are referring to as an overload condition, but it seems that you are watching CPU utilization on your server which appears fine, but some other indicator is telling you that OFBiz is overloaded, perhaps just running very slow???  Driving any J2EE application is a java virtual machine.  Are you monitoring the jvm in any way?  It is possible for the jvm to run out of resources while the physical platform still has plenty available.  For instance you may not have assigned the jvm enough physical memory so it gets stuck swapping objects to a swap file or executing excessive garbage collections.  Other common choke points include database calls and thread pools.  Do you have a way to monitor these?  Of course, it could be poor code, but if that were the case there would likely be lots of people complaining about the same issue.  I would take a close look at the jvm environment you are spinning up.  In my experience adding more memory for the jvm to utilize is the most likely way to fix performance related problems.  You can control this with the -Xms -Xmx -MaxPermSize parameters being passed during startup.  You can use the jstat utility to watch what the jvm memory is doing realtime.
Damon

> From: [hidden email]
> To: [hidden email]
> Subject: RE: Low utilization of OFBiz on multi core servers
> Date: Wed, 13 Aug 2014 15:47:59 +0000
>
> Hi Adrian,
>
> Thanks for your quick reply. It seems that this didn't resolve the problem.
> I also notice that once it gets overloaded the utilization drops from 80%
> to 60%. This is what puzzles me because I thought when it gets overloaded,
> the utilization should increase.
>
> While I was testing an 8 core server, the overloaded behaviour happens
> when the utilization is only 40%, which is quite low.
>
> Best wishes,
> Weikun
>
> ________________________________________
> From: Adrian Crum [[hidden email]]
> Sent: 13 August 2014 14:04
> To: [hidden email]
> Subject: Re: Low utilization of OFBiz on multi core servers
>
> Your are running your tests on a development deployment. Try setting up
> your deployment for production:
>
> https://cwiki.apache.org/confluence/display/OFBTECH/Apache+OFBiz+Technical+Production+Setup+Guide
>
> Adrian Crum
> Sandglass Software
> www.sandglass-software.com
>
> On 8/13/2014 1:12 PM, Wang, Weikun wrote:
> > Hi all,
> >
> > I'm using OFBiz to study the performance of web service in the cloud.
> > I found a problem when I deploy OFBiz on a 2 core server, which has
> > 4G memory. The OFBiz I used is the 12.04.03 version. The OS is ubuntu
> > 12.04.
> >
> > I was using OFBench to send requests to OFBiz. The OFBench is able
> > to send different kinds of requests, like login, logout, visit main page.
> > I was testing how many users I set in OFBench will saturate the server.
> > At some point I found that the server was overloaded when I set 20
> > users because the mean response time for the "main" request is over
> > 10 seconds. However, the CPU utilization is only 60%. I also did a test
> > for one core server. When it's overloaded, the utilization is almost 100%.
> >
> > I was using the default derby database. I monitor the memory of the
> > 2 core server, it was like 2GB used. So I guess memory is not a problem.
> > The other parameters are default. This is not the first time I notice this
> > problem. I also found this on other multi-core servers.
> >
> > Could anyone help me on this please?
> >
> > Thanks a lot in advance!!!!
> >
> > Best wishes,
> > Weikun
> >

[Wang, Weikun]

Hi Nick,

The link is great! I'm setting the Xms and Xmx, it does improve the performance!
I was looking at the response time. For a 1 core server with 80% utilization, the
response time for the main page changes from 0.87s to 0.79s! Now everything
looks perfect. Thanks a lot!

Best wishes,
Weikun

________________________________________
From: Nick Rosser [[hidden email]]
Sent: 13 August 2014 19:22
To: [hidden email]
Subject: Re: Low utilization of OFBiz on multi core servers

FWIW -- we went through a lot of performance pains implementing some
high-volume eComm apps. Ended up consulting directly with David Jones
for pointers / advise / strategy etc.

As a result we put together this doc, may be of some help / interest:

http://bigfish.solveda.com/help/guideCachingAndMemory.htm

First section is all about the caching aspects of OFBiz, all the good DB
caching stuff in the framework.

Scroll down to the "memory" section of the doc ... for a discussion
about JVM settings etc.

Nick

On 8/13/2014 12:53 PM, [hidden email] wrote:

> I agree that looking at just CPU is not a good guide.
> IO is usually the issue before cpu. If it was bad code the cpu might go 100%
> before the IO.
> But bus, memory etc all play a part.
> Did you switch to Postgres? Not sure if Derby is the best for load testing.
> I know with postgres each time a request is used it will use a new thread
> and can make use of multiple CPU.
> I am not sure if the JVM uses multiple core, but I imagine it has some form
> of using multiple threads.
>
> Any how I appreciate your sharing you found 20 concurrent users seems to be
> a max.
>
> Not 100% sure, but I know I had to support 250 at my last job, but I used
> many techniques.
> For one I had three web servers. My Database server was very high end with
> fiber SAN. I used different app servers for different parts of my app. I was
> using liferay as my portal engine, and I even had a separate server or some
> of our larger clients, while smaller clients used a shared instance.
> I guess my point is there are many ways to get an application to handle the
> load.
>
>
>
> -----
> Joel Fradkin
> --
> View this message in context: http://ofbiz.135035.n4.nabble.com/Low-utilization-of-OFBiz-on-multi-core-servers-tp4653440p4653445.html
> Sent from the OFBiz - User mailing list archive at Nabble.com.

[Jacques Le Roux]

Le 13/08/2014 20:22, Nick Rosser a écrit :

> FWIW -- we went through a lot of performance pains implementing some high-volume eComm apps. Ended up consulting directly with David Jones for
> pointers / advise / strategy etc.
>
> As a result we put together this doc, may be of some help / interest:
>
> http://bigfish.solveda.com/help/guideCachingAndMemory.htm
>
> First section is all about the caching aspects of OFBiz, all the good DB caching stuff in the framework.
>
> Scroll down to the "memory" section of the doc ... for a discussion about JVM settings etc.

Interesting, thanks Nick.

I noticed there is a known bug
<<TECH NOTE: The EXP and SR misses do not currently work on OFBiz due to a bug introduced a few years ago. All misses are logged as NF (Not Found).>>

Do you have more information about that, is it only on the UI side? Is it only information missing there, or? Is there a Jira issue open, etc?

Jacques

>
> Nick
>
> On 8/13/2014 12:53 PM, [hidden email] wrote:
>> I agree that looking at just CPU is not a good guide.
>> IO is usually the issue before cpu. If it was bad code the cpu might go 100%
>> before the IO.
>> But bus, memory etc all play a part.
>> Did you switch to Postgres? Not sure if Derby is the best for load testing.
>> I know with postgres each time a request is used it will use a new thread
>> and can make use of multiple CPU.
>> I am not sure if the JVM uses multiple core, but I imagine it has some form
>> of using multiple threads.
>>
>> Any how I appreciate your sharing you found 20 concurrent users seems to be
>> a max.
>>
>> Not 100% sure, but I know I had to support 250 at my last job, but I used
>> many techniques.
>> For one I had three web servers. My Database server was very high end with
>> fiber SAN. I used different app servers for different parts of my app. I was
>> using liferay as my portal engine, and I even had a separate server or some
>> of our larger clients, while smaller clients used a shared instance.
>> I guess my point is there are many ways to get an application to handle the
>> load.
>>
>>
>>
>> -----
>> Joel Fradkin
>> --
>> View this message in context: http://ofbiz.135035.n4.nabble.com/Low-utilization-of-OFBiz-on-multi-core-servers-tp4653440p4653445.html
>> Sent from the OFBiz - User mailing list archive at Nabble.com.
>

[Nick Rosser-2]

<<TECH NOTE: The EXP and SR misses do not currently work on OFBiz due to
a bug introduced a few years ago. All misses are logged as NF (Not
Found).>>

I have to think that this was direct feedback from David Jones. Perhaps
it is resolved now, not sure to be honest.

On 9/4/2014 8:22 AM, Jacques Le Roux wrote:

>
> Le 13/08/2014 20:22, Nick Rosser a écrit :
>> FWIW -- we went through a lot of performance pains implementing some
>> high-volume eComm apps. Ended up consulting directly with David Jones
>> for pointers / advise / strategy etc.
>>
>> As a result we put together this doc, may be of some help / interest:
>>
>> http://bigfish.solveda.com/help/guideCachingAndMemory.htm
>>
>> First section is all about the caching aspects of OFBiz, all the good
>> DB caching stuff in the framework.
>>
>> Scroll down to the "memory" section of the doc ... for a discussion
>> about JVM settings etc.
>
> Interesting, thanks Nick.
>
> I noticed there is a known bug
> <<TECH NOTE: The EXP and SR misses do not currently work on OFBiz due
> to a bug introduced a few years ago. All misses are logged as NF (Not
> Found).>>
>
> Do you have more information about that, is it only on the UI side? Is
> it only information missing there, or? Is there a Jira issue open, etc?
>
> Jacques
>
>>
>> Nick
>>
>> On 8/13/2014 12:53 PM, [hidden email] wrote:
>>> I agree that looking at just CPU is not a good guide.
>>> IO is usually the issue before cpu. If it was bad code the cpu might
>>> go 100%
>>> before the IO.
>>> But bus, memory etc all play a part.
>>> Did you switch to Postgres? Not sure if Derby is the best for load
>>> testing.
>>> I know with postgres each time a request is used it will use a new
>>> thread
>>> and can make use of multiple CPU.
>>> I am not sure if the JVM uses multiple core, but I imagine it has
>>> some form
>>> of using multiple threads.
>>>
>>> Any how I appreciate your sharing you found 20 concurrent users
>>> seems to be
>>> a max.
>>>
>>> Not 100% sure, but I know I had to support 250 at my last job, but I
>>> used
>>> many techniques.
>>> For one I had three web servers. My Database server was very high
>>> end with
>>> fiber SAN. I used different app servers for different parts of my
>>> app. I was
>>> using liferay as my portal engine, and I even had a separate server
>>> or some
>>> of our larger clients, while smaller clients used a shared instance.
>>> I guess my point is there are many ways to get an application to
>>> handle the
>>> load.
>>>
>>>
>>>
>>> -----
>>> Joel Fradkin
>>> --
>>> View this message in context:
>>> http://ofbiz.135035.n4.nabble.com/Low-utilization-of-OFBiz-on-multi-core-servers-tp4653440p4653445.html
>>> Sent from the OFBiz - User mailing list archive at Nabble.com.
>>

[Jacques Le Roux]

I believe it's only about the UI, so nothing really important, I did not check code though

I also noticed there is a small error in <<Back-End Database “Update”>> section
The modified ST_ID is 5 (from Yellow to Black Pants) not 6
The (stType = 2) is correct though

HTH

Jacques

Le 04/09/2014 15:17, Nick Rosser a écrit :

> <<TECH NOTE: The EXP and SR misses do not currently work on OFBiz due to a bug introduced a few years ago. All misses are logged as NF (Not Found).>>
>
> I have to think that this was direct feedback from David Jones. Perhaps it is resolved now, not sure to be honest.
>
> On 9/4/2014 8:22 AM, Jacques Le Roux wrote:
>>
>> Le 13/08/2014 20:22, Nick Rosser a écrit :
>>> FWIW -- we went through a lot of performance pains implementing some high-volume eComm apps. Ended up consulting directly with David Jones for
>>> pointers / advise / strategy etc.
>>>
>>> As a result we put together this doc, may be of some help / interest:
>>>
>>> http://bigfish.solveda.com/help/guideCachingAndMemory.htm
>>>
>>> First section is all about the caching aspects of OFBiz, all the good DB caching stuff in the framework.
>>>
>>> Scroll down to the "memory" section of the doc ... for a discussion about JVM settings etc.
>>
>> Interesting, thanks Nick.
>>
>> I noticed there is a known bug
>> <<TECH NOTE: The EXP and SR misses do not currently work on OFBiz due to a bug introduced a few years ago. All misses are logged as NF (Not Found).>>
>>
>> Do you have more information about that, is it only on the UI side? Is it only information missing there, or? Is there a Jira issue open, etc?
>>
>> Jacques
>>
>>>
>>> Nick
>>>
>>> On 8/13/2014 12:53 PM, [hidden email] wrote:
>>>> I agree that looking at just CPU is not a good guide.
>>>> IO is usually the issue before cpu. If it was bad code the cpu might go 100%
>>>> before the IO.
>>>> But bus, memory etc all play a part.
>>>> Did you switch to Postgres? Not sure if Derby is the best for load testing.
>>>> I know with postgres each time a request is used it will use a new thread
>>>> and can make use of multiple CPU.
>>>> I am not sure if the JVM uses multiple core, but I imagine it has some form
>>>> of using multiple threads.
>>>>
>>>> Any how I appreciate your sharing you found 20 concurrent users seems to be
>>>> a max.
>>>>
>>>> Not 100% sure, but I know I had to support 250 at my last job, but I used
>>>> many techniques.
>>>> For one I had three web servers. My Database server was very high end with
>>>> fiber SAN. I used different app servers for different parts of my app. I was
>>>> using liferay as my portal engine, and I even had a separate server or some
>>>> of our larger clients, while smaller clients used a shared instance.
>>>> I guess my point is there are many ways to get an application to handle the
>>>> load.
>>>>
>>>>
>>>>
>>>> -----
>>>> Joel Fradkin
>>>> --
>>>> View this message in context: http://ofbiz.135035.n4.nabble.com/Low-utilization-of-OFBiz-on-multi-core-servers-tp4653440p4653445.html
>>>> Sent from the OFBiz - User mailing list archive at Nabble.com.
>>>
>