Re: svn commit: r818321 - in /ofbiz/trunk: build.xml startofbiz.bat startofbiz.sh startofbizNoLog.bat

[Adam Heath-2]

[hidden email] wrote:
> Author: lektran
> Date: Thu Sep 24 00:05:05 2009
> New Revision: 818321
>
> URL: http://svn.apache.org/viewvc?rev=818321&view=rev
> Log:
> Added the maxPermSize jvm arg the various startup methods since OFBiz now seems to require more than the default 64m.  Also removed startofbizNoLog.bat since it seems somewhat superfluous, instead made startofbiz.bat log to the console by default and added a comment about how to change it if desired.

(nothing like delayed responses)

bother.  We had made a branch of ofbiz trunk Sep 5; we then used this
branch up until earlier this week.  This internal branch was able to
run fine with just a 64M permgen size.  However, once we jumped to the
latest trunk, we discovered that ofbiz suddenly used more memory, and
actually was throwing an OutOfMemory error.  This was due to not
having this particular change in the debian ofbiz init script.

Increasing the maxPermSize number is just not acceptable; having to
allocate so much more memory would increase the cost of hosting.

The discovery of this bug gave me a window of changes, to try and
track down the cause.  Near as I can tell, this memory issue is due to
 *all* service definitions having a custom, on-the-fly, class compiled
for them.  Each class is inside it's very own unique classloader.  So,
not only do you have all the internal memory for the class+classloader
combo(which includes a Vector with a default internal size of 10, but
only one element filled in the array), the class data itself is stored
in the permgen area.

Based on simple searching, I see 1926 class files(this number may not
be 100% accurate).  I then see 2595 service definitions.  Doing a jmap
-dump immediately after ofbiz starts, then running jhat on the
resulting dump file, shows tons and tons of classloaders loaded.

I'm working on a fix for this.  Since this bit of code is using
webslinger-invoker.jar, what I am doing in improving my test
coverage(I was at 80% initially, currently at 90%), then will import
the code in full into ofbiz(this was always the long-term plan with
the webslinger code anyways).  I'll then modify the invoker code to
support specifying a shared classloader, which should reduce the
memory usage.  I'll also be adding a way to turn off this compiled
class globally, per file, or per service.

[Tim Ruppert]

This is great Adam - thanks.

Cheers,
Ruppert

On Jan 26, 2010, at 11:49 AM, Adam Heath wrote:

> [hidden email] wrote:
>> Author: lektran
>> Date: Thu Sep 24 00:05:05 2009
>> New Revision: 818321
>>
>> URL: http://svn.apache.org/viewvc?rev=818321&view=rev
>> Log:
>> Added the maxPermSize jvm arg the various startup methods since OFBiz now seems to require more than the default 64m.  Also removed startofbizNoLog.bat since it seems somewhat superfluous, instead made startofbiz.bat log to the console by default and added a comment about how to change it if desired.
>
> (nothing like delayed responses)
>
> bother.  We had made a branch of ofbiz trunk Sep 5; we then used this
> branch up until earlier this week.  This internal branch was able to
> run fine with just a 64M permgen size.  However, once we jumped to the
> latest trunk, we discovered that ofbiz suddenly used more memory, and
> actually was throwing an OutOfMemory error.  This was due to not
> having this particular change in the debian ofbiz init script.
>
> Increasing the maxPermSize number is just not acceptable; having to
> allocate so much more memory would increase the cost of hosting.
>
> The discovery of this bug gave me a window of changes, to try and
> track down the cause.  Near as I can tell, this memory issue is due to
> *all* service definitions having a custom, on-the-fly, class compiled
> for them.  Each class is inside it's very own unique classloader.  So,
> not only do you have all the internal memory for the class+classloader
> combo(which includes a Vector with a default internal size of 10, but
> only one element filled in the array), the class data itself is stored
> in the permgen area.
>
> Based on simple searching, I see 1926 class files(this number may not
> be 100% accurate).  I then see 2595 service definitions.  Doing a jmap
> -dump immediately after ofbiz starts, then running jhat on the
> resulting dump file, shows tons and tons of classloaders loaded.
>
> I'm working on a fix for this.  Since this bit of code is using
> webslinger-invoker.jar, what I am doing in improving my test
> coverage(I was at 80% initially, currently at 90%), then will import
> the code in full into ofbiz(this was always the long-term plan with
> the webslinger code anyways).  I'll then modify the invoker code to
> support specifying a shared classloader, which should reduce the
> memory usage.  I'll also be adding a way to turn off this compiled
> class globally, per file, or per service.

[Adrian Crum]

Adam Heath wrote:

> [hidden email] wrote:
>> Author: lektran
>> Date: Thu Sep 24 00:05:05 2009
>> New Revision: 818321
>>
>> URL: http://svn.apache.org/viewvc?rev=818321&view=rev
>> Log:
>> Added the maxPermSize jvm arg the various startup methods since OFBiz now seems to require more than the default 64m.  Also removed startofbizNoLog.bat since it seems somewhat superfluous, instead made startofbiz.bat log to the console by default and added a comment about how to change it if desired.
>
> (nothing like delayed responses)
>
> bother.  We had made a branch of ofbiz trunk Sep 5; we then used this
> branch up until earlier this week.  This internal branch was able to
> run fine with just a 64M permgen size.  However, once we jumped to the
> latest trunk, we discovered that ofbiz suddenly used more memory, and
> actually was throwing an OutOfMemory error.  This was due to not
> having this particular change in the debian ofbiz init script.
>
> Increasing the maxPermSize number is just not acceptable; having to
> allocate so much more memory would increase the cost of hosting.
>
> The discovery of this bug gave me a window of changes, to try and
> track down the cause.  Near as I can tell, this memory issue is due to
>  *all* service definitions having a custom, on-the-fly, class compiled
> for them.  Each class is inside it's very own unique classloader.  So,
> not only do you have all the internal memory for the class+classloader
> combo(which includes a Vector with a default internal size of 10, but
> only one element filled in the array), the class data itself is stored
> in the permgen area.
>
> Based on simple searching, I see 1926 class files(this number may not
> be 100% accurate).  I then see 2595 service definitions.  Doing a jmap
> -dump immediately after ofbiz starts, then running jhat on the
> resulting dump file, shows tons and tons of classloaders loaded.
>
> I'm working on a fix for this.  Since this bit of code is using
> webslinger-invoker.jar, what I am doing in improving my test
> coverage(I was at 80% initially, currently at 90%), then will import
> the code in full into ofbiz(this was always the long-term plan with
> the webslinger code anyways).  I'll then modify the invoker code to
> support specifying a shared classloader, which should reduce the
> memory usage.  I'll also be adding a way to turn off this compiled
> class globally, per file, or per service.

Did you look into the CachedClassLoader class? It keeps a Map that
contains what are called "global" classes and separate Maps for each
component that contain something called "local" classes. I don't
understand why the separation is necessary and I wonder how many entries
those Maps get.

-Adrian

[Jacques Le Roux]

And as a second reminder http://people.apache.org/~bimargulies/memory-efficient-java-tutorial.pdf (I hope I will find enough time
one day to read all this in every details...)

Jacques

From: "Tim Ruppert" <[hidden email]>
This is great Adam - thanks.

Cheers,
Ruppert

On Jan 26, 2010, at 11:49 AM, Adam Heath wrote:

> [hidden email] wrote:
>> Author: lektran
>> Date: Thu Sep 24 00:05:05 2009
>> New Revision: 818321
>>
>> URL: http://svn.apache.org/viewvc?rev=818321&view=rev
>> Log:
>> Added the maxPermSize jvm arg the various startup methods since OFBiz now seems to require more than the default 64m.  Also
>> removed startofbizNoLog.bat since it seems somewhat superfluous, instead made startofbiz.bat log to the console by default and
>> added a comment about how to change it if desired.
>
> (nothing like delayed responses)
>
> bother.  We had made a branch of ofbiz trunk Sep 5; we then used this
> branch up until earlier this week.  This internal branch was able to
> run fine with just a 64M permgen size.  However, once we jumped to the
> latest trunk, we discovered that ofbiz suddenly used more memory, and
> actually was throwing an OutOfMemory error.  This was due to not
> having this particular change in the debian ofbiz init script.
>
> Increasing the maxPermSize number is just not acceptable; having to
> allocate so much more memory would increase the cost of hosting.
>
> The discovery of this bug gave me a window of changes, to try and
> track down the cause.  Near as I can tell, this memory issue is due to
> *all* service definitions having a custom, on-the-fly, class compiled
> for them.  Each class is inside it's very own unique classloader.  So,
> not only do you have all the internal memory for the class+classloader
> combo(which includes a Vector with a default internal size of 10, but
> only one element filled in the array), the class data itself is stored
> in the permgen area.
>
> Based on simple searching, I see 1926 class files(this number may not
> be 100% accurate).  I then see 2595 service definitions.  Doing a jmap
> -dump immediately after ofbiz starts, then running jhat on the
> resulting dump file, shows tons and tons of classloaders loaded.
>
> I'm working on a fix for this.  Since this bit of code is using
> webslinger-invoker.jar, what I am doing in improving my test
> coverage(I was at 80% initially, currently at 90%), then will import
> the code in full into ofbiz(this was always the long-term plan with
> the webslinger code anyways).  I'll then modify the invoker code to
> support specifying a shared classloader, which should reduce the
> memory usage.  I'll also be adding a way to turn off this compiled
> class globally, per file, or per service.

[Adam Heath-2]

Jacques Le Roux wrote:
> And as a second reminder
> http://people.apache.org/~bimargulies/memory-efficient-java-tutorial.pdf
> (I hope I will find enough time one day to read all this in every
> details...)

Java Concurrency in Practice, http://www.javaconcurrencyinpractice.com/.

Non-blocking programming, memory debugging, the works.  I know all
about memory optimization and complex programming techniques described
in this book.  It's a most wonderful read.

[Jacques Le Roux]

From: "Adam Heath" <[hidden email]>

> Jacques Le Roux wrote:
>> And as a second reminder
>> http://people.apache.org/~bimargulies/memory-efficient-java-tutorial.pdf
>> (I hope I will find enough time one day to read all this in every
>> details...)
>
> Java Concurrency in Practice, http://www.javaconcurrencyinpractice.com/.
>
> Non-blocking programming, memory debugging, the works.  I know all
> about memory optimization and complex programming techniques described
> in this book.  It's a most wonderful read.

I have already seen recommendation about it, another one in my list...

Thanks

Jacques