geronimo/dbcp memory leak

[Adam Heath-2]

I just had an OutOfMemoryError on one of our deployments.  I then ran
jmap, to do a heap dump, and looked at the resulting java.hprof binary
file using IBM's HeapAnalyzer(1), and jhat.

What I found, was a WeakHashMap inside dbcp, that contained 346000
WeakHashMap$Entry objects, taking up 125M of memory, out of 193M total
heap.  These Entry objects contain Transaction instances, not Connections.

Has anyone ran into something similiar?  This is against ofbiz version
595296.

ps: I'm currently running the ofbiz test cases in this ofbiz version on
an 8-way machine, with 8G of ram, underneath hprof, trying to track this
down.

1: http://www.alphaworks.ibm.com/tech/heapanalyzer

[Adam Heath-2]

Adam Heath wrote:

> I just had an OutOfMemoryError on one of our deployments.  I then ran
> jmap, to do a heap dump, and looked at the resulting java.hprof binary
> file using IBM's HeapAnalyzer(1), and jhat.
>
> What I found, was a WeakHashMap inside dbcp, that contained 346000
> WeakHashMap$Entry objects, taking up 125M of memory, out of 193M total
> heap.  These Entry objects contain Transaction instances, not Connections.
>
> Has anyone ran into something similiar?  This is against ofbiz version
> 595296.
>
> ps: I'm currently running the ofbiz test cases in this ofbiz version on
> an 8-way machine, with 8G of ram, underneath hprof, trying to track this
> down.
>
> 1: http://www.alphaworks.ibm.com/tech/heapanalyzer

Got it.  This version of ofbiz is using commons-dbcp 20070730, and a
leak fix was done on 20081012.  I'll retest, you can ignore this mail
for now.

[Adam Heath-2]

Adam Heath wrote:

> Adam Heath wrote:
>> I just had an OutOfMemoryError on one of our deployments.  I then ran
>> jmap, to do a heap dump, and looked at the resulting java.hprof binary
>> file using IBM's HeapAnalyzer(1), and jhat.
>>
>> What I found, was a WeakHashMap inside dbcp, that contained 346000
>> WeakHashMap$Entry objects, taking up 125M of memory, out of 193M total
>> heap.  These Entry objects contain Transaction instances, not Connections.
>>
>> Has anyone ran into something similiar?  This is against ofbiz version
>> 595296.
>>
>> ps: I'm currently running the ofbiz test cases in this ofbiz version on
>> an 8-way machine, with 8G of ram, underneath hprof, trying to track this
>> down.
>>
>> 1: http://www.alphaworks.ibm.com/tech/heapanalyzer
>
> Got it.  This version of ofbiz is using commons-dbcp 20070730, and a
> leak fix was done on 20081012.  I'll retest, you can ignore this mail
> for now.

20071012, in case anyone cares.

[Tim Ruppert]

Thanks Adam - this will be helpful.

Cheers,
Tim
--
Tim Ruppert
HotWax Media
http://www.hotwaxmedia.com

o:801.649.6594
f:801.649.6595

----- "Adam Heath" <[hidden email]> wrote:

> Adam Heath wrote:
> > I just had an OutOfMemoryError on one of our deployments.  I then
> ran
> > jmap, to do a heap dump, and looked at the resulting java.hprof
> binary
> > file using IBM's HeapAnalyzer(1), and jhat.
> >
> > What I found, was a WeakHashMap inside dbcp, that contained 346000
> > WeakHashMap$Entry objects, taking up 125M of memory, out of 193M
> total
> > heap.  These Entry objects contain Transaction instances, not
> Connections.
> >
> > Has anyone ran into something similiar?  This is against ofbiz
> version
> > 595296.
> >
> > ps: I'm currently running the ofbiz test cases in this ofbiz version
> on
> > an 8-way machine, with 8G of ram, underneath hprof, trying to track
> this
> > down.
> >
> > 1: http://www.alphaworks.ibm.com/tech/heapanalyzer
>
> Got it.  This version of ofbiz is using commons-dbcp 20070730, and a
> leak fix was done on 20081012.  I'll retest, you can ignore this mail
> for now.

[Adam Heath-2]

Adam Heath wrote:

> Adam Heath wrote:
>> I just had an OutOfMemoryError on one of our deployments.  I then ran
>> jmap, to do a heap dump, and looked at the resulting java.hprof binary
>> file using IBM's HeapAnalyzer(1), and jhat.
>>
>> What I found, was a WeakHashMap inside dbcp, that contained 346000
>> WeakHashMap$Entry objects, taking up 125M of memory, out of 193M total
>> heap.  These Entry objects contain Transaction instances, not Connections.
>>
>> Has anyone ran into something similiar?  This is against ofbiz version
>> 595296.
>>
>> ps: I'm currently running the ofbiz test cases in this ofbiz version on
>> an 8-way machine, with 8G of ram, underneath hprof, trying to track this
>> down.
>>
>> 1: http://www.alphaworks.ibm.com/tech/heapanalyzer
>
> Got it.  This version of ofbiz is using commons-dbcp 20070730, and a
> leak fix was done on 20081012.  I'll retest, you can ignore this mail
> for now.

In detail, there was a WeakHashMap, that contained a Transaction key,
pointed to a TransactionContext, that referenced the previous
Transaction key.

So, while the key itself was a WeakRef, the value had a hard reference
back to the key.  Insta-Leak(tm)!

[Jacques Le Roux]

Hi Adam,

Hopefully this should habe been fixed by the new DBCP version

Jacques

From: "Adam Heath" <[hidden email]>

>I just had an OutOfMemoryError on one of our deployments.  I then ran
> jmap, to do a heap dump, and looked at the resulting java.hprof binary
> file using IBM's HeapAnalyzer(1), and jhat.
>
> What I found, was a WeakHashMap inside dbcp, that contained 346000
> WeakHashMap$Entry objects, taking up 125M of memory, out of 193M total
> heap.  These Entry objects contain Transaction instances, not Connections.
>
> Has anyone ran into something similiar?  This is against ofbiz version
> 595296.
>
> ps: I'm currently running the ofbiz test cases in this ofbiz version on
> an 8-way machine, with 8G of ram, underneath hprof, trying to track this
> down.
>
> 1: http://www.alphaworks.ibm.com/tech/heapanalyzer
>

[Tim Ruppert]

This is a biggie - looking forward to seeing this fixed in a huge  
way.  We went to DBCP because of problems in Minerva :)

Cheers,
Ruppert
--
Tim Ruppert
HotWax Media
http://www.hotwaxmedia.com

o:801.649.6594
f:801.649.6595

On Nov 17, 2009, at 12:51 PM, Jacques Le Roux wrote:

> Hi Adam,
>
> Hopefully this should habe been fixed by the new DBCP version
>
> Jacques
>
> From: "Adam Heath" <[hidden email]>
>> I just had an OutOfMemoryError on one of our deployments.  I then ran
>> jmap, to do a heap dump, and looked at the resulting java.hprof  
>> binary
>> file using IBM's HeapAnalyzer(1), and jhat.
>> What I found, was a WeakHashMap inside dbcp, that contained 346000
>> WeakHashMap$Entry objects, taking up 125M of memory, out of 193M  
>> total
>> heap.  These Entry objects contain Transaction instances, not  
>> Connections.
>> Has anyone ran into something similiar?  This is against ofbiz  
>> version
>> 595296.
>> ps: I'm currently running the ofbiz test cases in this ofbiz  
>> version on
>> an 8-way machine, with 8G of ram, underneath hprof, trying to track  
>> this
>> down.
>> 1: http://www.alphaworks.ibm.com/tech/heapanalyzer
>>
>

[Adam Heath-2]

Jacques Le Roux wrote:
> Hi Adam,
>
> Hopefully this should habe been fixed by the new DBCP version

Actually, no, it's been fixed for a while now.  See OFBIZ-1818, which
was commited on 662852.

[Tim Ruppert]

Ouch :(  Jacques got my hopes up.  There are definitely still issues  
around this though ...

Cheers,
Ruppert

On Nov 17, 2009, at 1:35 PM, Adam Heath wrote:

> Jacques Le Roux wrote:
>> Hi Adam,
>>
>> Hopefully this should habe been fixed by the new DBCP version
>
> Actually, no, it's been fixed for a while now.  See OFBIZ-1818, which
> was commited on 662852.

[Jacques Le Roux]

Actually I was referring to https://issues.apache.org/jira/browse/OFBIZ-2599
Glad to know that the one Adam reported was already fixed

Jacques

From: "Tim Ruppert" <[hidden email]>

> Ouch :(  Jacques got my hopes up.  There are definitely still issues  
> around this though ...
>
> Cheers,
> Ruppert
>
> On Nov 17, 2009, at 1:35 PM, Adam Heath wrote:
>
>> Jacques Le Roux wrote:
>>> Hi Adam,
>>>
>>> Hopefully this should habe been fixed by the new DBCP version
>>
>> Actually, no, it's been fixed for a while now.  See OFBIZ-1818, which
>> was commited on 662852.
>
>