Sluggish server / Minerva exception

[John Martin]

We have an ofBiz application server that is running in a staging/demo
environment that gets little usage.  In December I updated the source code
via subversion from the trunk so the code is fairly up to date.  The
application starts out around 100MB but just sitting there grows to approx
300MB of memory over time under very lite loads.  It is accessing a mySQL
database.  I noticed today that when I went to login to the application it
was extremely slow.  After restarting the ofBiz application the performance
issue went away.

I looked into the ofbiz.log file and noticed an exception / error "There was
an error getting a Minerva datasource".  I don't know if this is the cause
of the performance issues or not but started going down this path since the
error was specific to the login:

=========================

 2007-01-12 14:56:19,025 (RMI TCP Connection(34512)-192.168.1.3) [
ConnectionFactory.java:82 :ERROR]
---- runtime exception report
--------------------------------------------------
There was an error getting a Minerva datasource.
Exception: java.lang.RuntimeException
Message: No ManagedConnections Available!
---- stack trace
---------------------------------------------------------------
java.lang.RuntimeException: No ManagedConnections Available!
org.ofbiz.minerva.pool.ObjectPool.getObject(ObjectPool.java:655)
org.ofbiz.minerva.pool.jdbc.xa.XAPoolDataSource.getConnection(
XAPoolDataSource.java:355)
org.ofbiz.entity.transaction.MinervaConnectionFactory.getConnection(
MinervaConnectionFactory.java:43)
org.ofbiz.entity.jdbc.ConnectionFactory.tryGenericConnectionSources(
ConnectionFactory.java:79)
org.ofbiz.geronimo.GeronimoTransactionFactory.getConnection(
GeronimoTransactionFactory.java:97)

.....

ConnectionFactory.java:71 :ERROR] ******* ERROR: No database connection
found for helperName "localmysql"

2007-01-12 14:56:19,025 (RMI TCP Connection(34512)-192.168.1.3) [
GenericDelegator.java:574:ERROR]

---- exception report
----------------------------------------------------------
Failure in create operation for entity [UserLoginHistory]:
org.ofbiz.entity.GenericEntityException: Exception while inserting the
following entity: [GenericEntity:UserLoginHistory][createdStamp,2007-01-12
14:55:59.025(java.sql.Timestamp)][createdTxStamp,2007-01-12

=========================
 In doing some research, I found that Minerva was replaced with Geronimo
back in Jun 06 so I was surprised and confused seeing this exception.  I
don't know if there is some configuration in my environment that is messed
up and therefore still using Minerva erroneously?

Any thoughts?

Thanks - John

[cjhowe]

Two issues there...
SVN:  You likely checked out using
svn.ofbiz.org/svn/ofbiz/trunk which is the old
repository.  The new svn repository is at
http://svn.apache.org/viewvc/ofbiz/

(OFBiz SVN has been off Minerva for nearly a year now)

The second issue, if you're using MySQL 5.0+, the auto
reconnect attribute was deprecated as a security
feature and so MySql will close the connection after 8
hours of inactivity.  For a work around please see
this recent discussion.

http://www.nabble.com/forum/ViewPost.jtp?post=8213433&framed=y


--- John Martin <[hidden email]> wrote:

> We have an ofBiz application server that is running
> in a staging/demo
> environment that gets little usage.  In December I
> updated the source code
> via subversion from the trunk so the code is fairly
> up to date.  The
> application starts out around 100MB but just sitting
> there grows to approx
> 300MB of memory over time under very lite loads.  It
> is accessing a mySQL
> database.  I noticed today that when I went to login
> to the application it
> was extremely slow.  After restarting the ofBiz
> application the performance
> issue went away.
>
> I looked into the ofbiz.log file and noticed an
> exception / error "There was
> an error getting a Minerva datasource".  I don't
> know if this is the cause
> of the performance issues or not but started going
> down this path since the
> error was specific to the login:
>
> =========================
>
>  2007-01-12 14:56:19,025 (RMI TCP
> Connection(34512)-192.168.1.3) [
> ConnectionFactory.java:82 :ERROR]
> ---- runtime exception report
> --------------------------------------------------
> There was an error getting a Minerva datasource.
> Exception: java.lang.RuntimeException
> Message: No ManagedConnections Available!
> ---- stack trace
>

---------------------------------------------------------------
> java.lang.RuntimeException: No ManagedConnections
> Available!
>
org.ofbiz.minerva.pool.ObjectPool.getObject(ObjectPool.java:655)
>
org.ofbiz.minerva.pool.jdbc.xa.XAPoolDataSource.getConnection(
> XAPoolDataSource.java:355)
>
org.ofbiz.entity.transaction.MinervaConnectionFactory.getConnection(
> MinervaConnectionFactory.java:43)
>
org.ofbiz.entity.jdbc.ConnectionFactory.tryGenericConnectionSources(
> ConnectionFactory.java:79)
>
org.ofbiz.geronimo.GeronimoTransactionFactory.getConnection(

> GeronimoTransactionFactory.java:97)
>
> .....
>
> ConnectionFactory.java:71 :ERROR] ******* ERROR: No
> database connection
> found for helperName "localmysql"
>
> 2007-01-12 14:56:19,025 (RMI TCP
> Connection(34512)-192.168.1.3) [
> GenericDelegator.java:574:ERROR]
>
> ---- exception report
>

----------------------------------------------------------
> Failure in create operation for entity
> [UserLoginHistory]:
> org.ofbiz.entity.GenericEntityException: Exception
> while inserting the
> following entity:
>
[GenericEntity:UserLoginHistory][createdStamp,2007-01-12
>
14:55:59.025(java.sql.Timestamp)][createdTxStamp,2007-01-12

>
> =========================
>  In doing some research, I found that Minerva was
> replaced with Geronimo
> back in Jun 06 so I was surprised and confused
> seeing this exception.  I
> don't know if there is some configuration in my
> environment that is messed
> up and therefore still using Minerva erroneously?
>
> Any thoughts?
>
> Thanks - John
>

[Jacques Le Roux]

Chris,

BTW you may put this *important* (and seemingly reccurring) autoreconnect MySQL 5.0+  tip in the SandBox for the moment :
http://docs.ofbiz.org/display/OFBIZ/Sandbox

And thanks for all your great and accurate help for users (and even devs ;o) !

Jacques

----- Original Message -----
From: "Chris Howe" <[hidden email]>
To: <[hidden email]>
Sent: Friday, January 12, 2007 10:19 PM
Subject: Re: Sluggish server / Minerva exception

> Two issues there...
> SVN:  You likely checked out using
> svn.ofbiz.org/svn/ofbiz/trunk which is the old
> repository.  The new svn repository is at
> http://svn.apache.org/viewvc/ofbiz/
>
> (OFBiz SVN has been off Minerva for nearly a year now)
>
> The second issue, if you're using MySQL 5.0+, the auto
> reconnect attribute was deprecated as a security
> feature and so MySql will close the connection after 8
> hours of inactivity.  For a work around please see
> this recent discussion.
>
> http://www.nabble.com/forum/ViewPost.jtp?post=8213433&framed=y
>
>
> --- John Martin <[hidden email]> wrote:
>
> > We have an ofBiz application server that is running
> > in a staging/demo
> > environment that gets little usage.  In December I
> > updated the source code
> > via subversion from the trunk so the code is fairly
> > up to date.  The
> > application starts out around 100MB but just sitting
> > there grows to approx
> > 300MB of memory over time under very lite loads.  It
> > is accessing a mySQL
> > database.  I noticed today that when I went to login
> > to the application it
> > was extremely slow.  After restarting the ofBiz
> > application the performance
> > issue went away.
> >
> > I looked into the ofbiz.log file and noticed an
> > exception / error "There was
> > an error getting a Minerva datasource".  I don't
> > know if this is the cause
> > of the performance issues or not but started going
> > down this path since the
> > error was specific to the login:
> >
> > =========================
> >
> >  2007-01-12 14:56:19,025 (RMI TCP
> > Connection(34512)-192.168.1.3) [
> > ConnectionFactory.java:82 :ERROR]
> > ---- runtime exception report
> > --------------------------------------------------
> > There was an error getting a Minerva datasource.
> > Exception: java.lang.RuntimeException
> > Message: No ManagedConnections Available!
> > ---- stack trace
> >
> ---------------------------------------------------------------
> > java.lang.RuntimeException: No ManagedConnections
> > Available!
> >
> org.ofbiz.minerva.pool.ObjectPool.getObject(ObjectPool.java:655)
> >
> org.ofbiz.minerva.pool.jdbc.xa.XAPoolDataSource.getConnection(
> > XAPoolDataSource.java:355)
> >
> org.ofbiz.entity.transaction.MinervaConnectionFactory.getConnection(
> > MinervaConnectionFactory.java:43)
> >
> org.ofbiz.entity.jdbc.ConnectionFactory.tryGenericConnectionSources(
> > ConnectionFactory.java:79)
> >
> org.ofbiz.geronimo.GeronimoTransactionFactory.getConnection(
> > GeronimoTransactionFactory.java:97)
> >
> > .....
> >
> > ConnectionFactory.java:71 :ERROR] ******* ERROR: No
> > database connection
> > found for helperName "localmysql"
> >
> > 2007-01-12 14:56:19,025 (RMI TCP
> > Connection(34512)-192.168.1.3) [
> > GenericDelegator.java:574:ERROR]
> >
> > ---- exception report
> >
> ----------------------------------------------------------
> > Failure in create operation for entity
> > [UserLoginHistory]:
> > org.ofbiz.entity.GenericEntityException: Exception
> > while inserting the
> > following entity:
> >
> [GenericEntity:UserLoginHistory][createdStamp,2007-01-12
> >
> 14:55:59.025(java.sql.Timestamp)][createdTxStamp,2007-01-12
> >
> > =========================
> >  In doing some research, I found that Minerva was
> > replaced with Geronimo
> > back in Jun 06 so I was surprised and confused
> > seeing this exception.  I
> > don't know if there is some configuration in my
> > environment that is messed
> > up and therefore still using Minerva erroneously?
> >
> > Any thoughts?
> >
> > Thanks - John
> >

[John Martin]

Thanks Chris -

I think you nailed my login error with the mySQL 5 connection being
terminated.  I have one question on this problem and your work around.

ofBiz caches n # of connections and if there are lite periods of time (i.e.
1am - 9am) the insert/delete scheduled task is only going to refresh one of
the connections.  The remaining connections will get stale and disconnected
by mySQL.  Over time, the # of active connections will dewindle to just a
couple.  The system will appear to be working but will be limping along on
just a few connections, sort of like a 8 cylinder engine running on 2
cylinders but without the pinging and backfiring.

It seems to me that the insert/delete work around is just a bandaid and in a
production environment that is not being pounded on 24x7 is going to run
into the dwindling connection pool issue.  To fix this problem, I think that
the connection pool manager might need a thread that just cycles throught
the connections and just reconnects them whenever they fail.  I haven't
confirmed this but would imaging that ofBiz would need to be restarted if I
were to recycle mySQL since all the connections in the pool would go stale.
Am I off on my assumption here?

As for the svn server, is there an easy way to switch from one svn server to
the other?  I've made numerous changes to the code, some that will be
submitted to the project and I don't want to loose those changes.

Thanks again.
John

[David E Jones]

On Jan 12, 2007, at 3:23 PM, John Martin wrote:

> As for the svn server, is there an easy way to switch from one svn  
> server to
> the other?  I've made numerous changes to the code, some that will be
> submitted to the project and I don't want to loose those changes.

The easiest way is to use the "svn switch" command to point to the  
new repo. This works fine with the recent OFBiz repository change.  
It's pretty easy to figure out, or just search the archives for  
examples.

-David

[cjhowe]

--- John Martin <[hidden email]> wrote:

<snip>

> It seems to me that the insert/delete work around is
> just a bandaid and in a
> production environment that is not being pounded on
> 24x7 is going to run
> into the dwindling connection pool issue.  To fix
> this problem, I think that
> the connection pool manager might need a thread that
> just cycles throught
> the connections and just reconnects them whenever
> they fail.  I haven't
> confirmed this but would imaging that ofBiz would
> need to be restarted if I
> were to recycle mySQL since all the connections in
> the pool would go stale.
> Am I off on my assumption here?

You caught me.  It does appear to be just a bandaid.
The only places where I've used this are deployments
where there aren't too many concurrent connections.
So it has worked so far.  It's those medium sized
sites this would be a problem for (too large and
you're using all of the connections inside of the 8
hours, too small and the one gimpy connection is good
enough :-) ).

If you can think of a better way to handle it, I'd be
more than happy to help test it out.  OFBiz handling
the database stuff is one of the reasons that
originally drew me to it. So, I've always been
apprehensive about messing with it.  

[cjhowe]

Thanks Jacques, I was looking for a place for it last
time around and didn't want to get in the way of the
moving effort.

It can be found here
http://docs.ofbiz.org/x/-QM

I'll be sure to update it if/when a more complete
solution is found.


--- Jacques Le Roux <[hidden email]>
wrote:

> Chris,
>
> BTW you may put this *important* (and seemingly
> reccurring) autoreconnect MySQL 5.0+  tip in the
> SandBox for the moment :
> http://docs.ofbiz.org/display/OFBIZ/Sandbox
>
> And thanks for all your great and accurate help for
> users (and even devs ;o) !
>
> Jacques
>
> ----- Original Message -----
> From: "Chris Howe" <[hidden email]>
> To: <[hidden email]>
> Sent: Friday, January 12, 2007 10:19 PM
> Subject: Re: Sluggish server / Minerva exception
>
>
> > Two issues there...
> > SVN:  You likely checked out using
> > svn.ofbiz.org/svn/ofbiz/trunk which is the old
> > repository.  The new svn repository is at
> > http://svn.apache.org/viewvc/ofbiz/
> >
> > (OFBiz SVN has been off Minerva for nearly a year
> now)
> >
> > The second issue, if you're using MySQL 5.0+, the
> auto
> > reconnect attribute was deprecated as a security
> > feature and so MySql will close the connection
> after 8
> > hours of inactivity.  For a work around please see
> > this recent discussion.
> >
> >
>

http://www.nabble.com/forum/ViewPost.jtp?post=8213433&framed=y

> >
> >
> > --- John Martin <[hidden email]> wrote:
> >
> > > We have an ofBiz application server that is
> running
> > > in a staging/demo
> > > environment that gets little usage.  In December
> I
> > > updated the source code
> > > via subversion from the trunk so the code is
> fairly
> > > up to date.  The
> > > application starts out around 100MB but just
> sitting
> > > there grows to approx
> > > 300MB of memory over time under very lite loads.
>  It
> > > is accessing a mySQL
> > > database.  I noticed today that when I went to
> login
> > > to the application it
> > > was extremely slow.  After restarting the ofBiz
> > > application the performance
> > > issue went away.
> > >
> > > I looked into the ofbiz.log file and noticed an
> > > exception / error "There was
> > > an error getting a Minerva datasource".  I don't
> > > know if this is the cause
> > > of the performance issues or not but started
> going
> > > down this path since the
> > > error was specific to the login:
> > >
> > > =========================
> > >
> > >  2007-01-12 14:56:19,025 (RMI TCP
> > > Connection(34512)-192.168.1.3) [
> > > ConnectionFactory.java:82 :ERROR]
> > > ---- runtime exception report
> > >
> --------------------------------------------------
> > > There was an error getting a Minerva datasource.
> > > Exception: java.lang.RuntimeException
> > > Message: No ManagedConnections Available!
> > > ---- stack trace
> > >
> >
>

---------------------------------------------------------------
> > > java.lang.RuntimeException: No
> ManagedConnections
> > > Available!
> > >
> >
>
org.ofbiz.minerva.pool.ObjectPool.getObject(ObjectPool.java:655)
> > >
> >
>
org.ofbiz.minerva.pool.jdbc.xa.XAPoolDataSource.getConnection(
> > > XAPoolDataSource.java:355)
> > >
> >
>
org.ofbiz.entity.transaction.MinervaConnectionFactory.getConnection(
> > > MinervaConnectionFactory.java:43)
> > >
> >
>
org.ofbiz.entity.jdbc.ConnectionFactory.tryGenericConnectionSources(
> > > ConnectionFactory.java:79)
> > >
> >
>
org.ofbiz.geronimo.GeronimoTransactionFactory.getConnection(

> > > GeronimoTransactionFactory.java:97)
> > >
> > > .....
> > >
> > > ConnectionFactory.java:71 :ERROR] ******* ERROR:
> No
> > > database connection
> > > found for helperName "localmysql"
> > >
> > > 2007-01-12 14:56:19,025 (RMI TCP
> > > Connection(34512)-192.168.1.3) [
> > > GenericDelegator.java:574:ERROR]
> > >
> > > ---- exception report
> > >
> >
>

----------------------------------------------------------
> > > Failure in create operation for entity
> > > [UserLoginHistory]:
> > > org.ofbiz.entity.GenericEntityException:
> Exception
> > > while inserting the
> > > following entity:
> > >
> >
>
[GenericEntity:UserLoginHistory][createdStamp,2007-01-12
> > >
> >
>
14:55:59.025(java.sql.Timestamp)][createdTxStamp,2007-01-12

> > >
> > > =========================
> > >  In doing some research, I found that Minerva
> was
> > > replaced with Geronimo
> > > back in Jun 06 so I was surprised and confused
> > > seeing this exception.  I
> > > don't know if there is some configuration in my
> > > environment that is messed
> > > up and therefore still using Minerva
> erroneously?
> > >
> > > Any thoughts?
> > >
> > > Thanks - John
> > >
>
>