Exception on high load

[Florin Popa]

Hello,

Using JMeter I am trying 500 concurrent requests on /ecommerce module

First of all, I had to disable the recording of server hits - because
the crash came too early, but as soon as I tried again, I got this:

  [java] 2009-06-18 18:12:32,780 (http-0.0.0.0-8080-Processor30)
[InheritableTransactionContext.java:311:ERROR] Unable to roll back
transaction
     [java] java.lang.IllegalStateException: Status is STATUS_NO_TRANSACTION
     [java]     at
org.apache.geronimo.transaction.manager.TransactionImpl.rollback(TransactionImpl.java:438)
     [java]     at
org.apache.geronimo.transaction.context.InheritableTransactionContext.rollbackAndThrow(InheritableTransactionContext.java:308)
     [java]     at
org.apache.geronimo.transaction.context.InheritableTransactionContext.complete(InheritableTransactionContext.java:199)
     [java]     at
org.apache.geronimo.transaction.context.InheritableTransactionContext.commit(InheritableTransactionContext.java:146)
     [java]     at
org.apache.geronimo.transaction.context.GeronimoTransactionManager.commit(GeronimoTransactionManager.java:81)
     [java]     at
org.ofbiz.entity.transaction.TransactionUtil.commit(TransactionUtil.java:179)
     [java]     at
org.ofbiz.entity.transaction.TransactionUtil.commit(TransactionUtil.java:165)
     [java]     at
org.ofbiz.widget.screen.ModelScreen.renderScreenString(ModelScreen.java:163)
     [java]     at
org.ofbiz.widget.screen.ScreenRenderer.render(ScreenRenderer.java:105)
     [java]     at
org.ofbiz.widget.screen.ScreenRenderer.render(ScreenRenderer.java:90)
     [java]     at
org.ofbiz.widget.screen.ScreenWidgetViewHandler.render(ScreenWidgetViewHandler.java:78)
     [java]     at
org.ofbiz.webapp.control.RequestHandler.renderView(RequestHandler.java:694)
     [java]     at
org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:469)
     [java]     at
org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:189)
     [java]     at
org.ofbiz.webapp.control.ControlServlet.doPost(ControlServlet.java:77)
     [java]     at
javax.servlet.http.HttpServlet.service(HttpServlet.java:615)
     [java]     at
javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
     [java]     at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
     [java]     at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
     [java]     at
org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:738)
     [java]     at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
     [java]     at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
     [java]     at
org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:371)
     [java]     at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
     [java]     at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
     [java]     at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
     [java]     at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
     [java]     at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
     [java]     at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
     [java]     at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
     [java]     at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541)
     [java]     at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
     [java]     at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
     [java]     at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
     [java]     at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
     [java]     at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
     [java]     at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
     [java]     at java.lang.Thread.run(Thread.java:619)
     [java] 2009-06-18 18:12:32,782 (http-0.0.0.0-8080-Processor30)
[        ModelScreen.java:165:ERROR]
     [java] ---- exception report
----------------------------------------------------------



Any help would be really appreciated

Best regards,
 Flopa

[David E. Jones-2]

Have you configured your server to handle this sort of load?

While most deployments don't require handling 500 simultaneous  
requests, it is certainly possible, but OFBiz doesn't come OOTB with  
settings like that.

Take a look at the Technical Production Setup Guide on docs.ofbiz.org,  
and make sure to configure at least the following (these are off the  
top of my head, so there may be more):

1. Tomcat HTTP listener threads (at least 500)
2. app server (OFBiz) database connection pool size (at least 500 plus  
a buffer, 20% over is usually good)
3. the number of database incoming connections available on your  
database server (postgres or whatever) (a few higher than for #2)

If you're trying to run this with Derby you may want to consider not  
doing that... I don't know if Derby in an embedded mode will do so  
well with 500 simultaneous threads.

-David


On Jun 18, 2009, at 9:15 AM, Florin Popa wrote:

> Hello,
>
> Using JMeter I am trying 500 concurrent requests on /ecommerce module
>
> First of all, I had to disable the recording of server hits -  
> because the crash came too early, but as soon as I tried again, I  
> got this:
>
> [java] 2009-06-18 18:12:32,780 (http-0.0.0.0-8080-Processor30)  
> [InheritableTransactionContext.java:311:ERROR] Unable to roll back  
> transaction
>    [java] java.lang.IllegalStateException: Status is  
> STATUS_NO_TRANSACTION
>    [java]     at  
> org
> .apache
> .geronimo
> .transaction.manager.TransactionImpl.rollback(TransactionImpl.java:
> 438)
>    [java]     at  
> org
> .apache
> .geronimo
> .transaction
> .context
> .InheritableTransactionContext
> .rollbackAndThrow(InheritableTransactionContext.java:308)
>    [java]     at  
> org
> .apache
> .geronimo
> .transaction
> .context
> .InheritableTransactionContext
> .complete(InheritableTransactionContext.java:199)
>    [java]     at  
> org
> .apache
> .geronimo
> .transaction
> .context
> .InheritableTransactionContext
> .commit(InheritableTransactionContext.java:146)
>    [java]     at  
> org
> .apache
> .geronimo
> .transaction
> .context
> .GeronimoTransactionManager.commit(GeronimoTransactionManager.java:81)
>    [java]     at  
> org
> .ofbiz
> .entity.transaction.TransactionUtil.commit(TransactionUtil.java:179)
>    [java]     at  
> org
> .ofbiz
> .entity.transaction.TransactionUtil.commit(TransactionUtil.java:165)
>    [java]     at  
> org
> .ofbiz.widget.screen.ModelScreen.renderScreenString(ModelScreen.java:
> 163)
>    [java]     at  
> org.ofbiz.widget.screen.ScreenRenderer.render(ScreenRenderer.java:105)
>    [java]     at  
> org.ofbiz.widget.screen.ScreenRenderer.render(ScreenRenderer.java:90)
>    [java]     at  
> org
> .ofbiz
> .widget
> .screen.ScreenWidgetViewHandler.render(ScreenWidgetViewHandler.java:
> 78)
>    [java]     at  
> org
> .ofbiz.webapp.control.RequestHandler.renderView(RequestHandler.java:
> 694)
>    [java]     at  
> org
> .ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:
> 469)
>    [java]     at  
> org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:189)
>    [java]     at  
> org.ofbiz.webapp.control.ControlServlet.doPost(ControlServlet.java:77)
>    [java]     at  
> javax.servlet.http.HttpServlet.service(HttpServlet.java:615)
>    [java]     at  
> javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
>    [java]     at  
> org
> .apache
> .catalina
> .core
> .ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
> 252)
>    [java]     at  
> org
> .apache
> .catalina
> .core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
>    [java]     at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter
> (UrlRewriteFilter.java:738)
>    [java]     at  
> org
> .apache
> .catalina
> .core
> .ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
> 202)
>    [java]     at  
> org
> .apache
> .catalina
> .core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
>    [java]     at  
> org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:
> 371)
>    [java]     at  
> org
> .apache
> .catalina
> .core
> .ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
> 202)
>    [java]     at  
> org
> .apache
> .catalina
> .core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
>    [java]     at  
> org
> .apache
> .catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:
> 213)
>    [java]     at  
> org
> .apache
> .catalina.core.StandardContextValve.invoke(StandardContextValve.java:
> 178)
>    [java]     at  
> org
> .apache
> .catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
>    [java]     at  
> org
> .apache
> .catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
>    [java]     at  
> org
> .apache
> .catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:
> 107)
>    [java]     at  
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:
> 541)
>    [java]     at  
> org
> .apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:
> 148)
>    [java]     at  
> org
> .apache.coyote.http11.Http11Processor.process(Http11Processor.java:
> 869)
>    [java]     at org.apache.coyote.http11.Http11BaseProtocol
> $Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:
> 664)
>    [java]     at  
> org
> .apache
> .tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:
> 527)
>    [java]     at  
> org
> .apache
> .tomcat
> .util
> .net
> .LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
>    [java]     at org.apache.tomcat.util.threads.ThreadPool
> $ControlRunnable.run(ThreadPool.java:684)
>    [java]     at java.lang.Thread.run(Thread.java:619)
>    [java] 2009-06-18 18:12:32,782 (http-0.0.0.0-8080-Processor30)  
> [        ModelScreen.java:165:ERROR]
>    [java] ---- exception report  
> ----------------------------------------------------------
>
>
>
> Any help would be really appreciated
>
> Best regards,
> Flopa

[Florin Popa]

Hello David,

Thanks a lot, it worked fine for 500 concurrent requests - just hitting
the first page which has view promo products.
I tried also 100 concurrent searches - having lucene integrated - it
seems I get the same exception after about 55 requests...Could it be
other setting to  be changed?

Best regards,
 Flopa

>
> Have you configured your server to handle this sort of load?
>
> While most deployments don't require handling 500 simultaneous
> requests, it is certainly possible, but OFBiz doesn't come OOTB with
> settings like that.
>
> Take a look at the Technical Production Setup Guide on docs.ofbiz.org,
> and make sure to configure at least the following (these are off the
> top of my head, so there may be more):
>
> 1. Tomcat HTTP listener threads (at least 500)
> 2. app server (OFBiz) database connection pool size (at least 500 plus
> a buffer, 20% over is usually good)
> 3. the number of database incoming connections available on your
> database server (postgres or whatever) (a few higher than for #2)
>
> If you're trying to run this with Derby you may want to consider not
> doing that... I don't know if Derby in an embedded mode will do so
> well with 500 simultaneous threads.
>
> -David
>
>
> On Jun 18, 2009, at 9:15 AM, Florin Popa wrote:
>
>> Hello,
>>
>> Using JMeter I am trying 500 concurrent requests on /ecommerce module
>>
>> First of all, I had to disable the recording of server hits - because
>> the crash came too early, but as soon as I tried again, I got this:
>>
>> [java] 2009-06-18 18:12:32,780 (http-0.0.0.0-8080-Processor30)
>> [InheritableTransactionContext.java:311:ERROR] Unable to roll back
>> transaction
>>    [java] java.lang.IllegalStateException: Status is
>> STATUS_NO_TRANSACTION
>>    [java]     at
>> org.apache.geronimo.transaction.manager.TransactionImpl.rollback(TransactionImpl.java:438)
>>
>>    [java]     at
>> org.apache.geronimo.transaction.context.InheritableTransactionContext.rollbackAndThrow(InheritableTransactionContext.java:308)
>>
>>    [java]     at
>> org.apache.geronimo.transaction.context.InheritableTransactionContext.complete(InheritableTransactionContext.java:199)
>>
>>    [java]     at
>> org.apache.geronimo.transaction.context.InheritableTransactionContext.commit(InheritableTransactionContext.java:146)
>>
>>    [java]     at
>> org.apache.geronimo.transaction.context.GeronimoTransactionManager.commit(GeronimoTransactionManager.java:81)
>>
>>    [java]     at
>> org.ofbiz.entity.transaction.TransactionUtil.commit(TransactionUtil.java:179)
>>
>>    [java]     at
>> org.ofbiz.entity.transaction.TransactionUtil.commit(TransactionUtil.java:165)
>>
>>    [java]     at
>> org.ofbiz.widget.screen.ModelScreen.renderScreenString(ModelScreen.java:163)
>>
>>    [java]     at
>> org.ofbiz.widget.screen.ScreenRenderer.render(ScreenRenderer.java:105)
>>    [java]     at
>> org.ofbiz.widget.screen.ScreenRenderer.render(ScreenRenderer.java:90)
>>    [java]     at
>> org.ofbiz.widget.screen.ScreenWidgetViewHandler.render(ScreenWidgetViewHandler.java:78)
>>
>>    [java]     at
>> org.ofbiz.webapp.control.RequestHandler.renderView(RequestHandler.java:694)
>>
>>    [java]     at
>> org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:469)
>>
>>    [java]     at
>> org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:189)
>>    [java]     at
>> org.ofbiz.webapp.control.ControlServlet.doPost(ControlServlet.java:77)
>>    [java]     at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:615)
>>    [java]     at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
>>    [java]     at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
>>
>>    [java]     at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
>>
>>    [java]     at
>> org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:738)
>>
>>    [java]     at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
>>
>>    [java]     at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
>>
>>    [java]     at
>> org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:371)
>>    [java]     at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
>>
>>    [java]     at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
>>
>>    [java]     at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
>>
>>    [java]     at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
>>
>>    [java]     at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
>>
>>    [java]     at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
>>
>>    [java]     at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
>>
>>    [java]     at
>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541)
>>
>>    [java]     at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
>>
>>    [java]     at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
>>
>>    [java]     at
>> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
>>
>>    [java]     at
>> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
>>
>>    [java]     at
>> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
>>
>>    [java]     at
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
>>
>>    [java]     at java.lang.Thread.run(Thread.java:619)
>>    [java] 2009-06-18 18:12:32,782 (http-0.0.0.0-8080-Processor30)
>> [        ModelScreen.java:165:ERROR]
>>    [java] ---- exception report
>> ----------------------------------------------------------
>>
>>
>>
>> Any help would be really appreciated
>>
>> Best regards,
>> Flopa
>
>