OFBiz › OFBiz - User

Ofbiz Demo TrunK Slow

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

14 messages Options

BJ Freeman

Ofbiz Demo TrunK Slow

did a
http://demo-trunk.ofbiz.apache.org/cmssite/cms/APACHE_OFBIZ_HTML
and got
Service Temporarily Unavailable
checked
http://monitoring.apache.org/status/

HTTP - Ofbiz Demo Trunk

WARNING 2010-12-06 21:08:58 0d 0h 6m 3s 6/10 HTTP WARNING: -
HTTP/1.1 200 OK - 13.380 second response time

=========================
BJ Freeman
Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=52>
Specialtymarket.com <http://www.specialtymarket.com/>
Systems Integrator-- Glad to Assist

Chat Y! messenger: bjfr33man

Jacques Le Roux

Re: Ofbiz Demo TrunK Slow

Administrator

In reply to this post by BJ Freeman

BJ Freeman wrote:

> OK was working on a patch to deactivate the full set generation.
> you beat me to it.
> I don't use the offical Demo trunk so expect the search engine have
> picked up the URLs
>
> This morning I rebooted my workstation.
> I do a restore of the Firefox pages.
> I have a page for demo-trunk and demo-stable.
> each page has about 5 URL that come up at once.
> It seems the demo-trunk buckled under this.
> so the problem shown by help is not the cause as such.
>
> BTW how do I see the threads like you shown. did not see the threads in
> the thread list in webtools. I have had about 7,000 hits in 30 days on
> my demo-trunk. Mostly Search engines

If you speak about org.apache.tomcat.util.threads.ThreadWithAttributes below, it comes from a heap dump I took (or was generated
automatically, I don't remember: the scripts use -XX:+HeapDumpOnOutOfMemoryError)

Jacques

> =========================
> BJ Freeman
> Strategic Power Office with Supplier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=52>
> Specialtymarket.com <http://www.specialtymarket.com/>
> Systems Integrator-- Glad to Assist
>
> Chat Y! messenger: bjfr33man
> Jacques Le Roux sent the following on 12/9/2010 10:04 AM:
>
>
>> At r1044053, I have temporarily deactivated the online help. It was not
>> working and slowing down too much the demo server...
>>
>> Jacques
>>
>> From: "Jacques Le Roux" <[hidden email]>
>>> It's interesting but which release.revision are you using? Also
>>> consider that there are maybe more users connected to the demo than to
>>> your server. For instance we can have
>>>
>>> 198 instances of
>>> "org.apache.tomcat.util.threads.ThreadWithAttributes", loaded by
>>> "java.net.URLClassLoader @ 0x7fb4b8f59220" occupy 286 111 256 (45,01%)
>>> bytes.
>>>
>>> I have tried the link and the help at large locally this afternoon
>>> (with last revision but a Postgres DB not cleared for some days) and
>>> it seems the process never end
>>> Same on demo were the trunk process was running near 100% all the
>>> afternoon. I just stopped an restarted it...
>>>
>>> This is why we got so much:
>>> *** ASF Nagios ***
>>> Notification Type: PROBLEM
>>> Host: ofbiz-vm.apache.org
>>> Address: 140.211.11.41
>>> Service: HTTP - Ofbiz Demo Trunk
>>> State: CRITICAL
>>> Info: CRITICAL - Socket timeout after 16 seconds
>>>
>>> Date/Time: Wed Dec 8 19:48:21 UTC 2010
>>>
>>> this afternoon...
>>>
>>> *Demo users: please don't use the online help before this problems is
>>> solved*
>>>
>>> Thanks
>>>
>>> Jacques
>>>
>>> From: "BJ Freeman" <[hidden email]>
>>>> it is interesting that I have more instances than on
>>>> ofbiz-vm.apache.org
>>>> including two trunk instances and can successfully run the full docbook
>>>>
>>>>
>>>>
>>>> =========================
>>>> BJ Freeman
>>>> Strategic Power Office with Supplier Automation
>>>> <http://www.businessesnetwork.com/automation/viewforum.php?f=52>
>>>> Specialtymarket.com <http://www.specialtymarket.com/>
>>>> Systems Integrator-- Glad to Assist
>>>>
>>>> Chat Y! messenger: bjfr33man
>>>> BJ Freeman sent the following on 12/8/2010 4:55 AM:
>>>>
>>>>> This is the top level ofbiz docbook link.
>>>>> yes it takes a lot of resources.
>>>>> it is doing a list of xls's for all two books in the set.
>>>>> I am focused on going through the docbooks.
>>>>> one is to split the tech and user docs into different sets.
>>>>>
>>>>>
>>>>> Jacques Le Roux sent the following on 12/8/2010 1:13 AM:
>>>>>
>>>>>
>>>>> =========================
>>>>> BJ Freeman
>>>>> Strategic Power Office with Supplier Automation
>>>>> <http://www.businessesnetwork.com/automation/viewforum.php?f=52>
>>>>> Specialtymarket.com <http://www.specialtymarket.com/>
>>>>> Systems Integrator-- Glad to Assist
>>>>>
>>>>> Chat Y! messenger: bjfr33man
>>>>>
>>>>>> OK, When I use this URL locally I get this
>>>>>>
>>>>>> 2010-12-08 10:02:15,609 (http-0.0.0.0-28080-1)
>>>>>> [ControlEventListener.java:64 :INFO ] Creating session:
>>>>>> 65D15F115C41C7719E22FEBE11BFE2F9.jvm1
>>>>>> 2010-12-08 10:02:15,609 (http-0.0.0.0-28080-1) [
>>>>>> ControlServlet.java:147:INFO ] [[[APACHE_OFBIZ_HTML] Request Begun,
>>>>>> encoding=[UTF-8]- total:0.0,since last(Begin):0.0]]
>>>>>> 2010-12-08 10:02:15,609 (http-0.0.0.0-28080-1) [
>>>>>> VisitHandler.java:232:INFO ] Found visitorId [10000] in cookie
>>>>>> 2010-12-08 10:02:15,625 (http-0.0.0.0-28080-1) [
>>>>>> ConfigXMLReader.java:120:INFO ] controller loaded: 0.0s, 1 requests, 0
>>>>>> views in jndi:/0.0.0.0/cmssite/WEB-INF/controller.xml
>>>>>> 2010-12-08 10:02:15,625 (http-0.0.0.0-28080-1) [
>>>>>> ConfigXMLReader.java:120:INFO ] controller loaded: 0.0s, 29
>>>>>> requests, 17
>>>>>> views in
>>>>>> file:/D:/workspace/ofbizDev/framework/common/webcommon/WEB-INF/common-cont
>>>>>>
>>>>>>
>>>>>> roller.xml
>>>>>> 2010-12-08 10:02:15,625 (http-0.0.0.0-28080-1) [
>>>>>> CmsEvents.java:145:INFO
>>>>>> ] Path INFO for Alias: APACHE_OFBIZ_HTML
>>>>>>
>>>>>> And then nothing in log but the Java process eats as much CPU
>>>>>> ressources
>>>>>> as it can. Please don't use this on demo!
>>>>>>
>>>>>> Jacques
>>>>>>
>>>>>> From: "Jacques Le Roux" <[hidden email]>
>>>>>>> This URL does not seem to be make much sense. How did you come to it?
>>>>>>>
>>>>>>> Jacques
>>>>>>>
>>>>>>> From: "BJ Freeman" <[hidden email]>
>>>>>>>> seems I blew it up again doing:
>>>>>>>> http://demo-trunk.ofbiz.apache.org/cmssite/cms/APACHE_OFBIZ_HTML
>>>>>>>>
>>>>>>>> HTTP - Ofbiz Demo Trunk
>>>>>>>>
>>>>>>>> This service has 1 comment associated with it This service is
>>>>>>>> flapping between states
>>>>>>>> CRITICAL 2010-12-08 02:01:58 0d 0h 33m 3s 10/10 CRITICAL - Socket
>>>>>>>> timeout after 16 seconds
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> =========================
>>>>>>>> BJ Freeman
>>>>>>>> Strategic Power Office with Supplier Automation
>>>>>>>> <http://www.businessesnetwork.com/automation/viewforum.php?f=52>
>>>>>>>> Specialtymarket.com <http://www.specialtymarket.com/>
>>>>>>>> Systems Integrator-- Glad to Assist
>>>>>>>>
>>>>>>>> Chat Y! messenger: bjfr33man
>>>>>>>> Jacques Le Roux sent the following on 12/7/2010 1:51 PM:
>>>>>>>>> We have already 2 memory dumps and the suspects so far are
>>>>>>>>>
>>>>>>>>> 1)
>>>>>>>>> 110 542 instances of
>>>>>>>>> "org.apache.derby.impl.jdbc.EmbedConnection40",
>>>>>>>>> loaded by "java.net.URLClassLoader @ 0x7f3681c111f0" occupy 415
>>>>>>>>> 680 440 (79,89%) bytes. These instances are referenced from one
>>>>>>>>> instance
>>>>>>>>> of "java.util.WeakHashMap$Entry[]", loaded by "<system
>>>>>>>>> class loader>"
>>>>>>>>> More at https://issues.apache.org/jira/browse/OFBIZ-4043
>>>>>>>>>
>>>>>>>>> 2)
>>>>>>>>> 198 instances of
>>>>>>>>> "org.apache.tomcat.util.threads.ThreadWithAttributes",
>>>>>>>>> loaded by "java.net.URLClassLoader @ 0x7fb4b8f59220" occupy
>>>>>>>>> 286 111 256 (45,01%) bytes.
>>>>>>>>>
>>>>>>>>> Biggest instances:
>>>>>>>>>
>>>>>>>>> a.. org.apache.tomcat.util.threads.ThreadWithAttributes @
>>>>>>>>> 0x7fb4c47e7818
>>>>>>>>> TP-Processor90 - 8 558 720 (1,35%) bytes.
>>>>>>>>> b.. org.apache.tomcat.util.threads.ThreadWithAttributes @
>>>>>>>>> 0x7fb4c35bb570
>>>>>>>>> TP-Processor70 - 8 197 640 (1,29%) bytes.
>>>>>>>>> c.. org.apache.tomcat.util.threads.ThreadWithAttributes @
>>>>>>>>> 0x7fb4c9f1b320
>>>>>>>>> TP-Processor122 - 8 195 856 (1,29%) bytes.
>>>>>>>>> d.. org.apache.tomcat.util.threads.ThreadWithAttributes @
>>>>>>>>> 0x7fb4c0719340
>>>>>>>>> TP-Processor27 - 7 108 272 (1,12%) bytes.
>>>>>>>>> e.. org.apache.tomcat.util.threads.ThreadWithAttributes @
>>>>>>>>> 0x7fb4ca6fd200
>>>>>>>>> TP-Processor129 - 6 912 832 (1,09%) bytes.
>>>>>>>>> f.. org.apache.tomcat.util.threads.ThreadWithAttributes @
>>>>>>>>> 0x7fb4cbfe2f60
>>>>>>>>> TP-Processor152 - 6 639 528 (1,04%) bytes.
>>>>>>>>> g.. org.apache.tomcat.util.threads.ThreadWithAttributes @
>>>>>>>>> 0x7fb4c80a7b98
>>>>>>>>> TP-Processor114 - 6 494 872 (1,02%) bytes.
>>>>>>>>> h.. org.apache.tomcat.util.threads.ThreadWithAttributes @
>>>>>>>>> 0x7fb4ca8bf9a8
>>>>>>>>> TP-Processor133 - 6 478 984 (1,02%) bytes.
>>>>>>>>>
>>>>>>>>> For the later it seems we simply miss enough memory to handle the
>>>>>>>>> requests (there are no threads taking a huge part, only 1,35% max)
>>>>>>>>>
>>>>>>>>> Most of the time one of the demo instances fails because there
>>>>>>>>> is not
>>>>>>>>> enough swap space. But this symptom means something is going wrong
>>>>>>>>> anyway. Because we have 2.5GB of memory and 1.5GB of swap space.
>>>>>>>>> The
>>>>>>>>> instances use
>>>>>>>>> branch9: -Xms128M -Xmx512M (default MaxPermSize is 64m for server)
>>>>>>>>> trunk: -Xms128M -Xmx768M -XX:MaxPermSize=256m (I have also tried
>>>>>>>>> with
>>>>>>>>> -XX:MaxPermSize=192m but it's the same)
>>>>>>>>>
>>>>>>>>> I suspect something from users (or bots) and the infra team
>>>>>>>>> suggested to
>>>>>>>>> use mod_log_forensic. I have not looked at it yet
>>>>>>>>>
>>>>>>>>> Jacques
>>>>>>>>>
>>>>>>>>> From: "BJ Freeman" <[hidden email]>
>>>>>>>>>> both my demos are working fine on Centos5
>>>>>>>>>> one other thing I noticed is that when the one failed the other
>>>>>>>>>> server
>>>>>>>>>> was up but there was no fail over to the working server.
>>>>>>>>>> Will look at the dumps and see if I can see anything and track
>>>>>>>>>> on my
>>>>>>>>>> demos.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> =========================
>>>>>>>>>> BJ Freeman
>>>>>>>>>> Strategic Power Office with Supplier Automation
>>>>>>>>>> <http://www.businessesnetwork.com/automation/viewforum.php?f=52>
>>>>>>>>>> Specialtymarket.com <http://www.specialtymarket.com/>
>>>>>>>>>> Systems Integrator-- Glad to Assist
>>>>>>>>>>
>>>>>>>>>> Chat Y! messenger: bjfr33man
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Jacques Le Roux sent the following on 12/7/2010 1:59 AM:
>>>>>>>>>>> I'm currently struggling to get both demo working smoothly.
>>>>>>>>>>> Though we
>>>>>>>>>>> did not change the running configuration, since the last OS
>>>>>>>>>>> update we
>>>>>>>>>>> had a lot of problems.
>>>>>>>>>>> I have changed the running configuration by minoring the memory
>>>>>>>>>>> used but
>>>>>>>>>>> there are still issues. I have already 2 heap dumps which show
>>>>>>>>>>> some
>>>>>>>>>>> leaks suspects. But I have not enough time to look into details
>>>>>>>>>>>
>>>>>>>>>>> Jacques
>>>>>>>>>>>
>>>>>>>>>>> From: "BJ Freeman" <[hidden email]>
>>>>>>>>>>>> did a
>>>>>>>>>>>> http://demo-trunk.ofbiz.apache.org/cmssite/cms/APACHE_OFBIZ_HTML
>>>>>>>>>>>> and got
>>>>>>>>>>>> Service Temporarily Unavailable
>>>>>>>>>>>> checked
>>>>>>>>>>>> http://monitoring.apache.org/status/
>>>>>>>>>>>>
>>>>>>>>>>>> HTTP - Ofbiz Demo Trunk
>>>>>>>>>>>>
>>>>>>>>>>>> WARNING 2010-12-06 21:08:58 0d 0h 6m 3s 6/10 HTTP WARNING: -
>>>>>>>>>>>> HTTP/1.1
>>>>>>>>>>>> 200 OK - 13.380 second response time
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> =========================
>>>>>>>>>>>> BJ Freeman
>>>>>>>>>>>> Strategic Power Office with Supplier Automation
>>>>>>>>>>>> <http://www.businessesnetwork.com/automation/viewforum.php?f=52>
>>>>>>>>>>>> Specialtymarket.com <http://www.specialtymarket.com/>
>>>>>>>>>>>> Systems Integrator-- Glad to Assist
>>>>>>>>>>>>
>>>>>>>>>>>> Chat Y! messenger: bjfr33man