[jira] [Commented] (OFBIZ-10592) OutOfMemory and stucked JobPoller issue

[Nicolas Malin (Jira)]

    [ https://issues.apache.org/jira/browse/OFBIZ-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648898#comment-16648898 ]

Giulio Speri commented on OFBIZ-10592:
--------------------------------------

After several profiling sessions (either on the remote production machine and on our local machines) we found that the results are basically the same (the only difference is in number, since on our local machines we cannot assign 20 GB of memory to the jvm heap, so the OOM appears earlier) .

We found that the service "autoDeleteAutoSaveShoppingList" is responsible for the allocation of the objects (that are mostly hash maps).

This service retrieves a list of shopping lists (with a findList) and for each list, checks if the maximum number of days to keep it (14) are passed by; if yes, get its shopping list items, run the service "removeShoppingListItem" and then "removeShoppingList", to delete the list.

Once found the service, we checked the dimension of the table "ShoppingList" : about 8.5M of records: with that number I can understand the OOM.

This situation led to a GC activity constant at 100%, an increasing objects allocation until the "death" of the application, with the service that have never gone over the findList method.

We then tried to reduce the dimensions of the table, keeping about 600K records.

We had a bit different profiling results: the number of recorded objects (hash maps) was always very high (about 30M), but the gc activity was no more constant to 100%, but it had a "saw tooth" shape and so had the memory.

The service kept on running for at least one hour (then we cut it off), but no shopping list was deleted.

The screenshots that I attach here are taken from this last profiling session.

They show the allocation tree of the biggest objects recorded (alloc_tree_600k_12102018), the number of the object recorded (recorded_object_600k_12102018) during the session and an overall view of the telemetries (telemetry_ovrl_600k_12102018).

 

Could this be only a "table dimension" problem or there could actually be a leak somewhere? 

 

!alloc_tree_600k_12102018.png!

 

 

 

> OutOfMemory and stucked JobPoller issue
> ---------------------------------------
>
>                 Key: OFBIZ-10592
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-10592
>             Project: OFBiz
>          Issue Type: Bug
>          Components: ALL APPLICATIONS
>    Affects Versions: Release Branch 13.07
>         Environment: Two instances Ofbiz installation on two machines, connected to an Apache HTTPD instance which acts as a proxy.
> Request to ofbiz instances are handled and load balanced.
> OFBiz version : 13.07.03 with Multitenant enabled
> OS: Ubuntu Linux 16.04 LTS
> RDBMS: MariaDB v10.1.24
>            Reporter: Giulio Speri
>            Assignee: Giulio Speri
>            Priority: Critical
>         Attachments: alloc_tree_600k_12102018.png, jvm_ofbiz1_profi_telem.png, jvm_prof_ofbiz1_telem2.png, ofbiz1_jvm_profil_nojobpoller.png, recorder_object_600k_12102018.png, telemetry_ovrl_600k_12102018.png
>
>
>  
> This installation is composed by two instances of OFBiz (v13.07.03), served via an Apache Tomcat webserver, along with a load balancer.
> The database server is MariaDB.
>  
> We had the first problems, about 3 weeks ago, when suddenly, the front1 (ofbiz instance 1), stopped serving web requests; front2, instead, was still working correctly.
>  
> Obviously we checked the log files, and we saw that async services were failing; the failure was accompanied by this error line:
>  
> *_Thread "AsyncAppender-async" java.lang.OutOfMemoryError: GC overhead limit exceeded_*
>  
> We analyzed the situation with our system specialists, and they told us that the application was highly stressing machine resources (cpu always at or near 100%, RAM usage rapidly increasing), until the jvm run out of memory.
> This "resource-high-consumption situation", occurred only when ofbiz1 instance was started with the JobPoller enabled; if the JobPoller was not enabled, ofbiz run with low resource usage. 
>  
> We then focused on the db, to check first of all the dimensions; the result was disconcerting; 45GB, mainly divided on four tables: SERVER_HIT (about 18 GB), VISIT (about 15 GB), ENTITY_SYNC_REMOVE (about 8 GB), VISITOR (about 2 GB).
> All the other tables had a size in the order of few MB each.
>  
> The first thing we did, was to clear all those tables, reducing considerably the db size.
> After the cleaning, we tried to start ofbiz1 again, with the JobPoller component enabled; this caused a lot of old scheduled/queued jobs, to execute.
> Except than for the start-up time, the resource usage of the machine, stabilized around normal to low values (cpu 1-10%).
> Ofbiz seemed to work (web request was served), but we noticed taht the JobPoller did not schedule or run jobs, anymore. 
> The number of job in "Pending" state in the JobSandbox entity was small (about 20); no Queued, no Failed, no jobs in other states.
> In addition to this, unfortunately, after few hours, jvm run out of memory again.
>  
> Our jvm has an heap maximum size of 20GB ( we have 32GB on the  machine), so it's not so small, I think.
> The next step we're going to do is set-up locally the application over the same production db to see what happens.
>  
> Now that I explained the situation, I am going to ask if, in your opinion/experience:
>  
> Could the JobPoller component be the root (and only) cause of the OutOfMemory of the jvm?
>  
> Could this issue be related to OFBIZ-5710?
>  
> Dumping and analyzing the heap of the jvm could help in some way to understand what or who fills the memory or is this operation a waste of time?
>  
> Is there something that we did not considered or missed during the whole process of problem analysis?
>  
>  
> I really thank you all for your attention and your help; any suggestion or advice would really be greatly appreciated.
>  
> Kind regards,
> Giulio

--
This message was sent by Atlassian JIRA
(v7.6.3#76005)