Extending multitenancy functionality added in svn commit: r1172989

[Justin-2]

One of my main aims has been to detach the services triggered by ant
targets on start up, in order to reduce the need to do server restarts on
the mt system for obvious reasons.
Now that I'm load testing I've discovered that the fact that the container
bound services to create/initialize new tenant
were tied to start up, hid memory issues.

To solve the "Exception in thread "RMI TCP Connection(idle)"
java.lang.OutOfMemoryError: Java heap space occurred." which tends to occur
when initializing (building db & loading the data) the 4th tenant I've been
looking for ways to insure that no references to delegators no longer in
active use exist thereby making GC possible & freeing up memory.

Alternatively I thought to look at classes who's state with specific regard
to the associated delegator is not important & maybe implementing a means
for such to be shared between delegators.

I've been looking at JobManager, JobPoller.destroyThreadPool(),
ServiceDispatcher, DispatchContext & the classloader to try come up with at
strategy of how memory with regards to delegators may be better managed.

I'm a bit overwhelmed at the moment & thought I might ask on the forum if
anyone has had any experience with this sort of thing before ... I've often
picked up a clue that helped me keep going here.

Thanks very much for your time.


Regards,
Justin

[Adrian Crum-3]

The references to delegators isn't the real issue - because they hold
very little data outside the data models. I would investigate to see if
multiple copies of the data models are being generated. If not, then
maybe the delegators aren't to blame.

Another thing to check is classes that hold references to delegator
objects instead of delegator names. There was some recent work to fix
those occurrences - maybe some areas were missed.

-Adrian

On 4/4/2012 5:24 PM, Justin wrote:

> One of my main aims has been to detach the services triggered by ant
> targets on start up, in order to reduce the need to do server restarts on
> the mt system for obvious reasons.
> Now that I'm load testing I've discovered that the fact that the container
> bound services to create/initialize new tenant
> were tied to start up, hid memory issues.
>
> To solve the "Exception in thread "RMI TCP Connection(idle)"
> java.lang.OutOfMemoryError: Java heap space occurred." which tends to occur
> when initializing (building db&  loading the data) the 4th tenant I've been
> looking for ways to insure that no references to delegators no longer in
> active use exist thereby making GC possible&  freeing up memory.
>
> Alternatively I thought to look at classes who's state with specific regard
> to the associated delegator is not important&  maybe implementing a means
> for such to be shared between delegators.
>
> I've been looking at JobManager, JobPoller.destroyThreadPool(),
> ServiceDispatcher, DispatchContext&  the classloader to try come up with at
> strategy of how memory with regards to delegators may be better managed.
>
> I'm a bit overwhelmed at the moment&  thought I might ask on the forum if
> anyone has had any experience with this sort of thing before ... I've often
> picked up a clue that helped me keep going here.
>
> Thanks very much for your time.
>
>
> Regards,
> Justin
>

[Justin-2]

*Thanks for your input Adrian.*

You're right it doesn't seem the delegator's are to blame after all.
I'm still trying to get my head around how the threading works, eg: the
cryptic comment on line 285 of org.ofbiz.entity.GenericDelegator.......
"//time to do some tricks with manual class loading that resolves circular
dependencies, like calling services..."

But at this point I have a hunch that the ServiceDispatcher & associated
classes are a more likely cause of the memory issues.

In the org.ofbiz.service.job.JobPoller there are properties set eg:
MAX_THREADS = 15; MAX_JOBS = 3
This together with the fact of there being a ServiceDispatcher for each new
tenant Delegator
Would lead to a situation where there are far more the 3 jobs running & a
lot more then 15 threads.

I certainly have a lot more reverse engineering to do before I can hope to
design an appropriate solution.

But do you think it might be feasible to have a singleton ServiceDispatcher
that was multitenant aware.

Would I in anyone's opinion, be barking up the wrong tree with such an
approach?

-Justin

On Wed, Apr 4, 2012 at 6:56 PM, Adrian Crum <
[hidden email]> wrote:

> The references to delegators isn't the real issue - because they hold very
> little data outside the data models. I would investigate to see if multiple
> copies of the data models are being generated. If not, then maybe the
> delegators aren't to blame.
>
> Another thing to check is classes that hold references to delegator
> objects instead of delegator names. There was some recent work to fix those
> occurrences - maybe some areas were missed.
>
> -Adrian
>
>
> On 4/4/2012 5:24 PM, Justin wrote:
>
>> One of my main aims has been to detach the services triggered by ant
>> targets on start up, in order to reduce the need to do server restarts on
>> the mt system for obvious reasons.
>> Now that I'm load testing I've discovered that the fact that the container
>> bound services to create/initialize new tenant
>> were tied to start up, hid memory issues.
>>
>> To solve the "Exception in thread "RMI TCP Connection(idle)"
>> java.lang.OutOfMemoryError: Java heap space occurred." which tends to
>> occur
>> when initializing (building db&  loading the data) the 4th tenant I've
>> been
>>
>> looking for ways to insure that no references to delegators no longer in
>> active use exist thereby making GC possible&  freeing up memory.
>>
>>
>> Alternatively I thought to look at classes who's state with specific
>> regard
>> to the associated delegator is not important&  maybe implementing a means
>>
>> for such to be shared between delegators.
>>
>> I've been looking at JobManager, JobPoller.destroyThreadPool(),
>> ServiceDispatcher, DispatchContext&  the classloader to try come up with
>> at
>>
>> strategy of how memory with regards to delegators may be better managed.
>>
>> I'm a bit overwhelmed at the moment&  thought I might ask on the forum if
>>
>> anyone has had any experience with this sort of thing before ... I've
>> often
>> picked up a clue that helped me keep going here.
>>
>> Thanks very much for your time.
>>
>>
>> Regards,
>> Justin
>>
>>