Re: svn commit: r1141288 - /ofbiz/trunk/framework/base/src/org/ofbiz/base/concurrent/ExecutionPool.java

[Adam Heath-2]

On 06/29/2011 05:12 PM, [hidden email] wrote:
> Author: doogie
> Date: Wed Jun 29 22:12:31 2011
> New Revision: 1141288
>
> URL: http://svn.apache.org/viewvc?rev=1141288&view=rev
> Log:
> FEATURE: Add a global executor, to be used during config type parsing.

I'm just waiting a little bit before checking in the next set, I'm
letting buildbot do it's work.

The next commit batch contains:

commit fac2be7ba22982fb2b6853764f2ec414c5885a52
Author: Adam Heath <[hidden email]>
Date:   Mon Jun 27 14:37:19 2011 -0500

     FEATURE: Load each configured catalina web context in parallel.

 
framework/catalina/src/org/ofbiz/catalina/container/CatalinaContainer.java
|   51 ++++++++++++++++++++++++++++--------
  1 files changed, 40 insertions(+), 11 deletions(-)

commit aea01ee1e73db9030ed644293e20f7ae6247daa7
Author: Adam Heath <[hidden email]>
Date:   Mon Jun 27 15:09:01 2011 -0500

     FEATURE: Parsing of secas.xml files are now done in parallel
using the
     global thread pool.

  framework/service/src/org/ofbiz/service/eca/ServiceEcaUtil.java |
19 +++++++++++++++----
  1 files changed, 15 insertions(+), 4 deletions(-)

commit da8f0f3d3d18d8cc4a3c5a651f42676c5be41ec7
Author: Adam Heath <[hidden email]>
Date:   Mon Jun 27 14:23:00 2011 -0500

     FEATURE: Parsing of eecas.xml files are now done in parallel,
using the
     global thread pool.

  framework/entityext/src/org/ofbiz/entityext/eca/EntityEcaUtil.java |
   19 +++++++++++++++----
  1 files changed, 15 insertions(+), 4 deletions(-)

commit 817086f9f0a907a5e7500fddcd422cfe88ffb7dd
Author: Adam Heath <[hidden email]>
Date:   Mon Jun 27 13:10:19 2011 -0500

     FEATURE: Parsing of service.xml files is now done in parallel,
using the
     global thread pool.

  framework/service/src/org/ofbiz/service/DispatchContext.java |   23
++++++++++++++++++-----
  1 files changed, 18 insertions(+), 5 deletions(-)

commit e2b7b294960dd2c2c07e2868c3bb77d42f545c3b
Author: Adam Heath <[hidden email]>
Date:   Mon Jun 27 17:14:13 2011 -0500

     FEATURE: Do GenericHelper initialization in parallel.

  framework/entity/src/org/ofbiz/entity/GenericDelegator.java |   16
+++++++++++++++-
  1 files changed, 15 insertions(+), 1 deletions(-)

>
> Modified:
>      ofbiz/trunk/framework/base/src/org/ofbiz/base/concurrent/ExecutionPool.java
>
> Modified: ofbiz/trunk/framework/base/src/org/ofbiz/base/concurrent/ExecutionPool.java
> URL: http://svn.apache.org/viewvc/ofbiz/trunk/framework/base/src/org/ofbiz/base/concurrent/ExecutionPool.java?rev=1141288&r1=1141287&r2=1141288&view=diff
> ==============================================================================
> --- ofbiz/trunk/framework/base/src/org/ofbiz/base/concurrent/ExecutionPool.java (original)
> +++ ofbiz/trunk/framework/base/src/org/ofbiz/base/concurrent/ExecutionPool.java Wed Jun 29 22:12:31 2011
> @@ -39,6 +39,7 @@ import org.ofbiz.base.util.Debug;
>   @SourceMonitored
>   public final class ExecutionPool {
>       public static final String module = ExecutionPool.class.getName();
> +    public static final ScheduledExecutorService GLOBAL_EXECUTOR = getExecutor(null, "ofbiz-config", -1, true);
>
>       protected static class ExecutionPoolThreadFactory implements ThreadFactory {
>           private final ThreadGroup group;
>
>

[Adam Heath-2]

On 06/29/2011 05:22 PM, Adam Heath wrote:

> On 06/29/2011 05:12 PM, [hidden email] wrote:
>> Author: doogie
>> Date: Wed Jun 29 22:12:31 2011
>> New Revision: 1141288
>>
>> URL: http://svn.apache.org/viewvc?rev=1141288&view=rev
>> Log:
>> FEATURE: Add a global executor, to be used during config type parsing.
>
> I'm just waiting a little bit before checking in the next set, I'm
> letting buildbot do it's work.
>
> The next commit batch contains:

And now I have committed them.

I do have one more parallel commit; this time, Seed data parsing in
parallelized.  However, it's rather convoluted, and I want to take
time to fix it.  Below are the complexities:

EntitySaxReader has support for transactions.  If the constructor is
used that allows you to set the timeout, and you set it to 0, then no
new transaction will be created.  This means that whatever existing
transaction is active will be used(this is an important thing to
remember).

Intitially, I tried to have a background thread that would write to
the delegator.  That is not valid, however, when we don't create our
own transaction.  The writing has to be done in the original, parent
thread.

So, that means that the parser.parse() call has to be in the
background.  Then, when end() is called, and either a singleton
create, or a batch of values is ready, it get's "posted" thru a
synchronous queue to the foreground thread.

Then, I noted that end() has support for transformations of the
xml(generally thru freemarker).  It creates a new, internal
EntitySaxReader to do this.  That then breaks again, because the
thread that creates the internal reader is running in the background,
and the transaction that gets used is not the same as the outer one
done by the initial caller.

Additionally, since the end() that creates the new internal reader is
already running the background, it makes no sense for the new reader
to also parse in the background.  Just the queue that it posts to has
to be sent back to the original parent thread.

I do have all of the above problems solved, and the test cases all run
happily.  However, I'd like to do some small refactors of the code
first, before committing the work as a whole.  If you look at the
other commits I did today, I tweaked each class with several prep type
commits, so that the final parallelization feature was rather small
and easier to  understand.