Improving OFBiz extensibility and distribution

[Mathieu Lirzin]

Hello,

The plugin mechanism has various issues that make distributing and
extending OFBiz inconvenient and complex.  Here are some examples of the
problems I identified:

1. Binary distribution of OFBiz needs to include both the Jar and the
   ‘ofbiz.home’ directory which make deployment more complex

2. Gradle multi-project feature is overly complex (try to understand
   “build.gradle”...) and proves to be limited and brittle in practice
   (ex. adding an extra Gradle plugin in a subproject can easily break
   the build with obscure errors)

3. The plugin mechanism is hardly coupled with Gradle which breaks its
   API regularly. This means that plugins cannot choose their build
   system (Maven, Leinigen, SBT) and cannot expect their plugins to work
   with various version of OFBiz reliably.

4. There is no flexibility to install plugins outside of the ‘plugins’
   directory of $OFBIZ_HOME.

5. Plugins and Framework bytecode is distributed in the same JAR when we
   would want instead to independently compile and distribute them.


Here is a *big* change that I am considering for OFBiz to fixes those
issues by leveraging the Java platform which already provides everything
that we need to fix those issues:

 - Distributing OFBiz in a Jar with its XML files inside to fix #1 would
   allow developpers to extend the framework without compiling it which
   removes the hard dependency on Gradle which fixes #2 and #3 and make
   the build process snappier.

 - the classpath can be used to compose an arbitrary list of plugins
   from various location which fixes #4.

 - The ServiceLoader class [1] is a standard and stable extensibility
   mechanism based on JAR metadata which can be used to scan the
   available OFBiz components available across multiple JARs that can
   replace Gradle complex component bundling to fix #5.

The biggest challenge with this solution is that we need to do a huge
work of adapting OFBiz to to use location independant resources [3] when
loading XML and GroovyScript files.

However what this effort would would bring is a convenient and decoupled
way to extend OFBiz by simply declaring a dependency to the framework
and dependant reusable plugins from the project plugin build tool like
that:

    // myApplicationProjectPlugin/build.gradle
    dependencies {
        implementation "org.apache.ofbiz:ofbiz-framework:18.12"
        implementation "org.example:reusable-ofbiz-plugin:0.8"
        ...
    }

Would people welcome such change?

Thanks.

[1] https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html
[2] https://docs.oracle.com/javase/8/docs/technotes/guides/lang/resources.html

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

[Pierre Smits-3]

Great initiative, Mathieu!

One of the things I found annoying (to say the least) since day 1 (about 10
years ago) of using OFBiz (and improving it), was the aspect that I was to
get everything even though I did (and do) not need it for my business
purposes (1). such as

   - various 3rd party Fintech integrations included in the accounting
   component
   - the humanres  and manufacturing component
   - the ZipSales & TaxWare integrations included in the order component
   - various 3td party Logistic integrations included in the product
   component
   - the various components (and applications) in the plugin repo (such as
   ebay, ebaystore, example, exampleext, multiflex, passport, pricat, solr and
   webpos).

And I suspect that this annoyance is felt by many of the OFBiz adopters,
and inherently felt by those charged with managing, maintaining and
improving the implemented code base (as opposed to those committing to the
OFBiz repos).  IMO, we should adopt the approach as implemented by the
Apache Commons project [2] (where each component has its own repo), instead
of forcing adopters and developers to accept everything in one repo (or 2
when having the need for something from the plugins repo). Over the years I
have advocated to have more disentanglement (leading to more choice for
adopters, and more focus for developers/contributors). Unfortunately it
didn't attract buy-in by committers/PMC Members.

Hopefully your initiative will lead to a more sane vision to what is
offered by the project, and how adopters (and developers/contributors)
benefit from more options and focus from the project.

[1] I differentiate here between what I use for my business needs and what
I work on to improve the user experience of

   1. my customers
   2. (the adopters of) OFBiz


Best regards,

Pierre Smits

*Apache Trafodion <https://trafodion.apache.org>, Vice President*
*Apache Directory <https://directory.apache.org>, PMC Member*
Apache Incubator <https://incubator.apache.org>, committer
*Apache OFBiz <https://ofbiz.apache.org>, contributor (without privileges)
since 2008*
Apache Steve <https://steve.apache.org>, committer


On Thu, Aug 22, 2019 at 10:53 AM Mathieu Lirzin <[hidden email]>
wrote:

> Hello,
>
> The plugin mechanism has various issues that make distributing and
> extending OFBiz inconvenient and complex.  Here are some examples of the
> problems I identified:
>
> 1. Binary distribution of OFBiz needs to include both the Jar and the
>    ‘ofbiz.home’ directory which make deployment more complex
>
> 2. Gradle multi-project feature is overly complex (try to understand
>    “build.gradle”...) and proves to be limited and brittle in practice
>    (ex. adding an extra Gradle plugin in a subproject can easily break
>    the build with obscure errors)
>
> 3. The plugin mechanism is hardly coupled with Gradle which breaks its
>    API regularly. This means that plugins cannot choose their build
>    system (Maven, Leinigen, SBT) and cannot expect their plugins to work
>    with various version of OFBiz reliably.
>
> 4. There is no flexibility to install plugins outside of the ‘plugins’
>    directory of $OFBIZ_HOME.
>
> 5. Plugins and Framework bytecode is distributed in the same JAR when we
>    would want instead to independently compile and distribute them.
>
>
> Here is a *big* change that I am considering for OFBiz to fixes those
> issues by leveraging the Java platform which already provides everything
> that we need to fix those issues:
>
>  - Distributing OFBiz in a Jar with its XML files inside to fix #1 would
>    allow developpers to extend the framework without compiling it which
>    removes the hard dependency on Gradle which fixes #2 and #3 and make
>    the build process snappier.
>
>  - the classpath can be used to compose an arbitrary list of plugins
>    from various location which fixes #4.
>
>  - The ServiceLoader class [1] is a standard and stable extensibility
>    mechanism based on JAR metadata which can be used to scan the
>    available OFBiz components available across multiple JARs that can
>    replace Gradle complex component bundling to fix #5.
>
> The biggest challenge with this solution is that we need to do a huge
> work of adapting OFBiz to to use location independant resources [3] when
> loading XML and GroovyScript files.
>
> However what this effort would would bring is a convenient and decoupled
> way to extend OFBiz by simply declaring a dependency to the framework
> and dependant reusable plugins from the project plugin build tool like
> that:
>
>     // myApplicationProjectPlugin/build.gradle
>     dependencies {
>         implementation "org.apache.ofbiz:ofbiz-framework:18.12"
>         implementation "org.example:reusable-ofbiz-plugin:0.8"
>         ...
>     }
>
> Would people welcome such change?
>
> Thanks.
>
> [1] https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html
> [2]
> https://docs.oracle.com/javase/8/docs/technotes/guides/lang/resources.html
>
> --
> Mathieu Lirzin
> GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37
>

[taher]

We have done many discussions in the past about publishing OFBiz as
deployable JAR and then the plugin mechanism would work as you
suggested and you won't need the build system anymore. I remember
joining our friend Adrian Crum before he passed away in an attempt to
do that but which did not bear fruit.

So, I am in agreement in theory with the concept, but I think
realistically this makes the whole "plugin" thing a secondary topic,
and the main topic is how to rewrite the core OFBiz framework as an
independent JAR with perhaps a single comprehensive webapp, and then
make the components deployable (possibly even at runtime) by having
this truly pluggable architecture. If you look at the commit history
when gradle was introduced, you will notice that in the past there was
a little top-level jar created by the start component and then every
component would hold its own jars and sources which are then compiled
and linked through the old build system (ant) and included through the
custom class loader. So it was a mix between ant + class loader to get
things going but was still suffering from being one big monolithic
thing.

So IMHO, perhaps a good approach would have the following preliminary
work even before starting to worry about the plugin architecture:

- Remove tomcat as an integral container and instead make the web
server independent of the framework and vice versa
- Delete the entire "container" architecture altogether and just keep
thing running in the core
- Make everything as a single webapp by default with possible opt-in
other webapps
- Customize the classloader to allow extending the classpath and
loading other components.
- Finally, the component concept itself should be redesigned such that
every component _is_ an outputted jar file.

One thing though, I think the concept of the subproject in gradle
would help, not hurt moving towards that goal. It's just that
currently, the implementation is not ideal and requires improvements.
I don't think ditching the entire subproject functionality is very
useful though.

Needless to say, this is probably a _lot_ of work. But a good
direction to have perhaps.


On Thu, Aug 22, 2019 at 11:53 AM Mathieu Lirzin
<[hidden email]> wrote:

>
> Hello,
>
> The plugin mechanism has various issues that make distributing and
> extending OFBiz inconvenient and complex.  Here are some examples of the
> problems I identified:
>
> 1. Binary distribution of OFBiz needs to include both the Jar and the
>    ‘ofbiz.home’ directory which make deployment more complex
>
> 2. Gradle multi-project feature is overly complex (try to understand
>    “build.gradle”...) and proves to be limited and brittle in practice
>    (ex. adding an extra Gradle plugin in a subproject can easily break
>    the build with obscure errors)
>
> 3. The plugin mechanism is hardly coupled with Gradle which breaks its
>    API regularly. This means that plugins cannot choose their build
>    system (Maven, Leinigen, SBT) and cannot expect their plugins to work
>    with various version of OFBiz reliably.
>
> 4. There is no flexibility to install plugins outside of the ‘plugins’
>    directory of $OFBIZ_HOME.
>
> 5. Plugins and Framework bytecode is distributed in the same JAR when we
>    would want instead to independently compile and distribute them.
>
>
> Here is a *big* change that I am considering for OFBiz to fixes those
> issues by leveraging the Java platform which already provides everything
> that we need to fix those issues:
>
>  - Distributing OFBiz in a Jar with its XML files inside to fix #1 would
>    allow developpers to extend the framework without compiling it which
>    removes the hard dependency on Gradle which fixes #2 and #3 and make
>    the build process snappier.
>
>  - the classpath can be used to compose an arbitrary list of plugins
>    from various location which fixes #4.
>
>  - The ServiceLoader class [1] is a standard and stable extensibility
>    mechanism based on JAR metadata which can be used to scan the
>    available OFBiz components available across multiple JARs that can
>    replace Gradle complex component bundling to fix #5.
>
> The biggest challenge with this solution is that we need to do a huge
> work of adapting OFBiz to to use location independant resources [3] when
> loading XML and GroovyScript files.
>
> However what this effort would would bring is a convenient and decoupled
> way to extend OFBiz by simply declaring a dependency to the framework
> and dependant reusable plugins from the project plugin build tool like
> that:
>
>     // myApplicationProjectPlugin/build.gradle
>     dependencies {
>         implementation "org.apache.ofbiz:ofbiz-framework:18.12"
>         implementation "org.example:reusable-ofbiz-plugin:0.8"
>         ...
>     }
>
> Would people welcome such change?
>
> Thanks.
>
> [1] https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html
> [2] https://docs.oracle.com/javase/8/docs/technotes/guides/lang/resources.html
>
> --
> Mathieu Lirzin
> GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

[Mathieu Lirzin]

Hello Pierre,

Pierre Smits <[hidden email]> writes:

> Great initiative, Mathieu!

Thanks the your enthusiasm. :-)

> IMO, we should adopt the approach as implemented by the Apache Commons
> project [2] (where each component has its own repo), instead of
> forcing adopters and developers to accept everything in one repo (or 2
> when having the need for something from the plugins repo). Over the
> years I have advocated to have more disentanglement (leading to more
> choice for adopters, and more focus for
> developers/contributors). Unfortunately it didn't attract buy-in by
> committers/PMC Members.
>
> Hopefully your initiative will lead to a more sane vision to what is
> offered by the project, and how adopters (and developers/contributors)
> benefit from more options and focus from the project.

I haven't though much about it but I confirm the disentanglement topic,
but I agree that providing more flexible ways to extend OFBiz would help
achieve such disentanglement.

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

[Mathieu Lirzin]

Hello Taher,

Taher Alkhateeb <[hidden email]> writes:

> We have done many discussions in the past about publishing OFBiz as
> deployable JAR and then the plugin mechanism would work as you
> suggested and you won't need the build system anymore. I remember
> joining our friend Adrian Crum before he passed away in an attempt to
> do that but which did not bear fruit.

Thanks for reminding that past effort,

> So, I am in agreement in theory with the concept, but I think
> realistically this makes the whole "plugin" thing a secondary topic,
> and the main topic is how to rewrite the core OFBiz framework as an
> independent JAR with perhaps a single comprehensive webapp, and then
> make the components deployable (possibly even at runtime) by having
> this truly pluggable architecture. If you look at the commit history
> when gradle was introduced, you will notice that in the past there was
> a little top-level jar created by the start component and then every
> component would hold its own jars and sources which are then compiled
> and linked through the old build system (ant) and included through the
> custom class loader. So it was a mix between ant + class loader to get
> things going but was still suffering from being one big monolithic
> thing.

I have used the old ant build in one of our customer project at Néréide
and noticed the separation in multiple jars, but I didn't investigate
the internals.

> So IMHO, perhaps a good approach would have the following preliminary
> work even before starting to worry about the plugin architecture:
>
> - Remove tomcat as an integral container and instead make the web
> server independent of the framework and vice versa
> - Delete the entire "container" architecture altogether and just keep
> thing running in the core
> - Make everything as a single webapp by default with possible opt-in
> other webapps
> - Customize the classloader to allow extending the classpath and
> loading other components.
> - Finally, the component concept itself should be redesigned such that
> every component _is_ an outputted jar file.

Maybe I am overlooking something but this seems like an interesting but
different objective to me. :-)

> One thing though, I think the concept of the subproject in gradle
> would help, not hurt moving towards that goal. It's just that
> currently, the implementation is not ideal and requires improvements.
> I don't think ditching the entire subproject functionality is very
> useful though.

Sure I am maybe throwing the baby with the bathwater. Gradle
multi-projet builds are indeed useful to solve the issue of coordinating
the local build and development of multiple source repositories.

My only critic is that using this Gradle facility should not be a
requirement for extending OFBiz.

Thanks for you answer.

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

[Mathieu Lirzin]

Mathieu Lirzin <[hidden email]> writes:

> Here is a *big* change that I am considering for OFBiz to fixes those
> issues by leveraging the Java platform which already provides everything
> that we need to fix those issues:
>
>  - Distributing OFBiz in a Jar with its XML files inside to fix #1 would
>    allow developpers to extend the framework without compiling it which
>    removes the hard dependency on Gradle which fixes #2 and #3 and make
>    the build process snappier.
>
>  - the classpath can be used to compose an arbitrary list of plugins
>    from various location which fixes #4.
>
>  - The ServiceLoader class [1] is a standard and stable extensibility
>    mechanism based on JAR metadata which can be used to scan the
>    available OFBiz components available across multiple JARs that can
>    replace Gradle complex component bundling to fix #5.
>
> The biggest challenge with this solution is that we need to do a huge
> work of adapting OFBiz to to use location independant resources [3] when
> loading XML and GroovyScript files.

I have opened OFBIZ-11161 [1] to implement that change.

[1] https://issues.apache.org/jira/browse/OFBIZ-11161

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

[taher]

Hi Mathieu,

It could be a different objective, but it serves in the provision of
an architecture where the framework is separated from the plugins and
they can be deployed at runtime. To do so, implementing what I
mentioned previously might help.

Anyhow, all initiatives are welcomed. It's a good start.

On Thu, Aug 22, 2019 at 8:06 PM Mathieu Lirzin
<[hidden email]> wrote:

>
> Hello Taher,
>
> Taher Alkhateeb <[hidden email]> writes:
>
> > We have done many discussions in the past about publishing OFBiz as
> > deployable JAR and then the plugin mechanism would work as you
> > suggested and you won't need the build system anymore. I remember
> > joining our friend Adrian Crum before he passed away in an attempt to
> > do that but which did not bear fruit.
>
> Thanks for reminding that past effort,
>
> > So, I am in agreement in theory with the concept, but I think
> > realistically this makes the whole "plugin" thing a secondary topic,
> > and the main topic is how to rewrite the core OFBiz framework as an
> > independent JAR with perhaps a single comprehensive webapp, and then
> > make the components deployable (possibly even at runtime) by having
> > this truly pluggable architecture. If you look at the commit history
> > when gradle was introduced, you will notice that in the past there was
> > a little top-level jar created by the start component and then every
> > component would hold its own jars and sources which are then compiled
> > and linked through the old build system (ant) and included through the
> > custom class loader. So it was a mix between ant + class loader to get
> > things going but was still suffering from being one big monolithic
> > thing.
>
> I have used the old ant build in one of our customer project at Néréide
> and noticed the separation in multiple jars, but I didn't investigate
> the internals.
>
> > So IMHO, perhaps a good approach would have the following preliminary
> > work even before starting to worry about the plugin architecture:
> >
> > - Remove tomcat as an integral container and instead make the web
> > server independent of the framework and vice versa
> > - Delete the entire "container" architecture altogether and just keep
> > thing running in the core
> > - Make everything as a single webapp by default with possible opt-in
> > other webapps
> > - Customize the classloader to allow extending the classpath and
> > loading other components.
> > - Finally, the component concept itself should be redesigned such that
> > every component _is_ an outputted jar file.
>
> Maybe I am overlooking something but this seems like an interesting but
> different objective to me. :-)
>
> > One thing though, I think the concept of the subproject in gradle
> > would help, not hurt moving towards that goal. It's just that
> > currently, the implementation is not ideal and requires improvements.
> > I don't think ditching the entire subproject functionality is very
> > useful though.
>
> Sure I am maybe throwing the baby with the bathwater. Gradle
> multi-projet builds are indeed useful to solve the issue of coordinating
> the local build and development of multiple source repositories.
>
> My only critic is that using this Gradle facility should not be a
> requirement for extending OFBiz.
>
> Thanks for you answer.
>
> --
> Mathieu Lirzin
> GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37