Multi-tenancy (and modularization) - once more

[gnomie]

As indicated in previous posts, I continued my little investigation into
multi-tenant OfBiz and modularization of OfBiz. I used OfBiz 4.0 for
that (at the time I started looking at it that seemed to be THE stable
release. At first glance v9.04 doesn't differ dramatically in structure
from 4.0 though and I hope what I say below is still valid).

This got a little lengthy and you should read this as an experience
report and brain dump (which it is) and some general architectural
proposals that I would like to bring forward (far down below).

My motivation is to use OfBiz as the multi-tenant E-Commerce / ERP
feature implementation on a non-OfBiz server framework.

So this post and its proposals may well be somewhat more radical than
the previous multi-tenancy discussions.

It is perfectly valid that OfBiz provides its own framework to embed
Tomcat (or Jetty) and manage the life cycle of its components. However,
there is a lot of other stuff I also want to integrate with and I don't
like OfBiz's framework to the extent that I want to integrate everything
with that one. So one part of the exercise was to turn OfBiz into
something I can easily integrate with my environment (which is not OSGi
but close in nature).

Secondly I wanted to use one OfBiz implementation to serve several
tenants in one VM following the basic approach that seems to be the
preferred choice currently (e.g. by Marc and Erwan): Tenant separation
via different database schemas.

Following is a short description of some modifications I did. Further
below I put some general proposals.

Modifications to not share one entityengine.xml configuration
-------------------------------------------------------------

In order to operate on OfBiz services and data you need a delegator and,
for services, a dispatcher.

The whole point is to have an OfBiz delegator per tenant. One way of
achieving that is to declare several delegators in the entityengine.xml
configuration file. As I would prefer to keep tenants self-contained I
would prefer to not change the entity engine configuration file for
every tenant.

Instead, as that configuration file gets resolved via the class path and
as in my case tenant "projects" are independent applications (with their
own classloaders), I would prefer to have every tenant own his own
entity engine configuration. To do so I had to modify GenericDelegator
in a way that the delegator map is partitioned by the thread's context
class loader. Similarly EntityConfigUtil had to be adapted.

Given an environment where OfBiz runs as is and there is additional web
applications that have classloaders delegating to the OfBiz classloader,
each of these applications may contain its own entity engine
configuration.

For dispatchers, the same problem exists: A static map of dispatchers.
So again, you would have to make this cache context aware.


Modifications for Modularization
--------------------------------

That was actually harder. I wanted to have a module (think bundle) per
OfBiz component - which simply follows and extends OfBiz's
modularization as its done today.

The OfBiz code is actually nicely modularized (as you can tell from the
build files) and splitting it up into modules was not so hard.

I put everything in config, scripts into class path and left the rest in
the component metadata (addressable via component:// URLs)

In order to support the OfBiz component model, I introduced that as a
component model on my platform (think OSGi Extender Model) so that all
component meta-data (entitydef and so on) is known to the system. That
required some serious modification of ComponentConfig but - looking
backwards - was straightforward.

The whole point of this is to be able to extend OfBiz with new
components that use other stuff that runs on my platform - using the
means of modularization available. I wanted to avoid a "server in a
server" situation.

Btw. I think what is in scripts, src, config is not component metadata
but component implementation, while the web apps are consumers of the
components but otherwise independent.


Modifications for Multi-Tenant Web Apps
---------------------------------------

There was some proposals to use virtual hosts (Marc) or additional user
data (Erwan) to identify the tenant associated with a request and then
pick the right delegator/dispatcher in the web apps that come with
OfBiz. Both work under the assumption that all tenants share the same
apps. In my case, that is only partially true. The maintenance apps
(like party manager and so on) are good for reuse. The E-Commerce front
end will most likely be tenant-specific.

Also, considering the modifications above, I found a some non-trivial
number of places in the web app code where dispatcher and delegators
where kept in instance variables so that they couldn't be exchanged on
the fly (I should revisit the code - Emforium must have fixed that
somehow).

Anyway... what I did in the end was to create web apps per tenant that
share the resources of the OOTB web apps but have their own entity
engine config. This means there is a /tenant_partymgr application for
example. Doing that however got me into trouble with some hard-coded
context paths. I changed UtilHTTP to have the context path in its
parameters and adapted a few places that had hard-coded paths.

Some more Problems (possibly 4.0 specific) I ran into were:

Cached Classloader:

I dumped that one as Jetty had a problem resolving the classpath for JSP
compilation (could be my Jetty version...). On a side-note: Are you sure
that class loader helps?

Freemarker paths:

At least SurveyWrapper uses UtilURL to resolve its freemarker templates
that hence could not referred to via "component://" URLs. I had to
change the paths and the resolution mechanism to user FlexibleLocation.
This is due to the fact that my file system layout is different that the
OOTB layout.

Bean Shell Scripts:

While they are part of the Web App resources, they are resolved via
"component://..." URLs rather than as resources of the web app (e.g.
ServletContext.getResource(...) or something similar).  I think it would
be most appropriate to treat those scripts as code and resolve them via
the class path. For now I kept a copy of them in both places, under the
component meta-data and in the web app (for simplicity).


Summary and Proposals
---------------------

While that (not 100% complete) list above may sound like it was a lot of
tweaking and work, I'd rather say that most of it was rather
straightforward and OfBiz is surprisingly well prepared for what I
wanted to achieve although it is not used that way today. So that's
good.

I believe there is a great potential in making OfBiz "embeddable". Of
course, the fact that it is usable OOTB is perfect. But other will start
where I am: Looking for an E-Commerce and ERP feature to integrate
rather than integrate with the Ofbiz platform.

The crucial integration points are (as far as I can tell now):

- Provisioning of Configuration (e.g. entity engine config)

- Provisioning for component metadata (that is shared across
configurations).

- Maintenance of an Application Context (maintaining delegators and
dispatchers etc.)

Imagine the providers abstracted by interfaces for which there are some
default implementations available OOTB.

When setting up an Application Context (that the calling application
owns) you would have to pass in a configuration and a component
provider. After that you can ask the Application Context for a delegator
or a dispatcher and get going.

All static caching of delegators and dispatchers would happen in the
Application Context.

In order to pass that around between entry points to the system, e.g.
different web apps, it can be registered by one unique key with the
system (some weak reference observer).

So you could achieve the same thing you have today but manage different
configurations the way you like and store component metadata the way you
like.

Essentially my proposal is to abstract everything that is deployment
specific.

In the end you should be able to run OfBiz as a set of bundles on OSGi
(e.g. tenant configuration and component metadata managed by an OSGi
extender) or as a set of libs and files in a standalone GUI app
(everything resolved on the classpath).

Ok, this got long enough for today. Hope anybody manages to read through
this all and gives it some thought.

Thanks,
  Henning

[Adrian Crum]

Henning wrote:
> Also, considering the modifications above, I found a some non-trivial
> number of places in the web app code where dispatcher and delegators
> where kept in instance variables so that they couldn't be exchanged on
> the fly (I should revisit the code - Emforium must have fixed that
> somehow).

Could you provide a patch and create a Jira issue for this?

> Freemarker paths:
>
> At least SurveyWrapper uses UtilURL to resolve its freemarker templates
> that hence could not referred to via "component://" URLs. I had to
> change the paths and the resolution mechanism to user FlexibleLocation.
> This is due to the fact that my file system layout is different that the
> OOTB layout.

Could you provide a patch and create a Jira issue for this?

-Adrian

[gnomie]

Am Montag, den 03.08.2009, 08:59 -0700 schrieb Adrian Crum:
> Henning wrote:
> > Also, considering the modifications above, I found a some non-trivial
> > number of places in the web app code where dispatcher and delegators
> > where kept in instance variables so that they couldn't be exchanged on
> > the fly (I should revisit the code - Emforium must have fixed that
> > somehow).
>
> Could you provide a patch and create a Jira issue for this?

Not sure that is a bug the way OfBiz is used today. I will revisit the
code and contact Marc to get some more details.

>
> > Freemarker paths:
> >
> > At least SurveyWrapper uses UtilURL to resolve its freemarker templates
> > that hence could not referred to via "component://" URLs. I had to
> > change the paths and the resolution mechanism to user FlexibleLocation.
> > This is due to the fact that my file system layout is different that the
> > OOTB layout.
>
> Could you provide a patch and create a Jira issue for this?

Sure.

>
> -Adrian

Henning

[gnomie]

Adrian,

  just found that the second issue has been fixed in 9.04.

Thanks,
  Henning


Am Montag, den 03.08.2009, 08:59 -0700 schrieb Adrian Crum:

> Henning wrote:
> > Also, considering the modifications above, I found a some non-trivial
> > number of places in the web app code where dispatcher and delegators
> > where kept in instance variables so that they couldn't be exchanged on
> > the fly (I should revisit the code - Emforium must have fixed that
> > somehow).
>
> Could you provide a patch and create a Jira issue for this?
>
> > Freemarker paths:
> >
> > At least SurveyWrapper uses UtilURL to resolve its freemarker templates
> > that hence could not referred to via "component://" URLs. I had to
> > change the paths and the resolution mechanism to user FlexibleLocation.
> > This is due to the fact that my file system layout is different that the
> > OOTB layout.
>
> Could you provide a patch and create a Jira issue for this?
>
> -Adrian