OFBIZ-12081: DB driver download and configuration based on gradle property

[Daniel Watford]

Hello,

I'ved added a PR for OFBIZ-12081 where I add a gradle runtime dependency to
download a specific database driver. Currently I have only added a driver
for Postgres, but it should be simple to nominate drivers for other DBMS
where they can be downloaded and distributed.

Are there any issues with having an ofbiz 'approved' version of a database
driver specified as part of the ofbiz build process?

If this is a reasonable thing to do please nominate dependencies for the
other DBMS supported by ofbiz.

The same PR also adds a new build task to modify values in entityengine.xml
corresponding to the DBMS selected in gradle.properties.

I propose these additions to support building multiple varieties of docker
images which are pre-loaded with the database driver for a particular DBMS.
These pre-configured docker images can then be used as part of ofbiz
published docker-compose specifications which couple a docker image with a
particular DBMS, all running in their own containers.

Please let me know if there are any concerns with pre-configuring builds in
this way.

Thanks,

Dan.

--
Daniel Watford

[NANJUNDA MURTHY]

Good evening i plan to install ofbiz but iam getting gradle error

On 2020/11/30 20:45:52, Daniel Watford <[hidden email]> wrote:

> Hello,
>
> I'ved added a PR for OFBIZ-12081 where I add a gradle runtime dependency to
> download a specific database driver. Currently I have only added a driver
> for Postgres, but it should be simple to nominate drivers for other DBMS
> where they can be downloaded and distributed.
>
> Are there any issues with having an ofbiz 'approved' version of a database
> driver specified as part of the ofbiz build process?
>
> If this is a reasonable thing to do please nominate dependencies for the
> other DBMS supported by ofbiz.
>
> The same PR also adds a new build task to modify values in entityengine.xml
> corresponding to the DBMS selected in gradle.properties.
>
> I propose these additions to support building multiple varieties of docker
> images which are pre-loaded with the database driver for a particular DBMS.
> These pre-configured docker images can then be used as part of ofbiz
> published docker-compose specifications which couple a docker image with a
> particular DBMS, all running in their own containers.
>
> Please let me know if there are any concerns with pre-configuring builds in
> this way.
>
> Thanks,
>
> Dan.
>
> --
> Daniel Watford
>

[Jacques Le Roux]

Hi NANJUNDA,

Your message has been moderated, else it would not have reached this Mailing List.

Please subscribe to the user ML for such questions and then use your email client.
See why here http://ofbiz.apache.org/mailing-lists.html.

You will get a better support, people can answer you on the ML.
The wider the audience the better the answers you might get.

Also it's more work for moderators who have to accept your messages as long as you have not subscribed.
I'll personally no longer accept them (other moderators still could).

Thanks

This said please post the log with the error to the user ML

Jacques

Le 10/12/2020 à 13:29, NANJUNDA MURTHY a écrit :

> Good evening i plan to install ofbiz but iam getting gradle error
>
> On 2020/11/30 20:45:52, Daniel Watford <[hidden email]> wrote:
>> Hello,
>>
>> I'ved added a PR for OFBIZ-12081 where I add a gradle runtime dependency to
>> download a specific database driver. Currently I have only added a driver
>> for Postgres, but it should be simple to nominate drivers for other DBMS
>> where they can be downloaded and distributed.
>>
>> Are there any issues with having an ofbiz 'approved' version of a database
>> driver specified as part of the ofbiz build process?
>>
>> If this is a reasonable thing to do please nominate dependencies for the
>> other DBMS supported by ofbiz.
>>
>> The same PR also adds a new build task to modify values in entityengine.xml
>> corresponding to the DBMS selected in gradle.properties.
>>
>> I propose these additions to support building multiple varieties of docker
>> images which are pre-loaded with the database driver for a particular DBMS.
>> These pre-configured docker images can then be used as part of ofbiz
>> published docker-compose specifications which couple a docker image with a
>> particular DBMS, all running in their own containers.
>>
>> Please let me know if there are any concerns with pre-configuring builds in
>> this way.
>>
>> Thanks,
>>
>> Dan.
>>
>> --
>> Daniel Watford
>>

[Daniel Watford]

Hello,

I'd like to get opinions from the devs on whether it is worth pursuing
changes to include database driver resolution in ofbiz builds.

PR246 (https://github.com/apache/ofbiz-framework/pull/246) was an
experiment to try and configure an instance of ofbiz from the gradle build,
covering database driver resolution and populating entityengine.xml with
username, password, etc needed to access a DBMS.

I've since learned about how ofbiz uses the Application gradle plugin to
create distributions (distTar / distZip) where it probably wouldn't be
appropriate to embed username and password at build time. Therefore I don't
think it is worth pursuing use of build-time properties to configure
database access.

However I do wonder if it is still worth using a build-time property to
target a build and its distributions for use with a particular DBMS type
(mysql, postgres, etc). Although they wouldn't be distributed by Apache,
another party might then be able to then create and host ofbiz
distributions for each of the major DBMS platforms, with the appropriate
client access DBMS library already included. I think this would assist in
creation of container images targeting different databases too.

We already specify versions of all other ofbiz dependencies, but we don't
specify the versions of DBMS access libraries that should be used. This
creates an extra deployment step for an ofbiz administrator and might
potentially be a source of confusion.

By making DBMS choice a build-time property we can declare, in code, which
libraries are officially supported by the project.

If this is worth pursuing the next step will be to identify the database
driver versions used by the community for their ofbiz deployments.

Please let me know what you think.

Thanks,

Dan

On Mon, 30 Nov 2020 at 20:45, Daniel Watford <[hidden email]> wrote:

> Hello,
>
> I'ved added a PR for OFBIZ-12081 where I add a gradle runtime dependency
> to download a specific database driver. Currently I have only added a
> driver for Postgres, but it should be simple to nominate drivers for other
> DBMS where they can be downloaded and distributed.
>
> Are there any issues with having an ofbiz 'approved' version of a database
> driver specified as part of the ofbiz build process?
>
> If this is a reasonable thing to do please nominate dependencies for the
> other DBMS supported by ofbiz.
>
> The same PR also adds a new build task to modify values in
> entityengine.xml corresponding to the DBMS selected in gradle.properties.
>
> I propose these additions to support building multiple varieties of docker
> images which are pre-loaded with the database driver for a particular DBMS.
> These pre-configured docker images can then be used as part of ofbiz
> published docker-compose specifications which couple a docker image with a
> particular DBMS, all running in their own containers.
>
> Please let me know if there are any concerns with pre-configuring builds
> in this way.
>
> Thanks,
>
> Dan.
>
> --
> Daniel Watford
>

--
Daniel Watford

[Eugen Stan]

Hello Daniel,

I'll re-post with updates my github comment here since I imagine not
everyone will read the PR comments.


I think trying to meddle with those configs at build time is brittle and
would complicate the build.

It would help to have the ability to add jars and configuration files to
the classpath - since that is where ofbiz looks for.

So the idea is to make it easy for people to add things at the BEGINING
of the classpath so they can override ofbiz configurations (and / or
libraries ?!)

I believe this can achieved for both gradle deploy and binary
(ofbiz.tar) deploy with minimal changes.

If you could add the code snippet bellow to build.gradle then people can
add files to config/ and lib-extra/ directories and they will be
available for ofbiz when it starts.

I've tested this and it works for my ofbiz docker build (I'm planning an
article these next 2 days and will share it. Spoiler: it works on ARM -
raspberry pi 4 ;) ).

I'm also adding the database drivers post build to lib-extra .

That way I can keep the ofbiz source unchanged and still get what I need.

(I don't have windows to test the classpath ).

tasks.startScripts {
     doLast {
         // Alter the start script for Unix systems.
         unixScript.text =
                 unixScript.text.replace('CLASSPATH=$APP_HOME/lib',
 
'CLASSPATH=$APP_HOME/config/:$APP_HOME/lib-extra/*:$APP_HOME/lib')

         // Alter the start script for Windows systems.
//        windowsScript.text =
//                windowsScript.text.replace('CLASSPATH=$APP_HOME/lib',
//
'CLASSPATH=$APP_HOME\\conf\\:$APP_HOME/lib-extra/*:$APP_HOME/lib')
     }
}


For adding things to the classpath when running gradle there is
something in place:

// Libraries downloaded manually
     implementation fileTree(dir: file("${rootDir}/lib"), include:
'**/*.jar')



--
Eugen Stan
[phone] / netdava.com

[Daniel Watford]

Hi Eugen,

I like the approach of a config directory which can be used to override
configuration files already compiled into a distribution (distTar/distZip).
It's probably worth opening a separate ticket for that since it would be
analogous to how the /lib directory is currently used for gradle builds.

But my main question was intended to deal with how ofbiz administrators
identify and retrieve the database driver libraries for their deployments.

In your case did you find a list of recommended versions for use with Ofbiz
or did you have a driver that you often use with your DBMS? Did it take
much time to choose the driver?

Even if it was easy for you to figure out the driver version, do you think
other less experienced ofbiz administrators would struggle? Having the
ofbiz project declare, in code, what library versions are known to work
with various DBMS would reduce the effort required for all administrators.

I agree that we probably don't want to change the content of configurations
at build time, but it might be appropriate to break entityengine.xml into
smaller DBMS specific files and then only include one of those files in a
binary distribution specific to a particular DBMS, hopefully making it a
bit easier for ofbiz administrators to get started.

My overall aim is to reduce the amount of additional library downloads and
configuration an ofbiz administrator needs to perform in order to get
started. Every step should help get closer to externalising configuration,
as I believe you are already working towards, which should make deploying
and upgrading ofbiz running in containers a bit easier.

Thanks,

Dan.

On Tue, 12 Jan 2021 at 12:21, Eugen Stan <[hidden email]> wrote:

> Hello Daniel,
>
> I'll re-post with updates my github comment here since I imagine not
> everyone will read the PR comments.
>
>
> I think trying to meddle with those configs at build time is brittle and
> would complicate the build.
>
> It would help to have the ability to add jars and configuration files to
> the classpath - since that is where ofbiz looks for.
>
> So the idea is to make it easy for people to add things at the BEGINING
> of the classpath so they can override ofbiz configurations (and / or
> libraries ?!)
>
> I believe this can achieved for both gradle deploy and binary
> (ofbiz.tar) deploy with minimal changes.
>
> If you could add the code snippet bellow to build.gradle then people can
> add files to config/ and lib-extra/ directories and they will be
> available for ofbiz when it starts.
>
> I've tested this and it works for my ofbiz docker build (I'm planning an
> article these next 2 days and will share it. Spoiler: it works on ARM -
> raspberry pi 4 ;) ).
>
> I'm also adding the database drivers post build to lib-extra .
>
> That way I can keep the ofbiz source unchanged and still get what I need.
>
> (I don't have windows to test the classpath ).
>
> tasks.startScripts {
>      doLast {
>          // Alter the start script for Unix systems.
>          unixScript.text =
>                  unixScript.text.replace('CLASSPATH=$APP_HOME/lib',
>
> 'CLASSPATH=$APP_HOME/config/:$APP_HOME/lib-extra/*:$APP_HOME/lib')
>
>          // Alter the start script for Windows systems.
> //        windowsScript.text =
> //                windowsScript.text.replace('CLASSPATH=$APP_HOME/lib',
> //
> 'CLASSPATH=$APP_HOME\\conf\\:$APP_HOME/lib-extra/*:$APP_HOME/lib')
>      }
> }
>
>
> For adding things to the classpath when running gradle there is
> something in place:
>
> // Libraries downloaded manually
>      implementation fileTree(dir: file("${rootDir}/lib"), include:
> '**/*.jar')
>
>
>
> --
> Eugen Stan
> [phone] / netdava.com
>

--
Daniel Watford

[James Yong-2]

Hi Daniel,

IMO, splitting entityengine.xml file into db-specific is good in general.

I have the following suggestions:

Gradle
Allow more 1 db driver to be downloaded. Different datasource can be in use at the same time, like mysql for olap and postgreql for others.

Non-Gradle
Allow ENTITY_ENGINE_XML_FILENAME=entityengine.xml in EntityConfig class to be configurable.
So we can switch to use entityengine.postgres.xml, entityengine.derby.xml, etc.

Regards,
James

On 2021/01/12 10:28:38, Daniel Watford <[hidden email]> wrote:

> Hello,
>
> I'd like to get opinions from the devs on whether it is worth pursuing
> changes to include database driver resolution in ofbiz builds.
>
> PR246 (https://github.com/apache/ofbiz-framework/pull/246) was an
> experiment to try and configure an instance of ofbiz from the gradle build,
> covering database driver resolution and populating entityengine.xml with
> username, password, etc needed to access a DBMS.
>
> I've since learned about how ofbiz uses the Application gradle plugin to
> create distributions (distTar / distZip) where it probably wouldn't be
> appropriate to embed username and password at build time. Therefore I don't
> think it is worth pursuing use of build-time properties to configure
> database access.
>
> However I do wonder if it is still worth using a build-time property to
> target a build and its distributions for use with a particular DBMS type
> (mysql, postgres, etc). Although they wouldn't be distributed by Apache,
> another party might then be able to then create and host ofbiz
> distributions for each of the major DBMS platforms, with the appropriate
> client access DBMS library already included. I think this would assist in
> creation of container images targeting different databases too.
>
> We already specify versions of all other ofbiz dependencies, but we don't
> specify the versions of DBMS access libraries that should be used. This
> creates an extra deployment step for an ofbiz administrator and might
> potentially be a source of confusion.
>
> By making DBMS choice a build-time property we can declare, in code, which
> libraries are officially supported by the project.
>
> If this is worth pursuing the next step will be to identify the database
> driver versions used by the community for their ofbiz deployments.
>
> Please let me know what you think.
>
> Thanks,
>
> Dan
>
> On Mon, 30 Nov 2020 at 20:45, Daniel Watford <[hidden email]> wrote:
>
> > Hello,
> >
> > I'ved added a PR for OFBIZ-12081 where I add a gradle runtime dependency
> > to download a specific database driver. Currently I have only added a
> > driver for Postgres, but it should be simple to nominate drivers for other
> > DBMS where they can be downloaded and distributed.
> >
> > Are there any issues with having an ofbiz 'approved' version of a database
> > driver specified as part of the ofbiz build process?
> >
> > If this is a reasonable thing to do please nominate dependencies for the
> > other DBMS supported by ofbiz.
> >
> > The same PR also adds a new build task to modify values in
> > entityengine.xml corresponding to the DBMS selected in gradle.properties.
> >
> > I propose these additions to support building multiple varieties of docker
> > images which are pre-loaded with the database driver for a particular DBMS.
> > These pre-configured docker images can then be used as part of ofbiz
> > published docker-compose specifications which couple a docker image with a
> > particular DBMS, all running in their own containers.
> >
> > Please let me know if there are any concerns with pre-configuring builds
> > in this way.
> >
> > Thanks,
> >
> > Dan.
> >
> > --
> > Daniel Watford
> >
>
>
> --
> Daniel Watford
>

[Eugen Stan]

Hi,

Please see my reply inline:

On 14.01.2021 16:21, Daniel Watford wrote:
> Hi Eugen,
>
> I like the approach of a config directory which can be used to override
> configuration files already compiled into a distribution (distTar/distZip).
> It's probably worth opening a separate ticket for that since it would be
> analogous to how the /lib directory is currently used for gradle builds.

https://issues.apache.org/jira/browse/OFBIZ-12136

lib/ is not enough. Gradle hardcodes the classpath. Adding jars there
has no effect IMO (see generated bin/ofbiz script). That is why I added
lib-extra/

> But my main question was intended to deal with how ofbiz administrators
> identify and retrieve the database driver libraries for their deployments.
>
> In your case did you find a list of recommended versions for use with Ofbiz
> or did you have a driver that you often use with your DBMS? Did it take
> much time to choose the driver?

When talking database drivers I try to pick the latest release.
It's important to match the driver version with database version and JVM
version. With PostgreSQL the latest driver is fine with the postgres 13
(which I am using) and JVM 8 or 11 which I am using.

For other databases it's the same: pick a driver that works with you
database version and your JVM.

>
> Even if it was easy for you to figure out the driver version, do you think
> other less experienced ofbiz administrators would struggle? Having the
> ofbiz project declare, in code, what library versions are known to work
> with various DBMS would reduce the effort required for all administrators.

Yes, for other people it might not be as easy - depending on their
knowledge and expertise - that is why we have community and consultancy
services.

> I agree that we probably don't want to change the content of configurations
> at build time, but it might be appropriate to break entityengine.xml into
> smaller DBMS specific files and then only include one of those files in a
> binary distribution specific to a particular DBMS, hopefully making it a
> bit easier for ofbiz administrators to get started.

I would split database connection out of the config or have the ability
to use environment properties or java system properties to configure the
connection.

This is better IMO than having to edit configuration files.
If you have an OFBiz build than you can change just supply different
params for DB instead of having to create config files.

Modern configuration libraries have functionality to interpolate
variables in the configuration file with env variables or system
properties. See for example https://github.com/lightbend/config .


> My overall aim is to reduce the amount of additional library downloads and
> configuration an ofbiz administrator needs to perform in order to get
> started. Every step should help get closer to externalising configuration,
> as I believe you are already working towards, which should make deploying
> and upgrading ofbiz running in containers a bit easier.

Thanks and I'll try to help you with this goal.
Docker makes these deployment issues more visible.
People handle deployment in many ways and with many tools.
I believe documentation is better here than opinionated solutions.


Regards,
--
Eugen Stan
[phone] / netdava.com

[Jacques Le Roux]

Hi Daniel, James,

Did we not recently discuss about splitting entityengine.xml and could not get a consensus? (I searched but did not find it)

Thanks

Jacques

Le 14/01/2021 à 15:21, Daniel Watford a écrit :

> Hi Eugen,
>
> I like the approach of a config directory which can be used to override
> configuration files already compiled into a distribution (distTar/distZip).
> It's probably worth opening a separate ticket for that since it would be
> analogous to how the /lib directory is currently used for gradle builds.
>
> But my main question was intended to deal with how ofbiz administrators
> identify and retrieve the database driver libraries for their deployments.
>
> In your case did you find a list of recommended versions for use with Ofbiz
> or did you have a driver that you often use with your DBMS? Did it take
> much time to choose the driver?
>
> Even if it was easy for you to figure out the driver version, do you think
> other less experienced ofbiz administrators would struggle? Having the
> ofbiz project declare, in code, what library versions are known to work
> with various DBMS would reduce the effort required for all administrators.
>
> I agree that we probably don't want to change the content of configurations
> at build time, but it might be appropriate to break entityengine.xml into
> smaller DBMS specific files and then only include one of those files in a
> binary distribution specific to a particular DBMS, hopefully making it a
> bit easier for ofbiz administrators to get started.
>
> My overall aim is to reduce the amount of additional library downloads and
> configuration an ofbiz administrator needs to perform in order to get
> started. Every step should help get closer to externalising configuration,
> as I believe you are already working towards, which should make deploying
> and upgrading ofbiz running in containers a bit easier.
>
> Thanks,
>
> Dan.
>
> On Tue, 12 Jan 2021 at 12:21, Eugen Stan <[hidden email]> wrote:
>
>> Hello Daniel,
>>
>> I'll re-post with updates my github comment here since I imagine not
>> everyone will read the PR comments.
>>
>>
>> I think trying to meddle with those configs at build time is brittle and
>> would complicate the build.
>>
>> It would help to have the ability to add jars and configuration files to
>> the classpath - since that is where ofbiz looks for.
>>
>> So the idea is to make it easy for people to add things at the BEGINING
>> of the classpath so they can override ofbiz configurations (and / or
>> libraries ?!)
>>
>> I believe this can achieved for both gradle deploy and binary
>> (ofbiz.tar) deploy with minimal changes.
>>
>> If you could add the code snippet bellow to build.gradle then people can
>> add files to config/ and lib-extra/ directories and they will be
>> available for ofbiz when it starts.
>>
>> I've tested this and it works for my ofbiz docker build (I'm planning an
>> article these next 2 days and will share it. Spoiler: it works on ARM -
>> raspberry pi 4 ;) ).
>>
>> I'm also adding the database drivers post build to lib-extra .
>>
>> That way I can keep the ofbiz source unchanged and still get what I need.
>>
>> (I don't have windows to test the classpath ).
>>
>> tasks.startScripts {
>>       doLast {
>>           // Alter the start script for Unix systems.
>>           unixScript.text =
>>                   unixScript.text.replace('CLASSPATH=$APP_HOME/lib',
>>
>> 'CLASSPATH=$APP_HOME/config/:$APP_HOME/lib-extra/*:$APP_HOME/lib')
>>
>>           // Alter the start script for Windows systems.
>> //        windowsScript.text =
>> //                windowsScript.text.replace('CLASSPATH=$APP_HOME/lib',
>> //
>> 'CLASSPATH=$APP_HOME\\conf\\:$APP_HOME/lib-extra/*:$APP_HOME/lib')
>>       }
>> }
>>
>>
>> For adding things to the classpath when running gradle there is
>> something in place:
>>
>> // Libraries downloaded manually
>>       implementation fileTree(dir: file("${rootDir}/lib"), include:
>> '**/*.jar')
>>
>>
>>
>> --
>> Eugen Stan
>> [phone] / netdava.com
>>
>

[Daniel Watford]

Hi Jacques,

I don't recall a conversation on the mailing list about breaking up
entityengine.xml, however Eugen did suggest creating some templates in the
context of the PR related to this thread (
https://github.com/apache/ofbiz-framework/pull/246) which I then
implemented for Postgres and Derby (in the PR).

On Thu, 14 Jan 2021 at 15:21, Jacques Le Roux <[hidden email]>
wrote:

> Hi Daniel, James,
>
> Did we not recently discuss about splitting entityengine.xml and could not
> get a consensus? (I searched but did not find it)
>
> Thanks
>
> Jacques
>
> Le 14/01/2021 à 15:21, Daniel Watford a écrit :
> > Hi Eugen,
> >
> > I like the approach of a config directory which can be used to override
> > configuration files already compiled into a distribution
> (distTar/distZip).
> > It's probably worth opening a separate ticket for that since it would be
> > analogous to how the /lib directory is currently used for gradle builds.
> >
> > But my main question was intended to deal with how ofbiz administrators
> > identify and retrieve the database driver libraries for their
> deployments.
> >
> > In your case did you find a list of recommended versions for use with
> Ofbiz
> > or did you have a driver that you often use with your DBMS? Did it take
> > much time to choose the driver?
> >
> > Even if it was easy for you to figure out the driver version, do you
> think
> > other less experienced ofbiz administrators would struggle? Having the
> > ofbiz project declare, in code, what library versions are known to work
> > with various DBMS would reduce the effort required for all
> administrators.
> >
> > I agree that we probably don't want to change the content of
> configurations
> > at build time, but it might be appropriate to break entityengine.xml into
> > smaller DBMS specific files and then only include one of those files in a
> > binary distribution specific to a particular DBMS, hopefully making it a
> > bit easier for ofbiz administrators to get started.
> >
> > My overall aim is to reduce the amount of additional library downloads
> and
> > configuration an ofbiz administrator needs to perform in order to get
> > started. Every step should help get closer to externalising
> configuration,
> > as I believe you are already working towards, which should make deploying
> > and upgrading ofbiz running in containers a bit easier.
> >
> > Thanks,
> >
> > Dan.
> >
> > On Tue, 12 Jan 2021 at 12:21, Eugen Stan <[hidden email]> wrote:
> >
> >> Hello Daniel,
> >>
> >> I'll re-post with updates my github comment here since I imagine not
> >> everyone will read the PR comments.
> >>
> >>
> >> I think trying to meddle with those configs at build time is brittle and
> >> would complicate the build.
> >>
> >> It would help to have the ability to add jars and configuration files to
> >> the classpath - since that is where ofbiz looks for.
> >>
> >> So the idea is to make it easy for people to add things at the BEGINING
> >> of the classpath so they can override ofbiz configurations (and / or
> >> libraries ?!)
> >>
> >> I believe this can achieved for both gradle deploy and binary
> >> (ofbiz.tar) deploy with minimal changes.
> >>
> >> If you could add the code snippet bellow to build.gradle then people can
> >> add files to config/ and lib-extra/ directories and they will be
> >> available for ofbiz when it starts.
> >>
> >> I've tested this and it works for my ofbiz docker build (I'm planning an
> >> article these next 2 days and will share it. Spoiler: it works on ARM -
> >> raspberry pi 4 ;) ).
> >>
> >> I'm also adding the database drivers post build to lib-extra .
> >>
> >> That way I can keep the ofbiz source unchanged and still get what I
> need.
> >>
> >> (I don't have windows to test the classpath ).
> >>
> >> tasks.startScripts {
> >>       doLast {
> >>           // Alter the start script for Unix systems.
> >>           unixScript.text =
> >>                   unixScript.text.replace('CLASSPATH=$APP_HOME/lib',
> >>
> >> 'CLASSPATH=$APP_HOME/config/:$APP_HOME/lib-extra/*:$APP_HOME/lib')
> >>
> >>           // Alter the start script for Windows systems.
> >> //        windowsScript.text =
> >> //                windowsScript.text.replace('CLASSPATH=$APP_HOME/lib',
> >> //
> >> 'CLASSPATH=$APP_HOME\\conf\\:$APP_HOME/lib-extra/*:$APP_HOME/lib')
> >>       }
> >> }
> >>
> >>
> >> For adding things to the classpath when running gradle there is
> >> something in place:
> >>
> >> // Libraries downloaded manually
> >>       implementation fileTree(dir: file("${rootDir}/lib"), include:
> >> '**/*.jar')
> >>
> >>
> >>
> >> --
> >> Eugen Stan
> >> [phone] / netdava.com
> >>
> >
>

--
Daniel Watford

[Daniel Watford]

Hi Eugen,

I think your point...

> It's important to match the driver version with database version

... is a good reason for the ofbiz project to NOT be opinionated about
database libraries and versions

Originally I had only thought we would need to specify one driver for one
DBMS type. I had failed to realise there would be lots of different
versions of any one DBMS in use, each potentially requiring a different
driver version.

Figuring out the 'good' driver version for each DBMS version potentially
used with ofbiz probably wouldn't be workable. I imagine at best we might
have a web page where ofbiz administrators can report on the combination of
DBMS version, driver and JVM they are successfully using.

You've changed my mind! :)   I'll drop the use of build-time properties to
resolve database driver libraries.

Thanks,

Dan.



On Thu, 14 Jan 2021 at 15:37, Eugen Stan <[hidden email]> wrote:

> Hi,
>
> Please see my reply inline:
>
> On 14.01.2021 16:21, Daniel Watford wrote:
> > Hi Eugen,
> >
> > I like the approach of a config directory which can be used to override
> > configuration files already compiled into a distribution
> (distTar/distZip).
> > It's probably worth opening a separate ticket for that since it would be
> > analogous to how the /lib directory is currently used for gradle builds.
>
> https://issues.apache.org/jira/browse/OFBIZ-12136
>
> lib/ is not enough. Gradle hardcodes the classpath. Adding jars there
> has no effect IMO (see generated bin/ofbiz script). That is why I added
> lib-extra/
>
> > But my main question was intended to deal with how ofbiz administrators
> > identify and retrieve the database driver libraries for their
> deployments.
> >
> > In your case did you find a list of recommended versions for use with
> Ofbiz
> > or did you have a driver that you often use with your DBMS? Did it take
> > much time to choose the driver?
>
> When talking database drivers I try to pick the latest release.
> It's important to match the driver version with database version and JVM
> version. With PostgreSQL the latest driver is fine with the postgres 13
> (which I am using) and JVM 8 or 11 which I am using.
>
> For other databases it's the same: pick a driver that works with you
> database version and your JVM.
>
> >
> > Even if it was easy for you to figure out the driver version, do you
> think
> > other less experienced ofbiz administrators would struggle? Having the
> > ofbiz project declare, in code, what library versions are known to work
> > with various DBMS would reduce the effort required for all
> administrators.
>
> Yes, for other people it might not be as easy - depending on their
> knowledge and expertise - that is why we have community and consultancy
> services.
>
> > I agree that we probably don't want to change the content of
> configurations
> > at build time, but it might be appropriate to break entityengine.xml into
> > smaller DBMS specific files and then only include one of those files in a
> > binary distribution specific to a particular DBMS, hopefully making it a
> > bit easier for ofbiz administrators to get started.
>
> I would split database connection out of the config or have the ability
> to use environment properties or java system properties to configure the
> connection.
>
> This is better IMO than having to edit configuration files.
> If you have an OFBiz build than you can change just supply different
> params for DB instead of having to create config files.
>
> Modern configuration libraries have functionality to interpolate
> variables in the configuration file with env variables or system
> properties. See for example https://github.com/lightbend/config .
>
>
> > My overall aim is to reduce the amount of additional library downloads
> and
> > configuration an ofbiz administrator needs to perform in order to get
> > started. Every step should help get closer to externalising
> configuration,
> > as I believe you are already working towards, which should make deploying
> > and upgrading ofbiz running in containers a bit easier.
>
> Thanks and I'll try to help you with this goal.
> Docker makes these deployment issues more visible.
> People handle deployment in many ways and with many tools.
> I believe documentation is better here than opinionated solutions.
>
>
> Regards,
> --
> Eugen Stan
> [phone] / netdava.com
>

--
Daniel Watford

[Eugen Stan]

Hi,

See my reply inline:

Regards,
Eugen

On 14.01.2021 18:02, Daniel Watford wrote:

> Hi Eugen,
>
> I think your point...
>
>  > It's important to match the driver version with database version
>
> ... is a good reason for the ofbiz project to NOT be opinionated about
> database libraries and versions
>
> Originally I had only thought we would need to specify one driver for
> one DBMS type. I had failed to realise there would be lots of different
> versions of any one DBMS in use, each potentially requiring a different
> driver version.
>
> Figuring out the 'good' driver version for each DBMS version potentially
> used with ofbiz probably wouldn't be workable. I imagine at best we
> might have a web page where ofbiz administrators can report on the
> combination of DBMS version, driver and JVM they are successfully using.
>
> You've changed my mind! :)   I'll drop the use of build-time properties
> to resolve database driver libraries.
>

If that is the case then the solution might be simpler:
We can set our own constraints.

For a lot of web applications the norm is: We support only the latest
2-4 versions of Browser X, Y and Z.

In Debian, the policy is to keep the major version of an app during the
full lifetime of the release. If Debian 10 comes with Gnome 3.34 - That
is the version people are going to use until Debian 11 comes ~ 2 years
later. Only minor upgrades / patches are updated. This makes sense
considering Debian packages ~ 65.000+ packages (?!).

OFBiz could deliver only the latest versions at the time of release and
upgrade only patch versions (?!) or during point releases.

It's best if this policy is made explicit in the documentation.

Let's say that OFBiz 18.x comes with
- postgresql-jdbc 42.x which supports PostgreSQL 9.x -> 13.x
- mysql connector 8.x - which supports MySQL and MariaDB 10.x +

OFBiz 21.x will come with the latest drivers at it's release time, and
so on.

Also since OFBiz has a base JVM version used for building, we could use
that version to match the drivers as well.

This database version compatibility and information about jvm version is
available on the provider website.

If people use something else - it's on them to make the changes - this
phrase should also be explicit.
IMO It's not the community's obligation to support every permutation out
there.

Because I build Docker images and I can add libraries to classpath - I'm
good either way.

Once https://issues.apache.org/jira/browse/OFBIZ-12136 is merged (will
submit a PR) - I'm all set for database drivers.

IMO one reason to add drivers to the projects is to make make it easier
to run integration tests for those databases OOTB.
I'm still not familiar with the project structure to figure out how to
run those tests.


--
Eugen Stan
[phone] / netdava.com