[PROPOSAL] Environment variable configuration

[Gil Portenseigne]

Hello all,

Working with different hosting companies, we used to have issues when
deploying OFBiz concerning technical configuration of the different
environments.
We are writing this mail to get feedback from the community and
eventually propose to improve OFBiz on this matter.

For a customer, we are working with 4 instances of a release 13.07
OFBiz, and are currently using a set of patches (with addonmanager...)
to manage environment specific configurations.
During each production deployment, the hosting company receive from our
jenkins a precompiled archive containing OFBiz codebase, and then apply
the set of patches to configure it to the environment needs, recompile
and relaunch...

This way of doing can cause issue when patch could not apply, after a
codebase modification (pretty rare but it happens).

We are not satisfied with this way of doing, we are currently thinking
about using environment variables to configure technical environment
properties (those are on the hosting company responsibility), and to
keep functional specifics into the code base.
If you have some experience or advice in this matter, you are welcome.

For our case, we currently have enhanced OFBiz to be able to get
environment variable from the operating system within property file and
some other configuration files (with default value if not set).

Examples :
*In Property file :
password=${env:ONE_CONF:ofbiz} -> environment variable ONE_CONF or ofbiz
if unset
other_config=${env:OTHER_CONF:${partyId}} -> environment variable
OTHER_CONF or ${partyId} if unset
*In entityengine.xml :
jdbc-uri="${env:DB_POSTGRES_URI:jdbc:mysql://127.0.0.1/ofbiz?autoReconnect=true}"
jdbc-username="${env:DB_POSTGRES_USER:ofbiz}"
jdbc-password="${env:DB_POSTGRES_PWD:ofbiz}"/>

That allow us to keep functional parameters stored within git branches.
Our jenkins now is able to build our 4 configured and compiled instances
and deliver it to the hosting company, that just have to set/check
environment variable (database access, activeMQ, log location, instance
id, etc.) before starting OFBiz app. Now we cannot have configuration
failure during deployment.

We will be glad to contribute it, if it's the good way to go !

Best Regards !

Gil Portenseigne

[Michael Brohl-3]

Hi Gil,

we have similar challenges and modified OFBiz to deal with it easily. We
offered to contribute this long time ago (2008) but it was decided
against [1]. It was suggested to use patches instead but I think it's
too complicated to manage several patch sets for different environments.

We now use a staged configure mechanism which uses a base build file,
auto detected machine name and provided parameters to decide which
configurations should be pulled for the environment. It's currently Ant
based and therefore does not fit into the current build mechanism (on
the todo list).

I like your approach also and I think it should be evaluated and discussed.

Best regards,

Michael Brohl
ecomify GmbH
www.ecomify.de

[1]
https://lists.apache.org/thread.html/b95e239250880d9a5b34268b3b711f0f8f7f0540a26bb41c5ced493a@1213087551@%3Cdev.ofbiz.apache.org%3E



Am 05.07.17 um 17:36 schrieb gil portenseigne:

> Hello all,
>
> Working with different hosting companies, we used to have issues when
> deploying OFBiz concerning technical configuration of the different
> environments.
> We are writing this mail to get feedback from the community and
> eventually propose to improve OFBiz on this matter.
>
> For a customer, we are working with 4 instances of a release 13.07
> OFBiz, and are currently using a set of patches (with addonmanager...)
> to manage environment specific configurations.
> During each production deployment, the hosting company receive from
> our jenkins a precompiled archive containing OFBiz codebase, and then
> apply the set of patches to configure it to the environment needs,
> recompile and relaunch...
>
> This way of doing can cause issue when patch could not apply, after a
> codebase modification (pretty rare but it happens).
>
> We are not satisfied with this way of doing, we are currently thinking
> about using environment variables to configure technical environment
> properties (those are on the hosting company responsibility), and to
> keep functional specifics into the code base.
> If you have some experience or advice in this matter, you are welcome.
>
> For our case, we currently have enhanced OFBiz to be able to get
> environment variable from the operating system within property file
> and some other configuration files (with default value if not set).
>
> Examples :
> *In Property file :
> password=${env:ONE_CONF:ofbiz} -> environment variable ONE_CONF or
> ofbiz if unset
> other_config=${env:OTHER_CONF:${partyId}} -> environment variable
> OTHER_CONF or ${partyId} if unset
> *In entityengine.xml :
> jdbc-uri="${env:DB_POSTGRES_URI:jdbc:mysql://127.0.0.1/ofbiz?autoReconnect=true}"
>
> jdbc-username="${env:DB_POSTGRES_USER:ofbiz}"
> jdbc-password="${env:DB_POSTGRES_PWD:ofbiz}"/>
>
> That allow us to keep functional parameters stored within git
> branches. Our jenkins now is able to build our 4 configured and
> compiled instances and deliver it to the hosting company, that just
> have to set/check environment variable (database access, activeMQ, log
> location, instance id, etc.) before starting OFBiz app. Now we cannot
> have configuration failure during deployment.
>
> We will be glad to contribute it, if it's the good way to go !
>
> Best Regards !
>
> Gil Portenseigne
>

[taher]

I faced something similar in the past (Not related to OFBiz). Many
systems require different paths of configuration depending on
different deployment scenarios.

There are multiple types of configurations I can think of including:
- platform dependent configurations (it depends on your server and the
environment)
- Sensitive information like passwords
- platform independent (test vs deployment environments for example)

I think the first two do not belong to the code base. And so the best
option for such problems that I used in the past is templates
(whatever technology you choose). Templates + Build system +
environment variables substantially simplify deployment when compared
with patches that are highly dependent on the current state of the
code base.

Now for the third kind (platform independent like multiple targets)
maybe we can get some good ideas from ruby-on-rails [1]. Essentially
we can tweak the build system and add a few environment files to allow
for multiple target environments.

[1] http://guides.rubyonrails.org/configuring.html

On Wed, Jul 5, 2017 at 6:56 PM, Michael Brohl <[hidden email]> wrote:

> Hi Gil,
>
> we have similar challenges and modified OFBiz to deal with it easily. We
> offered to contribute this long time ago (2008) but it was decided against
> [1]. It was suggested to use patches instead but I think it's too
> complicated to manage several patch sets for different environments.
>
> We now use a staged configure mechanism which uses a base build file, auto
> detected machine name and provided parameters to decide which configurations
> should be pulled for the environment. It's currently Ant based and therefore
> does not fit into the current build mechanism (on the todo list).
>
> I like your approach also and I think it should be evaluated and discussed.
>
> Best regards,
>
> Michael Brohl
> ecomify GmbH
> www.ecomify.de
>
> [1]
> https://lists.apache.org/thread.html/b95e239250880d9a5b34268b3b711f0f8f7f0540a26bb41c5ced493a@1213087551@%3Cdev.ofbiz.apache.org%3E
>
>
>
> Am 05.07.17 um 17:36 schrieb gil portenseigne:
>
>> Hello all,
>>
>> Working with different hosting companies, we used to have issues when
>> deploying OFBiz concerning technical configuration of the different
>> environments.
>> We are writing this mail to get feedback from the community and eventually
>> propose to improve OFBiz on this matter.
>>
>> For a customer, we are working with 4 instances of a release 13.07 OFBiz,
>> and are currently using a set of patches (with addonmanager...) to manage
>> environment specific configurations.
>> During each production deployment, the hosting company receive from our
>> jenkins a precompiled archive containing OFBiz codebase, and then apply the
>> set of patches to configure it to the environment needs, recompile and
>> relaunch...
>>
>> This way of doing can cause issue when patch could not apply, after a
>> codebase modification (pretty rare but it happens).
>>
>> We are not satisfied with this way of doing, we are currently thinking
>> about using environment variables to configure technical environment
>> properties (those are on the hosting company responsibility), and to keep
>> functional specifics into the code base.
>> If you have some experience or advice in this matter, you are welcome.
>>
>> For our case, we currently have enhanced OFBiz to be able to get
>> environment variable from the operating system within property file and some
>> other configuration files (with default value if not set).
>>
>> Examples :
>> *In Property file :
>> password=${env:ONE_CONF:ofbiz} -> environment variable ONE_CONF or ofbiz
>> if unset
>> other_config=${env:OTHER_CONF:${partyId}} -> environment variable
>> OTHER_CONF or ${partyId} if unset
>> *In entityengine.xml :
>>
>> jdbc-uri="${env:DB_POSTGRES_URI:jdbc:mysql://127.0.0.1/ofbiz?autoReconnect=true}"
>> jdbc-username="${env:DB_POSTGRES_USER:ofbiz}"
>> jdbc-password="${env:DB_POSTGRES_PWD:ofbiz}"/>
>>
>> That allow us to keep functional parameters stored within git branches.
>> Our jenkins now is able to build our 4 configured and compiled instances and
>> deliver it to the hosting company, that just have to set/check environment
>> variable (database access, activeMQ, log location, instance id, etc.) before
>> starting OFBiz app. Now we cannot have configuration failure during
>> deployment.
>>
>> We will be glad to contribute it, if it's the good way to go !
>>
>> Best Regards !
>>
>> Gil Portenseigne
>>
>
>

[James Yong-2]

Hi Gil and all,

Faced the same issues before.

I had different environments like SIT, UAT and PROD requiring different settings.
There was also specific requirements to remove the source folders and zip the final files for deployment.

So I created a folder which has subfolders named after each different environment and contains the respective environment-specific properties e.g.
- deployment
    - SIT
        - db.properties
        ...
    - UAT
        - db.properties
        ...

Also created a set of template files that contained placeholders to be replaced during deployment
- templates
    - entity-engine.xml
    ...

Finally there is a custom Ant target for deployment to each environment that will:
1) build OFBiz and copy the required files (leaving out the source folders) into a build folder;
2) read the template files and replace the placeholders with the right environment properties before copy to the correct location in the build folder; and
3) zip the contents of the build folder and append a timestamp.

Having said all these, your approach is convention over configuration, and keeps the environment properties separate from the OFBiz folder. So
+1
from me.

Regards,
James Yong

On 2017-07-05 23:36 (+0800), gil portenseigne <[hidden email]> wrote:

> Hello all,
>
> Working with different hosting companies, we used to have issues when
> deploying OFBiz concerning technical configuration of the different
> environments.
> We are writing this mail to get feedback from the community and
> eventually propose to improve OFBiz on this matter.
>
> For a customer, we are working with 4 instances of a release 13.07
> OFBiz, and are currently using a set of patches (with addonmanager...)
> to manage environment specific configurations.
> During each production deployment, the hosting company receive from our
> jenkins a precompiled archive containing OFBiz codebase, and then apply
> the set of patches to configure it to the environment needs, recompile
> and relaunch...
>
> This way of doing can cause issue when patch could not apply, after a
> codebase modification (pretty rare but it happens).
>
> We are not satisfied with this way of doing, we are currently thinking
> about using environment variables to configure technical environment
> properties (those are on the hosting company responsibility), and to
> keep functional specifics into the code base.
> If you have some experience or advice in this matter, you are welcome.
>
> For our case, we currently have enhanced OFBiz to be able to get
> environment variable from the operating system within property file and
> some other configuration files (with default value if not set).
>
> Examples :
> *In Property file :
> password=${env:ONE_CONF:ofbiz} -> environment variable ONE_CONF or ofbiz
> if unset
> other_config=${env:OTHER_CONF:${partyId}} -> environment variable
> OTHER_CONF or ${partyId} if unset
> *In entityengine.xml :
> jdbc-uri="${env:DB_POSTGRES_URI:jdbc:mysql://127.0.0.1/ofbiz?autoReconnect=true}"
> jdbc-username="${env:DB_POSTGRES_USER:ofbiz}"
> jdbc-password="${env:DB_POSTGRES_PWD:ofbiz}"/>
>
> That allow us to keep functional parameters stored within git branches.
> Our jenkins now is able to build our 4 configured and compiled instances
> and deliver it to the hosting company, that just have to set/check
> environment variable (database access, activeMQ, log location, instance
> id, etc.) before starting OFBiz app. Now we cannot have configuration
> failure during deployment.
>
> We will be glad to contribute it, if it's the good way to go !
>
> Best Regards !
>
> Gil Portenseigne
>

[James Yong-2]

Hi Gil,

Some afterthought questions with the proposed change:
1) Can it support multiple OFBiz instances on the same server?
2) Can we store encrypted values for some of the environment variables like database credentials?

Regards,
James Yong

On 2017-07-06 16:28 (+0800), "James Yong"<[hidden email]> wrote:

> Hi Gil and all,
>
> Faced the same issues before.
>
> I had different environments like SIT, UAT and PROD requiring different settings.
> There was also specific requirements to remove the source folders and zip the final files for deployment.
>
> So I created a folder which has subfolders named after each different environment and contains the respective environment-specific properties e.g.
> - deployment
>     - SIT
>         - db.properties
>         ...
>     - UAT
>         - db.properties
>         ...
>
> Also created a set of template files that contained placeholders to be replaced during deployment
> - templates
>     - entity-engine.xml
>     ...
>
> Finally there is a custom Ant target for deployment to each environment that will:
> 1) build OFBiz and copy the required files (leaving out the source folders) into a build folder;
> 2) read the template files and replace the placeholders with the right environment properties before copy to the correct location in the build folder; and
> 3) zip the contents of the build folder and append a timestamp.
>
> Having said all these, your approach is convention over configuration, and keeps the environment properties separate from the OFBiz folder. So
> +1
> from me.
>
> Regards,
> James Yong
>
> On 2017-07-05 23:36 (+0800), gil portenseigne <[hidden email]> wrote:
> > Hello all,
> >
> > Working with different hosting companies, we used to have issues when
> > deploying OFBiz concerning technical configuration of the different
> > environments.
> > We are writing this mail to get feedback from the community and
> > eventually propose to improve OFBiz on this matter.
> >
> > For a customer, we are working with 4 instances of a release 13.07
> > OFBiz, and are currently using a set of patches (with addonmanager...)
> > to manage environment specific configurations.
> > During each production deployment, the hosting company receive from our
> > jenkins a precompiled archive containing OFBiz codebase, and then apply
> > the set of patches to configure it to the environment needs, recompile
> > and relaunch...
> >
> > This way of doing can cause issue when patch could not apply, after a
> > codebase modification (pretty rare but it happens).
> >
> > We are not satisfied with this way of doing, we are currently thinking
> > about using environment variables to configure technical environment
> > properties (those are on the hosting company responsibility), and to
> > keep functional specifics into the code base.
> > If you have some experience or advice in this matter, you are welcome.
> >
> > For our case, we currently have enhanced OFBiz to be able to get
> > environment variable from the operating system within property file and
> > some other configuration files (with default value if not set).
> >
> > Examples :
> > *In Property file :
> > password=${env:ONE_CONF:ofbiz} -> environment variable ONE_CONF or ofbiz
> > if unset
> > other_config=${env:OTHER_CONF:${partyId}} -> environment variable
> > OTHER_CONF or ${partyId} if unset
> > *In entityengine.xml :
> > jdbc-uri="${env:DB_POSTGRES_URI:jdbc:mysql://127.0.0.1/ofbiz?autoReconnect=true}"
> > jdbc-username="${env:DB_POSTGRES_USER:ofbiz}"
> > jdbc-password="${env:DB_POSTGRES_PWD:ofbiz}"/>
> >
> > That allow us to keep functional parameters stored within git branches.
> > Our jenkins now is able to build our 4 configured and compiled instances
> > and deliver it to the hosting company, that just have to set/check
> > environment variable (database access, activeMQ, log location, instance
> > id, etc.) before starting OFBiz app. Now we cannot have configuration
> > failure during deployment.
> >
> > We will be glad to contribute it, if it's the good way to go !
> >
> > Best Regards !
> >
> > Gil Portenseigne
> >
>

[Gil Portenseigne]

Hi James,

inline

On 06/07/2017 12:12, James Yong wrote:
> Hi Gil,
>
> Some afterthought questions with the proposed change:
> 1) Can it support multiple OFBiz instances on the same server?
Using different shell process to launch the multiple OFBiz instances, i
think that it's possible to set environment variable dedicated to each
OFBiz instance.
> 2) Can we store encrypted values for some of the environment variables like database credentials?
You can store whatever value you want into environment variable, but you
must adapt OFBiz to decrypt the value.

Thanks all for your feedback, i'll work on trunk to make it available
soon (within a new Jira)

Regards

Gil

[Gil Portenseigne]

Hi Taher

inline
On 05/07/2017 18:41, Taher Alkhateeb wrote:
> I faced something similar in the past (Not related to OFBiz). Many
> systems require different paths of configuration depending on
> different deployment scenarios.
>
> There are multiple types of configurations I can think of including:
> - platform dependent configurations (it depends on your server and the
> environment)
> - Sensitive information like passwords
> - platform independent (test vs deployment environments for example)
That was exactly our analyze, and we thought about environment variable
to be a good place to store private platform dependent configurations.
Hosting just manage an environment file to "source" before launching OFBiz.
>
> I think the first two do not belong to the code base. And so the best
> option for such problems that I used in the past is templates
> (whatever technology you choose). Templates + Build system +
> environment variables substantially simplify deployment when compared
> with patches that are highly dependent on the current state of the
> code base.
True !
>
> Now for the third kind (platform independent like multiple targets)
> maybe we can get some good ideas from ruby-on-rails [1]. Essentially
> we can tweak the build system and add a few environment files to allow
> for multiple target environments.
We currently make use of git branches that jenkins merge when deploying.
We are just starting using it, but we are satisfied...
That offers the ability to developers to customize these configurations
with no need to have access to the environment and no impact on
deployment process.

Thanks Taher for your feedback

Gil
>
> [1] http://guides.rubyonrails.org/configuring.html
>

[Gil Portenseigne]

Hi Michael

On 05/07/2017 17:56, Michael Brohl wrote:
> Hi Gil,
>
> we have similar challenges and modified OFBiz to deal with it easily.
> We offered to contribute this long time ago (2008) but it was decided
> against [1]. It was suggested to use patches instead but I think it's
> too complicated to manage several patch sets for different environments.
Addonmanager was good to manage a set of patches, but i'm glad we found
another solution to make environment configuration more stable and
efficient.
>
> We now use a staged configure mechanism which uses a base build file,
> auto detected machine name and provided parameters to decide which
> configurations should be pulled for the environment. It's currently
> Ant based and therefore does not fit into the current build mechanism
> (on the todo list).
>
> I like your approach also and I think it should be evaluated and
> discussed.
Thanks, we'll be introducing it soon :)
Thanks !
Gil

[Gil Portenseigne]

Hello Guys,

I just created https://issues.apache.org/jira/browse/OFBIZ-9498

You can test it out with the provided patch.

And James, I just tested two launch to instances with different
environment properties with no issue (different JVM args and database
connections)

Cheers

Gil

On 06/07/2017 22:07, gil portenseigne wrote:

> Hi James,
>
> inline
>
> On 06/07/2017 12:12, James Yong wrote:
>> Hi Gil,
>>
>> Some afterthought questions with the proposed change:
>> 1) Can it support multiple OFBiz instances on the same server?
> Using different shell process to launch the multiple OFBiz instances,
> i think that it's possible to set environment variable dedicated to
> each OFBiz instance.
>> 2) Can we store encrypted values for some of the environment
>> variables like database credentials?
> You can store whatever value you want into environment variable, but
> you must adapt OFBiz to decrypt the value.
>
> Thanks all for your feedback, i'll work on trunk to make it available
> soon (within a new Jira)
>
> Regards
>
> Gil
>

[taher]

Hello Gil,

Thank you for working on this. I have reviewed the patch and I have a few notes:

First for gradle, I don't think you need to fetch environment
variables. You can do whatever you want by passing flags, which might
be a simpler solution. So you create your own script and modify gradle
for your environment. You can modify the project with -P flags and the
JVM with -D flags

Now with respect to the rest of the work, I'm not sure but I think
this makes the code a little more fragile or brittle. Essentially you
are injecting multiple code locations with variables that you fetch
from the environment.

This means you are putting "state" inside application logic. One of
thepain points for me when refactoring OFBiz was that I change some
code somewhere and I get something going in flames in a completely
different location. Why does that happen? Because of this sort of
_secret_ shared state.

Why not consider a different approach like either template generation
or multiple files. For example, instead of fetching variables and
introducing a new notation in the source files, instead you can have
test.general.properties, prod.general.properties,
develop.general.properties and so on. Alternatively, you can add a
template using freemarker, and make a new gradle task (e.g.
generateConfigurationFiles).

So for example, you can have something like this:
./gradlew generateConfigurationFiles -Penvironment=test
./gradlew generateConfigurationFiles -Penvironment=production
./gradlew generateConfigurationFiles -Penvironment=development
./gradlew generateConfigurationFiles -PsourceTemplate=mySourceTemplate.ftl
./gradlew generateConfigurationFiles -PoverrideProperties=myOverride.properties

Something like this would give you the ability to customize without
adding more moving parts. Wouldn't that achieve the same results
you're seeking? It might also have the advantage of maintaining
existing notation instead of adding this ${env: ...} notation?

WDYT

On Tue, Jul 18, 2017 at 11:11 PM, gil portenseigne
<[hidden email]> wrote:

> Hello Guys,
>
> I just created https://issues.apache.org/jira/browse/OFBIZ-9498
>
> You can test it out with the provided patch.
>
> And James, I just tested two launch to instances with different environment
> properties with no issue (different JVM args and database connections)
>
> Cheers
>
> Gil
>
>
> On 06/07/2017 22:07, gil portenseigne wrote:
>>
>> Hi James,
>>
>> inline
>>
>> On 06/07/2017 12:12, James Yong wrote:
>>>
>>> Hi Gil,
>>>
>>> Some afterthought questions with the proposed change:
>>> 1) Can it support multiple OFBiz instances on the same server?
>>
>> Using different shell process to launch the multiple OFBiz instances, i
>> think that it's possible to set environment variable dedicated to each OFBiz
>> instance.
>>>
>>> 2) Can we store encrypted values for some of the environment variables
>>> like database credentials?
>>
>> You can store whatever value you want into environment variable, but you
>> must adapt OFBiz to decrypt the value.
>>
>> Thanks all for your feedback, i'll work on trunk to make it available soon
>> (within a new Jira)
>>
>> Regards
>>
>> Gil
>>
>

[Gil Portenseigne]

Hello Taher,

My thoughts inline :)


On 20/07/2017 09:05, Taher Alkhateeb wrote:
> Hello Gil,
>
> Thank you for working on this. I have reviewed the patch and I have a few notes:
>
> First for gradle, I don't think you need to fetch environment
> variables. You can do whatever you want by passing flags, which might
> be a simpler solution. So you create your own script and modify gradle
> for your environment. You can modify the project with -P flags and the
> JVM with -D flags
We did the dev into a 13.07, so without gradle. I just re-did in gradle
what we have done with ant with our needs.
I agree we should use -D flags instead of env variable :).
> Now with respect to the rest of the work, I'm not sure but I think
> this makes the code a little more fragile or brittle. Essentially you
> are injecting multiple code locations with variables that you fetch
> from the environment.
Yes there are multi location modifications since we wanted to configure
not only classic property files, but entityengine.xml,
serviceengine.xml, web.xml, etc.
I do not measure the risk of such modifications, that you're talking
about, since default value are available.
But we indeed feel the need to check what configuration is available
within OFBiz to ensure its validity (hosting company job)...  that is
another kind of fragility.
> This means you are putting "state" inside application logic. One of
> thepain points for me when refactoring OFBiz was that I change some
> code somewhere and I get something going in flames in a completely
> different location. Why does that happen? Because of this sort of
> _secret_ shared state.
I understand your feeling and opinion, but i am not sure that these
small modifications are going to create flames in the app :).
I cannot yet visualize the "state" you mentioned, now it's just a
improvement of the way to get properties value, that should be use only
in very controlled production environment.
I will re-check if there is a way to minimize the several modification
locations.

> Why not consider a different approach like either template generation
> or multiple files. For example, instead of fetching variables and
> introducing a new notation in the source files, instead you can have
> test.general.properties, prod.general.properties,
> develop.general.properties and so on. Alternatively, you can add a
> template using freemarker, and make a new gradle task (e.g.
> generateConfigurationFiles).
>
> So for example, you can have something like this:
> ./gradlew generateConfigurationFiles -Penvironment=test
> ./gradlew generateConfigurationFiles -Penvironment=production
> ./gradlew generateConfigurationFiles -Penvironment=development
> ./gradlew generateConfigurationFiles -PsourceTemplate=mySourceTemplate.ftl
> ./gradlew generateConfigurationFiles -PoverrideProperties=myOverride.properties
>
> Something like this would give you the ability to customize without
> adding more moving parts. Wouldn't that achieve the same results
> you're seeking? It might also have the advantage of maintaining
> existing notation instead of adding this ${env: ...} notation?

The original needs was to avoid modifying files in production (via
patches/addons), to let hosting company sys admin to manage the
configurations changes within the environment without touching code base.
With your proposal, first we have to explain to them how to apply
specific configuration using gradle (not a big deal), then my concern is
that these template will have to be maintained by the sys admin, i.e.
when we are adding some config in a property file and this one is
already templated, communication is needed to explain what template
should be adapted etc.

With our proposal, no communication with the hosting company is needed,
but only when new environment variable is needed, and, no code base
modification is done on the VM after delivery.

I like the template proposal (currently we use git branches to manage
this need), this could be complementary with the "environment notation".

Thanks for your feedback !

Gil

> WDYT
>
> On Tue, Jul 18, 2017 at 11:11 PM, gil portenseigne
> <[hidden email]>  wrote:
>> Hello Guys,
>>
>> I just createdhttps://issues.apache.org/jira/browse/OFBIZ-9498
>>
>> You can test it out with the provided patch.
>>
>> And James, I just tested two launch to instances with different environment
>> properties with no issue (different JVM args and database connections)
>>
>> Cheers
>>
>> Gil
>>
>>
>> On 06/07/2017 22:07, gil portenseigne wrote:
>>> Hi James,
>>>
>>> inline
>>>
>>> On 06/07/2017 12:12, James Yong wrote:
>>>> Hi Gil,
>>>>
>>>> Some afterthought questions with the proposed change:
>>>> 1) Can it support multiple OFBiz instances on the same server?
>>> Using different shell process to launch the multiple OFBiz instances, i
>>> think that it's possible to set environment variable dedicated to each OFBiz
>>> instance.
>>>> 2) Can we store encrypted values for some of the environment variables
>>>> like database credentials?
>>> You can store whatever value you want into environment variable, but you
>>> must adapt OFBiz to decrypt the value.
>>>
>>> Thanks all for your feedback, i'll work on trunk to make it available soon
>>> (within a new Jira)
>>>
>>> Regards
>>>
>>> Gil
>>>

[taher]

Hi Gil, good feedback, inline ...

On Mon, Jul 24, 2017 at 10:17 AM, gil portenseigne
<[hidden email]> wrote:

> Hello Taher,
>
> My thoughts inline :)
>
>
> On 20/07/2017 09:05, Taher Alkhateeb wrote:
>>
>> Hello Gil,
>>
>> Thank you for working on this. I have reviewed the patch and I have a few
>> notes:
>>
>> First for gradle, I don't think you need to fetch environment
>> variables. You can do whatever you want by passing flags, which might
>> be a simpler solution. So you create your own script and modify gradle
>> for your environment. You can modify the project with -P flags and the
>> JVM with -D flags
>
> We did the dev into a 13.07, so without gradle. I just re-did in gradle what
> we have done with ant with our needs.
> I agree we should use -D flags instead of env variable :).
>>
>> Now with respect to the rest of the work, I'm not sure but I think
>> this makes the code a little more fragile or brittle. Essentially you
>> are injecting multiple code locations with variables that you fetch
>> from the environment.
>
> Yes there are multi location modifications since we wanted to configure not
> only classic property files, but entityengine.xml, serviceengine.xml,
> web.xml, etc.
> I do not measure the risk of such modifications, that you're talking about,
> since default value are available.
> But we indeed feel the need to check what configuration is available within
> OFBiz to ensure its validity (hosting company job)...  that is another kind
> of fragility.
>>
>> This means you are putting "state" inside application logic. One of
>> thepain points for me when refactoring OFBiz was that I change some
>> code somewhere and I get something going in flames in a completely
>> different location. Why does that happen? Because of this sort of
>> _secret_ shared state.
>
> I understand your feeling and opinion, but i am not sure that these small
> modifications are going to create flames in the app :).

Of course not :) it was an exaggeration (and my crappy attempt at some
humor). But you know what I mean right? We slowly introduce moving
parts (state) to the code which after years pile up to become a
monster. Most of my refactoring work lately is targeting removing
"state". It's very small things like this that keep getting bigger and
bigger.

> I cannot yet visualize the "state" you mentioned, now it's just a
> improvement of the way to get properties value, that should be use only in
> very controlled production environment.
> I will re-check if there is a way to minimize the several modification
> locations.
>

So allow my to explain what my interpretation of the word state. State
is essentially anything that is persisted in memory. An object is
state, a class instance field is state, an environment variable is
state, a thread is state, a process is state, a global variable is
state, and so on.

I am very much afraid of state and I try to minimize it in any code
base. Why do I do that? Well, if you have a variable (configuration in
this case) that might take multiple shapes, and you don't know at
runtime which shape would it be, then you've added a moving part. So
calling the same piece of code will not give you the same results
every time. This means code that is harder to understand, harder to
test (you need to do prep work) and harder to refactor. On the other
hand, if you do not touch the code at all, and instead do your
configuration preparation outside (using the build system for example
to generate configurations) then you avoid a whole set of problems
that come with this added logic.

>> Why not consider a different approach like either template generation
>> or multiple files. For example, instead of fetching variables and
>> introducing a new notation in the source files, instead you can have
>> test.general.properties, prod.general.properties,
>> develop.general.properties and so on. Alternatively, you can add a
>> template using freemarker, and make a new gradle task (e.g.
>> generateConfigurationFiles).
>>
>> So for example, you can have something like this:
>> ./gradlew generateConfigurationFiles -Penvironment=test
>> ./gradlew generateConfigurationFiles -Penvironment=production
>> ./gradlew generateConfigurationFiles -Penvironment=development
>> ./gradlew generateConfigurationFiles -PsourceTemplate=mySourceTemplate.ftl
>> ./gradlew generateConfigurationFiles
>> -PoverrideProperties=myOverride.properties
>>
>> Something like this would give you the ability to customize without
>> adding more moving parts. Wouldn't that achieve the same results
>> you're seeking? It might also have the advantage of maintaining
>> existing notation instead of adding this ${env: ...} notation?
>
> The original needs was to avoid modifying files in production (via
> patches/addons), to let hosting company sys admin to manage the
> configurations changes within the environment without touching code base.
> With your proposal, first we have to explain to them how to apply specific
> configuration using gradle (not a big deal), then my concern is that these
> template will have to be maintained by the sys admin, i.e. when we are
> adding some config in a property file and this one is already templated,
> communication is needed to explain what template should be adapted etc.

Okay, let's say we can have two types of templates. Ones that are
generic enough, and flexible enough, and can be maintained by us (the
community) because of their generic nature. For example
(production.properties, test.properties, develop.properties). Then we
have another type of templates that are very specific to your
environment. In that case, yes you need to maintain them yourself, but
your life would be much easier using a build system right? In fact we
can create generic tasks to generate specific templates as well.

>
> With our proposal, no communication with the hosting company is needed, but
> only when new environment variable is needed, and, no code base modification
> is done on the VM after delivery.

I'm not sure, but most modern deployments nowadays are fully
automated. You don't patch, or SSH to server to fix something. The
whole thing is managed by code. You commit something, the CI server
runs the tests and then automatically deploys to the production
environment a whole new fresh instance with the database and
everything. Communicating with the hosting company means this is done
very manually. Doesn't your hosting company have a set of web APIs to
control your VM instances?

>
> I like the template proposal (currently we use git branches to manage this
> need), this could be complementary with the "environment notation".

If you need any help with that just point me to where I might be of help.

>
> Thanks for your feedback !

Thank you for taking on this initiative :) I look forward to getting it done.

>
> Gil
>
>> WDYT
>>
>> On Tue, Jul 18, 2017 at 11:11 PM, gil portenseigne
>> <[hidden email]>  wrote:
>>>
>>> Hello Guys,
>>>
>>> I just createdhttps://issues.apache.org/jira/browse/OFBIZ-9498
>>>
>>> You can test it out with the provided patch.
>>>
>>> And James, I just tested two launch to instances with different
>>> environment
>>> properties with no issue (different JVM args and database connections)
>>>
>>> Cheers
>>>
>>> Gil
>>>
>>>
>>> On 06/07/2017 22:07, gil portenseigne wrote:
>>>>
>>>> Hi James,
>>>>
>>>> inline
>>>>
>>>> On 06/07/2017 12:12, James Yong wrote:
>>>>>
>>>>> Hi Gil,
>>>>>
>>>>> Some afterthought questions with the proposed change:
>>>>> 1) Can it support multiple OFBiz instances on the same server?
>>>>
>>>> Using different shell process to launch the multiple OFBiz instances, i
>>>> think that it's possible to set environment variable dedicated to each
>>>> OFBiz
>>>> instance.
>>>>>
>>>>> 2) Can we store encrypted values for some of the environment variables
>>>>> like database credentials?
>>>>
>>>> You can store whatever value you want into environment variable, but you
>>>> must adapt OFBiz to decrypt the value.
>>>>
>>>> Thanks all for your feedback, i'll work on trunk to make it available
>>>> soon
>>>> (within a new Jira)
>>>>
>>>> Regards
>>>>
>>>> Gil
>>>>
>

[Gil Portenseigne]

Hello Taher !

Thanks for your patience and your explanation, it's clearer now.

Inline (sorry if it's unclear i wrote it as i thought)

On 26/07/2017 07:58, Taher Alkhateeb wrote:
[...]
>> I understand your feeling and opinion, but i am not sure that these small
>> modifications are going to create flames in the app :).
> Of course not :) it was an exaggeration (and my crappy attempt at some
> humor). But you know what I mean right? We slowly introduce moving
> parts (state) to the code which after years pile up to become a
> monster. Most of my refactoring work lately is targeting removing
> "state". It's very small things like this that keep getting bigger and
> bigger.

Ok :)

>
>> I cannot yet visualize the "state" you mentioned, now it's just a
>> improvement of the way to get properties value, that should be use only in
>> very controlled production environment.
>> I will re-check if there is a way to minimize the several modification
>> locations.
>>
> So allow my to explain what my interpretation of the word state. State
> is essentially anything that is persisted in memory. An object is
> state, a class instance field is state, an environment variable is
> state, a thread is state, a process is state, a global variable is
> state, and so on.
>
> I am very much afraid of state and I try to minimize it in any code
> base. Why do I do that? Well, if you have a variable (configuration in
> this case) that might take multiple shapes, and you don't know at
> runtime which shape would it be, then you've added a moving part. So
> calling the same piece of code will not give you the same results
> every time. This means code that is harder to understand, harder to
> test (you need to do prep work) and harder to refactor. On the other
> hand, if you do not touch the code at all, and instead do your
> configuration preparation outside (using the build system for example
> to generate configurations) then you avoid a whole set of problems
> that come with this added logic.

Many thanks for the details, now i better grasp the state definition you
use, and fully understand the issue beneath it.
Since it's a very light add of code, that might be reduced, and moreover
the *state* being optional in the fact that it is only used for specific
needs that are intensely tested by the hosting company (preprod etc.)
I'm yet not convinced about giving it up :).
But i will continue the mail answer with the goal of finding another
suitable solution.

>
>>> Why not consider a different approach like either template generation
>>> or multiple files. For example, instead of fetching variables and
>>> introducing a new notation in the source files, instead you can have
>>> test.general.properties, prod.general.properties,
>>> develop.general.properties and so on. Alternatively, you can add a
>>> template using freemarker, and make a new gradle task (e.g.
>>> generateConfigurationFiles).
>>>
>>> So for example, you can have something like this:
>>> ./gradlew generateConfigurationFiles -Penvironment=test
>>> ./gradlew generateConfigurationFiles -Penvironment=production
>>> ./gradlew generateConfigurationFiles -Penvironment=development
>>> ./gradlew generateConfigurationFiles -PsourceTemplate=mySourceTemplate.ftl
>>> ./gradlew generateConfigurationFiles
>>> -PoverrideProperties=myOverride.properties
>>>
>>> Something like this would give you the ability to customize without
>>> adding more moving parts. Wouldn't that achieve the same results
>>> you're seeking? It might also have the advantage of maintaining
>>> existing notation instead of adding this ${env: ...} notation?
>> The original needs was to avoid modifying files in production (via
>> patches/addons), to let hosting company sys admin to manage the
>> configurations changes within the environment without touching code base.
>> With your proposal, first we have to explain to them how to apply specific
>> configuration using gradle (not a big deal), then my concern is that these
>> template will have to be maintained by the sys admin, i.e. when we are
>> adding some config in a property file and this one is already templated,
>> communication is needed to explain what template should be adapted etc.
> Okay, let's say we can have two types of templates. Ones that are
> generic enough, and flexible enough, and can be maintained by us (the
> community) because of their generic nature. For example
> (production.properties, test.properties, develop.properties). Then we
> have another type of templates that are very specific to your
> environment. In that case, yes you need to maintain them yourself, but
> your life would be much easier using a build system right? In fact we
> can create generic tasks to generate specific templates as well.

The major need i must serve is the resilience to configuration
modification, and we must work on these hosting config "templates" to
get it resilient enough...

>> With our proposal, no communication with the hosting company is needed, but
>> only when new environment variable is needed, and, no code base modification
>> is done on the VM after delivery.
> I'm not sure, but most modern deployments nowadays are fully
> automated. You don't patch, or SSH to server to fix something. The
> whole thing is managed by code. You commit something, the CI server
> runs the tests and then automatically deploys to the production
> environment a whole new fresh instance with the database and
> everything. Communicating with the hosting company means this is done
> very manually. Doesn't your hosting company have a set of web APIs to
> control your VM instances?

The communication i talked about occurred when automated deployment (via
rundeck) failed due to bad configuration system we used to have
(addonmanager), and when we did add new properties in our hot-deploy
component.
Moreover database and other configuration were managed by a set of
patches, and that could be where we failed.
With this observation we had to find an easy to manage solution.

After this thread discussion, i now see another solution :
* Using gradle to apply templated global environment configuration (we
currently do it with git branches)
* Find a way to use more templates to apply specific secret
configurations that are on the hosting company responsibility. (Env
variable)
** Those config must not be stored within our company SCM, but only
managed by hosting
** Those config must be simple to manage

The more I think about it, we must have mistaken the configuration
separation between pure technical and functional ones. That's where the
resilience lies.
Separating them better should allow to prepare for the hosting the
templates to store on the VM, to apply to each deployment.

Environment Variable will still be used within our customer project in
production, since it's a very *simple to manage* configuration (with ant
build system),  and it will remain available through Jira for those
interested.

With this work achieved, I think that a wiki page introducing these
concept should be a good way to introduce "how to manage configurations
with template in production environment"...  But only after building the
gradle tasks :).

>
>> I like the template proposal (currently we use git branches to manage this
>> need), this could be complementary with the "environment notation".
> If you need any help with that just point me to where I might be of help

Thanks again, i'd appreciate to have more feedback from those who
already are using templates, then I'll be glad to implement something
around this subject if Env Variable configuration is rejected :)

Regards,

Gil

>
>> Thanks for your feedback !
> Thank you for taking on this initiative :) I look forward to getting it done.
>
>> Gil
>>
>>> WDYT
>>>
>>> On Tue, Jul 18, 2017 at 11:11 PM, gil portenseigne
>>> <[hidden email]>  wrote:
>>>> Hello Guys,
>>>>
>>>> I just createdhttps://issues.apache.org/jira/browse/OFBIZ-9498
>>>>
>>>> You can test it out with the provided patch.
>>>>
>>>> And James, I just tested two launch to instances with different
>>>> environment
>>>> properties with no issue (different JVM args and database connections)
>>>>
>>>> Cheers
>>>>
>>>> Gil
>>>>
>>>>
>>>> On 06/07/2017 22:07, gil portenseigne wrote:
>>>>> Hi James,
>>>>>
>>>>> inline
>>>>>
>>>>> On 06/07/2017 12:12, James Yong wrote:
>>>>>> Hi Gil,
>>>>>>
>>>>>> Some afterthought questions with the proposed change:
>>>>>> 1) Can it support multiple OFBiz instances on the same server?
>>>>> Using different shell process to launch the multiple OFBiz instances, i
>>>>> think that it's possible to set environment variable dedicated to each
>>>>> OFBiz
>>>>> instance.
>>>>>> 2) Can we store encrypted values for some of the environment variables
>>>>>> like database credentials?
>>>>> You can store whatever value you want into environment variable, but you
>>>>> must adapt OFBiz to decrypt the value.
>>>>>
>>>>> Thanks all for your feedback, i'll work on trunk to make it available
>>>>> soon
>>>>> (within a new Jira)
>>>>>
>>>>> Regards
>>>>>
>>>>> Gil
>>>>>

[Michael Brohl-3]

Just an update triggered by the question from Swapnil [1]: our
configuration mechanism mentioned below is now on Gradle so it would be
compatible with 16.11 and later.

Regards,

Michael

[1]
https://lists.apache.org/thread.html/703f3e615a93a2a83fb92b122eb8275fb05aa27537d95342815dd043@%3Cdev.ofbiz.apache.org%3E


Am 05.07.17 um 17:56 schrieb Michael Brohl:

> Hi Gil,
>
> we have similar challenges and modified OFBiz to deal with it easily.
> We offered to contribute this long time ago (2008) but it was decided
> against [1]. It was suggested to use patches instead but I think it's
> too complicated to manage several patch sets for different environments.
>
> We now use a staged configure mechanism which uses a base build file,
> auto detected machine name and provided parameters to decide which
> configurations should be pulled for the environment. It's currently
> Ant based and therefore does not fit into the current build mechanism
> (on the todo list).
>
> I like your approach also and I think it should be evaluated and
> discussed.
>
> Best regards,
>
> Michael Brohl
> ecomify GmbH
> www.ecomify.de
>
> [1]
> https://lists.apache.org/thread.html/b95e239250880d9a5b34268b3b711f0f8f7f0540a26bb41c5ced493a@1213087551@%3Cdev.ofbiz.apache.org%3E
>
>
>
> Am 05.07.17 um 17:36 schrieb gil portenseigne:
>> Hello all,
>>
>> Working with different hosting companies, we used to have issues when
>> deploying OFBiz concerning technical configuration of the different
>> environments.
>> We are writing this mail to get feedback from the community and
>> eventually propose to improve OFBiz on this matter.
>>
>> For a customer, we are working with 4 instances of a release 13.07
>> OFBiz, and are currently using a set of patches (with
>> addonmanager...) to manage environment specific configurations.
>> During each production deployment, the hosting company receive from
>> our jenkins a precompiled archive containing OFBiz codebase, and then
>> apply the set of patches to configure it to the environment needs,
>> recompile and relaunch...
>>
>> This way of doing can cause issue when patch could not apply, after a
>> codebase modification (pretty rare but it happens).
>>
>> We are not satisfied with this way of doing, we are currently
>> thinking about using environment variables to configure technical
>> environment properties (those are on the hosting company
>> responsibility), and to keep functional specifics into the code base.
>> If you have some experience or advice in this matter, you are welcome.
>>
>> For our case, we currently have enhanced OFBiz to be able to get
>> environment variable from the operating system within property file
>> and some other configuration files (with default value if not set).
>>
>> Examples :
>> *In Property file :
>> password=${env:ONE_CONF:ofbiz} -> environment variable ONE_CONF or
>> ofbiz if unset
>> other_config=${env:OTHER_CONF:${partyId}} -> environment variable
>> OTHER_CONF or ${partyId} if unset
>> *In entityengine.xml :
>> jdbc-uri="${env:DB_POSTGRES_URI:jdbc:mysql://127.0.0.1/ofbiz?autoReconnect=true}"
>>
>> jdbc-username="${env:DB_POSTGRES_USER:ofbiz}"
>> jdbc-password="${env:DB_POSTGRES_PWD:ofbiz}"/>
>>
>> That allow us to keep functional parameters stored within git
>> branches. Our jenkins now is able to build our 4 configured and
>> compiled instances and deliver it to the hosting company, that just
>> have to set/check environment variable (database access, activeMQ,
>> log location, instance id, etc.) before starting OFBiz app. Now we
>> cannot have configuration failure during deployment.
>>
>> We will be glad to contribute it, if it's the good way to go !
>>
>> Best Regards !
>>
>> Gil Portenseigne
>>
>
>

[Jacques Le Roux]

That's quite interesting Michael,

Would you share in a Jira? Then we could get to merge all experiences and find a consensu.

Jacques


Le 03/11/2017 à 10:10, Michael Brohl a écrit :

> Just an update triggered by the question from Swapnil [1]: our configuration mechanism mentioned below is now on Gradle so it would be compatible
> with 16.11 and later.
>
> Regards,
>
> Michael
>
> [1] https://lists.apache.org/thread.html/703f3e615a93a2a83fb92b122eb8275fb05aa27537d95342815dd043@%3Cdev.ofbiz.apache.org%3E
>
>
> Am 05.07.17 um 17:56 schrieb Michael Brohl:
>> Hi Gil,
>>
>> we have similar challenges and modified OFBiz to deal with it easily. We offered to contribute this long time ago (2008) but it was decided against
>> [1]. It was suggested to use patches instead but I think it's too complicated to manage several patch sets for different environments.
>>
>> We now use a staged configure mechanism which uses a base build file, auto detected machine name and provided parameters to decide which
>> configurations should be pulled for the environment. It's currently Ant based and therefore does not fit into the current build mechanism (on the
>> todo list).
>>
>> I like your approach also and I think it should be evaluated and discussed.
>>
>> Best regards,
>>
>> Michael Brohl
>> ecomify GmbH
>> www.ecomify.de
>>
>> [1] https://lists.apache.org/thread.html/b95e239250880d9a5b34268b3b711f0f8f7f0540a26bb41c5ced493a@1213087551@%3Cdev.ofbiz.apache.org%3E
>>
>>
>>
>> Am 05.07.17 um 17:36 schrieb gil portenseigne:
>>> Hello all,
>>>
>>> Working with different hosting companies, we used to have issues when deploying OFBiz concerning technical configuration of the different
>>> environments.
>>> We are writing this mail to get feedback from the community and eventually propose to improve OFBiz on this matter.
>>>
>>> For a customer, we are working with 4 instances of a release 13.07 OFBiz, and are currently using a set of patches (with addonmanager...) to
>>> manage environment specific configurations.
>>> During each production deployment, the hosting company receive from our jenkins a precompiled archive containing OFBiz codebase, and then apply
>>> the set of patches to configure it to the environment needs, recompile and relaunch...
>>>
>>> This way of doing can cause issue when patch could not apply, after a codebase modification (pretty rare but it happens).
>>>
>>> We are not satisfied with this way of doing, we are currently thinking about using environment variables to configure technical environment
>>> properties (those are on the hosting company responsibility), and to keep functional specifics into the code base.
>>> If you have some experience or advice in this matter, you are welcome.
>>>
>>> For our case, we currently have enhanced OFBiz to be able to get environment variable from the operating system within property file and some
>>> other configuration files (with default value if not set).
>>>
>>> Examples :
>>> *In Property file :
>>> password=${env:ONE_CONF:ofbiz} -> environment variable ONE_CONF or ofbiz if unset
>>> other_config=${env:OTHER_CONF:${partyId}} -> environment variable OTHER_CONF or ${partyId} if unset
>>> *In entityengine.xml :
>>> jdbc-uri="${env:DB_POSTGRES_URI:jdbc:mysql://127.0.0.1/ofbiz?autoReconnect=true}"
>>> jdbc-username="${env:DB_POSTGRES_USER:ofbiz}"
>>> jdbc-password="${env:DB_POSTGRES_PWD:ofbiz}"/>
>>>
>>> That allow us to keep functional parameters stored within git branches. Our jenkins now is able to build our 4 configured and compiled instances
>>> and deliver it to the hosting company, that just have to set/check environment variable (database access, activeMQ, log location, instance id,
>>> etc.) before starting OFBiz app. Now we cannot have configuration failure during deployment.
>>>
>>> We will be glad to contribute it, if it's the good way to go !
>>>
>>> Best Regards !
>>>
>>> Gil Portenseigne
>>>
>>
>>
>
>

[Swapnil Mane]

Thank you Michael! :-)


- Best Regards,
Swapnil M Mane

On Fri, Nov 3, 2017 at 2:40 PM, Michael Brohl <[hidden email]>
wrote:

> Just an update triggered by the question from Swapnil [1]: our
> configuration mechanism mentioned below is now on Gradle so it would be
> compatible with 16.11 and later.
>
> Regards,
>
> Michael
>
> [1] https://lists.apache.org/thread.html/703f3e615a93a2a83fb92b1
> 22eb8275fb05aa27537d95342815dd043@%3Cdev.ofbiz.apache.org%3E
>
>
> Am 05.07.17 um 17:56 schrieb Michael Brohl:
>
> Hi Gil,
>>
>> we have similar challenges and modified OFBiz to deal with it easily. We
>> offered to contribute this long time ago (2008) but it was decided against
>> [1]. It was suggested to use patches instead but I think it's too
>> complicated to manage several patch sets for different environments.
>>
>> We now use a staged configure mechanism which uses a base build file,
>> auto detected machine name and provided parameters to decide which
>> configurations should be pulled for the environment. It's currently Ant
>> based and therefore does not fit into the current build mechanism (on the
>> todo list).
>>
>> I like your approach also and I think it should be evaluated and
>> discussed.
>>
>> Best regards,
>>
>> Michael Brohl
>> ecomify GmbH
>> www.ecomify.de
>>
>> [1] https://lists.apache.org/thread.html/b95e239250880d9a5b34268
>> b3b711f0f8f7f0540a26bb41c5ced493a@1213087551@%3Cdev.ofbiz.apache.org%3E
>>
>>
>>
>> Am 05.07.17 um 17:36 schrieb gil portenseigne:
>>
>>> Hello all,
>>>
>>> Working with different hosting companies, we used to have issues when
>>> deploying OFBiz concerning technical configuration of the different
>>> environments.
>>> We are writing this mail to get feedback from the community and
>>> eventually propose to improve OFBiz on this matter.
>>>
>>> For a customer, we are working with 4 instances of a release 13.07
>>> OFBiz, and are currently using a set of patches (with addonmanager...) to
>>> manage environment specific configurations.
>>> During each production deployment, the hosting company receive from our
>>> jenkins a precompiled archive containing OFBiz codebase, and then apply the
>>> set of patches to configure it to the environment needs, recompile and
>>> relaunch...
>>>
>>> This way of doing can cause issue when patch could not apply, after a
>>> codebase modification (pretty rare but it happens).
>>>
>>> We are not satisfied with this way of doing, we are currently thinking
>>> about using environment variables to configure technical environment
>>> properties (those are on the hosting company responsibility), and to keep
>>> functional specifics into the code base.
>>> If you have some experience or advice in this matter, you are welcome.
>>>
>>> For our case, we currently have enhanced OFBiz to be able to get
>>> environment variable from the operating system within property file and
>>> some other configuration files (with default value if not set).
>>>
>>> Examples :
>>> *In Property file :
>>> password=${env:ONE_CONF:ofbiz} -> environment variable ONE_CONF or ofbiz
>>> if unset
>>> other_config=${env:OTHER_CONF:${partyId}} -> environment variable
>>> OTHER_CONF or ${partyId} if unset
>>> *In entityengine.xml :
>>> jdbc-uri="${env:DB_POSTGRES_URI:jdbc:mysql://127.0.0.1/ofbiz
>>> ?autoReconnect=true}"
>>> jdbc-username="${env:DB_POSTGRES_USER:ofbiz}"
>>> jdbc-password="${env:DB_POSTGRES_PWD:ofbiz}"/>
>>>
>>> That allow us to keep functional parameters stored within git branches.
>>> Our jenkins now is able to build our 4 configured and compiled instances
>>> and deliver it to the hosting company, that just have to set/check
>>> environment variable (database access, activeMQ, log location, instance id,
>>> etc.) before starting OFBiz app. Now we cannot have configuration failure
>>> during deployment.
>>>
>>> We will be glad to contribute it, if it's the good way to go !
>>>
>>> Best Regards !
>>>
>>> Gil Portenseigne
>>>
>>>
>>
>>
>
>

[Jacques Le Roux]

Le 03/11/2017 à 11:39, Jacques Le Roux a écrit :
> That's quite interesting Michael,
>
> Would you share in a Jira? Then we could get to merge all experiences and find a consensu.
OK Gil created one already in July https://issues.apache.org/jira/browse/OFBIZ-9498

>
> Jacques
>
>
> Le 03/11/2017 à 10:10, Michael Brohl a écrit :
>> Just an update triggered by the question from Swapnil [1]: our configuration mechanism mentioned below is now on Gradle so it would be compatible
>> with 16.11 and later.
>>
>> Regards,
>>
>> Michael
>>
>> [1] https://lists.apache.org/thread.html/703f3e615a93a2a83fb92b122eb8275fb05aa27537d95342815dd043@%3Cdev.ofbiz.apache.org%3E
>>
>>
>> Am 05.07.17 um 17:56 schrieb Michael Brohl:
>>> Hi Gil,
>>>
>>> we have similar challenges and modified OFBiz to deal with it easily. We offered to contribute this long time ago (2008) but it was decided
>>> against [1]. It was suggested to use patches instead but I think it's too complicated to manage several patch sets for different environments.
>>>
>>> We now use a staged configure mechanism which uses a base build file, auto detected machine name and provided parameters to decide which
>>> configurations should be pulled for the environment. It's currently Ant based and therefore does not fit into the current build mechanism (on the
>>> todo list).
>>>
>>> I like your approach also and I think it should be evaluated and discussed.
>>>
>>> Best regards,
>>>
>>> Michael Brohl
>>> ecomify GmbH
>>> www.ecomify.de
>>>
>>> [1] https://lists.apache.org/thread.html/b95e239250880d9a5b34268b3b711f0f8f7f0540a26bb41c5ced493a@1213087551@%3Cdev.ofbiz.apache.org%3E
>>>
>>>
>>>
>>> Am 05.07.17 um 17:36 schrieb gil portenseigne:
>>>> Hello all,
>>>>
>>>> Working with different hosting companies, we used to have issues when deploying OFBiz concerning technical configuration of the different
>>>> environments.
>>>> We are writing this mail to get feedback from the community and eventually propose to improve OFBiz on this matter.
>>>>
>>>> For a customer, we are working with 4 instances of a release 13.07 OFBiz, and are currently using a set of patches (with addonmanager...) to
>>>> manage environment specific configurations.
>>>> During each production deployment, the hosting company receive from our jenkins a precompiled archive containing OFBiz codebase, and then apply
>>>> the set of patches to configure it to the environment needs, recompile and relaunch...
>>>>
>>>> This way of doing can cause issue when patch could not apply, after a codebase modification (pretty rare but it happens).
>>>>
>>>> We are not satisfied with this way of doing, we are currently thinking about using environment variables to configure technical environment
>>>> properties (those are on the hosting company responsibility), and to keep functional specifics into the code base.
>>>> If you have some experience or advice in this matter, you are welcome.
>>>>
>>>> For our case, we currently have enhanced OFBiz to be able to get environment variable from the operating system within property file and some
>>>> other configuration files (with default value if not set).
>>>>
>>>> Examples :
>>>> *In Property file :
>>>> password=${env:ONE_CONF:ofbiz} -> environment variable ONE_CONF or ofbiz if unset
>>>> other_config=${env:OTHER_CONF:${partyId}} -> environment variable OTHER_CONF or ${partyId} if unset
>>>> *In entityengine.xml :
>>>> jdbc-uri="${env:DB_POSTGRES_URI:jdbc:mysql://127.0.0.1/ofbiz?autoReconnect=true}"
>>>> jdbc-username="${env:DB_POSTGRES_USER:ofbiz}"
>>>> jdbc-password="${env:DB_POSTGRES_PWD:ofbiz}"/>
>>>>
>>>> That allow us to keep functional parameters stored within git branches. Our jenkins now is able to build our 4 configured and compiled instances
>>>> and deliver it to the hosting company, that just have to set/check environment variable (database access, activeMQ, log location, instance id,
>>>> etc.) before starting OFBiz app. Now we cannot have configuration failure during deployment.
>>>>
>>>> We will be glad to contribute it, if it's the good way to go !
>>>>
>>>> Best Regards !
>>>>
>>>> Gil Portenseigne
>>>>
>>>
>>>
>>
>>
>
>

[Michael Brohl-3]

Hi Jacques,

it takes some effort to make an OFBiz standard compatible patch of the
mechanism because we have several additions to the configurations. I
would take the effort if the community wants to adapt it but it's too
much work for just giving an idea.

I have explained the mechanism in [1]. There it is based on Ant but we
already use it in projects with Gradle.

We were looking for other solutions during the migration to Gradle but
haven't found a better approach considering all pros and cons (database
configuration, environment variables etc.). We use it for years on our
test- and production environments and it makes the handling of different
system specific configurations very easy.

We are thinking about the introduction of an additional configuration
level so that we have the base configuration (containing all
properties), a project configuration level and the system specific
configurations. This helps us to additionally maintain projects using
different sets of plugins.

I'm happy to explain more if something is unclear.

Regards,

Michael


[1]
https://lists.apache.org/thread.html/b95e239250880d9a5b34268b3b711f0f8f7f0540a26bb41c5ced493a@1213087551@%3Cdev.ofbiz.apache.org%3E


Am 03.11.17 um 11:39 schrieb Jacques Le Roux:

> That's quite interesting Michael,
>
> Would you share in a Jira? Then we could get to merge all experiences
> and find a consensu.
>
> Jacques
>
>
> Le 03/11/2017 à 10:10, Michael Brohl a écrit :
>> Just an update triggered by the question from Swapnil [1]: our
>> configuration mechanism mentioned below is now on Gradle so it would
>> be compatible with 16.11 and later.
>>
>> Regards,
>>
>> Michael
>>
>> [1]
>> https://lists.apache.org/thread.html/703f3e615a93a2a83fb92b122eb8275fb05aa27537d95342815dd043@%3Cdev.ofbiz.apache.org%3E
>>
>>
>> Am 05.07.17 um 17:56 schrieb Michael Brohl:
>>> Hi Gil,
>>>
>>> we have similar challenges and modified OFBiz to deal with it
>>> easily. We offered to contribute this long time ago (2008) but it
>>> was decided against [1]. It was suggested to use patches instead but
>>> I think it's too complicated to manage several patch sets for
>>> different environments.
>>>
>>> We now use a staged configure mechanism which uses a base build
>>> file, auto detected machine name and provided parameters to decide
>>> which configurations should be pulled for the environment. It's
>>> currently Ant based and therefore does not fit into the current
>>> build mechanism (on the todo list).
>>>
>>> I like your approach also and I think it should be evaluated and
>>> discussed.
>>>
>>> Best regards,
>>>
>>> Michael Brohl
>>> ecomify GmbH
>>> www.ecomify.de
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/b95e239250880d9a5b34268b3b711f0f8f7f0540a26bb41c5ced493a@1213087551@%3Cdev.ofbiz.apache.org%3E
>>>
>>>
>>>
>>> Am 05.07.17 um 17:36 schrieb gil portenseigne:
>>>> Hello all,
>>>>
>>>> Working with different hosting companies, we used to have issues
>>>> when deploying OFBiz concerning technical configuration of the
>>>> different environments.
>>>> We are writing this mail to get feedback from the community and
>>>> eventually propose to improve OFBiz on this matter.
>>>>
>>>> For a customer, we are working with 4 instances of a release 13.07
>>>> OFBiz, and are currently using a set of patches (with
>>>> addonmanager...) to manage environment specific configurations.
>>>> During each production deployment, the hosting company receive from
>>>> our jenkins a precompiled archive containing OFBiz codebase, and
>>>> then apply the set of patches to configure it to the environment
>>>> needs, recompile and relaunch...
>>>>
>>>> This way of doing can cause issue when patch could not apply, after
>>>> a codebase modification (pretty rare but it happens).
>>>>
>>>> We are not satisfied with this way of doing, we are currently
>>>> thinking about using environment variables to configure technical
>>>> environment properties (those are on the hosting company
>>>> responsibility), and to keep functional specifics into the code base.
>>>> If you have some experience or advice in this matter, you are welcome.
>>>>
>>>> For our case, we currently have enhanced OFBiz to be able to get
>>>> environment variable from the operating system within property file
>>>> and some other configuration files (with default value if not set).
>>>>
>>>> Examples :
>>>> *In Property file :
>>>> password=${env:ONE_CONF:ofbiz} -> environment variable ONE_CONF or
>>>> ofbiz if unset
>>>> other_config=${env:OTHER_CONF:${partyId}} -> environment variable
>>>> OTHER_CONF or ${partyId} if unset
>>>> *In entityengine.xml :
>>>> jdbc-uri="${env:DB_POSTGRES_URI:jdbc:mysql://127.0.0.1/ofbiz?autoReconnect=true}"
>>>>
>>>> jdbc-username="${env:DB_POSTGRES_USER:ofbiz}"
>>>> jdbc-password="${env:DB_POSTGRES_PWD:ofbiz}"/>
>>>>
>>>> That allow us to keep functional parameters stored within git
>>>> branches. Our jenkins now is able to build our 4 configured and
>>>> compiled instances and deliver it to the hosting company, that just
>>>> have to set/check environment variable (database access, activeMQ,
>>>> log location, instance id, etc.) before starting OFBiz app. Now we
>>>> cannot have configuration failure during deployment.
>>>>
>>>> We will be glad to contribute it, if it's the good way to go !
>>>>
>>>> Best Regards !
>>>>
>>>> Gil Portenseigne
>>>>
>>>
>>>
>>
>>
>

[Jacques Le Roux]

Thanks Michael,

I remember this discussion. My aim here is to replace the consensual advice we had with pre Gradle version, still at

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65865831#AddressingCustomRequirementsInOFBiz-PatchManagementUsingAntTargets

I think we can suggest more than one solution and people can then pick the one fitting more for them. Maybe you could begin to add yours in the
current AddressingCustomRequirementsInOFBiz page.

I'll suggest the same for Gil's with OFBIZ-9498.

Hope it clarifies things

Jacques


Le 17/11/2017 à 22:18, Michael Brohl a écrit :

> Hi Jacques,
>
> it takes some effort to make an OFBiz standard compatible patch of the mechanism because we have several additions to the configurations. I would
> take the effort if the community wants to adapt it but it's too much work for just giving an idea.
>
> I have explained the mechanism in [1]. There it is based on Ant but we already use it in projects with Gradle.
>
> We were looking for other solutions during the migration to Gradle but haven't found a better approach considering all pros and cons (database
> configuration, environment variables etc.). We use it for years on our test- and production environments and it makes the handling of different
> system specific configurations very easy.
>
> We are thinking about the introduction of an additional configuration level so that we have the base configuration (containing all properties), a
> project configuration level and the system specific configurations. This helps us to additionally maintain projects using different sets of plugins.
>
> I'm happy to explain more if something is unclear.
>
> Regards,
>
> Michael
>
>
> [1] https://lists.apache.org/thread.html/b95e239250880d9a5b34268b3b711f0f8f7f0540a26bb41c5ced493a@1213087551@%3Cdev.ofbiz.apache.org%3E
>
>
> Am 03.11.17 um 11:39 schrieb Jacques Le Roux:
>> That's quite interesting Michael,
>>
>> Would you share in a Jira? Then we could get to merge all experiences and find a consensu.
>>
>> Jacques
>>
>>
>> Le 03/11/2017 à 10:10, Michael Brohl a écrit :
>>> Just an update triggered by the question from Swapnil [1]: our configuration mechanism mentioned below is now on Gradle so it would be compatible
>>> with 16.11 and later.
>>>
>>> Regards,
>>>
>>> Michael
>>>
>>> [1] https://lists.apache.org/thread.html/703f3e615a93a2a83fb92b122eb8275fb05aa27537d95342815dd043@%3Cdev.ofbiz.apache.org%3E
>>>
>>>
>>> Am 05.07.17 um 17:56 schrieb Michael Brohl:
>>>> Hi Gil,
>>>>
>>>> we have similar challenges and modified OFBiz to deal with it easily. We offered to contribute this long time ago (2008) but it was decided
>>>> against [1]. It was suggested to use patches instead but I think it's too complicated to manage several patch sets for different environments.
>>>>
>>>> We now use a staged configure mechanism which uses a base build file, auto detected machine name and provided parameters to decide which
>>>> configurations should be pulled for the environment. It's currently Ant based and therefore does not fit into the current build mechanism (on the
>>>> todo list).
>>>>
>>>> I like your approach also and I think it should be evaluated and discussed.
>>>>
>>>> Best regards,
>>>>
>>>> Michael Brohl
>>>> ecomify GmbH
>>>> www.ecomify.de
>>>>
>>>> [1] https://lists.apache.org/thread.html/b95e239250880d9a5b34268b3b711f0f8f7f0540a26bb41c5ced493a@1213087551@%3Cdev.ofbiz.apache.org%3E
>>>>
>>>>
>>>>
>>>> Am 05.07.17 um 17:36 schrieb gil portenseigne:
>>>>> Hello all,
>>>>>
>>>>> Working with different hosting companies, we used to have issues when deploying OFBiz concerning technical configuration of the different
>>>>> environments.
>>>>> We are writing this mail to get feedback from the community and eventually propose to improve OFBiz on this matter.
>>>>>
>>>>> For a customer, we are working with 4 instances of a release 13.07 OFBiz, and are currently using a set of patches (with addonmanager...) to
>>>>> manage environment specific configurations.
>>>>> During each production deployment, the hosting company receive from our jenkins a precompiled archive containing OFBiz codebase, and then apply
>>>>> the set of patches to configure it to the environment needs, recompile and relaunch...
>>>>>
>>>>> This way of doing can cause issue when patch could not apply, after a codebase modification (pretty rare but it happens).
>>>>>
>>>>> We are not satisfied with this way of doing, we are currently thinking about using environment variables to configure technical environment
>>>>> properties (those are on the hosting company responsibility), and to keep functional specifics into the code base.
>>>>> If you have some experience or advice in this matter, you are welcome.
>>>>>
>>>>> For our case, we currently have enhanced OFBiz to be able to get environment variable from the operating system within property file and some
>>>>> other configuration files (with default value if not set).
>>>>>
>>>>> Examples :
>>>>> *In Property file :
>>>>> password=${env:ONE_CONF:ofbiz} -> environment variable ONE_CONF or ofbiz if unset
>>>>> other_config=${env:OTHER_CONF:${partyId}} -> environment variable OTHER_CONF or ${partyId} if unset
>>>>> *In entityengine.xml :
>>>>> jdbc-uri="${env:DB_POSTGRES_URI:jdbc:mysql://127.0.0.1/ofbiz?autoReconnect=true}"
>>>>> jdbc-username="${env:DB_POSTGRES_USER:ofbiz}"
>>>>> jdbc-password="${env:DB_POSTGRES_PWD:ofbiz}"/>
>>>>>
>>>>> That allow us to keep functional parameters stored within git branches. Our jenkins now is able to build our 4 configured and compiled instances
>>>>> and deliver it to the hosting company, that just have to set/check environment variable (database access, activeMQ, log location, instance id,
>>>>> etc.) before starting OFBiz app. Now we cannot have configuration failure during deployment.
>>>>>
>>>>> We will be glad to contribute it, if it's the good way to go !
>>>>>
>>>>> Best Regards !
>>>>>
>>>>> Gil Portenseigne
>>>>>
>>>>
>>>>
>>>
>>>
>>
>
>

[Michael Brohl-3]

Hi everyone,

the recent discussions reminded me of this thread again. There were a
few people who were curious how we do configuration management and
deployment (now in production for about 10 years). It is working for
projects based on releases 13, 16, 17, 18 and trunk (with either Ant or
Gradle).

Although we have our ways to maintain this as not being part of the
official OFBiz codebase, I still think it is a good approach worth
contributing.

Is this interesting for the community?

If there is significant interest I would try to re-write the proposal I
did in [1], updated to the newest Gradle mechanism and several aspects
for multi stage projects.

Cheers,

Michael Brohl

ecomify GmbH - www.ecomify.de


[1]
https://lists.apache.org/thread.html/b95e239250880d9a5b34268b3b711f0f8f7f0540a26bb41c5ced493a@1213087551@%3Cdev.ofbiz.apache.org%3E


Am 17.11.17 um 22:18 schrieb Michael Brohl:

> Hi Jacques,
>
> it takes some effort to make an OFBiz standard compatible patch of the
> mechanism because we have several additions to the configurations. I
> would take the effort if the community wants to adapt it but it's too
> much work for just giving an idea.
>
> I have explained the mechanism in [1]. There it is based on Ant but we
> already use it in projects with Gradle.
>
> We were looking for other solutions during the migration to Gradle but
> haven't found a better approach considering all pros and cons
> (database configuration, environment variables etc.). We use it for
> years on our test- and production environments and it makes the
> handling of different system specific configurations very easy.
>
> We are thinking about the introduction of an additional configuration
> level so that we have the base configuration (containing all
> properties), a project configuration level and the system specific
> configurations. This helps us to additionally maintain projects using
> different sets of plugins.
>
> I'm happy to explain more if something is unclear.
>
> Regards,
>
> Michael
>
>
> [1]
> https://lists.apache.org/thread.html/b95e239250880d9a5b34268b3b711f0f8f7f0540a26bb41c5ced493a@1213087551@%3Cdev.ofbiz.apache.org%3E
>
>
> Am 03.11.17 um 11:39 schrieb Jacques Le Roux:
>> That's quite interesting Michael,
>>
>> Would you share in a Jira? Then we could get to merge all experiences
>> and find a consensu.
>>
>> Jacques
>>
>>
>> Le 03/11/2017 à 10:10, Michael Brohl a écrit :
>>> Just an update triggered by the question from Swapnil [1]: our
>>> configuration mechanism mentioned below is now on Gradle so it would
>>> be compatible with 16.11 and later.
>>>
>>> Regards,
>>>
>>> Michael
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/703f3e615a93a2a83fb92b122eb8275fb05aa27537d95342815dd043@%3Cdev.ofbiz.apache.org%3E
>>>
>>>
>>> Am 05.07.17 um 17:56 schrieb Michael Brohl:
>>>> Hi Gil,
>>>>
>>>> we have similar challenges and modified OFBiz to deal with it
>>>> easily. We offered to contribute this long time ago (2008) but it
>>>> was decided against [1]. It was suggested to use patches instead
>>>> but I think it's too complicated to manage several patch sets for
>>>> different environments.
>>>>
>>>> We now use a staged configure mechanism which uses a base build
>>>> file, auto detected machine name and provided parameters to decide
>>>> which configurations should be pulled for the environment. It's
>>>> currently Ant based and therefore does not fit into the current
>>>> build mechanism (on the todo list).
>>>>
>>>> I like your approach also and I think it should be evaluated and
>>>> discussed.
>>>>
>>>> Best regards,
>>>>
>>>> Michael Brohl
>>>> ecomify GmbH
>>>> www.ecomify.de
>>>>
>>>> [1]
>>>> https://lists.apache.org/thread.html/b95e239250880d9a5b34268b3b711f0f8f7f0540a26bb41c5ced493a@1213087551@%3Cdev.ofbiz.apache.org%3E
>>>>
>>>>
>>>>
>>>> Am 05.07.17 um 17:36 schrieb gil portenseigne:
>>>>> Hello all,
>>>>>
>>>>> Working with different hosting companies, we used to have issues
>>>>> when deploying OFBiz concerning technical configuration of the
>>>>> different environments.
>>>>> We are writing this mail to get feedback from the community and
>>>>> eventually propose to improve OFBiz on this matter.
>>>>>
>>>>> For a customer, we are working with 4 instances of a release 13.07
>>>>> OFBiz, and are currently using a set of patches (with
>>>>> addonmanager...) to manage environment specific configurations.
>>>>> During each production deployment, the hosting company receive
>>>>> from our jenkins a precompiled archive containing OFBiz codebase,
>>>>> and then apply the set of patches to configure it to the
>>>>> environment needs, recompile and relaunch...
>>>>>
>>>>> This way of doing can cause issue when patch could not apply,
>>>>> after a codebase modification (pretty rare but it happens).
>>>>>
>>>>> We are not satisfied with this way of doing, we are currently
>>>>> thinking about using environment variables to configure technical
>>>>> environment properties (those are on the hosting company
>>>>> responsibility), and to keep functional specifics into the code base.
>>>>> If you have some experience or advice in this matter, you are
>>>>> welcome.
>>>>>
>>>>> For our case, we currently have enhanced OFBiz to be able to get
>>>>> environment variable from the operating system within property
>>>>> file and some other configuration files (with default value if not
>>>>> set).
>>>>>
>>>>> Examples :
>>>>> *In Property file :
>>>>> password=${env:ONE_CONF:ofbiz} -> environment variable ONE_CONF or
>>>>> ofbiz if unset
>>>>> other_config=${env:OTHER_CONF:${partyId}} -> environment variable
>>>>> OTHER_CONF or ${partyId} if unset
>>>>> *In entityengine.xml :
>>>>> jdbc-uri="${env:DB_POSTGRES_URI:jdbc:mysql://127.0.0.1/ofbiz?autoReconnect=true}"
>>>>>
>>>>> jdbc-username="${env:DB_POSTGRES_USER:ofbiz}"
>>>>> jdbc-password="${env:DB_POSTGRES_PWD:ofbiz}"/>
>>>>>
>>>>> That allow us to keep functional parameters stored within git
>>>>> branches. Our jenkins now is able to build our 4 configured and
>>>>> compiled instances and deliver it to the hosting company, that
>>>>> just have to set/check environment variable (database access,
>>>>> activeMQ, log location, instance id, etc.) before starting OFBiz
>>>>> app. Now we cannot have configuration failure during deployment.
>>>>>
>>>>> We will be glad to contribute it, if it's the good way to go !
>>>>>
>>>>> Best Regards !
>>>>>
>>>>> Gil Portenseigne
>>>>>
>>>>
>>>>
>>>
>>>
>>
>
>