Removing “base/config/component-load.xml”

[Jacques Le Roux]

Le 18/12/2019 à 21:09, Jacques Le Roux a écrit :

> Hi Samuel, All,
>
> Le 18/12/2019 à 14:00, Samuel Trégouët a écrit : Le 18/12/2019 à 12:09, Jacques Le Roux a écrit :
>>>> Mathieu asks Michael to provide  an "explanation regarding why it matters in production environments to be able to patch" component-load.xml files
>> yes we are still waiting for your answer Michael ;) In my opinion we
>> cannot go ahead in this discussion without your answer, without your
>> need about component-load.xml: are you trying to avoid loading a
>> particular framework component? do you patch a framework component and
>> need another one to be loaded first to make your patch works? ...
>
> That's an interesting point indeed. Have you your own framework component/s Michael ?

Note: I include applications components here


Jacques

[taher]

May I suggest inquiring from the community on whether this feature is
important to them? Pehaps on user@? This way maybe we can have a more
informed decision on whether to adopt this change.

On Thu, Dec 19, 2019, 10:13 AM Jacques Le Roux <[hidden email]>
wrote:

> Le 18/12/2019 à 21:09, Jacques Le Roux a écrit :
> > Hi Samuel, All,
> >
> > Le 18/12/2019 à 14:00, Samuel Trégouët a écrit : Le 18/12/2019 à 12:09,
> Jacques Le Roux a écrit :
> >>>> Mathieu asks Michael to provide  an "explanation regarding why it
> matters in production environments to be able to patch" component-load.xml
> files
> >> yes we are still waiting for your answer Michael ;) In my opinion we
> >> cannot go ahead in this discussion without your answer, without your
> >> need about component-load.xml: are you trying to avoid loading a
> >> particular framework component? do you patch a framework component and
> >> need another one to be loaded first to make your patch works? ...
> >
> > That's an interesting point indeed. Have you your own framework
> component/s Michael ?
>
> Note: I include applications components here
>
>
> Jacques
>

[taher]

Also, if there are disagreements on a direction among members then it is
usually the case that a -1 from a binding vote requires a revert and
starting a discussion. Then we can either have consensus, lazy consensus or
the issue is shut down.

I'm saying what should be done, just laying out the standard rules we
usually run by.

On Thu, Dec 19, 2019, 1:59 PM Taher Alkhateeb <[hidden email]>
wrote:

> May I suggest inquiring from the community on whether this feature is
> important to them? Pehaps on user@? This way maybe we can have a more
> informed decision on whether to adopt this change.
>
> On Thu, Dec 19, 2019, 10:13 AM Jacques Le Roux <
> [hidden email]> wrote:
>
>> Le 18/12/2019 à 21:09, Jacques Le Roux a écrit :
>> > Hi Samuel, All,
>> >
>> > Le 18/12/2019 à 14:00, Samuel Trégouët a écrit : Le 18/12/2019 à 12:09,
>> Jacques Le Roux a écrit :
>> >>>> Mathieu asks Michael to provide  an "explanation regarding why it
>> matters in production environments to be able to patch" component-load.xml
>> files
>> >> yes we are still waiting for your answer Michael ;) In my opinion we
>> >> cannot go ahead in this discussion without your answer, without your
>> >> need about component-load.xml: are you trying to avoid loading a
>> >> particular framework component? do you patch a framework component and
>> >> need another one to be loaded first to make your patch works? ...
>> >
>> > That's an interesting point indeed. Have you your own framework
>> component/s Michael ?
>>
>> Note: I include applications components here
>>
>>
>> Jacques
>>
>

[Jacques Le Roux]

Hi Taher,

Actually it has already been reverted

Jacques

Le 19/12/2019 à 12:04, Taher Alkhateeb a écrit :

> Also, if there are disagreements on a direction among members then it is
> usually the case that a -1 from a binding vote requires a revert and
> starting a discussion. Then we can either have consensus, lazy consensus or
> the issue is shut down.
>
> I'm saying what should be done, just laying out the standard rules we
> usually run by.
>
> On Thu, Dec 19, 2019, 1:59 PM Taher Alkhateeb <[hidden email]>
> wrote:
>
>> May I suggest inquiring from the community on whether this feature is
>> important to them? Pehaps on user@? This way maybe we can have a more
>> informed decision on whether to adopt this change.
>>
>> On Thu, Dec 19, 2019, 10:13 AM Jacques Le Roux <
>> [hidden email]> wrote:
>>
>>> Le 18/12/2019 à 21:09, Jacques Le Roux a écrit :
>>>> Hi Samuel, All,
>>>>
>>>> Le 18/12/2019 à 14:00, Samuel Trégouët a écrit : Le 18/12/2019 à 12:09,
>>> Jacques Le Roux a écrit :
>>>>>>> Mathieu asks Michael to provide  an "explanation regarding why it
>>> matters in production environments to be able to patch" component-load.xml
>>> files
>>>>> yes we are still waiting for your answer Michael ;) In my opinion we
>>>>> cannot go ahead in this discussion without your answer, without your
>>>>> need about component-load.xml: are you trying to avoid loading a
>>>>> particular framework component? do you patch a framework component and
>>>>> need another one to be loaded first to make your patch works? ...
>>>> That's an interesting point indeed. Have you your own framework
>>> component/s Michael ?
>>>
>>> Note: I include applications components here
>>>
>>>
>>> Jacques
>>>

[Mathieu Lirzin]

Hello Taher,

Taher Alkhateeb <[hidden email]> writes:

> Also, if there are disagreements on a direction among members then it is
> usually the case that a -1 from a binding vote requires a revert and
> starting a discussion. Then we can either have consensus, lazy consensus or
> the issue is shut down.
>
> I'm saying what should be done, just laying out the standard rules we
> usually run by.

For more precision, a veto for code-modification is not enough for
justifying a revert [1]:

--8<---------------cut here---------------start------------->8---
To prevent vetos from being used capriciously, they must be accompanied
by a technical justification showing why the change is bad (opens a
security exposure, negativelyaffects performance, etc. ). A veto without
a justification is invalid and has no weight.
--8<---------------cut here---------------end--------------->8---

[1] https://www.apache.org/foundation/voting.html#Veto

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

[Mathieu Lirzin]

Jacques Le Roux <[hidden email]> writes:

> Mathieu asks Michael to provide  an "explanation regarding why it
> matters in production environments to be able to patch"
> component-load.xml files

@Michael: ping

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

[Michael Brohl-3]

Hi Mathieu,

citing myself from the comment in
https://issues.apache.org/jira/browse/OFBIZ-11296:

Mathieu Lirzin, please consider that people have limited time and that
they have to put priorities on how they spent it. Putting pressure on
(non bugfixing) issues does not help good collaborations.

Please also refrain to put judgements in your statements like "why they
consider to mess with..." or else. Simply trust people who are around
for a long time and have deep, real life project experience with OFBiz
for a long time (for me it's coming to about 18 years now).

This project is not only about tech, it has a user base with serious
business running on base of OFBiz. This has always to be considered as
serious as good technical solutions should be considered. So we cannot
simply change things because single contributors like other technical
solutions better. We have to remain downwards compatible and manage
deprecation of features carefully.

Until now, there was not a single thumbs up from experienced
contributors for this change, but objections from others. This alone
should make you think about your approach.

Me fighting for the right approach also does not necessarily mean that I
have (current) personal or customer interests in mind. The user base is
much bigger than "my" user base.

If you read carefully what I previously wrote, there are several uses
for the applications component-load.xml:

  * deactivation of unused component(s) by commenting out the
    load-component entry (why load marketing resources if you don't use
    the component at the moment)
  * addition of components (yes, I've seen projects where this was not
    done through the hot-deploy mechanism)
  * ordering these components in the right load order

While you can argument that these might be "wrong" approaches, they are
technically valid and used in customer projects. Therefore we cannot
simply switch the mechanisms without a proper deprecation period.

For the plugins, all the above use cases are common in our projects. We
also use a multi-level configuration mechanism (standard default -
custom standard default - project default and targeted systems) where we
are able to do fine-grained configurations and generate the resulting
component-load.xml at build time.

My proposal would be to actively ask other contributors with significant
project experience for their input before re-commiting anything.

If there is a demand for yur solution, I would also propose to make the
solution compatible with the component-load mechanism and leave the old
component-load.xml in place, together with a deprecation announcement
and proper documentation on how to migrate. This would introduce the new
depends-on in the next release but does not change anything for existing
users if they want to stick with the component-load mechanism.

For the plugins, I object to introduce the mechanism at all for the
above stated reasons.

I hope this explains my point of view clear enough, please ask if it
does not.

Thanks,

Michael


Am 14.12.19 um 00:28 schrieb Mathieu Lirzin:

> Hello Michael,
>
> Michael Brohl <[hidden email]> writes:
>
>> I am still not sure why we need the new mechanism.
>>
>> And if we decide to use both, we should make sure that the old version
>> is the default at least for the lifecycle of one release with proper
>> an clear dopcumentation for users.
>>
>> Thanks,
>>
>> Michael
>>
>>
>> PS: I'm asking myself why some people have such a big problem
>> reverting their work if others object against it. If there was no
>> review/discussion/consensus for a new feature, it should simply not go
>> into the codebase and should at least be reverted if there are
>> objections.
>>
>> It's tiring to discuss this afterwards and if the people objecting are
>> not persistent enough, the code simply stays.
> I have personally no problem reverting things.  If you look at the ‘git
> log’ you will see some recent reverts I have made.  I just need to
> understand the actual objection before reverting [1].
>
> Since your argument seems to be about the “impacts on users” an
> explanation regarding what you or others are actually achieving when
> patching the “framework/component-load.xml” and
> “applications/component-load.xml” would help me understand the issue,
> because I honestly do not see why the loading order/mechanism of
> “framework” or “applications” should not be considered an implementation
> detail.
>
> By the way to give more context on my perspective, the usage of
> <depends-on> instead of “component-load.xml” in the
> framework/applications directories is related to the implementation of
> the work described in a previous discussion [2] because it defines a
> location independent an extensible component loading order.
>
> HTH,
>
> [1] https://github.com/apache/ofbiz-framework/commit/eeabe69813a1d9f42911dec70a912574046ef49b
> [2] https://lists.apache.org/thread.html/c2612f1e296b6ea15872185871d3a9d83d6a4afc6d2a76f7a336a126%40%3Cdev.ofbiz.apache.org%3E
>

[Mathieu Lirzin]

Hello,

The arguments provided by Michael are very general and go beyond the
specific question of “component-load.xml” so I am opening a general
discussion about how to make OFBiz evolve smoothly by precising the
extent of its public API.

I urge other contributors to join this discussion which is crucial to
define our capability to work together as a community and my willing to
continue to participate.

Michael Brohl <[hidden email]> writes:

> This project is not only about tech, it has a user base with serious
> business running on base of OFBiz. This has always to be considered as
> serious as good technical solutions should be considered. So we cannot
> simply change things because single contributors like other technical
> solutions better. We have to remain downwards compatible and manage
> deprecation of features carefully.

First to clarify things, making evolutions in the framework is not about
developers changing arbitrary stuff, it is about structuring internals
in an understandeable way to enable correctness and the inclusion of new
features that satisfies evolving requirements.

Backwards compability only makes sense when something has a public API
otherwise every evolution is a breaking change. In OFBiz we lack a
proper specification of what is a feature provided by the framework
subject to backward compatibility and what is an implementation detail
that can evolve/disappear between version silently. We rely on an
informal consensus to distinguish between the two.

The fact that some mechanism appears to be used in production is a valid
argument against its removal only if that mechanism is part of the
public API, otherwise it is up to the client code to adapt.

My broad understanding of what is part of OFBiz public API is:
 - the plugin mechanism
 - the data model and data access (Entity Engine)
 - The ability to call existing services and implement new ones (Service Engine)
 - the HTTP routing mechanism (Event Handler)
 - the various configuration files location in “{component}/config” directories.

> [...]
> If you read carefully what I previously wrote, there are several uses
> for the applications component-load.xml:
>
>  * deactivation of unused component(s) by commenting out the
>    load-component entry (why load marketing resources if you don't use
>    the component at the moment)
>  * addition of components (yes, I've seen projects where this was not
>    done through the hot-deploy mechanism)
>  * ordering these components in the right load order
>
> While you can argument that these might be "wrong" approaches, they
> are technically valid and used in customer projects. Therefore we
> cannot simply switch the mechanisms without a proper deprecation
> period.

The general problem here is not to know if things are wrong or
technically valid, it is to know if something is part of the public API
or is an implementation detail. This determines how to handle an
evolution on that part. Something wrong but part of the public API like
using XML for code should be handled with care (deprecation, migration
guides), but something technically valid but inappropriate like patching
framework Java source code from a plugin should be ignored.

In the case of ordering/enabling core components I consider it as an
implementation detail. If a component inside framework/applications is
effectively optional (like the marketing example you brought) it should
eventually be moved in the official plugins if we actually want to
provides the capability for users to disable it. However users should
not be entitled to think that they can freely desactivate/reorder/add
new components inside the framework/applications directory and that such
modifications will continue to work in a future release.

The larger question is about knowing if the internal organisation of the
files inside the "framework/applications" directories with the exception
of the “config” directories is considered part of OFBiz public API or
not?  What do people think?

For the record, Without the ability to safely refactor a large subset of
the codebase that have the status of “implementation details”, I will
simply stop contributing to OFBiz because I don't have time for endless
discussions with people blaming my community work because their
extensions happen to rely on unspecified behavior.

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

[Jacques Le Roux]

Hi Michael,

First I must say that I understand your POV about production and custom projects. I have been there too and when your business depends on it, things
really matter.

That's why I suggested that we could have both solutions in parallel for a while. With the current one being deprecated and the new one replacing it
w/o hurry. So the 1st question is to know if that's possible. I could be wrong, but I vaguely recall that Mathieu said it was possible. Once that
clarified the debate should cool down.

Before answering to Mathieu's last message, I'd like to note inline some points about yours.

Le 04/01/2020 à 15:47, Michael Brohl a écrit :
> Hi Mathieu,
>
> If you read carefully what I previously wrote, there are several uses for the applications component-load.xml:
>
>  * deactivation of unused component(s) by commenting out the
>    load-component entry (why load marketing resources if you don't use
>    the component at the moment)

Here I agree about Mathieu's perspective (in his last email). If an applications component can easily be put out by commenting out the  load-component
entry, it should not be in application but in plugins. Only mandatory applications should be in under the applications directory.


>  * addition of components (yes, I've seen projects where this was not
>    done through the hot-deploy mechanism)

This is no good and should not be done this way


>  * ordering these components in the right load order
>
> While you can argument that these might be "wrong" approaches, they are technically valid and used in customer projects. Therefore we cannot simply
> switch the mechanisms without a proper deprecation period.

As I said earlier I agree about a proper deprecation period.


>
> For the plugins, all the above use cases are common in our projects. We also use a multi-level configuration mechanism (standard default - custom
> standard default - project default and targeted systems) where we are able to do fine-grained configurations and generate the resulting
> component-load.xml at build time.

Is this not the main reason about your concerns on Mathieu's changes? Can't this be migrated using Mathieu's proposed solutions?


>
> My proposal would be to actively ask other contributors with significant project experience for their input before re-commiting anything.

I don't see any problems about committing new stuff as long as they don't disturb current functional behaviours. Note that those should be done in
trunk, and we are not supposed to use the trunk in production. So the impact should not be for today. If we refer to our current stable version it
would be at least 3 years before those being released... Would that not be a reasonable deprecation period?


>
> If there is a demand for yur solution, I would also propose to make the solution compatible with the component-load mechanism and leave the old
> component-load.xml in place, together with a deprecation announcement and proper documentation on how to migrate.

That sounds reasonable to me

> This would introduce the new depends-on in the next release
It would not be the next release, but one in at least 3 years...

> but does not change anything for existing users if they want to stick with the component-load mechanism.

Here we need to know if the cost of maintaining both solutions is worth it. At some point a deprecated mechanism can be removed if the newer solutions
offers more flexibility, better code, etc.


>
> For the plugins, I object to introduce the mechanism at all for the above stated reasons.

Could you elaborate on that, I don't get it

Thanks

Jacques

>
> I hope this explains my point of view clear enough, please ask if it does not.
>
> Thanks,
>
> Michael
>
>
> Am 14.12.19 um 00:28 schrieb Mathieu Lirzin:
>> Hello Michael,
>>
>> Michael Brohl <[hidden email]> writes:
>>
>>> I am still not sure why we need the new mechanism.
>>>
>>> And if we decide to use both, we should make sure that the old version
>>> is the default at least for the lifecycle of one release with proper
>>> an clear dopcumentation for users.
>>>
>>> Thanks,
>>>
>>> Michael
>>>
>>>
>>> PS: I'm asking myself why some people have such a big problem
>>> reverting their work if others object against it. If there was no
>>> review/discussion/consensus for a new feature, it should simply not go
>>> into the codebase and should at least be reverted if there are
>>> objections.
>>>
>>> It's tiring to discuss this afterwards and if the people objecting are
>>> not persistent enough, the code simply stays.
>> I have personally no problem reverting things.  If you look at the ‘git
>> log’ you will see some recent reverts I have made.  I just need to
>> understand the actual objection before reverting [1].
>>
>> Since your argument seems to be about the “impacts on users” an
>> explanation regarding what you or others are actually achieving when
>> patching the “framework/component-load.xml” and
>> “applications/component-load.xml” would help me understand the issue,
>> because I honestly do not see why the loading order/mechanism of
>> “framework” or “applications” should not be considered an implementation
>> detail.
>>
>> By the way to give more context on my perspective, the usage of
>> <depends-on> instead of “component-load.xml” in the
>> framework/applications directories is related to the implementation of
>> the work described in a previous discussion [2] because it defines a
>> location independent an extensible component loading order.
>>
>> HTH,
>>
>> [1] https://github.com/apache/ofbiz-framework/commit/eeabe69813a1d9f42911dec70a912574046ef49b
>> [2] https://lists.apache.org/thread.html/c2612f1e296b6ea15872185871d3a9d83d6a4afc6d2a76f7a336a126%40%3Cdev.ofbiz.apache.org%3E
>>
>

[Michael Brohl-3]

Hi Mathieu,

inline...

Am 05.01.20 um 18:32 schrieb Mathieu Lirzin:

> Hello,
>
> The arguments provided by Michael are very general and go beyond the
> specific question of “component-load.xml” so I am opening a general
> discussion about how to make OFBiz evolve smoothly by precising the
> extent of its public API.
>
> I urge other contributors to join this discussion which is crucial to
> define our capability to work together as a community and my willing to
> continue to participate.
>
> Michael Brohl <[hidden email]> writes:
>
>> This project is not only about tech, it has a user base with serious
>> business running on base of OFBiz. This has always to be considered as
>> serious as good technical solutions should be considered. So we cannot
>> simply change things because single contributors like other technical
>> solutions better. We have to remain downwards compatible and manage
>> deprecation of features carefully.
> First to clarify things, making evolutions in the framework is not about
> developers changing arbitrary stuff, it is about structuring internals
> in an understandeable way to enable correctness and the inclusion of new
> features that satisfies evolving requirements.
>
> Backwards compability only makes sense when something has a public API
> otherwise every evolution is a breaking change. In OFBiz we lack a
> proper specification of what is a feature provided by the framework
> subject to backward compatibility and what is an implementation detail
> that can evolve/disappear between version silently. We rely on an
> informal consensus to distinguish between the two.
>
> The fact that some mechanism appears to be used in production is a valid
> argument against its removal only if that mechanism is part of the
> public API, otherwise it is up to the client code to adapt.

OFBiz is not just a library or core framework, it is a multi-level project:

* a web development framework

* a web based ERP system on base of this framework

* highly flexible and extendable through various mechanisms.

OFBiz users are service providers, utilizing OFBiz to provide software
solutions as well as end users who are mainly using the applications.
There's also a mix in the case where company employees use OFBiz as a
web development framework to provide software solutions for their own
company.

So it cannot be simplified to a scenario where the framework is "ours"
and the users are proivided with the applications and a public API. So
if the project has provided a mechanism to configure how components are
loaded, we are also responsible for taking care of this if we want to
make changes.


>
> My broad understanding of what is part of OFBiz public API is:
>   - the plugin mechanism
>   - the data model and data access (Entity Engine)
>   - The ability to call existing services and implement new ones (Service Engine)
>   - the HTTP routing mechanism (Event Handler)
>   - the various configuration files location in “{component}/config” directories.

The component-load.xml is also a configuration option which is utilized
in projects.

There is some documentation on how to utilize OFBiz as a core framework
by deactivating all components (old but still valid, see [1]).

>
>> [...]
>> If you read carefully what I previously wrote, there are several uses
>> for the applications component-load.xml:
>>
>>   * deactivation of unused component(s) by commenting out the
>>     load-component entry (why load marketing resources if you don't use
>>     the component at the moment)
>>   * addition of components (yes, I've seen projects where this was not
>>     done through the hot-deploy mechanism)
>>   * ordering these components in the right load order
>>
>> While you can argument that these might be "wrong" approaches, they
>> are technically valid and used in customer projects. Therefore we
>> cannot simply switch the mechanisms without a proper deprecation
>> period.
> The general problem here is not to know if things are wrong or
> technically valid, it is to know if something is part of the public API
> or is an implementation detail. This determines how to handle an
> evolution on that part. Something wrong but part of the public API like
> using XML for code should be handled with care (deprecation, migration
> guides), but something technically valid but inappropriate like patching
> framework Java source code from a plugin should be ignored.

I don't think that patching Java code is/was part of the initial
discussion. We should not mix up things.


>
> In the case of ordering/enabling core components I consider it as an
> implementation detail. If a component inside framework/applications is

I don't agree, see above.


> effectively optional (like the marketing example you brought) it should
> eventually be moved in the official plugins if we actually want to
> provides the capability for users to disable it. However users should

Even it it would be a plugin, you still need a mechanism to
enable/disable it by configuration.

To my understanding, if we use depends-on exclusively for framework,
applications and plugins, this would not be possible anymore.


> not be entitled to think that they can freely desactivate/reorder/add
> new components inside the framework/applications directory and that such
> modifications will continue to work in a future release.

Users are also developers using OFBiz as a framework, so we should take
responsibility for this also.


>
> The larger question is about knowing if the internal organisation of the
> files inside the "framework/applications" directories with the exception
> of the “config” directories is considered part of OFBiz public API or
> not?  What do people think?

See my comments above. We cannot reduce the discussion to a public API
because of the web development framework functionality provided by OFBiz.

>
> For the record, Without the ability to safely refactor a large subset of
> the codebase that have the status of “implementation details”, I will
> simply stop contributing to OFBiz because I don't have time for endless
> discussions with people blaming my community work because their
> extensions happen to rely on unspecified behavior.

Noone is blaming your work. In the contrary, I expressed my appreciation
for your work in the beginning of this discussion.

A community project is serving a lot of different interests and is
natural that changes will bring up discussions. It is also natural that
improvements and chnages must be thoroughly discussed and decided.
Different people have different views on the project and in my opinion,
the best solutions come up if different people engage in the discussion
to find the best solution, both technically and functionally.

So it is essential that you are open for discussions and take into
account that things do not go through easily. It's in the best interest
of the project.

The depends-on discussion has shown this clearly: without my objection,
the regression put in the first implementation would have gone through
and potentially break productive projects after release.

Pressing people by rusing things into the codebase or even expressing
that you are not willing to contribute if you don't get your work into
the codebase easily does not help much to solve the challenges which lie
ahead uf us.

So I really hope that we can find a way for good collaboration and that
you can be patient enough to find the best solutions.

Thank you,

Michael


[1]
https://cwiki.apache.org/confluence/display/OFBIZ/Ofbiz+as+a+development+framework+-+release+10.04

[Samuel-2]

Hi all,

> Am 05.01.20 um 18:32 schrieb Mathieu Lirzin:
> >
> > I urge other contributors to join this discussion which is crucial to
> > define our capability to work together as a community and my willing to
> > continue to participate.

I also hope others contributors will eventually join (many thanks to
Jacques for joining!) since this discussion  seems to me to be larger
than a particular feature (component-load.xml): it is about the process
of contributing to OFBiz.

Michael has started to discuss because he had missed the commit which
removes component-load.xml in applications and framework and he claimed
[2] that we didn't discuss in [hidden email] before: completely
true! Why do we need to discuss such an implementation detail? Some
argue that we have to discuss before intruducing any *big* changement
:confused: What is a *big* changement? In software library/framework it
is quite easy to answer: a big changement is a breaking in public api.
So here is the question from Mathieu:

  what is OFBiz public API?

In my opinion we need an answer for this question otherwise we need to
discuss every single changement! which seems to be really cumbersome!
And even if we discuss every single changement how to decide it is good
for our community: *one* other contributor thumb-up is enough? maybe
*two*? do we need to wait forever if others don't care about a
particular changement?


> OFBiz is not just a library or core framework, it is a multi-level project:
>
> * a web development framework
>
> * a web based ERP system on base of this framework
>
> * highly flexible and extendable through various mechanisms.

Like so many frameworks, OFBiz is not different according to this
points.
And like so many frameworks which are extendable we need API to ease
extension.

>
> To my understanding, if we use depends-on exclusively for framework,
> applications and plugins, this would not be possible anymore.

This is where you're wrong. From the beginning using depends-on in
framework doesn't imply using it in plugins! The thing which drive
Mathieu to revert is that you cannot, in *framework* override depends-on
with a component-load.xml. And here we are with the actual discussion:

1. component-load.xml in plugin directory seems to be feature (nobody
   discuss this point)
2. what about component-load.xml in framework and applications
   directories? is it also a feature (in other words a public API) or an
   implementation detail

 
> [1]
> https://cwiki.apache.org/confluence/display/OFBIZ/Ofbiz+as+a+development+framework+-+release+10.04

This reference is a bit old and stated as wip so I will consider it
as irrelevant for our discussion ;)

cheers,

Samuel

[1]: https://lists.apache.org/thread.html/c2612f1e296b6ea15872185871d3a9d83d6a4afc6d2a76f7a336a126%40%3Cdev.ofbiz.apache.org%3E
[2]: https://lists.apache.org/thread.html/7eab3d2ae3bbeadb184b02f75f7b2b86263532485e88ecba4d4dc780%40%3Cdev.ofbiz.apache.org%3E

[Jacques Le Roux]

Hi Samuel,

Inline too...

Le 06/01/2020 à 10:29, Samuel Trégouët a écrit :

> Hi all,
> [snip...]
>
> Michael has started to discuss because he had missed the commit which
> removes component-load.xml in applications and framework and he claimed
> [2] that we didn't discuss in [hidden email] before: completely
> true! Why do we need to discuss such an implementation detail? Some
> argue that we have to discuss before intruducing any *big* changement
> :confused: What is a *big* changement? In software library/framework it
> is quite easy to answer: a big changement is a breaking in public api.
> So here is the question from Mathieu:
>
>    what is OFBiz public API?
>
> In my opinion we need an answer for this question otherwise we need to
> discuss every single changement! which seems to be really cumbersome!
> And even if we discuss every single changement how to decide it is good
> for our community: *one* other contributor thumb-up is enough? maybe
> *two*? do we need to wait forever if others don't care about a
> particular changement?

Mathieu and you make good points with the notion of "OFBiz public API".
It would be indeed good to have a such concept to officially (ie w/ a prior consensus) collect all parts that always need deeper discussions and consents.

But I fear it's not easy to define and this needs if not a deep discussion at least a long one (see the kinda recursion here?).

So before having all agreed about what the "OFBiz public API" is I think we need to cure the present issue. Except if Mathieu is pleased to wait
before this is agreed on.

Remember at the ASF most is about discussion and agreement. As the motto, which has has proved is efficiency over years, says: *"Community over code".*

> [snip...]
>
>> Michael's
>> =========
>> To my understanding, if we use depends-on exclusively for framework,
>> applications and plugins, this would not be possible anymore.
> This is where you're wrong. From the beginning using depends-on in
> framework doesn't imply using it in plugins! The thing which drive
> Mathieu to revert is that you cannot, in *framework* override depends-on
> with a component-load.xml.

So you say that depends-on and component-load.xml are incompatible?
This opens a new discussion to me. Because then we don't speak about deprecation but about replacement...


> And here we are with the actual discussion:
>
> 1. component-load.xml in plugin directory seems to be feature (nobody
>     discuss this point)
> 2. what about component-load.xml in framework and applications
>     directories? is it also a feature (in other words a public API) or an
>     implementation detail

I tend to think that it's not a feature. I'd though not say that it's an "implementation detail" ;) As often in "real-life" it's not black or white.
It's greyed and implications may be complex...


>> [1]
>> https://cwiki.apache.org/confluence/display/OFBIZ/Ofbiz+as+a+development+framework+-+release+10.04
> This reference is a bit old and stated as wip so I will consider it
> as irrelevant for our discussion ;)

I think we can indeed consider a 10 years old WIP reference a "bit old". I even don't see how what was wrote then still currently applies *in trunk*.
It even me wonder what I should to about OFBIZ-3500 :/ Certainly not close it!

My 2cts

Jacques

>
> cheers,
>
> Samuel
>
> [1]: https://lists.apache.org/thread.html/c2612f1e296b6ea15872185871d3a9d83d6a4afc6d2a76f7a336a126%40%3Cdev.ofbiz.apache.org%3E
> [2]: https://lists.apache.org/thread.html/7eab3d2ae3bbeadb184b02f75f7b2b86263532485e88ecba4d4dc780%40%3Cdev.ofbiz.apache.org%3E

[Mathieu Lirzin]

Hello,

Jacques Le Roux <[hidden email]> writes:

> Le 06/01/2020 à 10:29, Samuel Trégouët a écrit :
>>
>>    what is OFBiz public API?
>>
>> In my opinion we need an answer for this question otherwise we need to
>> discuss every single changement! which seems to be really cumbersome!
>> And even if we discuss every single changement how to decide it is good
>> for our community: *one* other contributor thumb-up is enough? maybe
>> *two*? do we need to wait forever if others don't care about a
>> particular changement?
>
> Mathieu and you make good points with the notion of "OFBiz public API".
> It would be indeed good to have a such concept to officially (ie w/ a prior consensus) collect all parts that always need deeper discussions and consents.
>
> But I fear it's not easy to define and this needs if not a deep discussion at least a long one (see the kinda recursion here?).
>
> So before havi ng all agreed about what the "OFBiz public API" is I
> think we need to cure the present issue. Except if Mathieu is pleased
> to wait before this is agreed on.

I think it is an important discussion that is overdue.

Given that I need a break to both get over the frustration of the recent
heated discussions and focus on my research work, As far as I am
concerned this is a good moment for this discussion to happen.

The goal of this discussion would be to define the boundaries between
deprecated/stable/experimental/internal code by considering both the
stability requirements that are important for production environments
(that need to be able to upgrade smoothly) and the capability for
developers to refactor code that is necessary to be able to implement
the features allowing OFBiz to remain relevant in the future.

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

[Pierre Smits-3]

I am +1,

as it is going to provide more clarity and devops simplicity for adopters

Best regards,

Pierre Smits

*Apache Trafodion <https://trafodion.apache.org>, Vice President*
*Apache Directory <https://directory.apache.org>, PMC Member*
Apache Incubator <https://incubator.apache.org>, committer
*Apache OFBiz <https://ofbiz.apache.org>, contributor (without privileges)
since 2008*
Apache Steve <https://steve.apache.org>, committer


On Sun, Dec 8, 2019 at 2:34 AM Mathieu Lirzin <[hidden email]>
wrote:

> Hello,
>
> We have in OFBiz two redundant ways to define the order in which
> components are loaded.
>
> * “component-load.xml” files stored in components directories meaning
>   directories containing single components. They define a total order
>   between every component inside that directory.
>
> * <depends-on> XML elements inside “ofbiz-component.xml” files stating
>   that a component ‘c’ requires a set of components ‘D’ to exist and to
>   be loaded before trying to load ‘c’.  Globally this constructs a
>   dependency graph which defines a partial order between available
>   components.
>
> Currently <depends-on> is used everywhere in the framework on ‘trunk’
> because it is more declarative and open for extensibility due the
> partial ordering capability.
>
> The only exception is “base/config/component-load.xml” which is a
> special components directory configuration defining the order of other
> components directories meaning “framework”, “themes”, “applications” and
> “plugins” in their respective dependency order.
>
> This file allows integrators to add and reorder components
> directories. This feature is not really useful nowadays considering that
> our plugin mechanism provides sufficient extensibility capability when
> combined with <depends-on> definitions.
>
> In order to simplify things which helps the endeavour towards
> transforming OFBiz in an extensible JVM based library, I would like to
> remove such configuration point and hard-code the list of component
> directories inside the code.
>
> I guess this low-level technical proposition is not interesting for most
> people but in case someone want to understand more or object before
> things are actually removed. I am opening a discussion on this ML.
>
> If nobody step-up in a week, I will go ahead.
>
> Thanks.
>
> --
> Mathieu Lirzin
> GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37
>

[Michael Brohl-3]

Hi Samuel,

inline...

Am 06.01.20 um 10:29 schrieb Samuel Trégouët:
> ...
> Michael has started to discuss because he had missed the commit which
> removes component-load.xml in applications and framework and he claimed
> [2] that we didn't discuss in [hidden email] before: completely
> true! Why do we need to discuss such an implementation detail? Some

I already explained why the applications component-load.xml is not an
implementation detail but used in real life projects.

> argue that we have to discuss before intruducing any *big* changement
> :confused: What is a *big* changement? In software library/framework it
> is quite easy to answer: a big changement is a breaking in public api.

It's not about being a big change, it's about breaking existing
mechanisms/configurations on the user side.

> So here is the question from Mathieu:
>
>    what is OFBiz public API?
>
> In my opinion we need an answer for this question otherwise we need to
> discuss every single changement! which seems to be really cumbersome!

It's common that you propose a change, ask for comments/review and maybe
advice if you work in a community, yes. It saves time and energy and
adds value to the final solution (mostly).

...

>> OFBiz is not just a library or core framework, it is a multi-level project:
>>
>> * a web development framework
>>
>> * a web based ERP system on base of this framework
>>
>> * highly flexible and extendable through various mechanisms.
> Like so many frameworks, OFBiz is not different according to this
> points.
> And like so many frameworks which are extendable we need API to ease
> extension.

Yes, but it makes a diffrence in the argumentation. The argument was,
that the load configuration is an implementation detail.

This might be true for pure ERP users but it is not users who utilize
existing mechanisms of the web development framework.

>> To my understanding, if we use depends-on exclusively for framework,
>> applications and plugins, this would not be possible anymore.
> This is where you're wrong. From the beginning using depends-on in
> framework doesn't imply using it in plugins! The thing which drive
> Mathieu to revert is that you cannot, in *framework* override depends-on
> with a component-load.xml. And here we are with the actual discussion:
>
> 1. component-load.xml in plugin directory seems to be feature (nobody
>     discuss this point)

That *might* be a misunderstanding (if Mathieu agrees on this point). My
understanding was that he first implemented and committed the change for
framework and applications (on Nov. 25).

 From his mail in dev [1] and also the issue title of [2] I understand
that the component-load mechanism should be removed *everywhere*
afterwards. The dev mail would not make sense otherwise because already
committed the work at the time of writing (two weeks before) and he
announces to go on if noone objects.

I apologize if I missed a point here, maybe Mathieu can clarify this?

I do not object against the change in framework, because I consider the
change of component-load in framework an exceptional use case.

For applications and plugins I have explained why we should not change
the mechanism.


> 2. what about component-load.xml in framework and applications
>     directories? is it also a feature (in other words a public API) or an
>     implementation detail
>
>  
>> [1]
>> https://cwiki.apache.org/confluence/display/OFBIZ/Ofbiz+as+a+development+framework+-+release+10.04
> This reference is a bit old and stated as wip so I will consider it
> as irrelevant for our discussion ;)

This reference was only given to document that the mechanism is meant to
be used in the way I described it. How old it is or if it is WIP does
not make the meaning less relevant.

> cheers,
>
> Samuel
>
> [1]: https://lists.apache.org/thread.html/c2612f1e296b6ea15872185871d3a9d83d6a4afc6d2a76f7a336a126%40%3Cdev.ofbiz.apache.org%3E
> [2]: https://lists.apache.org/thread.html/7eab3d2ae3bbeadb184b02f75f7b2b86263532485e88ecba4d4dc780%40%3Cdev.ofbiz.apache.org%3E

Thanks,

Michael


[1]
https://lists.apache.org/thread.html/441d038d1d000429dc1f09784db4b90bdc4cdd2b0e7c9bc4ccc9e48f%40%3Cdev.ofbiz.apache.org%3E

[2] https://issues.apache.org/jira/browse/OFBIZ-11296

[Jacques Le Roux]

Le 07/01/2020 à 00:56, Mathieu Lirzin a écrit :

> Hello,
>
> Jacques Le Roux <[hidden email]> writes:
>
>> Le 06/01/2020 à 10:29, Samuel Trégouët a écrit :
>>>     what is OFBiz public API?
>>>
>>> In my opinion we need an answer for this question otherwise we need to
>>> discuss every single changement! which seems to be really cumbersome!
>>> And even if we discuss every single changement how to decide it is good
>>> for our community: *one* other contributor thumb-up is enough? maybe
>>> *two*? do we need to wait forever if others don't care about a
>>> particular changement?
>> Mathieu and you make good points with the notion of "OFBiz public API".
>> It would be indeed good to have a such concept to officially (ie w/ a prior consensus) collect all parts that always need deeper discussions and consents.
>>
>> But I fear it's not easy to define and this needs if not a deep discussion at least a long one (see the kinda recursion here?).
>>
>> So before havi ng all agreed about what the "OFBiz public API" is I
>> think we need to cure the present issue. Except if Mathieu is pleased
>> to wait before this is agreed on.
> I think it is an important discussion that is overdue.
>
> Given that I need a break to both get over the frustration of the recent
> heated discussions and focus on my research work, As far as I am
> concerned this is a good moment for this discussion to happen.
>
> The goal of this discussion would be to define the boundaries between
> deprecated/stable/experimental/internal code by considering both the
> stability requirements that are important for production environments
> (that need to be able to upgrade smoothly) and the capability for
> developers to refactor code that is necessary to be able to implement
> the features allowing OFBiz to remain relevant in the future.

Hi Mathieu,

I started the "What is OFBiz public API?" thread

Actually I did yesterday but had to reboot my Internet box because of IP blocked issue and then ask barracudacentral.org to remove my IP from their
blocked list (I don't thanks my Internet Provider!)

Jacques

[Paul Foxworthy-2]

On Mon, 6 Jan 2020 at 20:29, Samuel Trégouët <[hidden email]>
wrote:


>   what is OFBiz public API?
>
> In my opinion we need an answer for this question otherwise we need to
> discuss every single changement! which seems to be really cumbersome!
> And even if we discuss every single changement how to decide it is good
> for our community: *one* other contributor thumb-up is enough? maybe
> *two*? do we need to wait forever if others don't care about a
> particular changement?
>

Hi Samuel,

I think this isn't a choice between having a public API and cumbersome
decision making. An API is a bits and bytes thing and won't magically
streamline decision making. A better architected, designed and understood
API will help **people** get an **initial impression** of what is easy to
refactor and re-implement.  Being clearer about it would be a good thing.
There is still the potential that a change that seems simple and
straightforward to one contributor is disruptive to others. And if it truly
is disruptive, we should collaborate on what comes next. Are the benefits
worth the disruption? Should there be some sort of deprecation, phase-out
or backwards compatibility option? I can't clearly define what "big" is,
but I know it when I see it.

And as a starting point for making these decisions, I suggest
https://www.apache.org/foundation/how-it-works.html#decision-making .

Cheers

Paul Foxworthy

--
Coherent Software Australia Pty Ltd
PO Box 2773
Cheltenham Vic 3192
Australia

Phone: [phone]
Web: http://www.coherentsoftware.com.au/
Email: [hidden email]

[Jacopo Cappellato-3]

There will always be a tension between guaranteeing backward compatibility
for the existing user base and the efforts to maintain our codebase,
enhance/refactor/innovate it.
Considering the peculiar nature of OFBiz, I don't think that trying to
define the areas that are part of the "public API" and then guarantee
backward compatibility only for them, will resolve this tension. In fact
there may be cases in which the "public API" should change in a
non-backward compatible way such as:
* the cost of maintaining a backward compatible feature is too high for our
community (or there is not enough interest in the active community in
maintaining it)
* the change is required to fix a flaw or a security vulnerability
* some fundamental (and important to the community) architectural change
may not be backward compatible
* etc...

Apart from this, even the definition of "public API" for OFBiz is
troublesome considering for example that potentially any OFBiz service can
be used by "client" code and as a consequence in theory our community
should never change the behavior (or remove) a service.

In an ideal World, our community would have infinite resources and would be
able to innovate and maintain our codebase while at the same time
guaranteeing backward compatibility for all the users... in our reality we
have to take the best decisions case by case. Potentially any change will
impact someone and we should accept this; our community is also here to
support and provide advices to users facing issues during migrations. I
don't think it is possible or even meaningful to ask that any
change/innovation must be backward compatible. This doesn't mean that we
should not care at all about backward compatibility because there are
changes that may have a large impact while not being so relevant for the
innovation.

There are cases (like the one that triggered this conversation) that may
involve some community discussion because it is difficult to figure out if
the cost (in terms of pains for our users) is worth the gain (in terms of
better/cleaner code): sometimes these discussions can be frustrating but if
we all try to stay open (to the other point of view), positive, flexible,
patient I am sure we will cope with them.

Just my 2 cents

Jacopo

[Jacques Le Roux]

Le 05/01/2020 à 18:32, Mathieu Lirzin a écrit :

> Hello,
>
> The arguments provided by Michael are very general and go beyond the
> specific question of “component-load.xml” so I am opening a general
> discussion about how to make OFBiz evolve smoothly by precising the
> extent of its public API.
>
> I urge other contributors to join this discussion which is crucial to
> define our capability to work together as a community and my willing to
> continue to participate.
>
> Michael Brohl <[hidden email]> writes:
>
>> This project is not only about tech, it has a user base with serious
>> business running on base of OFBiz. This has always to be considered as
>> serious as good technical solutions should be considered. So we cannot
>> simply change things because single contributors like other technical
>> solutions better. We have to remain downwards compatible and manage
>> deprecation of features carefully.
> First to clarify things, making evolutions in the framework is not about
> developers changing arbitrary stuff, it is about structuring internals
> in an understandeable way to enable correctness and the inclusion of new
> features that satisfies evolving requirements.
>
> Backwards compability only makes sense when something has a public API
> otherwise every evolution is a breaking change. In OFBiz we lack a
> proper specification of what is a feature provided by the framework
> subject to backward compatibility and what is an implementation detail
> that can evolve/disappear between version silently. We rely on an
> informal consensus to distinguish between the two.
>
> The fact that some mechanism appears to be used in production is a valid
> argument against its removal only if that mechanism is part of the
> public API, otherwise it is up to the client code to adapt.
>
> My broad understanding of what is part of OFBiz public API is:
>   - the plugin mechanism
>   - the data model and data access (Entity Engine)
>   - The ability to call existing services and implement new ones (Service Engine)
>   - the HTTP routing mechanism (Event Handler)
>   - the various configuration files location in “{component}/config” directories.
>
>> [...]
>> If you read carefully what I previously wrote, there are several uses
>> for the applications component-load.xml:
>>
>>   * deactivation of unused component(s) by commenting out the
>>     load-component entry (why load marketing resources if you don't use
>>     the component at the moment)
>>   * addition of components (yes, I've seen projects where this was not
>>     done through the hot-deploy mechanism)
>>   * ordering these components in the right load order
>>
>> While you can argument that these might be "wrong" approaches, they
>> are technically valid and used in customer projects. Therefore we
>> cannot simply switch the mechanisms without a proper deprecation
>> period.
> The general problem here is not to know if things are wrong or
> technically valid, it is to know if something is part of the public API
> or is an implementation detail. This determines how to handle an
> evolution on that part. Something wrong but part of the public API like
> using XML for code should be handled with care (deprecation, migration
> guides), but something technically valid but inappropriate like patching
> framework Java source code from a plugin should be ignored.
>
> In the case of ordering/enabling core components I consider it as an
> implementation detail. If a component inside framework/applications is
> effectively optional (like the marketing example you brought) it should
> eventually be moved in the official plugins if we actually want to
> provides the capability for users to disable it. However users should
> not be entitled to think that they can freely desactivate/reorder/add
> new components inside the framework/applications directory and that such
> modifications will continue to work in a future release.
>
> The larger question is about knowing if the internal organisation of the
> files inside the "framework/applications" directories with the exception
> of the “config” directories is considered part of OFBiz public API or
> not?  What do people think?
>
> For the record, Without the ability to safely refactor a large subset of
> the codebase that have the status of “implementation details”, I will
> simply stop contributing to OFBiz because I don't have time for endless
> discussions with people blaming my community work because their
> extensions happen to rely on unspecified behavior.

Hi All,

Following Mathieu's question about "OFBiz public API" and the points he already mentioned I decided to create a new thread.
I don't know what will happen with the other tread (mostly about removing/replacing component-load.xml). That's not the subject here.

Mathieu wrote:

    <<My broad understanding of what is part of OFBiz public API is:
      - the plugin mechanism
      - the data model and data access (Entity Engine)
      - The ability to call existing services and implement new ones (Service Engine)
      - the HTTP routing mechanism (Event Handler)
      - the various configuration files location in “{component}/config” directories.>>

I tend to agree with this list. From the top of my head, I'd add (maybe not a the same level and maybe we need to define level/s)

1. Obviously, the Java API !
2. All things (why not?) supported by XSD files.
    It includes Entity and Services Engines, not all feature are documented/supported by XSD files though.
    It also includes component-loader.xsd and ofbiz-component.xsd which are somehow parts of the subject of the other thread
3. OFBiz Gradle tasks

I don't know what to say about Groovy scripts which are replacing simple-method services.

What else?

Jacques

[holivier]

Hi, everybody,

It is not easy to add something in the discussion among the different arguments! (and on the different topics)

I will, however, try to give some summary of my position on the various subjects discussed.

1) As Jacques recall "Community over code" so :
  1.1 : It is important to trust all the members of the community in their will to improve OFBiz
  1.2 : the discussion cannot take place in 2 or 3 days, it is rather in a week or a month that it will take place.
(I don't have 2 or 3 hours to read (and answer) the mail every day, but I can find them during a week..
This is an advantage of mailing list discussions over chat discussions.

2) Deprecated (with documentation) before removing is a very good solution,
in a release migration process, when something has disappears (not yet move to the new solution), it's more easy to retrieve file when it was
deprecated to read associated documentation and know how to migrate.
  the rule is the same even it's simple util java method ;-)

3) I clearly not understand all implications/advantages/constrains ... or whatever about depend-on or component-load
what I only can be say is :
In a functional consultant point of view which want to configure an ofbiz with a lot of plugins (between 20 and 40) it's more easy to have a depend-on
on each plugin to be sure the loading order will be correct.

4) implementation detail or core choice, often, it's depend of point of view you use
On ERP development / implementation there are multiple type of people working on it with each one, specifics knowledge, usage, practice and point of view.
When someone says, it's a big change,
   start by trusting him and find out which process is affected to propose a solution;
   he doesn't want to block something, he wants to help you come up with a better solution that solves more cases.

5) patching is wrong : very often there are patch because hook or extend mechanism not exist so
When a plugin contains a patch, framework expert should explain
  or how to use existing extensibility mechanism to avoid the patch
  or how to improve the framework to provide a extensibility mechanism for this case
this point is also an example about previous point ;-)

6) what is OFBiz,
  - not only a public API ;-)
  - not only a framework
  - currently not a OOTB ERP  but I hope what, in future, there will contain some OOTB applications
In my long term view Apache OFBiz could be like linux, a core/kernel as small as possible with multiple components (the plugins) and so with some
distributions.
So, clearly, for me it is not possible to define the boundary between what is public (with backward compatibility) and what is only implementation.
Even on data model I can give some examples where not everyone will agree on what is in kernel and what is in components

just my 6 cents


Olivier

Le 06/01/2020 à 10:29, Samuel Trégouët a écrit :

>
> I also hope others contributors will eventually join (many thanks to
> Jacques for joining!) since this discussion  seems to me to be larger
> than a particular feature (component-load.xml): it is about the process
> of contributing to OFBiz.
>
>