Discussion: BuildBot component independence check

[Bruno Busco]

Hi devs,
I am trying to remove some OFBiz components but even removing few
specialpurpose components make the build or the data loading
impossible.
I think we need to gain back the components independence in a controlled way.
Even reviewing commits is not so easy to detect that a change
introduce a wrong dependence and then, when you try to separate things
its a complete mess.

What about to start a process that will take us to have a set of
different OFBiz configurations to be all built on BuildBot to be sure
that no wrong dependence has been introduced?

The first thing to do would be to agree to a dependence tree we want to rispect.
Then we could define one or more partial OFBiz configuration that
should work according to the approved dependence tree.
Then we could setup BuildBot to automatically build them for us.

Whet do you think about?

Thank you,
-Bruno

[Bruno Busco]

How could we define new ant targets to deploy partial OFBiz configurations?

I mean, an OFBiz configuration I would like to build and get working
is the one with the following applications:

- application/commonext
- application/content
- application/party
- application/securityext
- specialpurpose/myportal
- specialpurpose/ofbizwebsite

Does make sense to define a new ant target that removes all other
directories and then build what remains?
This new ant target could be executed in the BuildBot also.

-Bruno


2009/12/30 Bruno Busco <[hidden email]>:

> Hi devs,
> I am trying to remove some OFBiz components but even removing few
> specialpurpose components make the build or the data loading
> impossible.
> I think we need to gain back the components independence in a controlled way.
> Even reviewing commits is not so easy to detect that a change
> introduce a wrong dependence and then, when you try to separate things
> its a complete mess.
>
> What about to start a process that will take us to have a set of
> different OFBiz configurations to be all built on BuildBot to be sure
> that no wrong dependence has been introduced?
>
> The first thing to do would be to agree to a dependence tree we want to rispect.
> Then we could define one or more partial OFBiz configuration that
> should work according to the approved dependence tree.
> Then we could setup BuildBot to automatically build them for us.
>
> Whet do you think about?
>
> Thank you,
> -Bruno
>

[Adam Heath-2]

Bruno Busco wrote:

> How could we define new ant targets to deploy partial OFBiz configurations?
>
> I mean, an OFBiz configuration I would like to build and get working
> is the one with the following applications:
>
> - application/commonext
> - application/content
> - application/party
> - application/securityext
> - specialpurpose/myportal
> - specialpurpose/ofbizwebsite
>
> Does make sense to define a new ant target that removes all other
> directories and then build what remains?
> This new ant target could be executed in the BuildBot also.

For reference, the debian packages have a
-framework/-applications/-specialpurpose split.  I'd like to be able
to split each application and specialpurpose component into it's own
package, tho, and use dependencies/etc to bring them all back
together.  However, the components still aren't 100% separated yet.

[Bruno Busco]

Hi Adam,
thank you for the hint but I need some other (basic) information.
How should I use the debian package?
What is intended for?

I see a .pl script. Can I use it to get a partial (framework-only)
deploy directory from a checked out OFBiz working directory ?

Sorry for this basic questions.

-Bruno


2009/12/31 Adam Heath <[hidden email]>:

> Bruno Busco wrote:
>> How could we define new ant targets to deploy partial OFBiz configurations?
>>
>> I mean, an OFBiz configuration I would like to build and get working
>> is the one with the following applications:
>>
>> - application/commonext
>> - application/content
>> - application/party
>> - application/securityext
>> - specialpurpose/myportal
>> - specialpurpose/ofbizwebsite
>>
>> Does make sense to define a new ant target that removes all other
>> directories and then build what remains?
>> This new ant target could be executed in the BuildBot also.
>
> For reference, the debian packages have a
> -framework/-applications/-specialpurpose split.  I'd like to be able
> to split each application and specialpurpose component into it's own
> package, tho, and use dependencies/etc to bring them all back
> together.  However, the components still aren't 100% separated yet.
>
>

[Adam Heath-2]

Bruno Busco wrote:
> Hi Adam,
> thank you for the hint but I need some other (basic) information.
> How should I use the debian package?
> What is intended for?
>
> I see a .pl script. Can I use it to get a partial (framework-only)
> deploy directory from a checked out OFBiz working directory ?
>
> Sorry for this basic questions.

Update the debian/changelog, to at least contain the current svn
version. Update the date.  Do this by making a new stanza.

debian/rules clean
debian/rules binary

Install debs that get created in ..

There may be a few things that don't work, I've got a local git branch
that has a few very small improvements in debian/* that need to be added.

The packages do *not* support upgrades between different ofbiz
versions, because ofbiz doesn't support that itself.

[Bruno Busco]

Thank you Adam,
unfortunately I am not familiar with debian packages yet (consider
that I normally work under Windows).

In any case, what was my original goal, was to define some script or
whatever that could generate, starting from a fresh complete OFBiz
checkout, a modified OFBiz tree that could be used to build a partial
OFBiz configuration.

The script should be able to delete some directories and to apply some
SVN patches (tipically to change component-load.xml and build.xml
files).

I think that having these scripts in the SVN tree and using them in
the BuildBot to get these additional builds could help us to gradually
reduce components dependencies.

Should we simply add some more ant targets? It seems that ant can do
the job and works both on Linux and Windows.

-Bruno

2010/1/1 Adam Heath <[hidden email]>:

> Bruno Busco wrote:
>> Hi Adam,
>> thank you for the hint but I need some other (basic) information.
>> How should I use the debian package?
>> What is intended for?
>>
>> I see a .pl script. Can I use it to get a partial (framework-only)
>> deploy directory from a checked out OFBiz working directory ?
>>
>> Sorry for this basic questions.
>
> Update the debian/changelog, to at least contain the current svn
> version. Update the date.  Do this by making a new stanza.
>
> debian/rules clean
> debian/rules binary
>
> Install debs that get created in ..
>
> There may be a few things that don't work, I've got a local git branch
> that has a few very small improvements in debian/* that need to be added.
>
> The packages do *not* support upgrades between different ofbiz
> versions, because ofbiz doesn't support that itself.
>
>

[Jacopo Cappellato-4]

Instead of deleting folders the script could set the enabled="false" attribute in the ofbiz-component.xml files.

Jacopo

On Jan 2, 2010, at 10:27 AM, Bruno Busco wrote:

> Thank you Adam,
> unfortunately I am not familiar with debian packages yet (consider
> that I normally work under Windows).
>
> In any case, what was my original goal, was to define some script or
> whatever that could generate, starting from a fresh complete OFBiz
> checkout, a modified OFBiz tree that could be used to build a partial
> OFBiz configuration.
>
> The script should be able to delete some directories and to apply some
> SVN patches (tipically to change component-load.xml and build.xml
> files).
>
> I think that having these scripts in the SVN tree and using them in
> the BuildBot to get these additional builds could help us to gradually
> reduce components dependencies.
>
> Should we simply add some more ant targets? It seems that ant can do
> the job and works both on Linux and Windows.
>
> -Bruno
>
> 2010/1/1 Adam Heath <[hidden email]>:
>> Bruno Busco wrote:
>>> Hi Adam,
>>> thank you for the hint but I need some other (basic) information.
>>> How should I use the debian package?
>>> What is intended for?
>>>
>>> I see a .pl script. Can I use it to get a partial (framework-only)
>>> deploy directory from a checked out OFBiz working directory ?
>>>
>>> Sorry for this basic questions.
>>
>> Update the debian/changelog, to at least contain the current svn
>> version. Update the date.  Do this by making a new stanza.
>>
>> debian/rules clean
>> debian/rules binary
>>
>> Install debs that get created in ..
>>
>> There may be a few things that don't work, I've got a local git branch
>> that has a few very small improvements in debian/* that need to be added.
>>
>> The packages do *not* support upgrades between different ofbiz
>> versions, because ofbiz doesn't support that itself.
>>
>>

[Adam Heath-2]

Bruno Busco wrote:
> The script should be able to delete some directories and to apply some
> SVN patches (tipically to change component-load.xml and build.xml
> files).

Actually, that's not the best approach.  It would be better to modify
the build.xml and java classes to be more extensible, and not
hard-code things into framework/base/config/ofbiz-components.xml, that
reference hard-coded classes located in other components.(for example).

I'd love to be able to drop a component-into hot-deploy, and have a
new container automatically started.  This would make switching
between jetty and catalina simpler, and would allow for an asterisk
component(which requires constantly running code).

[Bruno Busco]

This is OK for the runtime but will still make available all the
packages during the build process.
Some unwanted java code dependencies would not be spotted out.

-Bruno

2010/1/2 Jacopo Cappellato <[hidden email]>:

> Instead of deleting folders the script could set the enabled="false" attribute in the ofbiz-component.xml files.
>
> Jacopo
>
> On Jan 2, 2010, at 10:27 AM, Bruno Busco wrote:
>
>> Thank you Adam,
>> unfortunately I am not familiar with debian packages yet (consider
>> that I normally work under Windows).
>>
>> In any case, what was my original goal, was to define some script or
>> whatever that could generate, starting from a fresh complete OFBiz
>> checkout, a modified OFBiz tree that could be used to build a partial
>> OFBiz configuration.
>>
>> The script should be able to delete some directories and to apply some
>> SVN patches (tipically to change component-load.xml and build.xml
>> files).
>>
>> I think that having these scripts in the SVN tree and using them in
>> the BuildBot to get these additional builds could help us to gradually
>> reduce components dependencies.
>>
>> Should we simply add some more ant targets? It seems that ant can do
>> the job and works both on Linux and Windows.
>>
>> -Bruno
>>
>> 2010/1/1 Adam Heath <[hidden email]>:
>>> Bruno Busco wrote:
>>>> Hi Adam,
>>>> thank you for the hint but I need some other (basic) information.
>>>> How should I use the debian package?
>>>> What is intended for?
>>>>
>>>> I see a .pl script. Can I use it to get a partial (framework-only)
>>>> deploy directory from a checked out OFBiz working directory ?
>>>>
>>>> Sorry for this basic questions.
>>>
>>> Update the debian/changelog, to at least contain the current svn
>>> version. Update the date.  Do this by making a new stanza.
>>>
>>> debian/rules clean
>>> debian/rules binary
>>>
>>> Install debs that get created in ..
>>>
>>> There may be a few things that don't work, I've got a local git branch
>>> that has a few very small improvements in debian/* that need to be added.
>>>
>>> The packages do *not* support upgrades between different ofbiz
>>> versions, because ofbiz doesn't support that itself.
>>>
>>>
>
>

[Scott Gray-2]

Doesn't the build order prevent this?  If one were to try and add a  
java dependency from the framework to the applications then the build  
would fail would it not?

While it is certainly possible, I'm entirely convinced of the need to  
use buildbot for this.  The reason buildbot is great for the tests is  
because it is so easy to unwittingly break them yet difficult to track  
down and fix after the fact.

With component dependencies all that is really required is a firm  
understanding by all committers of what the dependencies are and why  
they should be enforced.  I really think it's easy to avoid creating  
dependency problems by simply being aware of them and for the most  
part they are pretty easy to spot and to fix.  I would suggest that  
the main reason there are problems right now is because no one is  
making enough noise about the dependencies not being enforced.

Additionally, there is no point setting up a buildbot process that is  
going to fail so the dependencies would need to be fixed before this  
discussion can really come to any sort of resolution.  At that point  
the community may wish to consider completely separating the framework  
and applications as sub projects and if that happened then we'd need  
to setup completely different buildbot processes and the dependency  
issue would become somewhat irrelevant by complete separation.

Regards
Scott

HotWax Media
http://www.hotwaxmedia.com

On 2/01/2010, at 11:31 PM, Bruno Busco wrote:

> This is OK for the runtime but will still make available all the
> packages during the build process.
> Some unwanted java code dependencies would not be spotted out.
>
> -Bruno
>
> 2010/1/2 Jacopo Cappellato <[hidden email]>:
>> Instead of deleting folders the script could set the  
>> enabled="false" attribute in the ofbiz-component.xml files.
>>
>> Jacopo
>>
>> On Jan 2, 2010, at 10:27 AM, Bruno Busco wrote:
>>
>>> Thank you Adam,
>>> unfortunately I am not familiar with debian packages yet (consider
>>> that I normally work under Windows).
>>>
>>> In any case, what was my original goal, was to define some script or
>>> whatever that could generate, starting from a fresh complete OFBiz
>>> checkout, a modified OFBiz tree that could be used to build a  
>>> partial
>>> OFBiz configuration.
>>>
>>> The script should be able to delete some directories and to apply  
>>> some
>>> SVN patches (tipically to change component-load.xml and build.xml
>>> files).
>>>
>>> I think that having these scripts in the SVN tree and using them in
>>> the BuildBot to get these additional builds could help us to  
>>> gradually
>>> reduce components dependencies.
>>>
>>> Should we simply add some more ant targets? It seems that ant can do
>>> the job and works both on Linux and Windows.
>>>
>>> -Bruno
>>>
>>> 2010/1/1 Adam Heath <[hidden email]>:
>>>> Bruno Busco wrote:
>>>>> Hi Adam,
>>>>> thank you for the hint but I need some other (basic) information.
>>>>> How should I use the debian package?
>>>>> What is intended for?
>>>>>
>>>>> I see a .pl script. Can I use it to get a partial (framework-only)
>>>>> deploy directory from a checked out OFBiz working directory ?
>>>>>
>>>>> Sorry for this basic questions.
>>>>
>>>> Update the debian/changelog, to at least contain the current svn
>>>> version. Update the date.  Do this by making a new stanza.
>>>>
>>>> debian/rules clean
>>>> debian/rules binary
>>>>
>>>> Install debs that get created in ..
>>>>
>>>> There may be a few things that don't work, I've got a local git  
>>>> branch
>>>> that has a few very small improvements in debian/* that need to  
>>>> be added.
>>>>
>>>> The packages do *not* support upgrades between different ofbiz
>>>> versions, because ofbiz doesn't support that itself.
>>>>
>>>>
>>
>>

[Bruno Busco]

Scott,
if I proposed the BuildBot to be setup to check for the dependencies
is because I do not think that making noise about this is enough.
Framework separation is something we speak about since a lot of time
but no steps have been done because, I guess, it is not a community
high priority, but, only a wish.

Setting the BuildBot will:
1) make all aware of what really means for the community the
framework-only installation. We will start with a certain component
set installed, we will then discuss if this or that application should
be included or not and then we will end with a final decision.
2) for sure we will have a failed build at the beginning but we will
also have clear what are the dependence issues and all community can
work to reduce them (or at least to not increase theme)
3) when the framework-only build will be ok we could use the BuildBot
to daily build, test and package as we do with the complete OFBiz.

-Bruno

2010/1/2 Scott Gray <[hidden email]>:

> Doesn't the build order prevent this?  If one were to try and add a java
> dependency from the framework to the applications then the build would fail
> would it not?
>
> While it is certainly possible, I'm entirely convinced of the need to use
> buildbot for this.  The reason buildbot is great for the tests is because it
> is so easy to unwittingly break them yet difficult to track down and fix
> after the fact.
>
> With component dependencies all that is really required is a firm
> understanding by all committers of what the dependencies are and why they
> should be enforced.  I really think it's easy to avoid creating dependency
> problems by simply being aware of them and for the most part they are pretty
> easy to spot and to fix.  I would suggest that the main reason there are
> problems right now is because no one is making enough noise about the
> dependencies not being enforced.
>
> Additionally, there is no point setting up a buildbot process that is going
> to fail so the dependencies would need to be fixed before this discussion
> can really come to any sort of resolution.  At that point the community may
> wish to consider completely separating the framework and applications as sub
> projects and if that happened then we'd need to setup completely different
> buildbot processes and the dependency issue would become somewhat irrelevant
> by complete separation.
>
> Regards
> Scott
>
> HotWax Media
> http://www.hotwaxmedia.com
>
> On 2/01/2010, at 11:31 PM, Bruno Busco wrote:
>
>> This is OK for the runtime but will still make available all the
>> packages during the build process.
>> Some unwanted java code dependencies would not be spotted out.
>>
>> -Bruno
>>
>> 2010/1/2 Jacopo Cappellato <[hidden email]>:
>>>
>>> Instead of deleting folders the script could set the enabled="false"
>>> attribute in the ofbiz-component.xml files.
>>>
>>> Jacopo
>>>
>>> On Jan 2, 2010, at 10:27 AM, Bruno Busco wrote:
>>>
>>>> Thank you Adam,
>>>> unfortunately I am not familiar with debian packages yet (consider
>>>> that I normally work under Windows).
>>>>
>>>> In any case, what was my original goal, was to define some script or
>>>> whatever that could generate, starting from a fresh complete OFBiz
>>>> checkout, a modified OFBiz tree that could be used to build a partial
>>>> OFBiz configuration.
>>>>
>>>> The script should be able to delete some directories and to apply some
>>>> SVN patches (tipically to change component-load.xml and build.xml
>>>> files).
>>>>
>>>> I think that having these scripts in the SVN tree and using them in
>>>> the BuildBot to get these additional builds could help us to gradually
>>>> reduce components dependencies.
>>>>
>>>> Should we simply add some more ant targets? It seems that ant can do
>>>> the job and works both on Linux and Windows.
>>>>
>>>> -Bruno
>>>>
>>>> 2010/1/1 Adam Heath <[hidden email]>:
>>>>>
>>>>> Bruno Busco wrote:
>>>>>>
>>>>>> Hi Adam,
>>>>>> thank you for the hint but I need some other (basic) information.
>>>>>> How should I use the debian package?
>>>>>> What is intended for?
>>>>>>
>>>>>> I see a .pl script. Can I use it to get a partial (framework-only)
>>>>>> deploy directory from a checked out OFBiz working directory ?
>>>>>>
>>>>>> Sorry for this basic questions.
>>>>>
>>>>> Update the debian/changelog, to at least contain the current svn
>>>>> version. Update the date.  Do this by making a new stanza.
>>>>>
>>>>> debian/rules clean
>>>>> debian/rules binary
>>>>>
>>>>> Install debs that get created in ..
>>>>>
>>>>> There may be a few things that don't work, I've got a local git branch
>>>>> that has a few very small improvements in debian/* that need to be
>>>>> added.
>>>>>
>>>>> The packages do *not* support upgrades between different ofbiz
>>>>> versions, because ofbiz doesn't support that itself.
>>>>>
>>>>>
>>>
>>>
>
>

[Bruno Busco]

Bump this up since now seems others are interested.

-Bruno

2010/1/2 Bruno Busco <[hidden email]>:

> Scott,
> if I proposed the BuildBot to be setup to check for the dependencies
> is because I do not think that making noise about this is enough.
> Framework separation is something we speak about since a lot of time
> but no steps have been done because, I guess, it is not a community
> high priority, but, only a wish.
>
> Setting the BuildBot will:
> 1) make all aware of what really means for the community the
> framework-only installation. We will start with a certain component
> set installed, we will then discuss if this or that application should
> be included or not and then we will end with a final decision.
> 2) for sure we will have a failed build at the beginning but we will
> also have clear what are the dependence issues and all community can
> work to reduce them (or at least to not increase theme)
> 3) when the framework-only build will be ok we could use the BuildBot
> to daily build, test and package as we do with the complete OFBiz.
>
> -Bruno
>
> 2010/1/2 Scott Gray <[hidden email]>:
>> Doesn't the build order prevent this?  If one were to try and add a java
>> dependency from the framework to the applications then the build would fail
>> would it not?
>>
>> While it is certainly possible, I'm entirely convinced of the need to use
>> buildbot for this.  The reason buildbot is great for the tests is because it
>> is so easy to unwittingly break them yet difficult to track down and fix
>> after the fact.
>>
>> With component dependencies all that is really required is a firm
>> understanding by all committers of what the dependencies are and why they
>> should be enforced.  I really think it's easy to avoid creating dependency
>> problems by simply being aware of them and for the most part they are pretty
>> easy to spot and to fix.  I would suggest that the main reason there are
>> problems right now is because no one is making enough noise about the
>> dependencies not being enforced.
>>
>> Additionally, there is no point setting up a buildbot process that is going
>> to fail so the dependencies would need to be fixed before this discussion
>> can really come to any sort of resolution.  At that point the community may
>> wish to consider completely separating the framework and applications as sub
>> projects and if that happened then we'd need to setup completely different
>> buildbot processes and the dependency issue would become somewhat irrelevant
>> by complete separation.
>>
>> Regards
>> Scott
>>
>> HotWax Media
>> http://www.hotwaxmedia.com
>>
>> On 2/01/2010, at 11:31 PM, Bruno Busco wrote:
>>
>>> This is OK for the runtime but will still make available all the
>>> packages during the build process.
>>> Some unwanted java code dependencies would not be spotted out.
>>>
>>> -Bruno
>>>
>>> 2010/1/2 Jacopo Cappellato <[hidden email]>:
>>>>
>>>> Instead of deleting folders the script could set the enabled="false"
>>>> attribute in the ofbiz-component.xml files.
>>>>
>>>> Jacopo
>>>>
>>>> On Jan 2, 2010, at 10:27 AM, Bruno Busco wrote:
>>>>
>>>>> Thank you Adam,
>>>>> unfortunately I am not familiar with debian packages yet (consider
>>>>> that I normally work under Windows).
>>>>>
>>>>> In any case, what was my original goal, was to define some script or
>>>>> whatever that could generate, starting from a fresh complete OFBiz
>>>>> checkout, a modified OFBiz tree that could be used to build a partial
>>>>> OFBiz configuration.
>>>>>
>>>>> The script should be able to delete some directories and to apply some
>>>>> SVN patches (tipically to change component-load.xml and build.xml
>>>>> files).
>>>>>
>>>>> I think that having these scripts in the SVN tree and using them in
>>>>> the BuildBot to get these additional builds could help us to gradually
>>>>> reduce components dependencies.
>>>>>
>>>>> Should we simply add some more ant targets? It seems that ant can do
>>>>> the job and works both on Linux and Windows.
>>>>>
>>>>> -Bruno
>>>>>
>>>>> 2010/1/1 Adam Heath <[hidden email]>:
>>>>>>
>>>>>> Bruno Busco wrote:
>>>>>>>
>>>>>>> Hi Adam,
>>>>>>> thank you for the hint but I need some other (basic) information.
>>>>>>> How should I use the debian package?
>>>>>>> What is intended for?
>>>>>>>
>>>>>>> I see a .pl script. Can I use it to get a partial (framework-only)
>>>>>>> deploy directory from a checked out OFBiz working directory ?
>>>>>>>
>>>>>>> Sorry for this basic questions.
>>>>>>
>>>>>> Update the debian/changelog, to at least contain the current svn
>>>>>> version. Update the date.  Do this by making a new stanza.
>>>>>>
>>>>>> debian/rules clean
>>>>>> debian/rules binary
>>>>>>
>>>>>> Install debs that get created in ..
>>>>>>
>>>>>> There may be a few things that don't work, I've got a local git branch
>>>>>> that has a few very small improvements in debian/* that need to be
>>>>>> added.
>>>>>>
>>>>>> The packages do *not* support upgrades between different ofbiz
>>>>>> versions, because ofbiz doesn't support that itself.
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
>