Suggestions on improving Buildbot Builds

[gavin-105]

        Hi All,

So recently I worked on curing issues with your buildbot build at

http://ci.apache.org/builders/ofbiz-trunk 

Part of the build involves running Apache RAT on your code and producing a report at

http://ci.apache.org/projects/ofbiz/rat-output.html 

A summary of which also appears on the league table of participating projects at

http://ci.apache.org/projects/rat-master-summary.html 


You have quite a bit of code that RAT has to go through in order to compile these reports and
this takes up over an hour of your build on its own. Your build gets run on a per commit basis
so this RAT report is run each and every time  a commit is made.

I'd like to propose two suggestions.

Suggestion One :
==============

We remove the RAT report steps from your normal build and stick those steps
into its own build, separate of any other build steps. This build would run via a nightly trigger only.

This would bring your normal build down to just 15 minutes, and is then fine to continue to
run on a per commit basis, concentrating on compile checks, docs and uploading of snapshots.

Suggestion Two :
==============

Your RAT report says there are 2273 (currently) invalid or unlicensed files. There are obviously
some false reports in there, such as generated files that probably should not be checked.

The suggestion is, to go through the report at

http://ci.apache.org/projects/ofbiz/rat-output.html 

and see what files and/or directories can be excluded from the RAT check.
Lets get it down to a realistic figure that can then show you exactly what genuine files are missing
license headers.

This can be self-served by the addition of a rat-excludes file in your svn. I then tell RAT to read this
file and the excludes are automatic.

See the 'More Reading and links' section of

http://ci.apache.org/projects/rat-master-summary.html

for example rat-exclude files.

I am happy to get you started and create a patch with a starter rat-excludes file.

If you are happy with these suggestions, let me know and I'll get started.

Gav…

[Jacques Le Roux]

Hi Gavin,

Thanks for your comments, inline...

Le 03/09/2014 10:23, Gavin McDonald a écrit :

> Hi All,
>
> So recently I worked on curing issues with your buildbot build at
>
> http://ci.apache.org/builders/ofbiz-trunk
>
> Part of the build involves running Apache RAT on your code and producing a report at
>
> http://ci.apache.org/projects/ofbiz/rat-output.html
>
> A summary of which also appears on the league table of participating projects at
>
> http://ci.apache.org/projects/rat-master-summary.html
>
>
> You have quite a bit of code that RAT has to go through in order to compile these reports and
> this takes up over an hour of your build on its own. Your build gets run on a per commit basis
> so this RAT report is run each and every time  a commit is made.
>
> I'd like to propose two suggestions.
>
> Suggestion One :
> ==============
>
> We remove the RAT report steps from your normal build and stick those steps
> into its own build, separate of any other build steps. This build would run via a nightly trigger only.
>
> This would bring your normal build down to just 15 minutes, and is then fine to continue to
> run on a per commit basis, concentrating on compile checks, docs and uploading of snapshots.

I think in all cases we don't need to run the RAT report on each commit, nightly would be well enough

>
> Suggestion Two :
> ==============
>
> Your RAT report says there are 2273 (currently) invalid or unlicensed files. There are obviously
> some false reports in there, such as generated files that probably should not be checked.
>
> The suggestion is, to go through the report at
>
> http://ci.apache.org/projects/ofbiz/rat-output.html
>
> and see what files and/or directories can be excluded from the RAT check.
> Lets get it down to a realistic figure that can then show you exactly what genuine files are missing
> license headers.
>
> This can be self-served by the addition of a rat-excludes file in your svn. I then tell RAT to read this
> file and the excludes are automatic.
>
> See the 'More Reading and links' section of
>
> http://ci.apache.org/projects/rat-master-summary.html
>
> for example rat-exclude files.
>
> I am happy to get you started and create a patch with a starter rat-excludes file.
>
> If you are happy with these suggestions, let me know and I'll get started.

That would be great, even if at the moment we are not doing a great use of RAT.
This remembers me also this thread http://markmail.org/message/jkeqbdtf5im6zjlm I have completely forgottten it since :/ (as ever, bigger fishes...)

Jacques

>
> Gav…
>
>
>

[Jacques Le Roux]

Ha forgot, you may still  have a copy of an email  "Rat xsl sylesheet used in builbot scripts"  from [hidden email]
At least I have it here if you think it can behelpful

Jacques

Le 03/09/2014 11:57, Jacques Le Roux a écrit :

> Hi Gavin,
>
> Thanks for your comments, inline...
>
> Le 03/09/2014 10:23, Gavin McDonald a écrit :
>>     Hi All,
>>
>> So recently I worked on curing issues with your buildbot build at
>>
>> http://ci.apache.org/builders/ofbiz-trunk
>>
>> Part of the build involves running Apache RAT on your code and producing a report at
>>
>> http://ci.apache.org/projects/ofbiz/rat-output.html
>>
>> A summary of which also appears on the league table of participating projects at
>>
>> http://ci.apache.org/projects/rat-master-summary.html
>>
>>
>> You have quite a bit of code that RAT has to go through in order to compile these reports and
>> this takes up over an hour of your build on its own. Your build gets run on a per commit basis
>> so this RAT report is run each and every time  a commit is made.
>>
>> I'd like to propose two suggestions.
>>
>> Suggestion One :
>> ==============
>>
>> We remove the RAT report steps from your normal build and stick those steps
>> into its own build, separate of any other build steps. This build would run via a nightly trigger only.
>>
>> This would bring your normal build down to just 15 minutes, and is then fine to continue to
>> run on a per commit basis, concentrating on compile checks, docs and uploading of snapshots.
>
> I think in all cases we don't need to run the RAT report on each commit, nightly would be well enough
>
>>
>> Suggestion Two :
>> ==============
>>
>> Your RAT report says there are 2273 (currently) invalid or unlicensed files. There are obviously
>> some false reports in there, such as generated files that probably should not be checked.
>>
>> The suggestion is, to go through the report at
>>
>> http://ci.apache.org/projects/ofbiz/rat-output.html
>>
>> and see what files and/or directories can be excluded from the RAT check.
>> Lets get it down to a realistic figure that can then show you exactly what genuine files are missing
>> license headers.
>>
>> This can be self-served by the addition of a rat-excludes file in your svn. I then tell RAT to read this
>> file and the excludes are automatic.
>>
>> See the 'More Reading and links' section of
>>
>> http://ci.apache.org/projects/rat-master-summary.html
>>
>> for example rat-exclude files.
>>
>> I am happy to get you started and create a patch with a starter rat-excludes file.
>>
>> If you are happy with these suggestions, let me know and I'll get started.
>
> That would be great, even if at the moment we are not doing a great use of RAT.
> This remembers me also this thread http://markmail.org/message/jkeqbdtf5im6zjlm I have completely forgottten it since :/ (as ever, bigger fishes...)
>
> Jacques
>
>>
>> Gav…
>>
>>
>>
>

[Jacques Le Roux]

Hi Gavin,

We are still running RAT report (at least in trunk), normal?

Jacques

Le 03/09/2014 12:02, Jacques Le Roux a écrit :

> Ha forgot, you may still  have a copy of an email  "Rat xsl sylesheet used in builbot scripts"  from [hidden email]
> At least I have it here if you think it can behelpful
>
> Jacques
>
> Le 03/09/2014 11:57, Jacques Le Roux a écrit :
>> Hi Gavin,
>>
>> Thanks for your comments, inline...
>>
>> Le 03/09/2014 10:23, Gavin McDonald a écrit :
>>>     Hi All,
>>>
>>> So recently I worked on curing issues with your buildbot build at
>>>
>>> http://ci.apache.org/builders/ofbiz-trunk
>>>
>>> Part of the build involves running Apache RAT on your code and producing a report at
>>>
>>> http://ci.apache.org/projects/ofbiz/rat-output.html
>>>
>>> A summary of which also appears on the league table of participating projects at
>>>
>>> http://ci.apache.org/projects/rat-master-summary.html
>>>
>>>
>>> You have quite a bit of code that RAT has to go through in order to compile these reports and
>>> this takes up over an hour of your build on its own. Your build gets run on a per commit basis
>>> so this RAT report is run each and every time  a commit is made.
>>>
>>> I'd like to propose two suggestions.
>>>
>>> Suggestion One :
>>> ==============
>>>
>>> We remove the RAT report steps from your normal build and stick those steps
>>> into its own build, separate of any other build steps. This build would run via a nightly trigger only.
>>>
>>> This would bring your normal build down to just 15 minutes, and is then fine to continue to
>>> run on a per commit basis, concentrating on compile checks, docs and uploading of snapshots.
>>
>> I think in all cases we don't need to run the RAT report on each commit, nightly would be well enough
>>
>>>
>>> Suggestion Two :
>>> ==============
>>>
>>> Your RAT report says there are 2273 (currently) invalid or unlicensed files. There are obviously
>>> some false reports in there, such as generated files that probably should not be checked.
>>>
>>> The suggestion is, to go through the report at
>>>
>>> http://ci.apache.org/projects/ofbiz/rat-output.html
>>>
>>> and see what files and/or directories can be excluded from the RAT check.
>>> Lets get it down to a realistic figure that can then show you exactly what genuine files are missing
>>> license headers.
>>>
>>> This can be self-served by the addition of a rat-excludes file in your svn. I then tell RAT to read this
>>> file and the excludes are automatic.
>>>
>>> See the 'More Reading and links' section of
>>>
>>> http://ci.apache.org/projects/rat-master-summary.html
>>>
>>> for example rat-exclude files.
>>>
>>> I am happy to get you started and create a patch with a starter rat-excludes file.
>>>
>>> If you are happy with these suggestions, let me know and I'll get started.
>>
>> That would be great, even if at the moment we are not doing a great use of RAT.
>> This remembers me also this thread http://markmail.org/message/jkeqbdtf5im6zjlm I have completely forgottten it since :/ (as ever, bigger fishes...)
>>
>> Jacques
>>
>>>
>>> Gav…
>>>
>>>
>>>
>>
>