Executing unit tests with Cobertura code coverage metrics

[Bob Morley]

Got this running this morning.  Have created a minor set of tweaks as ticket OFBIZ-3651; if someone familiar with the Cobertura integration could review that would be helpful.

If someone is trying to run these here are a few little pointers:

- download Cobertura v1.9.3, drop the cobertura.jar into base/lib, rename to cobertura-1.9.3.jar
- update Eclipse .classpath to include this jar
- you can exercise your unit tests using Start with a test parameter and the jars should be instrumented and metrics should be collected in Eclipse OR you can simply run everything with the ant run-tests target.
- executing the ant cobertura-report target will create a very nice set of html reports

Here is my question -- when looking at the reports it showed 100% line code coverage in UtilValidate (for example) but this was for 111 lines.  Clearly this class has many more lines than that, and when I opened it up I saw that large portions of it were not marked green or red in the report.  What gives?  :)

[Adam Heath-2]

Bob Morley wrote:

> Got this running this morning.  Have created a minor set of tweaks as ticket
> OFBIZ-3651; if someone familiar with the Cobertura integration could review
> that would be helpful.
>
> If someone is trying to run these here are a few little pointers:
>
> - download Cobertura v1.9.3, drop the cobertura.jar into base/lib, rename to
> cobertura-1.9.3.jar
> - update Eclipse .classpath to include this jar
> - you can exercise your unit tests using Start with a test parameter and the
> jars should be instrumented and metrics should be collected in Eclipse OR
> you can simply run everything with the ant run-tests target.
> - executing the ant cobertura-report target will create a very nice set of
> html reports
>
> Here is my question -- when looking at the reports it showed 100% line code
> coverage in UtilValidate (for example) but this was for 111 lines.  Clearly
> this class has many more lines than that, and when I opened it up I saw that
> large portions of it were not marked green or red in the report.  What
> gives?  :)

Coverage in framework/base sucks when running run-tests at the
top-level.  There are overlapping classloaders, and this confuses
cobertura.

Running component tests directly in framework/base, doing 'ant tests'
does give correct coverage metrics.

[Bob Morley]

On Apr 6, 2010, at 1:50 PM, Adam Heath wrote:

> Bob Morley wrote:
>>
>> Here is my question -- when looking at the reports it showed 100%  
>> line code
>> coverage in UtilValidate (for example) but this was for 111 lines.  
>> Clearly
>> this class has many more lines than that, and when I opened it up I  
>> saw that
>> large portions of it were not marked green or red in the report.  
>> What
>> gives?  :)
>
>
> Coverage in framework/base sucks when running run-tests at the
> top-level.  There are overlapping classloaders, and this confuses
> cobertura.
>
> Running component tests directly in framework/base, doing 'ant tests'
> does give correct coverage metrics.

Does this only apply to framework/base?  When I looked at coverage of  
UtilAccounting (for example) it looks pretty accurate.

Also, we were talking in the office -- our understanding is that the  
Cobertura license would restrict Ofbiz from redistribution, but it  
should be able to use it as part of their build process.  Do you think  
there would be an issue include a target that downloads and deploys  
Cobertura and executes those targets as part of our build process  
internally?  That way we could get these metrics published as we move  
forward.  (Naturally we would have to fix the issue referred to  
above.  Thoughts?

[Adrian Crum]

Robert Morley wrote:
> Also, we were talking in the office -- our understanding is that the
> Cobertura license would restrict Ofbiz from redistribution, but it
> should be able to use it as part of their build process.  Do you think
> there would be an issue include a target that downloads and deploys
> Cobertura and executes those targets as part of our build process
> internally?  That way we could get these metrics published as we move
> forward.  (Naturally we would have to fix the issue referred to above.  
> Thoughts?

That's how Apache Commons does things. They pull down all those cool
tools - like CPD, RAT, etc.

[Scott Gray-2]

On 6/04/2010, at 11:59 AM, Robert Morley wrote:

>
> On Apr 6, 2010, at 1:50 PM, Adam Heath wrote:
>
>> Bob Morley wrote:
>>>
>>> Here is my question -- when looking at the reports it showed 100% line code
>>> coverage in UtilValidate (for example) but this was for 111 lines.  Clearly
>>> this class has many more lines than that, and when I opened it up I saw that
>>> large portions of it were not marked green or red in the report.  What
>>> gives?  :)
>>
>>
>> Coverage in framework/base sucks when running run-tests at the
>> top-level.  There are overlapping classloaders, and this confuses
>> cobertura.
>>
>> Running component tests directly in framework/base, doing 'ant tests'
>> does give correct coverage metrics.
>
> Does this only apply to framework/base?  When I looked at coverage of UtilAccounting (for example) it looks pretty accurate.
>
> Also, we were talking in the office -- our understanding is that the Cobertura license would restrict Ofbiz from redistribution, but it should be able to use it as part of their build process.  Do you think there would be an issue include a target that downloads and deploys Cobertura and executes those targets as part of our build process internally?  That way we could get these metrics published as we move forward.  (Naturally we would have to fix the issue referred to above.  Thoughts?

A target to download it should be fine and an optional dependency on it should also be fine, but I don't think it's okay to make it a required part of the build process even if the jar isn't being distributed with OFBiz.

Regards
Scott

[Adam Heath-2]

Robert Morley wrote:

>
> On Apr 6, 2010, at 1:50 PM, Adam Heath wrote:
>
>> Bob Morley wrote:
>>>
>>> Here is my question -- when looking at the reports it showed 100%
>>> line code
>>> coverage in UtilValidate (for example) but this was for 111 lines.
>>> Clearly
>>> this class has many more lines than that, and when I opened it up I
>>> saw that
>>> large portions of it were not marked green or red in the report.  What
>>> gives?  :)
>>
>>
>> Coverage in framework/base sucks when running run-tests at the
>> top-level.  There are overlapping classloaders, and this confuses
>> cobertura.
>>
>> Running component tests directly in framework/base, doing 'ant tests'
>> does give correct coverage metrics.
>
> Does this only apply to framework/base?  When I looked at coverage of
> UtilAccounting (for example) it looks pretty accurate.

ant run-tests at the top level has broken metrics on framework/base,
everything else is fine.

ant tests run in framework/base gives correct metrics on
framework/base, but nothing else, obviously.

> Also, we were talking in the office -- our understanding is that the
> Cobertura license would restrict Ofbiz from redistribution, but it
> should be able to use it as part of their build process.  Do you think
> there would be an issue include a target that downloads and deploys
> Cobertura and executes those targets as part of our build process
> internally?  That way we could get these metrics published as we move
> forward.  (Naturally we would have to fix the issue referred to above.
> Thoughts?

Don't know on this.

[Adam Heath-2]

Scott Gray wrote:

> On 6/04/2010, at 11:59 AM, Robert Morley wrote:
>
>> On Apr 6, 2010, at 1:50 PM, Adam Heath wrote:
>>
>>> Bob Morley wrote:
>>>> Here is my question -- when looking at the reports it showed 100% line code
>>>> coverage in UtilValidate (for example) but this was for 111 lines.  Clearly
>>>> this class has many more lines than that, and when I opened it up I saw that
>>>> large portions of it were not marked green or red in the report.  What
>>>> gives?  :)
>>>
>>> Coverage in framework/base sucks when running run-tests at the
>>> top-level.  There are overlapping classloaders, and this confuses
>>> cobertura.
>>>
>>> Running component tests directly in framework/base, doing 'ant tests'
>>> does give correct coverage metrics.
>> Does this only apply to framework/base?  When I looked at coverage of UtilAccounting (for example) it looks pretty accurate.
>>
>> Also, we were talking in the office -- our understanding is that the Cobertura license would restrict Ofbiz from redistribution, but it should be able to use it as part of their build process.  Do you think there would be an issue include a target that downloads and deploys Cobertura and executes those targets as part of our build process internally?  That way we could get these metrics published as we move forward.  (Naturally we would have to fix the issue referred to above.  Thoughts?
>
> A target to download it should be fine and an optional dependency on it should also be fine, but I don't think it's okay to make it a required part of the build process even if the jar isn't being distributed with OFBiz.

It's certainly not required.  If the jar is available at build time,
the class gets compiled.  I hard-code loading it(during run-tests
invocation), but if the class is not found, I ignore the error.  If
the class *is* found, but cobertura itself is not, then that error is
ignored too(which could happen if cobertura is removed after compiling).

[Jacques Le Roux]

From: "Adrian Crum" <[hidden email]>

> Robert Morley wrote:
>> Also, we were talking in the office -- our understanding is that the
>> Cobertura license would restrict Ofbiz from redistribution, but it
>> should be able to use it as part of their build process.  Do you think
>> there would be an issue include a target that downloads and deploys
>> Cobertura and executes those targets as part of our build process
>> internally?  That way we could get these metrics published as we move
>> forward.  (Naturally we would have to fix the issue referred to above.  
>> Thoughts?
>
> That's how Apache Commons does things. They pull down all those cool
> tools - like CPD, RAT, etc.

RAT is Apache but yes, sounds like a possible way indeed

Jacques