Failure with automated tests

[Robert Lor]

Hi,

I tried running the automated tests to see if my setup is correct and
noticed a number of failures.  Here's what I did:

1) svn co http://svn.apache.org/repos/asf/ofbiz/trunk ofbiz
2) setup db and modify entityengine.xml appropriately
2) ant run-install
3) java -jar ofbiz.jar -test

Are all the tests supposed to run successfully out of the box or am I
missing something in the setup?

The errors are slightly different between Postgres & MySQL.  Please see
the output in the attached file.


Thanks,
Robert

[BJ Freeman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

start first with the default DB
less things to track down
probably still get errors since there have been a lot of changes since
those were written


Robert Lor sent the following on 2/10/2009 3:15 PM:

> Hi,
>
> I tried running the automated tests to see if my setup is correct and
> noticed a number of failures.  Here's what I did:
>
> 1) svn co http://svn.apache.org/repos/asf/ofbiz/trunk ofbiz
> 2) setup db and modify entityengine.xml appropriately
> 2) ant run-install
> 3) java -jar ofbiz.jar -test
>
> Are all the tests supposed to run successfully out of the box or am I
> missing something in the setup?
>
> The errors are slightly different between Postgres & MySQL.  Please see
> the output in the attached file.
>
>
> Thanks,
> Robert

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJkjberP3NbaWWqE4RAmC8AJ4zAkfzpxK7Bxa6uj3ObBiXmbfHbACeJEdu
rhQgij+uN3UY0YjVdMPopA0=
=b26j
-----END PGP SIGNATURE-----

[Robert Lor]

BJ Freeman wrote:
> start first with the default DB
> less things to track down
> probably still get errors since there have been a lot of changes since
> those were written
>
>  
Just tried with Derby, and the errors are similar to those with MySQL
and Postgres.  I take it that the errors may be from the test cases
themselves. Is there a stable branch that passes all the test cases?

Thanks,
Robert

[BJ Freeman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

the rev 4.0 is the most stable but does not have the current enhancements.
Not sure the test cases have been verified on that version.
Since this is a community endeavor it would take someone to commit to
keeping the tests update.


Robert Lor sent the following on 2/10/2009 8:01 PM:

> BJ Freeman wrote:
>> start first with the default DB
>> less things to track down
>> probably still get errors since there have been a lot of changes since
>> those were written
>>
>>  
> Just tried with Derby, and the errors are similar to those with MySQL
> and Postgres.  I take it that the errors may be from the test cases
> themselves. Is there a stable branch that passes all the test cases?
>
> Thanks,
> Robert
>
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJkquUrP3NbaWWqE4RAsAgAKCOpOxZcDEvqPPKFo8gWqZPXhlwOQCdFN0H
11QVfFjvNzAux2kdFIEhQUw=
=x5qt
-----END PGP SIGNATURE-----

[Vikas Mayur-3]

Well, All the tests supposed to be run successfully. If you see the  
logs, note that all the test are not failed.
The only reason that I can think of is that In some cases, the data  
for all test cases are not defined properly. I mean few of the test  
case does not get the desired data and which is when changed by any  
earlier test causes them to fail.

Other reason could be recent changes in services that are being tested  
by test case.

Also the whole automated test-suite is supposed to run with fresh data  
each time.

Vikas

On Feb 11, 2009, at 4:45 AM, Robert Lor wrote:

> Hi,
>
> I tried running the automated tests to see if my setup is correct  
> and noticed a number of failures.  Here's what I did:
>
> 1) svn co http://svn.apache.org/repos/asf/ofbiz/trunk ofbiz
> 2) setup db and modify entityengine.xml appropriately
> 2) ant run-install
> 3) java -jar ofbiz.jar -test
>
> Are all the tests supposed to run successfully out of the box or am  
> I missing something in the setup?
>
> The errors are slightly different between Postgres & MySQL.  Please  
> see the output in the attached file.
>
>
> Thanks,
> Robert
> <test_log.tar.gz>

[David E Jones-3]

Yes, that's a good point Vikas, about the "fresh data". One of the  
issues with the current tests and test infrastructure is cleanup after  
running, and tools to make cleanup easier too.

For the current tests to run successfully you must have a fresh  
database with the OOTB demo data in place before testing each  
component. That means something like "ant clean-all; ant run-install;  
java -jar ofbiz.jar tests component=XXX; ant clean-all; ant run-
install; java -jar ...; ...".

There are quite a few things that have been discussed but either not  
developed or just not committed related to testing, so this is  
definitely an area that needs a lot of love! Since most things are  
ultimately driven by end-users of OFBiz it would help a lot if end-
users pushed for these, but that has only happened approximately 3  
times EVER for OFBiz AFAIK, as opposed to many hundreds (perhaps  
thousands) of times for other functionality and new features.

-David


On Feb 11, 2009, at 10:50 AM, Vikas Mayur wrote:

> Well, All the tests supposed to be run successfully. If you see the  
> logs, note that all the test are not failed.
> The only reason that I can think of is that In some cases, the data  
> for all test cases are not defined properly. I mean few of the test  
> case does not get the desired data and which is when changed by any  
> earlier test causes them to fail.
>
> Other reason could be recent changes in services that are being  
> tested by test case.
>
> Also the whole automated test-suite is supposed to run with fresh  
> data each time.
>
> Vikas
>
> On Feb 11, 2009, at 4:45 AM, Robert Lor wrote:
>
>> Hi,
>>
>> I tried running the automated tests to see if my setup is correct  
>> and noticed a number of failures.  Here's what I did:
>>
>> 1) svn co http://svn.apache.org/repos/asf/ofbiz/trunk ofbiz
>> 2) setup db and modify entityengine.xml appropriately
>> 2) ant run-install
>> 3) java -jar ofbiz.jar -test
>>
>> Are all the tests supposed to run successfully out of the box or am  
>> I missing something in the setup?
>>
>> The errors are slightly different between Postgres & MySQL.  Please  
>> see the output in the attached file.
>>
>>
>> Thanks,
>> Robert
>> <test_log.tar.gz>
>

[Vikas Mayur-3]

David, Thanks for bringing up this issue and indeed this is an area  
that need lot of contribution to make the whole bunch of test suite to  
a good level.

Robert (and anyone interested), For more information please refer to https://issues.apache.org/jira/browse/OFBIZ-1463 
.


Vikas


On Feb 12, 2009, at 1:14 AM, David E Jones wrote:

>
> Yes, that's a good point Vikas, about the "fresh data". One of the  
> issues with the current tests and test infrastructure is cleanup  
> after running, and tools to make cleanup easier too.
>
> For the current tests to run successfully you must have a fresh  
> database with the OOTB demo data in place before testing each  
> component. That means something like "ant clean-all; ant run-
> install; java -jar ofbiz.jar tests component=XXX; ant clean-all; ant  
> run-install; java -jar ...; ...".
>
> There are quite a few things that have been discussed but either not  
> developed or just not committed related to testing, so this is  
> definitely an area that needs a lot of love! Since most things are  
> ultimately driven by end-users of OFBiz it would help a lot if end-
> users pushed for these, but that has only happened approximately 3  
> times EVER for OFBiz AFAIK, as opposed to many hundreds (perhaps  
> thousands) of times for other functionality and new features.
>
> -David
>
>
> On Feb 11, 2009, at 10:50 AM, Vikas Mayur wrote:
>
>> Well, All the tests supposed to be run successfully. If you see the  
>> logs, note that all the test are not failed.
>> The only reason that I can think of is that In some cases, the data  
>> for all test cases are not defined properly. I mean few of the test  
>> case does not get the desired data and which is when changed by any  
>> earlier test causes them to fail.
>>
>> Other reason could be recent changes in services that are being  
>> tested by test case.
>>
>> Also the whole automated test-suite is supposed to run with fresh  
>> data each time.
>>
>> Vikas
>>
>> On Feb 11, 2009, at 4:45 AM, Robert Lor wrote:
>>
>>> Hi,
>>>
>>> I tried running the automated tests to see if my setup is correct  
>>> and noticed a number of failures.  Here's what I did:
>>>
>>> 1) svn co http://svn.apache.org/repos/asf/ofbiz/trunk ofbiz
>>> 2) setup db and modify entityengine.xml appropriately
>>> 2) ant run-install
>>> 3) java -jar ofbiz.jar -test
>>>
>>> Are all the tests supposed to run successfully out of the box or  
>>> am I missing something in the setup?
>>>
>>> The errors are slightly different between Postgres & MySQL.  
>>> Please see the output in the attached file.
>>>
>>>
>>> Thanks,
>>> Robert
>>> <test_log.tar.gz>
>>
>

[Robert Lor]

Vikas Mayur wrote:

> Well, All the tests supposed to be run successfully. If you see the
> logs, note that all the test are not failed.
> The only reason that I can think of is that In some cases, the data
> for all test cases are not defined properly. I mean few of the test
> case does not get the desired data and which is when changed by any
> earlier test causes them to fail.
>
> Other reason could be recent changes in services that are being tested
> by test case.
>
> Also the whole automated test-suite is supposed to run with fresh data
> each time.

Running the tests with fresh data does reduce but not eliminate the
number of failures as shown below.

bash-3.2$ grep "Pass:" log.mysql.clean
2009-02-12 16:37:24,112 (main) [     JunitContainer.java:73 :INFO ]
[JUNIT] Pass: true | # Tests: 4 | # Failed: 0 # Errors: 0
2009-02-12 16:38:34,015 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: true | # Tests: 22 | # Failed: 0 # Errors: 0
2009-02-12 16:38:34,600 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: false | # Tests: 20 | # Failed: 1 # Errors: 0
2009-02-12 16:38:36,424 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: false | # Tests: 1 | # Failed: 0 # Errors: 1
2009-02-12 16:38:36,824 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: true | # Tests: 5 | # Failed: 0 # Errors: 0
2009-02-12 16:38:38,476 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: false | # Tests: 10 | # Failed: 2 # Errors: 0
2009-02-12 16:38:39,109 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: true | # Tests: 2 | # Failed: 0 # Errors: 0
2009-02-12 16:38:43,475 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: false | # Tests: 7 | # Failed: 1 # Errors: 0
2009-02-12 16:38:43,523 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: false | # Tests: 1 | # Failed: 1 # Errors: 0
2009-02-12 16:38:44,479 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: false | # Tests: 12 | # Failed: 13 # Errors: 0
2009-02-12 16:38:45,400 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: true | # Tests: 2 | # Failed: 0 # Errors: 0
2009-02-12 16:38:45,503 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: true | # Tests: 1 | # Failed: 0 # Errors: 0

bash-3.2$ grep "Pass:" log/log.mysql.dirty
2009-02-12 16:42:33,133 (main) [     JunitContainer.java:73 :INFO ]
[JUNIT] Pass: true | # Tests: 4 | # Failed: 0 # Errors: 0
2009-02-12 16:43:45,049 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: false | # Tests: 22 | # Failed: 1 # Errors: 0
2009-02-12 16:43:45,618 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: false | # Tests: 20 | # Failed: 2 # Errors: 0
2009-02-12 16:43:46,115 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: false | # Tests: 1 | # Failed: 0 # Errors: 1
2009-02-12 16:43:46,322 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: true | # Tests: 5 | # Failed: 0 # Errors: 0
2009-02-12 16:43:47,376 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: false | # Tests: 10 | # Failed: 6 # Errors: 0
2009-02-12 16:43:47,800 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: true | # Tests: 2 | # Failed: 0 # Errors: 0
2009-02-12 16:43:49,633 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: false | # Tests: 7 | # Failed: 2 # Errors: 0
2009-02-12 16:43:49,697 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: false | # Tests: 1 | # Failed: 1 # Errors: 0
2009-02-12 16:43:51,305 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: false | # Tests: 12 | # Failed: 13 # Errors: 0
2009-02-12 16:43:52,333 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: true | # Tests: 2 | # Failed: 0 # Errors: 0
2009-02-12 16:43:52,386 (main) [   TestRunContainer.java:135:INFO ]
[JUNIT] Pass: true | # Tests: 1 | # Failed: 0 # Errors: 0
bash-3.2$

[Robert Lor]

David E Jones wrote:
>
> For the current tests to run successfully you must have a fresh
> database with the OOTB demo data in place before testing each
> component. That means something like "ant clean-all; ant run-install;
> java -jar ofbiz.jar tests component=XXX; ant clean-all; ant
> run-install; java -jar ...; ...".
>
With MySQL, all the tables also need to be dropped before re-running the
tests; otherwise one test case would fail. It doesn't happen with Derby
though.

-Robert

[Robert Lor]

This may be a stupid question, but I'm curious if it make sense to use
Cayenne ORM in OFBiz. My understanding is that the Entity Engine is very
flexible now in terms of providing an abstraction layer for OFBiz to
link with different databases and other data sources, but I thought
Cayenne ORM may offer some extra benefits
http://cayenne.apache.org/why-cayenne.html. Just an idea!

-Robert