test case definitions

[Adam Heath-2]

A junit test case is supposed to be completely stand-alone.  It should
not depend on any other test case running previously.  However, the
service test suite(framework/service/testdef/servicetests.xml)
requires the data load test case to run, then each test case to run in
series.  This is wrong.

I'm going to modify the test system to support having multiple
children.  <test-suite> currently only has <test-case> as a child.
I'm going to add <test-group>, which can then have multiple children.

[Tim Ruppert]

I'd back this up 100% btw.  I think that Scott has been working on some stuff here that may be of interest to everyone - with some really nice data rollback capabilities.  These dependent tests are not a good idea IMHO and we should make each one standalone.  If you need to load similar (or even the same) data, no worries, that's why the loads should be in data files not in the tests themselves - so that they're reusable.

Cheers,
Tim
--
Tim Ruppert
HotWax Media
http://www.hotwaxmedia.com

o:801.649.6594
f:801.649.6595

----- "Adam Heath" <[hidden email]> wrote:

> A junit test case is supposed to be completely stand-alone.  It
> should
> not depend on any other test case running previously.  However, the
> service test suite(framework/service/testdef/servicetests.xml)
> requires the data load test case to run, then each test case to run
> in
> series.  This is wrong.
>
> I'm going to modify the test system to support having multiple
> children.  <test-suite> currently only has <test-case> as a child.
> I'm going to add <test-group>, which can then have multiple children.

[Adam Heath-2]

Tim Ruppert wrote:
> I'd back this up 100% btw.  I think that Scott has been working
> on some stuff here that may be of interest to everyone - with
> some really nice data rollback capabilities.  These dependent
> tests are not a good idea IMHO and we should make each one
> standalone.  If you need to load similar (or even the same) data,
> no worries, that's why the loads should be in data files not in
> the tests themselves - so that they're reusable.

data rollback?  that's the complex solution.

I wrote a small container, that iterates all the defined tests, and
writes their names to a file.

Then, with a shell script, I clean all the data, do run-install, make
a backup copy of runtime/data, then run each test in a loop, restoring
the backup before each run.

On my machine, it takes 34 minutes to run all the test cases.  And
there are definitate failures.  I've already fixed some of them.

The shell script does java -jar test -component=foo -case=bar, and
saves the output xml in runtime/logs/test-results into a per-instance
directory.  I've attached it so others can look at it.