Database Test Suite

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Database Test Suite

John Martin
I'm looking for a good way of testing service methods that do CRUD on
various tables in the database.

I reviewed the framework/testtools and it didn't seem that there was
much if anything going on there nor did it appear to address
manipulation of test database data.

While poking around I ran into ProposedOFBizTestFramework document by
Brett Palmer that uses JXUnit
(http://ofbizwiki.go-integral.com/Wiki.jsp?page=ProposedOFBizTestFramework
).

This document looks quite promising and seems like it is exactly what
I am looking for but the ofbiz-test that it referenced no longer
exists at SourceForge.  Is this a dead project?  If not, where is it
now located?

What is the best practices for writing a test suite for new services
that are manipulating data?

Thanks!

John
Reply | Threaded
Open this post in threaded view
|

Re: Database Test Suite

Tim Ruppert
I was a committer for DbUnit back in the day, which does stage
information nicely into the database for running tests.  We used it in
conjunction with WebTest to drive all testing from ant.  It has worked
well in the past, but might not be the most up to date solution.

The only concern that I would have on this front is that there may be
too much data to stage, With large amounts of data, the DbUnit approach
could run into some performance issues with certain data sets.  It might
still be a viable approach for you John depending on your situation.

I'd still love to hear more about Brett's JXUnit experience whenever he
gets a chance.

Cheers,
Tim

--
Tim Ruppert
HotWax Media
http://www.hotwaxmedia.com

o:801.649.6594
f:801.649.6595

John Martin wrote:

> I'm looking for a good way of testing service methods that do CRUD on
> various tables in the database.
>
> I reviewed the framework/testtools and it didn't seem that there was
> much if anything going on there nor did it appear to address
> manipulation of test database data.
>
> While poking around I ran into ProposedOFBizTestFramework document by
> Brett Palmer that uses JXUnit
> (http://ofbizwiki.go-integral.com/Wiki.jsp?page=ProposedOFBizTestFramework 
>
> ).
>
> This document looks quite promising and seems like it is exactly what
> I am looking for but the ofbiz-test that it referenced no longer
> exists at SourceForge.  Is this a dead project?  If not, where is it
> now located?
>
> What is the best practices for writing a test suite for new services
> that are manipulating data?
>
> Thanks!
>
> John
Reply | Threaded
Open this post in threaded view
|

Re: Database Test Suite

Brett
Sorry,  there has not been a lot of work done on the JXUnit stuff
since its original creation.  The JXUnit project did not get a lot of
support on sourceforge and the original contributors moved on to other
projects.

I continued to use the ofbiz jxunit code for some of my own projects
but creating data driven tests were still difficult to create and
maintain.

Over the past year, I have been working with a different framework
called grinder (http://grinder.sourceforge.net/).  There have been a
few posts on the subject from David Jones and Al Byers.  Al has
recently been working on it and we have shared some ideas on how to
make it effective for ofbiz.

Here are some of the features that I like about grinder:

- Uses Jython as a the scripting language to implement tests.  This
makes it easier to create new tests without having to compile and
deploy each time.
- TCP Proxy listener to capture web requests and generates a test
script.  This script can then be modified to be more dynamic
- Ability to simulate multiple clients with various load and frequency options


Here are some things that are missing and what Al and I have been discussing:

- Need to be able to separate data from the test
- Need to develop some jython utilities that can read in a common XML
(or something else) format for a test and pass the data to the jython
test script


Al and David can probably share some of their ideas on testing.  We
need to have more test tools but it is always a matter of priorities
and everyone is always busy with other tasks.

Thoughts?


Brett

On 10/23/06, Tim Ruppert <[hidden email]> wrote:

> I was a committer for DbUnit back in the day, which does stage
> information nicely into the database for running tests.  We used it in
> conjunction with WebTest to drive all testing from ant.  It has worked
> well in the past, but might not be the most up to date solution.
>
> The only concern that I would have on this front is that there may be
> too much data to stage, With large amounts of data, the DbUnit approach
> could run into some performance issues with certain data sets.  It might
> still be a viable approach for you John depending on your situation.
>
> I'd still love to hear more about Brett's JXUnit experience whenever he
> gets a chance.
>
> Cheers,
> Tim
>
> --
> Tim Ruppert
> HotWax Media
> http://www.hotwaxmedia.com
>
> o:801.649.6594
> f:801.649.6595
>
> John Martin wrote:
> > I'm looking for a good way of testing service methods that do CRUD on
> > various tables in the database.
> >
> > I reviewed the framework/testtools and it didn't seem that there was
> > much if anything going on there nor did it appear to address
> > manipulation of test database data.
> >
> > While poking around I ran into ProposedOFBizTestFramework document by
> > Brett Palmer that uses JXUnit
> > (http://ofbizwiki.go-integral.com/Wiki.jsp?page=ProposedOFBizTestFramework
> >
> > ).
> >
> > This document looks quite promising and seems like it is exactly what
> > I am looking for but the ofbiz-test that it referenced no longer
> > exists at SourceForge.  Is this a dead project?  If not, where is it
> > now located?
> >
> > What is the best practices for writing a test suite for new services
> > that are manipulating data?
> >
> > Thanks!
> >
> > John
>
Reply | Threaded
Open this post in threaded view
|

Re: Database Test Suite

Tim Ruppert
Brett, I've looked into Grinder earlier this year (albeit only for an
hour or so) and was pleasantly surprised by it.  We used something
called qftestJUI (http://www.qfs.de/en/qftestJUI/index.html) while
developing a swing front end to a J2EE app - which had a lot of the
record and playback stuff that we were looking for.

Grinder seems to give you a lot of that, but some additional load
testing stuff as well. Is that about right?  Does anyone know how
Grinder compares to Selinium and it's new Firefox plugin?

Cheers,
Tim

--
Tim Ruppert
HotWax Media
http://www.hotwaxmedia.com

o:801.649.6594
f:801.649.6595

Brett Palmer wrote:

> Sorry,  there has not been a lot of work done on the JXUnit stuff
> since its original creation.  The JXUnit project did not get a lot of
> support on sourceforge and the original contributors moved on to other
> projects.
>
> I continued to use the ofbiz jxunit code for some of my own projects
> but creating data driven tests were still difficult to create and
> maintain.
>
> Over the past year, I have been working with a different framework
> called grinder (http://grinder.sourceforge.net/).  There have been a
> few posts on the subject from David Jones and Al Byers.  Al has
> recently been working on it and we have shared some ideas on how to
> make it effective for ofbiz.
>
> Here are some of the features that I like about grinder:
>
> - Uses Jython as a the scripting language to implement tests.  This
> makes it easier to create new tests without having to compile and
> deploy each time.
> - TCP Proxy listener to capture web requests and generates a test
> script.  This script can then be modified to be more dynamic
> - Ability to simulate multiple clients with various load and frequency
> options
>
>
> Here are some things that are missing and what Al and I have been
> discussing:
>
> - Need to be able to separate data from the test
> - Need to develop some jython utilities that can read in a common XML
> (or something else) format for a test and pass the data to the jython
> test script
>
>
> Al and David can probably share some of their ideas on testing.  We
> need to have more test tools but it is always a matter of priorities
> and everyone is always busy with other tasks.
>
> Thoughts?
>
>
> Brett
>
> On 10/23/06, Tim Ruppert <[hidden email]> wrote:
>> I was a committer for DbUnit back in the day, which does stage
>> information nicely into the database for running tests.  We used it in
>> conjunction with WebTest to drive all testing from ant.  It has worked
>> well in the past, but might not be the most up to date solution.
>>
>> The only concern that I would have on this front is that there may be
>> too much data to stage, With large amounts of data, the DbUnit approach
>> could run into some performance issues with certain data sets.  It might
>> still be a viable approach for you John depending on your situation.
>>
>> I'd still love to hear more about Brett's JXUnit experience whenever he
>> gets a chance.
>>
>> Cheers,
>> Tim
>>
>> --
>> Tim Ruppert
>> HotWax Media
>> http://www.hotwaxmedia.com
>>
>> o:801.649.6594
>> f:801.649.6595
>>
>> John Martin wrote:
>> > I'm looking for a good way of testing service methods that do CRUD on
>> > various tables in the database.
>> >
>> > I reviewed the framework/testtools and it didn't seem that there was
>> > much if anything going on there nor did it appear to address
>> > manipulation of test database data.
>> >
>> > While poking around I ran into ProposedOFBizTestFramework document by
>> > Brett Palmer that uses JXUnit
>> >
>> (http://ofbizwiki.go-integral.com/Wiki.jsp?page=ProposedOFBizTestFramework 
>>
>> >
>> > ).
>> >
>> > This document looks quite promising and seems like it is exactly what
>> > I am looking for but the ofbiz-test that it referenced no longer
>> > exists at SourceForge.  Is this a dead project?  If not, where is it
>> > now located?
>> >
>> > What is the best practices for writing a test suite for new services
>> > that are manipulating data?
>> >
>> > Thanks!
>> >
>> > John
>>
Reply | Threaded
Open this post in threaded view
|

Re: Database Test Suite

John Martin
Grinder does look really interesting and appears to me to be geared
more for testing the application for a UI and performance perspective.

What I am really after is something that is more database oriented to
allow a test suite to initialize data appropriately, run the test, and
then clean out the test data.  I'm writing new services to
insert/update data and want to have a simple way of building tests to
make sure that my code is working correctly.

Currently, I write a test script that invokes the service and I
inspect the results and walk through the code in the debugger.  I
agree with Brett that "creating data driven tests were still difficult
to create and maintain."  There's got to be an easier way...

Brett - I would be interested in the JXUnit integration with ofBiz
since that appears to be inline with what I'm trying to do.  Can point
me to where I can get what I would need to use it?

Thanks,

John
Reply | Threaded
Open this post in threaded view
|

Re: Database Test Suite

Tim Ruppert
John, this is exactly what I used DbUnit (http://www.dbunit.org/) for in
the past.  You can drive this from Ant or thru code, so it's as flexible
as you need it to be.

Check out some of the information up on their site - as well as this
(slightly dated) article written about them -
http://www.onjava.com/pub/a/onjava/2004/01/21/dbunit.html .  What we did
was extend their DatabaseTestCase and load and clear data based upon the
test case or test suite that we were running - using dynamically
generated resource paths based upon the package and class names of the
files.

If you're interested and I can be of help - I'm here to answer more Q's

Cheers,
Tim

--
Tim Ruppert
HotWax Media
http://www.hotwaxmedia.com

o:801.649.6594
f:801.649.6595

John Martin wrote:

> Grinder does look really interesting and appears to me to be geared
> more for testing the application for a UI and performance perspective.
>
> What I am really after is something that is more database oriented to
> allow a test suite to initialize data appropriately, run the test, and
> then clean out the test data.  I'm writing new services to
> insert/update data and want to have a simple way of building tests to
> make sure that my code is working correctly.
>
> Currently, I write a test script that invokes the service and I
> inspect the results and walk through the code in the debugger.  I
> agree with Brett that "creating data driven tests were still difficult
> to create and maintain."  There's got to be an easier way...
>
> Brett - I would be interested in the JXUnit integration with ofBiz
> since that appears to be inline with what I'm trying to do.  Can point
> me to where I can get what I would need to use it?
>
> Thanks,
>
> John
Reply | Threaded
Open this post in threaded view
|

Re: Database Test Suite

David E Jones-2
In reply to this post by John Martin

John,

Part of the test plan for OFBiz (that has been around for well over a  
year has not seen much attention) is to be able to easily to data  
level testing. We plan to use the OOTB demo and seed data as the  
initial database state, and then after tests are done the test  
database would simply be thrown away (avoiding the need to "undo" all  
test changes).

To validate results on the data level there is an existing feature in  
the simple-method that we plan to make easier to use in the test  
config XML files. What it does is read an entity engine XML file  
(which can exported from WebTools, or created manually) and validate  
it against the database by asserting that each element in the XML  
file matches a database record. It reports records that are missing  
and all fields that are different for existing records.

Is that more or less what you are looking for? The intent with these  
is to be intermixed with UI level tests and test services and such to  
provide testing facilities for the various architectural layers in  
OFBiz.

-David


On Oct 24, 2006, at 6:54 AM, John Martin wrote:

> Grinder does look really interesting and appears to me to be geared
> more for testing the application for a UI and performance perspective.
>
> What I am really after is something that is more database oriented to
> allow a test suite to initialize data appropriately, run the test, and
> then clean out the test data.  I'm writing new services to
> insert/update data and want to have a simple way of building tests to
> make sure that my code is working correctly.
>
> Currently, I write a test script that invokes the service and I
> inspect the results and walk through the code in the debugger.  I
> agree with Brett that "creating data driven tests were still difficult
> to create and maintain."  There's got to be an easier way...
>
> Brett - I would be interested in the JXUnit integration with ofBiz
> since that appears to be inline with what I'm trying to do.  Can point
> me to where I can get what I would need to use it?
>
> Thanks,
>
> John

Reply | Threaded
Open this post in threaded view
|

Re: Database Test Suite

John Martin
Hi David,

I think that the "throw away" database idea is probably out of the
question for my needs.  I code in an iterative process and need to
repeat the tests through the coding/debugging cycle.  When the code is
complete, so is the test suite for that service or set of services
which is then available for regression testing in the system.

During the development cycle I could run the tests 10-30 or more times
depending on the complexity.  If I understand your solution, one would
need to repopulate the database after each iteration which would be
very time consuming.  In addition, I could see issues with the XML
entity comparision with createXXX and lastUpdatedXXX datetime columns
as well as the autoincrement id columns.

What I am really looking for is the junit functionality where the test
suite has a build up and tear down section that is called before/after
the tests.  If this build up / tear down could be driven with XML,
that would be best.

It seems that the dbUnit project is more in line with what I am
looking for.  I'm going to explore this option and will colaborate
with Tim on this.

Thanks,

John

On 10/24/06, David E Jones <[hidden email]> wrote:

>
> John,
>
> Part of the test plan for OFBiz (that has been around for well over a
> year has not seen much attention) is to be able to easily to data
> level testing. We plan to use the OOTB demo and seed data as the
> initial database state, and then after tests are done the test
> database would simply be thrown away (avoiding the need to "undo" all
> test changes).
>
> To validate results on the data level there is an existing feature in
> the simple-method that we plan to make easier to use in the test
> config XML files. What it does is read an entity engine XML file
> (which can exported from WebTools, or created manually) and validate
> it against the database by asserting that each element in the XML
> file matches a database record. It reports records that are missing
> and all fields that are different for existing records.
>
> Is that more or less what you are looking for? The intent with these
> is to be intermixed with UI level tests and test services and such to
> provide testing facilities for the various architectural layers in
> OFBiz.
>
> -David
>
>
> On Oct 24, 2006, at 6:54 AM, John Martin wrote:
>
> > Grinder does look really interesting and appears to me to be geared
> > more for testing the application for a UI and performance perspective.
> >
> > What I am really after is something that is more database oriented to
> > allow a test suite to initialize data appropriately, run the test, and
> > then clean out the test data.  I'm writing new services to
> > insert/update data and want to have a simple way of building tests to
> > make sure that my code is working correctly.
> >
> > Currently, I write a test script that invokes the service and I
> > inspect the results and walk through the code in the debugger.  I
> > agree with Brett that "creating data driven tests were still difficult
> > to create and maintain."  There's got to be an easier way...
> >
> > Brett - I would be interested in the JXUnit integration with ofBiz
> > since that appears to be inline with what I'm trying to do.  Can point
> > me to where I can get what I would need to use it?
> >
> > Thanks,
> >
> > John
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Database Test Suite

David E Jones-2

John,

Okay, that's a good distinction. You're not looking for regression or  
system wide tests, which generally take enough time to run that  
building a test database (about 3 minutes on Derby on my laptop) is  
not a significant amount of time.

For quick tests like this I'd recommend creating test services that  
do the build up, code run, and results test all at once and then just  
roll back the transaction to do the cleanup. This tends to work  
fairly well and leverages existing transaction functionality that is  
pretty solid and does the "undo" easily. This is a lot easier than  
trying to save the state of the db and restoring it in the tear down.  
Of course, if you're not worried about developer efficiency, then  
code away and have a good time... ;)

-David


On Oct 24, 2006, at 6:07 PM, John Martin wrote:

> Hi David,
>
> I think that the "throw away" database idea is probably out of the
> question for my needs.  I code in an iterative process and need to
> repeat the tests through the coding/debugging cycle.  When the code is
> complete, so is the test suite for that service or set of services
> which is then available for regression testing in the system.
>
> During the development cycle I could run the tests 10-30 or more times
> depending on the complexity.  If I understand your solution, one would
> need to repopulate the database after each iteration which would be
> very time consuming.  In addition, I could see issues with the XML
> entity comparision with createXXX and lastUpdatedXXX datetime columns
> as well as the autoincrement id columns.
>
> What I am really looking for is the junit functionality where the test
> suite has a build up and tear down section that is called before/after
> the tests.  If this build up / tear down could be driven with XML,
> that would be best.
>
> It seems that the dbUnit project is more in line with what I am
> looking for.  I'm going to explore this option and will colaborate
> with Tim on this.
>
> Thanks,
>
> John
>
> On 10/24/06, David E Jones <[hidden email]> wrote:
>>
>> John,
>>
>> Part of the test plan for OFBiz (that has been around for well over a
>> year has not seen much attention) is to be able to easily to data
>> level testing. We plan to use the OOTB demo and seed data as the
>> initial database state, and then after tests are done the test
>> database would simply be thrown away (avoiding the need to "undo" all
>> test changes).
>>
>> To validate results on the data level there is an existing feature in
>> the simple-method that we plan to make easier to use in the test
>> config XML files. What it does is read an entity engine XML file
>> (which can exported from WebTools, or created manually) and validate
>> it against the database by asserting that each element in the XML
>> file matches a database record. It reports records that are missing
>> and all fields that are different for existing records.
>>
>> Is that more or less what you are looking for? The intent with these
>> is to be intermixed with UI level tests and test services and such to
>> provide testing facilities for the various architectural layers in
>> OFBiz.
>>
>> -David
>>
>>
>> On Oct 24, 2006, at 6:54 AM, John Martin wrote:
>>
>> > Grinder does look really interesting and appears to me to be geared
>> > more for testing the application for a UI and performance  
>> perspective.
>> >
>> > What I am really after is something that is more database  
>> oriented to
>> > allow a test suite to initialize data appropriately, run the  
>> test, and
>> > then clean out the test data.  I'm writing new services to
>> > insert/update data and want to have a simple way of building  
>> tests to
>> > make sure that my code is working correctly.
>> >
>> > Currently, I write a test script that invokes the service and I
>> > inspect the results and walk through the code in the debugger.  I
>> > agree with Brett that "creating data driven tests were still  
>> difficult
>> > to create and maintain."  There's got to be an easier way...
>> >
>> > Brett - I would be interested in the JXUnit integration with ofBiz
>> > since that appears to be inline with what I'm trying to do.  Can  
>> point
>> > me to where I can get what I would need to use it?
>> >
>> > Thanks,
>> >
>> > John
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Database Test Suite

John Martin
David,

Setting up the service and using transactions sound like a good idea.
The only negative that I see initially is that some times inspecting
the data visually is quite helpful to see what is going on.  In that
case, you have to let the transaction commit and then manually clean
up the mess.

Perhaps we could use the XML data import from with the build up method
and the tear-down could intellegently remove data from the XML.  Any
thoughts on that?

John
Reply | Threaded
Open this post in threaded view
|

Re: Database Test Suite

David E Jones-2

On Oct 24, 2006, at 6:42 PM, John Martin wrote:

> Setting up the service and using transactions sound like a good idea.
> The only negative that I see initially is that some times inspecting
> the data visually is quite helpful to see what is going on.  In that
> case, you have to let the transaction commit and then manually clean
> up the mess.
>
> Perhaps we could use the XML data import from with the build up method
> and the tear-down could intellegently remove data from the XML.  Any
> thoughts on that?

Just one: depending on the approach you take it's either really hard  
to do, or requires a very large overhead and only somewhat hard to do  
(which might be okay for test environments). Either way, it requires  
defining exactly what you want, including the granularity of changes  
you want to deal with. The tough part is the "intelligent" part...  
what would that mean?

-David

Reply | Threaded
Open this post in threaded view
|

Re: Database Test Suite

John Martin
Intelligent meaning that the teardown logic could refer to the test
data XML and perform the cleanup.  The XML probably would need some
syntax for the clean up.  It would be a nice feature if you could
disable the teardown so that you could visually inspect the data and
then invoke the teardown on demand to do the cleanup.