Delegator rollback per test case?
Locked
Delegator rollback per test case?
 
|
We've had this discussion before, here is last one I could find: http://markmail.org/message/ftvs45vnzlobo7hb
The solution I would like to propose for the problem David described is for us to make more use of the test-group child element of test- suite and have tests within those groups rollback as a group, for all other test cases we'd roll them back as soon as they complete. Here's an example of what I'm talking about since that description probably did nothing for you: <test-suite> <test-case/> <!-- Rollback here --> <test-case/> <!-- Rollback here --> <test-case/> <!-- Rollback here --> <!-- No rollbacks inside the test group --> <test-group> <test-case/> <test-case/> </test-group> <!-- Rollback here --> </test-suite> A common problem I'm seeing is that we want to run multiple tests on the same piece of demo data but can't because the previous test has already altered it. An example might be where you want to try shipping a single order in various different splits but you can't do that at the moment without having a separate demo order for each test. Thoughts? Thanks Scott HotWax Media http://www.hotwaxmedia.com |
smime.p7s (4K)
Re: Delegator rollback per test case?
|
|
Re: Delegator rollback per test case?
|
|
My intention was generally to have one test suite per sub-package e.g.
one for org.ofbiz.accounting.finaccount, one for org.ofbiz.accounting.fixedasset, etc. Lets say I want to run 50 different order processing tests, cancellations, edits, authorizations, packing, etc. To do this I have to either create 50 different demo orders which is time consuming or I have to split them up into different test suites which will quickly leave us with a huge number of test suites all doing very similar things. Aside from the above there are a few other things I don't like about having only one rollback per test suite: - When adding a new test case to a suite you have no idea what state the database will be in unless you inspect all of the other test cases in the suite and understand what they are doing. - It makes test cases more difficult to maintain because you have no idea where the dependencies are within a suite. With the change I'm suggesting you would need to be explicit about dependancies by putting the cases in a test group. - If you make some application code or demo data changes and 10 test cases in a suite start failing you have no idea if it's because a single case is failing and the others depend on it or because your changes are causing wide spread problems. Knowing the dependancies helps the developer to understand how big the problem is before they go through and examine every failed case. Ultimately my only goal here is to make writing tests easier, I'm not approaching it with any idealistic "this is how tests should be conducted" mentality. The easier we can make it, the more likely it is that people will actually write them. > I'm unclear about something in your message though: are you saying > you are doing it below and it's not meeting your needs, or that you > would like to be able to do it as below? I'm not sure what you mean, what I'm suggesting below isn't currently possible. The test-group tag actually does nothing at the moment (other than add the child test cases to the suite, which is the same as not having it at all). Regards Scott On 13/12/2009, at 3:44 PM, David E Jones wrote: > > My preference would still be to have one roll-back per test-suite, > which basically means one transaction per XML file. If you want the > tests to use the same data then put them in the same file. If you > want them to use different data, then they go in their own files. > > I'm unclear about something in your message though: are you saying > you are doing it below and it's not meeting your needs, or that you > would like to be able to do it as below? > > -David > > > On Dec 11, 2009, at 3:54 AM, Scott Gray wrote: > >> We've had this discussion before, here is last one I could find: http://markmail.org/message/ftvs45vnzlobo7hb >> >> The solution I would like to propose for the problem David >> described is for us to make more use of the test-group child >> element of test-suite and have tests within those groups rollback >> as a group, for all other test cases we'd roll them back as soon as >> they complete. Here's an example of what I'm talking about since >> that description probably did nothing for you: >> >> <test-suite> >> >> <test-case/> >> <!-- Rollback here --> >> <test-case/> >> <!-- Rollback here --> >> <test-case/> >> <!-- Rollback here --> >> >> <!-- No rollbacks inside the test group --> >> <test-group> >> <test-case/> >> <test-case/> >> </test-group> >> <!-- Rollback here --> >> >> </test-suite> >> >> A common problem I'm seeing is that we want to run multiple tests >> on the same piece of demo data but can't because the previous test >> has already altered it. An example might be where you want to try >> shipping a single order in various different splits but you can't >> do that at the moment without having a separate demo order for each >> test. >> >> Thoughts? >> >> Thanks >> Scott >> >> HotWax Media >> http://www.hotwaxmedia.com >> >> > |
Re: Delegator rollback per test case?
|
|
I'd really rather not have this idea die with a single response, I've
done my best to describe the perceived benefits and it would be good to get some additional feedback from anyone who is interested. Thanks Scott On 13/12/2009, at 8:20 PM, Scott Gray wrote: > My intention was generally to have one test suite per sub-package > e.g. one for org.ofbiz.accounting.finaccount, one for > org.ofbiz.accounting.fixedasset, etc. > > Lets say I want to run 50 different order processing tests, > cancellations, edits, authorizations, packing, etc. To do this I > have to either create 50 different demo orders which is time > consuming or I have to split them up into different test suites > which will quickly leave us with a huge number of test suites all > doing very similar things. > > Aside from the above there are a few other things I don't like about > having only one rollback per test suite: > - When adding a new test case to a suite you have no idea what state > the database will be in unless you inspect all of the other test > cases in the suite and understand what they are doing. > - It makes test cases more difficult to maintain because you have no > idea where the dependencies are within a suite. With the change I'm > suggesting you would need to be explicit about dependancies by > putting the cases in a test group. > - If you make some application code or demo data changes and 10 test > cases in a suite start failing you have no idea if it's because a > single case is failing and the others depend on it or because your > changes are causing wide spread problems. Knowing the dependancies > helps the developer to understand how big the problem is before they > go through and examine every failed case. > > Ultimately my only goal here is to make writing tests easier, I'm > not approaching it with any idealistic "this is how tests should be > conducted" mentality. The easier we can make it, the more likely it > is that people will actually write them. > >> I'm unclear about something in your message though: are you saying >> you are doing it below and it's not meeting your needs, or that you >> would like to be able to do it as below? > > I'm not sure what you mean, what I'm suggesting below isn't > currently possible. The test-group tag actually does nothing at the > moment (other than add the child test cases to the suite, which is > the same as not having it at all). > > Regards > Scott > > On 13/12/2009, at 3:44 PM, David E Jones wrote: > >> >> My preference would still be to have one roll-back per test-suite, >> which basically means one transaction per XML file. If you want the >> tests to use the same data then put them in the same file. If you >> want them to use different data, then they go in their own files. >> >> I'm unclear about something in your message though: are you saying >> you are doing it below and it's not meeting your needs, or that you >> would like to be able to do it as below? >> >> -David >> >> >> On Dec 11, 2009, at 3:54 AM, Scott Gray wrote: >> >>> We've had this discussion before, here is last one I could find: http://markmail.org/message/ftvs45vnzlobo7hb >>> >>> The solution I would like to propose for the problem David >>> described is for us to make more use of the test-group child >>> element of test-suite and have tests within those groups rollback >>> as a group, for all other test cases we'd roll them back as soon >>> as they complete. Here's an example of what I'm talking about >>> since that description probably did nothing for you: >>> >>> <test-suite> >>> >>> <test-case/> >>> <!-- Rollback here --> >>> <test-case/> >>> <!-- Rollback here --> >>> <test-case/> >>> <!-- Rollback here --> >>> >>> <!-- No rollbacks inside the test group --> >>> <test-group> >>> <test-case/> >>> <test-case/> >>> </test-group> >>> <!-- Rollback here --> >>> >>> </test-suite> >>> >>> A common problem I'm seeing is that we want to run multiple tests >>> on the same piece of demo data but can't because the previous test >>> has already altered it. An example might be where you want to try >>> shipping a single order in various different splits but you can't >>> do that at the moment without having a separate demo order for >>> each test. >>> >>> Thoughts? >>> >>> Thanks >>> Scott >>> >>> HotWax Media >>> http://www.hotwaxmedia.com >>> >>> >> > |
Re: Delegator rollback per test case?
|
|
Re: Delegator rollback per test case?
|
|
On 24/12/2009, at 9:50 PM, Adam Heath wrote:
> Scott Gray wrote: >> I'd really rather not have this idea die with a single response, I've >> done my best to describe the perceived benefits and it would be >> good to >> get some additional feedback from anyone who is interested. > > I don't care either way, however, I do have some suggestions on > implementation. > > Create an ofbiz test runner, if one doesn't exist. as I can see it's the only way to interact directly with the execution of a test suite (although it's been months since I looked at this stuff). > Then, decorate the > test suite with an annotation, that says that each test should be > rolled back. junit 4 may even have some support for doing this > automatically. Remember we're dealing with OFBiz tests here and the test suites are defined using XML. It was my intention to roll back each test case by default unless you explicitly wrap the test cases in a test-group element, as per my original email: >> Scott Gray wrote: >>> <test-suite> >>> >>> <test-case/> >>> <!-- Rollback here --> >>> <test-case/> >>> <!-- Rollback here --> >>> <test-case/> >>> <!-- Rollback here --> >>> >>> <!-- No rollbacks inside the test group --> >>> <test-group> >>> <test-case/> >>> <test-case/> >>> </test-group> >>> <!-- Rollback here --> >>> >>> </test-suite> prefer it we kept the test-suite documents as the single point of configuration. There's also the possibility of nesting the rollbacks if things need to get a bit more sophisticated: <test-group> <test-case name="test-1"/> <test-group> <test-case name="test-1a"/> <test-case name="test-1b"/> <test-case name="test-1c"/> </test-group> <!-- Rollback here to the end of test-1--> <test-case name="test-2"/> <test-case name="test-3"/> </test-group> <!-- Rollback test-1, test-2 and test-3 --> The main goal with all of this is to allow developers to spend more time writing tests and less time writing redundant demo data and/or test suites. Thanks Scott |
Re: Delegator rollback per test case?
|
|
Re: Delegator rollback per test case?
|
|
+1 - this needs to happen - the rollback is invaluable as Marc said.
Cheers, Ruppert -- Tim Ruppert HotWax Media http://www.hotwaxmedia.com o:801.649.6594 f:801.649.6595 On Dec 24, 2009, at 5:26 AM, Marc Morin wrote: > W make extensive use of the rollback pre test case paradigm for our proprietary application. It is invaluable. The test cases may run a little slower than if the setup/rollback was performed per suite, but the inter-testcase dependancies would be a nightmare for us to manage. > > > ----- Original Message ----- > From: "Scott Gray" <[hidden email]> > To: [hidden email] > Sent: Thursday, December 24, 2009 3:14:26 AM GMT -05:00 US/Canada Eastern > Subject: Re: Delegator rollback per test case? > > I'd really rather not have this idea die with a single response, I've > done my best to describe the perceived benefits and it would be good > to get some additional feedback from anyone who is interested. > > Thanks > Scott > > On 13/12/2009, at 8:20 PM, Scott Gray wrote: > >> My intention was generally to have one test suite per sub-package >> e.g. one for org.ofbiz.accounting.finaccount, one for >> org.ofbiz.accounting.fixedasset, etc. >> >> Lets say I want to run 50 different order processing tests, >> cancellations, edits, authorizations, packing, etc. To do this I >> have to either create 50 different demo orders which is time >> consuming or I have to split them up into different test suites >> which will quickly leave us with a huge number of test suites all >> doing very similar things. >> >> Aside from the above there are a few other things I don't like about >> having only one rollback per test suite: >> - When adding a new test case to a suite you have no idea what state >> the database will be in unless you inspect all of the other test >> cases in the suite and understand what they are doing. >> - It makes test cases more difficult to maintain because you have no >> idea where the dependencies are within a suite. With the change I'm >> suggesting you would need to be explicit about dependancies by >> putting the cases in a test group. >> - If you make some application code or demo data changes and 10 test >> cases in a suite start failing you have no idea if it's because a >> single case is failing and the others depend on it or because your >> changes are causing wide spread problems. Knowing the dependancies >> helps the developer to understand how big the problem is before they >> go through and examine every failed case. >> >> Ultimately my only goal here is to make writing tests easier, I'm >> not approaching it with any idealistic "this is how tests should be >> conducted" mentality. The easier we can make it, the more likely it >> is that people will actually write them. >> >>> I'm unclear about something in your message though: are you saying >>> you are doing it below and it's not meeting your needs, or that you >>> would like to be able to do it as below? >> >> I'm not sure what you mean, what I'm suggesting below isn't >> currently possible. The test-group tag actually does nothing at the >> moment (other than add the child test cases to the suite, which is >> the same as not having it at all). >> >> Regards >> Scott >> >> On 13/12/2009, at 3:44 PM, David E Jones wrote: >> >>> >>> My preference would still be to have one roll-back per test-suite, >>> which basically means one transaction per XML file. If you want the >>> tests to use the same data then put them in the same file. If you >>> want them to use different data, then they go in their own files. >>> >>> I'm unclear about something in your message though: are you saying >>> you are doing it below and it's not meeting your needs, or that you >>> would like to be able to do it as below? >>> >>> -David >>> >>> >>> On Dec 11, 2009, at 3:54 AM, Scott Gray wrote: >>> >>>> We've had this discussion before, here is last one I could find: http://markmail.org/message/ftvs45vnzlobo7hb >>>> >>>> The solution I would like to propose for the problem David >>>> described is for us to make more use of the test-group child >>>> element of test-suite and have tests within those groups rollback >>>> as a group, for all other test cases we'd roll them back as soon >>>> as they complete. Here's an example of what I'm talking about >>>> since that description probably did nothing for you: >>>> >>>> <test-suite> >>>> >>>> <test-case/> >>>> <!-- Rollback here --> >>>> <test-case/> >>>> <!-- Rollback here --> >>>> <test-case/> >>>> <!-- Rollback here --> >>>> >>>> <!-- No rollbacks inside the test group --> >>>> <test-group> >>>> <test-case/> >>>> <test-case/> >>>> </test-group> >>>> <!-- Rollback here --> >>>> >>>> </test-suite> >>>> >>>> A common problem I'm seeing is that we want to run multiple tests >>>> on the same piece of demo data but can't because the previous test >>>> has already altered it. An example might be where you want to try >>>> shipping a single order in various different splits but you can't >>>> do that at the moment without having a separate demo order for >>>> each test. >>>> >>>> Thoughts? >>>> >>>> Thanks >>>> Scott >>>> >>>> HotWax Media >>>> http://www.hotwaxmedia.com >>>> >>>> >>> >> > |
Re: Delegator rollback per test case?
|
|
Re: Delegator rollback per test case?
Administrator
|
Re: Delegator rollback per test case?
|
|
| Free forum by Nabble | Edit this page |