Re: svn commit: r750296 - in /ofbiz/trunk/framework/testtoo
Locked
 
|
Why not just use different test-suite elements for this? Why have an hierarchy in this? Actually... I'm not even sure what "this" is, ie what is the point of a group? Maybe we should discuss that first... -David On Mar 4, 2009, at 7:40 PM, [hidden email] wrote: > Author: doogie > Date: Thu Mar 5 02:40:06 2009 > New Revision: 750296 > > URL: http://svn.apache.org/viewvc?rev=750296&view=rev > Log: > Add a <test-group> element, that is a sibling of <test-case>; this > element can take a list of child test definitions. > > Modified: > ofbiz/trunk/framework/testtools/dtd/test-suite.xsd > ofbiz/trunk/framework/testtools/src/org/ofbiz/testtools/ > ModelTestSuite.java > ofbiz/trunk/framework/testtools/src/org/ofbiz/testtools/ > TestListContainer.java > > Modified: ofbiz/trunk/framework/testtools/dtd/test-suite.xsd > URL: http://svn.apache.org/viewvc/ofbiz/trunk/framework/testtools/dtd/test-suite.xsd?rev=750296&r1=750295&r2=750296&view=diff > = > = > = > = > = > = > = > = > ====================================================================== > --- ofbiz/trunk/framework/testtools/dtd/test-suite.xsd (original) > +++ ofbiz/trunk/framework/testtools/dtd/test-suite.xsd Thu Mar 5 > 02:40:06 2009 > @@ -36,9 +36,10 @@ > <!-- ELEMENTS start here --> > <xs:element name="test-suite"> > <xs:complexType> > - <xs:sequence> > - <xs:element minOccurs="1" maxOccurs="unbounded" > ref="test-case"/> > - </xs:sequence> > + <xs:choice minOccurs="1" maxOccurs="unbounded"> > + <xs:element ref="test-case"/> > + <xs:element ref="test-group"/> > + </xs:choice> > <xs:attributeGroup ref="attlist.test-suite"/> > </xs:complexType> > </xs:element> > @@ -57,6 +58,16 @@ > <xs:attributeGroup name="attlist.test-case"> > <xs:attribute type="xs:string" name="case-name" > use="required"/> > </xs:attributeGroup> > + <xs:element name="test-group"> > + <xs:annotation><xs:documentation></xs:documentation></ > xs:annotation> > + <xs:complexType> > + <xs:group minOccurs="1" maxOccurs="unbounded" > ref="AllTestCaseTypes"/> > + <xs:attributeGroup ref="attlist.test-group"/> > + </xs:complexType> > + </xs:element> > + <xs:attributeGroup name="attlist.test-group"> > + <xs:attribute type="xs:string" name="case-name" > use="required"/> > + </xs:attributeGroup> > > <xs:element name="junit-test-suite" > substitutionGroup="TestCaseTypes"> > <xs:annotation><xs:documentation>Used for JUnit test suites > written as a Java class.</xs:documentation></xs:annotation> > > Modified: ofbiz/trunk/framework/testtools/src/org/ofbiz/testtools/ > ModelTestSuite.java > URL: http://svn.apache.org/viewvc/ofbiz/trunk/framework/testtools/src/org/ofbiz/testtools/ModelTestSuite.java?rev=750296&r1=750295&r2=750296&view=diff > = > = > = > = > = > = > = > = > ====================================================================== > --- ofbiz/trunk/framework/testtools/src/org/ofbiz/testtools/ > ModelTestSuite.java (original) > +++ ofbiz/trunk/framework/testtools/src/org/ofbiz/testtools/ > ModelTestSuite.java Thu Mar 5 02:40:06 2009 > @@ -28,6 +28,7 @@ > > import org.ofbiz.base.util.Debug; > import org.ofbiz.base.util.ObjectType; > +import org.ofbiz.base.util.UtilMisc; > import org.ofbiz.base.util.UtilValidate; > import org.ofbiz.base.util.UtilXml; > import org.ofbiz.entity.GenericDelegator; > @@ -63,47 +64,60 @@ > this.delegator = > GenericDelegator.getGenericDelegator(this.delegatorName); > this.dispatcher = > GenericDispatcher.getLocalDispatcher(this.dispatcherName, delegator); > > - for (Element testCaseElement : > UtilXml.childElementList(mainElement, "test-case")) { > + for (Element testCaseElement : > UtilXml.childElementList(mainElement, UtilMisc.toSet("test-case", > "test-group"))) { > String caseName = testCaseElement.getAttribute("case- > name"); > + String nodeName = testCaseElement.getNodeName(); > if (testCase == null || caseName.equals(testCase)) { > - Element childElement = > UtilXml.firstChildElement(testCaseElement); > - String nodeName = childElement.getNodeName(); > - if ("junit-test-suite".equals(nodeName)) { > - String className = > childElement.getAttribute("class-name"); > - > - try { > - Class clz = ObjectType.loadClass(className); > - TestSuite suite = new TestSuite(); > - suite.addTestSuite(clz); > - Enumeration testEnum = suite.tests(); > - int testsAdded = 0; > - int casesAdded = 0; > - while (testEnum.hasMoreElements()) { > - Test tst = (Test) testEnum.nextElement(); > - this.testList.add(tst); > - casesAdded += tst.countTestCases(); > - testsAdded++; > - } > - Debug.logInfo("Added " + testsAdded + " > tests [" + casesAdded + " cases] from the class: " + className, > module); > - } catch (Exception e) { > - String errMsg = "Unable to load test suite > class : " + className; > - Debug.logError(e, errMsg, module); > + if (nodeName.equals("test-case")) { > + parseTestElement(caseName, > UtilXml.firstChildElement(testCaseElement)); > + } else if (nodeName.equals("test-group")) { > + int i = 0; > + for (Element childElement: > UtilXml.childElementList(testCaseElement)) { > + parseTestElement(caseName + '-' + i, > childElement); > + i++; > } > - } else if ("service-test".equals(nodeName)) { > - this.testList.add(new ServiceTest(caseName, > this, childElement)); > - } else if ("simple-method-test".equals(nodeName)) { > - this.testList.add(new > SimpleMethodTest(caseName, this, childElement)); > - } else if ("entity-xml".equals(nodeName)) { > - this.testList.add(new > EntityXmlAssertTest(caseName, this, childElement)); > - } else if ("entity-xml-assert".equals(nodeName)) { > - // this is the old, deprecated name for the > element, changed because it now does assert or load > - this.testList.add(new > EntityXmlAssertTest(caseName, this, childElement)); > - } else if ("jython-test".equals(nodeName)) { > - this.testList.add(new JythonTest(caseName, > this, childElement)); > } > } > } > } > + > + private void parseTestElement(String caseName, Element > testElement) { > + String nodeName = testElement.getNodeName(); > + if ("junit-test-suite".equals(nodeName)) { > + String className = testElement.getAttribute("class- > name"); > + > + try { > + Class clz = ObjectType.loadClass(className); > + TestSuite suite = new TestSuite(); > + suite.addTestSuite(clz); > + Enumeration testEnum = suite.tests(); > + int testsAdded = 0; > + int casesAdded = 0; > + while (testEnum.hasMoreElements()) { > + Test tst = (Test) testEnum.nextElement(); > + this.testList.add(tst); > + casesAdded += tst.countTestCases(); > + testsAdded++; > + } > + Debug.logInfo("Added " + testsAdded + " tests [" + > casesAdded + " cases] from the class: " + className, module); > + } catch (Exception e) { > + String errMsg = "Unable to load test suite class : > " + className; > + Debug.logError(e, errMsg, module); > + } > + } else if ("service-test".equals(nodeName)) { > + this.testList.add(new ServiceTest(caseName, this, > testElement)); > + } else if ("simple-method-test".equals(nodeName)) { > + this.testList.add(new SimpleMethodTest(caseName, this, > testElement)); > + } else if ("entity-xml".equals(nodeName)) { > + this.testList.add(new EntityXmlAssertTest(caseName, > this, testElement)); > + } else if ("entity-xml-assert".equals(nodeName)) { > + // this is the old, deprecated name for the element, > changed because it now does assert or load > + this.testList.add(new EntityXmlAssertTest(caseName, > this, testElement)); > + } else if ("jython-test".equals(nodeName)) { > + this.testList.add(new JythonTest(caseName, this, > testElement)); > + } > + > + } > > String getSuiteName() { > return this.suiteName; > > Modified: ofbiz/trunk/framework/testtools/src/org/ofbiz/testtools/ > TestListContainer.java > URL: http://svn.apache.org/viewvc/ofbiz/trunk/framework/testtools/src/org/ofbiz/testtools/TestListContainer.java?rev=750296&r1=750295&r2=750296&view=diff > = > = > = > = > = > = > = > = > ====================================================================== > --- ofbiz/trunk/framework/testtools/src/org/ofbiz/testtools/ > TestListContainer.java (original) > +++ ofbiz/trunk/framework/testtools/src/org/ofbiz/testtools/ > TestListContainer.java Thu Mar 5 02:40:06 2009 > @@ -27,6 +27,7 @@ > import org.ofbiz.base.container.Container; > import org.ofbiz.base.container.ContainerException; > import org.ofbiz.base.util.Debug; > +import org.ofbiz.base.util.UtilMisc; > import org.ofbiz.base.util.UtilXml; > > import java.io.File; > @@ -69,7 +70,7 @@ > try { > Document testSuiteDocument = > testSuiteResource.getDocument(); > Element documentElement = > testSuiteDocument.getDocumentElement(); > - for (Element testCaseElement : > UtilXml.childElementList(documentElement, "test-case")) { > + for (Element testCaseElement : > UtilXml.childElementList(documentElement, UtilMisc.toSet("test- > case", "test-group"))) { > String caseName = > testCaseElement.getAttribute("case-name"); > pout.println(componentName + ":" + caseName); > } > > |
Re: svn commit: r750296 - in /ofbiz/trunk/framework/testtoo
|
|
David E Jones wrote:
> > Why not just use different test-suite elements for this? Why have an > hierarchy in this? > > Actually... I'm not even sure what "this" is, ie what is the point of a > group? Maybe we should discuss that first... The service engine tests define several different test cases. The second test case does a data load. Then each subsequent test modifies the data state of the previous test. They all have to run in sequence, for them to pass. This means that they are *not* separate test cases, but a single test case. A test case is supposed to be self-contained, not depending on the state of anything else. It's supposed to start from empty, do whatever mutative changes it needs to, without any external dependencies. The service test cases did not follow this. I've got a script that runs each defined test case separately; it stops/starts ofbiz for each individual test case, and restores a previous saved copy of the derby data folder between each test. |
Test Practices and Patterns (was Re: svn commit: r750296 - in /ofbiz/trunk/framework/testtoo
|
|
On Mar 4, 2009, at 8:03 PM, Adam Heath wrote: > David E Jones wrote: >> >> Why not just use different test-suite elements for this? Why have an >> hierarchy in this? >> >> Actually... I'm not even sure what "this" is, ie what is the point >> of a >> group? Maybe we should discuss that first... > > The service engine tests define several different test cases. The > second test case does a data load. Then each subsequent test modifies > the data state of the previous test. They all have to run in > sequence, for them to pass. > > This means that they are *not* separate test cases, but a single test > case. A test case is supposed to be self-contained, not depending on > the state of anything else. It's supposed to start from empty, do > whatever mutative changes it needs to, without any external > dependencies. Who says? Why? The change you made doesn't change this, it just creates something new called a "test group" that I'm guessing is supposed to group tests together than depend on eachother. However, it doesn't make the test cases independent. Also, why introduce a test group concept when we already have the test- suite concept? Why not just use that? Run one suite at a time and only allow dependencies within that suite. Still, this hasn't been discussed. The only thing we've discussed and somewhat decided on so far (including the ApacheCon discussions I was involved in, which was the same as what was discussed before) is to only allow dependencies between test cases within components. We could restrict that more by only allowing dependencies between test cases within a suite (which I think is the more normal practice), and the terms "test case" and "test suite" are more known as those are used in JUnit and other things. > The service test cases did not follow this. Of course not. It's never been discussed or defined or established even as a recommended practice, let alone one we'll adopt... possibly because it doesn't make much sense? Backing up a little bit, you can make the assertion that each test case should be 100% independent, but that is only the beginning of the discussion, not the end of it. We do need to decide on what dependencies will be allowed, but I'm pretty sure won't end up with an agreed on answer of "no dependencies period". That leads to excessive redundancy. I guess one thing to keep in mind is that while we use JUnit underneath we're actually talking about something higher level than unit tests. These are general test cases and test suites, not necessarily just unit tests as those are really of limited utility. In fact, we're mostly testing processes with many steps and in the case of the applications tests these are meant to call a series of services that would normally be called through the UI, but we're dropping down to the service layer for them to test on that level. > I've got a script that runs each defined test case separately; it > stops/starts ofbiz for each individual test case, and restores a > previous saved copy of the derby data folder between each test. Yep, I saw that, looks cool. Either way, unless there is a good reason to have these test-groups, let's just stick with test-suite. I still haven't seen anything you've written about what the test-group will be used for, so at least that would be helpful. -David |
Re: Test Practices and Patterns (was Re: svn commit: r750296 - in /ofbiz/trunk/framework/testtoo
|
|
David E Jones wrote:
> > On Mar 4, 2009, at 8:03 PM, Adam Heath wrote: > >> David E Jones wrote: >>> >>> Why not just use different test-suite elements for this? Why have an >>> hierarchy in this? >>> >>> Actually... I'm not even sure what "this" is, ie what is the point of a >>> group? Maybe we should discuss that first... >> >> The service engine tests define several different test cases. The >> second test case does a data load. Then each subsequent test modifies >> the data state of the previous test. They all have to run in >> sequence, for them to pass. >> >> This means that they are *not* separate test cases, but a single test >> case. A test case is supposed to be self-contained, not depending on >> the state of anything else. It's supposed to start from empty, do >> whatever mutative changes it needs to, without any external >> dependencies. > > Who says? > > Why? > > The change you made doesn't change this, it just creates something new > called a "test group" that I'm guessing is supposed to group tests > together than depend on eachother. However, it doesn't make the test > cases independent. > > Also, why introduce a test group concept when we already have the > test-suite concept? Why not just use that? Run one suite at a time and > only allow dependencies within that suite. Well, TestRunContainer supports running a singly-defined test case from some testdef.xml file. If done against service tests, then you'll get false positives. I'm fine with not having this <test-group>, so long as we don't allow for running separate tests, that TestRunContainer currently allows. Currently, we can run all tests suites inside a component, or one separate test in a component, in whatever test suite it resides in. We have no way to restrict the lookup code to just a single test suite. > Still, this hasn't been discussed. The only thing we've discussed and > somewhat decided on so far (including the ApacheCon discussions I was > involved in, which was the same as what was discussed before) is to only > allow dependencies between test cases within components. Well, in some cases, multiple tests defined inside some test-suite.xml can *not* run in series, or in parallel. Each defined test case needs a *clean* database. In other cases, the test cases inside the test-suite.xml *do* require running each in series. > We could restrict that more by only allowing dependencies between test > cases within a suite (which I think is the more normal practice), and > the terms "test case" and "test suite" are more known as those are used > in JUnit and other things. > >> The service test cases did not follow this. > > Of course not. It's never been discussed or defined or established even > as a recommended practice, let alone one we'll adopt... possibly because > it doesn't make much sense? > > Backing up a little bit, you can make the assertion that each test case > should be 100% independent, but that is only the beginning of the > discussion, not the end of it. We do need to decide on what dependencies > will be allowed, but I'm pretty sure won't end up with an agreed on > answer of "no dependencies period". That leads to excessive redundancy. > > I guess one thing to keep in mind is that while we use JUnit underneath > we're actually talking about something higher level than unit tests. > These are general test cases and test suites, not necessarily just unit > tests as those are really of limited utility. In fact, we're mostly > testing processes with many steps and in the case of the applications > tests these are meant to call a series of services that would normally > be called through the UI, but we're dropping down to the service layer > for them to test on that level. You and I are on the same wave length. >> I've got a script that runs each defined test case separately; it >> stops/starts ofbiz for each individual test case, and restores a >> previous saved copy of the derby data folder between each test. > > Yep, I saw that, looks cool. > > Either way, unless there is a good reason to have these test-groups, > let's just stick with test-suite. I still haven't seen anything you've > written about what the test-group will be used for, so at least that > would be helpful. I had a later commit that changed the service tests over to it; however, that was just an automatic-type change, so that my script wouldn't break, by trying to run each one separately. |
Re: Test Practices and Patterns (was Re: svn commit: r750296 - in /ofbiz/trunk/framework/testtoo
|
|
*** First off Adam: it's great to see you working on this stuff, the testing stuff in general needs a LOT more work and hasn't seen much attention, so yeah... this is great. On Mar 4, 2009, at 9:25 PM, Adam Heath wrote: >>> The service engine tests define several different test cases. The >>> second test case does a data load. Then each subsequent test >>> modifies >>> the data state of the previous test. They all have to run in >>> sequence, for them to pass. >>> >>> This means that they are *not* separate test cases, but a single >>> test >>> case. A test case is supposed to be self-contained, not depending >>> on >>> the state of anything else. It's supposed to start from empty, do >>> whatever mutative changes it needs to, without any external >>> dependencies. >> >> Who says? >> >> Why? >> >> The change you made doesn't change this, it just creates something >> new >> called a "test group" that I'm guessing is supposed to group tests >> together than depend on eachother. However, it doesn't make the test >> cases independent. >> >> Also, why introduce a test group concept when we already have the >> test-suite concept? Why not just use that? Run one suite at a time >> and >> only allow dependencies within that suite. > > Well, TestRunContainer supports running a singly-defined test case > from some testdef.xml file. If done against service tests, then > you'll get false positives. > > I'm fine with not having this <test-group>, so long as we don't allow > for running separate tests, that TestRunContainer currently allows. > > Currently, we can run all tests suites inside a component, or one > separate test in a component, in whatever test suite it resides in. > We have no way to restrict the lookup code to just a single test > suite. What does all of this have to do with it? Are you just saying that we need a way to run one test suite at a time? If we're going to move toward the practice of having dependencies only within a single test suite, then yes, I'd agree this is necessary. At the minute the practice is to allow dependencies within a component. >> Still, this hasn't been discussed. The only thing we've discussed and >> somewhat decided on so far (including the ApacheCon discussions I was >> involved in, which was the same as what was discussed before) is to >> only >> allow dependencies between test cases within components. > > Well, in some cases, multiple tests defined inside some test-suite.xml > can *not* run in series, or in parallel. Each defined test case needs > a *clean* database. In other cases, the test cases inside the > test-suite.xml *do* require running each in series. Oh yeah? Where? Here's the scary thing that I'm starting to believe to be the case: because you didn't read and research before starting to change things you didn't try running the tests for a component all at once. You made incorrect assumptions about how they were written and how they were meant to be run. Your rants and complaints about the tests may be incorrect, and the "fixes" you've been making for them may be unnecessary. They are only true if you adopt your assertion that all test cases should be independent. Otherwise, they are not true and you're heading down a road that others may not appreciate. Could we maybe step back and talk about this before you cruise along too much more? -David |
Re: Test Practices and Patterns (was Re: svn commit: r750296 - in /ofbiz/trunk/framework/testtoo
|
|
David E Jones wrote:
> *** First off Adam: it's great to see you working on this stuff, the > testing stuff in general needs a LOT more work and hasn't seen much > attention, so yeah... this is great. Completely automatic tests with no side-effects is what I am striving for. I believe we are close. > On Mar 4, 2009, at 9:25 PM, Adam Heath wrote: > >>>> The service engine tests define several different test cases. The >>>> second test case does a data load. Then each subsequent test modifies >>>> the data state of the previous test. They all have to run in >>>> sequence, for them to pass. >>>> >>>> This means that they are *not* separate test cases, but a single test >>>> case. A test case is supposed to be self-contained, not depending on >>>> the state of anything else. It's supposed to start from empty, do >>>> whatever mutative changes it needs to, without any external >>>> dependencies. >>> >>> Who says? >>> >>> Why? >>> >>> The change you made doesn't change this, it just creates something new >>> called a "test group" that I'm guessing is supposed to group tests >>> together than depend on eachother. However, it doesn't make the test >>> cases independent. >>> >>> Also, why introduce a test group concept when we already have the >>> test-suite concept? Why not just use that? Run one suite at a time and >>> only allow dependencies within that suite. >> >> Well, TestRunContainer supports running a singly-defined test case >> from some testdef.xml file. If done against service tests, then >> you'll get false positives. >> >> I'm fine with not having this <test-group>, so long as we don't allow >> for running separate tests, that TestRunContainer currently allows. >> >> Currently, we can run all tests suites inside a component, or one >> separate test in a component, in whatever test suite it resides in. >> We have no way to restrict the lookup code to just a single test suite. > > What does all of this have to do with it? > > Are you just saying that we need a way to run one test suite at a time? > If we're going to move toward the practice of having dependencies only > within a single test suite, then yes, I'd agree this is necessary. > > At the minute the practice is to allow dependencies within a component. > >>> Still, this hasn't been discussed. The only thing we've discussed and >>> somewhat decided on so far (including the ApacheCon discussions I was >>> involved in, which was the same as what was discussed before) is to only >>> allow dependencies between test cases within components. >> >> Well, in some cases, multiple tests defined inside some test-suite.xml >> can *not* run in series, or in parallel. Each defined test case needs >> a *clean* database. In other cases, the test cases inside the >> test-suite.xml *do* require running each in series. > > Oh yeah? Where? > > Here's the scary thing that I'm starting to believe to be the case: > because you didn't read and research before starting to change things > you didn't try running the tests for a component all at once. You made > incorrect assumptions about how they were written and how they were > meant to be run. But I did do some. The service tests require running everything in the test suite. That I am absolutely certain of. The other thing we know, is that running some tests together causes problems. I'm not entirely certain if these same bad tests together exist in a single test suite, or if different test suites conflict. > Your rants and complaints about the tests may be incorrect, and the > "fixes" you've been making for them may be unnecessary. They are only > true if you adopt your assertion that all test cases should be > independent. Otherwise, they are not true and you're heading down a road > that others may not appreciate. > > Could we maybe step back and talk about this before you cruise along too > much more? I still stand by the <test-group> addition; it's a new feature, that seems usefull. It allows combining separately defined test cases into a single whole, without giving them each a separate name. It allows for code reuse, without having to write some java code, or a simple-method, to combine them. However, what we are heading towards is making the granularity be a test suite. In that case, the service test definition should be changed back, as well as changing my recent build.xml and TestListContainer policy. Just to restate, I will change the policy to be each suite gets run separately. |
Re: Test Practices and Patterns (was Re: svn commit: r750296 - in /ofbiz/trunk/framework/testtoo
|
|
Gentlemen, great debate. I will provide a bit more information a bit later as I don't want to answer without fully running thru all of the scenarios that everyone lays out. Adam - thanks for pushing on this - it's needed it for a long time.
Cheers, Tim -- Tim Ruppert HotWax Media http://www.hotwaxmedia.com o:801.649.6594 f:801.649.6595 ----- "Adam Heath" <[hidden email]> wrote: > David E Jones wrote: > > *** First off Adam: it's great to see you working on this stuff, > the > > testing stuff in general needs a LOT more work and hasn't seen much > > attention, so yeah... this is great. > > Completely automatic tests with no side-effects is what I am striving > for. I believe we are close. > > > On Mar 4, 2009, at 9:25 PM, Adam Heath wrote: > > > >>>> The service engine tests define several different test cases. > The > >>>> second test case does a data load. Then each subsequent test > modifies > >>>> the data state of the previous test. They all have to run in > >>>> sequence, for them to pass. > >>>> > >>>> This means that they are *not* separate test cases, but a single > test > >>>> case. A test case is supposed to be self-contained, not > depending on > >>>> the state of anything else. It's supposed to start from empty, > do > >>>> whatever mutative changes it needs to, without any external > >>>> dependencies. > >>> > >>> Who says? > >>> > >>> Why? > >>> > >>> The change you made doesn't change this, it just creates something > new > >>> called a "test group" that I'm guessing is supposed to group > tests > >>> together than depend on eachother. However, it doesn't make the > test > >>> cases independent. > >>> > >>> Also, why introduce a test group concept when we already have the > >>> test-suite concept? Why not just use that? Run one suite at a time > and > >>> only allow dependencies within that suite. > >> > >> Well, TestRunContainer supports running a singly-defined test case > >> from some testdef.xml file. If done against service tests, then > >> you'll get false positives. > >> > >> I'm fine with not having this <test-group>, so long as we don't > allow > >> for running separate tests, that TestRunContainer currently > allows. > >> > >> Currently, we can run all tests suites inside a component, or one > >> separate test in a component, in whatever test suite it resides > in. > >> We have no way to restrict the lookup code to just a single test > suite. > > > > What does all of this have to do with it? > > > > Are you just saying that we need a way to run one test suite at a > time? > > If we're going to move toward the practice of having dependencies > only > > within a single test suite, then yes, I'd agree this is necessary. > > > > At the minute the practice is to allow dependencies within a > component. > > > >>> Still, this hasn't been discussed. The only thing we've discussed > and > >>> somewhat decided on so far (including the ApacheCon discussions I > was > >>> involved in, which was the same as what was discussed before) is > to only > >>> allow dependencies between test cases within components. > >> > >> Well, in some cases, multiple tests defined inside some > test-suite.xml > >> can *not* run in series, or in parallel. Each defined test case > needs > >> a *clean* database. In other cases, the test cases inside the > >> test-suite.xml *do* require running each in series. > > > > Oh yeah? Where? > > > > Here's the scary thing that I'm starting to believe to be the case: > > because you didn't read and research before starting to change > things > > you didn't try running the tests for a component all at once. You > made > > incorrect assumptions about how they were written and how they were > > meant to be run. > > But I did do some. > > The service tests require running everything in the test suite. That > I am absolutely certain of. > > The other thing we know, is that running some tests together causes > problems. I'm not entirely certain if these same bad tests together > exist in a single test suite, or if different test suites conflict. > > > Your rants and complaints about the tests may be incorrect, and the > > "fixes" you've been making for them may be unnecessary. They are > only > > true if you adopt your assertion that all test cases should be > > independent. Otherwise, they are not true and you're heading down a > road > > that others may not appreciate. > > > > Could we maybe step back and talk about this before you cruise along > too > > much more? > > I still stand by the <test-group> addition; it's a new feature, that > seems usefull. It allows combining separately defined test cases > into > a single whole, without giving them each a separate name. It allows > for code reuse, without having to write some java code, or a > simple-method, to combine them. > > However, what we are heading towards is making the granularity be a > test suite. In that case, the service test definition should be > changed back, as well as changing my recent build.xml and > TestListContainer policy. > > Just to restate, I will change the policy to be each suite gets run > separately. |
Re: Test Practices and Patterns (was Re: svn commit: r750296 - in /ofbiz/trunk/framework/testtoo
|
|
I'm fine with the test-group, I think one test suite can have some groups
and one group can contain some test cases. It's good for cases organization. As my experience, the test cases can share some precondition and post condition in group scope, also the groups can share precondition and post condition too in current suite, that allows for code reuse as Adam say. And it's the bad habit if one case depend on the other one, we should avoid this.Test cases can share precondtion and post condition, but don't share the test case code each other. 2009/3/5 Tim Ruppert <[hidden email]> > Gentlemen, great debate. I will provide a bit more information a bit later > as I don't want to answer without fully running thru all of the scenarios > that everyone lays out. Adam - thanks for pushing on this - it's needed it > for a long time. > > Cheers, > Tim > -- > Tim Ruppert > HotWax Media > http://www.hotwaxmedia.com > > o:801.649.6594 > f:801.649.6595 > > ----- "Adam Heath" <[hidden email]> wrote: > > > David E Jones wrote: > > > *** First off Adam: it's great to see you working on this stuff, > > the > > > testing stuff in general needs a LOT more work and hasn't seen much > > > attention, so yeah... this is great. > > > > Completely automatic tests with no side-effects is what I am striving > > for. I believe we are close. > > > > > On Mar 4, 2009, at 9:25 PM, Adam Heath wrote: > > > > > >>>> The service engine tests define several different test cases. > > The > > >>>> second test case does a data load. Then each subsequent test > > modifies > > >>>> the data state of the previous test. They all have to run in > > >>>> sequence, for them to pass. > > >>>> > > >>>> This means that they are *not* separate test cases, but a single > > test > > >>>> case. A test case is supposed to be self-contained, not > > depending on > > >>>> the state of anything else. It's supposed to start from empty, > > do > > >>>> whatever mutative changes it needs to, without any external > > >>>> dependencies. > > >>> > > >>> Who says? > > >>> > > >>> Why? > > >>> > > >>> The change you made doesn't change this, it just creates something > > new > > >>> called a "test group" that I'm guessing is supposed to group > > tests > > >>> together than depend on eachother. However, it doesn't make the > > test > > >>> cases independent. > > >>> > > >>> Also, why introduce a test group concept when we already have the > > >>> test-suite concept? Why not just use that? Run one suite at a time > > and > > >>> only allow dependencies within that suite. > > >> > > >> Well, TestRunContainer supports running a singly-defined test case > > >> from some testdef.xml file. If done against service tests, then > > >> you'll get false positives. > > >> > > >> I'm fine with not having this <test-group>, so long as we don't > > allow > > >> for running separate tests, that TestRunContainer currently > > allows. > > >> > > >> Currently, we can run all tests suites inside a component, or one > > >> separate test in a component, in whatever test suite it resides > > in. > > >> We have no way to restrict the lookup code to just a single test > > suite. > > > > > > What does all of this have to do with it? > > > > > > Are you just saying that we need a way to run one test suite at a > > time? > > > If we're going to move toward the practice of having dependencies > > only > > > within a single test suite, then yes, I'd agree this is necessary. > > > > > > At the minute the practice is to allow dependencies within a > > component. > > > > > >>> Still, this hasn't been discussed. The only thing we've discussed > > and > > >>> somewhat decided on so far (including the ApacheCon discussions I > > was > > >>> involved in, which was the same as what was discussed before) is > > to only > > >>> allow dependencies between test cases within components. > > >> > > >> Well, in some cases, multiple tests defined inside some > > test-suite.xml > > >> can *not* run in series, or in parallel. Each defined test case > > needs > > >> a *clean* database. In other cases, the test cases inside the > > >> test-suite.xml *do* require running each in series. > > > > > > Oh yeah? Where? > > > > > > Here's the scary thing that I'm starting to believe to be the case: > > > because you didn't read and research before starting to change > > things > > > you didn't try running the tests for a component all at once. You > > made > > > incorrect assumptions about how they were written and how they were > > > meant to be run. > > > > But I did do some. > > > > The service tests require running everything in the test suite. That > > I am absolutely certain of. > > > > The other thing we know, is that running some tests together causes > > problems. I'm not entirely certain if these same bad tests together > > exist in a single test suite, or if different test suites conflict. > > > > > Your rants and complaints about the tests may be incorrect, and the > > > "fixes" you've been making for them may be unnecessary. They are > > only > > > true if you adopt your assertion that all test cases should be > > > independent. Otherwise, they are not true and you're heading down a > > road > > > that others may not appreciate. > > > > > > Could we maybe step back and talk about this before you cruise along > > too > > > much more? > > > > I still stand by the <test-group> addition; it's a new feature, that > > seems usefull. It allows combining separately defined test cases > > into > > a single whole, without giving them each a separate name. It allows > > for code reuse, without having to write some java code, or a > > simple-method, to combine them. > > > > However, what we are heading towards is making the granularity be a > > test suite. In that case, the service test definition should be > > changed back, as well as changing my recent build.xml and > > TestListContainer policy. > > > > Just to restate, I will change the policy to be each suite gets run > > separately. > |
«
Return to OFBiz - Dev
|
1 view|%1 views
| Free forum by Nabble | Edit this page |