OFBiz Testing Initiative
Locked
 
|
I am having trouble getting my messages posted to the list. This one
is taking a long time and I wonder if it is because it has HTML in the content and an attachment. I am pasting it in plain test. ... One of the presentations at the Users Conference will be me talking on my experiences with Grinder. With David there, we hope to go far past that to discuss strategies for adding testing functionality to OFBiz. I have no experience that qualifies me to be an expert in this area - just have a need. Rather than just wait for everyone to show up at the conference, I felt it would be helpful to try to treat some of the subjects ahead of time. Of course, not everything will be decided at the conference, so as much input as possible from everyone will be needed. Also, if there is anyone with strong Python and/or Grinder skills, their help would also be appreciated. OFBiz Testing Initiative GOAL BACKGROUND Problem with Design Documents Applicable Features of OFBiz Applicable Features of Grinder Applicable Features of Extreme Programming REQUIREMENTS Easy Generation Handle Complex Tests Adaptable per Application Rugged Handle Iterations Usable at Multiple Proficiency Levels Simplify Output Testing STRAWMAN PLAN Modify Test Script Generation Add Test Overlay Config File Source Code Control to Handle Iterations Appendices Standard Generated Grinder Test Script Possible Custom Generated Grinder Test Script Possible Overlay XML File GOAL The goal of the OFBiz Testing Initiative (OFBTI) is to add functionality to OFBiz that will streamline the functional testing process to the point that it will be cost-effective and beneficial to write comprehensive tests. Ideally, the tools could be used by non-programmers. A secondary goal would be to add tools to make design specs easier to write and more useful. BACKGROUND Problem with Design Documents Design documents, though necessary, have serious drawbacks. Expensive to create - should probably take as much time as the implementation. Never complete enough - always need to go back to client, anyway. No automation help in implementation and validation - as text documents, they must be manually translated into architecture documents. Easy to get out-of-date - while many times they are supposed to be "living" documents, in reality, they almost never are. Applicable Features of Extreme Programming Extreme programming addresses many of the shortcomings of the more conventional design document approach. Instead of a complete design doc or even use case up front, all that is required is a "story". The story needs only be complete enough to generate the first test and the first test is usually just a placeholder. Then in the course of many iterations, the functional test is enhanced, not the design document. The tests are living and serve the useful purpose of verifying that things have not broken as code gets changed. One of the very useful things about this approach is that when bugs are discovered, a test or subtest is added to guard against it ever creeping back into the code. Applicable Features of OFBiz OFBiz is different from other development environments, and it would be good to identify the features that could be used advantageously or which must be dealt with. OFBiz is highly configurable via XML formatted files and it would be good to continue that pattern. XML config files, because they have associated schema, make it feasible for non-programmers and those not intimately familiar with OFBiz to make changes and perform certain programming tasks. OFBiz has a general pattern of offering easily configurable options via data parameters, but always offering easy access to the lowest level of programming for those needed cases without requiring a huge environment adaptation. For instance, the screen widget config files allow screens to be created with miminal data and allow the user to be prompted via the schema in an XML editor, but if there is a construct that the widgets do not handle, the user can just throw in a call to an HTML component or a FreeMarker template - they do not need to abandon the screen widget framework. OFBiz is a service oriented framework and there are many tools that can be used aid in that process. If a request is processed by a service (in lieu of an event) then the input parameters can be automatically taken from the HTTP stream and automatically converted to the right type. Also, the form widget can build forms from the service definition. Applicable Features of Grinder There are multiple options for testing frameworks, but Grinder offers the following unique advantages. A robust test script generator - TCPProxy attempts to assign return parameters to variables and reuse them, rather than passing literals around. This means that it does not instantly break when keys are autogenerated. Grinder has convenient customization point - script generation is done via an XSLT stylesheet that makes use of Java extensions. This would be a natural place to make modifications to suit OFBTI. There are also filters for the requests and responses that can be swapped out via command line parameters. Jython - allows seamless switching between a highly productive scripting environment and regular Java code. Because of this feature, Jython has a lot of possibilities within OFBiz. Integrated functional and jUnit testing - the use of Jython allows for the same environment to be used to run HTTP client system tests and jUnit functional tests. REQUIREMENTS Easy Generation The use of a test script generator, such as Grinder's TCPProxy, allows test scripts to be created by capturing the input of a user at a browser. Handle Complex Tests A further enhancement would be to allow the creation of scripts by supplying a few data parameters or by easily modifying automatically generated scripts. One of the biggest drawbacks of end user testing environments is that when something changes, the associated test script is usually unusable. A big improvement would be to allow tests to be created by chaining together subtests. Adaptable per Application Many applications will have special characteristics that need to be handled specially. In one instance, I found that Grinder was using the '$' character to form Python variable names (which doesn't work) because that is how the HTTP parameters were named. Rugged Handle Iterations The general idea of XP is that the tests become more complex and comprehensive as the code becomes more complete. It would probably be a good idea to retain visibility to past test setups - though, at this time, I am not sure why. Usable at Multiple Proficiency Levels It would be good to have the testing environment useable, or at least understandable, by non-programmers. JUnit tests would not meet this requirement. Neither would Python scripts (though it could come close). For project managers, an XML-based configuration environment would be needed. Simplify Output Testing The analysis of the HTML pages that are returned by Grinder tests is one of the more problematic areas of end-user testing. The more common scenarios would need to be handled by the XML-based configuration environment mentioned above. The foreseeable cases would be matching of form values, existence or non-existence anywhere of a phrase or something that can be tested by an XPath expression. STRAWMAN PLAN What is discussed below is how I see tackling the testing problem. I am only interested in what will be supported by the community, so it can be changed. The overall approach is to use as much of Grinder as possible, and allow for the generated Grinder test scripts to be overwritten by a user generated XML test configuration file. Modify Test Script Generation The Grinder TCPProxy program will be used, but the XSLT stylesheet and the associated Java extension classes will be enhanced. The main change will be that the individual request tests that Grinder generates for each round trip to the server will be wrapped by code that will check to see if there are data overrides to be made coming from the user generated high level test config file. Each wrapper will also have a dictionary (ie. map) that defines tests to be made on each low level result. These also will be modified by the higher level config file. If there is not higher level config file, the script as generated by TCPProxy will run with no modifications. Input Dictionary The input directory will do more than just allow the user to supply literal values as test input. There will be helper functions for allowing the user to randomly pick values from a list. Also there will be helper functions for generating reasonably looking addresses. The user will be able to create Python scripts to generated special input. See the appendix for samples of what this would look like. Output Dictionary The output dictionary, which contains the test criteria, would have test helper functions of the following sort: Literal value Regular expression Form value XPath Custom Jython scripts Many of the test methods will be suitable for non-programmers, but the use of Jython scripts follows the OFBiz pattern of providing simple methods for simple tasks, but making it easy to drop down to the level needed to solve the problem. At some point (right away?), it will be necessary to allow complex joing of tests with AND, OR and NOT operators. Add Test Overlay Config File One of the modifications to the OOTB test script generated by TCPProxy will be that the script will look for the existence a file path as a parameters and use it to overwrite the default test script values (ie. the values typed in by the user when the script was being generated). The script could have multiple levels of complex scripts, but eventually, they must call one of the "request" tests. Keep in mind that the base test script could have a large number of "request" scripts, but they would not all need to be used by the user-defined config file; it could just use a subset. So the base test script may not, in fact, be totally generated by one session of user interaction with the system under test; it could be built up over time as new functionality is added without having to run thru the test. The OOTB behavior is to sequentially number the tests, but in this case, we may wish to name them with the request name that they interact with. These high level test scripts would need "include" functionality so that they could used standard building blocks. Source Code Control to Handle Iterations Appendices Standard Generated Grinder Test Script This is the actual output from a TCPProxy session: # The Grinder 3.0-beta30 # HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM from net.grinder.script import Test from net.grinder.script.Grinder import grinder from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest from HTTPClient import NVPair connectionDefaults = HTTPPluginControl.getConnectionDefaults() httpUtilities = HTTPPluginControl.getHTTPUtilities() # To use a proxy server, uncomment the next line and set the host and port. # connectionDefaults.setProxyServer("localhost", 8001) # These definitions at the top level of the file are evaluated once, # when the worker process is started. connectionDefaults.defaultHeaders = \ ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'), NVPair('Accept-Encoding', 'gzip,deflate'), NVPair('Accept-Language', 'en-us,en;q= 0.5'), NVPair('Accept-Charset', 'UTF-8,*'), NVPair('Accept', 'text/xml,application/xml,application/xhtml+xml,text/html;q= 0.9,text/plain;q=0.8,image/png,*/*;q=0.5'), ) headers0= \ ( ) headers1= \ ( NVPair('Referer', 'https://localhost:8443/webtools/control/main' ), ) headers2= \ ( NVPair('Referer', ' https://localhost:8443/webtools/control/checkLogin/main'), ) url0 = ' <a href="https://localhost:8443'">https://localhost:8443' # Create an HTTPRequest for each request, then replace the # reference to the HTTPRequest with an instrumented version. # You can access the unadorned instance using request101.__target__. request101 = HTTPRequest(url=url0, headers=headers0) request101 = Test(101, 'GET /').wrap(request101) request102 = HTTPRequest(url=url0, headers=headers0) request102 = Test(102, 'GET main').wrap(request102) request201 = HTTPRequest(url=url0, headers=headers1) request201 = Test(201, 'GET main').wrap(request201) request301 = HTTPRequest(url=url0, headers=headers2) request301 = Test(301, 'POST login').wrap(request301) class TestRunner: """A TestRunner instance is created for each worker thread.""" # A method for each recorded page. def page1(self): """GET / (requests 101-102).""" # Expecting 302 'Moved Temporarily' result = request101.GET('/webtools/') grinder.sleep(16) request102.GET('/webtools/control/main') return result def page2(self): """GET main (request 201).""" result = request201.GET('/webtools/control/checkLogin/main') return result def page3(self): """POST login (request 301).""" result = request301.POST('/webtools/control/login', ( NVPair('USERNAME', 'admin'), NVPair('PASSWORD', 'ofbiz'), ), ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), )) return result def __call__(self): """This method is called for every run performed by the worker thread.""" self.page1() # GET / (requests 101-102) grinder.sleep(4328) self.page2() # GET main (request 201) grinder.sleep(2109) self.page3() # POST login (request 301) def instrumentMethod(test, method_name, c=TestRunner): """Instrument a method with the given Test.""" unadorned = getattr(c, method_name) import new method = new.instancemethod (test.wrap(unadorned), None, c) setattr(c, method_name, method) # Replace each method with an instrumented version. # You can call the unadorned method using self.page1.__target__(). instrumentMethod(Test(100, 'Page 1'), 'page1') instrumentMethod(Test(200, 'Page 2'), 'page2') instrumentMethod(Test(300, 'Page 3'), 'page3') Possible Custom Generated Grinder Test Script # The Grinder 3.0-beta30 # HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM from net.grinder.script import Test from net.grinder.script.Grinder import grinder from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest from HTTPClient import NVPair import AGrinderTest connectionDefaults = HTTPPluginControl.getConnectionDefaults() httpUtilities = HTTPPluginControl.getHTTPUtilities() # To use a proxy server, uncomment the next line and set the host and port. # connectionDefaults.setProxyServer("localhost", 8001) # These definitions at the top level of the file are evaluated once, # when the worker process is started. connectionDefaults.defaultHeaders = \ ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'), NVPair('Accept-Encoding', 'gzip,deflate'), NVPair('Accept-Language', 'en-us,en;q= 0.5'), NVPair('Accept-Charset', 'UTF-8,*'), NVPair('Accept', 'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'), ) headers0= \ ( ) headers1= \ ( NVPair('Referer', 'https://localhost:8443/webtools/control/main'), ) headers2= \ ( NVPair('Referer', ' https://localhost:8443/webtools/control/checkLogin/main'), ) url0 = '<a href="https://localhost:8443'">https://localhost:8443' # Create an HTTPRequest for each request, then replace the # reference to the HTTPRequest with an instrumented version. # You can access the unadorned instance using request101.__target__. request101 = HTTPRequest(url=url0, headers=headers0) request101 = Test(101, 'GET /').wrap(request101) request102 = HTTPRequest(url=url0, headers=headers0) request102 = Test(102, 'GET main').wrap(request102) request201 = HTTPRequest(url=url0, headers=headers1) request201 = Test(201, 'GET main').wrap(request201) request301 = HTTPRequest(url=url0, headers=headers2) request301 = Test(301, 'POST login').wrap(request301) class TestRunner: """A TestRunner instance is created for each worker thread.""" # A method for each recorded page. def page1(self): """GET / (requests 101-102).""" # Expecting 302 'Moved Temporarily' result = request101.GET('/webtools/') grinder.sleep(16) request102.GET('/webtools/control/main') return result def page2(self): """GET main (request 201).""" result = request201.GET('/webtools/control/checkLogin/main') return result def page3(self): """POST login (request 301).""" result = request301.POST('/webtools/control/login', ( NVPair('USERNAME', 'admin'), NVPair('PASSWORD', 'ofbiz'), ), ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), )) return result """This is not working code. Just an idea of how the generated code would look""" def webtoolsWrap(self, overlayMap): inMap = {} outMap = {} self.agrinder.runWrappedTest(page1, inMap, outMap, overlayMap) def mainWrap(self, overlayMap): inMap = {} outMap = {} self.agrinder.runWrappedTest(page2, inMap, outMap, overlayMap) def loginWrap(self, overlayMap): inMap = {'USERNAME':'admin', 'PASSWORD':'ofbiz'} outMap = {} self.agrinder.runWrappedTest (page2, inMap, outMap, overlayMap) def __call__(self): """This method is called for every run performed by the worker thread.""" """I am going to do some handwaving here because I want to get this doc out today, but there would be code here to read in an argument from the command line and use it as a file path to read in the "test overlay" XML doc""" self.agrinder = AGrinderTest.AGrinder () testNode = self.agrinder.getOverlayTest("webtools", sys.argv) self.webtoolsWrap(testNode) # GET / (requests 101-102) grinder.sleep(4328) self.mainWrap(testNode) # GET main (request 201) grinder.sleep(2109) self.loginWrap(testNode) # POST login (request 301) def instrumentMethod(test, method_name, c=TestRunner): """Instrument a method with the given Test.""" unadorned = getattr(c, method_name) import new method = new.instancemethod(test.wrap(unadorned), None, c) setattr(c, method_name, method) # Replace each method with an instrumented version. # You can call the unadorned method using self.page1.__target__(). instrumentMethod(Test(100, 'Page 1'), 'page1') instrumentMethod(Test(200, 'Page 2'), 'page2') instrumentMethod(Test(300, 'Page 3'), 'page3') Possible Overlay XML File This example is probably not even close to what the final test overlay XML script will look like, but I think it is better to have something to go from. <tests> <test name="webtools"> <subtest name="login"> <input> <field name="USERNAME">jdoe</field> <field name="PASSWORD">id9Ed3jk</field> </input> <match> <field type="regex" op="not">not found</field> <field type="script" op="true">isSuccess()</field> </match> </subtest> </test> </tests> |
|
|
Al,
This is great that you're doing this. I think a testing framework is really what we need right now. My suggestion is instead of creating an xml language like you show at the bottom to make some API methods that could be called in Jython scripts. I think that might be easier for people to work with and would require less work over time in maintaining and extending the xml language. On Nov 1, 2006, at 10:49 AM, Al Byers wrote: > I am having trouble getting my messages posted to the list. This one > is taking a long time and I wonder if it is because it has HTML in the > content and an attachment. I am pasting it in plain test. > ... > One of the presentations at the Users Conference will be me talking on > my experiences with Grinder. With David there, we hope to go far past > that to discuss strategies for adding testing functionality to OFBiz. > I have no experience that qualifies me to be an expert in this area - > just have a need. Rather than just wait for everyone to show up at the > conference, I felt it would be helpful to try to treat some of the > subjects ahead of time. Of course, not everything will be decided at > the conference, so as much input as possible from everyone will be > needed. Also, if there is anyone with strong Python and/or Grinder > skills, their help would also be appreciated. > > > OFBiz Testing Initiative > GOAL > BACKGROUND > Problem with Design Documents > Applicable Features of OFBiz > Applicable Features of Grinder > Applicable Features of Extreme Programming > REQUIREMENTS > Easy Generation > Handle Complex Tests > Adaptable per Application > Rugged > Handle Iterations > Usable at Multiple Proficiency Levels > Simplify Output Testing > STRAWMAN PLAN > Modify Test Script Generation > Add Test Overlay Config File > Source Code Control to Handle Iterations > Appendices > Standard Generated Grinder Test Script > Possible Custom Generated Grinder Test Script > Possible Overlay XML File > GOAL The goal of the OFBiz Testing Initiative (OFBTI) is to add > functionality to OFBiz that will streamline the functional testing > process to the point that it will be cost-effective and beneficial to > write comprehensive tests. Ideally, the tools could be used by > non-programmers. > > A secondary goal would be to add tools to make design specs easier to > write and more useful. > BACKGROUND > Problem with Design Documents Design documents, though necessary, have > serious drawbacks. > Expensive to create - should probably take as much time as the > implementation. > > Never complete enough - always need to go back to client, anyway. > > No automation help in implementation and validation - as text > documents, they must be manually translated into architecture > documents. > > Easy to get out-of-date - while many times they are supposed to be > "living" documents, in reality, they almost never are. > > > Applicable Features of Extreme Programming Extreme programming > addresses many of the shortcomings of the more conventional design > document approach. Instead of a complete design doc or even use case > up front, all that is required is a "story". The story needs only be > complete enough to generate the first test and the first test is > usually just a placeholder. Then in the course of many iterations, the > functional test is enhanced, not the design document. The tests are > living and serve the useful purpose of verifying that things have not > broken as code gets changed. > > One of the very useful things about this approach is that when bugs > are discovered, a test or subtest is added to guard against it ever > creeping back into the code. > Applicable Features of OFBiz OFBiz is different from other development > environments, and it would be good to identify the features that could > be used advantageously or which must be dealt with. > > OFBiz is highly configurable via XML formatted files and it would be > good to continue that pattern. XML config files, because they have > associated schema, make it feasible for non-programmers and those not > intimately familiar with OFBiz to make changes and perform certain > programming tasks. > > OFBiz has a general pattern of offering easily configurable options > via data parameters, but always offering easy access to the lowest > level of programming for those needed cases without requiring a huge > environment adaptation. For instance, the screen widget config files > allow screens to be created with miminal data and allow the user to be > prompted via the schema in an XML editor, but if there is a construct > that the widgets do not handle, the user can just throw in a call to > an HTML component or a FreeMarker template - they do not need to > abandon the screen widget framework. > > OFBiz is a service oriented framework and there are many tools that > can be used aid in that process. If a request is processed by a > service (in lieu of an event) then the input parameters can be > automatically taken from the HTTP stream and automatically converted > to the right type. Also, the form widget can build forms from the > service definition. > Applicable Features of Grinder There are multiple options for testing > frameworks, but Grinder offers the following unique advantages. > A robust test script generator - TCPProxy attempts to assign return > parameters to variables and reuse them, rather than passing > literals around. This means that it does not instantly break when keys > are autogenerated. > Grinder has convenient customization point - script generation is done > via an XSLT stylesheet that makes use of Java extensions. This would > be a natural place to make modifications to suit OFBTI. There are also > filters for the requests and responses that can be swapped out via > command line parameters. > Jython - allows seamless switching between a highly productive > scripting environment and regular Java code. Because of this feature, > Jython has a lot of possibilities within OFBiz. > Integrated functional and jUnit testing - the use of Jython allows for > the same environment to be used to run HTTP client system tests and > jUnit functional tests. > REQUIREMENTS > Easy Generation The use of a test script generator, such as Grinder's > TCPProxy, allows test scripts to be created by capturing the input of > a user at a browser. > Handle Complex Tests A further enhancement would be to allow the > creation of scripts by supplying a few data parameters or by easily > modifying automatically generated scripts. One of the biggest > drawbacks of end user testing environments is that when something > changes, the associated test script is usually unusable. A big > improvement would be to allow tests to be created by chaining together > subtests. > Adaptable per Application Many applications will have special > characteristics that need to be handled specially. In one instance, I > found that Grinder was using the '$' character to form Python variable > names (which doesn't work) because that is how the HTTP parameters > were named. > Rugged > Handle Iterations The general idea of XP is that the tests become more > complex and comprehensive as the code becomes more complete. It would > probably be a good idea to retain visibility to past test setups - > though, at this time, I am not sure why. > Usable at Multiple Proficiency Levels It would be good to have the > testing environment useable, or at least understandable, by > non-programmers. JUnit tests would not meet this requirement. Neither > would Python scripts (though it could come close). For project > managers, an XML-based configuration environment would be needed. > Simplify Output Testing The analysis of the HTML pages that are > returned by Grinder tests is one of the more problematic areas of > end-user testing. The more common scenarios would need to be handled > by the XML-based configuration environment mentioned above. The > foreseeable cases would be matching of form values, existence or > non-existence anywhere of a phrase or something that can be tested by > an XPath expression. > STRAWMAN PLAN What is discussed below is how I see tackling the > testing problem. I am only interested in what will be supported by the > community, so it can be changed. > > The overall approach is to use as much of Grinder as possible, and > allow for the generated Grinder test scripts to be overwritten by a > user generated XML test configuration file. > Modify Test Script Generation The Grinder TCPProxy program will be > used, but the XSLT stylesheet and the associated Java extension > classes will be enhanced. The main change will be that the individual > request tests that Grinder generates for each round trip to the server > will be wrapped by code that will check to see if there are data > overrides to be made coming from the user generated high level test > config file. Each wrapper will also have a dictionary (ie. map) that > defines tests to be made on each low level result. These also will be > modified by the higher level config file. If there is not higher level > config file, the script as generated by TCPProxy will run with no > modifications. > Input Dictionary The input directory will do more than just allow the > user to supply literal values as test input. There will be helper > functions for allowing the user to randomly pick values from a list. > Also there will be helper functions for generating reasonably looking > addresses. The user will be able to create Python scripts to generated > special input. See the appendix for samples of what this would look > like. > Output Dictionary The output dictionary, which contains the test > criteria, would have test helper functions of the following sort: > Literal value > Regular expression > Form value > XPath > Custom Jython scripts Many of the test methods will be suitable for > non-programmers, but the use of Jython scripts follows the OFBiz > pattern of providing simple methods for simple tasks, but making it > easy to drop down to the level needed to solve the problem. At some > point (right away?), it will be necessary to allow complex joing of > tests with AND, OR and NOT operators. > Add Test Overlay Config File One of the modifications to the OOTB test > script generated by TCPProxy will be that the script will look for the > existence a file path as a parameters and use it to overwrite the > default test script values (ie. the values typed in by the user when > the script was being generated). The script could have multiple levels > of complex scripts, but eventually, they must call one of the > "request" tests. Keep in mind that the base test script could have a > large number of "request" scripts, but they would not all need to be > used by the user-defined config file; it could just use a subset. So > the base test script may not, in fact, be totally generated by one > session of user interaction with the system under test; it could be > built up over time as new functionality is added without having to run > thru the test. The OOTB behavior is to sequentially number the tests, > but in this case, we may wish to name them with the request name that > they interact with. > > These high level test scripts would need "include" functionality so > that they could used standard building blocks. > Source Code Control to Handle Iterations > Appendices > Standard Generated Grinder Test Script This is the actual output from > a TCPProxy session: > > # The Grinder 3.0-beta30 > # HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM > > from net.grinder.script import Test > from net.grinder.script.Grinder import grinder > from net.grinder.plugin.http > import HTTPPluginControl, HTTPRequest > from HTTPClient import NVPair > connectionDefaults = HTTPPluginControl.getConnectionDefaults() > httpUtilities = HTTPPluginControl.getHTTPUtilities() > > # To use a proxy server, uncomment the next line and set the host > and port. > > # connectionDefaults.setProxyServer("localhost", 8001) > > # These definitions at the top level of the file are evaluated once, > # when the worker process is started. > > connectionDefaults.defaultHeaders > = \ > ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; > en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'), > NVPair('Accept-Encoding', 'gzip,deflate'), > NVPair('Accept-Language', 'en-us,en;q= > 0.5'), > NVPair('Accept-Charset', 'UTF-8,*'), > NVPair('Accept', > 'text/xml,application/xml,application/xhtml+xml,text/html;q= > 0.9,text/plain;q=0.8,image/png,*/*;q=0.5'), ) > > headers0= \ > ( ) > > headers1= \ > > ( NVPair('Referer', 'https://localhost:8443/webtools/control/ > main' ), ) > > headers2= \ > ( NVPair('Referer', ' > https://localhost:8443/webtools/control/checkLogin/main'), ) > > url0 = ' <a href="https://localhost:8443'">https://localhost:8443' > > # Create an HTTPRequest for each request, then replace the > # reference to the HTTPRequest with an instrumented version. > > # You can access the unadorned instance using request101.__target__. > request101 = HTTPRequest(url=url0, headers=headers0) > request101 = Test(101, 'GET /').wrap(request101) > > request102 = HTTPRequest(url=url0, headers=headers0) > > request102 = Test(102, 'GET main').wrap(request102) > > request201 = HTTPRequest(url=url0, headers=headers1) > request201 = Test(201, 'GET main').wrap(request201) > > request301 = HTTPRequest(url=url0, headers=headers2) > > request301 = Test(301, 'POST login').wrap(request301) > > > class TestRunner: > """A TestRunner instance is created for each worker thread.""" > > # A method for each recorded page. > > def page1(self): > """GET / (requests 101-102).""" > > # Expecting 302 'Moved Temporarily' > result = request101.GET('/webtools/') > > grinder.sleep(16) > > request102.GET('/webtools/control/main') > > return result > > def page2(self): > """GET main (request 201).""" > result = request201.GET('/webtools/control/checkLogin/main') > > > return result > > def page3(self): > """POST login (request 301).""" > result = request301.POST('/webtools/control/login', > ( NVPair('USERNAME', 'admin'), > > NVPair('PASSWORD', 'ofbiz'), ), > ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), )) > > return result > > def __call__(self): > """This method is called for every run performed by the worker > thread.""" > > self.page1() # GET / (requests 101-102) > > grinder.sleep(4328) > self.page2() # GET main (request 201) > > grinder.sleep(2109) > self.page3() # POST login (request 301) > > > > def instrumentMethod(test, method_name, c=TestRunner): > """Instrument a method with the given Test.""" > unadorned = getattr(c, method_name) > import new > method = new.instancemethod > (test.wrap(unadorned), None, c) > setattr(c, method_name, method) > > # Replace each method with an instrumented version. > # You can call the unadorned method using self.page1.__target__(). > instrumentMethod(Test(100, 'Page 1'), 'page1') > > instrumentMethod(Test(200, 'Page 2'), 'page2') > instrumentMethod(Test(300, 'Page 3'), 'page3') > > > > Possible Custom Generated Grinder Test Script # The Grinder 3.0-beta30 > # HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM > > from net.grinder.script import Test > from net.grinder.script.Grinder import grinder > from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest > > from HTTPClient import NVPair > import AGrinderTest > > connectionDefaults = HTTPPluginControl.getConnectionDefaults() > httpUtilities = HTTPPluginControl.getHTTPUtilities() > > # To use a proxy server, uncomment the next line and set the host > and port. > > # connectionDefaults.setProxyServer("localhost", 8001) > > # These definitions at the top level of the file are evaluated once, > # when the worker process is started. > > connectionDefaults.defaultHeaders > = \ > ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; > en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'), > NVPair('Accept-Encoding', 'gzip,deflate'), > NVPair('Accept-Language', 'en-us,en;q= > 0.5'), > NVPair('Accept-Charset', 'UTF-8,*'), > NVPair('Accept', > 'text/xml,application/xml,application/xhtml+xml,text/ > html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'), > ) > > headers0= \ > ( ) > > headers1= \ > > ( NVPair('Referer', 'https://localhost:8443/webtools/control/ > main'), ) > > headers2= \ > ( NVPair('Referer', ' > https://localhost:8443/webtools/control/checkLogin/main'), ) > > url0 = '<a href="https://localhost:8443'">https://localhost:8443' > > # Create an HTTPRequest for each request, then replace the > # reference to the HTTPRequest with an instrumented version. > > # You can access the unadorned instance using request101.__target__. > request101 = HTTPRequest(url=url0, headers=headers0) > request101 = Test(101, 'GET /').wrap(request101) > > request102 = HTTPRequest(url=url0, headers=headers0) > > request102 = Test(102, 'GET main').wrap(request102) > > request201 = HTTPRequest(url=url0, headers=headers1) > request201 = Test(201, 'GET main').wrap(request201) > > request301 = HTTPRequest(url=url0, headers=headers2) > > request301 = Test(301, 'POST login').wrap(request301) > > > class TestRunner: > """A TestRunner instance is created for each worker thread.""" > > # A method for each recorded page. > > def page1(self): > """GET / (requests 101-102).""" > > # Expecting 302 'Moved Temporarily' > result = request101.GET('/webtools/') > > grinder.sleep(16) > > request102.GET('/webtools/control/main') > > return result > > def page2(self): > """GET main (request 201).""" > result = request201.GET('/webtools/control/checkLogin/main') > > > return result > > def page3(self): > """POST login (request 301).""" > result = request301.POST('/webtools/control/login', > ( NVPair('USERNAME', 'admin'), > > NVPair('PASSWORD', 'ofbiz'), ), > ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), )) > > return result > > > """This is not working code. Just an idea of how the generated > code would look""" > > def webtoolsWrap(self, overlayMap): > inMap = {} > outMap = {} > self.agrinder.runWrappedTest(page1, inMap, outMap, overlayMap) > > def mainWrap(self, overlayMap): > inMap = {} > outMap = {} > > self.agrinder.runWrappedTest(page2, inMap, outMap, overlayMap) > > def loginWrap(self, overlayMap): > inMap = {'USERNAME':'admin', 'PASSWORD':'ofbiz'} > outMap = {} > self.agrinder.runWrappedTest > (page2, inMap, outMap, overlayMap) > > def __call__(self): > """This method is called for every run performed by the worker > thread.""" > > """I am going to do some handwaving here because I want to get > this doc out today, > > but there would be code here to read in an argument from the > command line and > use it as a file path to read in the "test overlay" XML doc""" > self.agrinder = AGrinderTest.AGrinder > () > testNode = self.agrinder.getOverlayTest("webtools", sys.argv) > self.webtoolsWrap(testNode) # GET / (requests 101-102) > > grinder.sleep(4328) > self.mainWrap(testNode) # GET main (request 201) > > > grinder.sleep(2109) > self.loginWrap(testNode) # POST login (request 301) > > > def instrumentMethod(test, method_name, c=TestRunner): > """Instrument a method with the given Test.""" > > unadorned = getattr(c, method_name) > import new > method = new.instancemethod(test.wrap(unadorned), None, c) > setattr(c, method_name, method) > > # Replace each method with an instrumented version. > > # You can call the unadorned method using self.page1.__target__(). > instrumentMethod(Test(100, 'Page 1'), 'page1') > instrumentMethod(Test(200, 'Page 2'), 'page2') > instrumentMethod(Test(300, 'Page 3'), 'page3') > > > Possible Overlay XML File This example is probably not even close to > what the final test overlay XML script will look like, but I think it > is better to have something to go from. <tests> > <test name="webtools"> > <subtest name="login"> > <input> > <field name="USERNAME">jdoe</field> > <field name="PASSWORD">id9Ed3jk</field> > > </input> > <match> > <field type="regex" op="not">not found</field> > <field type="script" op="true">isSuccess()</field> > > </match> > </subtest> > </test> > </tests> Best Regards, Si [hidden email] |
|
|
Si,
Thanks for the feedback. I debated over the xml language. I think it is important to involve a different level of user. But the safe thing to do would be to create the api that you mention and then wrap it in an xml layer later on if there is demand. -Al On 11/2/06, Si Chen <[hidden email]> wrote: > Al, > > This is great that you're doing this. I think a testing framework is > really what we need right now. > > My suggestion is instead of creating an xml language like you show at > the bottom to make some API methods that could be called in Jython > scripts. I think that might be easier for people to work with and > would require less work over time in maintaining and extending the > xml language. > > On Nov 1, 2006, at 10:49 AM, Al Byers wrote: > > > I am having trouble getting my messages posted to the list. This one > > is taking a long time and I wonder if it is because it has HTML in the > > content and an attachment. I am pasting it in plain test. > > ... > > One of the presentations at the Users Conference will be me talking on > > my experiences with Grinder. With David there, we hope to go far past > > that to discuss strategies for adding testing functionality to OFBiz. > > I have no experience that qualifies me to be an expert in this area - > > just have a need. Rather than just wait for everyone to show up at the > > conference, I felt it would be helpful to try to treat some of the > > subjects ahead of time. Of course, not everything will be decided at > > the conference, so as much input as possible from everyone will be > > needed. Also, if there is anyone with strong Python and/or Grinder > > skills, their help would also be appreciated. > > > > > > OFBiz Testing Initiative > > GOAL > > BACKGROUND > > Problem with Design Documents > > Applicable Features of OFBiz > > Applicable Features of Grinder > > Applicable Features of Extreme Programming > > REQUIREMENTS > > Easy Generation > > Handle Complex Tests > > Adaptable per Application > > Rugged > > Handle Iterations > > Usable at Multiple Proficiency Levels > > Simplify Output Testing > > STRAWMAN PLAN > > Modify Test Script Generation > > Add Test Overlay Config File > > Source Code Control to Handle Iterations > > Appendices > > Standard Generated Grinder Test Script > > Possible Custom Generated Grinder Test Script > > Possible Overlay XML File > > GOAL The goal of the OFBiz Testing Initiative (OFBTI) is to add > > functionality to OFBiz that will streamline the functional testing > > process to the point that it will be cost-effective and beneficial to > > write comprehensive tests. Ideally, the tools could be used by > > non-programmers. > > > > A secondary goal would be to add tools to make design specs easier to > > write and more useful. > > BACKGROUND > > Problem with Design Documents Design documents, though necessary, have > > serious drawbacks. > > Expensive to create - should probably take as much time as the > > implementation. > > > > Never complete enough - always need to go back to client, anyway. > > > > No automation help in implementation and validation - as text > > documents, they must be manually translated into architecture > > documents. > > > > Easy to get out-of-date - while many times they are supposed to be > > "living" documents, in reality, they almost never are. > > > > > > Applicable Features of Extreme Programming Extreme programming > > addresses many of the shortcomings of the more conventional design > > document approach. Instead of a complete design doc or even use case > > up front, all that is required is a "story". The story needs only be > > complete enough to generate the first test and the first test is > > usually just a placeholder. Then in the course of many iterations, the > > functional test is enhanced, not the design document. The tests are > > living and serve the useful purpose of verifying that things have not > > broken as code gets changed. > > > > One of the very useful things about this approach is that when bugs > > are discovered, a test or subtest is added to guard against it ever > > creeping back into the code. > > Applicable Features of OFBiz OFBiz is different from other development > > environments, and it would be good to identify the features that could > > be used advantageously or which must be dealt with. > > > > OFBiz is highly configurable via XML formatted files and it would be > > good to continue that pattern. XML config files, because they have > > associated schema, make it feasible for non-programmers and those not > > intimately familiar with OFBiz to make changes and perform certain > > programming tasks. > > > > OFBiz has a general pattern of offering easily configurable options > > via data parameters, but always offering easy access to the lowest > > level of programming for those needed cases without requiring a huge > > environment adaptation. For instance, the screen widget config files > > allow screens to be created with miminal data and allow the user to be > > prompted via the schema in an XML editor, but if there is a construct > > that the widgets do not handle, the user can just throw in a call to > > an HTML component or a FreeMarker template - they do not need to > > abandon the screen widget framework. > > > > OFBiz is a service oriented framework and there are many tools that > > can be used aid in that process. If a request is processed by a > > service (in lieu of an event) then the input parameters can be > > automatically taken from the HTTP stream and automatically converted > > to the right type. Also, the form widget can build forms from the > > service definition. > > Applicable Features of Grinder There are multiple options for testing > > frameworks, but Grinder offers the following unique advantages. > > A robust test script generator - TCPProxy attempts to assign return > > parameters to variables and reuse them, rather than passing > > literals around. This means that it does not instantly break when keys > > are autogenerated. > > Grinder has convenient customization point - script generation is done > > via an XSLT stylesheet that makes use of Java extensions. This would > > be a natural place to make modifications to suit OFBTI. There are also > > filters for the requests and responses that can be swapped out via > > command line parameters. > > Jython - allows seamless switching between a highly productive > > scripting environment and regular Java code. Because of this feature, > > Jython has a lot of possibilities within OFBiz. > > Integrated functional and jUnit testing - the use of Jython allows for > > the same environment to be used to run HTTP client system tests and > > jUnit functional tests. > > REQUIREMENTS > > Easy Generation The use of a test script generator, such as Grinder's > > TCPProxy, allows test scripts to be created by capturing the input of > > a user at a browser. > > Handle Complex Tests A further enhancement would be to allow the > > creation of scripts by supplying a few data parameters or by easily > > modifying automatically generated scripts. One of the biggest > > drawbacks of end user testing environments is that when something > > changes, the associated test script is usually unusable. A big > > improvement would be to allow tests to be created by chaining together > > subtests. > > Adaptable per Application Many applications will have special > > characteristics that need to be handled specially. In one instance, I > > found that Grinder was using the '$' character to form Python variable > > names (which doesn't work) because that is how the HTTP parameters > > were named. > > Rugged > > Handle Iterations The general idea of XP is that the tests become more > > complex and comprehensive as the code becomes more complete. It would > > probably be a good idea to retain visibility to past test setups - > > though, at this time, I am not sure why. > > Usable at Multiple Proficiency Levels It would be good to have the > > testing environment useable, or at least understandable, by > > non-programmers. JUnit tests would not meet this requirement. Neither > > would Python scripts (though it could come close). For project > > managers, an XML-based configuration environment would be needed. > > Simplify Output Testing The analysis of the HTML pages that are > > returned by Grinder tests is one of the more problematic areas of > > end-user testing. The more common scenarios would need to be handled > > by the XML-based configuration environment mentioned above. The > > foreseeable cases would be matching of form values, existence or > > non-existence anywhere of a phrase or something that can be tested by > > an XPath expression. > > STRAWMAN PLAN What is discussed below is how I see tackling the > > testing problem. I am only interested in what will be supported by the > > community, so it can be changed. > > > > The overall approach is to use as much of Grinder as possible, and > > allow for the generated Grinder test scripts to be overwritten by a > > user generated XML test configuration file. > > Modify Test Script Generation The Grinder TCPProxy program will be > > used, but the XSLT stylesheet and the associated Java extension > > classes will be enhanced. The main change will be that the individual > > request tests that Grinder generates for each round trip to the server > > will be wrapped by code that will check to see if there are data > > overrides to be made coming from the user generated high level test > > config file. Each wrapper will also have a dictionary (ie. map) that > > defines tests to be made on each low level result. These also will be > > modified by the higher level config file. If there is not higher level > > config file, the script as generated by TCPProxy will run with no > > modifications. > > Input Dictionary The input directory will do more than just allow the > > user to supply literal values as test input. There will be helper > > functions for allowing the user to randomly pick values from a list. > > Also there will be helper functions for generating reasonably looking > > addresses. The user will be able to create Python scripts to generated > > special input. See the appendix for samples of what this would look > > like. > > Output Dictionary The output dictionary, which contains the test > > criteria, would have test helper functions of the following sort: > > Literal value > > Regular expression > > Form value > > XPath > > Custom Jython scripts Many of the test methods will be suitable for > > non-programmers, but the use of Jython scripts follows the OFBiz > > pattern of providing simple methods for simple tasks, but making it > > easy to drop down to the level needed to solve the problem. At some > > point (right away?), it will be necessary to allow complex joing of > > tests with AND, OR and NOT operators. > > Add Test Overlay Config File One of the modifications to the OOTB test > > script generated by TCPProxy will be that the script will look for the > > existence a file path as a parameters and use it to overwrite the > > default test script values (ie. the values typed in by the user when > > the script was being generated). The script could have multiple levels > > of complex scripts, but eventually, they must call one of the > > "request" tests. Keep in mind that the base test script could have a > > large number of "request" scripts, but they would not all need to be > > used by the user-defined config file; it could just use a subset. So > > the base test script may not, in fact, be totally generated by one > > session of user interaction with the system under test; it could be > > built up over time as new functionality is added without having to run > > thru the test. The OOTB behavior is to sequentially number the tests, > > but in this case, we may wish to name them with the request name that > > they interact with. > > > > These high level test scripts would need "include" functionality so > > that they could used standard building blocks. > > Source Code Control to Handle Iterations > > Appendices > > Standard Generated Grinder Test Script This is the actual output from > > a TCPProxy session: > > > > # The Grinder 3.0-beta30 > > # HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM > > > > from net.grinder.script import Test > > from net.grinder.script.Grinder import grinder > > from net.grinder.plugin.http > > import HTTPPluginControl, HTTPRequest > > from HTTPClient import NVPair > > connectionDefaults = HTTPPluginControl.getConnectionDefaults() > > httpUtilities = HTTPPluginControl.getHTTPUtilities() > > > > # To use a proxy server, uncomment the next line and set the host > > and port. > > > > # connectionDefaults.setProxyServer("localhost", 8001) > > > > # These definitions at the top level of the file are evaluated once, > > # when the worker process is started. > > > > connectionDefaults.defaultHeaders > > = \ > > ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; > > en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'), > > NVPair('Accept-Encoding', 'gzip,deflate'), > > NVPair('Accept-Language', 'en-us,en;q= > > 0.5'), > > NVPair('Accept-Charset', 'UTF-8,*'), > > NVPair('Accept', > > 'text/xml,application/xml,application/xhtml+xml,text/html;q= > > 0.9,text/plain;q=0.8,image/png,*/*;q=0.5'), ) > > > > headers0= \ > > ( ) > > > > headers1= \ > > > > ( NVPair('Referer', 'https://localhost:8443/webtools/control/ > > main' ), ) > > > > headers2= \ > > ( NVPair('Referer', ' > > https://localhost:8443/webtools/control/checkLogin/main'), ) > > > > url0 = ' <a href="https://localhost:8443'">https://localhost:8443' > > > > # Create an HTTPRequest for each request, then replace the > > # reference to the HTTPRequest with an instrumented version. > > > > # You can access the unadorned instance using request101.__target__. > > request101 = HTTPRequest(url=url0, headers=headers0) > > request101 = Test(101, 'GET /').wrap(request101) > > > > request102 = HTTPRequest(url=url0, headers=headers0) > > > > request102 = Test(102, 'GET main').wrap(request102) > > > > request201 = HTTPRequest(url=url0, headers=headers1) > > request201 = Test(201, 'GET main').wrap(request201) > > > > request301 = HTTPRequest(url=url0, headers=headers2) > > > > request301 = Test(301, 'POST login').wrap(request301) > > > > > > class TestRunner: > > """A TestRunner instance is created for each worker thread.""" > > > > # A method for each recorded page. > > > > def page1(self): > > """GET / (requests 101-102).""" > > > > # Expecting 302 'Moved Temporarily' > > result = request101.GET('/webtools/') > > > > grinder.sleep(16) > > > > request102.GET('/webtools/control/main') > > > > return result > > > > def page2(self): > > """GET main (request 201).""" > > result = request201.GET('/webtools/control/checkLogin/main') > > > > > > return result > > > > def page3(self): > > """POST login (request 301).""" > > result = request301.POST('/webtools/control/login', > > ( NVPair('USERNAME', 'admin'), > > > > NVPair('PASSWORD', 'ofbiz'), ), > > ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), )) > > > > return result > > > > def __call__(self): > > """This method is called for every run performed by the worker > > thread.""" > > > > self.page1() # GET / (requests 101-102) > > > > grinder.sleep(4328) > > self.page2() # GET main (request 201) > > > > grinder.sleep(2109) > > self.page3() # POST login (request 301) > > > > > > > > def instrumentMethod(test, method_name, c=TestRunner): > > """Instrument a method with the given Test.""" > > unadorned = getattr(c, method_name) > > import new > > method = new.instancemethod > > (test.wrap(unadorned), None, c) > > setattr(c, method_name, method) > > > > # Replace each method with an instrumented version. > > # You can call the unadorned method using self.page1.__target__(). > > instrumentMethod(Test(100, 'Page 1'), 'page1') > > > > instrumentMethod(Test(200, 'Page 2'), 'page2') > > instrumentMethod(Test(300, 'Page 3'), 'page3') > > > > > > > > Possible Custom Generated Grinder Test Script # The Grinder 3.0-beta30 > > # HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM > > > > from net.grinder.script import Test > > from net.grinder.script.Grinder import grinder > > from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest > > > > from HTTPClient import NVPair > > import AGrinderTest > > > > connectionDefaults = HTTPPluginControl.getConnectionDefaults() > > httpUtilities = HTTPPluginControl.getHTTPUtilities() > > > > # To use a proxy server, uncomment the next line and set the host > > and port. > > > > # connectionDefaults.setProxyServer("localhost", 8001) > > > > # These definitions at the top level of the file are evaluated once, > > # when the worker process is started. > > > > connectionDefaults.defaultHeaders > > = \ > > ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; > > en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'), > > NVPair('Accept-Encoding', 'gzip,deflate'), > > NVPair('Accept-Language', 'en-us,en;q= > > 0.5'), > > NVPair('Accept-Charset', 'UTF-8,*'), > > NVPair('Accept', > > 'text/xml,application/xml,application/xhtml+xml,text/ > > html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'), > > ) > > > > headers0= \ > > ( ) > > > > headers1= \ > > > > ( NVPair('Referer', 'https://localhost:8443/webtools/control/ > > main'), ) > > > > headers2= \ > > ( NVPair('Referer', ' > > https://localhost:8443/webtools/control/checkLogin/main'), ) > > > > url0 = '<a href="https://localhost:8443'">https://localhost:8443' > > > > # Create an HTTPRequest for each request, then replace the > > # reference to the HTTPRequest with an instrumented version. > > > > # You can access the unadorned instance using request101.__target__. > > request101 = HTTPRequest(url=url0, headers=headers0) > > request101 = Test(101, 'GET /').wrap(request101) > > > > request102 = HTTPRequest(url=url0, headers=headers0) > > > > request102 = Test(102, 'GET main').wrap(request102) > > > > request201 = HTTPRequest(url=url0, headers=headers1) > > request201 = Test(201, 'GET main').wrap(request201) > > > > request301 = HTTPRequest(url=url0, headers=headers2) > > > > request301 = Test(301, 'POST login').wrap(request301) > > > > > > class TestRunner: > > """A TestRunner instance is created for each worker thread.""" > > > > # A method for each recorded page. > > > > def page1(self): > > """GET / (requests 101-102).""" > > > > # Expecting 302 'Moved Temporarily' > > result = request101.GET('/webtools/') > > > > grinder.sleep(16) > > > > request102.GET('/webtools/control/main') > > > > return result > > > > def page2(self): > > """GET main (request 201).""" > > result = request201.GET('/webtools/control/checkLogin/main') > > > > > > return result > > > > def page3(self): > > """POST login (request 301).""" > > result = request301.POST('/webtools/control/login', > > ( NVPair('USERNAME', 'admin'), > > > > NVPair('PASSWORD', 'ofbiz'), ), > > ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), )) > > > > return result > > > > > > """This is not working code. Just an idea of how the generated > > code would look""" > > > > def webtoolsWrap(self, overlayMap): > > inMap = {} > > outMap = {} > > self.agrinder.runWrappedTest(page1, inMap, outMap, overlayMap) > > > > def mainWrap(self, overlayMap): > > inMap = {} > > outMap = {} > > > > self.agrinder.runWrappedTest(page2, inMap, outMap, overlayMap) > > > > def loginWrap(self, overlayMap): > > inMap = {'USERNAME':'admin', 'PASSWORD':'ofbiz'} > > outMap = {} > > self.agrinder.runWrappedTest > > (page2, inMap, outMap, overlayMap) > > > > def __call__(self): > > """This method is called for every run performed by the worker > > thread.""" > > > > """I am going to do some handwaving here because I want to get > > this doc out today, > > > > but there would be code here to read in an argument from the > > command line and > > use it as a file path to read in the "test overlay" XML doc""" > > self.agrinder = AGrinderTest.AGrinder > > () > > testNode = self.agrinder.getOverlayTest("webtools", sys.argv) > > self.webtoolsWrap(testNode) # GET / (requests 101-102) > > > > grinder.sleep(4328) > > self.mainWrap(testNode) # GET main (request 201) > > > > > > grinder.sleep(2109) > > self.loginWrap(testNode) # POST login (request 301) > > > > > > def instrumentMethod(test, method_name, c=TestRunner): > > """Instrument a method with the given Test.""" > > > > unadorned = getattr(c, method_name) > > import new > > method = new.instancemethod(test.wrap(unadorned), None, c) > > setattr(c, method_name, method) > > > > # Replace each method with an instrumented version. > > > > # You can call the unadorned method using self.page1.__target__(). > > instrumentMethod(Test(100, 'Page 1'), 'page1') > > instrumentMethod(Test(200, 'Page 2'), 'page2') > > instrumentMethod(Test(300, 'Page 3'), 'page3') > > > > > > Possible Overlay XML File This example is probably not even close to > > what the final test overlay XML script will look like, but I think it > > is better to have something to go from. <tests> > > <test name="webtools"> > > <subtest name="login"> > > <input> > > <field name="USERNAME">jdoe</field> > > <field name="PASSWORD">id9Ed3jk</field> > > > > </input> > > <match> > > <field type="regex" op="not">not found</field> > > <field type="script" op="true">isSuccess()</field> > > > > </match> > > </subtest> > > </test> > > </tests> > > Best Regards, > > Si > [hidden email] > > > > > |
|
|
Yes, that sounds like a good idea. We should have an abstracted api
layer and then be able to call it either from jython or use the xml wrapper, whichever one is easier for the job. On Nov 2, 2006, at 10:31 AM, Al Byers wrote: > e Best Regards, Si [hidden email] |
Administrator
|
In reply to this post by byersa
Yes I agree with Al, having both is surely better (of course more work also...)
Jacques From: "Al Byers" <[hidden email]> > Si, > > Thanks for the feedback. I debated over the xml language. I think it > is important to involve a different level of user. But the safe thing > to do would be to create the api that you mention and then wrap it in > an xml layer later on if there is demand. > > -Al > > On 11/2/06, Si Chen <[hidden email]> wrote: > > Al, > > > > This is great that you're doing this. I think a testing framework is > > really what we need right now. > > > > My suggestion is instead of creating an xml language like you show at > > the bottom to make some API methods that could be called in Jython > > scripts. I think that might be easier for people to work with and > > would require less work over time in maintaining and extending the > > xml language. > > > > On Nov 1, 2006, at 10:49 AM, Al Byers wrote: > > > > > I am having trouble getting my messages posted to the list. This one > > > is taking a long time and I wonder if it is because it has HTML in the > > > content and an attachment. I am pasting it in plain test. > > > ... > > > One of the presentations at the Users Conference will be me talking on > > > my experiences with Grinder. With David there, we hope to go far past > > > that to discuss strategies for adding testing functionality to OFBiz. > > > I have no experience that qualifies me to be an expert in this area - > > > just have a need. Rather than just wait for everyone to show up at the > > > conference, I felt it would be helpful to try to treat some of the > > > subjects ahead of time. Of course, not everything will be decided at > > > the conference, so as much input as possible from everyone will be > > > needed. Also, if there is anyone with strong Python and/or Grinder > > > skills, their help would also be appreciated. > > > > > > > > > OFBiz Testing Initiative > > > GOAL > > > BACKGROUND > > > Problem with Design Documents > > > Applicable Features of OFBiz > > > Applicable Features of Grinder > > > Applicable Features of Extreme Programming > > > REQUIREMENTS > > > Easy Generation > > > Handle Complex Tests > > > Adaptable per Application > > > Rugged > > > Handle Iterations > > > Usable at Multiple Proficiency Levels > > > Simplify Output Testing > > > STRAWMAN PLAN > > > Modify Test Script Generation > > > Add Test Overlay Config File > > > Source Code Control to Handle Iterations > > > Appendices > > > Standard Generated Grinder Test Script > > > Possible Custom Generated Grinder Test Script > > > Possible Overlay XML File > > > GOAL The goal of the OFBiz Testing Initiative (OFBTI) is to add > > > functionality to OFBiz that will streamline the functional testing > > > process to the point that it will be cost-effective and beneficial to > > > write comprehensive tests. Ideally, the tools could be used by > > > non-programmers. > > > > > > A secondary goal would be to add tools to make design specs easier to > > > write and more useful. > > > BACKGROUND > > > Problem with Design Documents Design documents, though necessary, have > > > serious drawbacks. > > > Expensive to create - should probably take as much time as the > > > implementation. > > > > > > Never complete enough - always need to go back to client, anyway. > > > > > > No automation help in implementation and validation - as text > > > documents, they must be manually translated into architecture > > > documents. > > > > > > Easy to get out-of-date - while many times they are supposed to be > > > "living" documents, in reality, they almost never are. > > > > > > > > > Applicable Features of Extreme Programming Extreme programming > > > addresses many of the shortcomings of the more conventional design > > > document approach. Instead of a complete design doc or even use case > > > up front, all that is required is a "story". The story needs only be > > > complete enough to generate the first test and the first test is > > > usually just a placeholder. Then in the course of many iterations, the > > > functional test is enhanced, not the design document. The tests are > > > living and serve the useful purpose of verifying that things have not > > > broken as code gets changed. > > > > > > One of the very useful things about this approach is that when bugs > > > are discovered, a test or subtest is added to guard against it ever > > > creeping back into the code. > > > Applicable Features of OFBiz OFBiz is different from other development > > > environments, and it would be good to identify the features that could > > > be used advantageously or which must be dealt with. > > > > > > OFBiz is highly configurable via XML formatted files and it would be > > > good to continue that pattern. XML config files, because they have > > > associated schema, make it feasible for non-programmers and those not > > > intimately familiar with OFBiz to make changes and perform certain > > > programming tasks. > > > > > > OFBiz has a general pattern of offering easily configurable options > > > via data parameters, but always offering easy access to the lowest > > > level of programming for those needed cases without requiring a huge > > > environment adaptation. For instance, the screen widget config files > > > allow screens to be created with miminal data and allow the user to be > > > prompted via the schema in an XML editor, but if there is a construct > > > that the widgets do not handle, the user can just throw in a call to > > > an HTML component or a FreeMarker template - they do not need to > > > abandon the screen widget framework. > > > > > > OFBiz is a service oriented framework and there are many tools that > > > can be used aid in that process. If a request is processed by a > > > service (in lieu of an event) then the input parameters can be > > > automatically taken from the HTTP stream and automatically converted > > > to the right type. Also, the form widget can build forms from the > > > service definition. > > > Applicable Features of Grinder There are multiple options for testing > > > frameworks, but Grinder offers the following unique advantages. > > > A robust test script generator - TCPProxy attempts to assign return > > > parameters to variables and reuse them, rather than passing > > > literals around. This means that it does not instantly break when keys > > > are autogenerated. > > > Grinder has convenient customization point - script generation is done > > > via an XSLT stylesheet that makes use of Java extensions. This would > > > be a natural place to make modifications to suit OFBTI. There are also > > > filters for the requests and responses that can be swapped out via > > > command line parameters. > > > Jython - allows seamless switching between a highly productive > > > scripting environment and regular Java code. Because of this feature, > > > Jython has a lot of possibilities within OFBiz. > > > Integrated functional and jUnit testing - the use of Jython allows for > > > the same environment to be used to run HTTP client system tests and > > > jUnit functional tests. > > > REQUIREMENTS > > > Easy Generation The use of a test script generator, such as Grinder's > > > TCPProxy, allows test scripts to be created by capturing the input of > > > a user at a browser. > > > Handle Complex Tests A further enhancement would be to allow the > > > creation of scripts by supplying a few data parameters or by easily > > > modifying automatically generated scripts. One of the biggest > > > drawbacks of end user testing environments is that when something > > > changes, the associated test script is usually unusable. A big > > > improvement would be to allow tests to be created by chaining together > > > subtests. > > > Adaptable per Application Many applications will have special > > > characteristics that need to be handled specially. In one instance, I > > > found that Grinder was using the '$' character to form Python variable > > > names (which doesn't work) because that is how the HTTP parameters > > > were named. > > > Rugged > > > Handle Iterations The general idea of XP is that the tests become more > > > complex and comprehensive as the code becomes more complete. It would > > > probably be a good idea to retain visibility to past test setups - > > > though, at this time, I am not sure why. > > > Usable at Multiple Proficiency Levels It would be good to have the > > > testing environment useable, or at least understandable, by > > > non-programmers. JUnit tests would not meet this requirement. Neither > > > would Python scripts (though it could come close). For project > > > managers, an XML-based configuration environment would be needed. > > > Simplify Output Testing The analysis of the HTML pages that are > > > returned by Grinder tests is one of the more problematic areas of > > > end-user testing. The more common scenarios would need to be handled > > > by the XML-based configuration environment mentioned above. The > > > foreseeable cases would be matching of form values, existence or > > > non-existence anywhere of a phrase or something that can be tested by > > > an XPath expression. > > > STRAWMAN PLAN What is discussed below is how I see tackling the > > > testing problem. I am only interested in what will be supported by the > > > community, so it can be changed. > > > > > > The overall approach is to use as much of Grinder as possible, and > > > allow for the generated Grinder test scripts to be overwritten by a > > > user generated XML test configuration file. > > > Modify Test Script Generation The Grinder TCPProxy program will be > > > used, but the XSLT stylesheet and the associated Java extension > > > classes will be enhanced. The main change will be that the individual > > > request tests that Grinder generates for each round trip to the server > > > will be wrapped by code that will check to see if there are data > > > overrides to be made coming from the user generated high level test > > > config file. Each wrapper will also have a dictionary (ie. map) that > > > defines tests to be made on each low level result. These also will be > > > modified by the higher level config file. If there is not higher level > > > config file, the script as generated by TCPProxy will run with no > > > modifications. > > > Input Dictionary The input directory will do more than just allow the > > > user to supply literal values as test input. There will be helper > > > functions for allowing the user to randomly pick values from a list. > > > Also there will be helper functions for generating reasonably looking > > > addresses. The user will be able to create Python scripts to generated > > > special input. See the appendix for samples of what this would look > > > like. > > > Output Dictionary The output dictionary, which contains the test > > > criteria, would have test helper functions of the following sort: > > > Literal value > > > Regular expression > > > Form value > > > XPath > > > Custom Jython scripts Many of the test methods will be suitable for > > > non-programmers, but the use of Jython scripts follows the OFBiz > > > pattern of providing simple methods for simple tasks, but making it > > > easy to drop down to the level needed to solve the problem. At some > > > point (right away?), it will be necessary to allow complex joing of > > > tests with AND, OR and NOT operators. > > > Add Test Overlay Config File One of the modifications to the OOTB test > > > script generated by TCPProxy will be that the script will look for the > > > existence a file path as a parameters and use it to overwrite the > > > default test script values (ie. the values typed in by the user when > > > the script was being generated). The script could have multiple levels > > > of complex scripts, but eventually, they must call one of the > > > "request" tests. Keep in mind that the base test script could have a > > > large number of "request" scripts, but they would not all need to be > > > used by the user-defined config file; it could just use a subset. So > > > the base test script may not, in fact, be totally generated by one > > > session of user interaction with the system under test; it could be > > > built up over time as new functionality is added without having to run > > > thru the test. The OOTB behavior is to sequentially number the tests, > > > but in this case, we may wish to name them with the request name that > > > they interact with. > > > > > > These high level test scripts would need "include" functionality so > > > that they could used standard building blocks. > > > Source Code Control to Handle Iterations > > > Appendices > > > Standard Generated Grinder Test Script This is the actual output from > > > a TCPProxy session: > > > > > > # The Grinder 3.0-beta30 > > > # HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM > > > > > > from net.grinder.script import Test > > > from net.grinder.script.Grinder import grinder > > > from net.grinder.plugin.http > > > import HTTPPluginControl, HTTPRequest > > > from HTTPClient import NVPair > > > connectionDefaults = HTTPPluginControl.getConnectionDefaults() > > > httpUtilities = HTTPPluginControl.getHTTPUtilities() > > > > > > # To use a proxy server, uncomment the next line and set the host > > > and port. > > > > > > # connectionDefaults.setProxyServer("localhost", 8001) > > > > > > # These definitions at the top level of the file are evaluated once, > > > # when the worker process is started. > > > > > > connectionDefaults.defaultHeaders > > > = \ > > > ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; > > > en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'), > > > NVPair('Accept-Encoding', 'gzip,deflate'), > > > NVPair('Accept-Language', 'en-us,en;q= > > > 0.5'), > > > NVPair('Accept-Charset', 'UTF-8,*'), > > > NVPair('Accept', > > > 'text/xml,application/xml,application/xhtml+xml,text/html;q= > > > 0.9,text/plain;q=0.8,image/png,*/*;q=0.5'), ) > > > > > > headers0= \ > > > ( ) > > > > > > headers1= \ > > > > > > ( NVPair('Referer', 'https://localhost:8443/webtools/control/ > > > main' ), ) > > > > > > headers2= \ > > > ( NVPair('Referer', ' > > > https://localhost:8443/webtools/control/checkLogin/main'), ) > > > > > > url0 = ' <a href="https://localhost:8443'">https://localhost:8443' > > > > > > # Create an HTTPRequest for each request, then replace the > > > # reference to the HTTPRequest with an instrumented version. > > > > > > # You can access the unadorned instance using request101.__target__. > > > request101 = HTTPRequest(url=url0, headers=headers0) > > > request101 = Test(101, 'GET /').wrap(request101) > > > > > > request102 = HTTPRequest(url=url0, headers=headers0) > > > > > > request102 = Test(102, 'GET main').wrap(request102) > > > > > > request201 = HTTPRequest(url=url0, headers=headers1) > > > request201 = Test(201, 'GET main').wrap(request201) > > > > > > request301 = HTTPRequest(url=url0, headers=headers2) > > > > > > request301 = Test(301, 'POST login').wrap(request301) > > > > > > > > > class TestRunner: > > > """A TestRunner instance is created for each worker thread.""" > > > > > > # A method for each recorded page. > > > > > > def page1(self): > > > """GET / (requests 101-102).""" > > > > > > # Expecting 302 'Moved Temporarily' > > > result = request101.GET('/webtools/') > > > > > > grinder.sleep(16) > > > > > > request102.GET('/webtools/control/main') > > > > > > return result > > > > > > def page2(self): > > > """GET main (request 201).""" > > > result = request201.GET('/webtools/control/checkLogin/main') > > > > > > > > > return result > > > > > > def page3(self): > > > """POST login (request 301).""" > > > result = request301.POST('/webtools/control/login', > > > ( NVPair('USERNAME', 'admin'), > > > > > > NVPair('PASSWORD', 'ofbiz'), ), > > > ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), )) > > > > > > return result > > > > > > def __call__(self): > > > """This method is called for every run performed by the worker > > > thread.""" > > > > > > self.page1() # GET / (requests 101-102) > > > > > > grinder.sleep(4328) > > > self.page2() # GET main (request 201) > > > > > > grinder.sleep(2109) > > > self.page3() # POST login (request 301) > > > > > > > > > > > > def instrumentMethod(test, method_name, c=TestRunner): > > > """Instrument a method with the given Test.""" > > > unadorned = getattr(c, method_name) > > > import new > > > method = new.instancemethod > > > (test.wrap(unadorned), None, c) > > > setattr(c, method_name, method) > > > > > > # Replace each method with an instrumented version. > > > # You can call the unadorned method using self.page1.__target__(). > > > instrumentMethod(Test(100, 'Page 1'), 'page1') > > > > > > instrumentMethod(Test(200, 'Page 2'), 'page2') > > > instrumentMethod(Test(300, 'Page 3'), 'page3') > > > > > > > > > > > > Possible Custom Generated Grinder Test Script # The Grinder 3.0-beta30 > > > # HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM > > > > > > from net.grinder.script import Test > > > from net.grinder.script.Grinder import grinder > > > from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest > > > > > > from HTTPClient import NVPair > > > import AGrinderTest > > > > > > connectionDefaults = HTTPPluginControl.getConnectionDefaults() > > > httpUtilities = HTTPPluginControl.getHTTPUtilities() > > > > > > # To use a proxy server, uncomment the next line and set the host > > > and port. > > > > > > # connectionDefaults.setProxyServer("localhost", 8001) > > > > > > # These definitions at the top level of the file are evaluated once, > > > # when the worker process is started. > > > > > > connectionDefaults.defaultHeaders > > > = \ > > > ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; > > > en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'), > > > NVPair('Accept-Encoding', 'gzip,deflate'), > > > NVPair('Accept-Language', 'en-us,en;q= > > > 0.5'), > > > NVPair('Accept-Charset', 'UTF-8,*'), > > > NVPair('Accept', > > > 'text/xml,application/xml,application/xhtml+xml,text/ > > > html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'), > > > ) > > > > > > headers0= \ > > > ( ) > > > > > > headers1= \ > > > > > > ( NVPair('Referer', 'https://localhost:8443/webtools/control/ > > > main'), ) > > > > > > headers2= \ > > > ( NVPair('Referer', ' > > > https://localhost:8443/webtools/control/checkLogin/main'), ) > > > > > > url0 = '<a href="https://localhost:8443'">https://localhost:8443' > > > > > > # Create an HTTPRequest for each request, then replace the > > > # reference to the HTTPRequest with an instrumented version. > > > > > > # You can access the unadorned instance using request101.__target__. > > > request101 = HTTPRequest(url=url0, headers=headers0) > > > request101 = Test(101, 'GET /').wrap(request101) > > > > > > request102 = HTTPRequest(url=url0, headers=headers0) > > > > > > request102 = Test(102, 'GET main').wrap(request102) > > > > > > request201 = HTTPRequest(url=url0, headers=headers1) > > > request201 = Test(201, 'GET main').wrap(request201) > > > > > > request301 = HTTPRequest(url=url0, headers=headers2) > > > > > > request301 = Test(301, 'POST login').wrap(request301) > > > > > > > > > class TestRunner: > > > """A TestRunner instance is created for each worker thread.""" > > > > > > # A method for each recorded page. > > > > > > def page1(self): > > > """GET / (requests 101-102).""" > > > > > > # Expecting 302 'Moved Temporarily' > > > result = request101.GET('/webtools/') > > > > > > grinder.sleep(16) > > > > > > request102.GET('/webtools/control/main') > > > > > > return result > > > > > > def page2(self): > > > """GET main (request 201).""" > > > result = request201.GET('/webtools/control/checkLogin/main') > > > > > > > > > return result > > > > > > def page3(self): > > > """POST login (request 301).""" > > > result = request301.POST('/webtools/control/login', > > > ( NVPair('USERNAME', 'admin'), > > > > > > NVPair('PASSWORD', 'ofbiz'), ), > > > ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), )) > > > > > > return result > > > > > > > > > """This is not working code. Just an idea of how the generated > > > code would look""" > > > > > > def webtoolsWrap(self, overlayMap): > > > inMap = {} > > > outMap = {} > > > self.agrinder.runWrappedTest(page1, inMap, outMap, overlayMap) > > > > > > def mainWrap(self, overlayMap): > > > inMap = {} > > > outMap = {} > > > > > > self.agrinder.runWrappedTest(page2, inMap, outMap, overlayMap) > > > > > > def loginWrap(self, overlayMap): > > > inMap = {'USERNAME':'admin', 'PASSWORD':'ofbiz'} > > > outMap = {} > > > self.agrinder.runWrappedTest > > > (page2, inMap, outMap, overlayMap) > > > > > > def __call__(self): > > > """This method is called for every run performed by the worker > > > thread.""" > > > > > > """I am going to do some handwaving here because I want to get > > > this doc out today, > > > > > > but there would be code here to read in an argument from the > > > command line and > > > use it as a file path to read in the "test overlay" XML doc""" > > > self.agrinder = AGrinderTest.AGrinder > > > () > > > testNode = self.agrinder.getOverlayTest("webtools", sys.argv) > > > self.webtoolsWrap(testNode) # GET / (requests 101-102) > > > > > > grinder.sleep(4328) > > > self.mainWrap(testNode) # GET main (request 201) > > > > > > > > > grinder.sleep(2109) > > > self.loginWrap(testNode) # POST login (request 301) > > > > > > > > > def instrumentMethod(test, method_name, c=TestRunner): > > > """Instrument a method with the given Test.""" > > > > > > unadorned = getattr(c, method_name) > > > import new > > > method = new.instancemethod(test.wrap(unadorned), None, c) > > > setattr(c, method_name, method) > > > > > > # Replace each method with an instrumented version. > > > > > > # You can call the unadorned method using self.page1.__target__(). > > > instrumentMethod(Test(100, 'Page 1'), 'page1') > > > instrumentMethod(Test(200, 'Page 2'), 'page2') > > > instrumentMethod(Test(300, 'Page 3'), 'page3') > > > > > > > > > Possible Overlay XML File This example is probably not even close to > > > what the final test overlay XML script will look like, but I think it > > > is better to have something to go from. <tests> > > > <test name="webtools"> > > > <subtest name="login"> > > > <input> > > > <field name="USERNAME">jdoe</field> > > > <field name="PASSWORD">id9Ed3jk</field> > > > > > > </input> > > > <match> > > > <field type="regex" op="not">not found</field> > > > <field type="script" op="true">isSuccess()</field> > > > > > > </match> > > > </subtest> > > > </test> > > > </tests> > > > > Best Regards, > > > > Si > > [hidden email] |
|
|
In reply to this post by byersa
Maybe the data files could be simple property files with a specific
naming convention. Then the jython script would just refer to a specific resource and the data would be loaded and executed. The data files could contains expected results and validation data. The thing with XML is if it is too complex a tester won't create the data files. Brett On 11/2/06, Al Byers <[hidden email]> wrote: > Si, > > Thanks for the feedback. I debated over the xml language. I think it > is important to involve a different level of user. But the safe thing > to do would be to create the api that you mention and then wrap it in > an xml layer later on if there is demand. > > -Al > > On 11/2/06, Si Chen <[hidden email]> wrote: > > Al, > > > > This is great that you're doing this. I think a testing framework is > > really what we need right now. > > > > My suggestion is instead of creating an xml language like you show at > > the bottom to make some API methods that could be called in Jython > > scripts. I think that might be easier for people to work with and > > would require less work over time in maintaining and extending the > > xml language. > > > > On Nov 1, 2006, at 10:49 AM, Al Byers wrote: > > > > > I am having trouble getting my messages posted to the list. This one > > > is taking a long time and I wonder if it is because it has HTML in the > > > content and an attachment. I am pasting it in plain test. > > > ... > > > One of the presentations at the Users Conference will be me talking on > > > my experiences with Grinder. With David there, we hope to go far past > > > that to discuss strategies for adding testing functionality to OFBiz. > > > I have no experience that qualifies me to be an expert in this area - > > > just have a need. Rather than just wait for everyone to show up at the > > > conference, I felt it would be helpful to try to treat some of the > > > subjects ahead of time. Of course, not everything will be decided at > > > the conference, so as much input as possible from everyone will be > > > needed. Also, if there is anyone with strong Python and/or Grinder > > > skills, their help would also be appreciated. > > > > > > > > > OFBiz Testing Initiative > > > GOAL > > > BACKGROUND > > > Problem with Design Documents > > > Applicable Features of OFBiz > > > Applicable Features of Grinder > > > Applicable Features of Extreme Programming > > > REQUIREMENTS > > > Easy Generation > > > Handle Complex Tests > > > Adaptable per Application > > > Rugged > > > Handle Iterations > > > Usable at Multiple Proficiency Levels > > > Simplify Output Testing > > > STRAWMAN PLAN > > > Modify Test Script Generation > > > Add Test Overlay Config File > > > Source Code Control to Handle Iterations > > > Appendices > > > Standard Generated Grinder Test Script > > > Possible Custom Generated Grinder Test Script > > > Possible Overlay XML File > > > GOAL The goal of the OFBiz Testing Initiative (OFBTI) is to add > > > functionality to OFBiz that will streamline the functional testing > > > process to the point that it will be cost-effective and beneficial to > > > write comprehensive tests. Ideally, the tools could be used by > > > non-programmers. > > > > > > A secondary goal would be to add tools to make design specs easier to > > > write and more useful. > > > BACKGROUND > > > Problem with Design Documents Design documents, though necessary, have > > > serious drawbacks. > > > Expensive to create - should probably take as much time as the > > > implementation. > > > > > > Never complete enough - always need to go back to client, anyway. > > > > > > No automation help in implementation and validation - as text > > > documents, they must be manually translated into architecture > > > documents. > > > > > > Easy to get out-of-date - while many times they are supposed to be > > > "living" documents, in reality, they almost never are. > > > > > > > > > Applicable Features of Extreme Programming Extreme programming > > > addresses many of the shortcomings of the more conventional design > > > document approach. Instead of a complete design doc or even use case > > > up front, all that is required is a "story". The story needs only be > > > complete enough to generate the first test and the first test is > > > usually just a placeholder. Then in the course of many iterations, the > > > functional test is enhanced, not the design document. The tests are > > > living and serve the useful purpose of verifying that things have not > > > broken as code gets changed. > > > > > > One of the very useful things about this approach is that when bugs > > > are discovered, a test or subtest is added to guard against it ever > > > creeping back into the code. > > > Applicable Features of OFBiz OFBiz is different from other development > > > environments, and it would be good to identify the features that could > > > be used advantageously or which must be dealt with. > > > > > > OFBiz is highly configurable via XML formatted files and it would be > > > good to continue that pattern. XML config files, because they have > > > associated schema, make it feasible for non-programmers and those not > > > intimately familiar with OFBiz to make changes and perform certain > > > programming tasks. > > > > > > OFBiz has a general pattern of offering easily configurable options > > > via data parameters, but always offering easy access to the lowest > > > level of programming for those needed cases without requiring a huge > > > environment adaptation. For instance, the screen widget config files > > > allow screens to be created with miminal data and allow the user to be > > > prompted via the schema in an XML editor, but if there is a construct > > > that the widgets do not handle, the user can just throw in a call to > > > an HTML component or a FreeMarker template - they do not need to > > > abandon the screen widget framework. > > > > > > OFBiz is a service oriented framework and there are many tools that > > > can be used aid in that process. If a request is processed by a > > > service (in lieu of an event) then the input parameters can be > > > automatically taken from the HTTP stream and automatically converted > > > to the right type. Also, the form widget can build forms from the > > > service definition. > > > Applicable Features of Grinder There are multiple options for testing > > > frameworks, but Grinder offers the following unique advantages. > > > A robust test script generator - TCPProxy attempts to assign return > > > parameters to variables and reuse them, rather than passing > > > literals around. This means that it does not instantly break when keys > > > are autogenerated. > > > Grinder has convenient customization point - script generation is done > > > via an XSLT stylesheet that makes use of Java extensions. This would > > > be a natural place to make modifications to suit OFBTI. There are also > > > filters for the requests and responses that can be swapped out via > > > command line parameters. > > > Jython - allows seamless switching between a highly productive > > > scripting environment and regular Java code. Because of this feature, > > > Jython has a lot of possibilities within OFBiz. > > > Integrated functional and jUnit testing - the use of Jython allows for > > > the same environment to be used to run HTTP client system tests and > > > jUnit functional tests. > > > REQUIREMENTS > > > Easy Generation The use of a test script generator, such as Grinder's > > > TCPProxy, allows test scripts to be created by capturing the input of > > > a user at a browser. > > > Handle Complex Tests A further enhancement would be to allow the > > > creation of scripts by supplying a few data parameters or by easily > > > modifying automatically generated scripts. One of the biggest > > > drawbacks of end user testing environments is that when something > > > changes, the associated test script is usually unusable. A big > > > improvement would be to allow tests to be created by chaining together > > > subtests. > > > Adaptable per Application Many applications will have special > > > characteristics that need to be handled specially. In one instance, I > > > found that Grinder was using the '$' character to form Python variable > > > names (which doesn't work) because that is how the HTTP parameters > > > were named. > > > Rugged > > > Handle Iterations The general idea of XP is that the tests become more > > > complex and comprehensive as the code becomes more complete. It would > > > probably be a good idea to retain visibility to past test setups - > > > though, at this time, I am not sure why. > > > Usable at Multiple Proficiency Levels It would be good to have the > > > testing environment useable, or at least understandable, by > > > non-programmers. JUnit tests would not meet this requirement. Neither > > > would Python scripts (though it could come close). For project > > > managers, an XML-based configuration environment would be needed. > > > Simplify Output Testing The analysis of the HTML pages that are > > > returned by Grinder tests is one of the more problematic areas of > > > end-user testing. The more common scenarios would need to be handled > > > by the XML-based configuration environment mentioned above. The > > > foreseeable cases would be matching of form values, existence or > > > non-existence anywhere of a phrase or something that can be tested by > > > an XPath expression. > > > STRAWMAN PLAN What is discussed below is how I see tackling the > > > testing problem. I am only interested in what will be supported by the > > > community, so it can be changed. > > > > > > The overall approach is to use as much of Grinder as possible, and > > > allow for the generated Grinder test scripts to be overwritten by a > > > user generated XML test configuration file. > > > Modify Test Script Generation The Grinder TCPProxy program will be > > > used, but the XSLT stylesheet and the associated Java extension > > > classes will be enhanced. The main change will be that the individual > > > request tests that Grinder generates for each round trip to the server > > > will be wrapped by code that will check to see if there are data > > > overrides to be made coming from the user generated high level test > > > config file. Each wrapper will also have a dictionary (ie. map) that > > > defines tests to be made on each low level result. These also will be > > > modified by the higher level config file. If there is not higher level > > > config file, the script as generated by TCPProxy will run with no > > > modifications. > > > Input Dictionary The input directory will do more than just allow the > > > user to supply literal values as test input. There will be helper > > > functions for allowing the user to randomly pick values from a list. > > > Also there will be helper functions for generating reasonably looking > > > addresses. The user will be able to create Python scripts to generated > > > special input. See the appendix for samples of what this would look > > > like. > > > Output Dictionary The output dictionary, which contains the test > > > criteria, would have test helper functions of the following sort: > > > Literal value > > > Regular expression > > > Form value > > > XPath > > > Custom Jython scripts Many of the test methods will be suitable for > > > non-programmers, but the use of Jython scripts follows the OFBiz > > > pattern of providing simple methods for simple tasks, but making it > > > easy to drop down to the level needed to solve the problem. At some > > > point (right away?), it will be necessary to allow complex joing of > > > tests with AND, OR and NOT operators. > > > Add Test Overlay Config File One of the modifications to the OOTB test > > > script generated by TCPProxy will be that the script will look for the > > > existence a file path as a parameters and use it to overwrite the > > > default test script values (ie. the values typed in by the user when > > > the script was being generated). The script could have multiple levels > > > of complex scripts, but eventually, they must call one of the > > > "request" tests. Keep in mind that the base test script could have a > > > large number of "request" scripts, but they would not all need to be > > > used by the user-defined config file; it could just use a subset. So > > > the base test script may not, in fact, be totally generated by one > > > session of user interaction with the system under test; it could be > > > built up over time as new functionality is added without having to run > > > thru the test. The OOTB behavior is to sequentially number the tests, > > > but in this case, we may wish to name them with the request name that > > > they interact with. > > > > > > These high level test scripts would need "include" functionality so > > > that they could used standard building blocks. > > > Source Code Control to Handle Iterations > > > Appendices > > > Standard Generated Grinder Test Script This is the actual output from > > > a TCPProxy session: > > > > > > # The Grinder 3.0-beta30 > > > # HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM > > > > > > from net.grinder.script import Test > > > from net.grinder.script.Grinder import grinder > > > from net.grinder.plugin.http > > > import HTTPPluginControl, HTTPRequest > > > from HTTPClient import NVPair > > > connectionDefaults = HTTPPluginControl.getConnectionDefaults() > > > httpUtilities = HTTPPluginControl.getHTTPUtilities() > > > > > > # To use a proxy server, uncomment the next line and set the host > > > and port. > > > > > > # connectionDefaults.setProxyServer("localhost", 8001) > > > > > > # These definitions at the top level of the file are evaluated once, > > > # when the worker process is started. > > > > > > connectionDefaults.defaultHeaders > > > = \ > > > ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; > > > en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'), > > > NVPair('Accept-Encoding', 'gzip,deflate'), > > > NVPair('Accept-Language', 'en-us,en;q= > > > 0.5'), > > > NVPair('Accept-Charset', 'UTF-8,*'), > > > NVPair('Accept', > > > 'text/xml,application/xml,application/xhtml+xml,text/html;q= > > > 0.9,text/plain;q=0.8,image/png,*/*;q=0.5'), ) > > > > > > headers0= \ > > > ( ) > > > > > > headers1= \ > > > > > > ( NVPair('Referer', 'https://localhost:8443/webtools/control/ > > > main' ), ) > > > > > > headers2= \ > > > ( NVPair('Referer', ' > > > https://localhost:8443/webtools/control/checkLogin/main'), ) > > > > > > url0 = ' <a href="https://localhost:8443'">https://localhost:8443' > > > > > > # Create an HTTPRequest for each request, then replace the > > > # reference to the HTTPRequest with an instrumented version. > > > > > > # You can access the unadorned instance using request101.__target__. > > > request101 = HTTPRequest(url=url0, headers=headers0) > > > request101 = Test(101, 'GET /').wrap(request101) > > > > > > request102 = HTTPRequest(url=url0, headers=headers0) > > > > > > request102 = Test(102, 'GET main').wrap(request102) > > > > > > request201 = HTTPRequest(url=url0, headers=headers1) > > > request201 = Test(201, 'GET main').wrap(request201) > > > > > > request301 = HTTPRequest(url=url0, headers=headers2) > > > > > > request301 = Test(301, 'POST login').wrap(request301) > > > > > > > > > class TestRunner: > > > """A TestRunner instance is created for each worker thread.""" > > > > > > # A method for each recorded page. > > > > > > def page1(self): > > > """GET / (requests 101-102).""" > > > > > > # Expecting 302 'Moved Temporarily' > > > result = request101.GET('/webtools/') > > > > > > grinder.sleep(16) > > > > > > request102.GET('/webtools/control/main') > > > > > > return result > > > > > > def page2(self): > > > """GET main (request 201).""" > > > result = request201.GET('/webtools/control/checkLogin/main') > > > > > > > > > return result > > > > > > def page3(self): > > > """POST login (request 301).""" > > > result = request301.POST('/webtools/control/login', > > > ( NVPair('USERNAME', 'admin'), > > > > > > NVPair('PASSWORD', 'ofbiz'), ), > > > ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), )) > > > > > > return result > > > > > > def __call__(self): > > > """This method is called for every run performed by the worker > > > thread.""" > > > > > > self.page1() # GET / (requests 101-102) > > > > > > grinder.sleep(4328) > > > self.page2() # GET main (request 201) > > > > > > grinder.sleep(2109) > > > self.page3() # POST login (request 301) > > > > > > > > > > > > def instrumentMethod(test, method_name, c=TestRunner): > > > """Instrument a method with the given Test.""" > > > unadorned = getattr(c, method_name) > > > import new > > > method = new.instancemethod > > > (test.wrap(unadorned), None, c) > > > setattr(c, method_name, method) > > > > > > # Replace each method with an instrumented version. > > > # You can call the unadorned method using self.page1.__target__(). > > > instrumentMethod(Test(100, 'Page 1'), 'page1') > > > > > > instrumentMethod(Test(200, 'Page 2'), 'page2') > > > instrumentMethod(Test(300, 'Page 3'), 'page3') > > > > > > > > > > > > Possible Custom Generated Grinder Test Script # The Grinder 3.0-beta30 > > > # HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM > > > > > > from net.grinder.script import Test > > > from net.grinder.script.Grinder import grinder > > > from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest > > > > > > from HTTPClient import NVPair > > > import AGrinderTest > > > > > > connectionDefaults = HTTPPluginControl.getConnectionDefaults() > > > httpUtilities = HTTPPluginControl.getHTTPUtilities() > > > > > > # To use a proxy server, uncomment the next line and set the host > > > and port. > > > > > > # connectionDefaults.setProxyServer("localhost", 8001) > > > > > > # These definitions at the top level of the file are evaluated once, > > > # when the worker process is started. > > > > > > connectionDefaults.defaultHeaders > > > = \ > > > ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; > > > en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'), > > > NVPair('Accept-Encoding', 'gzip,deflate'), > > > NVPair('Accept-Language', 'en-us,en;q= > > > 0.5'), > > > NVPair('Accept-Charset', 'UTF-8,*'), > > > NVPair('Accept', > > > 'text/xml,application/xml,application/xhtml+xml,text/ > > > html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'), > > > ) > > > > > > headers0= \ > > > ( ) > > > > > > headers1= \ > > > > > > ( NVPair('Referer', 'https://localhost:8443/webtools/control/ > > > main'), ) > > > > > > headers2= \ > > > ( NVPair('Referer', ' > > > https://localhost:8443/webtools/control/checkLogin/main'), ) > > > > > > url0 = '<a href="https://localhost:8443'">https://localhost:8443' > > > > > > # Create an HTTPRequest for each request, then replace the > > > # reference to the HTTPRequest with an instrumented version. > > > > > > # You can access the unadorned instance using request101.__target__. > > > request101 = HTTPRequest(url=url0, headers=headers0) > > > request101 = Test(101, 'GET /').wrap(request101) > > > > > > request102 = HTTPRequest(url=url0, headers=headers0) > > > > > > request102 = Test(102, 'GET main').wrap(request102) > > > > > > request201 = HTTPRequest(url=url0, headers=headers1) > > > request201 = Test(201, 'GET main').wrap(request201) > > > > > > request301 = HTTPRequest(url=url0, headers=headers2) > > > > > > request301 = Test(301, 'POST login').wrap(request301) > > > > > > > > > class TestRunner: > > > """A TestRunner instance is created for each worker thread.""" > > > > > > # A method for each recorded page. > > > > > > def page1(self): > > > """GET / (requests 101-102).""" > > > > > > # Expecting 302 'Moved Temporarily' > > > result = request101.GET('/webtools/') > > > > > > grinder.sleep(16) > > > > > > request102.GET('/webtools/control/main') > > > > > > return result > > > > > > def page2(self): > > > """GET main (request 201).""" > > > result = request201.GET('/webtools/control/checkLogin/main') > > > > > > > > > return result > > > > > > def page3(self): > > > """POST login (request 301).""" > > > result = request301.POST('/webtools/control/login', > > > ( NVPair('USERNAME', 'admin'), > > > > > > NVPair('PASSWORD', 'ofbiz'), ), > > > ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), )) > > > > > > return result > > > > > > > > > """This is not working code. Just an idea of how the generated > > > code would look""" > > > > > > def webtoolsWrap(self, overlayMap): > > > inMap = {} > > > outMap = {} > > > self.agrinder.runWrappedTest(page1, inMap, outMap, overlayMap) > > > > > > def mainWrap(self, overlayMap): > > > inMap = {} > > > outMap = {} > > > > > > self.agrinder.runWrappedTest(page2, inMap, outMap, overlayMap) > > > > > > def loginWrap(self, overlayMap): > > > inMap = {'USERNAME':'admin', 'PASSWORD':'ofbiz'} > > > outMap = {} > > > self.agrinder.runWrappedTest > > > (page2, inMap, outMap, overlayMap) > > > > > > def __call__(self): > > > """This method is called for every run performed by the worker > > > thread.""" > > > > > > """I am going to do some handwaving here because I want to get > > > this doc out today, > > > > > > but there would be code here to read in an argument from the > > > command line and > > > use it as a file path to read in the "test overlay" XML doc""" > > > self.agrinder = AGrinderTest.AGrinder > > > () > > > testNode = self.agrinder.getOverlayTest("webtools", sys.argv) > > > self.webtoolsWrap(testNode) # GET / (requests 101-102) > > > > > > grinder.sleep(4328) > > > self.mainWrap(testNode) # GET main (request 201) > > > > > > > > > grinder.sleep(2109) > > > self.loginWrap(testNode) # POST login (request 301) > > > > > > > > > def instrumentMethod(test, method_name, c=TestRunner): > > > """Instrument a method with the given Test.""" > > > > > > unadorned = getattr(c, method_name) > > > import new > > > method = new.instancemethod(test.wrap(unadorned), None, c) > > > setattr(c, method_name, method) > > > > > > # Replace each method with an instrumented version. > > > > > > # You can call the unadorned method using self.page1.__target__(). > > > instrumentMethod(Test(100, 'Page 1'), 'page1') > > > instrumentMethod(Test(200, 'Page 2'), 'page2') > > > instrumentMethod(Test(300, 'Page 3'), 'page3') > > > > > > > > > Possible Overlay XML File This example is probably not even close to > > > what the final test overlay XML script will look like, but I think it > > > is better to have something to go from. <tests> > > > <test name="webtools"> > > > <subtest name="login"> > > > <input> > > > <field name="USERNAME">jdoe</field> > > > <field name="PASSWORD">id9Ed3jk</field> > > > > > > </input> > > > <match> > > > <field type="regex" op="not">not found</field> > > > <field type="script" op="true">isSuccess()</field> > > > > > > </match> > > > </subtest> > > > </test> > > > </tests> > > > > Best Regards, > > > > Si > > [hidden email] > > > > > > > > > > > |
|
|
I don't if you guys have any experience with Canoo's Webtest, but we had
tester's writing these with no problems back in the day. Brett's right, we just need to ensure that the syntax makes sense or no one will use it. Cheers, Tim -- Tim Ruppert HotWax Media http://www.hotwaxmedia.com o:801.649.6594 f:801.649.6595 Brett Palmer wrote: > Maybe the data files could be simple property files with a specific > naming convention. Then the jython script would just refer to a > specific resource and the data would be loaded and executed. > > The data files could contains expected results and validation data. > > The thing with XML is if it is too complex a tester won't create the > data files. > > Brett > > On 11/2/06, Al Byers <[hidden email]> wrote: >> Si, >> >> Thanks for the feedback. I debated over the xml language. I think it >> is important to involve a different level of user. But the safe thing >> to do would be to create the api that you mention and then wrap it in >> an xml layer later on if there is demand. >> >> -Al >> >> On 11/2/06, Si Chen <[hidden email]> wrote: >> > Al, >> > >> > This is great that you're doing this. I think a testing framework is >> > really what we need right now. >> > >> > My suggestion is instead of creating an xml language like you show at >> > the bottom to make some API methods that could be called in Jython >> > scripts. I think that might be easier for people to work with and >> > would require less work over time in maintaining and extending the >> > xml language. >> > >> > On Nov 1, 2006, at 10:49 AM, Al Byers wrote: >> > >> > > I am having trouble getting my messages posted to the list. This one >> > > is taking a long time and I wonder if it is because it has HTML >> in the >> > > content and an attachment. I am pasting it in plain test. >> > > ... >> > > One of the presentations at the Users Conference will be me >> talking on >> > > my experiences with Grinder. With David there, we hope to go far >> past >> > > that to discuss strategies for adding testing functionality to >> OFBiz. >> > > I have no experience that qualifies me to be an expert in this >> area - >> > > just have a need. Rather than just wait for everyone to show up >> at the >> > > conference, I felt it would be helpful to try to treat some of the >> > > subjects ahead of time. Of course, not everything will be decided at >> > > the conference, so as much input as possible from everyone will be >> > > needed. Also, if there is anyone with strong Python and/or Grinder >> > > skills, their help would also be appreciated. >> > > >> > > >> > > OFBiz Testing Initiative >> > > GOAL >> > > BACKGROUND >> > > Problem with Design Documents >> > > Applicable Features of OFBiz >> > > Applicable Features of Grinder >> > > Applicable Features of Extreme Programming >> > > REQUIREMENTS >> > > Easy Generation >> > > Handle Complex Tests >> > > Adaptable per Application >> > > Rugged >> > > Handle Iterations >> > > Usable at Multiple Proficiency Levels >> > > Simplify Output Testing >> > > STRAWMAN PLAN >> > > Modify Test Script Generation >> > > Add Test Overlay Config File >> > > Source Code Control to Handle Iterations >> > > Appendices >> > > Standard Generated Grinder Test Script >> > > Possible Custom Generated Grinder Test Script >> > > Possible Overlay XML File >> > > GOAL The goal of the OFBiz Testing Initiative (OFBTI) is to add >> > > functionality to OFBiz that will streamline the functional testing >> > > process to the point that it will be cost-effective and >> beneficial to >> > > write comprehensive tests. Ideally, the tools could be used by >> > > non-programmers. >> > > >> > > A secondary goal would be to add tools to make design specs >> easier to >> > > write and more useful. >> > > BACKGROUND >> > > Problem with Design Documents Design documents, though necessary, >> have >> > > serious drawbacks. >> > > Expensive to create - should probably take as much time as the >> > > implementation. >> > > >> > > Never complete enough - always need to go back to client, anyway. >> > > >> > > No automation help in implementation and validation - as text >> > > documents, they must be manually translated into architecture >> > > documents. >> > > >> > > Easy to get out-of-date - while many times they are supposed to be >> > > "living" documents, in reality, they almost never are. >> > > >> > > >> > > Applicable Features of Extreme Programming Extreme programming >> > > addresses many of the shortcomings of the more conventional design >> > > document approach. Instead of a complete design doc or even use case >> > > up front, all that is required is a "story". The story needs only be >> > > complete enough to generate the first test and the first test is >> > > usually just a placeholder. Then in the course of many >> iterations, the >> > > functional test is enhanced, not the design document. The tests are >> > > living and serve the useful purpose of verifying that things have >> not >> > > broken as code gets changed. >> > > >> > > One of the very useful things about this approach is that when bugs >> > > are discovered, a test or subtest is added to guard against it ever >> > > creeping back into the code. >> > > Applicable Features of OFBiz OFBiz is different from other >> development >> > > environments, and it would be good to identify the features that >> could >> > > be used advantageously or which must be dealt with. >> > > >> > > OFBiz is highly configurable via XML formatted files and it would be >> > > good to continue that pattern. XML config files, because they have >> > > associated schema, make it feasible for non-programmers and those >> not >> > > intimately familiar with OFBiz to make changes and perform certain >> > > programming tasks. >> > > >> > > OFBiz has a general pattern of offering easily configurable options >> > > via data parameters, but always offering easy access to the lowest >> > > level of programming for those needed cases without requiring a huge >> > > environment adaptation. For instance, the screen widget config files >> > > allow screens to be created with miminal data and allow the user >> to be >> > > prompted via the schema in an XML editor, but if there is a >> construct >> > > that the widgets do not handle, the user can just throw in a call to >> > > an HTML component or a FreeMarker template - they do not need to >> > > abandon the screen widget framework. >> > > >> > > OFBiz is a service oriented framework and there are many tools that >> > > can be used aid in that process. If a request is processed by a >> > > service (in lieu of an event) then the input parameters can be >> > > automatically taken from the HTTP stream and automatically converted >> > > to the right type. Also, the form widget can build forms from the >> > > service definition. >> > > Applicable Features of Grinder There are multiple options for >> testing >> > > frameworks, but Grinder offers the following unique advantages. >> > > A robust test script generator - TCPProxy attempts to assign return >> > > parameters to variables and reuse them, rather than passing >> > > literals around. This means that it does not instantly break when >> keys >> > > are autogenerated. >> > > Grinder has convenient customization point - script generation is >> done >> > > via an XSLT stylesheet that makes use of Java extensions. This would >> > > be a natural place to make modifications to suit OFBTI. There are >> also >> > > filters for the requests and responses that can be swapped out via >> > > command line parameters. >> > > Jython - allows seamless switching between a highly productive >> > > scripting environment and regular Java code. Because of this >> feature, >> > > Jython has a lot of possibilities within OFBiz. >> > > Integrated functional and jUnit testing - the use of Jython >> allows for >> > > the same environment to be used to run HTTP client system tests and >> > > jUnit functional tests. >> > > REQUIREMENTS >> > > Easy Generation The use of a test script generator, such as >> Grinder's >> > > TCPProxy, allows test scripts to be created by capturing the >> input of >> > > a user at a browser. >> > > Handle Complex Tests A further enhancement would be to allow the >> > > creation of scripts by supplying a few data parameters or by easily >> > > modifying automatically generated scripts. One of the biggest >> > > drawbacks of end user testing environments is that when something >> > > changes, the associated test script is usually unusable. A big >> > > improvement would be to allow tests to be created by chaining >> together >> > > subtests. >> > > Adaptable per Application Many applications will have special >> > > characteristics that need to be handled specially. In one >> instance, I >> > > found that Grinder was using the '$' character to form Python >> variable >> > > names (which doesn't work) because that is how the HTTP parameters >> > > were named. >> > > Rugged >> > > Handle Iterations The general idea of XP is that the tests become >> more >> > > complex and comprehensive as the code becomes more complete. It >> would >> > > probably be a good idea to retain visibility to past test setups - >> > > though, at this time, I am not sure why. >> > > Usable at Multiple Proficiency Levels It would be good to have the >> > > testing environment useable, or at least understandable, by >> > > non-programmers. JUnit tests would not meet this requirement. >> Neither >> > > would Python scripts (though it could come close). For project >> > > managers, an XML-based configuration environment would be needed. >> > > Simplify Output Testing The analysis of the HTML pages that are >> > > returned by Grinder tests is one of the more problematic areas of >> > > end-user testing. The more common scenarios would need to be handled >> > > by the XML-based configuration environment mentioned above. The >> > > foreseeable cases would be matching of form values, existence or >> > > non-existence anywhere of a phrase or something that can be >> tested by >> > > an XPath expression. >> > > STRAWMAN PLAN What is discussed below is how I see tackling the >> > > testing problem. I am only interested in what will be supported >> by the >> > > community, so it can be changed. >> > > >> > > The overall approach is to use as much of Grinder as possible, and >> > > allow for the generated Grinder test scripts to be overwritten by a >> > > user generated XML test configuration file. >> > > Modify Test Script Generation The Grinder TCPProxy program will be >> > > used, but the XSLT stylesheet and the associated Java extension >> > > classes will be enhanced. The main change will be that the >> individual >> > > request tests that Grinder generates for each round trip to the >> server >> > > will be wrapped by code that will check to see if there are data >> > > overrides to be made coming from the user generated high level test >> > > config file. Each wrapper will also have a dictionary (ie. map) that >> > > defines tests to be made on each low level result. These also >> will be >> > > modified by the higher level config file. If there is not higher >> level >> > > config file, the script as generated by TCPProxy will run with no >> > > modifications. >> > > Input Dictionary The input directory will do more than just allow >> the >> > > user to supply literal values as test input. There will be helper >> > > functions for allowing the user to randomly pick values from a list. >> > > Also there will be helper functions for generating reasonably >> looking >> > > addresses. The user will be able to create Python scripts to >> generated >> > > special input. See the appendix for samples of what this would look >> > > like. >> > > Output Dictionary The output dictionary, which contains the test >> > > criteria, would have test helper functions of the following sort: >> > > Literal value >> > > Regular expression >> > > Form value >> > > XPath >> > > Custom Jython scripts Many of the test methods will be suitable for >> > > non-programmers, but the use of Jython scripts follows the OFBiz >> > > pattern of providing simple methods for simple tasks, but making it >> > > easy to drop down to the level needed to solve the problem. At some >> > > point (right away?), it will be necessary to allow complex joing of >> > > tests with AND, OR and NOT operators. >> > > Add Test Overlay Config File One of the modifications to the OOTB >> test >> > > script generated by TCPProxy will be that the script will look >> for the >> > > existence a file path as a parameters and use it to overwrite the >> > > default test script values (ie. the values typed in by the user when >> > > the script was being generated). The script could have multiple >> levels >> > > of complex scripts, but eventually, they must call one of the >> > > "request" tests. Keep in mind that the base test script could have a >> > > large number of "request" scripts, but they would not all need to be >> > > used by the user-defined config file; it could just use a subset. So >> > > the base test script may not, in fact, be totally generated by one >> > > session of user interaction with the system under test; it could be >> > > built up over time as new functionality is added without having >> to run >> > > thru the test. The OOTB behavior is to sequentially number the >> tests, >> > > but in this case, we may wish to name them with the request name >> that >> > > they interact with. >> > > >> > > These high level test scripts would need "include" functionality so >> > > that they could used standard building blocks. >> > > Source Code Control to Handle Iterations >> > > Appendices >> > > Standard Generated Grinder Test Script This is the actual output >> from >> > > a TCPProxy session: >> > > >> > > # The Grinder 3.0-beta30 >> > > # HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM >> > > >> > > from net.grinder.script import Test >> > > from net.grinder.script.Grinder import grinder >> > > from net.grinder.plugin.http >> > > import HTTPPluginControl, HTTPRequest >> > > from HTTPClient import NVPair >> > > connectionDefaults = HTTPPluginControl.getConnectionDefaults() >> > > httpUtilities = HTTPPluginControl.getHTTPUtilities() >> > > >> > > # To use a proxy server, uncomment the next line and set the host >> > > and port. >> > > >> > > # connectionDefaults.setProxyServer("localhost", 8001) >> > > >> > > # These definitions at the top level of the file are evaluated once, >> > > # when the worker process is started. >> > > >> > > connectionDefaults.defaultHeaders >> > > = \ >> > > ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; >> > > en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'), >> > > NVPair('Accept-Encoding', 'gzip,deflate'), >> > > NVPair('Accept-Language', 'en-us,en;q= >> > > 0.5'), >> > > NVPair('Accept-Charset', 'UTF-8,*'), >> > > NVPair('Accept', >> > > 'text/xml,application/xml,application/xhtml+xml,text/html;q= >> > > 0.9,text/plain;q=0.8,image/png,*/*;q=0.5'), ) >> > > >> > > headers0= \ >> > > ( ) >> > > >> > > headers1= \ >> > > >> > > ( NVPair('Referer', 'https://localhost:8443/webtools/control/ >> > > main' ), ) >> > > >> > > headers2= \ >> > > ( NVPair('Referer', ' >> > > https://localhost:8443/webtools/control/checkLogin/main'), ) >> > > >> > > url0 = ' <a href="https://localhost:8443'">https://localhost:8443' >> > > >> > > # Create an HTTPRequest for each request, then replace the >> > > # reference to the HTTPRequest with an instrumented version. >> > > >> > > # You can access the unadorned instance using request101.__target__. >> > > request101 = HTTPRequest(url=url0, headers=headers0) >> > > request101 = Test(101, 'GET /').wrap(request101) >> > > >> > > request102 = HTTPRequest(url=url0, headers=headers0) >> > > >> > > request102 = Test(102, 'GET main').wrap(request102) >> > > >> > > request201 = HTTPRequest(url=url0, headers=headers1) >> > > request201 = Test(201, 'GET main').wrap(request201) >> > > >> > > request301 = HTTPRequest(url=url0, headers=headers2) >> > > >> > > request301 = Test(301, 'POST login').wrap(request301) >> > > >> > > >> > > class TestRunner: >> > > """A TestRunner instance is created for each worker thread.""" >> > > >> > > # A method for each recorded page. >> > > >> > > def page1(self): >> > > """GET / (requests 101-102).""" >> > > >> > > # Expecting 302 'Moved Temporarily' >> > > result = request101.GET('/webtools/') >> > > >> > > grinder.sleep(16) >> > > >> > > request102.GET('/webtools/control/main') >> > > >> > > return result >> > > >> > > def page2(self): >> > > """GET main (request 201).""" >> > > result = request201.GET('/webtools/control/checkLogin/main') >> > > >> > > >> > > return result >> > > >> > > def page3(self): >> > > """POST login (request 301).""" >> > > result = request301.POST('/webtools/control/login', >> > > ( NVPair('USERNAME', 'admin'), >> > > >> > > NVPair('PASSWORD', 'ofbiz'), ), >> > > ( NVPair('Content-Type', >> 'application/x-www-form-urlencoded'), )) >> > > >> > > return result >> > > >> > > def __call__(self): >> > > """This method is called for every run performed by the worker >> > > thread.""" >> > > >> > > self.page1() # GET / (requests 101-102) >> > > >> > > grinder.sleep(4328) >> > > self.page2() # GET main (request 201) >> > > >> > > grinder.sleep(2109) >> > > self.page3() # POST login (request 301) >> > > >> > > >> > > >> > > def instrumentMethod(test, method_name, c=TestRunner): >> > > """Instrument a method with the given Test.""" >> > > unadorned = getattr(c, method_name) >> > > import new >> > > method = new.instancemethod >> > > (test.wrap(unadorned), None, c) >> > > setattr(c, method_name, method) >> > > >> > > # Replace each method with an instrumented version. >> > > # You can call the unadorned method using self.page1.__target__(). >> > > instrumentMethod(Test(100, 'Page 1'), 'page1') >> > > >> > > instrumentMethod(Test(200, 'Page 2'), 'page2') >> > > instrumentMethod(Test(300, 'Page 3'), 'page3') >> > > >> > > >> > > >> > > Possible Custom Generated Grinder Test Script # The Grinder >> 3.0-beta30 >> > > # HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM >> > > >> > > from net.grinder.script import Test >> > > from net.grinder.script.Grinder import grinder >> > > from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest >> > > >> > > from HTTPClient import NVPair >> > > import AGrinderTest >> > > >> > > connectionDefaults = HTTPPluginControl.getConnectionDefaults() >> > > httpUtilities = HTTPPluginControl.getHTTPUtilities() >> > > >> > > # To use a proxy server, uncomment the next line and set the host >> > > and port. >> > > >> > > # connectionDefaults.setProxyServer("localhost", 8001) >> > > >> > > # These definitions at the top level of the file are evaluated once, >> > > # when the worker process is started. >> > > >> > > connectionDefaults.defaultHeaders >> > > = \ >> > > ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; >> > > en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'), >> > > NVPair('Accept-Encoding', 'gzip,deflate'), >> > > NVPair('Accept-Language', 'en-us,en;q= >> > > 0.5'), >> > > NVPair('Accept-Charset', 'UTF-8,*'), >> > > NVPair('Accept', >> > > 'text/xml,application/xml,application/xhtml+xml,text/ >> > > html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'), >> > > ) >> > > >> > > headers0= \ >> > > ( ) >> > > >> > > headers1= \ >> > > >> > > ( NVPair('Referer', 'https://localhost:8443/webtools/control/ >> > > main'), ) >> > > >> > > headers2= \ >> > > ( NVPair('Referer', ' >> > > https://localhost:8443/webtools/control/checkLogin/main'), ) >> > > >> > > url0 = '<a href="https://localhost:8443'">https://localhost:8443' >> > > >> > > # Create an HTTPRequest for each request, then replace the >> > > # reference to the HTTPRequest with an instrumented version. >> > > >> > > # You can access the unadorned instance using request101.__target__. >> > > request101 = HTTPRequest(url=url0, headers=headers0) >> > > request101 = Test(101, 'GET /').wrap(request101) >> > > >> > > request102 = HTTPRequest(url=url0, headers=headers0) >> > > >> > > request102 = Test(102, 'GET main').wrap(request102) >> > > >> > > request201 = HTTPRequest(url=url0, headers=headers1) >> > > request201 = Test(201, 'GET main').wrap(request201) >> > > >> > > request301 = HTTPRequest(url=url0, headers=headers2) >> > > >> > > request301 = Test(301, 'POST login').wrap(request301) >> > > >> > > >> > > class TestRunner: >> > > """A TestRunner instance is created for each worker thread.""" >> > > >> > > # A method for each recorded page. >> > > >> > > def page1(self): >> > > """GET / (requests 101-102).""" >> > > >> > > # Expecting 302 'Moved Temporarily' >> > > result = request101.GET('/webtools/') >> > > >> > > grinder.sleep(16) >> > > >> > > request102.GET('/webtools/control/main') >> > > >> > > return result >> > > >> > > def page2(self): >> > > """GET main (request 201).""" >> > > result = request201.GET('/webtools/control/checkLogin/main') >> > > >> > > >> > > return result >> > > >> > > def page3(self): >> > > """POST login (request 301).""" >> > > result = request301.POST('/webtools/control/login', >> > > ( NVPair('USERNAME', 'admin'), >> > > >> > > NVPair('PASSWORD', 'ofbiz'), ), >> > > ( NVPair('Content-Type', >> 'application/x-www-form-urlencoded'), )) >> > > >> > > return result >> > > >> > > >> > > """This is not working code. Just an idea of how the generated >> > > code would look""" >> > > >> > > def webtoolsWrap(self, overlayMap): >> > > inMap = {} >> > > outMap = {} >> > > self.agrinder.runWrappedTest(page1, inMap, outMap, overlayMap) >> > > >> > > def mainWrap(self, overlayMap): >> > > inMap = {} >> > > outMap = {} >> > > >> > > self.agrinder.runWrappedTest(page2, inMap, outMap, overlayMap) >> > > >> > > def loginWrap(self, overlayMap): >> > > inMap = {'USERNAME':'admin', 'PASSWORD':'ofbiz'} >> > > outMap = {} >> > > self.agrinder.runWrappedTest >> > > (page2, inMap, outMap, overlayMap) >> > > >> > > def __call__(self): >> > > """This method is called for every run performed by the worker >> > > thread.""" >> > > >> > > """I am going to do some handwaving here because I want to get >> > > this doc out today, >> > > >> > > but there would be code here to read in an argument from the >> > > command line and >> > > use it as a file path to read in the "test overlay" XML doc""" >> > > self.agrinder = AGrinderTest.AGrinder >> > > () >> > > testNode = self.agrinder.getOverlayTest("webtools", sys.argv) >> > > self.webtoolsWrap(testNode) # GET / (requests 101-102) >> > > >> > > grinder.sleep(4328) >> > > self.mainWrap(testNode) # GET main (request 201) >> > > >> > > >> > > grinder.sleep(2109) >> > > self.loginWrap(testNode) # POST login (request 301) >> > > >> > > >> > > def instrumentMethod(test, method_name, c=TestRunner): >> > > """Instrument a method with the given Test.""" >> > > >> > > unadorned = getattr(c, method_name) >> > > import new >> > > method = new.instancemethod(test.wrap(unadorned), None, c) >> > > setattr(c, method_name, method) >> > > >> > > # Replace each method with an instrumented version. >> > > >> > > # You can call the unadorned method using self.page1.__target__(). >> > > instrumentMethod(Test(100, 'Page 1'), 'page1') >> > > instrumentMethod(Test(200, 'Page 2'), 'page2') >> > > instrumentMethod(Test(300, 'Page 3'), 'page3') >> > > >> > > >> > > Possible Overlay XML File This example is probably not even close to >> > > what the final test overlay XML script will look like, but I >> think it >> > > is better to have something to go from. <tests> >> > > <test name="webtools"> >> > > <subtest name="login"> >> > > <input> >> > > <field name="USERNAME">jdoe</field> >> > > <field name="PASSWORD">id9Ed3jk</field> >> > > >> > > </input> >> > > <match> >> > > <field type="regex" op="not">not found</field> >> > > <field type="script" op="true">isSuccess()</field> >> > > >> > > </match> >> > > </subtest> >> > > </test> >> > > </tests> >> > >> > Best Regards, >> > >> > Si >> > [hidden email] >> > >> > >> > >> > >> > >> |
|
|
I thought the XML would be an improvement in terms of ease of use.
Glad we have some people with experience chiming in. Should be a good discussion on Saturday. -Al |
Re: OFBiz Testing Initiative
|
|
In reply to this post by byersa
Al,
this is really very interesting, thanks for sharing your ideas and research results. Just one point (after a very quick review of your document) I see that we will need to enhance the Grinder tool to better support the tests we will need in OFBiz. That's great, however I think that it is worth noticing (even if it is probably what you have already planned to do) that it would beneficial for the two projects (OFBiz and The Grinder) if all the enhancements that are *not* strictly connected with OFBiz, will be discussed/designed/implemented (also) in the Grinder community. In this way the workload for the OFBiz community will be limited to the integration stuff and the risks of having an incomplete tool and a lot of new code to maintain will be lower. Jacopo |
|
|
For anyone following this thread, the mini conference went extremely
well and the thing that came out of this portion was the suggestion by Tim Ruppert that we look at WebTest (http://webtest.canoo.com/webtest/manual/WebTestHome.html) for doing this sort of testing. It certainly seems to be what we were looking for. There will probably still be a place for Grinder for doing load testing. -Al On 11/3/06, Jacopo Cappellato <[hidden email]> wrote: > Al, > > this is really very interesting, thanks for sharing your ideas and > research results. > > Just one point (after a very quick review of your document) I see that > we will need to enhance the Grinder tool to better support the tests we > will need in OFBiz. > That's great, however I think that it is worth noticing (even if it is > probably what you have already planned to do) that it would beneficial > for the two projects (OFBiz and The Grinder) if all the enhancements > that are *not* strictly connected with OFBiz, will be > discussed/designed/implemented (also) in the Grinder community. > In this way the workload for the OFBiz community will be limited to the > integration stuff and the risks of having an incomplete tool and a lot > of new code to maintain will be lower. > > Jacopo > > |
|
Administrator
|
I did not yet found time time to test/use it but I know that Neogia people use Selenium for this job and seem quite happy with it.
http://www.openqa.org/selenium/ Jacques ----- Original Message ----- From: "Al Byers" <[hidden email]> To: <[hidden email]> Sent: Monday, November 06, 2006 2:30 PM Subject: Re: OFBiz Testing Initiative > For anyone following this thread, the mini conference went extremely > well and the thing that came out of this portion was the suggestion by > Tim Ruppert that we look at WebTest > (http://webtest.canoo.com/webtest/manual/WebTestHome.html) for doing > this sort of testing. It certainly seems to be what we were looking > for. There will probably still be a place for Grinder for doing load > testing. > > -Al > > On 11/3/06, Jacopo Cappellato <[hidden email]> wrote: > > Al, > > > > this is really very interesting, thanks for sharing your ideas and > > research results. > > > > Just one point (after a very quick review of your document) I see that > > we will need to enhance the Grinder tool to better support the tests we > > will need in OFBiz. > > That's great, however I think that it is worth noticing (even if it is > > probably what you have already planned to do) that it would beneficial > > for the two projects (OFBiz and The Grinder) if all the enhancements > > that are *not* strictly connected with OFBiz, will be > > discussed/designed/implemented (also) in the Grinder community. > > In this way the workload for the OFBiz community will be limited to the > > integration stuff and the risks of having an incomplete tool and a lot > > of new code to maintain will be lower. > > > > Jacopo > > > > |
|
|
Right now, I'm preparing to do a little Webtest / Selenium testing
either tomorrow or first thing next week. Selenium's got a pretty slick Firefox IDE in addition to their APIs for doing testing. I'll try and get everyone one feedback in the not too distant future. Cheers, Tim -- Tim Ruppert HotWax Media http://www.hotwaxmedia.com o:801.649.6594 f:801.649.6595 Jacques Le Roux wrote: > I did not yet found time time to test/use it but I know that Neogia people use Selenium for this job and seem quite happy with it. > > http://www.openqa.org/selenium/ > > Jacques > > ----- Original Message ----- > From: "Al Byers" <[hidden email]> > To: <[hidden email]> > Sent: Monday, November 06, 2006 2:30 PM > Subject: Re: OFBiz Testing Initiative > > > >> For anyone following this thread, the mini conference went extremely >> well and the thing that came out of this portion was the suggestion by >> Tim Ruppert that we look at WebTest >> (http://webtest.canoo.com/webtest/manual/WebTestHome.html) for doing >> this sort of testing. It certainly seems to be what we were looking >> for. There will probably still be a place for Grinder for doing load >> testing. >> >> -Al >> >> On 11/3/06, Jacopo Cappellato <[hidden email]> wrote: >> >>> Al, >>> >>> this is really very interesting, thanks for sharing your ideas and >>> research results. >>> >>> Just one point (after a very quick review of your document) I see that >>> we will need to enhance the Grinder tool to better support the tests we >>> will need in OFBiz. >>> That's great, however I think that it is worth noticing (even if it is >>> probably what you have already planned to do) that it would beneficial >>> for the two projects (OFBiz and The Grinder) if all the enhancements >>> that are *not* strictly connected with OFBiz, will be >>> discussed/designed/implemented (also) in the Grinder community. >>> In this way the workload for the OFBiz community will be limited to the >>> integration stuff and the risks of having an incomplete tool and a lot >>> of new code to maintain will be lower. >>> >>> Jacopo >>> >>> >>> |
|
Administrator
|
Yes Tim,
The Firefox IDE and versatile aspect of Selenium seems very interesting. Looking forward for the results of you comparative(?) tests. Thanks Jacques From: "Tim Ruppert" <[hidden email]> > Right now, I'm preparing to do a little Webtest / Selenium testing > either tomorrow or first thing next week. Selenium's got a pretty slick > Firefox IDE in addition to their APIs for doing testing. I'll try and > get everyone one feedback in the not too distant future. > > Cheers, > Tim > > -- > Tim Ruppert > HotWax Media > http://www.hotwaxmedia.com > > o:801.649.6594 > f:801.649.6595 > > Jacques Le Roux wrote: > > I did not yet found time time to test/use it but I know that Neogia people use Selenium for this job and seem quite happy with > > > > http://www.openqa.org/selenium/ > > > > Jacques > > > > ----- Original Message ----- > > From: "Al Byers" <[hidden email]> > > To: <[hidden email]> > > Sent: Monday, November 06, 2006 2:30 PM > > Subject: Re: OFBiz Testing Initiative > > > > > > > >> For anyone following this thread, the mini conference went extremely > >> well and the thing that came out of this portion was the suggestion by > >> Tim Ruppert that we look at WebTest > >> (http://webtest.canoo.com/webtest/manual/WebTestHome.html) for doing > >> this sort of testing. It certainly seems to be what we were looking > >> for. There will probably still be a place for Grinder for doing load > >> testing. > >> > >> -Al > >> > >> On 11/3/06, Jacopo Cappellato <[hidden email]> wrote: > >> > >>> Al, > >>> > >>> this is really very interesting, thanks for sharing your ideas and > >>> research results. > >>> > >>> Just one point (after a very quick review of your document) I see that > >>> we will need to enhance the Grinder tool to better support the tests we > >>> will need in OFBiz. > >>> That's great, however I think that it is worth noticing (even if it is > >>> probably what you have already planned to do) that it would beneficial > >>> for the two projects (OFBiz and The Grinder) if all the enhancements > >>> that are *not* strictly connected with OFBiz, will be > >>> discussed/designed/implemented (also) in the Grinder community. > >>> In this way the workload for the OFBiz community will be limited to the > >>> integration stuff and the risks of having an incomplete tool and a lot > >>> of new code to maintain will be lower. > >>> > >>> Jacopo > >>> > >>> > >>> |
|
|
Testing Update . . . I just wanted to let everyone know that while I
have things set up, i have not had a chance to get much farther than due to my company's demand on my time. As a result, I am unsure if I will get much done on this until after the Thanksgiving break, but know that this is not forgotten and will get completed. My apologies in advance for the tardiness. Cheers, Tim -- Tim Ruppert HotWax Media http://www.hotwaxmedia.com o:801.649.6594 f:801.649.6595 Jacques Le Roux wrote: > Yes Tim, > > The Firefox IDE and versatile aspect of Selenium seems very interesting. > Looking forward for the results of you comparative(?) tests. > > Thanks > > Jacques > > From: "Tim Ruppert" <[hidden email]> > >> Right now, I'm preparing to do a little Webtest / Selenium testing >> either tomorrow or first thing next week. Selenium's got a pretty slick >> Firefox IDE in addition to their APIs for doing testing. I'll try and >> get everyone one feedback in the not too distant future. >> >> Cheers, >> Tim >> >> -- >> Tim Ruppert >> HotWax Media >> http://www.hotwaxmedia.com >> >> o:801.649.6594 >> f:801.649.6595 >> >> Jacques Le Roux wrote: >> >>> I did not yet found time time to test/use it but I know that Neogia people use Selenium for this job and seem quite happy with >>> > it. > >>> http://www.openqa.org/selenium/ >>> >>> Jacques >>> >>> ----- Original Message ----- >>> From: "Al Byers" <[hidden email]> >>> To: <[hidden email]> >>> Sent: Monday, November 06, 2006 2:30 PM >>> Subject: Re: OFBiz Testing Initiative >>> >>> >>> >>> >>>> For anyone following this thread, the mini conference went extremely >>>> well and the thing that came out of this portion was the suggestion by >>>> Tim Ruppert that we look at WebTest >>>> (http://webtest.canoo.com/webtest/manual/WebTestHome.html) for doing >>>> this sort of testing. It certainly seems to be what we were looking >>>> for. There will probably still be a place for Grinder for doing load >>>> testing. >>>> >>>> -Al >>>> >>>> On 11/3/06, Jacopo Cappellato <[hidden email]> wrote: >>>> >>>> >>>>> Al, >>>>> >>>>> this is really very interesting, thanks for sharing your ideas and >>>>> research results. >>>>> >>>>> Just one point (after a very quick review of your document) I see that >>>>> we will need to enhance the Grinder tool to better support the tests we >>>>> will need in OFBiz. >>>>> That's great, however I think that it is worth noticing (even if it is >>>>> probably what you have already planned to do) that it would beneficial >>>>> for the two projects (OFBiz and The Grinder) if all the enhancements >>>>> that are *not* strictly connected with OFBiz, will be >>>>> discussed/designed/implemented (also) in the Grinder community. >>>>> In this way the workload for the OFBiz community will be limited to the >>>>> integration stuff and the risks of having an incomplete tool and a lot >>>>> of new code to maintain will be lower. >>>>> >>>>> Jacopo >>>>> >>>>> >>>>> >>>>> |
«
Return to OFBiz - Dev
|
1 view|%1 views
| Free forum by Nabble | Edit this page |