OFBiz Testing Initiative

[byersa]

One of the presentations at the Users Conference will be me talking on my experiences with Grinder. With David there, we hope to go far past that to discuss strategies for adding testing functionality to OFBiz. I have no experience that qualifies me to be an expert in this area - just have a need. Rather than just wait for everyone to show up at the conference, I felt it would be helpful to try to treat some of the subjects ahead of time. Of course, not everything will be decided at the conference, so as much input as possible from everyone will be needed. Also, if there is anyone with strong Python and/or Grinder skills, their help would also be appreciated.

I have attached the whitepaper HTML file, but I am also going to paste it in here.

-Al Byers


<H1 align="center">OFBiz Testing Initiative</H1>

<ul>
<li><a href="#goal">GOAL</a></li>

<li><a href="#background">BACKGROUND</a></li>
<ul>
<li><a href="#specs">Problem with Design Documents</a></li>
<li><a href="#features">Applicable Features of OFBiz</a></li>
<li><a href="#grinder">Applicable Features of Grinder</a></li>
<li><a href="#xp">Applicable Features of Extreme Programming</a></li>
</ul>

<li><a href="#requirements">REQUIREMENTS</a></li>
<ul>
<li><a href="#generation">Easy Generation</a></li>
<li><a href="#complex">Handle Complex Tests</a></li>
<li><a href="#adaptable">Adaptable per Application</a></li>
<li><a href="#rugged">Rugged</a></li>
<li><a href="#iterations">Handle Iterations</a></li>
<li><a href="#proficiency">Usable at Multiple Proficiency Levels</a></li>
<li><a href="#output">Simplify Output Testing</a></li>
</ul>

<li><a href="#plan">STRAWMAN PLAN</a></li>
<ul>
<li><a href="#modify">Modify Test Script Generation</a></li>
<li><a href="#higher">Add Test Overlay Config File</a></li>
<li><a href="#scc">Source Code Control to Handle Iterations</a></li>
</ul>
<li><a href="#appendices">Appendices</a></li>
<ul>
<li><a href="#stdcode">Standard Generated Grinder Test Script</a></li>
<li><a href="#customcode">Possible Custom Generated Grinder Test Script</a></li>
<li><a href="#overlay">Possible Overlay XML File</a></li>
</ul>
</ul>

<H2><a name="goal">GOAL</a></H2>
The goal of the OFBiz Testing Initiative (OFBTI) is to add functionality to OFBiz that will streamline
the functional testing process to the point that it will be cost-effective and beneficial to write
comprehensive tests. Ideally, the tools could be used by non-programmers.
<p/>
A secondary goal would be to add tools to make design specs easier to write and
more useful.

<H2><a name="background">BACKGROUND</a></H2>

<H3><a name="specs">Problem with Design Documents</a></H3>
Design documents, though necessary, have serious drawbacks.
<ul>
<li><b>Expensive to create</b> - should probably take as much time as the implementation.<li>
<li><b>Never complete enough</b> - always need to go back to client, anyway.<li>
<li><b>No automation help in implementation and validation</b> - as text documents,
they must be manually translated into architecture documents.<li>
<li><b>Easy to get out-of-date</b> - while many times they are supposed to be "living" documents,
in reality, they almost never are.<li>
</ul>

<H3><a name="xp">Applicable Features of Extreme Programming</a></H3>
Extreme programming addresses many of the shortcomings of the more conventional design document approach.
Instead of a complete design doc or even use case up front, all that is required is a "story".
The story needs only be complete enough to generate the first test and the first test is usually just a placeholder.
Then in the course of many iterations, the functional test is enhanced, not the design document.
The tests are living and serve the useful purpose of verifying that things have not broken as code gets changed.
<p/>
One of the very useful things about this approach is that when bugs are discovered, a test or subtest is added to
guard against it ever creeping back into the code.

<H3><a name="features">Applicable Features of OFBiz</a></H3>
OFBiz is different from other development environments, and it would be good to identify the features that could be used
advantageously or which must be dealt with.
<p/>
OFBiz is highly configurable via XML formatted files and it would be good to continue that pattern.
XML config files, because they have associated schema, make it feasible for non-programmers and those not intimately
familiar with OFBiz to make changes and perform certain programming tasks.
<p/>
OFBiz has a general pattern of offering easily configurable options via data parameters, but always offering easy
access to the lowest level of programming for those needed cases without requiring a huge environment adaptation.
For instance, the screen widget config files allow screens to be created with miminal data and allow the user to be
prompted via the schema in an XML editor, but if there is a construct that the widgets do not handle, the user
can just throw in a call to an HTML component or a FreeMarker template - they do not need to abandon the screen widget framework.
<p/>
OFBiz is a service oriented framework and there are many tools that can be used aid in that process.
If a request is processed by a service (in lieu of an event) then the input parameters can be automatically taken from
the HTTP stream and automatically converted to the right type. Also, the form widget can build forms from the service definition.

<H3><a name="grinder">Applicable Features of Grinder</a></H3>
There are multiple options for testing frameworks, but Grinder offers the following unique advantages.
<ul>
<li><b>A robust test script generator</b> - TCPProxy attempts to assign return parameters to variables and
    reuse them, rather than passing literals around. This means that it does not instantly break when keys are autogenerated.</li>
<li><b>Grinder has convenient customization point</b> - script generation is done via an XSLT stylesheet that makes use of Java extensions.
 This would be a natural place to make modifications to suit OFBTI. There are also filters for the requests and responses that can be
swapped out via command line parameters.</li>
<li><b>Jython</b> - allows seamless switching between a highly productive scripting environment and regular Java code.
Because of this feature, Jython has a lot of possibilities within OFBiz. </li>
<li><b>Integrated functional and jUnit testing</b> - the use of Jython allows for the same environment to be used to run HTTP client system tests
and jUnit functional tests.</li>
</ul>

<H2><a name="requirements">REQUIREMENTS</a></H2>
<H3><a name="generation">Easy Generation</a></H3>
The use of a test script generator, such as Grinder's TCPProxy, allows test scripts to be created by capturing the
input of a user at a browser.

<H3><a name="complex">Handle Complex Tests</a></H3>
A further enhancement would be to allow the creation of scripts by supplying a few data parameters or by
easily modifying automatically generated scripts.
One of the biggest drawbacks of end user testing environments is that when something changes, the associated test script
is usually unusable.
A big improvement would be to allow tests to be created by chaining together subtests.

<H3><a name="adaptable">Adaptable per Application</a></H3>
Many applications will have special characteristics that need to be handled specially.
In one instance, I found that Grinder was using the '$' character to form Python variable names (which doesn't work) because
that is how the HTTP parameters were named.
<H3><a name="rugged">Rugged</a></H3>
<H3><a name="iterations">Handle Iterations</a></H3>
The general idea of XP is that the tests become more complex and comprehensive as the code becomes more complete.
It would probably be a good idea to retain visibility to past test setups - though, at this time, I am not sure why.

<H3><a name="proficiency">Usable at Multiple Proficiency Levels</a></H3>
It would be good to have the testing environment useable, or at least understandable, by non-programmers.
JUnit tests would not meet this requirement.
Neither would Python scripts (though it could come close).
For project managers, an XML-based configuration environment would be needed.
<H3><a name="output">Simplify Output Testing</a></H3>
The analysis of the HTML pages that are returned by Grinder tests is one of the more problematic areas of end-user testing.
The more common scenarios would need to be handled by the XML-based configuration environment mentioned above.
The foreseeable cases would be matching of form values, existence or non-existence anywhere of a phrase or something
that can be tested by an XPath expression.

<H2><a name="plan">STRAWMAN PLAN</a></H2>
What is discussed below is how I see tackling the testing problem.
I am only interested in what will be supported by the community, so it can be changed.
<p/>
The overall approach is to use as much of Grinder as possible, and allow for the generated Grinder test scripts to be
overwritten by a user generated XML test configuration file.

<H3><a name="modify">Modify Test Script Generation</a></H3>
The Grinder TCPProxy program will be used, but the XSLT stylesheet and the associated Java extension classes will be enhanced.
The main change will be that the individual request tests that Grinder generates for each round trip to the server will be
wrapped by code that will check to see if there are data overrides to be made coming from the user generated high level test config file.
Each wrapper will also have a dictionary (ie. map) that defines tests to be made on each low level result.
These also will be modified by the higher level config file.
If there is not higher level config file, the script as generated by TCPProxy will run with no modifications.

<H4><a name="inoutmaps">Input Dictionary</a></H4>
The input directory will do more than just allow the user to supply literal values as test input.
There will be helper functions for allowing the user to randomly pick values from a list.
Also there will be helper functions for generating reasonably looking addresses.
The user will be able to create Python scripts to generated special input.
See the appendix for samples of what this would look like.
<H4><a name="inoutmaps">Output Dictionary</a></H4>
The output dictionary, which contains the test criteria, would have test helper functions of the following sort:
<ul>
<li><b>Literal value</b></li>
<li><b>Regular expression</b></li>
<li><b>Form value</b></li>
<li><b>XPath</b></li>
<li><b>Custom Jython scripts</b></li>
</ul>
Many of the test methods will be suitable for non-programmers, but the use of Jython scripts follows the OFBiz pattern
of providing simple methods for simple tasks, but making it easy to drop down to the level needed to solve the problem.
At some point (right away?), it will be necessary to allow complex joing of tests with AND, OR and NOT operators.
<H3><a name="higher">Add Test Overlay Config File</a></H3>
One of the modifications to the OOTB test script generated by TCPProxy will be that the script will look for the existence
a file path as a parameters and use it to overwrite the default test script values (ie. the values typed in by the user
when the script was being generated).
The script could have multiple levels of complex scripts, but eventually, they must call one of the "request" tests.
Keep in mind that the base test script could have a large number of "request" scripts, but they would not all need
to be used by the user-defined config file; it could just use a subset. So the base test script may not, in fact, be totally
generated by one session of user interaction with the system under test; it could be built up over time as new functionality
is added without having to run thru the test. The OOTB behavior is to sequentially number the tests, but in this case,
we may wish to name them with the request name that they interact with.
<p/>
These high level test scripts would need "include" functionality so that they could used standard building blocks.
<H3><a name="scc">Source Code Control to Handle Iterations</a></H3>
<H2><a name="appendices">Appendices</a></H2>
<H3><a name="stdcode">Standard Generated Grinder Test Script</a></H3>
This is the actual output from a TCPProxy session:
<p/>
<pre>
# The Grinder 3.0-beta30
# HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM

from net.grinder.script import Test
from net.grinder.script.Grinder import grinder
from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest
from HTTPClient import NVPair
connectionDefaults = HTTPPluginControl.getConnectionDefaults()
httpUtilities = HTTPPluginControl.getHTTPUtilities()

# To use a proxy server, uncomment the next line and set the host and port.
# connectionDefaults.setProxyServer("localhost", 8001)

# These definitions at the top level of the file are evaluated once,
# when the worker process is started.

connectionDefaults.defaultHeaders = \
  ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'),
    NVPair('Accept-Encoding', 'gzip,deflate'),
    NVPair('Accept-Language', 'en-us,en;q=0.5'),
    NVPair('Accept-Charset', 'UTF-8,*'),
    NVPair('Accept', 'text/xml,application/xml,application/xhtml+xml,text/html;q= 0.9,text/plain;q=0.8,image/png,*/*;q=0.5'), )

headers0= \
  ( )

headers1= \
  ( NVPair('Referer', 'https://localhost:8443/webtools/control/main' ), )

headers2= \
  ( NVPair('Referer', 'https://localhost:8443/webtools/control/checkLogin/main'), )

url0 = ' https://localhost:8443'

# Create an HTTPRequest for each request, then replace the
# reference to the HTTPRequest with an instrumented version.
# You can access the unadorned instance using request101.__target__.
request101 = HTTPRequest(url=url0, headers=headers0)
request101 = Test(101, 'GET /').wrap(request101)

request102 = HTTPRequest(url=url0, headers=headers0)
request102 = Test(102, 'GET main').wrap(request102)

request201 = HTTPRequest(url=url0, headers=headers1)
request201 = Test(201, 'GET main').wrap(request201)

request301 = HTTPRequest(url=url0, headers=headers2)
request301 = Test(301, 'POST login').wrap(request301)


class TestRunner:
  """A TestRunner instance is created for each worker thread."""

  # A method for each recorded page.
  def page1(self):
    """GET / (requests 101-102)."""
   
    # Expecting 302 'Moved Temporarily'
    result = request101.GET('/webtools/')

    grinder.sleep(16)
    request102.GET('/webtools/control/main')

    return result

  def page2(self):
    """GET main (request 201)."""
    result = request201.GET('/webtools/control/checkLogin/main')

    return result

  def page3(self):
    """POST login (request 301)."""
    result = request301.POST('/webtools/control/login',
      ( NVPair('USERNAME', 'admin'),
        NVPair('PASSWORD', 'ofbiz'), ),
      ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), ))

    return result

  def __call__(self):
    """This method is called for every run performed by the worker thread."""
    self.page1()      # GET / (requests 101-102)

    grinder.sleep(4328)
    self.page2()      # GET main (request 201)

    grinder.sleep(2109)
    self.page3()      # POST login (request 301)


def instrumentMethod(test, method_name, c=TestRunner):
  """Instrument a method with the given Test."""
  unadorned = getattr(c, method_name)
  import new
  method = new.instancemethod(test.wrap(unadorned), None, c)
  setattr(c, method_name, method)

# Replace each method with an instrumented version.
# You can call the unadorned method using self.page1.__target__().
instrumentMethod(Test(100, 'Page 1'), 'page1')
instrumentMethod(Test(200, 'Page 2'), 'page2')
instrumentMethod(Test(300, 'Page 3'), 'page3')

</pre>
<H3><a name="custmcode">Possible Custom Generated Grinder Test Script</a></H3>
<pre>
# The Grinder 3.0-beta30
# HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM

from net.grinder.script import Test
from net.grinder.script.Grinder import grinder
from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest
from HTTPClient import NVPair
import AGrinderTest

connectionDefaults = HTTPPluginControl.getConnectionDefaults()
httpUtilities = HTTPPluginControl.getHTTPUtilities()

# To use a proxy server, uncomment the next line and set the host and port.
# connectionDefaults.setProxyServer("localhost", 8001)

# These definitions at the top level of the file are evaluated once,
# when the worker process is started.

connectionDefaults.defaultHeaders = \
  ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'),
    NVPair('Accept-Encoding', 'gzip,deflate'),
    NVPair('Accept-Language', 'en-us,en;q= 0.5'),
    NVPair('Accept-Charset', 'UTF-8,*'),
    NVPair('Accept', 'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'), )

headers0= \
  ( )

headers1= \
  ( NVPair('Referer', 'https://localhost:8443/webtools/control/main'), )

headers2= \
  ( NVPair('Referer', ' https://localhost:8443/webtools/control/checkLogin/main'), )

url0 = 'https://localhost:8443'

# Create an HTTPRequest for each request, then replace the
# reference to the HTTPRequest with an instrumented version.
# You can access the unadorned instance using request101.__target__.
request101 = HTTPRequest(url=url0, headers=headers0)
request101 = Test(101, 'GET /').wrap(request101)

request102 = HTTPRequest(url=url0, headers=headers0)
request102 = Test(102, 'GET main').wrap(request102)

request201 = HTTPRequest(url=url0, headers=headers1)
request201 = Test(201, 'GET main').wrap(request201)

request301 = HTTPRequest(url=url0, headers=headers2)
request301 = Test(301, 'POST login').wrap(request301)


class TestRunner:
  """A TestRunner instance is created for each worker thread."""

  # A method for each recorded page.
  def page1(self):
    """GET / (requests 101-102)."""
   
    # Expecting 302 'Moved Temporarily'
    result = request101.GET('/webtools/')

    grinder.sleep(16)
    request102.GET('/webtools/control/main')

    return result

  def page2(self):
    """GET main (request 201)."""
    result = request201.GET('/webtools/control/checkLogin/main')

    return result

  def page3(self):
    """POST login (request 301)."""
    result = request301.POST('/webtools/control/login',
      ( NVPair('USERNAME', 'admin'),
        NVPair('PASSWORD', 'ofbiz'), ),
      ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), ))

    return result

 
    """This is not working code. Just an idea of how the generated code would look"""
  def webtoolsWrap(self, overlayMap):
    inMap = {}
    outMap = {}
    self.agrinder.runWrappedTest(page1, inMap, outMap, overlayMap)

  def mainWrap(self, overlayMap):
    inMap = {}
    outMap = {}
    self.agrinder.runWrappedTest(page2, inMap, outMap, overlayMap)

  def loginWrap(self, overlayMap):
    inMap = {'USERNAME':'admin', 'PASSWORD':'ofbiz'}
    outMap = {}
    self.agrinder.runWrappedTest (page2, inMap, outMap, overlayMap)

  def __call__(self):
    """This method is called for every run performed by the worker thread."""
   
    """I am going to do some handwaving here because I want to get this doc out today,
        but there would be code here to read in an argument from the command line and
    use it as a file path to read in the "test overlay" XML doc"""
    self.agrinder = AGrinderTest.AGrinder ()
    testNode = self.agrinder.getOverlayTest("webtools", sys.argv)
    self.webtoolsWrap(testNode)      # GET / (requests 101-102)

    grinder.sleep(4328)
    self.mainWrap(testNode)      # GET main (request 201)

    grinder.sleep(2109)
    self.loginWrap(testNode)      # POST login (request 301)


def instrumentMethod(test, method_name, c=TestRunner):
  """Instrument a method with the given Test."""
  unadorned = getattr(c, method_name)
  import new
  method = new.instancemethod(test.wrap(unadorned), None, c)
  setattr(c, method_name, method)

# Replace each method with an instrumented version.
# You can call the unadorned method using self.page1.__target__().
instrumentMethod(Test(100, 'Page 1'), 'page1')
instrumentMethod(Test(200, 'Page 2'), 'page2')
instrumentMethod(Test(300, 'Page 3'), 'page3')
</pre>
<H3><a name="overlay">Possible Overlay XML File</a></H3>
This example is probably not even close to what the final test overlay XML script will look like,
but I think it is better to have something to go from.
<pre>
&lt;tests&gt;
  &lt;test name="webtools"&gt;
    &lt;subtest name="login"&gt;
      &lt;input&gt;
        &lt;field name="USERNAME"&gt;jdoe&lt;/field&gt;
        &lt;field name="PASSWORD"&gt;id9Ed3jk&lt;/field&gt;
      &lt;/input&gt;
      &lt;match&gt;
        &lt;field type="regex" op="not"&gt;not found&lt;/field&gt;
        &lt;field type="script" op="true"&gt;isSuccess()&lt;/field&gt;
      &lt;/match&gt;
    &lt;/subtest&gt;
  &lt;/test&gt;
&lt;/tests&gt;
</pre>