More on testing framework

[Cameron Smith-6]

Hi all, I have been following the recent discussions about Ofbiz testing framework(s) with interest, and I thought I would share our experiences.  I intentionally include ofbiz-dev as there seems to be dev work ongoing on the test framework at the moment.  We are integrating an existing application with Ofbiz, effectively it runs as an Ofbiz module.  Our development approach is fairly test intensive, so we have spent a while trying to be able to maintain that approach with Ofbiz - it was not enough simply to get the integration working at the level of the running application.

========== current situation =================

We have 3 kinds of tests-
 *Unit Tests proper - run as JUnit tests
 *White box integration tests - also run as JUnit tests, but they run components via the same Spring configuration as the actual application, to test that too
*Black box web tests - run via Watir/Ruby which exercises the browser, which allows us to test DHTML behaviour also.   Watir with Ruby is a very similar idea to Grinder with Python - it is VERY useful to be able to express the tests in a high-level language, and code logical testing constructs as high level entities.  We are a Java shop here but Ruby is MUCH faster than Java for preparing tests, in fact we now have so little extraneous syntax (which would be inescapable in Java), that the test scripts are pretty much readable as documents.  I won't discuss Ruby vs. Python here, I think they are both good languages.  The big idea is not to clunk around in Java!  And I think a "normal syntax" imperative language like Rb or Py wins out here over XML, for certain things like if/else or looping, XML-based languages are just an ugly pain - look at Ant for example, when you have to put any control logic in.

*Data Loading. For data loading, we use a custom script build on top of Ruby/Rake which calls MySQL command -line executables.  Definitely less flexible than Entity Engine, nor does it do as much but very fast (3s on a dev machine to rebuild and repopulate about 100 tables).

========== the Ofbiz Experience =================

The Unit Tests proper, and the Web tests, basically ran with almost no alterations, once we had the application running inside Ofbiz.

The interesting parts were the integration tests, and the data loading.  Our key problem was time - we run tests constantly, and a particular rule is that a dev must run all tests succesfully before committing.  In a typical 4hr session he may commit 3 times, so if the tests take 5mins longer, in a team of 4 we can easily lose 1hr per session.

*Data loading.
We tweaked the main build.xml inside Ofbiz to give us a data load task which does NOT rebuild Ofbiz (as we donºt alter Ofbiz itself, just use it), and ONLY loads seed data files (in Entity Engine format) from a certain directory. This was easy to do as the Entity Engine Data Loader already offers flexibility in this regard. We still need to put data cleaning in there.  

*Integration tests.
We use the new testtools stuff prepared by David.  On the whole it works but we have a few tweaks and suggestions.
**Tweaks.
In framework/base/config/test-containers.xml we removed everything except component-container, classloader-container, testtools-container.  This improves startup time by about 10%

**Suggestions (David, if you think these have merit, I'd be happy to code them and submit for your consideration.  I am running against ofbiz_opentaps_453703 in case any have already been done).
1. You should be able to tell TestRunContainer to load only tests from a certain dir or even file.  This would allow people to run test cycles at the level they want, obviously with speed benefits.
2. It should be possible to tell the Entity Engine, via component-container, NOT to do an Enitty check against the DB when it starts, as 9 times out of 10 nothing has changed and the dev knows it.  Again, the objective is to speed up the test cycle.  Perhaps this option already exists and I just couldnºt find out how to set it!

cameron

P.S. good luck with the conference, sounds good.  I live v. far away so won't be attending!



       
       
               
___________________________________________________________
All new Yahoo! Mail "The new Interface is stunning in its simplicity and ease of use." - PC Magazine
http://uk.docs.yahoo.com/nowyoucan.html