ofbiz testing capabilities

[Wojciech Biela]

Hello

I recently looked at ofbiz and am now evaluating whether or not we
should go with it or not. We are a software development company.

The key thing for me at this point is testability (both unit- and
functional-level). I have a great concern because I hear that only
recently people are starting to think about it on the ofbiz list, and
the project dates back to 2001, so there is a whole lot of legacy code
that is not tested in any (automated) way?

I read Cameron Smith's post from Nov 1 2006 "More on testing
framework" where he said that he had "Unit Tests proper - run as JUnit
tests", could any of you (or you Cameron) elaborate a bit about those
tests in the context of ofbiz
- how painful was it
- what components did you test
- what things did you mock
- what do you suggest

Unfortunately I didn't find this information in the documentation and
a preliminary google search on this subject didn't reveal any
revelations either. I only found that you are taking about functional
testing. But then I ask what about unit level tests? I see junit.jar
in the sources but I then again I see the build process does not
invoke unit tests (there is a separate ant task for it), I read that
there is a total of 24 unit tests.... There was some info that the
topic is covered in the advanced (commercial?) documentation by
Undersun, is there anything apache-licenced ;) on that?

Is the architecture of ofbiz testable or not ? Is it possible to write
unit test (i.e. isolated unit tests) as well as integration tests
(also through junit).

This is very important to me as we try to follow the Test-Driven
Development practice and thus testability is the key concern to me. I
like very much what you've done here, but I am afraid whether we will
be able to safely play around with it and alter it for our customer
without killing it.

Concerning functional and acceptance testing have anyone considered
using FitNesse, it's a really glamorous tool for holding acceptance
criteria and running them against the system under test, additionally
it's a great collaboration tool (it's a wiki). If you really really
(as in: you rarely should) need to check some things though the
browser interface then there are perfect examples of calling Selenium
from FitNesse if anyone is interested I may point you to such
materials.

This is especially important if you would like to clearly specify
functionality (define acceptance tests) ahead of implementing this
functionality, recording stuff with the Selenium recorder is more or
less fine for legacy stuff but that's not always the case.

BTW please tell me how hard will it be to degrade the process of
adding products, promotions to something simpler, our client wants
some specific bits of what I see in the demo app, so I'm thinking
about simplifying things for him as his main request is simplicity.

Wojtek Biela
www.exorigo.pl

[Jacques Le Roux]

Hi,

Just some thoughts. OFBiz is complex at the technological level *and* at the bisnness (functionnal) level.
Perhaps a difficulty for testing with a project like OFBiz is that there are not only java programs but also plenty of minilang, bsh
and ftl... So something like Selenium is also needed.

I think that parts like Entity Engine and Service Engine are already well validated (used in Jira for instance for EE). Of course
any modifications may introduce regression. And that does not mean that we should not do more in this field and testing in general
and there are aleready some plans.

You found http://www.nabble.com/More-on-testing-tf2743100.html
there are also http://www.nabble.com/More-on-testing-framework-tf2551536.html
http://www.nabble.com/Database-Test-Suite-tf2496760.html
http://www.nabble.com/OFBiz-Testing-Initiative-tf2555099.html
http://www.nabble.com/Dev---Automated-Testing-in-OFBiz-tf1781243.html
http://www.nabble.com/Automation-testing-in-ofbiz-tf2169543.html
http://www.nabble.com/More-on-automating-testing-tf2183494.html
and certainly more but these are the most recent (ordered from recent to older)

There is also a point about this in Developers Conference, early 2007
http://www.nabble.com/Developers-Conference%2C-early-2007-tf2925184.html

I let more advanded people in this field answer accuratly at you questions.

Of course any efforts from your part will be greatly appreciated, remember it's open source...

Jacques



----- Original Message -----
From: "Wojciech Biela" <[hidden email]>
To: <[hidden email]>
Cc: <[hidden email]>
Sent: Monday, January 15, 2007 11:45 AM
Subject: ofbiz testing capabilities

> Hello
>
> I recently looked at ofbiz and am now evaluating whether or not we
> should go with it or not. We are a software development company.
>
> The key thing for me at this point is testability (both unit- and
> functional-level). I have a great concern because I hear that only
> recently people are starting to think about it on the ofbiz list, and
> the project dates back to 2001, so there is a whole lot of legacy code
> that is not tested in any (automated) way?
>
> I read Cameron Smith's post from Nov 1 2006 "More on testing
> framework" where he said that he had "Unit Tests proper - run as JUnit
> tests", could any of you (or you Cameron) elaborate a bit about those
> tests in the context of ofbiz
> - how painful was it
> - what components did you test
> - what things did you mock
> - what do you suggest
>
> Unfortunately I didn't find this information in the documentation and
> a preliminary google search on this subject didn't reveal any
> revelations either. I only found that you are taking about functional
> testing. But then I ask what about unit level tests? I see junit.jar
> in the sources but I then again I see the build process does not
> invoke unit tests (there is a separate ant task for it), I read that
> there is a total of 24 unit tests.... There was some info that the
> topic is covered in the advanced (commercial?) documentation by
> Undersun, is there anything apache-licenced ;) on that?
>
> Is the architecture of ofbiz testable or not ? Is it possible to write
> unit test (i.e. isolated unit tests) as well as integration tests
> (also through junit).
>
> This is very important to me as we try to follow the Test-Driven
> Development practice and thus testability is the key concern to me. I
> like very much what you've done here, but I am afraid whether we will
> be able to safely play around with it and alter it for our customer
> without killing it.
>
> Concerning functional and acceptance testing have anyone considered
> using FitNesse, it's a really glamorous tool for holding acceptance
> criteria and running them against the system under test, additionally
> it's a great collaboration tool (it's a wiki). If you really really
> (as in: you rarely should) need to check some things though the
> browser interface then there are perfect examples of calling Selenium
> from FitNesse if anyone is interested I may point you to such
> materials.
>
> This is especially important if you would like to clearly specify
> functionality (define acceptance tests) ahead of implementing this
> functionality, recording stuff with the Selenium recorder is more or
> less fine for legacy stuff but that's not always the case.
>
> BTW please tell me how hard will it be to degrade the process of
> adding products, promotions to something simpler, our client wants
> some specific bits of what I see in the demo app, so I'm thinking
> about simplifying things for him as his main request is simplicity.
>
> Wojtek Biela
> www.exorigo.pl

[Wojciech Biela]

Hello

> programs but also plenty of minilang, bsh
> and ftl... So something like Selenium is also needed.

well bsh could be tested from pure FitNesse with no problem, page
templates obviously would have to be rendered and tested through
Selenium, but my point was that you can run Selenium (Selenium RC)
tests from FitNesse statically or dynamically, and I think I don't
have to point out the benefit of having one tool/place to gather the
functional and acceptance tests (as opposed to multiple high level
test repositories)

BTW why do you use BSH and not Groovy (which now becomes the de facto
scripting language for the Java VM)? is there any problem with it, or
wasn't it just mature enough when you started?

> I think that parts like Entity Engine and Service Engine are already well validated

manually or automatically?

> and certainly more but these are the most recent (ordered from recent to older)
thank you, will look into it right away

> There is also a point about this in Developers Conference, early 2007
> http://www.nabble.com/Developers-Conference%2C-early-2007-tf2925184.html

cool, I'll be happy to see how it goes as I'm stationed in Europe so I
can't come, (as in I don't have the budged available, I'm saving it
for www.xp2007.org ;) )

> I let more advanded people in this field answer accuratly at you questions.

can't wait to hear from them!

> Of course any efforts from your part will be greatly appreciated, remember it's open source...

Sure, I'll be glad to chip in some ideas/exp on automated testing ...
and possibly code when I come to that point

best regards
Wojtek

>
> ----- Original Message -----
> From: "Wojciech Biela" <[hidden email]>
> To: <[hidden email]>
> Cc: <[hidden email]>
> Sent: Monday, January 15, 2007 11:45 AM
> Subject: ofbiz testing capabilities
>
>
> > Hello
> >
> > I recently looked at ofbiz and am now evaluating whether or not we
> > should go with it or not. We are a software development company.
> >
> > The key thing for me at this point is testability (both unit- and
> > functional-level). I have a great concern because I hear that only
> > recently people are starting to think about it on the ofbiz list, and
> > the project dates back to 2001, so there is a whole lot of legacy code
> > that is not tested in any (automated) way?
> >
> > I read Cameron Smith's post from Nov 1 2006 "More on testing
> > framework" where he said that he had "Unit Tests proper - run as JUnit
> > tests", could any of you (or you Cameron) elaborate a bit about those
> > tests in the context of ofbiz
> > - how painful was it
> > - what components did you test
> > - what things did you mock
> > - what do you suggest
> >
> > Unfortunately I didn't find this information in the documentation and
> > a preliminary google search on this subject didn't reveal any
> > revelations either. I only found that you are taking about functional
> > testing. But then I ask what about unit level tests? I see junit.jar
> > in the sources but I then again I see the build process does not
> > invoke unit tests (there is a separate ant task for it), I read that
> > there is a total of 24 unit tests.... There was some info that the
> > topic is covered in the advanced (commercial?) documentation by
> > Undersun, is there anything apache-licenced ;) on that?
> >
> > Is the architecture of ofbiz testable or not ? Is it possible to write
> > unit test (i.e. isolated unit tests) as well as integration tests
> > (also through junit).
> >
> > This is very important to me as we try to follow the Test-Driven
> > Development practice and thus testability is the key concern to me. I
> > like very much what you've done here, but I am afraid whether we will
> > be able to safely play around with it and alter it for our customer
> > without killing it.
> >
> > Concerning functional and acceptance testing have anyone considered
> > using FitNesse, it's a really glamorous tool for holding acceptance
> > criteria and running them against the system under test, additionally
> > it's a great collaboration tool (it's a wiki). If you really really
> > (as in: you rarely should) need to check some things though the
> > browser interface then there are perfect examples of calling Selenium
> > from FitNesse if anyone is interested I may point you to such
> > materials.
> >
> > This is especially important if you would like to clearly specify
> > functionality (define acceptance tests) ahead of implementing this
> > functionality, recording stuff with the Selenium recorder is more or
> > less fine for legacy stuff but that's not always the case.
> >
> > BTW please tell me how hard will it be to degrade the process of
> > adding products, promotions to something simpler, our client wants
> > some specific bits of what I see in the demo app, so I'm thinking
> > about simplifying things for him as his main request is simplicity.
> >
> > Wojtek Biela
> > www.exorigo.pl
>
>

--
Wojtek Biela

[Jacques Le Roux]

Wojciech,

From: "Wojciech Biela" <[hidden email]>
> > Entity Engine is used by thousand of users all the time (do you know Jira ?). Service Engine is used in all OFBiz
implementations
> > around the world without known problems.
>
> yeah, I know Jira and it's popularity ;) my concern is not having an
> automated test suite verifying that we don't break anything during our
> further development, would not want to be the one chased by those
> "thousand users all the time" because I broke something with my code
> ... but as you are (we are ;) ) moving towards a testing tool for
> ofbiz that will change I hope

Frightening ;o)

> as for functional/acceptance tests they will provide smoke tests to
> tell us "something is wrong", but I would really really like to have
> unit level tests for the code I write (and modify!!!) to keep me going
> and refactoring fast. that's why I also asked about your exp with
> this, how easy is it to mock/decouple the code you write from the
> framework and test it thoroughly at the unit level, or even better
> test-drive it


I must aknowledge that I yet never used any testing tools in OFBiz (and I'm sure I'm not alone in this case).
So apart the infos I sent you I will be of little help in this area.
However I clearly understand importance of this and would be happy too to escape from thousands of angry users ;o)

[...snip...]

> > Como sounds great ;o)
>
> it sure does !! :D you're all welcome :)

Jacques

> --
> Wojtek

[David E Jones]

Wojtek,

While it's great that Jacques has answered and shared his thoughts  
and experience on this, please don't consider that a full  
representation of the state of testing in OFBiz.

You should know that testing has been considered since day 1, but to  
date I personally have not had a single client willing to sponsor any  
sort of testing effort. Combine this with the fact that the framework  
is made up mostly of business-level tools and we are fortunately in a  
place where there are few low level bugs that require low-level unit  
tests. We still definitely have a need for 2 things in the testing area:

1. regression testing for applications, mostly on the service level  
(even if driven through the web-based UI)
2. tools and infrastructure in place for pre-development acceptance  
testing

Note that there is already a pretty good test infrastructure in place  
and it is used right now to run the JUnit tests for various of the  
framework components (mostly the entity and service engines). It just  
needs to be extended to support easy execution of testing scripts  
created by other tools. The main ones we are considering are Grinder  
and Canoo WebTest.

Whether you use OFBiz or not is your choice and if it works well for  
you, great, if not that's fine too.

A lot of people talk about testing, but so far little has been  
contributed in this area, largely I think because it has always been  
a nice to have, and no one with sufficient resources has invested  
sufficiently in it. That's really all it comes down to. I developed  
the current testing framework in the testtools component and while  
functional for junit tests it is far from complete. I have nothing  
else to invest in this as other obligations with higher priority  
won't allow it, so help from others is, as with most things, necessary.

But yes, OFBiz is inherently testable and includes a great deal of  
functionality for creating and automatically running sets of tests.  
There are even nice tools built around the entity engine for adding  
data to the database and asserting that the data in the database  
matches a certain state.

-David


On Jan 15, 2007, at 3:45 AM, Wojciech Biela wrote:

> Hello
>
> I recently looked at ofbiz and am now evaluating whether or not we
> should go with it or not. We are a software development company.
>
> The key thing for me at this point is testability (both unit- and
> functional-level). I have a great concern because I hear that only
> recently people are starting to think about it on the ofbiz list, and
> the project dates back to 2001, so there is a whole lot of legacy code
> that is not tested in any (automated) way?
>
> I read Cameron Smith's post from Nov 1 2006 "More on testing
> framework" where he said that he had "Unit Tests proper - run as JUnit
> tests", could any of you (or you Cameron) elaborate a bit about those
> tests in the context of ofbiz
> - how painful was it
> - what components did you test
> - what things did you mock
> - what do you suggest
>
> Unfortunately I didn't find this information in the documentation and
> a preliminary google search on this subject didn't reveal any
> revelations either. I only found that you are taking about functional
> testing. But then I ask what about unit level tests? I see junit.jar
> in the sources but I then again I see the build process does not
> invoke unit tests (there is a separate ant task for it), I read that
> there is a total of 24 unit tests.... There was some info that the
> topic is covered in the advanced (commercial?) documentation by
> Undersun, is there anything apache-licenced ;) on that?
>
> Is the architecture of ofbiz testable or not ? Is it possible to write
> unit test (i.e. isolated unit tests) as well as integration tests
> (also through junit).
>
> This is very important to me as we try to follow the Test-Driven
> Development practice and thus testability is the key concern to me. I
> like very much what you've done here, but I am afraid whether we will
> be able to safely play around with it and alter it for our customer
> without killing it.
>
> Concerning functional and acceptance testing have anyone considered
> using FitNesse, it's a really glamorous tool for holding acceptance
> criteria and running them against the system under test, additionally
> it's a great collaboration tool (it's a wiki). If you really really
> (as in: you rarely should) need to check some things though the
> browser interface then there are perfect examples of calling Selenium
> from FitNesse if anyone is interested I may point you to such
> materials.
>
> This is especially important if you would like to clearly specify
> functionality (define acceptance tests) ahead of implementing this
> functionality, recording stuff with the Selenium recorder is more or
> less fine for legacy stuff but that's not always the case.
>
> BTW please tell me how hard will it be to degrade the process of
> adding products, promotions to something simpler, our client wants
> some specific bits of what I see in the demo app, so I'm thinking
> about simplifying things for him as his main request is simplicity.
>
> Wojtek Biela
> www.exorigo.pl

[jonwimp]

Wojtek,

Looks like you're asking for 2 things here: testability and extensibility.

Please note that my answers do not represent OFBiz community's, just my opinion.

For testability, you need documentation of OFBiz. At least functional docs. Such docs are really
sparse at the moment, so the issue of testability is rendered moot given the lack of this
prerequisite.

I would suggest you create your own set of test cases based on your own business logics. After
all, you want to test your customized OFBiz against your client's set of requirements, not the
OFBiz community's.

(At some point, perhaps even now, the community could be working to amass docs on OFBiz,
functionally speaking.)

For extensibility, I can just say 1 word: total. I've not seen anything as extensible as this.
Between the entity engine, the widget engine, and the tried-and-tested integration of Beanshell,
FOP, Freemarker, etc, OFBiz is a staggeringly large piece of "tutorial by example". Of course,
it's being opensource is also a major factor.

I would suggest you spend some time to get up to speed with the OFBiz framework, perhaps through
the training videos or buy some training seminars from the community. Once you get a "once
through" (end-to-end) tutorial, you'll be hooked on OFBiz.

(No, I do not provide such training seminars. I've never bought any training from the community or
undersun consulting. I've never been paid by anyone here. I took apart OFBiz myself like I would a
ham radio, but that's my personal preference to sweat things out myself big time.)

Jonathon

David E. Jones wrote:

>
> Wojtek,
>
> While it's great that Jacques has answered and shared his thoughts and
> experience on this, please don't consider that a full representation of
> the state of testing in OFBiz.
>
> You should know that testing has been considered since day 1, but to
> date I personally have not had a single client willing to sponsor any
> sort of testing effort. Combine this with the fact that the framework is
> made up mostly of business-level tools and we are fortunately in a place
> where there are few low level bugs that require low-level unit tests. We
> still definitely have a need for 2 things in the testing area:
>
> 1. regression testing for applications, mostly on the service level
> (even if driven through the web-based UI)
> 2. tools and infrastructure in place for pre-development acceptance testing
>
> Note that there is already a pretty good test infrastructure in place
> and it is used right now to run the JUnit tests for various of the
> framework components (mostly the entity and service engines). It just
> needs to be extended to support easy execution of testing scripts
> created by other tools. The main ones we are considering are Grinder and
> Canoo WebTest.
>
> Whether you use OFBiz or not is your choice and if it works well for
> you, great, if not that's fine too.
>
> A lot of people talk about testing, but so far little has been
> contributed in this area, largely I think because it has always been a
> nice to have, and no one with sufficient resources has invested
> sufficiently in it. That's really all it comes down to. I developed the
> current testing framework in the testtools component and while
> functional for junit tests it is far from complete. I have nothing else
> to invest in this as other obligations with higher priority won't allow
> it, so help from others is, as with most things, necessary.
>
> But yes, OFBiz is inherently testable and includes a great deal of
> functionality for creating and automatically running sets of tests.
> There are even nice tools built around the entity engine for adding data
> to the database and asserting that the data in the database matches a
> certain state.
>
> -David
>
>
> On Jan 15, 2007, at 3:45 AM, Wojciech Biela wrote:
>
>> Hello
>>
>> I recently looked at ofbiz and am now evaluating whether or not we
>> should go with it or not. We are a software development company.
>>
>> The key thing for me at this point is testability (both unit- and
>> functional-level). I have a great concern because I hear that only
>> recently people are starting to think about it on the ofbiz list, and
>> the project dates back to 2001, so there is a whole lot of legacy code
>> that is not tested in any (automated) way?
>>
>> I read Cameron Smith's post from Nov 1 2006 "More on testing
>> framework" where he said that he had "Unit Tests proper - run as JUnit
>> tests", could any of you (or you Cameron) elaborate a bit about those
>> tests in the context of ofbiz
>> - how painful was it
>> - what components did you test
>> - what things did you mock
>> - what do you suggest
>>
>> Unfortunately I didn't find this information in the documentation and
>> a preliminary google search on this subject didn't reveal any
>> revelations either. I only found that you are taking about functional
>> testing. But then I ask what about unit level tests? I see junit.jar
>> in the sources but I then again I see the build process does not
>> invoke unit tests (there is a separate ant task for it), I read that
>> there is a total of 24 unit tests.... There was some info that the
>> topic is covered in the advanced (commercial?) documentation by
>> Undersun, is there anything apache-licenced ;) on that?
>>
>> Is the architecture of ofbiz testable or not ? Is it possible to write
>> unit test (i.e. isolated unit tests) as well as integration tests
>> (also through junit).
>>
>> This is very important to me as we try to follow the Test-Driven
>> Development practice and thus testability is the key concern to me. I
>> like very much what you've done here, but I am afraid whether we will
>> be able to safely play around with it and alter it for our customer
>> without killing it.
>>
>> Concerning functional and acceptance testing have anyone considered
>> using FitNesse, it's a really glamorous tool for holding acceptance
>> criteria and running them against the system under test, additionally
>> it's a great collaboration tool (it's a wiki). If you really really
>> (as in: you rarely should) need to check some things though the
>> browser interface then there are perfect examples of calling Selenium
>> from FitNesse if anyone is interested I may point you to such
>> materials.
>>
>> This is especially important if you would like to clearly specify
>> functionality (define acceptance tests) ahead of implementing this
>> functionality, recording stuff with the Selenium recorder is more or
>> less fine for legacy stuff but that's not always the case.
>>
>> BTW please tell me how hard will it be to degrade the process of
>> adding products, promotions to something simpler, our client wants
>> some specific bits of what I see in the demo app, so I'm thinking
>> about simplifying things for him as his main request is simplicity.
>>
>> Wojtek Biela
>> www.exorigo.pl
>

[Cameron Smith-6]

Hi Wojtek.

In reply to your general question, I do believe that
OFBiz is eminently testable, in that you can identify
distinct pieces of functionality at various levels and
test them by automatic means.

Re David's point about the need for practical
intervention and not just talk, I did in fact offer to
help with this (improvements to the testing framework)
a while back.  However the feedback I got from David
and others to the suggestions I made (before I
implemented them) was basically nil.

Given this lack of interest, I left it for the moment,
however in the next couple of months I will have to
improve the test framework anyway for my company's
internal purposes and I would be happy to contribute
that back to OFBiz.

Meanwhile, I suggest that the best way to go forward
would be to look at the existing tests and work out
how to use the framework from there, as there does not
appear to be any up-to-date documentation.  Basically
the starting point is the ofbiz-component.xml file for
a particular module, which points at unit test
definition files, via a <test-resource> tag.  (I am
away from the office at the mo so cannot check the
exact tag name).

If you have specific problems using the framework,
post them here and I will respond, but, like David, I
am not up for general testing discussions.  Pretty
much everything general there is to say, has been
discussed already on the mailing list archives

cameron

> Wojtek,
>
> While it's great that Jacques has answered and
> shared his thoughts  
> and experience on this, please don't consider that a
> full  
> representation of the state of testing in OFBiz.
>
> You should know that testing has been considered
> since day 1, but to  
> date I personally have not had a single client
> willing to sponsor any  
> sort of testing effort. Combine this with the fact
> that the framework  
> is made up mostly of business-level tools and we are
> fortunately in a  
> place where there are few low level bugs that
> require low-level unit  
> tests. We still definitely have a need for 2 things
> in the testing area:
>
> 1. regression testing for applications, mostly on
> the service level  
> (even if driven through the web-based UI)
> 2. tools and infrastructure in place for
> pre-development acceptance  
> testing
>
> Note that there is already a pretty good test
> infrastructure in place  
> and it is used right now to run the JUnit tests for
> various of the  
> framework components (mostly the entity and service
> engines). It just  
> needs to be extended to support easy execution of
> testing scripts  
> created by other tools. The main ones we are
> considering are Grinder  
> and Canoo WebTest.
>
> Whether you use OFBiz or not is your choice and if
> it works well for  
> you, great, if not that's fine too.
>
> A lot of people talk about testing, but so far
> little has been  
> contributed in this area, largely I think because it
> has always been  
> a nice to have, and no one with sufficient resources
> has invested  
> sufficiently in it. That's really all it comes down
> to. I developed  
> the current testing framework in the testtools
> component and while  
> functional for junit tests it is far from complete.
> I have nothing  
> else to invest in this as other obligations with
> higher priority  
> won't allow it, so help from others is, as with most
> things, necessary.
>
> But yes, OFBiz is inherently testable and includes a
> great deal of  
> functionality for creating and automatically running
> sets of tests.  
> There are even nice tools built around the entity
> engine for adding  
> data to the database and asserting that the data in
> the database  
> matches a certain state.
>
> -David
>
>
> On Jan 15, 2007, at 3:45 AM, Wojciech Biela wrote:
>
> > Hello
> >
> > I recently looked at ofbiz and am now evaluating
> whether or not we
> > should go with it or not. We are a software
> development company.
> >
> > The key thing for me at this point is testability
> (both unit- and
> > functional-level). I have a great concern because
> I hear that only
> > recently people are starting to think about it on
> the ofbiz list, and
> > the project dates back to 2001, so there is a
> whole lot of legacy code
> > that is not tested in any (automated) way?
> >
> > I read Cameron Smith's post from Nov 1 2006 "More
> on testing
> > framework" where he said that he had "Unit Tests
> proper - run as JUnit
> > tests", could any of you (or you Cameron)
> elaborate a bit about those
> > tests in the context of ofbiz
> > - how painful was it
> > - what components did you test
> > - what things did you mock
> > - what do you suggest
> >
> > Unfortunately I didn't find this information in
> the documentation and
> > a preliminary google search on this subject didn't
> reveal any
> > revelations either. I only found that you are
> taking about functional
> > testing. But then I ask what about unit level
> tests? I see junit.jar
> > in the sources but I then again I see the build
> process does not
> > invoke unit tests (there is a separate ant task
> for it), I read that
> > there is a total of 24 unit tests.... There was
> some info that the
> > topic is covered in the advanced (commercial?)
> documentation by
> > Undersun, is there anything apache-licenced ;) on
> that?
> >
> > Is the architecture of ofbiz testable or not ? Is
> it possible to write
> > unit test (i.e. isolated unit tests) as well as
> integration tests
> > (also through junit).
> >
> > This is very important to me as we try to follow
> the Test-Driven
> > Development practice and thus testability is the
> key concern to me. I
> > like very much what you've done here, but I am
> afraid whether we will
> > be able to safely play around with it and alter it
> for our customer
> > without killing it.
> >
> > Concerning functional and acceptance testing have
> anyone considered
> > using FitNesse, it's a really glamorous tool for
> holding acceptance
> > criteria and running them against the system under
> test, additionally
> > it's a great collaboration tool (it's a wiki). If
> you really really
> > (as in: you rarely should) need to check some
> things though the
> > browser interface then there are perfect examples
> of calling Selenium
> > from FitNesse if anyone is interested I may point
> you to such
> > materials.
> >
> > This is especially important if you would like to
> clearly specify
> > functionality (define acceptance tests) ahead of
> implementing this
> > functionality, recording stuff with the Selenium
> recorder is more or
> > less fine for legacy stuff but that's not always
> the case.
> >
> > BTW please tell me how hard will it be to degrade
> the process of
> > adding products, promotions to something simpler,
> our client wants
> > some specific bits of what I see in the demo app,
> so I'm thinking
> > about simplifying things for him as his main
> request is simplicity.
> >
> > Wojtek Biela
> > www.exorigo.pl
>
>

               
___________________________________________________________
What kind of emailer are you? Find out today - get a free analysis of your email personality. Take the quiz at the Yahoo! Mail Championship.
http://uk.rd.yahoo.com/evt=44106/*http://mail.yahoo.net/uk 

[Wojciech Biela]

Hello Jonathon

> For testability, you need documentation of OFBiz. At least functional docs. Such docs are really
> sparse at the moment, so the issue of testability is rendered moot given the lack of this
> prerequisite.

Sigh.

> I would suggest you create your own set of test cases based on your own business logics. After
> all, you want to test your customized OFBiz against your client's set of requirements, not the
> OFBiz community's.

true, at least for the functional/acceptance layer of testing

> For extensibility, I can just say 1 word: total. I've not seen anything as extensible as this.

that frightens me a bit, I've seen stuff very extensible and very
useless because totally complicated to do simple stuff and changes.
Don't shoot! I don't say ofBiz is like that ;) I just say what I'm
afraid of, that all the configuration will kill development (as did
e.g. a load of descriptors kill EJB (pre 3.0)),

then again I like what I see, really, though I don't necessarily
understand yet e.g. the absence of OR mapping I hope to realise that
soon

I just hope I'll be able to properly test the components I write (have
automated tests).

> it's being opensource is also a major factor.

it's what matters to me, otherwise I'd be developing apps in WS
Commerce by now ;)


> I would suggest you spend some time to get up to speed with the OFBiz framework, perhaps through
> the training videos or buy some training seminars from the community. Once you get a "once
> through" (end-to-end) tutorial, you'll be hooked on OFBiz.

am right in the middle of that ;)

> (No, I do not provide such training seminars. I've never bought any training from the community or
> undersun consulting. I've never been paid by anyone here. I took apart OFBiz myself like I would a
> ham radio, but that's my personal preference to sweat things out myself big time.)

cool :)

thanks a lot
Wojtek

> >
> David E. Jones wrote:
> >
> > Wojtek,
> >
> > While it's great that Jacques has answered and shared his thoughts and
> > experience on this, please don't consider that a full representation of
> > the state of testing in OFBiz.
> >
> > You should know that testing has been considered since day 1, but to
> > date I personally have not had a single client willing to sponsor any
> > sort of testing effort. Combine this with the fact that the framework is
> > made up mostly of business-level tools and we are fortunately in a place
> > where there are few low level bugs that require low-level unit tests. We
> > still definitely have a need for 2 things in the testing area:
> >
> > 1. regression testing for applications, mostly on the service level
> > (even if driven through the web-based UI)
> > 2. tools and infrastructure in place for pre-development acceptance testing
> >
> > Note that there is already a pretty good test infrastructure in place
> > and it is used right now to run the JUnit tests for various of the
> > framework components (mostly the entity and service engines). It just
> > needs to be extended to support easy execution of testing scripts
> > created by other tools. The main ones we are considering are Grinder and
> > Canoo WebTest.
> >
> > Whether you use OFBiz or not is your choice and if it works well for
> > you, great, if not that's fine too.
> >
> > A lot of people talk about testing, but so far little has been
> > contributed in this area, largely I think because it has always been a
> > nice to have, and no one with sufficient resources has invested
> > sufficiently in it. That's really all it comes down to. I developed the
> > current testing framework in the testtools component and while
> > functional for junit tests it is far from complete. I have nothing else
> > to invest in this as other obligations with higher priority won't allow
> > it, so help from others is, as with most things, necessary.
> >
> > But yes, OFBiz is inherently testable and includes a great deal of
> > functionality for creating and automatically running sets of tests.
> > There are even nice tools built around the entity engine for adding data
> > to the database and asserting that the data in the database matches a
> > certain state.
> >
> > -David
> >
> >
> > On Jan 15, 2007, at 3:45 AM, Wojciech Biela wrote:
> >
> >> Hello
> >>
> >> I recently looked at ofbiz and am now evaluating whether or not we
> >> should go with it or not. We are a software development company.
> >>
> >> The key thing for me at this point is testability (both unit- and
> >> functional-level). I have a great concern because I hear that only
> >> recently people are starting to think about it on the ofbiz list, and
> >> the project dates back to 2001, so there is a whole lot of legacy code
> >> that is not tested in any (automated) way?
> >>
> >> I read Cameron Smith's post from Nov 1 2006 "More on testing
> >> framework" where he said that he had "Unit Tests proper - run as JUnit
> >> tests", could any of you (or you Cameron) elaborate a bit about those
> >> tests in the context of ofbiz
> >> - how painful was it
> >> - what components did you test
> >> - what things did you mock
> >> - what do you suggest
> >>
> >> Unfortunately I didn't find this information in the documentation and
> >> a preliminary google search on this subject didn't reveal any
> >> revelations either. I only found that you are taking about functional
> >> testing. But then I ask what about unit level tests? I see junit.jar
> >> in the sources but I then again I see the build process does not
> >> invoke unit tests (there is a separate ant task for it), I read that
> >> there is a total of 24 unit tests.... There was some info that the
> >> topic is covered in the advanced (commercial?) documentation by
> >> Undersun, is there anything apache-licenced ;) on that?
> >>
> >> Is the architecture of ofbiz testable or not ? Is it possible to write
> >> unit test (i.e. isolated unit tests) as well as integration tests
> >> (also through junit).
> >>
> >> This is very important to me as we try to follow the Test-Driven
> >> Development practice and thus testability is the key concern to me. I
> >> like very much what you've done here, but I am afraid whether we will
> >> be able to safely play around with it and alter it for our customer
> >> without killing it.
> >>
> >> Concerning functional and acceptance testing have anyone considered
> >> using FitNesse, it's a really glamorous tool for holding acceptance
> >> criteria and running them against the system under test, additionally
> >> it's a great collaboration tool (it's a wiki). If you really really
> >> (as in: you rarely should) need to check some things though the
> >> browser interface then there are perfect examples of calling Selenium
> >> from FitNesse if anyone is interested I may point you to such
> >> materials.
> >>
> >> This is especially important if you would like to clearly specify
> >> functionality (define acceptance tests) ahead of implementing this
> >> functionality, recording stuff with the Selenium recorder is more or
> >> less fine for legacy stuff but that's not always the case.
> >>
> >> BTW please tell me how hard will it be to degrade the process of
> >> adding products, promotions to something simpler, our client wants
> >> some specific bits of what I see in the demo app, so I'm thinking
> >> about simplifying things for him as his main request is simplicity.
> >>
> >> Wojtek Biela
> >> www.exorigo.pl
> >
>
>

--
Wojtek Biela

[Wojciech Biela]

Hello David

> You should know that testing has been considered since day 1,

considered as in "it would be fun to have it" or "let's decouple this
from that so we'll be able to test that in isolation"?

> but to
> date I personally have not had a single client willing to sponsor any
> sort of testing effort.

hmmm, funny, I'll tell you how it looks like from where I stand, it's
a bit off-topic but never mind. Don't consider it offensive please
(!), I just want to share. When I tell the customer whether or not he
wants tests in place (acceptance tests) and talk for a quarter about
the consequences of not having such test he always is more than happy
to have those tests, and is even willing to help specify (or at least
sign off) those tests.  As for programmer tests (unit tests) they're
non-negotiable for me, it's just part of programming, period. I just
want to sleep well, once I got used to them I just can't do without
them, it's like heroine (although I never tried the latter :) just
seen the movies)

> Combine this with the fact that the framework
> is made up mostly of business-level tools and we are fortunately in a
> place where there are few low level bugs that require low-level unit
> tests.

true, if we plan to freeze that code and don't plan to refactor, smoke
tests (e.g. acceptance tests, business level tests) are not too good
as they just tell you, something went wrong, where unit tests tell you
this method here doesn't return a correct value. That said I know
right now covering ofBiz with unit tests just like that is insane and
smoke tests are the only acceptable solution for the moment.

> We still definitely have a need for 2 things in the testing area:
>
> 1. regression testing for applications, mostly on the service level
> (even if driven through the web-based UI)

true

> 2. tools and infrastructure in place for pre-development acceptance
> testing

pre-development acc testing, I don't get it, please elaborate

> Note that there is already a pretty good test infrastructure in place
> and it is used right now to run the JUnit tests for various of the
> framework components (mostly the entity and service engines).

24 tests (as you outlined on the mailing list a while ago)?

>  It just
> needs to be extended to support easy execution of testing scripts
> created by other tools. The main ones we are considering are Grinder
> and Canoo WebTest.

please consider also Selenium, or better Selenium RC ran from
FitNesse, just take a look at it, if you want I can elaborate on both

> Whether you use OFBiz or not is your choice and if it works well for
> you, great, if not that's fine too.

as in: "I don't care"? :))) no offence, yeah I know, I really hope it
works well, we are starting the bidding process in a few days so it'll
be the client's decision if we are to do it or not

> A lot of people talk about testing, but so far little has been
> contributed in this area, largely I think because it has always been
> a nice to have, and no one with sufficient resources has invested
> sufficiently in it. That's really all it comes down to.

I know exactly what you mean.

> I developed
> the current testing framework in the testtools component and while
> functional for junit tests it is far from complete. I have nothing
> else to invest in this as other obligations with higher priority
> won't allow it, so help from others is, as with most things, necessary.

I can't say I will, because I'm not sure if the client will choose us,
nevertheless for the time I am happy to discuss

> But yes, OFBiz is inherently testable and includes a great deal of
> functionality for creating and automatically running sets of tests.

see? that's what makes me happy :) you should have started with that :)

> There are even nice tools built around the entity engine for adding
> data to the database and asserting that the data in the database
> matches a certain state.

cool

thanks a lot David!
regards
Wojtek

>
> -David
>
>
> On Jan 15, 2007, at 3:45 AM, Wojciech Biela wrote:
>
> > Hello
> >
> > I recently looked at ofbiz and am now evaluating whether or not we
> > should go with it or not. We are a software development company.
> >
> > The key thing for me at this point is testability (both unit- and
> > functional-level). I have a great concern because I hear that only
> > recently people are starting to think about it on the ofbiz list, and
> > the project dates back to 2001, so there is a whole lot of legacy code
> > that is not tested in any (automated) way?
> >
> > I read Cameron Smith's post from Nov 1 2006 "More on testing
> > framework" where he said that he had "Unit Tests proper - run as JUnit
> > tests", could any of you (or you Cameron) elaborate a bit about those
> > tests in the context of ofbiz
> > - how painful was it
> > - what components did you test
> > - what things did you mock
> > - what do you suggest
> >
> > Unfortunately I didn't find this information in the documentation and
> > a preliminary google search on this subject didn't reveal any
> > revelations either. I only found that you are taking about functional
> > testing. But then I ask what about unit level tests? I see junit.jar
> > in the sources but I then again I see the build process does not
> > invoke unit tests (there is a separate ant task for it), I read that
> > there is a total of 24 unit tests.... There was some info that the
> > topic is covered in the advanced (commercial?) documentation by
> > Undersun, is there anything apache-licenced ;) on that?
> >
> > Is the architecture of ofbiz testable or not ? Is it possible to write
> > unit test (i.e. isolated unit tests) as well as integration tests
> > (also through junit).
> >
> > This is very important to me as we try to follow the Test-Driven
> > Development practice and thus testability is the key concern to me. I
> > like very much what you've done here, but I am afraid whether we will
> > be able to safely play around with it and alter it for our customer
> > without killing it.
> >
> > Concerning functional and acceptance testing have anyone considered
> > using FitNesse, it's a really glamorous tool for holding acceptance
> > criteria and running them against the system under test, additionally
> > it's a great collaboration tool (it's a wiki). If you really really
> > (as in: you rarely should) need to check some things though the
> > browser interface then there are perfect examples of calling Selenium
> > from FitNesse if anyone is interested I may point you to such
> > materials.
> >
> > This is especially important if you would like to clearly specify
> > functionality (define acceptance tests) ahead of implementing this
> > functionality, recording stuff with the Selenium recorder is more or
> > less fine for legacy stuff but that's not always the case.
> >
> > BTW please tell me how hard will it be to degrade the process of
> > adding products, promotions to something simpler, our client wants
> > some specific bits of what I see in the demo app, so I'm thinking
> > about simplifying things for him as his main request is simplicity.
> >
> > Wojtek Biela
> > www.exorigo.pl
>
>
>
>

--
Wojtek Biela

[Wojciech Biela]

Hello Cameron

> In reply to your general question, I do believe that
> OFBiz is eminently testable, in that you can identify
> distinct pieces of functionality at various levels and
> test them by automatic means.

good to hear that

> Re David's point about the need for practical
> intervention and not just talk, I did in fact offer to
> help with this (improvements to the testing framework)
> a while back.  However the feedback I got from David
> and others to the suggestions I made (before I
> implemented them) was basically nil.

:(

> Given this lack of interest, I left it for the moment,
> however in the next couple of months I will have to
> improve the test framework anyway for my company's
> internal purposes and I would be happy to contribute
> that back to OFBiz.

cool !!!

> Meanwhile, I suggest that the best way to go forward
> would be to look at the existing tests and work out
> how to use the framework from there, as there does not
> appear to be any up-to-date documentation.  Basically
> the starting point is the ofbiz-component.xml file for
> a particular module, which points at unit test
> definition files, via a <test-resource> tag.  (I am
> away from the office at the mo so cannot check the
> exact tag name).

OK, that seems a good place to start

> If you have specific problems using the framework,
> post them here and I will respond, but, like David, I
> am not up for general testing discussions.

OK, point taken

> Pretty
> much everything general there is to say, has been
> discussed already on the mailing list archives

:D there's always more :D

thanks for the tips!
regards
Wojtek

>
> cameron
>
> > Wojtek,
> >
> > While it's great that Jacques has answered and
> > shared his thoughts
> > and experience on this, please don't consider that a
> > full
> > representation of the state of testing in OFBiz.
> >
> > You should know that testing has been considered
> > since day 1, but to
> > date I personally have not had a single client
> > willing to sponsor any
> > sort of testing effort. Combine this with the fact
> > that the framework
> > is made up mostly of business-level tools and we are
> > fortunately in a
> > place where there are few low level bugs that
> > require low-level unit
> > tests. We still definitely have a need for 2 things
> > in the testing area:
> >
> > 1. regression testing for applications, mostly on
> > the service level
> > (even if driven through the web-based UI)
> > 2. tools and infrastructure in place for
> > pre-development acceptance
> > testing
> >
> > Note that there is already a pretty good test
> > infrastructure in place
> > and it is used right now to run the JUnit tests for
> > various of the
> > framework components (mostly the entity and service
> > engines). It just
> > needs to be extended to support easy execution of
> > testing scripts
> > created by other tools. The main ones we are
> > considering are Grinder
> > and Canoo WebTest.
> >
> > Whether you use OFBiz or not is your choice and if
> > it works well for
> > you, great, if not that's fine too.
> >
> > A lot of people talk about testing, but so far
> > little has been
> > contributed in this area, largely I think because it
> > has always been
> > a nice to have, and no one with sufficient resources
> > has invested
> > sufficiently in it. That's really all it comes down
> > to. I developed
> > the current testing framework in the testtools
> > component and while
> > functional for junit tests it is far from complete.
> > I have nothing
> > else to invest in this as other obligations with
> > higher priority
> > won't allow it, so help from others is, as with most
> > things, necessary.
> >
> > But yes, OFBiz is inherently testable and includes a
> > great deal of
> > functionality for creating and automatically running
> > sets of tests.
> > There are even nice tools built around the entity
> > engine for adding
> > data to the database and asserting that the data in
> > the database
> > matches a certain state.
> >
> > -David
> >
> >
> > On Jan 15, 2007, at 3:45 AM, Wojciech Biela wrote:
> >
> > > Hello
> > >
> > > I recently looked at ofbiz and am now evaluating
> > whether or not we
> > > should go with it or not. We are a software
> > development company.
> > >
> > > The key thing for me at this point is testability
> > (both unit- and
> > > functional-level). I have a great concern because
> > I hear that only
> > > recently people are starting to think about it on
> > the ofbiz list, and
> > > the project dates back to 2001, so there is a
> > whole lot of legacy code
> > > that is not tested in any (automated) way?
> > >
> > > I read Cameron Smith's post from Nov 1 2006 "More
> > on testing
> > > framework" where he said that he had "Unit Tests
> > proper - run as JUnit
> > > tests", could any of you (or you Cameron)
> > elaborate a bit about those
> > > tests in the context of ofbiz
> > > - how painful was it
> > > - what components did you test
> > > - what things did you mock
> > > - what do you suggest
> > >
> > > Unfortunately I didn't find this information in
> > the documentation and
> > > a preliminary google search on this subject didn't
> > reveal any
> > > revelations either. I only found that you are
> > taking about functional
> > > testing. But then I ask what about unit level
> > tests? I see junit.jar
> > > in the sources but I then again I see the build
> > process does not
> > > invoke unit tests (there is a separate ant task
> > for it), I read that
> > > there is a total of 24 unit tests.... There was
> > some info that the
> > > topic is covered in the advanced (commercial?)
> > documentation by
> > > Undersun, is there anything apache-licenced ;) on
> > that?
> > >
> > > Is the architecture of ofbiz testable or not ? Is
> > it possible to write
> > > unit test (i.e. isolated unit tests) as well as
> > integration tests
> > > (also through junit).
> > >
> > > This is very important to me as we try to follow
> > the Test-Driven
> > > Development practice and thus testability is the
> > key concern to me. I
> > > like very much what you've done here, but I am
> > afraid whether we will
> > > be able to safely play around with it and alter it
> > for our customer
> > > without killing it.
> > >
> > > Concerning functional and acceptance testing have
> > anyone considered
> > > using FitNesse, it's a really glamorous tool for
> > holding acceptance
> > > criteria and running them against the system under
> > test, additionally
> > > it's a great collaboration tool (it's a wiki). If
> > you really really
> > > (as in: you rarely should) need to check some
> > things though the
> > > browser interface then there are perfect examples
> > of calling Selenium
> > > from FitNesse if anyone is interested I may point
> > you to such
> > > materials.
> > >
> > > This is especially important if you would like to
> > clearly specify
> > > functionality (define acceptance tests) ahead of
> > implementing this
> > > functionality, recording stuff with the Selenium
> > recorder is more or
> > > less fine for legacy stuff but that's not always
> > the case.
> > >
> > > BTW please tell me how hard will it be to degrade
> > the process of
> > > adding products, promotions to something simpler,
> > our client wants
> > > some specific bits of what I see in the demo app,
> > so I'm thinking
> > > about simplifying things for him as his main
> > request is simplicity.
> > >
> > > Wojtek Biela
> > > www.exorigo.pl
> >
> >
>
>
>
>
> ___________________________________________________________
> What kind of emailer are you? Find out today - get a free analysis of your email personality. Take the quiz at the Yahoo! Mail Championship.
> http://uk.rd.yahoo.com/evt=44106/*http://mail.yahoo.net/uk
>

--
Wojtek Biela

[jonwimp]

Hi Wojciech,

>> For extensibility, I can just say 1 word: total. I've not seen
>> anything as extensible as this.
>
> that frightens me a bit, I've seen stuff very extensible and very
> useless because totally complicated to do simple stuff and changes.
> Don't shoot! I don't say ofBiz is like that ;) I just say what I'm
> afraid of, that all the configuration will kill development (as did
> e.g. a load of descriptors kill EJB (pre 3.0)),

The entity framework in OFBiz is not like EJB. And it's not the sheer number of descriptors that
killed EJB. One of the reasons I'm on to OFBiz now is mainly because it's a nice change from EJB.
I believe some veterans here may point you to old discussions about how OFBiz compares with EJB.

EJB's performance is horrible.

Jonathon

[Wojciech Biela]

> The entity framework in OFBiz is not like EJB. And it's not the sheer number of descriptors that
> killed EJB.

yeah I know ;) I just couldn't resist bringing that up, I'm of those
persons who type faster than think ;)

> One of the reasons I'm on to OFBiz now is mainly because it's a nice change from EJB.
> I believe some veterans here may point you to old discussions about how OFBiz compares with EJB.
>
> EJB's performance is horrible.

regards
--
Wojtek Biela