We miss a QA team, we face too much regressions
Locked
We miss a QA team, we face too much regressions
 
Administrator
|
Hi,
With the considerable HW effort, a lot of things are going on recently, and it's hard to follow. I though noticed that we experience more and more regressions (not all related to HW effort, far from it). Fortunately it's so far mostly minor points and often related with the UI, OFBIZ-7346 and OFBIZ-7363 being counter examples (OFBIZ-7346 can be critical) From my experience, w/o a QA person or team, it's very hard to detect those side effects at the UI level when you refactor or fix it. I remember the (ex) Neogia team (mostly Erwan) tried to maintain a Selenium/Webdriver set of tests. I don't know if they continue/d. Since we spoke about Junit and unit tests recently, some prefer TestNG, at least coupled with Selenium http://testng.org/doc/selenium.html Does it make sense, do you think it's only an utopia? Thanks Jacques |
|
Hi Jacques,
Selenium tests cannot be unit tests in OFBiz because it requires firing up the server. You can consider them part of the integration tests (existing functionality). In fact, I would consider selenium tests to be functional tests (higher than integration) -> https://en.wikipedia.org/wiki/Functional_testing So yeah we can add them, but I don't think we can do that to the raw unit-tests (at least in the context discussed in the other proposal thread) Taher Alkhateeb On Thu, Jun 16, 2016 at 10:40 PM, Jacques Le Roux < [hidden email]> wrote: > Hi, > > With the considerable HW effort, a lot of things are going on recently, > and it's hard to follow. I though noticed that we experience more and more > regressions (not all related to HW effort, far from it). > > Fortunately it's so far mostly minor points and often related with the UI, > OFBIZ-7346 and OFBIZ-7363 being counter examples (OFBIZ-7346 can be > critical) > > From my experience, w/o a QA person or team, it's very hard to detect > those side effects at the UI level when you refactor or fix it. I remember > the (ex) Neogia team (mostly Erwan) tried to maintain a Selenium/Webdriver > set of tests. I don't know if they continue/d. > > Since we spoke about Junit and unit tests recently, some prefer TestNG, at > least coupled with Selenium http://testng.org/doc/selenium.html > > Does it make sense, do you think it's only an utopia? > > Thanks > > Jacques > > |
Re: We miss a QA team, we face too much regressions
|
Administrator
|
Yes, I agree Selenium does not fit with unit tests and is the higher level of tests as UI is the higher level of code. From my experience running
tests from this level helps also to detect bugs at a lower levels, notably regressions. I spoke about unit tests only because I wanted to say that some prefer to use TestNG with Selenium, this to see if nobody had an opinion or/and use Selenium for their own projects. About TestNG, I actually believe you can do the same with JUnit 4. I must say I never used Selenium with either JUnit or TestNG. Only as a simple user creating tests for myself. That was long ago and it then saved me a lot of time... Jacques Le 16/06/2016 à 21:48, Taher Alkhateeb a écrit : > Hi Jacques, > > Selenium tests cannot be unit tests in OFBiz because it requires firing up > the server. You can consider them part of the integration tests (existing > functionality). In fact, I would consider selenium tests to be functional > tests (higher than integration) -> > https://en.wikipedia.org/wiki/Functional_testing > > So yeah we can add them, but I don't think we can do that to the raw > unit-tests (at least in the context discussed in the other proposal thread) > > Taher Alkhateeb > > On Thu, Jun 16, 2016 at 10:40 PM, Jacques Le Roux < > [hidden email]> wrote: > >> Hi, >> >> With the considerable HW effort, a lot of things are going on recently, >> and it's hard to follow. I though noticed that we experience more and more >> regressions (not all related to HW effort, far from it). >> >> Fortunately it's so far mostly minor points and often related with the UI, >> OFBIZ-7346 and OFBIZ-7363 being counter examples (OFBIZ-7346 can be >> critical) >> >> From my experience, w/o a QA person or team, it's very hard to detect >> those side effects at the UI level when you refactor or fix it. I remember >> the (ex) Neogia team (mostly Erwan) tried to maintain a Selenium/Webdriver >> set of tests. I don't know if they continue/d. >> >> Since we spoke about Junit and unit tests recently, some prefer TestNG, at >> least coupled with Selenium http://testng.org/doc/selenium.html >> >> Does it make sense, do you think it's only an utopia? >> >> Thanks >> >> Jacques >> >> |
Re: We miss a QA team, we face too much regressions
|
In reply to this post by taher
One of the side benefits of having a small number of committer's is that
you prevent bad designs and poorly tested code getting into the trunk. The disadvantage is that the committers are easily overwhelmed by an active contributor community. You may want to put in some rules about unit tests so that code without adequate test coverage can not be updated until the unit tests are sufficient for the committer to feel confident about accepting it. This may cause people to work on tests for stuff that they did not write but are considered key functionality in the modules being updated. There is no free ride and if you allow people to build up the technical debt of the project in order to meet their own deadlines, you will eventually have to face a large debt that comes due. Taher is paying off the debt in the framework which is a great contribution. It may be that others are going to have to take up the challenge in the application side. You may have to have a short moratorium on enhancements until the debt is reduced to a manageable level. There may also be the issue of people modifying too many layers at once so changes affect a lot of different services so unpleasant side-effects are easier to generate. Are the regressions caused by a small group of contributors or from updates going through a few committers? It is an open source project so there has to be some sensitivity about asking people to do a bit more to clean up old debt but if that is a problem and it is not addressed, it can be a big mess. Ron On 16/06/2016 3:48 PM, Taher Alkhateeb wrote: > Hi Jacques, > > Selenium tests cannot be unit tests in OFBiz because it requires firing up > the server. You can consider them part of the integration tests (existing > functionality). In fact, I would consider selenium tests to be functional > tests (higher than integration) -> > https://en.wikipedia.org/wiki/Functional_testing > > So yeah we can add them, but I don't think we can do that to the raw > unit-tests (at least in the context discussed in the other proposal thread) > > Taher Alkhateeb > > On Thu, Jun 16, 2016 at 10:40 PM, Jacques Le Roux < > [hidden email]> wrote: > >> Hi, >> >> With the considerable HW effort, a lot of things are going on recently, >> and it's hard to follow. I though noticed that we experience more and more >> regressions (not all related to HW effort, far from it). >> >> Fortunately it's so far mostly minor points and often related with the UI, >> OFBIZ-7346 and OFBIZ-7363 being counter examples (OFBIZ-7346 can be >> critical) >> >> From my experience, w/o a QA person or team, it's very hard to detect >> those side effects at the UI level when you refactor or fix it. I remember >> the (ex) Neogia team (mostly Erwan) tried to maintain a Selenium/Webdriver >> set of tests. I don't know if they continue/d. >> >> Since we spoke about Junit and unit tests recently, some prefer TestNG, at >> least coupled with Selenium http://testng.org/doc/selenium.html >> >> Does it make sense, do you think it's only an utopia? >> >> Thanks >> >> Jacques >> >> -- Ron Wheeler President Artifact Software Inc email: [hidden email] skype: ronaldmwheeler phone: 866-970-2435, ext 102 |
Re: We miss a QA team, we face too much regressions
|
|
In reply to this post by Jacques Le Roux
Hi,
Neogia selenium/webdriver test suite have been abandonned, and we decided at Néréide to stop doing Selenium UI testing since it needs a lot of effort to maintain, they are very long to pass, and *weren't stable enough* (maybe that's because we used a lot of ajax area update...) in the way that not changing code make the test go red/blue... Technically I used to convert my selenium test case to Junit4 class... No problem with this technology From my experience, UI testing is a good way too ensure one or more customer core process of an application through UI, but i faced so many problems with it, that the word Utopia ring a bell to me. I'd rather like integration tests like they are in OFBiz yet. Moreover, OOTB UI in OFBiz is not intended to be used directly for a customer/user, without adapting it to his needs, i cannot imagine testsuites covering all the processes provided by OFBiz, and maintaining it... I already have problem to find time to contribute in OFBiz as I wish, I won't spend this time maintaining selenium testcase... It's a quite negative opinion here, i know that Olivier is still using it for its project, he might have a different POV. Best Regards Gil P.S. : Why not some selenium tests in example component, to test widgets and show user how to implement selenium testing within OFBiz, but that's not the goal of the discussion. On 16/06/2016 21:40, Jacques Le Roux
wrote:
Hi, |
|
|
Hi Gil,
For me it's a simple rule, lots of unit tests, less integration tests, even less functional tests (selenium). But I think all are useful if done in proper quantity I like this diagram in a way summarizes it -> http://i.stack.imgur.com/fjQvQ.png Taher Alkhateeb On Fri, Jun 17, 2016 at 9:19 AM, gil portenseigne < [hidden email]> wrote: > Hi, > > Neogia selenium/webdriver test suite have been abandonned, and we decided > at Néréide to stop doing Selenium UI testing since it needs a lot of effort > to maintain, they are very long to pass, and *weren't stable enough* (maybe > that's because we used a lot of ajax area update...) in the way that not > changing code make the test go red/blue... > > Technically I used to convert my selenium test case to Junit4 class... No > problem with this technology > > From my experience, UI testing is a good way too ensure one or more > customer core process of an application through UI, but i faced so many > problems with it, that the word Utopia ring a bell to me. I'd rather like > integration tests like they are in OFBiz yet. > > Moreover, OOTB UI in OFBiz is not intended to be used directly for a > customer/user, without adapting it to his needs, i cannot imagine > testsuites covering all the processes provided by OFBiz, and maintaining > it... > I already have problem to find time to contribute in OFBiz as I wish, I > won't spend this time maintaining selenium testcase... > > It's a quite negative opinion here, i know that Olivier is still using it > for its project, he might have a different POV. > > Best Regards > > Gil > > P.S. : Why not some selenium tests in example component, to test widgets > and show user how to implement selenium testing within OFBiz, but that's > not the goal of the discussion. > > On 16/06/2016 21:40, Jacques Le Roux wrote: > > Hi, > > With the considerable HW effort, a lot of things are going on recently, > and it's hard to follow. I though noticed that we experience more and more > regressions (not all related to HW effort, far from it). > > Fortunately it's so far mostly minor points and often related with the UI, > OFBIZ-7346 and OFBIZ-7363 being counter examples (OFBIZ-7346 can be > critical) > > From my experience, w/o a QA person or team, it's very hard to detect > those side effects at the UI level when you refactor or fix it. I remember > the (ex) Neogia team (mostly Erwan) tried to maintain a Selenium/Webdriver > set of tests. I don't know if they continue/d. > > Since we spoke about Junit and unit tests recently, some prefer TestNG, at > least coupled with Selenium <http://testng.org/doc/selenium.html> > http://testng.org/doc/selenium.html > > Does it make sense, do you think it's only an utopia? > > Thanks > > Jacques > > > |
Re: We miss a QA team, we face too much regressions
|
Administrator
|
In reply to this post by Gil Portenseigne
Thanks for the feedback Gil,
Nothing better than a life experience! Mine was not direct because I worked with a QA engineer in charge of all these aspects. From my POV, it was great to have it, but I had not to deal with it ;) Jacques Le 17/06/2016 à 08:19, gil portenseigne a écrit : > Hi, > > Neogia selenium/webdriver test suite have been abandonned, and we decided at Néréide to stop doing Selenium UI testing since it needs a lot of > effort to maintain, they are very long to pass, and *weren't stable enough* (maybe that's because we used a lot of ajax area update...) in the way > that not changing code make the test go red/blue... > > Technically I used to convert my selenium test case to Junit4 class... No problem with this technology > > From my experience, UI testing is a good way too ensure one or more customer core process of an application through UI, but i faced so many problems > with it, that the word Utopia ring a bell to me. I'd rather like integration tests like they are in OFBiz yet. > > Moreover, OOTB UI in OFBiz is not intended to be used directly for a customer/user, without adapting it to his needs, i cannot imagine testsuites > covering all the processes provided by OFBiz, and maintaining it... > I already have problem to find time to contribute in OFBiz as I wish, I won't spend this time maintaining selenium testcase... > > It's a quite negative opinion here, i know that Olivier is still using it for its project, he might have a different POV. > > Best Regards > > Gil > > P.S. : Why not some selenium tests in example component, to test widgets and show user how to implement selenium testing within OFBiz, but that's > not the goal of the discussion. > > On 16/06/2016 21:40, Jacques Le Roux wrote: >> Hi, >> >> With the considerable HW effort, a lot of things are going on recently, and it's hard to follow. I though noticed that we experience more and more >> regressions (not all related to HW effort, far from it). >> >> Fortunately it's so far mostly minor points and often related with the UI, OFBIZ-7346 and OFBIZ-7363 being counter examples (OFBIZ-7346 can be >> critical) >> >> From my experience, w/o a QA person or team, it's very hard to detect those side effects at the UI level when you refactor or fix it. I remember >> the (ex) Neogia team (mostly Erwan) tried to maintain a Selenium/Webdriver set of tests. I don't know if they continue/d. >> >> Since we spoke about Junit and unit tests recently, some prefer TestNG, at least coupled with Selenium http://testng.org/doc/selenium.html >> >> Does it make sense, do you think it's only an utopia? >> >> Thanks >> >> Jacques >> > |
Re: We miss a QA team, we face too much regressions
|
Administrator
|
In reply to this post by taher
Thanks Taher,
The diagram says it all indeed. There are certainly crucial parts of the UI we would like to cover, like order creation (in both order manager and variants in ecommerce)... and such... Jacques Le 17/06/2016 à 08:26, Taher Alkhateeb a écrit : > Hi Gil, > > For me it's a simple rule, lots of unit tests, less integration tests, even > less functional tests (selenium). But I think all are useful if done in > proper quantity > > I like this diagram in a way summarizes it -> > http://i.stack.imgur.com/fjQvQ.png > > Taher Alkhateeb > > On Fri, Jun 17, 2016 at 9:19 AM, gil portenseigne < > [hidden email]> wrote: > >> Hi, >> >> Neogia selenium/webdriver test suite have been abandonned, and we decided >> at Néréide to stop doing Selenium UI testing since it needs a lot of effort >> to maintain, they are very long to pass, and *weren't stable enough* (maybe >> that's because we used a lot of ajax area update...) in the way that not >> changing code make the test go red/blue... >> >> Technically I used to convert my selenium test case to Junit4 class... No >> problem with this technology >> >> From my experience, UI testing is a good way too ensure one or more >> customer core process of an application through UI, but i faced so many >> problems with it, that the word Utopia ring a bell to me. I'd rather like >> integration tests like they are in OFBiz yet. >> >> Moreover, OOTB UI in OFBiz is not intended to be used directly for a >> customer/user, without adapting it to his needs, i cannot imagine >> testsuites covering all the processes provided by OFBiz, and maintaining >> it... >> I already have problem to find time to contribute in OFBiz as I wish, I >> won't spend this time maintaining selenium testcase... >> >> It's a quite negative opinion here, i know that Olivier is still using it >> for its project, he might have a different POV. >> >> Best Regards >> >> Gil >> >> P.S. : Why not some selenium tests in example component, to test widgets >> and show user how to implement selenium testing within OFBiz, but that's >> not the goal of the discussion. >> >> On 16/06/2016 21:40, Jacques Le Roux wrote: >> >> Hi, >> >> With the considerable HW effort, a lot of things are going on recently, >> and it's hard to follow. I though noticed that we experience more and more >> regressions (not all related to HW effort, far from it). >> >> Fortunately it's so far mostly minor points and often related with the UI, >> OFBIZ-7346 and OFBIZ-7363 being counter examples (OFBIZ-7346 can be >> critical) >> >> From my experience, w/o a QA person or team, it's very hard to detect >> those side effects at the UI level when you refactor or fix it. I remember >> the (ex) Neogia team (mostly Erwan) tried to maintain a Selenium/Webdriver >> set of tests. I don't know if they continue/d. >> >> Since we spoke about Junit and unit tests recently, some prefer TestNG, at >> least coupled with Selenium <http://testng.org/doc/selenium.html> >> http://testng.org/doc/selenium.html >> >> Does it make sense, do you think it's only an utopia? >> >> Thanks >> >> Jacques >> >> >> |
|
|
Hi Jacques,
Yeah makes sense, I like your idea. One thing at a time though, we badly badly need unit tests right now and I recommend we focus most of our efforts in the next few months in building the unit tests as soon as I'm done with implementing it (hopefully very soon) Then again I also think the order logic is overly complex and should also be eventually simplified. I am drifting too much aren't I? Okay I will zip it :) Taher Alkhateeb On Jun 17, 2016 11:16 AM, "Jacques Le Roux" <[hidden email]> wrote: > Thanks Taher, > > The diagram says it all indeed. There are certainly crucial parts of the > UI we would like to cover, like order creation (in both order manager and > variants in ecommerce)... and such... > > Jacques > > > Le 17/06/2016 à 08:26, Taher Alkhateeb a écrit : > >> Hi Gil, >> >> For me it's a simple rule, lots of unit tests, less integration tests, >> even >> less functional tests (selenium). But I think all are useful if done in >> proper quantity >> >> I like this diagram in a way summarizes it -> >> http://i.stack.imgur.com/fjQvQ.png >> >> Taher Alkhateeb >> >> On Fri, Jun 17, 2016 at 9:19 AM, gil portenseigne < >> [hidden email]> wrote: >> >> Hi, >>> >>> Neogia selenium/webdriver test suite have been abandonned, and we decided >>> at Néréide to stop doing Selenium UI testing since it needs a lot of >>> effort >>> to maintain, they are very long to pass, and *weren't stable enough* >>> (maybe >>> that's because we used a lot of ajax area update...) in the way that not >>> changing code make the test go red/blue... >>> >>> Technically I used to convert my selenium test case to Junit4 class... No >>> problem with this technology >>> >>> From my experience, UI testing is a good way too ensure one or more >>> customer core process of an application through UI, but i faced so many >>> problems with it, that the word Utopia ring a bell to me. I'd rather like >>> integration tests like they are in OFBiz yet. >>> >>> Moreover, OOTB UI in OFBiz is not intended to be used directly for a >>> customer/user, without adapting it to his needs, i cannot imagine >>> testsuites covering all the processes provided by OFBiz, and maintaining >>> it... >>> I already have problem to find time to contribute in OFBiz as I wish, I >>> won't spend this time maintaining selenium testcase... >>> >>> It's a quite negative opinion here, i know that Olivier is still using it >>> for its project, he might have a different POV. >>> >>> Best Regards >>> >>> Gil >>> >>> P.S. : Why not some selenium tests in example component, to test widgets >>> and show user how to implement selenium testing within OFBiz, but that's >>> not the goal of the discussion. >>> >>> On 16/06/2016 21:40, Jacques Le Roux wrote: >>> >>> Hi, >>> >>> With the considerable HW effort, a lot of things are going on recently, >>> and it's hard to follow. I though noticed that we experience more and >>> more >>> regressions (not all related to HW effort, far from it). >>> >>> Fortunately it's so far mostly minor points and often related with the >>> UI, >>> OFBIZ-7346 and OFBIZ-7363 being counter examples (OFBIZ-7346 can be >>> critical) >>> >>> From my experience, w/o a QA person or team, it's very hard to detect >>> those side effects at the UI level when you refactor or fix it. I >>> remember >>> the (ex) Neogia team (mostly Erwan) tried to maintain a >>> Selenium/Webdriver >>> set of tests. I don't know if they continue/d. >>> >>> Since we spoke about Junit and unit tests recently, some prefer TestNG, >>> at >>> least coupled with Selenium <http://testng.org/doc/selenium.html> >>> http://testng.org/doc/selenium.html >>> >>> Does it make sense, do you think it's only an utopia? >>> >>> Thanks >>> >>> Jacques >>> >>> >>> >>> > |
Re: We miss a QA team, we face too much regressions
|
|
I suggest that we, in order to keep this on the radar and not sink away in
the ml swamp, we create placeholder issues in JIRA to track progress and thoughts. Best regards, Pierre Smits ORRTIZ.COM <http://www.orrtiz.com> OFBiz based solutions & services OFBiz Extensions Marketplace http://oem.ofbizci.net/oci-2/ On Fri, Jun 17, 2016 at 10:34 AM, Taher Alkhateeb < [hidden email]> wrote: > Hi Jacques, > > Yeah makes sense, I like your idea. > > One thing at a time though, we badly badly need unit tests right now and I > recommend we focus most of our efforts in the next few months in building > the unit tests as soon as I'm done with implementing it (hopefully very > soon) > > Then again I also think the order logic is overly complex and should also > be eventually simplified. I am drifting too much aren't I? Okay I will zip > it :) > > Taher Alkhateeb > On Jun 17, 2016 11:16 AM, "Jacques Le Roux" <[hidden email]> > wrote: > > > Thanks Taher, > > > > The diagram says it all indeed. There are certainly crucial parts of the > > UI we would like to cover, like order creation (in both order manager and > > variants in ecommerce)... and such... > > > > Jacques > > > > > > Le 17/06/2016 à 08:26, Taher Alkhateeb a écrit : > > > >> Hi Gil, > >> > >> For me it's a simple rule, lots of unit tests, less integration tests, > >> even > >> less functional tests (selenium). But I think all are useful if done in > >> proper quantity > >> > >> I like this diagram in a way summarizes it -> > >> http://i.stack.imgur.com/fjQvQ.png > >> > >> Taher Alkhateeb > >> > >> On Fri, Jun 17, 2016 at 9:19 AM, gil portenseigne < > >> [hidden email]> wrote: > >> > >> Hi, > >>> > >>> Neogia selenium/webdriver test suite have been abandonned, and we > decided > >>> at Néréide to stop doing Selenium UI testing since it needs a lot of > >>> effort > >>> to maintain, they are very long to pass, and *weren't stable enough* > >>> (maybe > >>> that's because we used a lot of ajax area update...) in the way that > not > >>> changing code make the test go red/blue... > >>> > >>> Technically I used to convert my selenium test case to Junit4 class... > No > >>> problem with this technology > >>> > >>> From my experience, UI testing is a good way too ensure one or more > >>> customer core process of an application through UI, but i faced so many > >>> problems with it, that the word Utopia ring a bell to me. I'd rather > like > >>> integration tests like they are in OFBiz yet. > >>> > >>> Moreover, OOTB UI in OFBiz is not intended to be used directly for a > >>> customer/user, without adapting it to his needs, i cannot imagine > >>> testsuites covering all the processes provided by OFBiz, and > maintaining > >>> it... > >>> I already have problem to find time to contribute in OFBiz as I wish, I > >>> won't spend this time maintaining selenium testcase... > >>> > >>> It's a quite negative opinion here, i know that Olivier is still using > it > >>> for its project, he might have a different POV. > >>> > >>> Best Regards > >>> > >>> Gil > >>> > >>> P.S. : Why not some selenium tests in example component, to test > widgets > >>> and show user how to implement selenium testing within OFBiz, but > that's > >>> not the goal of the discussion. > >>> > >>> On 16/06/2016 21:40, Jacques Le Roux wrote: > >>> > >>> Hi, > >>> > >>> With the considerable HW effort, a lot of things are going on recently, > >>> and it's hard to follow. I though noticed that we experience more and > >>> more > >>> regressions (not all related to HW effort, far from it). > >>> > >>> Fortunately it's so far mostly minor points and often related with the > >>> UI, > >>> OFBIZ-7346 and OFBIZ-7363 being counter examples (OFBIZ-7346 can be > >>> critical) > >>> > >>> From my experience, w/o a QA person or team, it's very hard to detect > >>> those side effects at the UI level when you refactor or fix it. I > >>> remember > >>> the (ex) Neogia team (mostly Erwan) tried to maintain a > >>> Selenium/Webdriver > >>> set of tests. I don't know if they continue/d. > >>> > >>> Since we spoke about Junit and unit tests recently, some prefer TestNG, > >>> at > >>> least coupled with Selenium <http://testng.org/doc/selenium.html> > >>> http://testng.org/doc/selenium.html > >>> > >>> Does it make sense, do you think it's only an utopia? > >>> > >>> Thanks > >>> > >>> Jacques > >>> > >>> > >>> > >>> > > > |
Re: We miss a QA team, we face too much regressions
|
Administrator
|
In reply to this post by Ron Wheeler
Le 16/06/2016 à 22:53, Ron Wheeler a écrit :
> One of the side benefits of having a small number of committer's is that you prevent bad designs and poorly tested code getting into the trunk. > The disadvantage is that the committers are easily overwhelmed by an active contributor community. Would you say that with 31 committers (most active) we are currently a small number of committers? > You may want to put in some rules about unit tests so that code without adequate test coverage can not be updated until the unit tests are > sufficient for the committer to feel confident about accepting it. This may cause people to work on tests for stuff that they did not write but are > considered key functionality in the modules being updated. > There is no free ride and if you allow people to build up the technical debt of the project in order to meet their own deadlines, you will > eventually have to face a large debt that comes due. > > Taher is paying off the debt in the framework which is a great contribution. > It may be that others are going to have to take up the challenge in the application side. > You may have to have a short moratorium on enhancements until the debt is reduced to a manageable level. > > There may also be the issue of people modifying too many layers at once so changes affect a lot of different services so unpleasant side-effects are > easier to generate. > > Are the regressions caused by a small group of contributors or from updates going through a few committers? As I said it's recently fortunately small things. For now it's hard to answer to your question, because the HW effort is rip-roaring. I guess when it will settle a lot of things will be better/fixed, in the meantime me will certainly face some uncertainty. My question was not about pointing finger put how to prevent issues. Hence my question about Selenium because our current set of tests is obviously not enough. Your suggestion about more unit tests and rules is certainly to consider. I'd wait the end of the rip-roaring HW effort, for things to stabilise, and then try to introduce more constraints, or should we discuss it right away, community? > > It is an open source project so there has to be some sensitivity about asking people to do a bit more to clean up old debt but if that is a problem > and it is not addressed, it can be a big mess. I see a lot of skilled good will and clearly success for the last few years. I think we can achieve it, OFBiz is here to stay! Jacques > > > Ron > > > On 16/06/2016 3:48 PM, Taher Alkhateeb wrote: >> Hi Jacques, >> >> Selenium tests cannot be unit tests in OFBiz because it requires firing up >> the server. You can consider them part of the integration tests (existing >> functionality). In fact, I would consider selenium tests to be functional >> tests (higher than integration) -> >> https://en.wikipedia.org/wiki/Functional_testing >> >> So yeah we can add them, but I don't think we can do that to the raw >> unit-tests (at least in the context discussed in the other proposal thread) >> >> Taher Alkhateeb >> >> On Thu, Jun 16, 2016 at 10:40 PM, Jacques Le Roux < >> [hidden email]> wrote: >> >>> Hi, >>> >>> With the considerable HW effort, a lot of things are going on recently, >>> and it's hard to follow. I though noticed that we experience more and more >>> regressions (not all related to HW effort, far from it). >>> >>> Fortunately it's so far mostly minor points and often related with the UI, >>> OFBIZ-7346 and OFBIZ-7363 being counter examples (OFBIZ-7346 can be >>> critical) >>> >>> From my experience, w/o a QA person or team, it's very hard to detect >>> those side effects at the UI level when you refactor or fix it. I remember >>> the (ex) Neogia team (mostly Erwan) tried to maintain a Selenium/Webdriver >>> set of tests. I don't know if they continue/d. >>> >>> Since we spoke about Junit and unit tests recently, some prefer TestNG, at >>> least coupled with Selenium http://testng.org/doc/selenium.html >>> >>> Does it make sense, do you think it's only an utopia? >>> >>> Thanks >>> >>> Jacques >>> >>> > > |
Re: We miss a QA team, we face too much regressions
|
Administrator
|
In reply to this post by Pierre Smits
Good point Pierre, I'll do
Jacques Le 17/06/2016 à 11:03, Pierre Smits a écrit : > I suggest that we, in order to keep this on the radar and not sink away in > the ml swamp, we create placeholder issues in JIRA to track progress and > thoughts. > > Best regards, > > Pierre Smits > > ORRTIZ.COM <http://www.orrtiz.com> > OFBiz based solutions & services > > OFBiz Extensions Marketplace > http://oem.ofbizci.net/oci-2/ > > On Fri, Jun 17, 2016 at 10:34 AM, Taher Alkhateeb < > [hidden email]> wrote: > >> Hi Jacques, >> >> Yeah makes sense, I like your idea. >> >> One thing at a time though, we badly badly need unit tests right now and I >> recommend we focus most of our efforts in the next few months in building >> the unit tests as soon as I'm done with implementing it (hopefully very >> soon) >> >> Then again I also think the order logic is overly complex and should also >> be eventually simplified. I am drifting too much aren't I? Okay I will zip >> it :) >> >> Taher Alkhateeb >> On Jun 17, 2016 11:16 AM, "Jacques Le Roux" <[hidden email]> >> wrote: >> >>> Thanks Taher, >>> >>> The diagram says it all indeed. There are certainly crucial parts of the >>> UI we would like to cover, like order creation (in both order manager and >>> variants in ecommerce)... and such... >>> >>> Jacques >>> >>> >>> Le 17/06/2016 à 08:26, Taher Alkhateeb a écrit : >>> >>>> Hi Gil, >>>> >>>> For me it's a simple rule, lots of unit tests, less integration tests, >>>> even >>>> less functional tests (selenium). But I think all are useful if done in >>>> proper quantity >>>> >>>> I like this diagram in a way summarizes it -> >>>> http://i.stack.imgur.com/fjQvQ.png >>>> >>>> Taher Alkhateeb >>>> >>>> On Fri, Jun 17, 2016 at 9:19 AM, gil portenseigne < >>>> [hidden email]> wrote: >>>> >>>> Hi, >>>>> Neogia selenium/webdriver test suite have been abandonned, and we >> decided >>>>> at Néréide to stop doing Selenium UI testing since it needs a lot of >>>>> effort >>>>> to maintain, they are very long to pass, and *weren't stable enough* >>>>> (maybe >>>>> that's because we used a lot of ajax area update...) in the way that >> not >>>>> changing code make the test go red/blue... >>>>> >>>>> Technically I used to convert my selenium test case to Junit4 class... >> No >>>>> problem with this technology >>>>> >>>>> From my experience, UI testing is a good way too ensure one or more >>>>> customer core process of an application through UI, but i faced so many >>>>> problems with it, that the word Utopia ring a bell to me. I'd rather >> like >>>>> integration tests like they are in OFBiz yet. >>>>> >>>>> Moreover, OOTB UI in OFBiz is not intended to be used directly for a >>>>> customer/user, without adapting it to his needs, i cannot imagine >>>>> testsuites covering all the processes provided by OFBiz, and >> maintaining >>>>> it... >>>>> I already have problem to find time to contribute in OFBiz as I wish, I >>>>> won't spend this time maintaining selenium testcase... >>>>> >>>>> It's a quite negative opinion here, i know that Olivier is still using >> it >>>>> for its project, he might have a different POV. >>>>> >>>>> Best Regards >>>>> >>>>> Gil >>>>> >>>>> P.S. : Why not some selenium tests in example component, to test >> widgets >>>>> and show user how to implement selenium testing within OFBiz, but >> that's >>>>> not the goal of the discussion. >>>>> >>>>> On 16/06/2016 21:40, Jacques Le Roux wrote: >>>>> >>>>> Hi, >>>>> >>>>> With the considerable HW effort, a lot of things are going on recently, >>>>> and it's hard to follow. I though noticed that we experience more and >>>>> more >>>>> regressions (not all related to HW effort, far from it). >>>>> >>>>> Fortunately it's so far mostly minor points and often related with the >>>>> UI, >>>>> OFBIZ-7346 and OFBIZ-7363 being counter examples (OFBIZ-7346 can be >>>>> critical) >>>>> >>>>> From my experience, w/o a QA person or team, it's very hard to detect >>>>> those side effects at the UI level when you refactor or fix it. I >>>>> remember >>>>> the (ex) Neogia team (mostly Erwan) tried to maintain a >>>>> Selenium/Webdriver >>>>> set of tests. I don't know if they continue/d. >>>>> >>>>> Since we spoke about Junit and unit tests recently, some prefer TestNG, >>>>> at >>>>> least coupled with Selenium <http://testng.org/doc/selenium.html> >>>>> http://testng.org/doc/selenium.html >>>>> >>>>> Does it make sense, do you think it's only an utopia? >>>>> >>>>> Thanks >>>>> >>>>> Jacques >>>>> >>>>> >>>>> >>>>> |
Re: We miss a QA team, we face too much regressions
|
Administrator
|
In reply to this post by taher
Yes I agree unit tests 1st , as the diagram shows: you don't build on sand.
Jacques Le 17/06/2016 à 10:34, Taher Alkhateeb a écrit : > Hi Jacques, > > Yeah makes sense, I like your idea. > > One thing at a time though, we badly badly need unit tests right now and I > recommend we focus most of our efforts in the next few months in building > the unit tests as soon as I'm done with implementing it (hopefully very > soon) > > Then again I also think the order logic is overly complex and should also > be eventually simplified. I am drifting too much aren't I? Okay I will zip > it :) > > Taher Alkhateeb > On Jun 17, 2016 11:16 AM, "Jacques Le Roux" <[hidden email]> > wrote: > >> Thanks Taher, >> >> The diagram says it all indeed. There are certainly crucial parts of the >> UI we would like to cover, like order creation (in both order manager and >> variants in ecommerce)... and such... >> >> Jacques >> >> >> Le 17/06/2016 à 08:26, Taher Alkhateeb a écrit : >> >>> Hi Gil, >>> >>> For me it's a simple rule, lots of unit tests, less integration tests, >>> even >>> less functional tests (selenium). But I think all are useful if done in >>> proper quantity >>> >>> I like this diagram in a way summarizes it -> >>> http://i.stack.imgur.com/fjQvQ.png >>> >>> Taher Alkhateeb >>> >>> On Fri, Jun 17, 2016 at 9:19 AM, gil portenseigne < >>> [hidden email]> wrote: >>> >>> Hi, >>>> Neogia selenium/webdriver test suite have been abandonned, and we decided >>>> at Néréide to stop doing Selenium UI testing since it needs a lot of >>>> effort >>>> to maintain, they are very long to pass, and *weren't stable enough* >>>> (maybe >>>> that's because we used a lot of ajax area update...) in the way that not >>>> changing code make the test go red/blue... >>>> >>>> Technically I used to convert my selenium test case to Junit4 class... No >>>> problem with this technology >>>> >>>> From my experience, UI testing is a good way too ensure one or more >>>> customer core process of an application through UI, but i faced so many >>>> problems with it, that the word Utopia ring a bell to me. I'd rather like >>>> integration tests like they are in OFBiz yet. >>>> >>>> Moreover, OOTB UI in OFBiz is not intended to be used directly for a >>>> customer/user, without adapting it to his needs, i cannot imagine >>>> testsuites covering all the processes provided by OFBiz, and maintaining >>>> it... >>>> I already have problem to find time to contribute in OFBiz as I wish, I >>>> won't spend this time maintaining selenium testcase... >>>> >>>> It's a quite negative opinion here, i know that Olivier is still using it >>>> for its project, he might have a different POV. >>>> >>>> Best Regards >>>> >>>> Gil >>>> >>>> P.S. : Why not some selenium tests in example component, to test widgets >>>> and show user how to implement selenium testing within OFBiz, but that's >>>> not the goal of the discussion. >>>> >>>> On 16/06/2016 21:40, Jacques Le Roux wrote: >>>> >>>> Hi, >>>> >>>> With the considerable HW effort, a lot of things are going on recently, >>>> and it's hard to follow. I though noticed that we experience more and >>>> more >>>> regressions (not all related to HW effort, far from it). >>>> >>>> Fortunately it's so far mostly minor points and often related with the >>>> UI, >>>> OFBIZ-7346 and OFBIZ-7363 being counter examples (OFBIZ-7346 can be >>>> critical) >>>> >>>> From my experience, w/o a QA person or team, it's very hard to detect >>>> those side effects at the UI level when you refactor or fix it. I >>>> remember >>>> the (ex) Neogia team (mostly Erwan) tried to maintain a >>>> Selenium/Webdriver >>>> set of tests. I don't know if they continue/d. >>>> >>>> Since we spoke about Junit and unit tests recently, some prefer TestNG, >>>> at >>>> least coupled with Selenium <http://testng.org/doc/selenium.html> >>>> http://testng.org/doc/selenium.html >>>> >>>> Does it make sense, do you think it's only an utopia? >>>> >>>> Thanks >>>> >>>> Jacques >>>> >>>> >>>> >>>> |
Re: We miss a QA team, we face too much regressions
|
|
In reply to this post by Jacques Le Roux
On 17/06/2016 5:19 AM, Jacques Le Roux wrote:
> Le 16/06/2016 à 22:53, Ron Wheeler a écrit : >> One of the side benefits of having a small number of committer's is >> that you prevent bad designs and poorly tested code getting into the >> trunk. >> The disadvantage is that the committers are easily overwhelmed by an >> active contributor community. > > Would you say that with 31 committers (most active) we are currently a > small number of committers? Are the committers able to verify the code committed? How many of the regressions should have been detected before the code was committed? How many of the regressions were caused by lack of documentation of existing features so that people broke things that were "hidden" relationships? It is hard to build and maintain a bullet-proof integration test suite so human engineering is still a big part of the solution. > >> You may want to put in some rules about unit tests so that code >> without adequate test coverage can not be updated until the unit >> tests are sufficient for the committer to feel confident about >> accepting it. This may cause people to work on tests for stuff that >> they did not write but are considered key functionality in the >> modules being updated. >> There is no free ride and if you allow people to build up the >> technical debt of the project in order to meet their own deadlines, >> you will eventually have to face a large debt that comes due. >> >> Taher is paying off the debt in the framework which is a great >> contribution. >> It may be that others are going to have to take up the challenge in >> the application side. >> You may have to have a short moratorium on enhancements until the >> debt is reduced to a manageable level. >> >> There may also be the issue of people modifying too many layers at >> once so changes affect a lot of different services so unpleasant >> side-effects are easier to generate. >> >> Are the regressions caused by a small group of contributors or from >> updates going through a few committers? > > As I said it's recently fortunately small things. For now it's hard to > answer to your question, because the HW effort is rip-roaring. I guess > when it will settle a lot of things will be better/fixed, in the > meantime me will certainly face some uncertainty. > My question was not about pointing finger put how to prevent issues. > Hence my question about Selenium because our current set of tests is > obviously not enough. > Your suggestion about more unit tests and rules is certainly to > consider. I'd wait the end of the rip-roaring HW effort, for things to > stabilise, and then try to introduce more constraints, or should we > discuss it right away, community? > When you are in a hole, the first think to do is to stop digging! >> >> It is an open source project so there has to be some sensitivity >> about asking people to do a bit more to clean up old debt but if that >> is a problem and it is not addressed, it can be a big mess. > > I see a lot of skilled good will and clearly success for the last few > years. I think we can achieve it, OFBiz is here to stay! > > Jacques > >> >> >> Ron >> >> >> On 16/06/2016 3:48 PM, Taher Alkhateeb wrote: >>> Hi Jacques, >>> >>> Selenium tests cannot be unit tests in OFBiz because it requires >>> firing up >>> the server. You can consider them part of the integration tests >>> (existing >>> functionality). In fact, I would consider selenium tests to be >>> functional >>> tests (higher than integration) -> >>> https://en.wikipedia.org/wiki/Functional_testing >>> >>> So yeah we can add them, but I don't think we can do that to the raw >>> unit-tests (at least in the context discussed in the other proposal >>> thread) >>> >>> Taher Alkhateeb >>> >>> On Thu, Jun 16, 2016 at 10:40 PM, Jacques Le Roux < >>> [hidden email]> wrote: >>> >>>> Hi, >>>> >>>> With the considerable HW effort, a lot of things are going on >>>> recently, >>>> and it's hard to follow. I though noticed that we experience more >>>> and more >>>> regressions (not all related to HW effort, far from it). >>>> >>>> Fortunately it's so far mostly minor points and often related with >>>> the UI, >>>> OFBIZ-7346 and OFBIZ-7363 being counter examples (OFBIZ-7346 can be >>>> critical) >>>> >>>> From my experience, w/o a QA person or team, it's very hard to detect >>>> those side effects at the UI level when you refactor or fix it. I >>>> remember >>>> the (ex) Neogia team (mostly Erwan) tried to maintain a >>>> Selenium/Webdriver >>>> set of tests. I don't know if they continue/d. >>>> >>>> Since we spoke about Junit and unit tests recently, some prefer >>>> TestNG, at >>>> least coupled with Selenium http://testng.org/doc/selenium.html >>>> >>>> Does it make sense, do you think it's only an utopia? >>>> >>>> Thanks >>>> >>>> Jacques >>>> >>>> >> >> > > -- Ron Wheeler President Artifact Software Inc email: [hidden email] skype: ronaldmwheeler phone: 866-970-2435, ext 102 |
Re: We miss a QA team, we face too much regressions
|
Administrator
|
Le 17/06/2016 à 14:03, Ron Wheeler a écrit : > On 17/06/2016 5:19 AM, Jacques Le Roux wrote: >> Le 16/06/2016 à 22:53, Ron Wheeler a écrit : >>> One of the side benefits of having a small number of committer's is that you prevent bad designs and poorly tested code getting into the trunk. >>> The disadvantage is that the committers are easily overwhelmed by an active contributor community. >> >> Would you say that with 31 committers (most active) we are currently a small number of committers? > > Are the committers able to verify the code committed? I believe so > How many of the regressions should have been detected before the code was committed? I have no ideas > > How many of the regressions were caused by lack of documentation of existing features so that people broke things that were "hidden" relationships? One part of the project which cruelly lacks documentation is the UI of the content component. But the problems appeared mostly which changes related with FOP and Birt because upgrading/refactoring/improving code is not always a task as easy as it may look > > It is hard to build and maintain a bullet-proof integration test suite so human engineering is still a big part of the solution. Right, I'm still convinced some high level Selenium tests would help > >> >>> You may want to put in some rules about unit tests so that code without adequate test coverage can not be updated until the unit tests are >>> sufficient for the committer to feel confident about accepting it. This may cause people to work on tests for stuff that they did not write but >>> are considered key functionality in the modules being updated. >>> There is no free ride and if you allow people to build up the technical debt of the project in order to meet their own deadlines, you will >>> eventually have to face a large debt that comes due. >>> >>> Taher is paying off the debt in the framework which is a great contribution. >>> It may be that others are going to have to take up the challenge in the application side. >>> You may have to have a short moratorium on enhancements until the debt is reduced to a manageable level. >>> >>> There may also be the issue of people modifying too many layers at once so changes affect a lot of different services so unpleasant side-effects >>> are easier to generate. >>> >>> Are the regressions caused by a small group of contributors or from updates going through a few committers? >> >> As I said it's recently fortunately small things. For now it's hard to answer to your question, because the HW effort is rip-roaring. I guess when >> it will settle a lot of things will be better/fixed, in the meantime me will certainly face some uncertainty. >> My question was not about pointing finger put how to prevent issues. Hence my question about Selenium because our current set of tests is obviously >> not enough. >> Your suggestion about more unit tests and rules is certainly to consider. I'd wait the end of the rip-roaring HW effort, for things to stabilise, >> and then try to introduce more constraints, or should we discuss it right away, community? >> > > When you are in a hole, the first think to do is to stop digging! I'm not sure how to interpret this injunction ;) Jacques > > >>> >>> It is an open source project so there has to be some sensitivity about asking people to do a bit more to clean up old debt but if that is a >>> problem and it is not addressed, it can be a big mess. >> >> I see a lot of skilled good will and clearly success for the last few years. I think we can achieve it, OFBiz is here to stay! >> >> Jacques >> >>> >>> >>> Ron >>> >>> >>> On 16/06/2016 3:48 PM, Taher Alkhateeb wrote: >>>> Hi Jacques, >>>> >>>> Selenium tests cannot be unit tests in OFBiz because it requires firing up >>>> the server. You can consider them part of the integration tests (existing >>>> functionality). In fact, I would consider selenium tests to be functional >>>> tests (higher than integration) -> >>>> https://en.wikipedia.org/wiki/Functional_testing >>>> >>>> So yeah we can add them, but I don't think we can do that to the raw >>>> unit-tests (at least in the context discussed in the other proposal thread) >>>> >>>> Taher Alkhateeb >>>> >>>> On Thu, Jun 16, 2016 at 10:40 PM, Jacques Le Roux < >>>> [hidden email]> wrote: >>>> >>>>> Hi, >>>>> >>>>> With the considerable HW effort, a lot of things are going on recently, >>>>> and it's hard to follow. I though noticed that we experience more and more >>>>> regressions (not all related to HW effort, far from it). >>>>> >>>>> Fortunately it's so far mostly minor points and often related with the UI, >>>>> OFBIZ-7346 and OFBIZ-7363 being counter examples (OFBIZ-7346 can be >>>>> critical) >>>>> >>>>> From my experience, w/o a QA person or team, it's very hard to detect >>>>> those side effects at the UI level when you refactor or fix it. I remember >>>>> the (ex) Neogia team (mostly Erwan) tried to maintain a Selenium/Webdriver >>>>> set of tests. I don't know if they continue/d. >>>>> >>>>> Since we spoke about Junit and unit tests recently, some prefer TestNG, at >>>>> least coupled with Selenium http://testng.org/doc/selenium.html >>>>> >>>>> Does it make sense, do you think it's only an utopia? >>>>> >>>>> Thanks >>>>> >>>>> Jacques >>>>> >>>>> >>> >>> >> >> > > |
Re: We miss a QA team, we face too much regressions
|
|
Jacques and others,
Thanks for your kind support to team of developers working so many tickets at the same time. Some of the active engineers from HW are also skilled at QA practices and tools to automate testing and they will be super happy to start contributing selenium tests. Before we can start writing selenium tests it will be good to have community agree on expected system behavior and document them. We do have good start on it, https://cwiki.apache.org/confluence/display/OFBIZ/Business+Process+Stories+and+Use+Cases+Library <https://cwiki.apache.org/confluence/display/OFBIZ/Business+Process+Stories+and+Use+Cases+Library> Is the fact that the use cases are posted on OFBiz confluence, enough to say that these are the expected system behavior OR Do we need to take time to review them and declare them accepted (we may use Jira ticket to track the review activity and then close the ticket when we have the agreement)? The developer team at HW is excited to do whatever it takes to increase OFBiz adoption and they will be happy to develop selenium tests. Thanks and Regards Anil Patel COO Hotwax Systems http://www.hotwaxsystems.com/ <http://www.hotwaxmedia.com/> Cell: +1 509 398 3120 > On Jun 18, 2016, at 12:10 AM, Jacques Le Roux <[hidden email]> wrote: > > > Le 17/06/2016 à 14:03, Ron Wheeler a écrit : >> On 17/06/2016 5:19 AM, Jacques Le Roux wrote: >>> Le 16/06/2016 à 22:53, Ron Wheeler a écrit : >>>> One of the side benefits of having a small number of committer's is that you prevent bad designs and poorly tested code getting into the trunk. >>>> The disadvantage is that the committers are easily overwhelmed by an active contributor community. >>> >>> Would you say that with 31 committers (most active) we are currently a small number of committers? >> >> Are the committers able to verify the code committed? > > I believe so > >> How many of the regressions should have been detected before the code was committed? > > I have no ideas > >> >> How many of the regressions were caused by lack of documentation of existing features so that people broke things that were "hidden" relationships? > > One part of the project which cruelly lacks documentation is the UI of the content component. But the problems appeared mostly which changes related with FOP and Birt because upgrading/refactoring/improving code is not always a task as easy as it may look > >> >> It is hard to build and maintain a bullet-proof integration test suite so human engineering is still a big part of the solution. > > Right, I'm still convinced some high level Selenium tests would help > >> >>> >>>> You may want to put in some rules about unit tests so that code without adequate test coverage can not be updated until the unit tests are sufficient for the committer to feel confident about accepting it. This may cause people to work on tests for stuff that they did not write but are considered key functionality in the modules being updated. >>>> There is no free ride and if you allow people to build up the technical debt of the project in order to meet their own deadlines, you will eventually have to face a large debt that comes due. >>>> >>>> Taher is paying off the debt in the framework which is a great contribution. >>>> It may be that others are going to have to take up the challenge in the application side. >>>> You may have to have a short moratorium on enhancements until the debt is reduced to a manageable level. >>>> >>>> There may also be the issue of people modifying too many layers at once so changes affect a lot of different services so unpleasant side-effects are easier to generate. >>>> >>>> Are the regressions caused by a small group of contributors or from updates going through a few committers? >>> >>> As I said it's recently fortunately small things. For now it's hard to answer to your question, because the HW effort is rip-roaring. I guess when it will settle a lot of things will be better/fixed, in the meantime me will certainly face some uncertainty. >>> My question was not about pointing finger put how to prevent issues. Hence my question about Selenium because our current set of tests is obviously not enough. >>> Your suggestion about more unit tests and rules is certainly to consider. I'd wait the end of the rip-roaring HW effort, for things to stabilise, and then try to introduce more constraints, or should we discuss it right away, community? >>> >> >> When you are in a hole, the first think to do is to stop digging! > > I'm not sure how to interpret this injunction ;) > > Jacques > >> >> >>>> >>>> It is an open source project so there has to be some sensitivity about asking people to do a bit more to clean up old debt but if that is a problem and it is not addressed, it can be a big mess. >>> >>> I see a lot of skilled good will and clearly success for the last few years. I think we can achieve it, OFBiz is here to stay! >>> >>> Jacques >>> >>>> >>>> >>>> Ron >>>> >>>> >>>> On 16/06/2016 3:48 PM, Taher Alkhateeb wrote: >>>>> Hi Jacques, >>>>> >>>>> Selenium tests cannot be unit tests in OFBiz because it requires firing up >>>>> the server. You can consider them part of the integration tests (existing >>>>> functionality). In fact, I would consider selenium tests to be functional >>>>> tests (higher than integration) -> >>>>> https://en.wikipedia.org/wiki/Functional_testing >>>>> >>>>> So yeah we can add them, but I don't think we can do that to the raw >>>>> unit-tests (at least in the context discussed in the other proposal thread) >>>>> >>>>> Taher Alkhateeb >>>>> >>>>> On Thu, Jun 16, 2016 at 10:40 PM, Jacques Le Roux < >>>>> [hidden email]> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> With the considerable HW effort, a lot of things are going on recently, >>>>>> and it's hard to follow. I though noticed that we experience more and more >>>>>> regressions (not all related to HW effort, far from it). >>>>>> >>>>>> Fortunately it's so far mostly minor points and often related with the UI, >>>>>> OFBIZ-7346 and OFBIZ-7363 being counter examples (OFBIZ-7346 can be >>>>>> critical) >>>>>> >>>>>> From my experience, w/o a QA person or team, it's very hard to detect >>>>>> those side effects at the UI level when you refactor or fix it. I remember >>>>>> the (ex) Neogia team (mostly Erwan) tried to maintain a Selenium/Webdriver >>>>>> set of tests. I don't know if they continue/d. >>>>>> >>>>>> Since we spoke about Junit and unit tests recently, some prefer TestNG, at >>>>>> least coupled with Selenium http://testng.org/doc/selenium.html >>>>>> >>>>>> Does it make sense, do you think it's only an utopia? >>>>>> >>>>>> Thanks >>>>>> >>>>>> Jacques |
Re: We miss a QA team, we face too much regressions
|
Administrator
|
Hi Anil,
Le 18/06/2016 à 22:21, Anil Patel a écrit : > Jacques and others, > > Thanks for your kind support to team of developers working so many tickets at the same time. I'm very happy to see such an activity, and I must confess it's hard to follow all progresses but definitely exciting! > Some of the active engineers from HW are also skilled at QA practices and tools to automate testing and they will be super happy to start contributing selenium tests. Looking forward :) > Before we can start writing selenium tests it will be good to have community agree on expected system behavior and document them. We do have good start on it, > https://cwiki.apache.org/confluence/display/OFBIZ/Business+Process+Stories+and+Use+Cases+Library <https://cwiki.apache.org/confluence/display/OFBIZ/Business+Process+Stories+and+Use+Cases+Library> > > Is the fact that the use cases are posted on OFBiz confluence, enough to say that these are the expected system behavior > > OR > > Do we need to take time to review them and declare them accepted (we may use Jira ticket to track the review activity and then close the ticket when we have the agreement)? I think we can trust this work, and complete/step in when necessary > The developer team at HW is excited to do whatever it takes to increase OFBiz adoption and they will be happy to develop selenium tests. Sounds like a start, great! Jacques > > > Thanks and Regards > Anil Patel > COO > Hotwax Systems > http://www.hotwaxsystems.com/ <http://www.hotwaxmedia.com/> > Cell: +1 509 398 3120 > >> On Jun 18, 2016, at 12:10 AM, Jacques Le Roux <[hidden email]> wrote: >> >> >> Le 17/06/2016 à 14:03, Ron Wheeler a écrit : >>> On 17/06/2016 5:19 AM, Jacques Le Roux wrote: >>>> Le 16/06/2016 à 22:53, Ron Wheeler a écrit : >>>>> One of the side benefits of having a small number of committer's is that you prevent bad designs and poorly tested code getting into the trunk. >>>>> The disadvantage is that the committers are easily overwhelmed by an active contributor community. >>>> Would you say that with 31 committers (most active) we are currently a small number of committers? >>> Are the committers able to verify the code committed? >> I believe so >> >>> How many of the regressions should have been detected before the code was committed? >> I have no ideas >> >>> How many of the regressions were caused by lack of documentation of existing features so that people broke things that were "hidden" relationships? >> One part of the project which cruelly lacks documentation is the UI of the content component. But the problems appeared mostly which changes related with FOP and Birt because upgrading/refactoring/improving code is not always a task as easy as it may look >> >>> It is hard to build and maintain a bullet-proof integration test suite so human engineering is still a big part of the solution. >> Right, I'm still convinced some high level Selenium tests would help >> >>>>> You may want to put in some rules about unit tests so that code without adequate test coverage can not be updated until the unit tests are sufficient for the committer to feel confident about accepting it. This may cause people to work on tests for stuff that they did not write but are considered key functionality in the modules being updated. >>>>> There is no free ride and if you allow people to build up the technical debt of the project in order to meet their own deadlines, you will eventually have to face a large debt that comes due. >>>>> >>>>> Taher is paying off the debt in the framework which is a great contribution. >>>>> It may be that others are going to have to take up the challenge in the application side. >>>>> You may have to have a short moratorium on enhancements until the debt is reduced to a manageable level. >>>>> >>>>> There may also be the issue of people modifying too many layers at once so changes affect a lot of different services so unpleasant side-effects are easier to generate. >>>>> >>>>> Are the regressions caused by a small group of contributors or from updates going through a few committers? >>>> As I said it's recently fortunately small things. For now it's hard to answer to your question, because the HW effort is rip-roaring. I guess when it will settle a lot of things will be better/fixed, in the meantime me will certainly face some uncertainty. >>>> My question was not about pointing finger put how to prevent issues. Hence my question about Selenium because our current set of tests is obviously not enough. >>>> Your suggestion about more unit tests and rules is certainly to consider. I'd wait the end of the rip-roaring HW effort, for things to stabilise, and then try to introduce more constraints, or should we discuss it right away, community? >>>> >>> When you are in a hole, the first think to do is to stop digging! >> I'm not sure how to interpret this injunction ;) >> >> Jacques >> >>> >>>>> It is an open source project so there has to be some sensitivity about asking people to do a bit more to clean up old debt but if that is a problem and it is not addressed, it can be a big mess. >>>> I see a lot of skilled good will and clearly success for the last few years. I think we can achieve it, OFBiz is here to stay! >>>> >>>> Jacques >>>> >>>>> >>>>> Ron >>>>> >>>>> >>>>> On 16/06/2016 3:48 PM, Taher Alkhateeb wrote: >>>>>> Hi Jacques, >>>>>> >>>>>> Selenium tests cannot be unit tests in OFBiz because it requires firing up >>>>>> the server. You can consider them part of the integration tests (existing >>>>>> functionality). In fact, I would consider selenium tests to be functional >>>>>> tests (higher than integration) -> >>>>>> https://en.wikipedia.org/wiki/Functional_testing >>>>>> >>>>>> So yeah we can add them, but I don't think we can do that to the raw >>>>>> unit-tests (at least in the context discussed in the other proposal thread) >>>>>> >>>>>> Taher Alkhateeb >>>>>> >>>>>> On Thu, Jun 16, 2016 at 10:40 PM, Jacques Le Roux < >>>>>> [hidden email]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> With the considerable HW effort, a lot of things are going on recently, >>>>>>> and it's hard to follow. I though noticed that we experience more and more >>>>>>> regressions (not all related to HW effort, far from it). >>>>>>> >>>>>>> Fortunately it's so far mostly minor points and often related with the UI, >>>>>>> OFBIZ-7346 and OFBIZ-7363 being counter examples (OFBIZ-7346 can be >>>>>>> critical) >>>>>>> >>>>>>> From my experience, w/o a QA person or team, it's very hard to detect >>>>>>> those side effects at the UI level when you refactor or fix it. I remember >>>>>>> the (ex) Neogia team (mostly Erwan) tried to maintain a Selenium/Webdriver >>>>>>> set of tests. I don't know if they continue/d. >>>>>>> >>>>>>> Since we spoke about Junit and unit tests recently, some prefer TestNG, at >>>>>>> least coupled with Selenium http://testng.org/doc/selenium.html >>>>>>> >>>>>>> Does it make sense, do you think it's only an utopia? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Jacques > |
«
Return to OFBiz - Dev
|
1 view|%1 views
| Free forum by Nabble | Edit this page |