Integration tests take a very long time

[taher]

Hi Folks,

As you may recall, we introduced unit-tests to OFBiz in addition to the
existing integration tests to try and balance the long development cycle
that we currently suffer from.

The total number of tests on the latest version of Trunk on my machine is
1929, and the total time it takes to complete all tests using the command
./gradlew cleanAll loadDefault testIntegration is almost 20 minutes!
Something is wrong, and in my opinion it is that we have way too many
integration tests and very few unit tests. Integration tests are resource
hungry and a strain on the system and heavily slow down the development
cycle.

So I urge everyone interested in writing tests to prefer unit tests over
integration tests where possible, or even better, to try rewrite some of
the integration tests as unit tests, or to increase the efficiency of the
algorithms for testing. Our development cycle is becoming very slow because
of the mountain of integration tests that reside in the framework.

If you are interested, Jacopo already wrote a few tests with mocking using
Mockito. You can use these tests as a reference to help you in writing
efficient and fast unit tests.

Cheers,

Taher Alkhateeb

[Scott Gray-3]

FYI latest trunk takes almost exactly 5 minutes on my machine.

Regards
Scott

On 16 November 2016 at 06:34, Taher Alkhateeb <[hidden email]>
wrote:

> Hi Folks,
>
> As you may recall, we introduced unit-tests to OFBiz in addition to the
> existing integration tests to try and balance the long development cycle
> that we currently suffer from.
>
> The total number of tests on the latest version of Trunk on my machine is
> 1929, and the total time it takes to complete all tests using the command
> ./gradlew cleanAll loadDefault testIntegration is almost 20 minutes!
> Something is wrong, and in my opinion it is that we have way too many
> integration tests and very few unit tests. Integration tests are resource
> hungry and a strain on the system and heavily slow down the development
> cycle.
>
> So I urge everyone interested in writing tests to prefer unit tests over
> integration tests where possible, or even better, to try rewrite some of
> the integration tests as unit tests, or to increase the efficiency of the
> algorithms for testing. Our development cycle is becoming very slow because
> of the mountain of integration tests that reside in the framework.
>
> If you are interested, Jacopo already wrote a few tests with mocking using
> Mockito. You can use these tests as a reference to help you in writing
> efficient and fast unit tests.
>
> Cheers,
>
> Taher Alkhateeb
>

[Jacques Le Roux]

I see only 1292 tests on BuildBot https://ci.apache.org/projects/ofbiz/logs/trunk/html/ (I guess you had a typo)

It's  4 mins, 25 secs for https://ci.apache.org/builders/ofbiz-trunk/builds/1745 (trunk HEAD, step 3)

It was the same a month ago (can't get back more)

It's  9 to 10 mins on my machine, was around 6 mins two months ago and I remember around 3 mins 6 months ago

So yes it's longer. Now we maybe don't need to run all tests all the time but on Buildbot. I mean it's a common practise to subdivide tests in
different levels (some speak about 4 levels). The same could be applied to what we call integration testing now. In order to not run the longer tests
all the time. Then we need to decide if it's possible and if possible where/what to cut and separate. I mean not only to transform current integration
tests to unit tests which is certainly a good idea where applicable.

BTW unit tests also slow down builds. For the moment not that much and it's always possible to not run them with each build (gradlew build -x test).
You can also bypass integration tests for simple changes and rely on BuildBot, at least for OFBiz repo commits. Or you may run only a component test
suite if your changes concern only a component, or several test suites but not all, etc.

Improving tests efficiency is also a good idea, but we then need to profile and see where it's interesting to do so.

Jacques


Le 15/11/2016 à 22:59, Scott Gray a écrit :

> FYI latest trunk takes almost exactly 5 minutes on my machine.
>
> Regards
> Scott
>
> On 16 November 2016 at 06:34, Taher Alkhateeb <[hidden email]>
> wrote:
>
>> Hi Folks,
>>
>> As you may recall, we introduced unit-tests to OFBiz in addition to the
>> existing integration tests to try and balance the long development cycle
>> that we currently suffer from.
>>
>> The total number of tests on the latest version of Trunk on my machine is
>> 1929, and the total time it takes to complete all tests using the command
>> ./gradlew cleanAll loadDefault testIntegration is almost 20 minutes!
>> Something is wrong, and in my opinion it is that we have way too many
>> integration tests and very few unit tests. Integration tests are resource
>> hungry and a strain on the system and heavily slow down the development
>> cycle.
>>
>> So I urge everyone interested in writing tests to prefer unit tests over
>> integration tests where possible, or even better, to try rewrite some of
>> the integration tests as unit tests, or to increase the efficiency of the
>> algorithms for testing. Our development cycle is becoming very slow because
>> of the mountain of integration tests that reside in the framework.
>>
>> If you are interested, Jacopo already wrote a few tests with mocking using
>> Mockito. You can use these tests as a reference to help you in writing
>> efficient and fast unit tests.
>>
>> Cheers,
>>
>> Taher Alkhateeb
>>

[taher]

Hi Scott. Does that include loading data? Do you run ./gradlew cleanAll
loadDefault testIntegration? Or are you referring only to the tests?

Regardless of execution time, it is definitely _longer_ than it was in the
past, which is the reason why I started this thread in the first place.

On Wed, Nov 16, 2016 at 12:59 AM, Scott Gray <[hidden email]>
wrote:

> FYI latest trunk takes almost exactly 5 minutes on my machine.
>
> Regards
> Scott
>
> On 16 November 2016 at 06:34, Taher Alkhateeb <[hidden email]>
> wrote:
>
> > Hi Folks,
> >
> > As you may recall, we introduced unit-tests to OFBiz in addition to the
> > existing integration tests to try and balance the long development cycle
> > that we currently suffer from.
> >
> > The total number of tests on the latest version of Trunk on my machine is
> > 1929, and the total time it takes to complete all tests using the command
> > ./gradlew cleanAll loadDefault testIntegration is almost 20 minutes!
> > Something is wrong, and in my opinion it is that we have way too many
> > integration tests and very few unit tests. Integration tests are resource
> > hungry and a strain on the system and heavily slow down the development
> > cycle.
> >
> > So I urge everyone interested in writing tests to prefer unit tests over
> > integration tests where possible, or even better, to try rewrite some of
> > the integration tests as unit tests, or to increase the efficiency of the
> > algorithms for testing. Our development cycle is becoming very slow
> because
> > of the mountain of integration tests that reside in the framework.
> >
> > If you are interested, Jacopo already wrote a few tests with mocking
> using
> > Mockito. You can use these tests as a reference to help you in writing
> > efficient and fast unit tests.
> >
> > Cheers,
> >
> > Taher Alkhateeb
> >
>

[Jacopo Cappellato-5]

./gradlew cleanAll loadDefault testIntegration

takes 5 minutes and 20 seconds in my Mac as well (which is similar to what
Scott is using I think) but I know that it can take much longer in other
boxes: I think the main reason is because our tests performs a lot of IO
and on systems using SSD the time is much shorter.
IO traffic is mostly generated by db calls and logging and I think we could
improve the performance by disabling most of the logging when we are
running the tests.
Taher, I agree that now that we have a good framework to run unit tests we
should review our integration tests and identify the ones that should
become unit tests, and convert them to the new style.

Jacopo


On Wed, Nov 16, 2016 at 5:20 PM, Taher Alkhateeb <[hidden email]
> wrote:

> Hi Scott. Does that include loading data? Do you run ./gradlew cleanAll
> loadDefault testIntegration? Or are you referring only to the tests?
>
> Regardless of execution time, it is definitely _longer_ than it was in the
> past, which is the reason why I started this thread in the first place.
>
> On Wed, Nov 16, 2016 at 12:59 AM, Scott Gray <[hidden email]
> >
> wrote:
>
> > FYI latest trunk takes almost exactly 5 minutes on my machine.
> >
> > Regards
> > Scott
> >
> > On 16 November 2016 at 06:34, Taher Alkhateeb <
> [hidden email]>
> > wrote:
> >
> > > Hi Folks,
> > >
> > > As you may recall, we introduced unit-tests to OFBiz in addition to the
> > > existing integration tests to try and balance the long development
> cycle
> > > that we currently suffer from.
> > >
> > > The total number of tests on the latest version of Trunk on my machine
> is
> > > 1929, and the total time it takes to complete all tests using the
> command
> > > ./gradlew cleanAll loadDefault testIntegration is almost 20 minutes!
> > > Something is wrong, and in my opinion it is that we have way too many
> > > integration tests and very few unit tests. Integration tests are
> resource
> > > hungry and a strain on the system and heavily slow down the development
> > > cycle.
> > >
> > > So I urge everyone interested in writing tests to prefer unit tests
> over
> > > integration tests where possible, or even better, to try rewrite some
> of
> > > the integration tests as unit tests, or to increase the efficiency of
> the
> > > algorithms for testing. Our development cycle is becoming very slow
> > because
> > > of the mountain of integration tests that reside in the framework.
> > >
> > > If you are interested, Jacopo already wrote a few tests with mocking
> > using
> > > Mockito. You can use these tests as a reference to help you in writing
> > > efficient and fast unit tests.
> > >
> > > Cheers,
> > >
> > > Taher Alkhateeb
> > >
> >
>

[Jacques Le Roux]

I have an SSD also (but my machine is 3 years old...). I noticed that when using Ubuntu in a VM on my machine it's faster (7 mins instead of 9 to 10),
we all know NTFS is slower than EXT3

I agree it's quite a good pragmatic idea to disable most of the logging when running tests, maybe by creating a testIntegration option ?

Jacques


Le 16/11/2016 à 18:00, Jacopo Cappellato a écrit :

> ./gradlew cleanAll loadDefault testIntegration
>
> takes 5 minutes and 20 seconds in my Mac as well (which is similar to what
> Scott is using I think) but I know that it can take much longer in other
> boxes: I think the main reason is because our tests performs a lot of IO
> and on systems using SSD the time is much shorter.
> IO traffic is mostly generated by db calls and logging and I think we could
> improve the performance by disabling most of the logging when we are
> running the tests.
> Taher, I agree that now that we have a good framework to run unit tests we
> should review our integration tests and identify the ones that should
> become unit tests, and convert them to the new style.
>
> Jacopo
>
>
> On Wed, Nov 16, 2016 at 5:20 PM, Taher Alkhateeb <[hidden email]
>> wrote:
>> Hi Scott. Does that include loading data? Do you run ./gradlew cleanAll
>> loadDefault testIntegration? Or are you referring only to the tests?
>>
>> Regardless of execution time, it is definitely _longer_ than it was in the
>> past, which is the reason why I started this thread in the first place.
>>
>> On Wed, Nov 16, 2016 at 12:59 AM, Scott Gray <[hidden email]
>> wrote:
>>
>>> FYI latest trunk takes almost exactly 5 minutes on my machine.
>>>
>>> Regards
>>> Scott
>>>
>>> On 16 November 2016 at 06:34, Taher Alkhateeb <
>> [hidden email]>
>>> wrote:
>>>
>>>> Hi Folks,
>>>>
>>>> As you may recall, we introduced unit-tests to OFBiz in addition to the
>>>> existing integration tests to try and balance the long development
>> cycle
>>>> that we currently suffer from.
>>>>
>>>> The total number of tests on the latest version of Trunk on my machine
>> is
>>>> 1929, and the total time it takes to complete all tests using the
>> command
>>>> ./gradlew cleanAll loadDefault testIntegration is almost 20 minutes!
>>>> Something is wrong, and in my opinion it is that we have way too many
>>>> integration tests and very few unit tests. Integration tests are
>> resource
>>>> hungry and a strain on the system and heavily slow down the development
>>>> cycle.
>>>>
>>>> So I urge everyone interested in writing tests to prefer unit tests
>> over
>>>> integration tests where possible, or even better, to try rewrite some
>> of
>>>> the integration tests as unit tests, or to increase the efficiency of
>> the
>>>> algorithms for testing. Our development cycle is becoming very slow
>>> because
>>>> of the mountain of integration tests that reside in the framework.
>>>>
>>>> If you are interested, Jacopo already wrote a few tests with mocking
>>> using
>>>> Mockito. You can use these tests as a reference to help you in writing
>>>> efficient and fast unit tests.
>>>>
>>>> Cheers,
>>>>
>>>> Taher Alkhateeb
>>>>