Re: buildbot failure in on ofbizTrunkFrameworkPlugins

[taher]

I am suspecting this is still a broken build system rather than a
genuine error. Can you confirm? If tests are running okay locally then
we still have a faulty build logic.

On Sun, Jan 7, 2018 at 5:20 PM,  <[hidden email]> wrote:

> The Buildbot has detected a new failure on builder ofbizTrunkFrameworkPlugins while building . Full details are available at:
>     https://ci.apache.org/builders/ofbizTrunkFrameworkPlugins/builds/10
>
> Buildbot URL: https://ci.apache.org/
>
> Buildslave for this Build: lares_ubuntu
>
> Build Reason: downstream
> Build Source Stamp: [branch ofbiz/ofbiz-framework/trunk] 1820457
> Blamelist: jleroux
>
> BUILD FAILED: failed shell_4
>
> Sincerely,
>  -The Buildbot
>
>
>

[Jacques Le Roux]

Le 07/01/2018 à 15:44, Taher Alkhateeb a écrit :
> I am suspecting this is still a broken build system rather than a
> genuine error. Can you confirm?
It's not a "broken build system" but was a transient error as you can see there
https://ci.apache.org/builders/ofbizTrunkFrameworkPlugins

> If tests are running okay locally
Yes they are, at least now...

> then we still have a faulty build logic.
Nope, it's not a "faulty build logic." just a transient error on build servers

When you see that tests pass locally and not on Buildbot it's most certainly an issue with servers.
Those are hard workers and make errors from time to time (which shows that not only human make errors)

When something like that happens, I get to IRC (using https://webchat.freenode.net)
There using a recognisable username (I use jleroux) I get to the ofbiz channel.
I wait for ofbiz-bot to appear, click on query and then I can make a request in the chat line to restart a scheduler.
A request is of the form
"force build ofbizTrunkFrameworkPlugins"
you put the text you want to appear after this expression, something like
"forces manual build after Buildbot weird 85 errors on labels :-o"

I hope this clarify. I'll put a .md file in tools\documentation\BuildBot and will import is as HTML in a new wiki page for BuildBot

I hope this will help us to get to a fruitful collaboration :)

BTW we are still waiting for something to happen, see https://issues.apache.org/jira/browse/INFRA-15785

Jacques

> On Sun, Jan 7, 2018 at 5:20 PM,  <[hidden email]> wrote:
>> The Buildbot has detected a new failure on builder ofbizTrunkFrameworkPlugins while building . Full details are available at:
>>      https://ci.apache.org/builders/ofbizTrunkFrameworkPlugins/builds/10
>>
>> Buildbot URL: https://ci.apache.org/
>>
>> Buildslave for this Build: lares_ubuntu
>>
>> Build Reason: downstream
>> Build Source Stamp: [branch ofbiz/ofbiz-framework/trunk] 1820457
>> Blamelist: jleroux
>>
>> BUILD FAILED: failed shell_4
>>
>> Sincerely,
>>   -The Buildbot
>>
>>
>>

[Jacques Le Roux]

Le 07/01/2018 à 18:41, Jacques Le Roux a écrit :
> A request is of the form
> "force build ofbizTrunkFrameworkPlugins"
> you put the text you want to appear after this expression, something like
> "forces manual build after Buildbot weird 85 errors on labels :-o"
Forgot to say that with our last config (see INFRA-15394) the ofbizTrunkFrameworkPlugins and ofbizBranch17FrameworkPlugins builders are dependent and
respectively automatically launched by the ofbizTrunkFramework and ofbizBranch17Framework builders but only on commits.
So if you use an IRC command like
"force build ofbizTrunkFramework" only this builder will be launched not the dependent ofbizTrunkFrameworkPlugins

HTH

Jacques

[taher]

Sorry, I'm not sure what you mean by a transient error? How did you
determine that it is a transient error, and did you identify what is
causing it? Why are we having more of these errors than in the past?

On Sun, Jan 7, 2018 at 8:41 PM, Jacques Le Roux
<[hidden email]> wrote:

> Le 07/01/2018 à 15:44, Taher Alkhateeb a écrit :
>>
>> I am suspecting this is still a broken build system rather than a
>> genuine error. Can you confirm?
>
> It's not a "broken build system" but was a transient error as you can see
> there
> https://ci.apache.org/builders/ofbizTrunkFrameworkPlugins
>
>> If tests are running okay locally
>
> Yes they are, at least now...
>
>> then we still have a faulty build logic.
>
> Nope, it's not a "faulty build logic." just a transient error on build
> servers
>
> When you see that tests pass locally and not on Buildbot it's most certainly
> an issue with servers.
> Those are hard workers and make errors from time to time (which shows that
> not only human make errors)
>
> When something like that happens, I get to IRC (using
> https://webchat.freenode.net)
> There using a recognisable username (I use jleroux) I get to the ofbiz
> channel.
> I wait for ofbiz-bot to appear, click on query and then I can make a request
> in the chat line to restart a scheduler.
> A request is of the form
> "force build ofbizTrunkFrameworkPlugins"
> you put the text you want to appear after this expression, something like
> "forces manual build after Buildbot weird 85 errors on labels :-o"
>
> I hope this clarify. I'll put a .md file in tools\documentation\BuildBot and
> will import is as HTML in a new wiki page for BuildBot
>
> I hope this will help us to get to a fruitful collaboration :)
>
> BTW we are still waiting for something to happen, see
> https://issues.apache.org/jira/browse/INFRA-15785
>
> Jacques
>
>
>> On Sun, Jan 7, 2018 at 5:20 PM,  <[hidden email]> wrote:
>>>
>>> The Buildbot has detected a new failure on builder
>>> ofbizTrunkFrameworkPlugins while building . Full details are available at:
>>>      https://ci.apache.org/builders/ofbizTrunkFrameworkPlugins/builds/10
>>>
>>> Buildbot URL: https://ci.apache.org/
>>>
>>> Buildslave for this Build: lares_ubuntu
>>>
>>> Build Reason: downstream
>>> Build Source Stamp: [branch ofbiz/ofbiz-framework/trunk] 1820457
>>> Blamelist: jleroux
>>>
>>> BUILD FAILED: failed shell_4
>>>
>>> Sincerely,
>>>   -The Buildbot
>>>
>>>
>>>
>

[Jacques Le Roux]

Le 07/01/2018 à 21:08, Taher Alkhateeb a écrit :
> Sorry, I'm not sure what you mean by a transient error?
A test error which happens only once.  The same code (no commits between) is used few minutes later and it works.
If you want to know more about possible such issues please ask infra.
I was told in the pass that such things happen. Why do you think I use IRC? I did not discover this by myself.

> How did you determine that it is a transient error,
Because it works locally and, when it's launched from IRC just after, the same code passes the tests on Buildbot also.

> and did you identify what is causing it?
That I have not ideas and you need to ask infra. There is one sentence I often heard when asking: "shit happen" (you certainly know about the fan too)
I was also told that disk and especially network issues happen. The ASF is far from owning all the machines and IIRW the Buildbot machines:
https://ci.apache.org/

> Why are we having more of these errors than in the past?
I think you mix repetitive false warning, which was fixed with INFRA-15394, with these kinds of issues:

 1. svn fails to update
 2. tests pass locally but not on Buildbot

which happen from time to time and not specially more now. Then IRC, with the restriction I explained below, is the solution.

Jacques

>
> On Sun, Jan 7, 2018 at 8:41 PM, Jacques Le Roux
> <[hidden email]> wrote:
>> Le 07/01/2018 à 15:44, Taher Alkhateeb a écrit :
>>> I am suspecting this is still a broken build system rather than a
>>> genuine error. Can you confirm?
>> It's not a "broken build system" but was a transient error as you can see
>> there
>> https://ci.apache.org/builders/ofbizTrunkFrameworkPlugins
>>
>>> If tests are running okay locally
>> Yes they are, at least now...
>>
>>> then we still have a faulty build logic.
>> Nope, it's not a "faulty build logic." just a transient error on build
>> servers
>>
>> When you see that tests pass locally and not on Buildbot it's most certainly
>> an issue with servers.
>> Those are hard workers and make errors from time to time (which shows that
>> not only human make errors)
>>
>> When something like that happens, I get to IRC (using
>> https://webchat.freenode.net)
>> There using a recognisable username (I use jleroux) I get to the ofbiz
>> channel.
>> I wait for ofbiz-bot to appear, click on query and then I can make a request
>> in the chat line to restart a scheduler.
>> A request is of the form
>> "force build ofbizTrunkFrameworkPlugins"
>> you put the text you want to appear after this expression, something like
>> "forces manual build after Buildbot weird 85 errors on labels :-o"
>>
>> I hope this clarify. I'll put a .md file in tools\documentation\BuildBot and
>> will import is as HTML in a new wiki page for BuildBot
>>
>> I hope this will help us to get to a fruitful collaboration :)
>>
>> BTW we are still waiting for something to happen, see
>> https://issues.apache.org/jira/browse/INFRA-15785
>>
>> Jacques
>>
>>
>>> On Sun, Jan 7, 2018 at 5:20 PM,  <[hidden email]> wrote:
>>>> The Buildbot has detected a new failure on builder
>>>> ofbizTrunkFrameworkPlugins while building . Full details are available at:
>>>>       https://ci.apache.org/builders/ofbizTrunkFrameworkPlugins/builds/10
>>>>
>>>> Buildbot URL: https://ci.apache.org/
>>>>
>>>> Buildslave for this Build: lares_ubuntu
>>>>
>>>> Build Reason: downstream
>>>> Build Source Stamp: [branch ofbiz/ofbiz-framework/trunk] 1820457
>>>> Blamelist: jleroux
>>>>
>>>> BUILD FAILED: failed shell_4
>>>>
>>>> Sincerely,
>>>>    -The Buildbot
>>>>
>>>>
>>>>

[Jacques Le Roux]

Hi Taher, All,

I put a small BuildBot documentation at https://svn.apache.org/repos/asf/ofbiz/tools/documentation/BuildBot I'll now create a wiki page to import it

Everybody is welcome to try and improve it

Jacques

Le 07/01/2018 à 23:54, Jacques Le Roux a écrit :

>
> Le 07/01/2018 à 21:08, Taher Alkhateeb a écrit :
>> Sorry, I'm not sure what you mean by a transient error?
> A test error which happens only once.  The same code (no commits between) is used few minutes later and it works.
> If you want to know more about possible such issues please ask infra.
> I was told in the pass that such things happen. Why do you think I use IRC? I did not discover this by myself.
>
>> How did you determine that it is a transient error,
> Because it works locally and, when it's launched from IRC just after, the same code passes the tests on Buildbot also.
>
>> and did you identify what is causing it?
> That I have not ideas and you need to ask infra. There is one sentence I often heard when asking: "shit happen" (you certainly know about the fan too)
> I was also told that disk and especially network issues happen. The ASF is far from owning all the machines and IIRW the Buildbot machines:
> https://ci.apache.org/
>
>> Why are we having more of these errors than in the past?
> I think you mix repetitive false warning, which was fixed with INFRA-15394, with these kinds of issues:
>
> 1. svn fails to update
> 2. tests pass locally but not on Buildbot
>
> which happen from time to time and not specially more now. Then IRC, with the restriction I explained below, is the solution.
>
> Jacques
>
>>
>> On Sun, Jan 7, 2018 at 8:41 PM, Jacques Le Roux
>> <[hidden email]> wrote:
>>> Le 07/01/2018 à 15:44, Taher Alkhateeb a écrit :
>>>> I am suspecting this is still a broken build system rather than a
>>>> genuine error. Can you confirm?
>>> It's not a "broken build system" but was a transient error as you can see
>>> there
>>> https://ci.apache.org/builders/ofbizTrunkFrameworkPlugins
>>>
>>>> If tests are running okay locally
>>> Yes they are, at least now...
>>>
>>>> then we still have a faulty build logic.
>>> Nope, it's not a "faulty build logic." just a transient error on build
>>> servers
>>>
>>> When you see that tests pass locally and not on Buildbot it's most certainly
>>> an issue with servers.
>>> Those are hard workers and make errors from time to time (which shows that
>>> not only human make errors)
>>>
>>> When something like that happens, I get to IRC (using
>>> https://webchat.freenode.net)
>>> There using a recognisable username (I use jleroux) I get to the ofbiz
>>> channel.
>>> I wait for ofbiz-bot to appear, click on query and then I can make a request
>>> in the chat line to restart a scheduler.
>>> A request is of the form
>>> "force build ofbizTrunkFrameworkPlugins"
>>> you put the text you want to appear after this expression, something like
>>> "forces manual build after Buildbot weird 85 errors on labels :-o"
>>>
>>> I hope this clarify. I'll put a .md file in tools\documentation\BuildBot and
>>> will import is as HTML in a new wiki page for BuildBot
>>>
>>> I hope this will help us to get to a fruitful collaboration :)
>>>
>>> BTW we are still waiting for something to happen, see
>>> https://issues.apache.org/jira/browse/INFRA-15785
>>>
>>> Jacques
>>>
>>>
>>>> On Sun, Jan 7, 2018 at 5:20 PM, <[hidden email]> wrote:
>>>>> The Buildbot has detected a new failure on builder
>>>>> ofbizTrunkFrameworkPlugins while building . Full details are available at:
>>>>> https://ci.apache.org/builders/ofbizTrunkFrameworkPlugins/builds/10
>>>>>
>>>>> Buildbot URL: https://ci.apache.org/
>>>>>
>>>>> Buildslave for this Build: lares_ubuntu
>>>>>
>>>>> Build Reason: downstream
>>>>> Build Source Stamp: [branch ofbiz/ofbiz-framework/trunk] 1820457
>>>>> Blamelist: jleroux
>>>>>
>>>>> BUILD FAILED: failed shell_4
>>>>>
>>>>> Sincerely,
>>>>>    -The Buildbot
>>>>>
>>>>>
>>>>>
>
>

[taher]

So it seems like we have faulty buildbot scripts again. I feel inclined to
rewrite them from scratch. Anyone interested in helping out?

On Jan 11, 2018 3:16 PM, <[hidden email]> wrote:

> The Buildbot has detected a new failure on builder
> ofbizTrunkFrameworkPlugins while building . Full details are available at:
>     https://ci.apache.org/builders/ofbizTrunkFrameworkPlugins/builds/16
>
> Buildbot URL: https://ci.apache.org/
>
> Buildslave for this Build: orcus_ubuntu
>
> Build Reason: downstream
> Build Source Stamp: [branch ofbiz/ofbiz-framework/trunk] 1820862
> Blamelist: jleroux
>
> BUILD FAILED: failed shell_4
>
> Sincerely,
>  -The Buildbot
>
>
>
>

[Jacques Le Roux]

Please follow, this is related to http://markmail.org/message/vltmfn6x7wsfpss2

Jacques


Le 11/01/2018 à 13:18, Taher Alkhateeb a écrit :

> So it seems like we have faulty buildbot scripts again. I feel inclined to
> rewrite them from scratch. Anyone interested in helping out?
>
> On Jan 11, 2018 3:16 PM, <[hidden email]> wrote:
>
>> The Buildbot has detected a new failure on builder
>> ofbizTrunkFrameworkPlugins while building . Full details are available at:
>>      https://ci.apache.org/builders/ofbizTrunkFrameworkPlugins/builds/16
>>
>> Buildbot URL: https://ci.apache.org/
>>
>> Buildslave for this Build: orcus_ubuntu
>>
>> Build Reason: downstream
>> Build Source Stamp: [branch ofbiz/ofbiz-framework/trunk] 1820862
>> Blamelist: jleroux
>>
>> BUILD FAILED: failed shell_4
>>
>> Sincerely,
>>   -The Buildbot
>>
>>
>>
>>