[LLVMdev] Confusing buildbot failure in LLVM on sanitizer-x86_64-linux

[David Blaikie]

On Wed, May 13, 2015 at 10:39 AM, Reid Kleckner <rnk at google.com> wrote:

> It's a 20m timeout without output.
>
> If you back up to the build and look at the 'annotate' step output,
> there's this text:
>
> http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/17916/steps/annotate/logs/stdio
>
> -- Testing: 258 tests, 16 threads --
> Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
> command timed out: 1200 seconds without output, attempting to kill
> process killed by signal 9
> program finished with exit code -1
> elapsedTime=3507.624426
>
> The annotator should probably include that timeout text in the failing
> step, so that sounds like a bug.
>
> Another issue is that tsan times out sometimes.
>

Also - how often are the timeouts actually indicative of regressions.
Perhaps we could flag them as "exceptional" results, shown in purple (&
possibly not emailing anyone except the buildbot owner) - rather than red
failures somehow.


> Should we be sending tsan build failures to upstream developers? How often
> do they break tsan? I suspect that when LLVM breaks tsan, it also breaks
> ASan, which isn't as flaky. It might be better to mail the tsan failures to
> Dmitry or someone and not upstream LLVM devs.
>
> On Wed, May 13, 2015 at 9:59 AM, Diego Novillo <dnovillo at google.com>
> wrote:
>
>> Alexey, I got mail from one of the tsan buildbots, claiming a breakage
>> in tsan tests. But I cannot see anything on the logs it has for the
>> build.
>>
>>
>> http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/17916/steps/run%2064-bit%20tsan%20unit%20tests/logs/stdio
>>
>> Any ideas?  Thanks. Diego.
>>
>>
>> ---------- Forwarded message ----------
>> From:  <llvm.buildmaster at lab.llvm.org>
>> Date: Wed, May 13, 2015 at 12:53 PM
>> Subject: buildbot failure in LLVM on sanitizer-x86_64-linux
>> To: Brendon Cahoon <bcahoon at codeaurora.org>, Diego Novillo
>> <dnovillo at google.com>, Teresa Johnson <tejohnson at google.com>, Yaron
>> Keren <yaron.keren at gmail.com>
>> Cc: gkistanova at gmail.com
>>
>>
>> The Buildbot has detected a new failure on builder
>> sanitizer-x86_64-linux while building llvm.
>> Full details are available at:
>>  http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/17916
>>
>> Buildbot URL: http://lab.llvm.org:8011/
>>
>> Buildslave for this Build: sanitizer-buildbot1
>>
>> Build Reason: scheduler
>> Build Source Stamp: [branch trunk] 237261
>> Blamelist: bcahoon,dnovillo,tejohnson,yrnkrn
>>
>> BUILD FAILED: failed annotate failed run 64-bit tsan unit tests
>>
>> sincerely,
>>  -The Buildbot
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150513/b9023ebc/attachment.html>