[llvm-dev] Buildbots do not detect test-suite failures!!!

[David Greene via llvm-dev]

Speaking of LNT, we've had a lot of trouble setting it up locally.  We
can get it to run basic pass/fail tests all right, but any performance
tracking is pretty hopeless.  From talking with others, their
experiences seem to have been the same.

It would be nice to have a non-bespoke system to test larger
applications and track performance for them.  Has anyone looked into
options?  Would such an effort be welcome, or are people happy with LNT?

                        -David

Nemanja Ivanovic via llvm-dev <llvm-dev at lists.llvm.org> writes:

> I have sent this a bit over a month ago and got no response. Does anyone know if this issue has been resolved? I hope that we are no longer in a situation where
> we may have LNT failures on build bots that aren't reported as failures.
>
> On Thu, Feb 14, 2019 at 6:30 AM Nemanja Ivanovic <nemanja.i.ibm at gmail.com> wrote:
>
>  It has just been brought to my attention that our buildbots no longer report failures to compile/link files in the test-suite as failures for the build. As a result, we
>  have at least one bot that is currently showing a green status, yet it aborts when building 3 test cases.
>
>  I assume that something has changed with how (presumably LNT) reports these failures and the Zorg code does not adequately detect this (I think the
>  buildbot scripts grep for a string such as "FAIL:" in the log or something along those lines). Previously, a failure to compile would show up as two failures (a
>  failure at compile time and a failure at link time). Now it just shows up as NOEXE.
>
>  This really seems like something we should fix ASAP.
>
>  For an example:
>  http://lab.llvm.org:8011/builders/clang-s390x-linux-lnt/builds/11252/steps/test-suite/logs/stdio
>
>  The Vector-build test case shows up as NOEXE and we're clearly hitting an UNREACHABLE while compiling the test:
> Unexpected node to lower
> UNREACHABLE executed at ...
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev