[LLVMdev] FNT testers reporting success even though they failed

[David Blaikie]

On Tue, Dec 11, 2012 at 1:27 AM, Duncan Sands <baldrick at free.fr> wrote:
> All the LLVM lab FNT builders were reporting failures in the same set of
> tests
> (scimark2, LivermooreLoops, some others).  Now they've all turned green but
> the
> same tests are still failing.  I don't see any commits to deliberately make
> the
> FNT pass, so it looks like someone broke failure detection...  Note that the
> FNT "report" text became empty at the same time.  Here's what "report" was
> like
> before:
>
>
> http://lab.llvm.org:8011/builders/clang-x86_64-debian-fnt/builds/12575/steps/make.test-suite/logs/report
>
> and after:
>
> http://lab.llvm.org:8011/builders/clang-x86_64-debian-fnt/builds/12576/steps/make.test-suite/logs/report
>
> The first green builds are:
>   http://lab.llvm.org:8011/builders/clang-x86_64-debian-fnt/builds/12576
>   http://lab.llvm.org:8011/builders/clang-x86_64-linux-fnt/builds/57
>
> http://lab.llvm.org:8011/builders/dragonegg-x86_64-linux-gcc-4.6-fnt/builds/1983
>
> Does anyone know what might have caused this?

[Thought it might've been my change (r169769) but that doesn't look
like it intersects with this particular test task]

I think you're on the right track with the logs. If the log that
reaches the buildmaster is empty then we won't find any failures &
we'll report success.

Have you checked any of the slaves to see if the log files are empty
in the builds themselves?

Have you run a local execution of the test suite uisng roughly the
same commands & observed whether the reports are empty coming out of
that?

- David