ppc64 clang LNT buildbot / reference_output / test failures

Renato Golin renato.golin at linaro.org
Fri Apr 26 06:30:24 PDT 2013


On 26 April 2013 00:35, Hal Finkel <hfinkel at anl.gov> wrote:

> Attached. Thanks!
>

The tar ball attached doesn't contain the original bug (alti.stepfft), and
contains lots of directories in which I could look. Can you send me just
the ones that are failing (just the testname.out-simple file)?


When running without USE_REFERENCE_OUTPUT=1 (so that it compares to the
> system gcc), then all tests pass.
>

This means that the problem is probably not in the compiler, but in the
test, which is good news. But it also means it could take longer for you to
reduce a case or find where the problem is.

I know that running against GCC is very tempting, after all we all want the
bots green! But this is a false sense of security. If there is a bug in the
libraries, or if both GCC and LLVM have similar bugs, it'll go undetected.
It's not that difficult to get similar bugs, since we're all running
similar benchmarks and we're all compiler engineers, so we'll probably find
similar ways to implement a feature.

Also, people are free to "look up" GCC in order to understand how a
particular target-specific feature is implemented, and do it on the same
grounds. These problems are more common that we'd like to believe.
Furthermore, if a bug is detected in GCC, but not fixed, it'll take a long
time for us to detect as well, since there aren't many people monitoring
and syncing both Bugzillas... ;)

Another way to fix the test is to produce output that is deterministic.
I've fixed several tests by sorting tables before output.

All in all, I tend to agree with Daniel, and unless I see a case where it
really is impossible to have a reference output, I'd rather keep them.

cheers,
--renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20130426/403b6a5f/attachment.html>


More information about the llvm-commits mailing list