[LLVMdev] LNT BenchmarkGame

[Renato Golin]

On 12 March 2013 14:36, Tim Northover <t.p.northover at gmail.com> wrote:

> If Clang and GCC disagree on the same source, same machine and with
> the same libraries, that certainly is odd.


They don't. That's the odd bit. GCC and Clang agree on the output on both
ARM and x86_64, and neither agree with the reference_output file.

What could be happening is that the version of the libraries on that
buildbot is old, and both ARM and x86_64 have been updated.

I'm not suggesting we should keep replacing the "golden" file for the new
value, but that we should disable checking the reference_output at all, and
rely on a GCC vs. Clang comparison.

I agree that the comparison is no better than a reference file (since it,
too, could be wrong), but comparing both outputs eliminate any library
mismatch and it's less likely that both GCC and Clang will be wrong about
exactly the same thing at the same time.

Is there a way to turn off the check against the reference_output and make
it check against a GCC executable output?

cheers,
--renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130312/33e6bab3/attachment.html>