[LLVMdev] LNT BenchmarkGame

[Daniel Dunbar]

On Tue, Mar 12, 2013 at 7:24 AM, Tim Northover <t.p.northover at gmail.com>wrote:

> > Is there any chance that the results are not being checked correctly? Any
> > other ideas?
>
> I think I vaguely convinced myself that the infrastructure didn't
> actually check whether tests it classified as benchmarks passed or
> failed. Not sure I had any good evidence for it other than things like
> you're seeing.
>

This is false.

Every test gets compared against some kind of expected output file (which
includes the exit code). The correct output is either:
 a. a reference output file
or
 b. the output from a natively run executable
depending on some of the test parameters.

 - Daniel


>
> > I'm tempted to just change the reference output and see what
> > happens with the other bots...
>
> Could be worth a try. But if that thing really is generating random
> numbers I'm not sure replacing one genuine cast-iron random number
> with another is the best solution long-term.
>
> Tim.
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130312/adf91ffd/attachment.html>