[LLVMdev] LNT BenchmarkGame
daniel at zuster.org
Tue Mar 12 09:45:23 PDT 2013
On Tue, Mar 12, 2013 at 7:24 AM, Tim Northover <t.p.northover at gmail.com>wrote:
> > Is there any chance that the results are not being checked correctly? Any
> > other ideas?
> I think I vaguely convinced myself that the infrastructure didn't
> actually check whether tests it classified as benchmarks passed or
> failed. Not sure I had any good evidence for it other than things like
> you're seeing.
This is false.
Every test gets compared against some kind of expected output file (which
includes the exit code). The correct output is either:
a. a reference output file
b. the output from a natively run executable
depending on some of the test parameters.
> > I'm tempted to just change the reference output and see what
> > happens with the other bots...
> Could be worth a try. But if that thing really is generating random
> numbers I'm not sure replacing one genuine cast-iron random number
> with another is the best solution long-term.
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the llvm-dev