ppc64 clang LNT buildbot / reference_output / test failures

[Hal Finkel]

----- Original Message -----
> From: "Renato Golin" <renato.golin at linaro.org>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "Daniel Dunbar" <daniel at zuster.org>, "Commit Messages and Patches for LLVM" <llvm-commits at cs.uiuc.edu>
> Sent: Friday, April 26, 2013 8:30:24 AM
> Subject: Re: ppc64 clang LNT buildbot / reference_output / test failures
> 
> 
> On 26 April 2013 00:35, Hal Finkel < hfinkel at anl.gov > wrote:
> 
> 
> 
> 
> 
> Attached. Thanks!
> 
> 
> The tar ball attached doesn't contain the original bug
> (alti.stepfft), and contains lots of directories in which I could
> look. Can you send me just the ones that are failing (just the
> testname.out-simple file)?

What do you mean? These are the failing cases, and I've included the 'native' and simple outputs (in this case, 'native' is just a copy of the reference output). 

To be clear, after I ran the test suite, I did this:
for f in $(grep '\*' report.simple.txt | awk '{ print $1}'); do ls $(dirname $f)/Output/$(basename $f).out-{nat,simple}; done > ~/tmp/ts_output_files.txt

and then:
cat ~/tmp/ts_output_files.txt | xargs tar -czvf ~/tmp/ts_output_files.tar.gz

Are you looking for something else?

> 
> 
> 
> 
> 
> 
> When running without USE_REFERENCE_OUTPUT=1 (so that it compares to
> the system gcc), then all tests pass.
> 
> 
> This means that the problem is probably not in the compiler, but in
> the test, which is good news. But it also means it could take longer
> for you to reduce a case or find where the problem is.
> 
> 
> I know that running against GCC is very tempting, after all we all
> want the bots green! But this is a false sense of security. If there
> is a bug in the libraries, or if both GCC and LLVM have similar
> bugs, it'll go undetected. It's not that difficult to get similar
> bugs, since we're all running similar benchmarks and we're all
> compiler engineers, so we'll probably find similar ways to implement
> a feature.
> 
> 
> Also, people are free to "look up" GCC in order to understand how a
> particular target-specific feature is implemented, and do it on the
> same grounds. These problems are more common that we'd like to
> believe. Furthermore, if a bug is detected in GCC, but not fixed,
> it'll take a long time for us to detect as well, since there aren't
> many people monitoring and syncing both Bugzillas... ;)
> 
> 
> Another way to fix the test is to produce output that is
> deterministic. I've fixed several tests by sorting tables before
> output.
> 
> 

I agree that we should have good reference outputs, I'm arguing only for a short-term solution while we audit the test cases.

Thanks again,
Hal

> 
> All in all, I tend to agree with Daniel, and unless I see a case
> where it really is impossible to have a reference output, I'd rather
> keep them.
> 
> 
> cheers,
> --renato