[LLVMdev] Running gcc tests using Clang

[Jim Grosbach]

All,

Here's a bit of old shell script I've used for comparing test results from various GCC testsuite runs. It's nothing fancy, but does cut down a bit of the noise and makes it a bit more convenient. Just point it at two directories containing the *.sum result files. YMMV and all that.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: compare_test_results.sh
Type: application/octet-stream
Size: 1475 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20101004/f9a250ab/attachment.obj>
-------------- next part --------------


-Jim


On Oct 2, 2010, at 9:18 AM, Daniel Dunbar wrote:

> Hi John,
> 
> On Wed, Sep 22, 2010 at 11:32 AM, John Thompson
> <john.thompson.jtsoftware at gmail.com> wrote:
>> I've run the tests from clang-tests/trunk/gcc-4_2-testsuite on a Ubuntu
>> x86-64 Linux box with the following results:
>> 
>>         === gcc Summary ===
>> 
>> # of expected passes        29946
>> # of unexpected failures    9938
>> # of unexpected successes    29
>> # of expected failures        28
>> # of unresolved testcases    1451
>> # of untested testcases        273
>> # of unsupported tests        811
>> pid is 4456 -4456
>> output is clang version 2.9 (trunk 113841)
>> Target: x86_64-unknown-linux-gnu
>> Thread model: posix
>>  status 0
>> /home/john/llvm/Release+Asserts/bin/clang  version 2.9 (trunk 113841)
>> 
>> This seems to be an improvement over the results I had previously when using
>> the i386 configuration.  However, it sounded like you had more success on a
>> Darwin box, with only 1 unexpected failure (mine, which should be fixed
>> now).  Should I expect that with this configuration?
> 
> No, that is unfortunately not the case.
> 
> What I do for testing is keep arounds results (see the 'make
> save-results' utility target) and compare the diffs from run to run.
> This is a real pain, but the best tool we have at the moment.
> 
> - Daniel
> 
>> I found that I had to exit most other running applications to be able to run
>> the tests without running out of memory.
>> 
>> I've enclosed my hacked makefile, hacked to change the configuration and add
>> the -v (verbose) flag.
>> 
>> I have a 6 mb log file from this run if needed.
>> 
>> ~/gcc-4_2-testsuite$ find . -print | wc
>>   36392   36392 1687901
>> 
>> Some output from the first few failed tests:
>> 
>> Testing gcc.apple/5490617.c
>> doing compile
>> pid is 17502 -17502
>> output is  status 0
>> FAIL: gcc.apple/5490617.c scan-assembler .lcomm __my_endbss
>> 
>> Testing gcc.apple/4104248.c
>> doing compile
>> pid is 17445 -17445
>> close result is 17445 exp8 0 1
>> output is ilp3217400.c:2: error: array size is negative
>>  status 1
>> 
>> Testing gcc.apple/4641942.c
>> doing compile
>> pid is 17481 -17481
>> output is  status 0
>> FAIL: gcc.apple/4641942.c  (test for warnings, line 20)
>> 
>> Testing gcc.apple/5597292.c
>> doing compile
>> pid is 17507 -17507
>> close result is 17507 exp8 0 1
>> output is /home/john/gcc-4_2-testsuite/src/gcc.apple/5597292.c:7: error:
>> fields must have a constant size: 'variable length array in structure'
>> extension will never be supported
>>  status 1
>> compiler exited with status 1
>> FAIL: gcc.apple/5597292.c (test for excess errors)
>> 
>> Testing gcc.apple/6286881.c
>> doing compile
>> pid is 17572 -17572
>> close result is 17572 exp8 0 1
>> output is /home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: warning: ^
>> has lower precedence than !=; != will be evaluated first [-Wparentheses]
>> /home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: note: place
>> parentheses around the ^ expression to evaluate it first
>> /home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: note: place
>> parentheses around the != expression to silence this warning
>> /home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: error: address
>> expression must be an lvalue or a function designator
>>  status 1
>> compiler exited with status 1
>> FAIL: gcc.apple/6286881.c  (test for errors, line 8)
>> 
>> etc.
>> 
>> -John
>> 
>> --
>> John Thompson
>> John.Thompson.JTSoftware at gmail.com
>> 
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>> 
>> 
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev