[cfe-dev] 2.7 Pre-release1 available for testing

[Tanya Lattner]

On Mar 24, 2010, at 2:47 PM, Török Edwin wrote:

> On 03/17/2010 10:12 PM, Tanya Lattner wrote:
>> The 2.7 binaries are available for testing:
>> http://llvm.org/pre-releases/2.7/pre-release1/
>> 
>> You will also find the source tarballs there as well.
>> 
>> We rely on the community to help make our releases great, so please help
>> test 2.7 if you can. Please follow these instructions to test 2.7:
>> 
>> /To test llvm-gcc:/
>> 
>> 1) Compile llvm from source and untar the llvm-test in the projects  
>> directory (name it llvm-test or test-suite). Choose to use a pre- 
>> compiled llvm-gcc or re-compile it yourself.  
>> 
>> 2) Run make check, report any failures (FAIL or unexpected pass). Note  
>> that you need to reconfigure llvm with llvm-gcc in your path or with -- with-llvmgccdir 
>> 
>> 3) Run "make TEST=nightly report". Compare these results to a 2.6 llvm-test nightly report or send the results to the list. For supported targets, we'll try to examine the results, but its best if you can do the comparison yourself. 
>> 
> 
> Hi Tanya,
> 
> Attached are the nightly test results when run with llvm-gcc
> (report.nightly.txt), and when run with clang (clang-report.nightly.txt).
> 

Thanks for testing the release!

> Tests were run on x86-64, Debian unstable, Linux 2.6.33, gcc 4.4.3,
> 64-bit. I built srcdir == objdir, I have built llvm and clang myself,
> and used the binaries for llvm-gcc.
> 
> 1. llvm-gcc 2.7 vs 2.6
> compared to my results from Aug 31 2009, ignoring CBE failures:
> 
> new JIT failures:
> MultiSource/Applications/spiff/spiff
> SingleSource/Regression/C/2004-03-15-IndirectGoto
> 

Yes, I'm seeing the second regression on darwin too. Please file a bug for the other one if you havent already.

> 2. llvm-gcc 2.7 vs clang 2.7
> When comparing the 2.7 llvm-gcc and clang results I see these
> differences (is llvm-gcc considered baseline for clang?):
> ALL FAIL (pass in llvm-gcc):
> MultiSource/Benchmarks/PAQ8p/paq8p
> MultiSource/Benchmarks/tramp3d-v4/tramp3d-v4
> MultiSource/Benchmarks/Prolangs-C/archie-client/archie
> MultiSource/Benchmarks/Prolangs-C/cdecl/cdecl
> SingleSource/Benchmarks/Misc-C++/bigfib
> SingleSource/Regression/C++/EH/ConditionalExpr
> SingleSource/Regression/C++/EH/ctor_dtor_count-2
> SingleSource/Regression/C++/EH/function_try_block
> SingleSource/Regression/C++/EH/simple_throw
> SingleSource/UnitTests/2006-12-04-DynAllocAndRestore
> SingleSource/UnitTests/Vector/SSE/sse.expandfft
> SingleSource/UnitTests/Vector/SSE/sse.stepfft
> 
> JIT failures in clang, pass in llvm-gcc:
> MultiSource/Applications/sqlite3/sqlite3
> SingleSource/Regression/C++/ofstream_ctor
> 

This isn't part of our release criteria. So, these are not release blockers.

> 3. Some performance regressions GCC/LLC  (2.6 -> 2.7), but keep in mind
> that I wasn't using GCC 4.4.3 as comparison for llvm 2.6!
> 
> MultiSource/Applications/hexxagon/hexxagon  1.22 -> 1.14
> MultiSource/Applications/lua/lua  0.91 -> 0.84
> MultiSource/Applications/obsequi/Obsequi  0.93 -> 0.86
> MultiSource/Benchmarks/ASC_Sequoia/CrystalMk/CrystalMk  1.01 -> 0.91
> MultiSource/Benchmarks/FreeBench/fourinarow/fourinarow     0.94 -> 0.75
> MultiSource/Benchmarks/FreeBench/neural/neural   1.0 -> 0.9
> MultiSource/Benchmarks/MiBench/telecomm-gsm/telecomm-gsm   1.06 -> 0.9
> MultiSource/Benchmarks/Olden/treeadd/treeadd  11.44 -> 9.89
> MultiSource/Benchmarks/Olden/tsp/tsp  1.14 -> 1.02
> MultiSource/Benchmarks/Ptrdist/anagram/anagram 1.33 -> 1.23
> SingleSource/Benchmarks/Dhrystone/dry  7.32 -> 5.16
> SingleSource/Benchmarks/Dhrystone/fldry   8.02 -> 6.65
> ....
> 

Unfortunately, we just don't have enough man power to have performance be a release criteria at this time. We also need a better infrastructure in place to track this stuff (Daniel is working on it).

> I'll have to write a script to compare the results, its boring and
> inaccurate to do by hand.
> 
> Will go through the bugzilla tomorrow and see if I need to open new bugs
> for this stuff.
> 
>> 
>>  /To test clang:/ 
>> 
>> 1) Compile llvm and clang from source. 
>> 
>> 2) Run make check for llvm. 
>> 
>> 3) Run make  -C tools/clang-2.6 test VERBOSE=1 (report any failures or  
>> unexpected passes) 
> 
> Surely you meant tools/clang-2.7
> 

Yes.

> FYI I pulled the following revisions for ClamAV's llvm on top of 2.7:
> r98349
> r98410
> r98447
> r98508
> r99143
> r99146
> r99147
> r99160
> r99400
> 
> I don't know if any of these qualify as regression fixes for 2.7, I'll
> leave it up to you to decide if you want to put them into 2.7 or not.
> 

I'll have to discuss with Chris about these. Its technically not a release blocker.

Thanks,
-Tanya


> Best regards,
> --Edwin
> <report.nightly.txt><clang-report.nightly.txt>