[LLVMdev] polyhedron 2005 results for llvm svn
Tanya M. Lattner
tonic at nondot.org
Fri Oct 31 09:47:39 PDT 2008
The rules for each release are mentioned here:
http://llvm.org/docs/HowToReleaseLLVM.html
I'm going to expand the first two sections and move them to another
document since no one reads this one. Its is missing details on targets
and languages we use as criteria, which I will update.
However, the release process is documented.
I am not saying that building gfortran is not important but you do realize
that we have not included this as a criteria for releases _ever_ and
secondly we branched on Oct. 6th and prerelease1 testing closed on Oct.
19th. In order for us to get a release out in a timely manner, we have
strict rules on what gets merged in and what the cutoff point is. For
every patch that gets merged in, I have to do a full round of all the
testing for every target we are supporting. It takes a lot of time.
Lastly, we are not the FSF. We have our own rules for releases and I
actually think we are even more strict when it comes to regressions. We
try to release a high quality product in a timely manner. Its a difficult
thing to balance.
I'm sorry you are unhappy with this decision.
-Tanya
On Fri, 31 Oct 2008, Jack Howarth wrote:
> I am finding with the patch that all of the Polyhedron 2005
> benchmarks pass on i686-apple-darwin9. Could someone clarify the
> regression rules for releases? Not building a secondary language
> on a primary target is usually considered a P1 regression for
> FSF gcc. Not doing so here gives one the impression that llvm.org
> isn't playing by the same rules. No one is ever going to want to
> use these releases if they don't follow similar release guidelines
> for maintaining usability between releases.
> Jack
> ps I did a test run of make check-gfortran for the build but it
> gets a lot of false positives on failures due to the bogus flags
> on from darwin.h.
>
> ================================================================================
> Date & Time : 31 Oct 2008 0:08:43
> Test Name : llvm_gfortran_lin_p4
> Compile Command : llvm-gfortran -ffast-math -funroll-loops -msse3 -O3 %n.f90 -o %n
> Benchmarks : ac aermod air capacita channel doduc fatigue gas_dyn induct linpk mdbx nf protein rnflow test_fpu tfft
> Maximum Times : 2000.0
> Target Error % : 0.100
> Minimum Repeats : 10
> Maximum Repeats : 100
>
> Benchmark Compile Executable Ave Run Number Estim
> Name (secs) (bytes) (secs) Repeats Err %
> --------- ------- ---------- ------- ------- ------
> ac 3.68 10000 19.33 21 0.0819
> aermod 33.81 10000 22.02 25 0.0841
> air 5.53 10000 14.30 100 0.1575
> capacita 4.40 10000 63.65 14 0.0896
> channel 0.82 10000 4.68 12 0.0350
> doduc 2.82 10000 50.60 14 0.0857
> fatigue 4.70 10000 17.14 12 0.0816
> gas_dyn 3.76 10000 29.24 37 0.0909
> induct 6.86 10000 68.63 10 0.0870
> linpk 3.01 10000 30.77 28 0.0879
> mdbx 4.30 10000 18.30 23 0.0993
> nf 3.01 10000 35.01 19 0.0329
> protein 1.39 10000 55.97 10 0.0415
> rnflow 1.92 10000 47.68 10 0.0257
> test_fpu 1.60 10000 16.57 10 0.0571
> tfft 0.60 10000 2.87 14 0.0849
>
> Geometric Mean Execution Time = 23.37 seconds
>
> ================================================================================
> Polyhedron Benchmark Validator
> Copyright (C) Polyhedron Software Ltd - 2004 - All rights reserved
>
> Test File : llvm_gfortran_lin_p4.run
> Instruction File : ../../f90valid.in
>
>> Value= 14159 Target= 14159 Tolerance= 0
>> Value= 16 Target= 16 Tolerance= 0
>> Value= 1000000 Target= 1000000 Tolerance= 0
>> Value=0.44068679351 Target=0.44068679400 Tolerance=0.10000000000E-06
>> Value= 10000 Target= 10000 Tolerance= 0
>> Value= 1.4530259062 Target= 1.4530259100 Tolerance=0.10000000000E-02
>> Value=0.60312607314E-07 Target=0.60312607300E-07 Tolerance=0.10000000000E-06
>> Value=0.54549700079 Target=0.54549700100 Tolerance=0.30000000000E-03
>> Value=0.25209045155E-04 Target=0.25209045200E-04 Tolerance=0.10000000000E-06
>> Value= 139.68826600 Target= 139.68826600 Tolerance=0.10000000000
>> Value=0.15138495928E-01 Target=0.15138495900E-01 Tolerance=0.20000000000E-04
>
> ac PASSED 11 tests - no failures
>
>> Value= 2191.1142600 Target= 2191.1145000 Tolerance=0.10000000000E-02
>> Value= 34310.414060 Target= 34310.421880 Tolerance=0.30000000000E-01
>> Value= 4260.9589800 Target= 4260.9716800 Tolerance=0.50000000000E-01
>> Value= 37094.359380 Target= 37094.375000 Tolerance=0.20000000000E-01
>> Value= 7924.8818400 Target= 7924.8842800 Tolerance=0.30000000000E-01
>> Value= 37094.359380 Target= 37094.375000 Tolerance= 1.0000000000
>
> aermod PASSED 6 tests - no failures
>
> = Value=1737 Target=1737
>> Value=0.99606856460E-02 Target=0.99606856460E-02 Tolerance=0.10000000000E-06
>> Value=0.10000000000E-01 Target=0.10000000000E-01 Tolerance=0.10000000000E-06
>> Value=0.41528600000 Target=0.41528600000 Tolerance=0.10000000000E-06
>> Value=0.57795248195E-05 Target=0.57795248200E-05 Tolerance=0.10000000000E-06
>> Value=0.10001137147E-01 Target=0.10001137100E-01 Tolerance=0.10000000000E-06
> = Value=1744 Target=1744
>
> air PASSED 7 tests - no failures
>
>> Value=0.25749207000E-04 Target= 0.0000000000 Tolerance=0.60000000000E-04
>> Value= 35.406150000 Target= 35.408000000 Tolerance=0.20000000000E-02
>> Value=0.16246922000E-04 Target= 0.0000000000 Tolerance=0.60000000000E-04
>> Value= 5.8967730000 Target= 5.8980000000 Tolerance=0.20000000000E-02
>
> capacita PASSED 4 tests - no failures
>
>> Value=0.42933170509E-01 Target=0.42933170400E-01 Tolerance=0.10000000000E-08
>> Value= 667696.66776 Target= 667696.66000 Tolerance=0.10000000000E-01
>> Value= 9.7805700000 Target= 9.7805700000 Tolerance=0.10000000000E-04
>> Value= 6.4000000000 Target= 6.4000000000 Tolerance=0.10000000000
>> Value= 6.9000000000 Target= 6.9000000000 Tolerance=0.10000000000
>> Value= 7.4000000000 Target= 7.4000000000 Tolerance=0.10000000000
>> Value= 7.8000000000 Target= 7.8000000000 Tolerance=0.10000000000
>> Value= 8.2000000000 Target= 8.2000000000 Tolerance=0.10000000000
>> Value= 8.5000000000 Target= 8.5000000000 Tolerance=0.10000000000
>> Value= 8.7000000000 Target= 8.7000000000 Tolerance=0.10000000000
>> Value= 8.6000000000 Target= 8.6000000000 Tolerance=0.10000000000
>> Value= 8.5000000000 Target= 8.5000000000 Tolerance=0.10000000000
>> Value= 7.9000000000 Target= 7.9000000000 Tolerance=0.10000000000
>> Value= 7.4000000000 Target= 7.4000000000 Tolerance=0.10000000000
>> Value= 6.1000000000 Target= 6.1000000000 Tolerance=0.10000000000
>> Value= 5.0000000000 Target= 5.0000000000 Tolerance=0.10000000000
>> Value= 2.9000000000 Target= 2.9000000000 Tolerance=0.10000000000
>> Value=0.60000000000 Target=0.60000000000 Tolerance=0.10000000000
>
> channel PASSED 18 tests - no failures
>
>> Value= 2000.0018055 Target= 2000.0024122 Tolerance=0.10000000000E-01
>> Value= 533028 Target= 533050 Tolerance= 5000
>
> doduc PASSED 2 tests - no failures
>
> = Value=10 Target=10
>> Value=0.60325000000E-04 Target=0.60325000000E-04 Tolerance=0.10000000000E-08
> = Value=9 Target=9
>> Value=0.53975000000E-04 Target=0.53975000000E-04 Tolerance=0.10000000000E-08
> = Value=8 Target=8
>> Value=0.47625000000E-04 Target=0.47625000000E-04 Tolerance=0.10000000000E-08
>> Value= 17.558100000 Target= 17.558100000 Tolerance=0.10000000000E-03
>> Value=0.53463000000 Target=0.53463000000 Tolerance=0.10000000000E-05
>
> fatigue PASSED 8 tests - no failures
>
>> Value=0.86774900000E-06 Target=0.86774900000E-06 Tolerance=0.10000000000E-10
>
> gas_dyn PASSED 1 tests - no failures
>
> = Value=10 Target=10
>> Value=0.13333333333E-04 Target=0.13333333333E-04 Tolerance=0.10000000000E-11
>> Value=0.33333333333E-03 Target=0.33333333333E-03 Tolerance=0.10000000000E-10
>> Value=0.22940707222E-10 Target=0.22940707222E-10 Tolerance=0.10000000000E-16
>> Value=0.57351768056E-09 Target=0.57351768056E-09 Tolerance=0.10000000000E-16
>> Value=0.10000000000E+21 Target=0.10000000000E+21 Tolerance= 1000.0000000
>> Value=-.10000000000E+21 Target=-.10000000000E+21 Tolerance= 1000.0000000
>> Value=0.10000000000E+21 Target=0.10000000000E+21 Tolerance= 1000.0000000
>> Value=-.10000000000E+21 Target=-.10000000000E+21 Tolerance= 1000.0000000
>> Value= 0.0000000000 Target= 0.0000000000 Tolerance=0.10000000000E-11
>> Value=0.59537942844E-01 Target=0.59537942844E-01 Tolerance=0.10000000000E-09
>> Value= 0.0000000000 Target= 0.0000000000 Tolerance=0.10000000000E-11
>> Value=0.96399525024E-01 Target=0.96399525024E-01 Tolerance=0.10000000000E-09
> = Value=120000 Target=120000
> = Value=0 Target=0
> = Value=0 Target=0
> = Value=360000 Target=360000
> = Value=241200 Target=241200
> = Value=120000 Target=120000
> = Value=80400 Target=80400
>
> induct PASSED 20 tests - no failures
>
>> Value= 25.095297500 Target= 23.100000000 Tolerance= 2.0000000000
>> Value=0.27858826400E-10 Target=0.27858826400E-10 Tolerance=0.10000000000E-09
>> Value=0.22204460500E-15 Target=0.22204460500E-15 Tolerance=0.10000000000E-14
>> Value= 1.0000000000 Target= 1.0000000000 Tolerance=0.10000000000E-07
>> Value= 1.0000000000 Target= 1.0000000000 Tolerance=0.10000000000E-07
>
> linpk PASSED 5 tests - no failures
>
> = Value=1000 Target=1000
>> Value=0.27060000000 Target=0.27000000000 Tolerance=0.10000000000E-01
>> Value=-3.3184000000 Target=-3.3200000000 Tolerance=0.20000000000E-01
>> Value=-3.0478000000 Target=-3.0450000000 Tolerance=0.50000000000E-02
>> Value= 16.930600000 Target= 17.100000000 Tolerance= 1.0000000000
>> Value=0.80000000000E-13 Target= 0.0000000000 Tolerance=0.10000000000E-11
>> Value=0.10000000000E-12 Target= 0.0000000000 Tolerance=0.10000000000E-11
>> Value=-.40000000000E-15 Target= 0.0000000000 Tolerance=0.10000000000E-11
> = Value=100 Target=100
>> Value=0.29620000000 Target=0.29620000000 Tolerance=0.20000000000E-03
>> Value=-3.3440000000 Target=-3.3440000000 Tolerance=0.20000000000E-03
>> Value=-3.0478000000 Target=-3.0478000000 Tolerance=0.50000000000E-03
>> Value= 1.0012000000 Target= 1.0012000000 Tolerance=0.20000000000E-03
>> Value=0.90000000000E-14 Target= 0.0000000000 Tolerance=0.10000000000E-11
>> Value=0.90000000000E-14 Target= 0.0000000000 Tolerance=0.10000000000E-11
>> Value=0.10000000000E-13 Target= 0.0000000000 Tolerance=0.10000000000E-11
> = Value=100 Target=100
>> Value=0.29620000000 Target=0.29620000000 Tolerance=0.20000000000E-03
>> Value=-3.3440000000 Target=-3.3440000000 Tolerance=0.20000000000E-03
>> Value=-3.0478000000 Target=-3.0478000000 Tolerance=0.50000000000E-03
>> Value= 1.0012000000 Target= 1.0012000000 Tolerance=0.20000000000E-03
>> Value=0.50000000000E-13 Target= 0.0000000000 Tolerance=0.10000000000E-11
>> Value=0.20000000000E-13 Target= 0.0000000000 Tolerance=0.10000000000E-11
>> Value=-.30000000000E-14 Target= 0.0000000000 Tolerance=0.10000000000E-11
> = Value=1000 Target=1000
>> Value=0.27570000000 Target=0.27450000000 Tolerance=0.10000000000E-01
>> Value=-3.3233000000 Target=-3.3220000000 Tolerance=0.10000000000E-01
>> Value=-3.0476000000 Target=-3.0477000000 Tolerance=0.50000000000E-03
>> Value= 18.349300000 Target= 18.400000000 Tolerance=0.80000000000
>> Value=-.70000000000E-13 Target= 0.0000000000 Tolerance=0.10000000000E-11
>> Value=0.60000000000E-13 Target= 0.0000000000 Tolerance=0.10000000000E-11
>> Value=-.80000000000E-13 Target= 0.0000000000 Tolerance=0.10000000000E-11
> = Value=100 Target=100
>> Value=0.28970000000 Target=0.28970000000 Tolerance=0.50000000000E-03
>> Value=-3.3413000000 Target=-3.3413000000 Tolerance=0.10000000000E-02
>> Value=-3.0517000000 Target=-3.0517000000 Tolerance=0.50000000000E-03
>> Value= 1.1007000000 Target= 1.1007000000 Tolerance=0.10000000000E-02
>> Value=-.70000000000E-13 Target= 0.0000000000 Tolerance=0.10000000000E-11
>> Value=0.20000000000E-13 Target= 0.0000000000 Tolerance=0.10000000000E-11
>> Value=-.40000000000E-14 Target= 0.0000000000 Tolerance=0.10000000000E-11
>
> mdbx PASSED 40 tests - no failures
>
>> Value= 24 Target= 24 Tolerance= 0
>> Value=0.21401000000 Target=0.21400000000 Tolerance=0.10000000000E-02
>> Value=0.22295000000 Target=0.22295000000 Tolerance=0.10000000000E-03
>> Value=-.49013470000E-13 Target= 0.0000000000 Tolerance=0.20000000000E-12
>> Value= 38 Target= 37 Tolerance= 1
>> Value= 27 Target= 27 Tolerance= 1
>> Value= 44 Target= 44 Tolerance= 1
>
> nf PASSED 7 tests - no failures
>
>> Value= 5714.0000000 Target= 5714.0000000 Tolerance=0.10000000000
>> Value= 2214644.0000 Target= 2214644.0000 Tolerance=0.10000000000
>> Value= 22856.000000 Target= 22856.000000 Tolerance=0.10000000000
>> Value= 8858576.0000 Target= 8858576.0000 Tolerance=0.10000000000
>> Value= 22856.000000 Target= 22856.000000 Tolerance=0.10000000000
>> Value= 8858576.0000 Target= 8858576.0000 Tolerance=0.10000000000
>> Value= 22856.000000 Target= 22856.000000 Tolerance=0.10000000000
>> Value= 8858576.0000 Target= 8858576.0000 Tolerance=0.10000000000
>> Value= 22856.000000 Target= 22856.000000 Tolerance=0.10000000000
>> Value= 8858576.0000 Target= 8858576.0000 Tolerance=0.10000000000
>
> protein PASSED 10 tests - no failures
>
>> Value= 61 Target= 61 Tolerance= 0
> = Value=189 Target=189
> = Value=13230 Target=13230
> = Value=13363 Target=13363
>> Value= 1.0000000000 Target= 1.0000000000 Tolerance=0.10000000000E-07
>> Value= 85 Target= 85 Tolerance= 0
> = Value=101 Target=101
>> Value= 357316 Target= 357316 Tolerance= 1
>> Value= 356376 Target= 356376 Tolerance= 1
>> Value=0.26000000000 Target=0.26000000000 Tolerance=0.10000000000E-07
>> Value= 109 Target= 109 Tolerance= 0
> = Value=125 Target=125
>> Value= 1179816 Target= 1179816 Tolerance= 1
>> Value= 1180126 Target= 1180126 Tolerance= 1
>> Value=0.29999999000E-01 Target=0.30000000000E-01 Tolerance=0.10000000000E-07
>> Value= 205 Target= 205 Tolerance= 0
> = Value=213 Target=213
> = Value=3278 Target=3278
> = Value=3208 Target=3208
>> Value= 2.1400000000 Target= 2.1400000000 Tolerance=0.10000000000E-05
>
> rnflow PASSED 20 tests - no failures
>
>> Value=0.60000000000E-14 Target= 0.0000000000 Tolerance=0.10000000000E-10
>> Value=0.23000000000E-13 Target= 0.0000000000 Tolerance=0.10000000000E-10
>> Value=0.31000000000E-13 Target= 0.0000000000 Tolerance=0.10000000000E-10
>> Value=0.25000000000E-12 Target= 0.0000000000 Tolerance=0.10000000000E-10
>
> test_fpu PASSED 4 tests - no failures
>
>> Value=0.30791526000E-07 Target=0.30000000000E-07 Tolerance=0.30000000000E-07
>
> tfft PASSED 1 tests - no failures
>
> Finished Testing 16 benchmarks - 16 passed, and 0 failed
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
More information about the llvm-dev
mailing list