[llvm-dev] Performance Testing BasicAA and CFL AA

[Brian Homerding via llvm-dev]

Hello llvm-dev,

We recently collected some data on the test-suite using cfl=steens AA
compared to BasicAA and wanted to report the results.  There is one failing
test (SingleSource/Benchmarks/Shootout/Shootout-lists.exec) which we are
looking into.  All unlisted tests show no statistical difference.  There
are additional details about the testing methodology at the bottom of the
email.

- Brian Homerding

The format of the results is: (negative means BasicAA is better)
*Test*
Absolute Difference
Percentage Difference

**************** Execution Differences ****************
*SingleSource/Benchmarks/Shootout/Shootout-matrix.exe*
0.104083 +/- 0.0617407
7.14886% +/- 4.2406%

*SingleSource/Benchmarks/CoyoteBench/huffbench.exec*
0.408713 +/- 0.174997
3.21233% +/- 1.37541%

**************** Compilation Differences ****************

*MultiSource/Benchmarks/DOE-ProxyApps-C/SimpleMOC/SimpleMOC.compile*

-0.176327 +/- 0.0750171

-6.18574% +/- 2.63169%


*MultiSource/Benchmarks/tramp3d-v4/tramp3d-v4.compile*

-0.599067 +/- 0.49799

-1.23398% +/- 1.02577%


*MultiSource/Benchmarks/nbench/nbench.compile*

0.31194 +/- 0.137726

7.0723% +/- 3.12252%


*MultiSource/Benchmarks/MallocBench/gs/gs.compile*

-0.0901467 +/- 0.0737197

-0.85973% +/- 0.703066%

*MultiSource/Benchmarks/MiBench/consumer-typeset/consumer-typeset.compile*
-0.542867 +/- 0.511135
-1.72053% +/- 1.61996%

**************** Link Differences ****************
*MultiSource/Benchmarks/TSVC/ControlFlow-dbl/ControlFlow-dbl-link.compile*
-0.00284333 +/- 0.0026716
-9.62319% +/- 9.04198%

*SingleSource/Benchmarks/Shootout-C++/**Shootout-C++-ary3-link.compile*
-0.0039 +/- 0.00380256
-8.40819% +/- 8.19811%

*MultiSource/Benchmarks/SciMark2-C/s**cimark2-link.compile*
-0.00320667 +/- 0.00242367
-12.8335% +/- 9.69987%

*MultiSource/Benchmarks/tramp3d-v4/t**ramp3d-v4-link.compile*
-0.00392 +/- 0.0036359
-4.30517% +/- 3.99315%

*MultiSource/Benchmarks/TSVC/NodeSplitting-dbl/*
*NodeSplitting-dbl-link.compile*
-0.00362667 +/- 0.00287988
-12.9957% +/- 10.3197%


** Notes on the testing methodology **
We collected 30 compilation and execution samples on an Intel(R) Xeon(R)
Platinum 8180M CPU @ 2.50GHz Skylake machine using revision 348151.  Beyond
the "-use-cfl-aa=steens" flag we also used "-march=native -O3".  The
results were compared using a python version of the ministat tool.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181219/bd2fa874/attachment.html>