[LLVMdev] [RFC] Benchmarking subset of the test suite

[Hal Finkel]

At the LLVM Developers' Meeting in November, I promised to work on isolating a subset of the current test suite that is useful for benchmarking. Having looked at this in more detail, most of the applications and benchmarks in the test suite are useful for benchmarking, and so I think that a better way of phrasing it is that we should construct a list of programs in the test suite that are not useful for benchmarking.

My proposed exclusion list is provided below. I constructed this exclusion list primarily based on the following experiment: I ran the test suite 10 times in three configurations: 1) On an IBM POWER7 (P7) with -O3 -mvsx, 2) On a P7 at -O0 and 3) On an Intel Xeon E5430 with -O3 all using make -j6. I then used the ministat utility (which performs a T test) to compare the timings of the two P7 configurations against each other and the Xeon configuration, requiring a detectable difference at 99.5% confidence. I looked for tests that showed no significant difference in all three comparisons. The running configuration here is purposefully noisy, the idea is to eliminate those tests that are significantly sensitive to startup time, file I/O time, memory bandwidth, etc., or just too short, and by running many tests in parallel (non-deterministically), my hope is to eliminate those tests can cannot usefully serve as benchmarks in a "normal" environment.

I'll admit being somewhat surprised by so many of the Prolangs and Shootout "benchmarks" seemingly not serving as useful benchmarks; perhaps someone can look into improving the problem size, etc. of these.

Without further ado, I propose that a test-suite configuration designed for benchmarking exclude the following:

MultiSource/Applications/Burg/burg
MultiSource/Applications/treecc/treecc
MultiSource/Benchmarks/MiBench/office-ispell/office-ispell
MultiSource/Benchmarks/MiBench/security-blowfish/security-blowfish
MultiSource/Benchmarks/Prolangs-C/allroots/allroots
MultiSource/Benchmarks/Prolangs-C/archie-client/archie
MultiSource/Benchmarks/Prolangs-C/assembler/assembler
MultiSource/Benchmarks/Prolangs-C/compiler/compiler
MultiSource/Benchmarks/Prolangs-C++/deriv1/deriv1
MultiSource/Benchmarks/Prolangs-C++/deriv2/deriv2
MultiSource/Benchmarks/Prolangs-C++/family/family
MultiSource/Benchmarks/Prolangs-C/fixoutput/fixoutput
MultiSource/Benchmarks/Prolangs-C/football/football
MultiSource/Benchmarks/Prolangs-C++/fsm/fsm
MultiSource/Benchmarks/Prolangs-C++/garage/garage
MultiSource/Benchmarks/Prolangs-C++/NP/np
MultiSource/Benchmarks/Prolangs-C++/objects/objects
MultiSource/Benchmarks/Prolangs-C++/office/office
MultiSource/Benchmarks/Prolangs-C/plot2fig/plot2fig
MultiSource/Benchmarks/Prolangs-C++/shapes/shapes
MultiSource/Benchmarks/Prolangs-C/simulator/simulator
MultiSource/Benchmarks/Prolangs-C/TimberWolfMC/timberwolfmc
MultiSource/Benchmarks/Prolangs-C++/trees/trees
MultiSource/Benchmarks/Prolangs-C/unix-smail/unix-smail
MultiSource/Benchmarks/Prolangs-C/unix-tbl/unix-tbl
MultiSource/Benchmarks/Prolangs-C++/vcirc/vcirc
SingleSource/Benchmarks/McGill/exptree
SingleSource/Benchmarks/Shootout-C++/hello
SingleSource/Benchmarks/Shootout-C++/reversefile
SingleSource/Benchmarks/Shootout-C++/spellcheck
SingleSource/Benchmarks/Shootout-C++/sumcol
SingleSource/Benchmarks/Shootout-C++/wc
SingleSource/Benchmarks/Shootout-C++/wordfreq
SingleSource/Benchmarks/Shootout/hello
SingleSource/Regression/C++/2003-05-14-array-init
SingleSource/Regression/C++/2003-05-14-expr_stmt
SingleSource/Regression/C/2003-05-14-initialize-string
SingleSource/Regression/C/2003-05-21-BitfieldHandling
SingleSource/Regression/C/2003-05-21-UnionBitfields
SingleSource/Regression/C/2003-05-21-UnionTest
SingleSource/Regression/C/2003-05-22-LocalTypeTest
SingleSource/Regression/C/2003-05-22-VarSizeArray
SingleSource/Regression/C/2003-05-23-TransparentUnion
SingleSource/Regression/C++/2003-06-08-BaseType
SingleSource/Regression/C++/2003-06-08-VirtualFunctions
SingleSource/Regression/C++/2003-06-13-Crasher
SingleSource/Regression/C/2003-06-16-InvalidInitializer
SingleSource/Regression/C/2003-06-16-VolatileError
SingleSource/Regression/C++/2003-08-20-EnumSizeProblem
SingleSource/Regression/C++/2003-09-29-NonPODsByValue
SingleSource/Regression/C/2003-10-12-GlobalVarInitializers
SingleSource/Regression/C/2004-02-03-AggregateCopy
SingleSource/Regression/C/2004-03-15-IndirectGoto
SingleSource/Regression/C/2005-05-06-LongLongSignedShift
SingleSource/Regression/C/2008-01-07-LongDouble
SingleSource/Regression/C++/2008-01-29-ParamAliasesReturn
SingleSource/Regression/C++/2011-03-28-Bitfield
SingleSource/Regression/C/badidx
SingleSource/Regression/C/bigstack
SingleSource/Regression/C++/BuiltinTypeInfo
SingleSource/Regression/C/callargs
SingleSource/Regression/C/casts
SingleSource/Regression/C/compare
SingleSource/Regression/C/ConstructorDestructorAttributes
SingleSource/Regression/C/DuffsDevice
SingleSource/Regression/C++/EH/class_hierarchy
SingleSource/Regression/C++/EH/ConditionalExpr
SingleSource/Regression/C++/EH/ctor_dtor_count
SingleSource/Regression/C++/EH/ctor_dtor_count-2
SingleSource/Regression/C++/EH/dead_try_block
SingleSource/Regression/C++/EH/exception_spec_test
SingleSource/Regression/C++/EH/function_try_block
SingleSource/Regression/C++/EH/inlined_cleanup
SingleSource/Regression/C++/EH/recursive-throw
SingleSource/Regression/C++/EH/simple_rethrow
SingleSource/Regression/C++/EH/simple_throw
SingleSource/Regression/C++/EH/throw_rethrow_test
SingleSource/Regression/C++/fixups
SingleSource/Regression/C++/global_ctor
SingleSource/Regression/C/globalrefs
SingleSource/Regression/C++/global_type
SingleSource/Regression/C++/ofstream_ctor
SingleSource/Regression/C/pointer_arithmetic
SingleSource/Regression/C++/pointer_member
SingleSource/Regression/C++/pointer_method
SingleSource/Regression/C++/pointer_method2
SingleSource/Regression/C/PR10189
SingleSource/Regression/C/PR1386
SingleSource/Regression/C/PR491
SingleSource/Regression/C/PR640
SingleSource/Regression/C++/short_circuit_dtor
SingleSource/Regression/C/sumarray
SingleSource/Regression/C/sumarraymalloc
SingleSource/Regression/C/testtrace
SingleSource/UnitTests/2002-04-17-PrintfChar
SingleSource/UnitTests/2002-05-02-ArgumentTest
SingleSource/UnitTests/2002-05-02-CastTest
SingleSource/UnitTests/2002-05-02-CastTest1
SingleSource/UnitTests/2002-05-02-CastTest2
SingleSource/UnitTests/2002-05-02-CastTest3
SingleSource/UnitTests/2002-05-02-ManyArguments
SingleSource/UnitTests/2002-05-03-NotTest
SingleSource/UnitTests/2002-05-19-DivTest
SingleSource/UnitTests/2002-08-02-CastTest
SingleSource/UnitTests/2002-08-02-CastTest2
SingleSource/UnitTests/2002-08-19-CodegenBug
SingleSource/UnitTests/2002-10-09-ArrayResolution
SingleSource/UnitTests/2002-10-12-StructureArgs
SingleSource/UnitTests/2002-10-12-StructureArgsSimple
SingleSource/UnitTests/2002-10-13-BadLoad
SingleSource/UnitTests/2002-12-13-MishaTest
SingleSource/UnitTests/2003-04-22-Switch
SingleSource/UnitTests/2003-05-02-DependentPHI
SingleSource/UnitTests/2003-05-07-VarArgs
SingleSource/UnitTests/2003-05-12-MinIntProblem
SingleSource/UnitTests/2003-05-14-AtExit
SingleSource/UnitTests/2003-05-26-Shorts
SingleSource/UnitTests/2003-05-31-CastToBool
SingleSource/UnitTests/2003-05-31-LongShifts
SingleSource/UnitTests/2003-07-06-IntOverflow
SingleSource/UnitTests/2003-07-08-BitOpsTest
SingleSource/UnitTests/2003-07-09-LoadShorts
SingleSource/UnitTests/2003-07-09-SignedArgs
SingleSource/UnitTests/2003-07-10-SignConversions
SingleSource/UnitTests/2003-08-05-CastFPToUint
SingleSource/UnitTests/2003-08-11-VaListArg
SingleSource/UnitTests/2003-08-20-FoldBug
SingleSource/UnitTests/2003-09-18-BitFieldTest
SingleSource/UnitTests/2003-10-13-SwitchTest
SingleSource/UnitTests/2003-10-29-ScalarReplBug
SingleSource/UnitTests/2004-02-02-NegativeZero
SingleSource/UnitTests/2004-06-20-StaticBitfieldInit
SingleSource/UnitTests/2004-11-28-GlobalBoolLayout
SingleSource/UnitTests/2005-05-11-Popcount-ffs-fls
SingleSource/UnitTests/2005-05-12-Int64ToFP
SingleSource/UnitTests/2005-05-13-SDivTwo
SingleSource/UnitTests/2005-07-17-INT-To-FP
SingleSource/UnitTests/2005-11-29-LongSwitch
SingleSource/UnitTests/2006-01-29-SimpleIndirectCall
SingleSource/UnitTests/2006-02-04-DivRem
SingleSource/UnitTests/2006-12-01-float_varg
SingleSource/UnitTests/2006-12-04-DynAllocAndRestore
SingleSource/UnitTests/2006-12-07-Compare64BitConstant
SingleSource/UnitTests/2006-12-11-LoadConstants
SingleSource/UnitTests/2007-01-04-KNR-Args
SingleSource/UnitTests/2007-03-02-VaCopy
SingleSource/UnitTests/2007-04-25-weak
SingleSource/UnitTests/2008-04-18-LoopBug
SingleSource/UnitTests/2008-04-20-LoopBug2
SingleSource/UnitTests/2008-07-13-InlineSetjmp
SingleSource/UnitTests/2009-04-16-BitfieldInitialization
SingleSource/UnitTests/2009-12-07-StructReturn
SingleSource/UnitTests/2010-05-24-BitfieldTest
SingleSource/UnitTests/AtomicOps
SingleSource/UnitTests/block-byref-cxxobj-test
SingleSource/UnitTests/block-byref-test
SingleSource/UnitTests/block-call-r7674133
SingleSource/UnitTests/block-copied-in-cxxobj
SingleSource/UnitTests/block-copied-in-cxxobj-1
SingleSource/UnitTests/blockstret
SingleSource/UnitTests/byval-alignment
SingleSource/UnitTests/conditional-gnu-ext
SingleSource/UnitTests/conditional-gnu-ext-cxx
SingleSource/UnitTests/DefaultInitDynArrays
SingleSource/UnitTests/FloatPrecision
SingleSource/UnitTests/initp1
SingleSource/UnitTests/member-function-pointers
SingleSource/UnitTests/printargs
SingleSource/UnitTests/SignlessTypes/cast2
SingleSource/UnitTests/SignlessTypes/cast-bug
SingleSource/UnitTests/SignlessTypes/ccc
SingleSource/UnitTests/SignlessTypes/div
SingleSource/UnitTests/SignlessTypes/factor
SingleSource/UnitTests/SignlessTypes/Large/cast
SingleSource/UnitTests/SignlessTypes/shr
SingleSource/UnitTests/stmtexpr
SingleSource/UnitTests/StructModifyTest
SingleSource/UnitTests/TestLoop
SingleSource/UnitTests/Threads/2010-12-08-tls
SingleSource/UnitTests/Threads/tls
SingleSource/UnitTests/Vector/build
SingleSource/UnitTests/Vector/divides
SingleSource/UnitTests/Vector/sumarray
SingleSource/UnitTests/Vector/sumarray-dbl
SingleSource/UnitTests/vla

SingleSource/UnitTests/Vector/Altivec/2007-01-07-lvsl-lvsr-Regression
SingleSource/UnitTests/Vector/Altivec/alti.sdot
SingleSource/UnitTests/Vector/Altivec/casts
SingleSource/UnitTests/Vector/Altivec/test1

SingleSource/UnitTests/ms_struct-bitfield
SingleSource/UnitTests/ms_struct-bitfield-1
SingleSource/UnitTests/ms_struct-bitfield-init
SingleSource/UnitTests/ms_struct-bitfield-init-1
SingleSource/UnitTests/ms_struct_pack_layout
SingleSource/UnitTests/ms_struct_pack_layout-1

If anyone has any proposed modifications to this list, please let me know. Otherwise, I'll construct a patch to the test-suite makefiles and mail it for review soon.

Thanks in advance,
Hal

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory