[PATCH] Add benchmarking-only mode to the test suite

Thu May 15 05:59:57 PDT 2014

----- Original Message -----
> From: "Yi Kong" <kongy.dev at gmail.com>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "Eric Christopher" <echristo at gmail.com>, "llvm-commits" <llvm-commits at cs.uiuc.edu>, "Tobias Grosser"
> <tobias at grosser.es>
> Sent: Thursday, May 15, 2014 5:26:54 AM
> Subject: Re: [PATCH] Add benchmarking-only mode to the test suite
> 
> Hi Hal Finkel,
> 
> What's the criteria you use to to decide useful benchmarks?

Please refer to the LLVMDev thread "[RFC] Benchmarking subset of the test suite" in which I explain my methadology in detail.

> 
> I suggest you to also have a look at the standard deviation or MAD.

Of course this has already been considered and taken into account ;)

> Some of the tests have really large variance that we may not want to
> include when benchmarking, eg.
> Polybench/linear-algebra/kernels/3mm/3mm. I've attached a patch which
> makes tables sortable so that it is easier to investigate.

If you feel that there is a test or tests that have too large of a variance for useful benchmarking, please compose a list, explain your criteria, and we'll merge in some useful way.

> 
> I see you categorized some tests as not useful for benchmarking,
> maybe
> a better way to do is to extend running time by increasing problem
> size? For example, blowfish can be modified to have more Feistel
> rounds.

Yes, this is certainly true. My goal here is to represent things as they are now; as tests are updated to be more useful for benchmarking, they can be removed from the exclusion list.

Thanks for the feedback!

 -Hal

> 
> Cheers,
> Yi Kong
> 
> On 14 May 2014 17:34, Hal Finkel <hfinkel at anl.gov> wrote:
> > ----- Original Message -----
> >> From: "Eric Christopher" <echristo at gmail.com>
> >> To: "Hal Finkel" <hfinkel at anl.gov>
> >> Cc: "llvm-commits" <llvm-commits at cs.uiuc.edu>
> >> Sent: Wednesday, May 14, 2014 11:33:15 AM
> >> Subject: Re: [PATCH] Add benchmarking-only mode to the test suite
> >>
> >> This looks good to me. One request would be for a
> >> README.BENCHMARKING
> >> file at the top that describes what's going on and what someone
> >> should
> >> do when adding a new test to the testsuite.
> >
> > That's a good idea; will do. Thanks!
> >
> >  -Hal
> >
> >>
> >> -eric
> >>
> >> On Sun, May 11, 2014 at 4:39 PM, Hal Finkel <hfinkel at anl.gov>
> >> wrote:
> >> > Hello everyone,
> >> >
> >> > To follow-up on last week's thread on adding a benchmarking-only
> >> > mode to the test suite, which excludes those programs unlikely
> >> > to
> >> > use useful as benchmarks, here's a patch which provides an
> >> > implementation. When BENCHMARKING_ONLY is defined, the following
> >> > programs should be excluded:
> >> >
> >> > MultiSource/Applications/Burg/burg
> >> > MultiSource/Applications/treecc/treecc
> >> > MultiSource/Benchmarks/MiBench/office-ispell/office-ispell
> >> > MultiSource/Benchmarks/MiBench/security-blowfish/security-blowfish
> >> > MultiSource/Benchmarks/Prolangs-C/allroots/allroots
> >> > MultiSource/Benchmarks/Prolangs-C/archie-client/archie
> >> > MultiSource/Benchmarks/Prolangs-C/assembler/assembler
> >> > MultiSource/Benchmarks/Prolangs-C/compiler/compiler
> >> > MultiSource/Benchmarks/Prolangs-C++/deriv1/deriv1
> >> > MultiSource/Benchmarks/Prolangs-C++/deriv2/deriv2
> >> > MultiSource/Benchmarks/Prolangs-C++/family/family
> >> > MultiSource/Benchmarks/Prolangs-C/fixoutput/fixoutput
> >> > MultiSource/Benchmarks/Prolangs-C/football/football
> >> > MultiSource/Benchmarks/Prolangs-C++/fsm/fsm
> >> > MultiSource/Benchmarks/Prolangs-C++/garage/garage
> >> > MultiSource/Benchmarks/Prolangs-C++/NP/np
> >> > MultiSource/Benchmarks/Prolangs-C++/objects/objects
> >> > MultiSource/Benchmarks/Prolangs-C++/office/office
> >> > MultiSource/Benchmarks/Prolangs-C/plot2fig/plot2fig
> >> > MultiSource/Benchmarks/Prolangs-C++/shapes/shapes
> >> > MultiSource/Benchmarks/Prolangs-C/simulator/simulator
> >> > MultiSource/Benchmarks/Prolangs-C/TimberWolfMC/timberwolfmc
> >> > MultiSource/Benchmarks/Prolangs-C++/trees/trees
> >> > MultiSource/Benchmarks/Prolangs-C/unix-smail/unix-smail
> >> > MultiSource/Benchmarks/Prolangs-C/unix-tbl/unix-tbl
> >> > MultiSource/Benchmarks/Prolangs-C++/vcirc/vcirc
> >> > SingleSource/Benchmarks/McGill/exptree
> >> > SingleSource/Benchmarks/Shootout-C++/hello
> >> > SingleSource/Benchmarks/Shootout-C++/reversefile
> >> > SingleSource/Benchmarks/Shootout-C++/spellcheck
> >> > SingleSource/Benchmarks/Shootout-C++/sumcol
> >> > SingleSource/Benchmarks/Shootout-C++/wc
> >> > SingleSource/Benchmarks/Shootout-C++/wordfreq
> >> > SingleSource/Benchmarks/Shootout/hello
> >> > SingleSource/Regression/C++/2003-05-14-array-init
> >> > SingleSource/Regression/C++/2003-05-14-expr_stmt
> >> > SingleSource/Regression/C/2003-05-14-initialize-string
> >> > SingleSource/Regression/C/2003-05-21-BitfieldHandling
> >> > SingleSource/Regression/C/2003-05-21-UnionBitfields
> >> > SingleSource/Regression/C/2003-05-21-UnionTest
> >> > SingleSource/Regression/C/2003-05-22-LocalTypeTest
> >> > SingleSource/Regression/C/2003-05-22-VarSizeArray
> >> > SingleSource/Regression/C/2003-05-23-TransparentUnion
> >> > SingleSource/Regression/C++/2003-06-08-BaseType
> >> > SingleSource/Regression/C++/2003-06-08-VirtualFunctions
> >> > SingleSource/Regression/C++/2003-06-13-Crasher
> >> > SingleSource/Regression/C/2003-06-16-InvalidInitializer
> >> > SingleSource/Regression/C/2003-06-16-VolatileError
> >> > SingleSource/Regression/C++/2003-08-20-EnumSizeProblem
> >> > SingleSource/Regression/C++/2003-09-29-NonPODsByValue
> >> > SingleSource/Regression/C/2003-10-12-GlobalVarInitializers
> >> > SingleSource/Regression/C/2004-02-03-AggregateCopy
> >> > SingleSource/Regression/C/2004-03-15-IndirectGoto
> >> > SingleSource/Regression/C/2005-05-06-LongLongSignedShift
> >> > SingleSource/Regression/C/2008-01-07-LongDouble
> >> > SingleSource/Regression/C++/2008-01-29-ParamAliasesReturn
> >> > SingleSource/Regression/C++/2011-03-28-Bitfield
> >> > SingleSource/Regression/C/badidx
> >> > SingleSource/Regression/C/bigstack
> >> > SingleSource/Regression/C++/BuiltinTypeInfo
> >> > SingleSource/Regression/C/callargs
> >> > SingleSource/Regression/C/casts
> >> > SingleSource/Regression/C/compare
> >> > SingleSource/Regression/C/ConstructorDestructorAttributes
> >> > SingleSource/Regression/C/DuffsDevice
> >> > SingleSource/Regression/C++/EH/class_hierarchy
> >> > SingleSource/Regression/C++/EH/ConditionalExpr
> >> > SingleSource/Regression/C++/EH/ctor_dtor_count
> >> > SingleSource/Regression/C++/EH/ctor_dtor_count-2
> >> > SingleSource/Regression/C++/EH/dead_try_block
> >> > SingleSource/Regression/C++/EH/exception_spec_test
> >> > SingleSource/Regression/C++/EH/function_try_block
> >> > SingleSource/Regression/C++/EH/inlined_cleanup
> >> > SingleSource/Regression/C++/EH/recursive-throw
> >> > SingleSource/Regression/C++/EH/simple_rethrow
> >> > SingleSource/Regression/C++/EH/simple_throw
> >> > SingleSource/Regression/C++/EH/throw_rethrow_test
> >> > SingleSource/Regression/C++/fixups
> >> > SingleSource/Regression/C++/global_ctor
> >> > SingleSource/Regression/C/globalrefs
> >> > SingleSource/Regression/C++/global_type
> >> > SingleSource/Regression/C++/ofstream_ctor
> >> > SingleSource/Regression/C/pointer_arithmetic
> >> > SingleSource/Regression/C++/pointer_member
> >> > SingleSource/Regression/C++/pointer_method
> >> > SingleSource/Regression/C++/pointer_method2
> >> > SingleSource/Regression/C/PR10189
> >> > SingleSource/Regression/C/PR1386
> >> > SingleSource/Regression/C/PR491
> >> > SingleSource/Regression/C/PR640
> >> > SingleSource/Regression/C++/short_circuit_dtor
> >> > SingleSource/Regression/C/sumarray
> >> > SingleSource/Regression/C/sumarraymalloc
> >> > SingleSource/Regression/C/testtrace
> >> > SingleSource/UnitTests/2002-04-17-PrintfChar
> >> > SingleSource/UnitTests/2002-05-02-ArgumentTest
> >> > SingleSource/UnitTests/2002-05-02-CastTest
> >> > SingleSource/UnitTests/2002-05-02-CastTest1
> >> > SingleSource/UnitTests/2002-05-02-CastTest2
> >> > SingleSource/UnitTests/2002-05-02-CastTest3
> >> > SingleSource/UnitTests/2002-05-02-ManyArguments
> >> > SingleSource/UnitTests/2002-05-03-NotTest
> >> > SingleSource/UnitTests/2002-05-19-DivTest
> >> > SingleSource/UnitTests/2002-08-02-CastTest
> >> > SingleSource/UnitTests/2002-08-02-CastTest2
> >> > SingleSource/UnitTests/2002-08-19-CodegenBug
> >> > SingleSource/UnitTests/2002-10-09-ArrayResolution
> >> > SingleSource/UnitTests/2002-10-12-StructureArgs
> >> > SingleSource/UnitTests/2002-10-12-StructureArgsSimple
> >> > SingleSource/UnitTests/2002-10-13-BadLoad
> >> > SingleSource/UnitTests/2002-12-13-MishaTest
> >> > SingleSource/UnitTests/2003-04-22-Switch
> >> > SingleSource/UnitTests/2003-05-02-DependentPHI
> >> > SingleSource/UnitTests/2003-05-07-VarArgs
> >> > SingleSource/UnitTests/2003-05-12-MinIntProblem
> >> > SingleSource/UnitTests/2003-05-14-AtExit
> >> > SingleSource/UnitTests/2003-05-26-Shorts
> >> > SingleSource/UnitTests/2003-05-31-CastToBool
> >> > SingleSource/UnitTests/2003-05-31-LongShifts
> >> > SingleSource/UnitTests/2003-07-06-IntOverflow
> >> > SingleSource/UnitTests/2003-07-08-BitOpsTest
> >> > SingleSource/UnitTests/2003-07-09-LoadShorts
> >> > SingleSource/UnitTests/2003-07-09-SignedArgs
> >> > SingleSource/UnitTests/2003-07-10-SignConversions
> >> > SingleSource/UnitTests/2003-08-05-CastFPToUint
> >> > SingleSource/UnitTests/2003-08-11-VaListArg
> >> > SingleSource/UnitTests/2003-08-20-FoldBug
> >> > SingleSource/UnitTests/2003-09-18-BitFieldTest
> >> > SingleSource/UnitTests/2003-10-13-SwitchTest
> >> > SingleSource/UnitTests/2003-10-29-ScalarReplBug
> >> > SingleSource/UnitTests/2004-02-02-NegativeZero
> >> > SingleSource/UnitTests/2004-06-20-StaticBitfieldInit
> >> > SingleSource/UnitTests/2004-11-28-GlobalBoolLayout
> >> > SingleSource/UnitTests/2005-05-11-Popcount-ffs-fls
> >> > SingleSource/UnitTests/2005-05-12-Int64ToFP
> >> > SingleSource/UnitTests/2005-05-13-SDivTwo
> >> > SingleSource/UnitTests/2005-07-17-INT-To-FP
> >> > SingleSource/UnitTests/2005-11-29-LongSwitch
> >> > SingleSource/UnitTests/2006-01-29-SimpleIndirectCall
> >> > SingleSource/UnitTests/2006-02-04-DivRem
> >> > SingleSource/UnitTests/2006-12-01-float_varg
> >> > SingleSource/UnitTests/2006-12-04-DynAllocAndRestore
> >> > SingleSource/UnitTests/2006-12-07-Compare64BitConstant
> >> > SingleSource/UnitTests/2006-12-11-LoadConstants
> >> > SingleSource/UnitTests/2007-01-04-KNR-Args
> >> > SingleSource/UnitTests/2007-03-02-VaCopy
> >> > SingleSource/UnitTests/2007-04-25-weak
> >> > SingleSource/UnitTests/2008-04-18-LoopBug
> >> > SingleSource/UnitTests/2008-04-20-LoopBug2
> >> > SingleSource/UnitTests/2008-07-13-InlineSetjmp
> >> > SingleSource/UnitTests/2009-04-16-BitfieldInitialization
> >> > SingleSource/UnitTests/2009-12-07-StructReturn
> >> > SingleSource/UnitTests/2010-05-24-BitfieldTest
> >> > SingleSource/UnitTests/AtomicOps
> >> > SingleSource/UnitTests/block-byref-cxxobj-test
> >> > SingleSource/UnitTests/block-byref-test
> >> > SingleSource/UnitTests/block-call-r7674133
> >> > SingleSource/UnitTests/block-copied-in-cxxobj
> >> > SingleSource/UnitTests/block-copied-in-cxxobj-1
> >> > SingleSource/UnitTests/blockstret
> >> > SingleSource/UnitTests/byval-alignment
> >> > SingleSource/UnitTests/conditional-gnu-ext
> >> > SingleSource/UnitTests/conditional-gnu-ext-cxx
> >> > SingleSource/UnitTests/DefaultInitDynArrays
> >> > SingleSource/UnitTests/FloatPrecision
> >> > SingleSource/UnitTests/initp1
> >> > SingleSource/UnitTests/member-function-pointers
> >> > SingleSource/UnitTests/printargs
> >> > SingleSource/UnitTests/SignlessTypes/cast2
> >> > SingleSource/UnitTests/SignlessTypes/cast-bug
> >> > SingleSource/UnitTests/SignlessTypes/ccc
> >> > SingleSource/UnitTests/SignlessTypes/div
> >> > SingleSource/UnitTests/SignlessTypes/factor
> >> > SingleSource/UnitTests/SignlessTypes/Large/cast
> >> > SingleSource/UnitTests/SignlessTypes/shr
> >> > SingleSource/UnitTests/stmtexpr
> >> > SingleSource/UnitTests/StructModifyTest
> >> > SingleSource/UnitTests/TestLoop
> >> > SingleSource/UnitTests/Threads/2010-12-08-tls
> >> > SingleSource/UnitTests/Threads/tls
> >> > SingleSource/UnitTests/Vector/build
> >> > SingleSource/UnitTests/Vector/divides
> >> > SingleSource/UnitTests/Vector/sumarray
> >> > SingleSource/UnitTests/Vector/sumarray-dbl
> >> > SingleSource/UnitTests/vla
> >> >
> >> > SingleSource/UnitTests/Vector/Altivec/2007-01-07-lvsl-lvsr-Regression
> >> > SingleSource/UnitTests/Vector/Altivec/alti.sdot
> >> > SingleSource/UnitTests/Vector/Altivec/casts
> >> > SingleSource/UnitTests/Vector/Altivec/test1
> >> >
> >> > SingleSource/UnitTests/ms_struct-bitfield
> >> > SingleSource/UnitTests/ms_struct-bitfield-1
> >> > SingleSource/UnitTests/ms_struct-bitfield-init
> >> > SingleSource/UnitTests/ms_struct-bitfield-init-1
> >> > SingleSource/UnitTests/ms_struct_pack_layout
> >> > SingleSource/UnitTests/ms_struct_pack_layout-1
> >> >
> >> > Please review.
> >> >
> >> > Thanks again,
> >> > Hal
> >> >
> >> > --
> >> > Hal Finkel
> >> > Assistant Computational Scientist
> >> > Leadership Computing Facility
> >> > Argonne National Laboratory
> >> >
> >> > _______________________________________________
> >> > llvm-commits mailing list
> >> > llvm-commits at cs.uiuc.edu
> >> > http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> >> >
> >>
> >
> > --
> > Hal Finkel
> > Assistant Computational Scientist
> > Leadership Computing Facility
> > Argonne National Laboratory
> > _______________________________________________
> > llvm-commits mailing list
> > llvm-commits at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory