[LLVMdev] [RFC] Benchmarking subset of the test suite

[Hal Finkel]

----- Original Message -----
> From: "Chris Lattner" <clattner at apple.com>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "LLVM Developers Mailing List" <llvmdev at cs.uiuc.edu>
> Sent: Sunday, May 4, 2014 1:37:42 PM
> Subject: Re: [LLVMdev] [RFC] Benchmarking subset of the test suite
> 
> On May 4, 2014, at 5:39 AM, Hal Finkel <hfinkel at anl.gov> wrote:
> > Without further ado, I propose that a test-suite configuration
> > designed for benchmarking exclude the following:
> 
> Unrelatedly to your point, I think we should just outright remove the
> bulk of the SingleSource/Regression / SingleSource/UnitTests tests.
>  They largely date back to before Clang existed and aren’t serving
> much useful value anymore.
> 
> Any of them that are useful (e.g. the EH tests?) could be saved or
> moved somewhere else.

I have mixed feelings about this. On the one hand, the regression and unit tests don't belong with the others, and if we start running applications multiple times for benchmarking, we'd likely want to exclude these correctness-only tests. On the other hand, I've found some of these tests quite useful in catching miscompiles (the EH tests, the bitfield tests, vector intrinsics tests and the va_args tests specifically come to mind). It might be useful to separate out some set of 'ABI' tests, even if we remove the others.

 -Hal

> 
> -Chris
> 
> 
> > 
> > SingleSource/Regression/C++/2003-05-14-array-init
> > SingleSource/Regression/C++/2003-05-14-expr_stmt
> > SingleSource/Regression/C/2003-05-14-initialize-string
> > SingleSource/Regression/C/2003-05-21-BitfieldHandling
> > SingleSource/Regression/C/2003-05-21-UnionBitfields
> > SingleSource/Regression/C/2003-05-21-UnionTest
> > SingleSource/Regression/C/2003-05-22-LocalTypeTest
> > SingleSource/Regression/C/2003-05-22-VarSizeArray
> > SingleSource/Regression/C/2003-05-23-TransparentUnion
> > SingleSource/Regression/C++/2003-06-08-BaseType
> > SingleSource/Regression/C++/2003-06-08-VirtualFunctions
> > SingleSource/Regression/C++/2003-06-13-Crasher
> > SingleSource/Regression/C/2003-06-16-InvalidInitializer
> > SingleSource/Regression/C/2003-06-16-VolatileError
> > SingleSource/Regression/C++/2003-08-20-EnumSizeProblem
> > SingleSource/Regression/C++/2003-09-29-NonPODsByValue
> > SingleSource/Regression/C/2003-10-12-GlobalVarInitializers
> > SingleSource/Regression/C/2004-02-03-AggregateCopy
> > SingleSource/Regression/C/2004-03-15-IndirectGoto
> > SingleSource/Regression/C/2005-05-06-LongLongSignedShift
> > SingleSource/Regression/C/2008-01-07-LongDouble
> > SingleSource/Regression/C++/2008-01-29-ParamAliasesReturn
> > SingleSource/Regression/C++/2011-03-28-Bitfield
> > SingleSource/Regression/C/badidx
> > SingleSource/Regression/C/bigstack
> > SingleSource/Regression/C++/BuiltinTypeInfo
> > SingleSource/Regression/C/callargs
> > SingleSource/Regression/C/casts
> > SingleSource/Regression/C/compare
> > SingleSource/Regression/C/ConstructorDestructorAttributes
> > SingleSource/Regression/C/DuffsDevice
> > SingleSource/Regression/C++/EH/class_hierarchy
> > SingleSource/Regression/C++/EH/ConditionalExpr
> > SingleSource/Regression/C++/EH/ctor_dtor_count
> > SingleSource/Regression/C++/EH/ctor_dtor_count-2
> > SingleSource/Regression/C++/EH/dead_try_block
> > SingleSource/Regression/C++/EH/exception_spec_test
> > SingleSource/Regression/C++/EH/function_try_block
> > SingleSource/Regression/C++/EH/inlined_cleanup
> > SingleSource/Regression/C++/EH/recursive-throw
> > SingleSource/Regression/C++/EH/simple_rethrow
> > SingleSource/Regression/C++/EH/simple_throw
> > SingleSource/Regression/C++/EH/throw_rethrow_test
> > SingleSource/Regression/C++/fixups
> > SingleSource/Regression/C++/global_ctor
> > SingleSource/Regression/C/globalrefs
> > SingleSource/Regression/C++/global_type
> > SingleSource/Regression/C++/ofstream_ctor
> > SingleSource/Regression/C/pointer_arithmetic
> > SingleSource/Regression/C++/pointer_member
> > SingleSource/Regression/C++/pointer_method
> > SingleSource/Regression/C++/pointer_method2
> > SingleSource/Regression/C/PR10189
> > SingleSource/Regression/C/PR1386
> > SingleSource/Regression/C/PR491
> > SingleSource/Regression/C/PR640
> > SingleSource/Regression/C++/short_circuit_dtor
> > SingleSource/Regression/C/sumarray
> > SingleSource/Regression/C/sumarraymalloc
> > SingleSource/Regression/C/testtrace
> > SingleSource/UnitTests/2002-04-17-PrintfChar
> > SingleSource/UnitTests/2002-05-02-ArgumentTest
> > SingleSource/UnitTests/2002-05-02-CastTest
> > SingleSource/UnitTests/2002-05-02-CastTest1
> > SingleSource/UnitTests/2002-05-02-CastTest2
> > SingleSource/UnitTests/2002-05-02-CastTest3
> > SingleSource/UnitTests/2002-05-02-ManyArguments
> > SingleSource/UnitTests/2002-05-03-NotTest
> > SingleSource/UnitTests/2002-05-19-DivTest
> > SingleSource/UnitTests/2002-08-02-CastTest
> > SingleSource/UnitTests/2002-08-02-CastTest2
> > SingleSource/UnitTests/2002-08-19-CodegenBug
> > SingleSource/UnitTests/2002-10-09-ArrayResolution
> > SingleSource/UnitTests/2002-10-12-StructureArgs
> > SingleSource/UnitTests/2002-10-12-StructureArgsSimple
> > SingleSource/UnitTests/2002-10-13-BadLoad
> > SingleSource/UnitTests/2002-12-13-MishaTest
> > SingleSource/UnitTests/2003-04-22-Switch
> > SingleSource/UnitTests/2003-05-02-DependentPHI
> > SingleSource/UnitTests/2003-05-07-VarArgs
> > SingleSource/UnitTests/2003-05-12-MinIntProblem
> > SingleSource/UnitTests/2003-05-14-AtExit
> > SingleSource/UnitTests/2003-05-26-Shorts
> > SingleSource/UnitTests/2003-05-31-CastToBool
> > SingleSource/UnitTests/2003-05-31-LongShifts
> > SingleSource/UnitTests/2003-07-06-IntOverflow
> > SingleSource/UnitTests/2003-07-08-BitOpsTest
> > SingleSource/UnitTests/2003-07-09-LoadShorts
> > SingleSource/UnitTests/2003-07-09-SignedArgs
> > SingleSource/UnitTests/2003-07-10-SignConversions
> > SingleSource/UnitTests/2003-08-05-CastFPToUint
> > SingleSource/UnitTests/2003-08-11-VaListArg
> > SingleSource/UnitTests/2003-08-20-FoldBug
> > SingleSource/UnitTests/2003-09-18-BitFieldTest
> > SingleSource/UnitTests/2003-10-13-SwitchTest
> > SingleSource/UnitTests/2003-10-29-ScalarReplBug
> > SingleSource/UnitTests/2004-02-02-NegativeZero
> > SingleSource/UnitTests/2004-06-20-StaticBitfieldInit
> > SingleSource/UnitTests/2004-11-28-GlobalBoolLayout
> > SingleSource/UnitTests/2005-05-11-Popcount-ffs-fls
> > SingleSource/UnitTests/2005-05-12-Int64ToFP
> > SingleSource/UnitTests/2005-05-13-SDivTwo
> > SingleSource/UnitTests/2005-07-17-INT-To-FP
> > SingleSource/UnitTests/2005-11-29-LongSwitch
> > SingleSource/UnitTests/2006-01-29-SimpleIndirectCall
> > SingleSource/UnitTests/2006-02-04-DivRem
> > SingleSource/UnitTests/2006-12-01-float_varg
> > SingleSource/UnitTests/2006-12-04-DynAllocAndRestore
> > SingleSource/UnitTests/2006-12-07-Compare64BitConstant
> > SingleSource/UnitTests/2006-12-11-LoadConstants
> > SingleSource/UnitTests/2007-01-04-KNR-Args
> > SingleSource/UnitTests/2007-03-02-VaCopy
> > SingleSource/UnitTests/2007-04-25-weak
> > SingleSource/UnitTests/2008-04-18-LoopBug
> > SingleSource/UnitTests/2008-04-20-LoopBug2
> > SingleSource/UnitTests/2008-07-13-InlineSetjmp
> > SingleSource/UnitTests/2009-04-16-BitfieldInitialization
> > SingleSource/UnitTests/2009-12-07-StructReturn
> > SingleSource/UnitTests/2010-05-24-BitfieldTest
> > SingleSource/UnitTests/AtomicOps
> > SingleSource/UnitTests/block-byref-cxxobj-test
> > SingleSource/UnitTests/block-byref-test
> > SingleSource/UnitTests/block-call-r7674133
> > SingleSource/UnitTests/block-copied-in-cxxobj
> > SingleSource/UnitTests/block-copied-in-cxxobj-1
> > SingleSource/UnitTests/blockstret
> > SingleSource/UnitTests/byval-alignment
> > SingleSource/UnitTests/conditional-gnu-ext
> > SingleSource/UnitTests/conditional-gnu-ext-cxx
> > SingleSource/UnitTests/DefaultInitDynArrays
> > SingleSource/UnitTests/FloatPrecision
> > SingleSource/UnitTests/initp1
> > SingleSource/UnitTests/member-function-pointers
> > SingleSource/UnitTests/printargs
> > SingleSource/UnitTests/SignlessTypes/cast2
> > SingleSource/UnitTests/SignlessTypes/cast-bug
> > SingleSource/UnitTests/SignlessTypes/ccc
> > SingleSource/UnitTests/SignlessTypes/div
> > SingleSource/UnitTests/SignlessTypes/factor
> > SingleSource/UnitTests/SignlessTypes/Large/cast
> > SingleSource/UnitTests/SignlessTypes/shr
> > SingleSource/UnitTests/stmtexpr
> > SingleSource/UnitTests/StructModifyTest
> > SingleSource/UnitTests/TestLoop
> > SingleSource/UnitTests/Threads/2010-12-08-tls
> > SingleSource/UnitTests/Threads/tls
> > SingleSource/UnitTests/Vector/build
> > SingleSource/UnitTests/Vector/divides
> > SingleSource/UnitTests/Vector/sumarray
> > SingleSource/UnitTests/Vector/sumarray-dbl
> > SingleSource/UnitTests/vla
> > 
> > SingleSource/UnitTests/Vector/Altivec/2007-01-07-lvsl-lvsr-Regression
> > SingleSource/UnitTests/Vector/Altivec/alti.sdot
> > SingleSource/UnitTests/Vector/Altivec/casts
> > SingleSource/UnitTests/Vector/Altivec/test1
> > 
> > SingleSource/UnitTests/ms_struct-bitfield
> > SingleSource/UnitTests/ms_struct-bitfield-1
> > SingleSource/UnitTests/ms_struct-bitfield-init
> > SingleSource/UnitTests/ms_struct-bitfield-init-1
> > SingleSource/UnitTests/ms_struct_pack_layout
> > SingleSource/UnitTests/ms_struct_pack_layout-1
> 
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory