[test-suite] r210251 - Add a benchmarking-only mode to the test suite
Hal Finkel
hfinkel at anl.gov
Thu Jun 12 16:02:39 PDT 2014
----- Original Message -----
> From: "Andrew Trick" <atrick at apple.com>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "llvm commits" <llvm-commits at cs.uiuc.edu>
> Sent: Thursday, June 12, 2014 5:58:51 PM
> Subject: Re: [test-suite] r210251 - Add a benchmarking-only mode to the test suite
>
> I plan to remove the old BENCHMARK_ONLY flag now and use yours
> instead. I think I was the only one using it.
>
> Do you by chance want to rename yours to BENCHMARK_ONLY, which is
> less typing?
How about BENCHMARKS_ONLY? Using the singular seems confusing. What do you think?
-Hal
>
> -Andy
>
> On Jun 5, 2014, at 2:24 AM, Hal Finkel <hfinkel at anl.gov> wrote:
>
> > Author: hfinkel
> > Date: Thu Jun 5 04:24:05 2014
> > New Revision: 210251
> >
> > URL: http://llvm.org/viewvc/llvm-project?rev=210251&view=rev
> > Log:
> > Add a benchmarking-only mode to the test suite
> >
> > This adds a benchmarking-only mode to the test suite, which
> > excludes those
> > programs unlikely to use useful as benchmarks. When
> > BENCHMARKING_ONLY is
> > defined, the following programs should be excluded:
> >
> > MultiSource/Applications/Burg/burg
> > MultiSource/Applications/treecc/treecc
> > MultiSource/Benchmarks/MiBench/office-ispell/office-ispell
> > MultiSource/Benchmarks/MiBench/security-blowfish/security-blowfish
> > MultiSource/Benchmarks/Prolangs-C/allroots/allroots
> > MultiSource/Benchmarks/Prolangs-C/archie-client/archie
> > MultiSource/Benchmarks/Prolangs-C/assembler/assembler
> > MultiSource/Benchmarks/Prolangs-C/compiler/compiler
> > MultiSource/Benchmarks/Prolangs-C++/deriv1/deriv1
> > MultiSource/Benchmarks/Prolangs-C++/deriv2/deriv2
> > MultiSource/Benchmarks/Prolangs-C++/family/family
> > MultiSource/Benchmarks/Prolangs-C/fixoutput/fixoutput
> > MultiSource/Benchmarks/Prolangs-C/football/football
> > MultiSource/Benchmarks/Prolangs-C++/fsm/fsm
> > MultiSource/Benchmarks/Prolangs-C++/garage/garage
> > MultiSource/Benchmarks/Prolangs-C++/NP/np
> > MultiSource/Benchmarks/Prolangs-C++/objects/objects
> > MultiSource/Benchmarks/Prolangs-C++/office/office
> > MultiSource/Benchmarks/Prolangs-C/plot2fig/plot2fig
> > MultiSource/Benchmarks/Prolangs-C++/shapes/shapes
> > MultiSource/Benchmarks/Prolangs-C/simulator/simulator
> > MultiSource/Benchmarks/Prolangs-C/TimberWolfMC/timberwolfmc
> > MultiSource/Benchmarks/Prolangs-C++/trees/trees
> > MultiSource/Benchmarks/Prolangs-C/unix-smail/unix-smail
> > MultiSource/Benchmarks/Prolangs-C/unix-tbl/unix-tbl
> > MultiSource/Benchmarks/Prolangs-C++/vcirc/vcirc
> > SingleSource/Benchmarks/McGill/exptree
> > SingleSource/Benchmarks/Shootout-C++/hello
> > SingleSource/Benchmarks/Shootout-C++/reversefile
> > SingleSource/Benchmarks/Shootout-C++/spellcheck
> > SingleSource/Benchmarks/Shootout-C++/sumcol
> > SingleSource/Benchmarks/Shootout-C++/wc
> > SingleSource/Benchmarks/Shootout-C++/wordfreq
> > SingleSource/Benchmarks/Shootout/hello
> > SingleSource/Regression/C++/2003-05-14-array-init
> > SingleSource/Regression/C++/2003-05-14-expr_stmt
> > SingleSource/Regression/C/2003-05-14-initialize-string
> > SingleSource/Regression/C/2003-05-21-BitfieldHandling
> > SingleSource/Regression/C/2003-05-21-UnionBitfields
> > SingleSource/Regression/C/2003-05-21-UnionTest
> > SingleSource/Regression/C/2003-05-22-LocalTypeTest
> > SingleSource/Regression/C/2003-05-22-VarSizeArray
> > SingleSource/Regression/C/2003-05-23-TransparentUnion
> > SingleSource/Regression/C++/2003-06-08-BaseType
> > SingleSource/Regression/C++/2003-06-08-VirtualFunctions
> > SingleSource/Regression/C++/2003-06-13-Crasher
> > SingleSource/Regression/C/2003-06-16-InvalidInitializer
> > SingleSource/Regression/C/2003-06-16-VolatileError
> > SingleSource/Regression/C++/2003-08-20-EnumSizeProblem
> > SingleSource/Regression/C++/2003-09-29-NonPODsByValue
> > SingleSource/Regression/C/2003-10-12-GlobalVarInitializers
> > SingleSource/Regression/C/2004-02-03-AggregateCopy
> > SingleSource/Regression/C/2004-03-15-IndirectGoto
> > SingleSource/Regression/C/2005-05-06-LongLongSignedShift
> > SingleSource/Regression/C/2008-01-07-LongDouble
> > SingleSource/Regression/C++/2008-01-29-ParamAliasesReturn
> > SingleSource/Regression/C++/2011-03-28-Bitfield
> > SingleSource/Regression/C/badidx
> > SingleSource/Regression/C/bigstack
> > SingleSource/Regression/C++/BuiltinTypeInfo
> > SingleSource/Regression/C/callargs
> > SingleSource/Regression/C/casts
> > SingleSource/Regression/C/compare
> > SingleSource/Regression/C/ConstructorDestructorAttributes
> > SingleSource/Regression/C/DuffsDevice
> > SingleSource/Regression/C++/EH/class_hierarchy
> > SingleSource/Regression/C++/EH/ConditionalExpr
> > SingleSource/Regression/C++/EH/ctor_dtor_count
> > SingleSource/Regression/C++/EH/ctor_dtor_count-2
> > SingleSource/Regression/C++/EH/dead_try_block
> > SingleSource/Regression/C++/EH/exception_spec_test
> > SingleSource/Regression/C++/EH/function_try_block
> > SingleSource/Regression/C++/EH/inlined_cleanup
> > SingleSource/Regression/C++/EH/recursive-throw
> > SingleSource/Regression/C++/EH/simple_rethrow
> > SingleSource/Regression/C++/EH/simple_throw
> > SingleSource/Regression/C++/EH/throw_rethrow_test
> > SingleSource/Regression/C++/fixups
> > SingleSource/Regression/C++/global_ctor
> > SingleSource/Regression/C/globalrefs
> > SingleSource/Regression/C++/global_type
> > SingleSource/Regression/C++/ofstream_ctor
> > SingleSource/Regression/C/pointer_arithmetic
> > SingleSource/Regression/C++/pointer_member
> > SingleSource/Regression/C++/pointer_method
> > SingleSource/Regression/C++/pointer_method2
> > SingleSource/Regression/C/PR10189
> > SingleSource/Regression/C/PR1386
> > SingleSource/Regression/C/PR491
> > SingleSource/Regression/C/PR640
> > SingleSource/Regression/C++/short_circuit_dtor
> > SingleSource/Regression/C/sumarray
> > SingleSource/Regression/C/sumarraymalloc
> > SingleSource/Regression/C/testtrace
> > SingleSource/UnitTests/2002-04-17-PrintfChar
> > SingleSource/UnitTests/2002-05-02-ArgumentTest
> > SingleSource/UnitTests/2002-05-02-CastTest
> > SingleSource/UnitTests/2002-05-02-CastTest1
> > SingleSource/UnitTests/2002-05-02-CastTest2
> > SingleSource/UnitTests/2002-05-02-CastTest3
> > SingleSource/UnitTests/2002-05-02-ManyArguments
> > SingleSource/UnitTests/2002-05-03-NotTest
> > SingleSource/UnitTests/2002-05-19-DivTest
> > SingleSource/UnitTests/2002-08-02-CastTest
> > SingleSource/UnitTests/2002-08-02-CastTest2
> > SingleSource/UnitTests/2002-08-19-CodegenBug
> > SingleSource/UnitTests/2002-10-09-ArrayResolution
> > SingleSource/UnitTests/2002-10-12-StructureArgs
> > SingleSource/UnitTests/2002-10-12-StructureArgsSimple
> > SingleSource/UnitTests/2002-10-13-BadLoad
> > SingleSource/UnitTests/2002-12-13-MishaTest
> > SingleSource/UnitTests/2003-04-22-Switch
> > SingleSource/UnitTests/2003-05-02-DependentPHI
> > SingleSource/UnitTests/2003-05-07-VarArgs
> > SingleSource/UnitTests/2003-05-12-MinIntProblem
> > SingleSource/UnitTests/2003-05-14-AtExit
> > SingleSource/UnitTests/2003-05-26-Shorts
> > SingleSource/UnitTests/2003-05-31-CastToBool
> > SingleSource/UnitTests/2003-05-31-LongShifts
> > SingleSource/UnitTests/2003-07-06-IntOverflow
> > SingleSource/UnitTests/2003-07-08-BitOpsTest
> > SingleSource/UnitTests/2003-07-09-LoadShorts
> > SingleSource/UnitTests/2003-07-09-SignedArgs
> > SingleSource/UnitTests/2003-07-10-SignConversions
> > SingleSource/UnitTests/2003-08-05-CastFPToUint
> > SingleSource/UnitTests/2003-08-11-VaListArg
> > SingleSource/UnitTests/2003-08-20-FoldBug
> > SingleSource/UnitTests/2003-09-18-BitFieldTest
> > SingleSource/UnitTests/2003-10-13-SwitchTest
> > SingleSource/UnitTests/2003-10-29-ScalarReplBug
> > SingleSource/UnitTests/2004-02-02-NegativeZero
> > SingleSource/UnitTests/2004-06-20-StaticBitfieldInit
> > SingleSource/UnitTests/2004-11-28-GlobalBoolLayout
> > SingleSource/UnitTests/2005-05-11-Popcount-ffs-fls
> > SingleSource/UnitTests/2005-05-12-Int64ToFP
> > SingleSource/UnitTests/2005-05-13-SDivTwo
> > SingleSource/UnitTests/2005-07-17-INT-To-FP
> > SingleSource/UnitTests/2005-11-29-LongSwitch
> > SingleSource/UnitTests/2006-01-29-SimpleIndirectCall
> > SingleSource/UnitTests/2006-02-04-DivRem
> > SingleSource/UnitTests/2006-12-01-float_varg
> > SingleSource/UnitTests/2006-12-04-DynAllocAndRestore
> > SingleSource/UnitTests/2006-12-07-Compare64BitConstant
> > SingleSource/UnitTests/2006-12-11-LoadConstants
> > SingleSource/UnitTests/2007-01-04-KNR-Args
> > SingleSource/UnitTests/2007-03-02-VaCopy
> > SingleSource/UnitTests/2007-04-25-weak
> > SingleSource/UnitTests/2008-04-18-LoopBug
> > SingleSource/UnitTests/2008-04-20-LoopBug2
> > SingleSource/UnitTests/2008-07-13-InlineSetjmp
> > SingleSource/UnitTests/2009-04-16-BitfieldInitialization
> > SingleSource/UnitTests/2009-12-07-StructReturn
> > SingleSource/UnitTests/2010-05-24-BitfieldTest
> > SingleSource/UnitTests/AtomicOps
> > SingleSource/UnitTests/block-byref-cxxobj-test
> > SingleSource/UnitTests/block-byref-test
> > SingleSource/UnitTests/block-call-r7674133
> > SingleSource/UnitTests/block-copied-in-cxxobj
> > SingleSource/UnitTests/block-copied-in-cxxobj-1
> > SingleSource/UnitTests/blockstret
> > SingleSource/UnitTests/byval-alignment
> > SingleSource/UnitTests/conditional-gnu-ext
> > SingleSource/UnitTests/conditional-gnu-ext-cxx
> > SingleSource/UnitTests/DefaultInitDynArrays
> > SingleSource/UnitTests/FloatPrecision
> > SingleSource/UnitTests/initp1
> > SingleSource/UnitTests/member-function-pointers
> > SingleSource/UnitTests/printargs
> > SingleSource/UnitTests/SignlessTypes/cast2
> > SingleSource/UnitTests/SignlessTypes/cast-bug
> > SingleSource/UnitTests/SignlessTypes/ccc
> > SingleSource/UnitTests/SignlessTypes/div
> > SingleSource/UnitTests/SignlessTypes/factor
> > SingleSource/UnitTests/SignlessTypes/Large/cast
> > SingleSource/UnitTests/SignlessTypes/shr
> > SingleSource/UnitTests/stmtexpr
> > SingleSource/UnitTests/StructModifyTest
> > SingleSource/UnitTests/TestLoop
> > SingleSource/UnitTests/Threads/2010-12-08-tls
> > SingleSource/UnitTests/Threads/tls
> > SingleSource/UnitTests/Vector/build
> > SingleSource/UnitTests/Vector/divides
> > SingleSource/UnitTests/Vector/sumarray
> > SingleSource/UnitTests/Vector/sumarray-dbl
> > SingleSource/UnitTests/vla
> >
> > SingleSource/UnitTests/Vector/Altivec/2007-01-07-lvsl-lvsr-Regression
> > SingleSource/UnitTests/Vector/Altivec/alti.sdot
> > SingleSource/UnitTests/Vector/Altivec/casts
> > SingleSource/UnitTests/Vector/Altivec/test1
> >
> > SingleSource/UnitTests/ms_struct-bitfield
> > SingleSource/UnitTests/ms_struct-bitfield-1
> > SingleSource/UnitTests/ms_struct-bitfield-init
> > SingleSource/UnitTests/ms_struct-bitfield-init-1
> > SingleSource/UnitTests/ms_struct_pack_layout
> > SingleSource/UnitTests/ms_struct_pack_layout-1
> >
> > I constructed this exclusion list primarily based on the following
> > experiment:
> > I ran the test suite 10 times in three configurations: 1) On an IBM
> > POWER7 (P7)
> > with -O3 -mvsx, 2) On a P7 at -O0 and 3) On an Intel Xeon E5430
> > with -O3 all
> > using make -j6. I then used the ministat utility (which performs a
> > T test) to
> > compare the timings of the two P7 configurations against each other
> > and the
> > Xeon configuration, requiring a detectable difference at 99.5%
> > confidence. I
> > looked for tests that showed no significant difference in all three
> > comparisons. The running configuration here is purposefully noisy,
> > the idea is
> > to eliminate those tests that are significantly sensitive to
> > startup time, file
> > I/O time, memory bandwidth, etc., or just too short, and by running
> > many tests
> > in parallel (non-deterministically), my hope is to eliminate those
> > tests can
> > cannot usefully serve as benchmarks in a "normal" environment.
> >
> > Modified:
> > test-suite/trunk/MultiSource/Applications/Makefile
> > test-suite/trunk/MultiSource/Benchmarks/MiBench/Makefile
> > test-suite/trunk/MultiSource/Benchmarks/Prolangs-C++/Makefile
> > test-suite/trunk/MultiSource/Benchmarks/Prolangs-C/Makefile
> > test-suite/trunk/SingleSource/Benchmarks/McGill/Makefile
> > test-suite/trunk/SingleSource/Benchmarks/Shootout-C++/Makefile
> > test-suite/trunk/SingleSource/Benchmarks/Shootout/Makefile
> > test-suite/trunk/SingleSource/Regression/C++/EH/Makefile
> > test-suite/trunk/SingleSource/Regression/C++/Makefile
> > test-suite/trunk/SingleSource/Regression/C/Makefile
> > test-suite/trunk/SingleSource/UnitTests/Makefile
> > test-suite/trunk/SingleSource/UnitTests/SignlessTypes/Large/Makefile
> > test-suite/trunk/SingleSource/UnitTests/SignlessTypes/Makefile
> > test-suite/trunk/SingleSource/UnitTests/Threads/Makefile
> > test-suite/trunk/SingleSource/UnitTests/Vector/Altivec/Makefile
> > test-suite/trunk/SingleSource/UnitTests/Vector/Makefile
> >
> > Modified: test-suite/trunk/MultiSource/Applications/Makefile
> > URL:
> > http://llvm.org/viewvc/llvm-project/test-suite/trunk/MultiSource/Applications/Makefile?rev=210251&r1=210250&r2=210251&view=diff
> > ==============================================================================
> > --- test-suite/trunk/MultiSource/Applications/Makefile (original)
> > +++ test-suite/trunk/MultiSource/Applications/Makefile Thu Jun 5
> > 04:24:05 2014
> > @@ -42,4 +42,9 @@ PARALLEL_DIRS := $(filter-out obsequi, $
> > PARALLEL_DIRS := $(filter-out lemon, $(PARALLEL_DIRS))
> > endif
> >
> > +ifdef BENCHMARKING_ONLY
> > +PARALLEL_DIRS := $(filter-out Burg, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out treecc, $(PARALLEL_DIRS))
> > +endif
> > +
> > include $(LEVEL)/Makefile.programs
> >
> > Modified: test-suite/trunk/MultiSource/Benchmarks/MiBench/Makefile
> > URL:
> > http://llvm.org/viewvc/llvm-project/test-suite/trunk/MultiSource/Benchmarks/MiBench/Makefile?rev=210251&r1=210250&r2=210251&view=diff
> > ==============================================================================
> > --- test-suite/trunk/MultiSource/Benchmarks/MiBench/Makefile
> > (original)
> > +++ test-suite/trunk/MultiSource/Benchmarks/MiBench/Makefile Thu
> > Jun 5 04:24:05 2014
> > @@ -35,4 +35,9 @@ ifeq ($(ARCH),XCore)
> > PARALLEL_DIRS := $(filter-out office-ispell, $(PARALLEL_DIRS))
> > endif
> >
> > +ifdef BENCHMARKING_ONLY
> > +PARALLEL_DIRS := $(filter-out office-ispell, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out security-blowfish, $(PARALLEL_DIRS))
> > +endif
> > +
> > include $(LEVEL)/Makefile.programs
> >
> > Modified:
> > test-suite/trunk/MultiSource/Benchmarks/Prolangs-C++/Makefile
> > URL:
> > http://llvm.org/viewvc/llvm-project/test-suite/trunk/MultiSource/Benchmarks/Prolangs-C%2B%2B/Makefile?rev=210251&r1=210250&r2=210251&view=diff
> > ==============================================================================
> > --- test-suite/trunk/MultiSource/Benchmarks/Prolangs-C++/Makefile
> > (original)
> > +++ test-suite/trunk/MultiSource/Benchmarks/Prolangs-C++/Makefile
> > Thu Jun 5 04:24:05 2014
> > @@ -1,4 +1,19 @@
> > LEVEL = ../../..
> > PARALLEL_DIRS := NP city deriv1 deriv2 employ family fsm garage \
> > life objects ocean office primes shapes simul
> > trees vcirc
> > +
> > +ifdef BENCHMARKING_ONLY
> > +PARALLEL_DIRS := $(filter-out deriv1, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out deriv2, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out family, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out fsm, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out garage, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out NP, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out objects, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out office, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out shapes, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out trees, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out vcirc, $(PARALLEL_DIRS))
> > +endif
> > +
> > include $(LEVEL)/Makefile.programs
> >
> > Modified:
> > test-suite/trunk/MultiSource/Benchmarks/Prolangs-C/Makefile
> > URL:
> > http://llvm.org/viewvc/llvm-project/test-suite/trunk/MultiSource/Benchmarks/Prolangs-C/Makefile?rev=210251&r1=210250&r2=210251&view=diff
> > ==============================================================================
> > --- test-suite/trunk/MultiSource/Benchmarks/Prolangs-C/Makefile
> > (original)
> > +++ test-suite/trunk/MultiSource/Benchmarks/Prolangs-C/Makefile Thu
> > Jun 5 04:24:05 2014
> > @@ -22,4 +22,18 @@ PARALLEL_DIRS := $(filter-out archie-cli
> > PARALLEL_DIRS := $(filter-out unix-smail, $(PARALLEL_DIRS))
> > endif
> >
> > +ifdef BENCHMARKING_ONLY
> > +PARALLEL_DIRS := $(filter-out allroots, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out archie-client, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out assembler, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out compiler, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out fixoutput, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out football, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out plot2fig, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out simulator, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out TimberWolfMC, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out unix-smail, $(PARALLEL_DIRS))
> > +PARALLEL_DIRS := $(filter-out unix-tbl, $(PARALLEL_DIRS))
> > +endif
> > +
> > include $(LEVEL)/Makefile.programs
> >
> > Modified: test-suite/trunk/SingleSource/Benchmarks/McGill/Makefile
> > URL:
> > http://llvm.org/viewvc/llvm-project/test-suite/trunk/SingleSource/Benchmarks/McGill/Makefile?rev=210251&r1=210250&r2=210251&view=diff
> > ==============================================================================
> > --- test-suite/trunk/SingleSource/Benchmarks/McGill/Makefile
> > (original)
> > +++ test-suite/trunk/SingleSource/Benchmarks/McGill/Makefile Thu
> > Jun 5 04:24:05 2014
> > @@ -2,4 +2,8 @@ LEVEL = ../../..
> > LDFLAGS += -lm
> > FP_TOLERANCE := 0.001
> >
> > +ifdef BENCHMARKING_ONLY
> > +PROGRAMS_TO_SKIP += exptree
> > +endif
> > +
> > include $(LEVEL)/SingleSource/Makefile.singlesrc
> >
> > Modified:
> > test-suite/trunk/SingleSource/Benchmarks/Shootout-C++/Makefile
> > URL:
> > http://llvm.org/viewvc/llvm-project/test-suite/trunk/SingleSource/Benchmarks/Shootout-C%2B%2B/Makefile?rev=210251&r1=210250&r2=210251&view=diff
> > ==============================================================================
> > --- test-suite/trunk/SingleSource/Benchmarks/Shootout-C++/Makefile
> > (original)
> > +++ test-suite/trunk/SingleSource/Benchmarks/Shootout-C++/Makefile
> > Thu Jun 5 04:24:05 2014
> > @@ -13,5 +13,9 @@ RUNTIMELIMIT:=10800
> > endif
> > endif
> >
> > +ifdef BENCHMARKING_ONLY
> > +PROGRAMS_TO_SKIP += hello reversefile spellcheck sumcol wc
> > wordfreq
> > +endif
> > +
> > LDFLAGS += -lstdc++
> > include $(LEVEL)/SingleSource/Makefile.singlesrc
> >
> > Modified:
> > test-suite/trunk/SingleSource/Benchmarks/Shootout/Makefile
> > URL:
> > http://llvm.org/viewvc/llvm-project/test-suite/trunk/SingleSource/Benchmarks/Shootout/Makefile?rev=210251&r1=210250&r2=210251&view=diff
> > ==============================================================================
> > --- test-suite/trunk/SingleSource/Benchmarks/Shootout/Makefile
> > (original)
> > +++ test-suite/trunk/SingleSource/Benchmarks/Shootout/Makefile Thu
> > Jun 5 04:24:05 2014
> > @@ -5,4 +5,8 @@ ifeq ($(ARCH),XCore)
> > XCORE_TARGET_NEEDS_MEMORY := 256
> > endif
> >
> > +ifdef BENCHMARKING_ONLY
> > +PROGRAMS_TO_SKIP += hello
> > +endif
> > +
> > include $(LEVEL)/SingleSource/Makefile.singlesrc
> >
> > Modified: test-suite/trunk/SingleSource/Regression/C++/EH/Makefile
> > URL:
> > http://llvm.org/viewvc/llvm-project/test-suite/trunk/SingleSource/Regression/C%2B%2B/EH/Makefile?rev=210251&r1=210250&r2=210251&view=diff
> > ==============================================================================
> > --- test-suite/trunk/SingleSource/Regression/C++/EH/Makefile
> > (original)
> > +++ test-suite/trunk/SingleSource/Regression/C++/EH/Makefile Thu
> > Jun 5 04:24:05 2014
> > @@ -3,5 +3,21 @@ REQUIRES_EH_SUPPORT = 1
> >
> > include $(LEVEL)/Makefile.config
> >
> > +ifdef BENCHMARKING_ONLY
> > +PROGRAMS_TO_SKIP += \
> > + class_hierarchy \
> > + ConditionalExpr \
> > + ctor_dtor_count \
> > + ctor_dtor_count-2 \
> > + dead_try_block \
> > + exception_spec_test \
> > + function_try_block \
> > + inlined_cleanup \
> > + recursive-throw \
> > + simple_rethrow \
> > + simple_throw \
> > + throw_rethrow_test
> > +endif
> > +
> > LDFLAGS += -lstdc++
> > include $(LEVEL)/SingleSource/Makefile.singlesrc
> >
> > Modified: test-suite/trunk/SingleSource/Regression/C++/Makefile
> > URL:
> > http://llvm.org/viewvc/llvm-project/test-suite/trunk/SingleSource/Regression/C%2B%2B/Makefile?rev=210251&r1=210250&r2=210251&view=diff
> > ==============================================================================
> > --- test-suite/trunk/SingleSource/Regression/C++/Makefile
> > (original)
> > +++ test-suite/trunk/SingleSource/Regression/C++/Makefile Thu Jun
> > 5 04:24:05 2014
> > @@ -8,5 +8,27 @@ LEVEL = ../../..
> >
> > DIRS=EH
> >
> > +ifdef BENCHMARKING_ONLY
> > +PROGRAMS_TO_SKIP += \
> > + 2003-05-14-array-init \
> > + 2003-05-14-expr_stmt \
> > + 2003-06-08-BaseType \
> > + 2003-06-08-VirtualFunctions \
> > + 2003-06-13-Crasher \
> > + 2003-08-20-EnumSizeProblem \
> > + 2003-09-29-NonPODsByValue \
> > + 2008-01-29-ParamAliasesReturn \
> > + 2011-03-28-Bitfield \
> > + BuiltinTypeInfo \
> > + fixups \
> > + global_ctor \
> > + global_type \
> > + ofstream_ctor \
> > + pointer_member \
> > + pointer_method \
> > + pointer_method2 \
> > + short_circuit_dtor
> > +endif
> > +
> > LDFLAGS += -lstdc++
> > include $(LEVEL)/SingleSource/Makefile.singlesrc
> >
> > Modified: test-suite/trunk/SingleSource/Regression/C/Makefile
> > URL:
> > http://llvm.org/viewvc/llvm-project/test-suite/trunk/SingleSource/Regression/C/Makefile?rev=210251&r1=210250&r2=210251&view=diff
> > ==============================================================================
> > --- test-suite/trunk/SingleSource/Regression/C/Makefile (original)
> > +++ test-suite/trunk/SingleSource/Regression/C/Makefile Thu Jun 5
> > 04:24:05 2014
> > @@ -6,5 +6,39 @@ EXEC_XFAILS = casts
> > endif
> > endif
> >
> > +ifdef BENCHMARKING_ONLY
> > +PROGRAMS_TO_SKIP += \
> > + 2003-05-14-initialize-string \
> > + 2003-05-21-BitfieldHandling \
> > + 2003-05-21-UnionBitfields \
> > + 2003-05-21-UnionTest \
> > + 2003-05-22-LocalTypeTest \
> > + 2003-05-22-VarSizeArray \
> > + 2003-05-23-TransparentUnion \
> > + 2003-06-16-InvalidInitializer \
> > + 2003-06-16-VolatileError \
> > + 2003-10-12-GlobalVarInitializers \
> > + 2004-02-03-AggregateCopy \
> > + 2004-03-15-IndirectGoto \
> > + 2005-05-06-LongLongSignedShift \
> > + 2008-01-07-LongDouble \
> > + badidx \
> > + bigstack \
> > + callargs \
> > + casts \
> > + compare \
> > + ConstructorDestructorAttributes \
> > + DuffsDevice \
> > + globalrefs \
> > + pointer_arithmetic \
> > + PR10189 \
> > + PR1386 \
> > + PR491 \
> > + PR640 \
> > + sumarray \
> > + sumarraymalloc \
> > + testtrace
> > +endif
> > +
> > include $(LEVEL)/SingleSource/Makefile.singlesrc
> >
> >
> > Modified: test-suite/trunk/SingleSource/UnitTests/Makefile
> > URL:
> > http://llvm.org/viewvc/llvm-project/test-suite/trunk/SingleSource/UnitTests/Makefile?rev=210251&r1=210250&r2=210251&view=diff
> > ==============================================================================
> > --- test-suite/trunk/SingleSource/UnitTests/Makefile (original)
> > +++ test-suite/trunk/SingleSource/UnitTests/Makefile Thu Jun 5
> > 04:24:05 2014
> > @@ -48,5 +48,93 @@ PROGRAMS_TO_SKIP += initp1
> > XCORE_TARGET_NEEDS_MEMORY := 1
> > endif
> >
> > +ifdef BENCHMARKING_ONLY
> > +PROGRAMS_TO_SKIP += \
> > + 2002-04-17-PrintfChar \
> > + 2002-05-02-ArgumentTest \
> > + 2002-05-02-CastTest \
> > + 2002-05-02-CastTest1 \
> > + 2002-05-02-CastTest2 \
> > + 2002-05-02-CastTest3 \
> > + 2002-05-02-ManyArguments \
> > + 2002-05-03-NotTest \
> > + 2002-05-19-DivTest \
> > + 2002-08-02-CastTest \
> > + 2002-08-02-CastTest2 \
> > + 2002-08-19-CodegenBug \
> > + 2002-10-09-ArrayResolution \
> > + 2002-10-12-StructureArgs \
> > + 2002-10-12-StructureArgsSimple \
> > + 2002-10-13-BadLoad \
> > + 2002-12-13-MishaTest \
> > + 2003-04-22-Switch \
> > + 2003-05-02-DependentPHI \
> > + 2003-05-07-VarArgs \
> > + 2003-05-12-MinIntProblem \
> > + 2003-05-14-AtExit \
> > + 2003-05-26-Shorts \
> > + 2003-05-31-CastToBool \
> > + 2003-05-31-LongShifts \
> > + 2003-07-06-IntOverflow \
> > + 2003-07-08-BitOpsTest \
> > + 2003-07-09-LoadShorts \
> > + 2003-07-09-SignedArgs \
> > + 2003-07-10-SignConversions \
> > + 2003-08-05-CastFPToUint \
> > + 2003-08-11-VaListArg \
> > + 2003-08-20-FoldBug \
> > + 2003-09-18-BitFieldTest \
> > + 2003-10-13-SwitchTest \
> > + 2003-10-29-ScalarReplBug \
> > + 2004-02-02-NegativeZero \
> > + 2004-06-20-StaticBitfieldInit \
> > + 2004-11-28-GlobalBoolLayout \
> > + 2005-05-11-Popcount-ffs-fls \
> > + 2005-05-12-Int64ToFP \
> > + 2005-05-13-SDivTwo \
> > + 2005-07-17-INT-To-FP \
> > + 2005-11-29-LongSwitch \
> > + 2006-01-29-SimpleIndirectCall \
> > + 2006-02-04-DivRem \
> > + 2006-12-01-float_varg \
> > + 2006-12-04-DynAllocAndRestore \
> > + 2006-12-07-Compare64BitConstant \
> > + 2006-12-11-LoadConstants \
> > + 2007-01-04-KNR-Args \
> > + 2007-03-02-VaCopy \
> > + 2007-04-25-weak \
> > + 2008-04-18-LoopBug \
> > + 2008-04-20-LoopBug2 \
> > + 2008-07-13-InlineSetjmp \
> > + 2009-04-16-BitfieldInitialization \
> > + 2009-12-07-StructReturn \
> > + 2010-05-24-BitfieldTest \
> > + AtomicOps \
> > + block-byref-cxxobj-test \
> > + block-byref-test \
> > + block-call-r7674133 \
> > + block-copied-in-cxxobj \
> > + block-copied-in-cxxobj-1 \
> > + blockstret \
> > + byval-alignment \
> > + conditional-gnu-ext \
> > + conditional-gnu-ext-cxx \
> > + DefaultInitDynArrays \
> > + FloatPrecision \
> > + initp1 \
> > + member-function-pointers \
> > + printargs \
> > + stmtexpr \
> > + StructModifyTest \
> > + TestLoop \
> > + vla \
> > + ms_struct-bitfield \
> > + ms_struct-bitfield-1 \
> > + ms_struct-bitfield-init \
> > + ms_struct-bitfield-init-1 \
> > + ms_struct_pack_layout \
> > + ms_struct_pack_layout-1
> > +endif
> > +
> > LDFLAGS += -lstdc++
> > include $(LEVEL)/SingleSource/Makefile.singlesrc
> >
> > Modified:
> > test-suite/trunk/SingleSource/UnitTests/SignlessTypes/Large/Makefile
> > URL:
> > http://llvm.org/viewvc/llvm-project/test-suite/trunk/SingleSource/UnitTests/SignlessTypes/Large/Makefile?rev=210251&r1=210250&r2=210251&view=diff
> > ==============================================================================
> > ---
> > test-suite/trunk/SingleSource/UnitTests/SignlessTypes/Large/Makefile
> > (original)
> > +++
> > test-suite/trunk/SingleSource/UnitTests/SignlessTypes/Large/Makefile
> > Thu Jun 5 04:24:05 2014
> > @@ -9,4 +9,8 @@ EXEC_XFAILS = cast
> > endif
> > endif
> >
> > +ifdef BENCHMARKING_ONLY
> > +PROGRAMS_TO_SKIP += cast
> > +endif
> > +
> > include $(LEVEL)/SingleSource/Makefile.singlesrc
> >
> > Modified:
> > test-suite/trunk/SingleSource/UnitTests/SignlessTypes/Makefile
> > URL:
> > http://llvm.org/viewvc/llvm-project/test-suite/trunk/SingleSource/UnitTests/SignlessTypes/Makefile?rev=210251&r1=210250&r2=210251&view=diff
> > ==============================================================================
> > --- test-suite/trunk/SingleSource/UnitTests/SignlessTypes/Makefile
> > (original)
> > +++ test-suite/trunk/SingleSource/UnitTests/SignlessTypes/Makefile
> > Thu Jun 5 04:24:05 2014
> > @@ -11,5 +11,9 @@ SEED := $(shell date +%j)
> > # the same results regardless of whether you make clean or not.
> > RUN_OPTIONS := 31415926
> >
> > +ifdef BENCHMARKING_ONLY
> > +PROGRAMS_TO_SKIP += cast2 cast-bug ccc div factor shr
> > +endif
> > +
> > LDFLAGS += -lstdc++
> > include $(LEVEL)/SingleSource/Makefile.singlesrc
> >
> > Modified: test-suite/trunk/SingleSource/UnitTests/Threads/Makefile
> > URL:
> > http://llvm.org/viewvc/llvm-project/test-suite/trunk/SingleSource/UnitTests/Threads/Makefile?rev=210251&r1=210250&r2=210251&view=diff
> > ==============================================================================
> > --- test-suite/trunk/SingleSource/UnitTests/Threads/Makefile
> > (original)
> > +++ test-suite/trunk/SingleSource/UnitTests/Threads/Makefile Thu
> > Jun 5 04:24:05 2014
> > @@ -15,4 +15,8 @@ ifeq ($(TARGET_OS),Darwin)
> > PROGRAMS_TO_SKIP := tls 2010-12-08-tls
> > endif
> >
> > +ifdef BENCHMARKING_ONLY
> > +PROGRAMS_TO_SKIP += 2010-12-08-tls tls
> > +endif
> > +
> > include $(LEVEL)/SingleSource/Makefile.singlesrc
> >
> > Modified:
> > test-suite/trunk/SingleSource/UnitTests/Vector/Altivec/Makefile
> > URL:
> > http://llvm.org/viewvc/llvm-project/test-suite/trunk/SingleSource/UnitTests/Vector/Altivec/Makefile?rev=210251&r1=210250&r2=210251&view=diff
> > ==============================================================================
> > --- test-suite/trunk/SingleSource/UnitTests/Vector/Altivec/Makefile
> > (original)
> > +++ test-suite/trunk/SingleSource/UnitTests/Vector/Altivec/Makefile
> > Thu Jun 5 04:24:05 2014
> > @@ -13,4 +13,10 @@ TARGET_FLAGS += -maltivec
> > TEST_TARGET_FLAGS += -maltivec
> > LCCFLAGS += -maltivec
> >
> > +ifdef BENCHMARKING_ONLY
> > +PROGRAMS_TO_SKIP += \
> > + 2007-01-07-lvsl-lvsr-Regression alti.sdot \
> > + casts test1
> > +endif
> > +
> > include $(LEVEL)/SingleSource/Makefile.singlesrc
> >
> > Modified: test-suite/trunk/SingleSource/UnitTests/Vector/Makefile
> > URL:
> > http://llvm.org/viewvc/llvm-project/test-suite/trunk/SingleSource/UnitTests/Vector/Makefile?rev=210251&r1=210250&r2=210251&view=diff
> > ==============================================================================
> > --- test-suite/trunk/SingleSource/UnitTests/Vector/Makefile
> > (original)
> > +++ test-suite/trunk/SingleSource/UnitTests/Vector/Makefile Thu Jun
> > 5 04:24:05 2014
> > @@ -37,6 +37,10 @@ RUNTIMELIMIT:=7200
> > endif
> > endif
> >
> > +ifdef BENCHMARKING_ONLY
> > +PROGRAMS_TO_SKIP += build divides sumarray sumarray-dbl
> > +endif
> > +
> > include $(LEVEL)/SingleSource/Makefile.singlesrc
> >
> >
> >
> >
> > _______________________________________________
> > llvm-commits mailing list
> > llvm-commits at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
>
--
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory
More information about the llvm-commits
mailing list