[PATCH][TEST-SUITE] Marking more programs as not to be run in benchmark-only mode

[Kristof Beyls]

Thanks, committed as r237777.

> -----Original Message-----
> From: Chris Matthews [mailto:chris.matthews at apple.com]
> Sent: 19 May 2015 18:33
> To: Kristof Beyls
> Cc: Hal Finkel; Tobias Grosser; llvm-commits
> Subject: Re: [PATCH][TEST-SUITE] Marking more programs as not to be run
> in benchmark-only mode
> 
> I think this is great! I totally agree.
> 
> > On May 19, 2015, at 10:28 AM, Kristof Beyls <kristof.beyls at arm.com>
> wrote:
> >
> > Hi,
> >
> > One substantial source of performance noisiness of the LNT test-suite
> > are programs running very shortly.
> > The attached patch disables more of the programs in benchmark-only
> > mode, based on analysis that I've done on the programs that run less
> > than 10ms on http://llvm.org/perf/db_default/v4/nts/machine/39.
> >
> > I think it is debatable whether some of these should remain to be run
> > in benchmark-only mode, as we also have code in LNT to just ignore all
> > benchmarks running less than
> > 10 ms.
> >
> > Here is the list of programs that are removed in benchmark-only mode
> > by the attached patch.
> >
> > The programs that clearly don't have value as a benchmark:
> > * SingleSource/UnitTests/Vector: constpool simple:
> >  both don't have any loops in the code.
> > * SingleSource/UnitTests/Vector/AArch64: aarch64_neon_intrinsics:
> >  doesn't have any loop in the code.
> > * SingleSource/UnitTests/Vector/NEON: simple:
> >  doesn't have any loop in the code.
> > * SingleSource/UnitTests: 2005-07-15-Bitfield-ABI 2006-01-23-UnionInit
> > 2007-04-10-BitfieldTest:
> >  doesn't have any loop in the code.
> > * MultiSource/Benchmarks/Prolangs-C: loader:
> >  This program exits immediately because no arguments are given on the
> > command line. Unless  someone creates inputs for this program, this
> > should not be considered a benchmark.
> >
> >
> > The programs for which it's debatable whether they have value or not.
> > I think none of these do enough work to be considered as a benchmark:
> > * MultiSource/Benchmarks/McCat/:
> >  - 15-trie produces trie data structures, but the program has no loops
> > - so it's probably IO
> >    bound, and therefore shouldn't be considered as a benchmark.
> > * MultiSource/Benchmarks/Prolangs-C:
> >  - cdecl: The benchmark parses about 70 C declarations
> > * MultiSource/Benchmarks/MiBench:
> >  - office-stringsearch searches a few hundred substrings in a set of a
> > few hundred strings.
> >  - telecom-adpcm seems to spend most of its time in IO - and should
> > have a loop to do the main data
> >    transformation multiple times on the same buffer if run as a
> benchmark.
> > * SingleSource/Benchmarks/Stanford:
> >  - IntMM: does 10 matrix multiplications of size 40x40.
> > * SingleSource/Regression/C/Makefile: matrixTranspose,  sumarray2d,
> > test_indvars:
> >  - matrixTranspose: transposes a 32x32 matrix 10 times.
> >  - sumarray2d: creates a 100x100 matrix and sums all the elements in
> it.
> >  - test_indvars: traverses a 20000-element matrix twice.
> > * SingleSource/UnitTests/SignlessTypes:
> >  - rem: does 100 gcd computations.
> >
> > The following programs also run very shortly, but I do think they have
> > value as a benchmark - so the attached patch doesn't disable them in
> > benchmark
> > mode:
> > * SingleSource/Benchmarks/Misc/lowercase
> > * SingleSource/Benchmarks/Shootout/objinst
> > * SingleSource/Benchmarks/Shootout-C++/objinst
> > For all three, it seems that the main computation in the benchmark is
> > completely optimized away. Keeping these running as benchmarks should
> > allow us to catch if llvm ever regresses in being able to optimize
> > away the main computation in these programs.
> >
> >
> > What do you think?
> >
> > Thanks,
> >
> > Kristof
> > <exclude_short_running_programs_from_benchmark_mode.diff>