[llvm-bugs] [Bug 28801] New: Benchmark Quality: Short running tests

[via llvm-bugs]

https://llvm.org/bugs/show_bug.cgi?id=28801

            Bug ID: 28801
           Summary: Benchmark Quality: Short running tests
           Product: Test Suite
           Version: trunk
          Hardware: PC
                OS: All
            Status: NEW
          Severity: normal
          Priority: P
         Component: Nightly Tester
          Assignee: unassignedbugs at nondot.org
          Reporter: matze at braunis.de
                CC: llvm-bugs at lists.llvm.org
    Classification: Unclassified

We have a number of benchmarks in the test-suite that have too short run times,
this makes them a bad fit for benchmarking as operating system timers have a
limited granularity, warm/cold caches has a big effect on runtime, etc.

Ways to deal with these benchmarks:
- Do not use them for benchmarking but just as a correctness test (the
BENCHMARKING_ONLY flag allows that).
- Increase the size of the input data or the parameters with which the
benchmark is run
- Modify the sourcecode to allow the benchmark to be run multiple times
- Ideally we would put a mechanism in place that allows to scale the input
size/iteration count based on a fixed number. This way we can adapt the
benchmark to slower/faster systems and change the default scaling in the future
as machines become faster.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20160801/775ac559/attachment-0001.html>