<html>
<head>
<base href="https://llvm.org/bugs/" />
</head>
<body><table border="1" cellspacing="0" cellpadding="8">
<tr>
<th>Bug ID</th>
<td><a class="bz_bug_link
bz_status_NEW "
title="NEW --- - Benchmark Quality: Short running tests"
href="https://llvm.org/bugs/show_bug.cgi?id=28801">28801</a>
</td>
</tr>
<tr>
<th>Summary</th>
<td>Benchmark Quality: Short running tests
</td>
</tr>
<tr>
<th>Product</th>
<td>Test Suite
</td>
</tr>
<tr>
<th>Version</th>
<td>trunk
</td>
</tr>
<tr>
<th>Hardware</th>
<td>PC
</td>
</tr>
<tr>
<th>OS</th>
<td>All
</td>
</tr>
<tr>
<th>Status</th>
<td>NEW
</td>
</tr>
<tr>
<th>Severity</th>
<td>normal
</td>
</tr>
<tr>
<th>Priority</th>
<td>P
</td>
</tr>
<tr>
<th>Component</th>
<td>Nightly Tester
</td>
</tr>
<tr>
<th>Assignee</th>
<td>unassignedbugs@nondot.org
</td>
</tr>
<tr>
<th>Reporter</th>
<td>matze@braunis.de
</td>
</tr>
<tr>
<th>CC</th>
<td>llvm-bugs@lists.llvm.org
</td>
</tr>
<tr>
<th>Classification</th>
<td>Unclassified
</td>
</tr></table>
<p>
<div>
<pre>We have a number of benchmarks in the test-suite that have too short run times,
this makes them a bad fit for benchmarking as operating system timers have a
limited granularity, warm/cold caches has a big effect on runtime, etc.
Ways to deal with these benchmarks:
- Do not use them for benchmarking but just as a correctness test (the
BENCHMARKING_ONLY flag allows that).
- Increase the size of the input data or the parameters with which the
benchmark is run
- Modify the sourcecode to allow the benchmark to be run multiple times
- Ideally we would put a mechanism in place that allows to scale the input
size/iteration count based on a fixed number. This way we can adapt the
benchmark to slower/faster systems and change the default scaling in the future
as machines become faster.</pre>
</div>
</p>
<hr>
<span>You are receiving this mail because:</span>
<ul>
<li>You are on the CC list for the bug.</li>
</ul>
</body>
</html>