<html>
<head>
<base href="https://llvm.org/bugs/" />
</head>
<body><table border="1" cellspacing="0" cellpadding="8">
<tr>
<th>Bug ID</th>
<td><a class="bz_bug_link
bz_status_NEW "
title="NEW --- - [libcxx] Add C++ library benchmarks"
href="https://llvm.org/bugs/show_bug.cgi?id=31084">31084</a>
</td>
</tr>
<tr>
<th>Summary</th>
<td>[libcxx] Add C++ library benchmarks
</td>
</tr>
<tr>
<th>Product</th>
<td>Test Suite
</td>
</tr>
<tr>
<th>Version</th>
<td>trunk
</td>
</tr>
<tr>
<th>Hardware</th>
<td>PC
</td>
</tr>
<tr>
<th>OS</th>
<td>Linux
</td>
</tr>
<tr>
<th>Status</th>
<td>NEW
</td>
</tr>
<tr>
<th>Severity</th>
<td>normal
</td>
</tr>
<tr>
<th>Priority</th>
<td>P
</td>
</tr>
<tr>
<th>Component</th>
<td>Programs Tests
</td>
</tr>
<tr>
<th>Assignee</th>
<td>unassignedbugs@nondot.org
</td>
</tr>
<tr>
<th>Reporter</th>
<td>renato.golin@linaro.org
</td>
</tr>
<tr>
<th>CC</th>
<td>llvm-bugs@lists.llvm.org
</td>
</tr>
<tr>
<th>Classification</th>
<td>Unclassified
</td>
</tr></table>
<p>
<div>
<pre>We have plenty of C++ library tests in the libcxx repo, but no benchmarks at
all. During the "Libcxx performance" BoF at the US LLVM 2016, we discussed that
adding such a benchmark to the test-suite would be a great idea. However, I
have no idea how to even start.
First, there were some talks about integrating the Google Benchmark [1] into
the test-suite and (maybe) convert the current benchmarks to use it. Regardless
of a potential move, since we're creating a whole new benchmark, I'd assume
using a tested tool is better than using gettimeofday()...
Second, what kinds of tests do we want? I can think of two approaches:
1. We test the "number of iterations" that algorithms and containers perform on
small to medium datasets and make sure they're scaling as documented. I'm not
sure how to do that other than instrumenting libcxx and only enabling the
instrumentation in the benchmark. Other users can enable this on their own
programs, for debug?
2. We test the actual "wall time" on a single core (threading is a separate
problem), and make sure that not only we don't regress from past runs, but that
the actual time spent *also* scales with the guarantees.
What we cannot do is to compare with other standard libraries, unless we want
to include their sources (and build) in the test-suite. Relying on whatever is
in the system will never work across the board.
Finally, the "complicated set". Threading, atomics and localisation tend to be
hard to test and heavily dependent on the architecture and OS. We already have
tests for that, but I fear the run time of those tests, if measured, will have
a large deviation, even on the same architecture/OS combination.
Sub-architecture and OS configureations will play a heavy role on them.
So, how do we start? Who want's to help?
cheers,
--renato
[1] <a href="https://github.com/google/benchmark">https://github.com/google/benchmark</a></pre>
</div>
</p>
<hr>
<span>You are receiving this mail because:</span>
<ul>
<li>You are on the CC list for the bug.</li>
</ul>
</body>
</html>