Benchmarking some lld revisions

[Sean Silva via llvm-commits]

On Fri, May 26, 2017 at 4:03 PM, Rafael Espíndola <
rafael.espindola at gmail.com> wrote:

> Now that I have a workstation setup again for doing really stable
> benchmarks I decided to try a quick experiment and benchmark linking
> firefox with the last 1000 revisions or so.
>
> I used the last 1000 revision, regardless of what they changed. That
> way we can get an idea of the fluctuation to expect from measurement
> bias when unrelated things change.
>
> The results are attached. Data has 3 columns. The first one is the
> revision, the second one is the time and the third is the percentage
> std dev reported by perf over 10 runs.
>
> The plot was created with gnuplot:
>
> plot "data" using 1:2:($3 * $2 /100) title "time" with errorlines
>
> It is interesting to see that the variation from commit to commit is
> much bigger than the variation in a single commit,


This is a great observation. Good idea adding that into the visualization.
To be fair, you also have a very reproducible setup ;)


> but the graph
> allows us to find some significant commits. In this case, they were:
>
> r303689: The test has --buildit ... --build-id=none, and now the
> second one is used.
>
> r303925: Uses CachedHashStringRef again for comdat signatures.
>

Awesome, thanks for doing the measurements and the visualization/analysis!
Our performance seems very healthy.


>
> Cheers,
> Rafael
>
> P.S.: It would be truly awesome if someone could setup a bot that does
> this over various tests.
>

Scanning around internally here at Google a while back, it seems like we
have some infrastructure that should allow doing this fairly easily, modulo
a bit of yak shaving. IIRC, it required fiddling around with the internal
build system to fix some layering violations due to libLLDELF.a's elf::link
function being exported by some random header in include/ instead of a
header in ELF/ where the cc_library is defined (which would require some
CMake changes upstream too). It's on my todo list though to be honest it's
probably a "I'll hopefully get to it this year" type thing.
(the benchmarking infrastructure hooks into the build system and runs C++
functions so it would need to go through elf::link).

I'm not sure if the results of this infrastructure can be plugged into some
automatic external-facing report, but it would at least make it easy for
me, Rui, and any other interested googlers to periodically check in on it.
The granularity would be limited to the weekly (or whatever) LLVM import
though.

-- Sean Silva
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20170531/a4c28bfa/attachment.html>