[cfe-dev] [LLVMdev] Performance Tracking

[Chandler Carruth]

On Wed, Nov 16, 2011 at 11:53 AM, David Chisnall <csdavec at swan.ac.uk> wrote:

> On 16 Nov 2011, at 19:45, Matthieu Monrocq wrote:
>
> > Many thanks David, it had been a while (6 months I guess) since the last
> benchmark I saw and I was wondering how the new Clang/LLVM compared to GCC!
> >
> > One comment though, the graphs are great, however the alternance of
> "less is better"/"more is better" makes for a difficult read: it's not
> obvious at a glance which is performing better and it's difficult to get a
> quick overview surveying the few graphs available.
>
> To clarify - I didn't create these benchmarks and am not affiliated in any
> way with the site that did, someone sent me the link and asked me if I knew
> what accounted for the differences between the three compilers tested.
>
> After looking at them, I see that there are some improvements and some
> regressions between 2.9 and 3.0.  I am interested in us setting up
> something that ensures that 3.1 contains only improvements and not
> regressions.
>
> Running benchmarks like these on (at least some of) the buildbots and
> sending mails to people for any commit that resulted in a slowdown would be
> a good start.  I believe most other compilers do something along these
> lines...


I completely agree, and I think Evan has described the right approach. Look
at the way these benchmarks run, and port them to the LLVM test suite.
There are bots that run nightly and dashboards that track regressions.

However, I can only spot two regressions:

FLAC encoding regresses by maybe 2% -- is that actually within the noise?

John The Ripper: Blowfish regresses by over 5%; that one actually looks
interesting.

The rest seem to have improved, or to have some error in running...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20111116/a2215bb1/attachment.html>