[PATCH][LNT] Add support for more detailed profiling to lnt.

[Daniel Dunbar]

While people are putting feature bugs in your ear, one thing I have wanted
LNT to do for a long time is to track the hash of the generated binary.
Since the compiler is usually deterministic, the binary hashes change much
less frequently than the compiler changes, and having access to that
information makes some additional things possible (like easily detecting
when something is just noise).

The other advantage of tracking this is that, in theory, another server
could be responsible for archiving all of the generated de-duplicated
binaries. The de-dupped content shouldn't actually be that large, and this
would be an easy thing to host on some cloud based provider that can
accommodate the storage requirements. That server could then be responsible
for having features like providing on-demand disassembly of a given
executable.

 - Daniel


On Thu, Jun 25, 2015 at 8:12 AM, Kristof Beyls <kristof.beyls at arm.com>
wrote:

> Yes, absolutely!
>
>
>
> Right now, we’re slowly getting to the point where we’re seeing the
> significant performance changes on http://llvm.org/perf.
> I’m currently focussing mainly on improving the daily report page (
> http://llvm.org/perf/db_default/v4/nts/daily_report) as that
> seems the most useful report to get an overview of the significant changes.
>
>
>
> But once that is working well, the next step is to make it easy to
> understand what caused the significant performance change.
>
> Indeed, something visual that helps to intuitively understand the main
> changes in the profile + a way to also see the corresponding
> changes in assembly code.
>
>
>
> Measuring the profile + recording the assembly code for all programs at
> every run will likely be prohibitively expensive – both
> due to the performance tracking bots slowing down and the storage capacity
> needed to store the information for all runs on the server.
> But maybe the rerun-functionality could be extended so that for the
> significant changes, the bot is asked to do extra profiling runs
> of the programs with significant performance deltas and is asked to also
> submit the assembly produced. That will need to be done
> for both the current version of clang under test + the previous version.
> How to make sure the bot caches a number of previous versions
> (or rebuilds it) is probably not entirely trivial. We’ll also need to make
> sure that the profiles and the assembly code also don’t
> massively increase the storage needs on the server side. But all-in-all,
> off the top of my head, this seems to be the most straightforward
> way to get to a system like you describe below.
>
>
>
> So, yes, absolutely, being able to detect significant performance deltas
> is not the end goal, but rather a system like you describe below.
>
>
>
> Kristof
>
>
>
> *From:* Sean Silva [mailto:chisophugis at gmail.com]
> *Sent:* 25 June 2015 03:23
> *To:* Kristof Beyls
> *Cc:* llvm-commits; Tobias Grosser
> *Subject:* Re: [PATCH][LNT] Add support for more detailed profiling to
> lnt.
>
>
>
> It would be awesome if when we detect a performance change of one of the
> benchmarks (not the LNT server), we did a similar analysis across the two
> versions and presented a graph like this, but where the weight is the
> performance diff of the function. Bonus points for making the graph nodes
> clickable and showing an annotated assembly diff.
>
>
>
> -- Sean Silva
>
>
>
> On Wed, Jun 24, 2015 at 10:15 AM, Kristof Beyls <kristof.beyls at arm.com>
> wrote:
>
> Hi,
>
>
>
> The attached patch adds support for getting call-graph profiles from lnt
> runserver. It does this by adding
> a new –profiler-dir command line option, which results in a pstat profile
> to be stored to a separate file
> in that sub directory, per web request. I find the extra call graph info
> very
> useful when trying to understand what the bottlenecks are in the rendering
> of pages by the
> lnt webserver.
>
>
>
> The pstat files can easily be converted into something graphical by a
> command like:
>
> $ gprof2dot -f pstats profile_dir/POST.submitRun.000591ms.1435075964.prof
> |  dot -Tsvg > submitRun.svg
>
>
>
> I’ve attached such an svg graph for a request of the daily report page to
> give an idea of
> the kind of information this more detailed profiling can give.
>
>
>
> Does the patch look OK to commit?
>
>
>
> Thanks,
>
>
>
> Kristof
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20150625/6c5522e2/attachment.html>