[PATCH][LNT] Add support for more detailed profiling to lnt.

[Sean Silva]

On Thu, Jun 25, 2015 at 8:18 AM, Daniel Dunbar <daniel at zuster.org> wrote:

> While people are putting feature bugs in your ear, one thing I have wanted
> LNT to do for a long time is to track the hash of the generated binary.
> Since the compiler is usually deterministic, the binary hashes change much
> less frequently than the compiler changes, and having access to that
> information makes some additional things possible (like easily detecting
> when something is just noise).
>
> The other advantage of tracking this is that, in theory, another server
> could be responsible for archiving all of the generated de-duplicated
> binaries. The de-dupped content shouldn't actually be that large, and this
> would be an easy thing to host on some cloud based provider that can
> accommodate the storage requirements. That server could then be responsible
> for having features like providing on-demand disassembly of a given
> executable.
>

Absolutely.

Recently in the thread "[LLVMdev] Proposal: change LNT’s regression
detection algorithm and how it is used to reduce false positives" I posted
some findings (a ways downthread). For the programs I looked at, the
binaries changed about once in every 1000 revisions (looking at a window of
10,000 revisions).

-- Sean Silva


>
>  - Daniel
>
>
> On Thu, Jun 25, 2015 at 8:12 AM, Kristof Beyls <kristof.beyls at arm.com>
> wrote:
>
>> Yes, absolutely!
>>
>>
>>
>> Right now, we’re slowly getting to the point where we’re seeing the
>> significant performance changes on http://llvm.org/perf.
>> I’m currently focussing mainly on improving the daily report page (
>> http://llvm.org/perf/db_default/v4/nts/daily_report) as that
>> seems the most useful report to get an overview of the significant
>> changes.
>>
>>
>>
>> But once that is working well, the next step is to make it easy to
>> understand what caused the significant performance change.
>>
>> Indeed, something visual that helps to intuitively understand the main
>> changes in the profile + a way to also see the corresponding
>> changes in assembly code.
>>
>>
>>
>> Measuring the profile + recording the assembly code for all programs at
>> every run will likely be prohibitively expensive – both
>> due to the performance tracking bots slowing down and the storage
>> capacity needed to store the information for all runs on the server.
>> But maybe the rerun-functionality could be extended so that for the
>> significant changes, the bot is asked to do extra profiling runs
>> of the programs with significant performance deltas and is asked to also
>> submit the assembly produced. That will need to be done
>> for both the current version of clang under test + the previous version.
>> How to make sure the bot caches a number of previous versions
>> (or rebuilds it) is probably not entirely trivial. We’ll also need to
>> make sure that the profiles and the assembly code also don’t
>> massively increase the storage needs on the server side. But all-in-all,
>> off the top of my head, this seems to be the most straightforward
>> way to get to a system like you describe below.
>>
>>
>>
>> So, yes, absolutely, being able to detect significant performance deltas
>> is not the end goal, but rather a system like you describe below.
>>
>>
>>
>> Kristof
>>
>>
>>
>> *From:* Sean Silva [mailto:chisophugis at gmail.com]
>> *Sent:* 25 June 2015 03:23
>> *To:* Kristof Beyls
>> *Cc:* llvm-commits; Tobias Grosser
>> *Subject:* Re: [PATCH][LNT] Add support for more detailed profiling to
>> lnt.
>>
>>
>>
>> It would be awesome if when we detect a performance change of one of the
>> benchmarks (not the LNT server), we did a similar analysis across the two
>> versions and presented a graph like this, but where the weight is the
>> performance diff of the function. Bonus points for making the graph nodes
>> clickable and showing an annotated assembly diff.
>>
>>
>>
>> -- Sean Silva
>>
>>
>>
>> On Wed, Jun 24, 2015 at 10:15 AM, Kristof Beyls <kristof.beyls at arm.com>
>> wrote:
>>
>> Hi,
>>
>>
>>
>> The attached patch adds support for getting call-graph profiles from lnt
>> runserver. It does this by adding
>> a new –profiler-dir command line option, which results in a pstat profile
>> to be stored to a separate file
>> in that sub directory, per web request. I find the extra call graph info
>> very
>> useful when trying to understand what the bottlenecks are in the
>> rendering of pages by the
>> lnt webserver.
>>
>>
>>
>> The pstat files can easily be converted into something graphical by a
>> command like:
>>
>> $ gprof2dot -f pstats profile_dir/POST.submitRun.000591ms.1435075964.prof
>> |  dot -Tsvg > submitRun.svg
>>
>>
>>
>> I’ve attached such an svg graph for a request of the daily report page to
>> give an idea of
>> the kind of information this more detailed profiling can give.
>>
>>
>>
>> Does the patch look OK to commit?
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Kristof
>>
>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20150625/55223dfd/attachment.html>