[llvm-dev] New LLD performance builder

Thu Feb 22 13:56:48 PST 2018

Thanks a lot for setting this up!

By using the "mean as aggregation" option one can see the noise in the
results better:

http://lnt.llvm.org/db_default/v4/link/graph?switch_min_mean=yes&moving_window_size=10&plot.9=1.9.7&submit=Update

There are a few benchmarknig tips in https://www.llvm.org/docs/Benchmarking.html.

For example, from looking at

http://lab.llvm.org:8011/builders/lld-perf-testsuite/builds/285/steps/cmake-configure/logs/stdio

It seems the produced lld binary is not being statically linked.

A tip to make the bot a bit faster is that it could run "ninja bin/lld"
instead of just "ninja":

http://lab.llvm.org:8011/builders/lld-perf-testsuite/builds/285/steps/build-unified-tree/logs/stdio

Is lld-speed-test in a tmpfs?

Is lld-benchmark.py a copy of lld/utils/benchmark.py?

Thanks,
Rafael

Galina Kistanova via llvm-dev <llvm-dev at lists.llvm.org> writes:

> Hello George,
>
> Sorry, somehow hit a send button too soon. Please ignore the previous
> e-mail.
>
> The bot does 10 runs for each of the benchmarks (those dots in the logs are
> meaningful). We can increase the number of runs if proven that this would
> significantly increase the accuracy. I didn't see the increase in accuracy when
> have been staging the bot, which would justify the extra time and larger
> gaps between the tested revisions.  10 runs seems give a good balance. But
> I'm open for suggestions.
>
> It seems the statistics are quite stable if you would look over number of
> revisions.
> And in this particular case the picture seems quite clear.
>
> At http://lnt.llvm.org/db_default/v4/link/104, the list of Performance
> Regressions suggests that the most hit was with the linux-kernel. The
> regressed metrics - branches, branch-misses, instructions,
> cycles, seconds-elapsed, task-clock. Some other benchmarks shows
> regressions in branches and branch-misses, some shows improvements.
>
> The metrics are consistent before and after the commit, so, I do not think
> this one is an outliner.
> For example, if one would take a look at the linux-kernel branches -
> http://lnt.llvm.org/db_default/v4/link/graph?plot.0=1.12.2&highlight_run=104,
> it gets obvious that the number of branches increased significantly as a
> result of the r325313. The metric is very stable around the impacted commit
> and does not go down after. The  branch-misses is more volatile, but still
> consistently shows the regression as the result of this commit.
>
> Now someone should see why this particular commit has resulted in
> significant increase of branching with the Linux Kernel.
>
> As of how to use LNT web UI, I'm sure you have checked that, but, just in
> case, here is the link to the LNT doc - http://llvm.org/docs/lnt/con
> tents.html.
>
>> task-clock results are available for "linux-kernel" and "llvm-as-fsds"
> only and all other
>> tests has blank field. Should it mean there was no noticable difference
> in results ?
>
> If you would go to http://lnt.llvm.org/db_default/v4/link/104#task-clock
> (or go to  http://lnt.llvm.org/db_default/v4/link/104 and select the task-clock
> on the left, which is the same), you would see the list of actual values in
> the "Current" column. All of them populated, none is blank. The column "%"
> contains the difference from the previous run in percents, or dash for no
> measured difference.
>
>>  Also, "Graph" and "Matrix" buttons whatever they should do show errors
> atm.
>
> I guess you didn't select what to graph or what to show as a matrix, did
> you?
>
> Besides reporting to the lnt.llvm.org, each build contains in the log all
> the reported data, so you could process it whatever you want and find
> helpful.
>
> Hope this helps.
>
> Thanks
>
> Galina
>
>
> On Fri, Feb 16, 2018 at 1:55 AM, George Rimar via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> >Hello everyone,
>> >
>> >I have added a new public LLD performance builder at
>> >http://lab.llvm.org:8011/builders/lld-perf-testsuite.
>> >It builds LLVM and LLD by the latest releaed Clang and runs a set of
>> >perfromance tests.
>> >
>> >The builder is reliable. Please pay attention on the failures.
>> >
>> >The performance statistics are here:
>> >http://lnt.llvm.org/db_default/v4/link/recent_activity
>> >
>> >Thanks
>> >
>> >Galina
>>
>> Great news, thanks !
>>
>> Looking on results I am not sure how to explain them though.
>>
>> For example r325313 fixes "use after free", it should not give any
>> performance
>> slowdowns or boosts. Though if I read results right, they show 23.65%
>> slowdown
>> for time of linking linux kernel (http://lnt.llvm.org/db_
>> default/v4/link/104).
>>
>> I guess such variation can happen for example if bot do only single link
>> iteration for tests,
>> so that final time is just a error mostly probably.
>>
>> task-clock results are available for "linux-kernel" and "llvm-as-fsds"
>> only and all other
>> tests has blank field. Should it mean there was no noticable difference in
>> results ?
>>
>> Also, "Graph" and "Matrix" buttons whatever they should do show errors atm.
>> ("Nothing to graph." and "Not Found: Request requires some data
>> arguments.").
>>
>> Best regards,
>> George | Developer | Access Softek, Inc
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev