[llvm-dev] RFC: LNT/Test-suite support for custom metrics and test parameterization

[Kristof Beyls via llvm-dev]

On 22 Apr 2016, at 11:14, Mehdi Amini <mehdi.amini at apple.com<mailto:mehdi.amini at apple.com>> wrote:


On Apr 22, 2016, at 12:45 AM, Kristof Beyls via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:


On 21 Apr 2016, at 17:44, Sergey Yakoushkin <sergey.yakoushkin at gmail.com<mailto:sergey.yakoushkin at gmail.com>> wrote:

Hi Kristof,

       The way we use LNT, we would run different configuration (e.g. -O3 vs -Os) as different "machines" in LNT's model.

O2/O3 is indeed bad example. We're also using different machines for Os/O3 - such parameters apply to all tests and we don't propose major changes.
Elena was only extending LNT interface a bit to ease LLVM-testsuite execution with different compiler or HW flags.

Oh I see, this boils down to extending the lnt runtest interface to be able to specify a set of configurations, rather than a single configuration and making
sure configurations get submitted under different machine names? We kick off the different configuration runs through a script invoking lnt runtest multiple
times. I don't see a big problem with extending the lnt runtest interface to do this, assuming it doesn't break the underlying concepts assumed throughout
LNT. Maybe the only downside is that this will add even more command line options to lnt runtest, which already has a lot (too many?) command line
options.

Maybe some changes are required to analyze and compare metrics between "machines": e.g. code size/performance between Os/O2/O3.
Do you perform such comparisons?

We typically do these kinds of comparisons when we test our patches pre-commit, i.e. comparing for example '-O3' with '-O3 'mllvm -enable-my-new-pass'.
To stick with the LNT concepts, tests enabling new passes are stored as a different "machine".
The only way I know to be able to do a comparison between runs on 2 different "machine"s is to manually edit the URL for run vs run comparison
and fill in the runids of the 2 runs you want to compare.
For example, the following URL is a comparison of green-dragon-07-x86_64-O3-flto vs green-dragon-06-x86_64-O0-g on the public llvm.org/perf<http://llvm.org/perf> server:
http://llvm.org/perf/db_default/v4/nts/70644?compare_to=70634
I had to manually look up and fill in the run ids 70644 and 70634.
It would be great if there was a better way to be able to do these kind of comparisons - i.e. not having to manually fill in run ids, but having a webui to easily find and pick the runs you want to compare.
(As an aside: I find it intriguing that the URL above suggests that there are quite a few cases where "-O0 -g" produces faster code than "-O3 -flto").

Can you be more explicit which ones? I don't see any regression (other than compared to the baseline, or on the compile time).

--
Mehdi

D'Oh! I was misinterpreting the compile time differences as execution time differences. Indeed, there is no unexpected result in there.
Sorry for the noise!

Kristof

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160422/350248e0/attachment.html>