<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 21 Apr 2016, at 17:44, Sergey Yakoushkin <<a href="mailto:sergey.yakoushkin@gmail.com" class="">sergey.yakoushkin@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">Hi Kristof,
<div class=""><br class="">
</div>
<div class=""><font color="#500050" face="Calibri, sans-serif" class=""><span style="font-size:14.6667px" class=""> The way we use LNT, we would run different configuration (e.g. -O3 vs -Os) as different "machines" in LNT's model.</span></font></div>
<div class=""><br class="">
</div>
<div class="">O2/O3 is indeed bad example. We're also using different machines for Os/O3 - such parameters apply to all tests and we don't propose major changes.</div>
<div class="">
<div class="">Elena was only extending LNT interface a bit to ease LLVM-testsuite execution with different compiler or HW flags.<br class="">
</div>
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>Oh I see, this boils down to extending the lnt runtest interface to be able to specify a set of configurations, rather than a single configuration and making</div>
<div>sure configurations get submitted under different machine names? We kick off the different configuration runs through a script invoking lnt runtest multiple</div>
<div>times. I don't see a big problem with extending the lnt runtest interface to do this, assuming it doesn't break the underlying concepts assumed throughout</div>
<div>LNT. Maybe the only downside is that this will add even more command line options to lnt runtest, which already has a lot (too many?) command line</div>
<div>options.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div class=""></div>
<div class="">Maybe some changes are required to analyze and compare metrics between "machines": e.g. code size/performance between Os/O2/O3.</div>
<div class="">Do you perform such comparisons?</div>
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>We typically do these kinds of comparisons when we test our patches pre-commit, i.e. comparing for example '-O3' with '-O3 'mllvm -enable-my-new-pass'.</div>
<div>To stick with the LNT concepts, tests enabling new passes are stored as a different "machine".</div>
<div>The only way I know to be able to do a comparison between runs on 2 different "machine"s is to manually edit the URL for run vs run comparison</div>
<div>and fill in the runids of the 2 runs you want to compare.</div>
<div>For example, the following URL is a comparison of green-dragon-07-x86_64-O3-flto vs green-dragon-06-x86_64-O0-g on the public
<a href="http://llvm.org/perf" class="">llvm.org/perf</a> server:</div>
<div><a href="http://llvm.org/perf/db_default/v4/nts/70644?compare_to=70634" class="">http://llvm.org/perf/db_default/v4/nts/70644?compare_to=70634</a></div>
<div>I had to manually look up and fill in the run ids 70644 and 70634.</div>
<div>It would be great if there was a better way to be able to do these kind of comparisons - i.e. not having to manually fill in run ids, but having a webui to easily find and pick the runs you want to compare.</div>
<div>(As an aside: I find it intriguing that the URL above suggests that there are quite a few cases where "-O0 -g" produces faster code than "-O3 -flto").</div>
<div><br class="">
</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">"test parameters" are different, they allow exploring multiple variants of the same test case. E.g. can be:<br class="">
</div>
<div class="">* index of input data sets, length of input vector, size of matrix, etc;<br class="">
</div>
<div class="">* macro that affect source code such as changing 1) static data allocation to dynamic or 2) constants to variables (compile-time unknown)<br class="">
</div>
<div class="">* extra sets of internal compilation options that are relevant only for particular test case<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Same parameters can apply to multiple tests with different value sets:</div>
<div class=""><br class="">
</div>
<div class="">test1: param1={v1,v2,v3}<br class="">
</div>
<div class="">test2: param1={v2,v4}<br class="">
</div>
<div class="">test3:</div>
<div class=""><br class="">
</div>
<div class="">Of course, original test cases can be duplicated (copied under different names) - that is enough to execute tests.</div>
<div class="">Explicit "test parameters" allow exploring dependencies between test parameters and metrics.</div>
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>Right. In the new cmake+lit way of driving the test-suite, some of these test parameters are input to cmake (like macros) and others will be input to lit (like changing inputs), I think.</div>
<div>We see this also in e.g. running SPEC with ref vs train vs test data sets. TBH, I'm not quite sure how to best drive this. I guess Mathhias may have better ideas than me here.</div>
<div>I do think that to comply with LNT's current conceptual model, tests being run with different parameters will have to have different test names in the LNT view.</div>
<div><br class="">
</div>
<div>Thanks,</div>
<div><br class="">
</div>
<div>Kristof</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div class=""><br class="">
</div>
</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Thu, Apr 21, 2016 at 4:36 PM, Kristof Beyls via llvm-dev
<span dir="ltr" class=""><<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class=""><br class="">
<div class=""><span class="">
<blockquote type="cite" class="">
<div class="">On 21 Apr 2016, at 15:00, Elena Lepilkina <<a href="mailto:Elena.Lepilkina@synopsys.com" target="_blank" class="">Elena.Lepilkina@synopsys.com</a>> wrote:</div>
<br class="">
<div class="">
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="">
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif" class="">
<span style="font-size:11pt;font-family:Calibri,sans-serif" class="">Hi Kristof and Daniel,<u class=""></u><u class=""></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif" class="">
<span style="font-size:11pt;font-family:Calibri,sans-serif" class=""> </span></div>
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif" class="">
<span style="font-size:11pt;font-family:Calibri,sans-serif" class="">Thanks for your answers.<u class=""></u><u class=""></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif" class="">
<span style="font-size:11pt;font-family:Calibri,sans-serif" class=""> </span></div>
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif" class="">
<span style="font-size:11pt;font-family:Calibri,sans-serif" class="">Unfortunately I haven’t tried scaling up to a large data set before. Today I tried and results are quite bad.<u class=""></u><u class=""></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif" class="">
<span style="font-size:11pt;font-family:Calibri,sans-serif" class="">So database scheme should be rebuild. Now I thought about creating one sample table for each test-suite, but not cloning all tables in packet. As I see other tables can be same for all testsuites.
I mean if user run tests of new testsuite, new sample table would be created during importing data from json, if it doesn’t exist. Are there some problems with this solution? May be, I don’t know some details.</span></div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
</span>
<div class="">It's unfortunate to see performance doesn't scale with the proposed initial schema, but not entirely surprising. I don't really have much feedback on how the schema could be adapted otherwise as I haven't worked much on that. I hope Daniel will
have more insights to share here.</div>
<span class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="">
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif" class="">
<span style="font-size:11pt;font-family:Calibri,sans-serif" class=""><u class=""></u><u class=""></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif" class="">
<span style="font-size:11pt;font-family:Calibri,sans-serif" class="">Moreover, I have question about compile tests. Are compile tests runnable? In<span class=""> </span><a href="http://llvm.org/perf" style="color:purple;text-decoration:underline" target="_blank" class="">http://llvm.org/perf</a><span class=""> </span>there
is no compile test. Does that mean that they are deprecated for now?<u class=""></u><u class=""></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif" class="">
<span style="font-size:11pt;font-family:Calibri,sans-serif" class=""> </span></div>
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif" class="">
<span style="font-size:11pt;font-family:Calibri,sans-serif" class="">About test parameters, for example, we would like to have opportunity to compare benchmark results of test compiled with -O3 and -Os in context of one run.</span></div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
</span>
<div class="">The way we use LNT, we would run different configuration (e.g. -O3 vs -Os) as different "machines" in LNT's model. This is also explained in LNT's documentation, see</div>
</div>
<a href="https://github.com/llvm-mirror/lnt/blob/master/docs/concepts.rst" target="_blank" class="">https://github.com/llvm-mirror/lnt/blob/master/docs/concepts.rst</a>. Unfortunately, this version of the documentation hasn't found it's way yet to <a href="http://llvm.org/docs/lnt/contents.html" target="_blank" class="">http://llvm.org/docs/lnt/contents.html</a>.
<div class="">Is there a reason why storing different configurations as different "machines" in the LNT model doesn't work for you?</div>
<div class="">I assume that there are a number of places in LNT's analyses that assume that different runs coming from the same "machine" are always produced by the same configuration. But I'm not entirely sure about that.</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class=""><br class="">
</div>
<div class="">Kristof</div>
</div>
<br class="">
_______________________________________________<br class="">
LLVM Developers mailing list<br class="">
<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a><br class="">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="">
<br class="">
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>