<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Apr 22, 2016, at 12:45 AM, Kristof Beyls via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" class=""><div class=""><br class="Apple-interchange-newline">On 21 Apr 2016, at 17:44, Sergey Yakoushkin <<a href="mailto:sergey.yakoushkin@gmail.com" class="">sergey.yakoushkin@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hi Kristof,<div class=""><br class=""></div><div class=""><font color="#500050" face="Calibri, sans-serif" class=""><span class="" style="font-size: 14.6667px;"> The way we use LNT, we would run different configuration (e.g. -O3 vs -Os) as different "machines" in LNT's model.</span></font></div><div class=""><br class=""></div><div class="">O2/O3 is indeed bad example. We're also using different machines for Os/O3 - such parameters apply to all tests and we don't propose major changes.</div><div class=""><div class="">Elena was only extending LNT interface a bit to ease LLVM-testsuite execution with different compiler or HW flags.<br class=""></div></div></div></div></blockquote><div class=""><br class=""></div><div class="">Oh I see, this boils down to extending the lnt runtest interface to be able to specify a set of configurations, rather than a single configuration and making</div><div class="">sure configurations get submitted under different machine names? We kick off the different configuration runs through a script invoking lnt runtest multiple</div><div class="">times. I don't see a big problem with extending the lnt runtest interface to do this, assuming it doesn't break the underlying concepts assumed throughout</div><div class="">LNT. Maybe the only downside is that this will add even more command line options to lnt runtest, which already has a lot (too many?) command line</div><div class="">options.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><div class=""></div><div class="">Maybe some changes are required to analyze and compare metrics between "machines": e.g. code size/performance between Os/O2/O3.</div><div class="">Do you perform such comparisons?</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">We typically do these kinds of comparisons when we test our patches pre-commit, i.e. comparing for example '-O3' with '-O3 'mllvm -enable-my-new-pass'.</div><div class="">To stick with the LNT concepts, tests enabling new passes are stored as a different "machine".</div><div class="">The only way I know to be able to do a comparison between runs on 2 different "machine"s is to manually edit the URL for run vs run comparison</div><div class="">and fill in the runids of the 2 runs you want to compare.</div><div class="">For example, the following URL is a comparison of green-dragon-07-x86_64-O3-flto vs green-dragon-06-x86_64-O0-g on the public<span class="Apple-converted-space"> </span><a href="http://llvm.org/perf" class="">llvm.org/perf</a> server:</div><div class=""><a href="http://llvm.org/perf/db_default/v4/nts/70644?compare_to=70634" class="">http://llvm.org/perf/db_default/v4/nts/70644?compare_to=70634</a></div><div class="">I had to manually look up and fill in the run ids 70644 and 70634.</div><div class="">It would be great if there was a better way to be able to do these kind of comparisons - i.e. not having to manually fill in run ids, but having a webui to easily find and pick the runs you want to compare.</div><div class="">(As an aside: I find it intriguing that the URL above suggests that there are quite a few cases where "-O0 -g" produces faster code than "-O3 -flto").</div></div></div></blockquote><div><br class=""></div><div>Can you be more explicit which ones? I don't see any regression (other than compared to the baseline, or on the compile time).</div><div><br class=""></div><div>-- </div><div>Mehdi</div><div><br class=""></div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class=""><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class="">"test parameters" are different, they allow exploring multiple variants of the same test case. E.g. can be:<br class=""></div><div class="">* index of input data sets, length of input vector, size of matrix, etc;<br class=""></div><div class="">* macro that affect source code such as changing 1) static data allocation to dynamic or 2) constants to variables (compile-time unknown)<br class=""></div><div class="">* extra sets of internal compilation options that are relevant only for particular test case<br class=""></div><div class=""><br class=""></div><div class="">Same parameters can apply to multiple tests with different value sets:</div><div class=""><br class=""></div><div class="">test1: param1={v1,v2,v3}<br class=""></div><div class="">test2: param1={v2,v4}<br class=""></div><div class="">test3:</div><div class=""><br class=""></div><div class="">Of course, original test cases can be duplicated (copied under different names) - that is enough to execute tests.</div><div class="">Explicit "test parameters" allow exploring dependencies between test parameters and metrics.</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">Right. In the new cmake+lit way of driving the test-suite, some of these test parameters are input to cmake (like macros) and others will be input to lit (like changing inputs), I think.</div><div class="">We see this also in e.g. running SPEC with ref vs train vs test data sets. TBH, I'm not quite sure how to best drive this. I guess Mathhias may have better ideas than me here.</div><div class="">I do think that to comply with LNT's current conceptual model, tests being run with different parameters will have to have different test names in the LNT view.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class=""><br class=""></div><div class="">Kristof</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><div class=""><br class=""></div></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Thu, Apr 21, 2016 at 4:36 PM, Kristof Beyls via llvm-dev<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div class="" style="word-wrap: break-word;"><br class=""><div class=""><span class=""><blockquote type="cite" class=""><div class="">On 21 Apr 2016, at 15:00, Elena Lepilkina <<a href="mailto:Elena.Lepilkina@synopsys.com" target="_blank" class="">Elena.Lepilkina@synopsys.com</a>> wrote:</div><br class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-position: normal; font-variant-caps: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div class="" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span class="" style="font-size: 11pt; font-family: Calibri, sans-serif;">Hi Kristof and Daniel,<u class=""></u><u class=""></u></span></div><div class="" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span class="" style="font-size: 11pt; font-family: Calibri, sans-serif;"> </span></div><div class="" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span class="" style="font-size: 11pt; font-family: Calibri, sans-serif;">Thanks for your answers.<u class=""></u><u class=""></u></span></div><div class="" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span class="" style="font-size: 11pt; font-family: Calibri, sans-serif;"> </span></div><div class="" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span class="" style="font-size: 11pt; font-family: Calibri, sans-serif;">Unfortunately I haven’t tried scaling up to a large data set before. Today I tried and results are quite bad.<u class=""></u><u class=""></u></span></div><div class="" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span class="" style="font-size: 11pt; font-family: Calibri, sans-serif;">So database scheme should be rebuild. Now I thought about creating one sample table for each test-suite, but not cloning all tables in packet. As I see other tables can be same for all testsuites. I mean if user run tests of new testsuite, new sample table would be created during importing data from json, if it doesn’t exist. Are there some problems with this solution? May be, I don’t know some details.</span></div></div></div></blockquote><div class=""><br class=""></div></span><div class="">It's unfortunate to see performance doesn't scale with the proposed initial schema, but not entirely surprising. I don't really have much feedback on how the schema could be adapted otherwise as I haven't worked much on that. I hope Daniel will have more insights to share here.</div><span class=""><br class=""><blockquote type="cite" class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-position: normal; font-variant-caps: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div class="" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span class="" style="font-size: 11pt; font-family: Calibri, sans-serif;"><u class=""></u><u class=""></u></span></div><div class="" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span class="" style="font-size: 11pt; font-family: Calibri, sans-serif;">Moreover, I have question about compile tests. Are compile tests runnable? In<span class=""> </span><a href="http://llvm.org/perf" target="_blank" class="" style="color: purple; text-decoration: underline;">http://llvm.org/perf</a><span class=""> </span>there is no compile test. Does that mean that they are deprecated for now?<u class=""></u><u class=""></u></span></div><div class="" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span class="" style="font-size: 11pt; font-family: Calibri, sans-serif;"> </span></div><div class="" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span class="" style="font-size: 11pt; font-family: Calibri, sans-serif;">About test parameters, for example, we would like to have opportunity to compare benchmark results of test compiled with -O3 and -Os in context of one run.</span></div></div></div></blockquote><div class=""><br class=""></div></span><div class="">The way we use LNT, we would run different configuration (e.g. -O3 vs -Os) as different "machines" in LNT's model. This is also explained in LNT's documentation, see</div></div><a href="https://github.com/llvm-mirror/lnt/blob/master/docs/concepts.rst" target="_blank" class="">https://github.com/llvm-mirror/lnt/blob/master/docs/concepts.rst</a>. Unfortunately, this version of the documentation hasn't found it's way yet to <a href="http://llvm.org/docs/lnt/contents.html" target="_blank" class="">http://llvm.org/docs/lnt/contents.html</a>.<div class="">Is there a reason why storing different configurations as different "machines" in the LNT model doesn't work for you?</div><div class="">I assume that there are a number of places in LNT's analyses that assume that different runs coming from the same "machine" are always produced by the same configuration. But I'm not entirely sure about that.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class=""><br class=""></div><div class="">Kristof</div></div><br class="">_______________________________________________<br class="">LLVM Developers mailing list<br class=""><a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a><br class=""><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class=""><br class=""></blockquote></div><br class=""></div></div></blockquote></div><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">LLVM Developers mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""><a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a></span></div></blockquote></div><br class=""></body></html>