<div dir="ltr">Hi Kristof,<div><br></div><div><font color="#500050" face="Calibri, sans-serif"><span style="font-size:14.6667px"> The way we use LNT, we would run different configuration (e.g. -O3 vs -Os) as different "machines" in LNT's model.</span></font></div><div><br></div><div>O2/O3 is indeed bad example. We're also using different machines for Os/O3 - such parameters apply to all tests and we don't propose major changes.</div><div><div>Elena was only extending LNT interface a bit to ease LLVM-testsuite execution with different compiler or HW flags.<br></div><div></div><div>Maybe some changes are required to analyze and compare metrics between "machines": e.g. code size/performance between Os/O2/O3.</div><div>Do you perform such comparisons?</div><div><br></div><div><br></div><div>"test parameters" are different, they allow exploring multiple variants of the same test case. E.g. can be:<br></div><div>* index of input data sets, length of input vector, size of matrix, etc;<br></div><div>* macro that affect source code such as changing 1) static data allocation to dynamic or 2) constants to variables (compile-time unknown)<br></div><div>* extra sets of internal compilation options that are relevant only for particular test case<br></div><div><br></div><div>Same parameters can apply to multiple tests with different value sets:</div><div><br></div><div>test1: param1={v1,v2,v3}<br></div><div>test2: param1={v2,v4}<br></div><div>test3:</div><div><br></div><div>Of course, original test cases can be duplicated (copied under different names) - that is enough to execute tests.</div><div>Explicit "test parameters" allow exploring dependencies between test parameters and metrics.</div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 21, 2016 at 4:36 PM, Kristof Beyls via llvm-dev <span dir="ltr"><<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<br>
<div><span class="">
<blockquote type="cite">
<div>On 21 Apr 2016, at 15:00, Elena Lepilkina <<a href="mailto:Elena.Lepilkina@synopsys.com" target="_blank">Elena.Lepilkina@synopsys.com</a>> wrote:</div>
<br>
<div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<span style="font-size:11pt;font-family:Calibri,sans-serif">Hi Kristof and Daniel,<u></u><u></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<span style="font-size:11pt;font-family:Calibri,sans-serif"> </span></div>
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<span style="font-size:11pt;font-family:Calibri,sans-serif">Thanks for your answers.<u></u><u></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<span style="font-size:11pt;font-family:Calibri,sans-serif"> </span></div>
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<span style="font-size:11pt;font-family:Calibri,sans-serif">Unfortunately I haven’t tried scaling up to a large data set before. Today I tried and results are quite bad.<u></u><u></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<span style="font-size:11pt;font-family:Calibri,sans-serif">So database scheme should be rebuild. Now I thought about creating one sample table for each test-suite, but not cloning all tables in packet. As I see other tables can be same for all
testsuites. I mean if user run tests of new testsuite, new sample table would be created during importing data from json, if it doesn’t exist. Are there some problems with this solution? May be, I don’t know some details.</span></div>
</div>
</div>
</blockquote>
<div><br>
</div>
</span><div>It's unfortunate to see performance doesn't scale with the proposed initial schema, but not entirely surprising. I don't really have much feedback on how the schema could be adapted otherwise as I haven't worked much on that. I hope Daniel will have more
insights to share here.</div><span class="">
<br>
<blockquote type="cite">
<div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<span style="font-size:11pt;font-family:Calibri,sans-serif"><u></u><u></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<span style="font-size:11pt;font-family:Calibri,sans-serif">Moreover, I have question about compile tests. Are compile tests runnable? In<span> </span><a href="http://llvm.org/perf" style="color:purple;text-decoration:underline" target="_blank">http://llvm.org/perf</a><span> </span>there
is no compile test. Does that mean that they are deprecated for now?<u></u><u></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<span style="font-size:11pt;font-family:Calibri,sans-serif"> </span></div>
<div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman',serif">
<span style="font-size:11pt;font-family:Calibri,sans-serif">About test parameters, for example, we would like to have opportunity to compare benchmark results of test compiled with -O3 and -Os in context of one run.</span></div>
</div>
</div>
</blockquote>
<div><br>
</div>
</span><div>The way we use LNT, we would run different configuration (e.g. -O3 vs -Os) as different "machines" in LNT's model. This is also explained in LNT's documentation, see</div>
</div>
<a href="https://github.com/llvm-mirror/lnt/blob/master/docs/concepts.rst" target="_blank">https://github.com/llvm-mirror/lnt/blob/master/docs/concepts.rst</a>. Unfortunately, this version of the documentation hasn't found it's way yet to <a href="http://llvm.org/docs/lnt/contents.html" target="_blank">http://llvm.org/docs/lnt/contents.html</a>.
<div>Is there a reason why storing different configurations as different "machines" in the LNT model doesn't work for you?</div>
<div>I assume that there are a number of places in LNT's analyses that assume that different runs coming from the same "machine" are always produced by the same configuration. But I'm not entirely sure about that.</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Kristof</div>
</div>
<br>_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
<br></blockquote></div><br></div>