<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:56.7pt 42.5pt 56.7pt 85.05pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thank you for answers.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">About last question. Now there are compile and nt(simple) tests in LNT. We suggested to add opportunity to run and save in database any test-suite(nightly and etc test-suites
described with Makefile+.report file). Each test-suite has name: simple, nightly, etc. And when we run test we choose some test-suite. Now if we use Cmake and test modules we create test-suite as we want and it has no name. Moreover, I can run tests always
with different test modules. If we want to save results in LNT database we need to compare all metrics and then understood if we have such test-suite before or not.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">And how should we show these test-suites?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Now there are on main page<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Compile<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Nt<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">New custom metrics should be shown to and they should have name.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">One decision for this problem is to add file with description of test modules and association with this group of test modules some name.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">There is no such opportunity now?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Elena.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Matthias Braun [mailto:matze@braunis.de]
<br>
<b>Sent:</b> Friday, May 27, 2016 2:56 AM<br>
<b>To:</b> Elena Lepilkina <Elena.Lepilkina@synopsys.com><br>
<b>Cc:</b> llvm-dev <llvm-dev@lists.llvm.org>; nd <nd@arm.com><br>
<b>Subject:</b> Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test parameterization<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On May 26, 2016, at 7:08 AM, Elena Lepilkina via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I understood your modules and I see as them can be used in LNT. But there are some question about old features.</span><o:p></o:p></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">1.</span><span style="font-size:7.0pt"> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">With
Makefiles we can write pass and collect some statistics. There was an example of branch pass. Metrics can be collected by</span><o:p></o:p></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif">@-$(LOPT) -load dcc888$(SHLIBEXT) -branch-counter -stats \ -time-passes -disable-output $< 2>>$@</span><o:p></o:p></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif">in makefile. In report file we write how data should be parsed. Can we do same things now with cmake+lit?</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">This question hits several points at once:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- Collecting metrics at compiletime: The Makefiles have definitely been more flexible in this regard. The flip side of the medal however is that the makefiles became so complicated that you typically needed to pass several barely documented
flags to them for a simple task like compiling and running the benchmarks without any further actions.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">cmake on the other hand restricts us a bit in how much we can intercept and modify the build process of the benchmark (and just to point this out: the running process in lit is no problem and really flexible).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">The best worst solution I have come up with so far is using a python wrapper that understands the same arguments as clang and slightly modifies the arguments (adding stats and a providing a new output filename for the stats) before passing
it along to clang. I have some personal script running which does this but it is not in a cleaned publishable state yet.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">As for the reporting in the .report files: Instead of doing perl magic you can just as well read the json produced by lit and process it in your favorite scripting language. Minimalistic example in python (that just prints all metrics per
benchmark):<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">import json<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">import sys<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">data = json.load(open(sys.argv[1]))<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">for test in data['tests']:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> print "%-10s:" % test['name']<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> for name, value in test['metrics'].items():<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> print "\t%s: %s" % (name, value)<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">2.</span><span style="font-size:7.0pt"> <span class="apple-converted-space"> </span></span><span style="font-family:"Calibri",sans-serif">As
I saw in LNT code there is no opportunity to compile but not execute tests. Some metrics cab be collected without execution, than run can be faster.</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The lnt test module only exposes a limited set of features possible with the cmake/lit testsuite. IMO we should break the odd relationship that lnt drives the running of the benchmark, but should just read the result file after the user
has the testsuite in whatever configuration he wanted. I started a patch for this here: <a href="http://reviews.llvm.org/D19949">http://reviews.llvm.org/D19949</a> but I need to find some time to rework it.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">If you run the test-suite manually: Just remove the "run" module in the config.test_modules list in the lit.site.cfg of your builddir. The benchmarks will no longer be executed but the codesize, hash and compiletime modules will still collect
those metrics. We can add a cmake configuration parameter for this if this is a generally useful configuration.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">3.</span><span style="font-size:7.0pt"> <span class="apple-converted-space"> </span></span><span style="font-family:"Calibri",sans-serif">Before
each group of metrics has name. Now if we would like to integrate cmake with custom metrics in LNT, we should compare all metrics of each run to make sure if this is the same test-suite or not. And we should give some random names to each group.</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I don't understand what you mean here, can you give an example/typical use case?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">- Matthias<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>