<div dir="ltr">Hi Elena,<div><br></div><div>> <span style="font-family:calibri,sans-serif;font-size:11pt;color:rgb(33,33,33);line-height:1.5">First of all, now LNT uses make-style of running tests and parse results from result csv file. Are there any plans to go to cmake?</span></div><br class="inbox-inbox-Apple-interchange-newline"><div><br></div><div>There are two test drivers in LNT. "lnt runtests nt" uses the old Makefile-based system and "lnt runtests test-suite" uses the new cmake-based system. It sounds like you are using the former and should switch to the latter.</div><div><br></div><div>Cheers,</div><div><br></div><div>James</div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, 25 May 2016 at 10:04 Elena Lepilkina via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Hi Matthias,<u></u><u></u></span></p></div></div><div lang="EN-US" link="blue" vlink="purple"><div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thank you for your answer.
<u></u><u></u></span></p>
</div></div><div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">But can you answer for some more questions?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">First of all, now LNT uses make-style of running tests and parse results from result csv file. Are there any plans to go to cmake?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">As I understood lit will run and collect all metrics, but there is no opportunity to make any settings for choosing what metrics I would like to collect. Test reports files
allow to choose what report I would like. One time I can use one, second time I can use another. I can do this with cmake only by changing test_modules in file.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">So I can’t group some metrics and give them some name.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Am I right?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Elena.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Matthias Braun [mailto:<a href="mailto:matze@braunis.de" target="_blank">matze@braunis.de</a>]
<br>
<b>Sent:</b> Friday, May 20, 2016 8:03 PM<br>
<b>To:</b> Elena Lepilkina <<a href="mailto:Elena.Lepilkina@synopsys.com" target="_blank">Elena.Lepilkina@synopsys.com</a>><br>
<b>Cc:</b> Daniel Dunbar <<a href="mailto:daniel@zuster.org" target="_blank">daniel@zuster.org</a>>; llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>>; nd <<a href="mailto:nd@arm.com" target="_blank">nd@arm.com</a>></span></p></div></div></div></div><div lang="EN-US" link="blue" vlink="purple"><div><div><div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
<b>Subject:</b> Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test parameterization<u></u><u></u></span></p></div></div></div></div><div lang="EN-US" link="blue" vlink="purple"><div><div><div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On May 12, 2016, at 11:21 PM, Elena Lepilkina via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<u></u><u></u></p>
</div></blockquote></div></div></div><div lang="EN-US" link="blue" vlink="purple"><div><div><blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Hi all,</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">As we understood great changes will be done in LNT, so we are waiting to new LNT version and stopped our work in LNT.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">One more question about using test-suite separately with cmake. Cmake can only build all tests and generate lit tests. After that we can run LIT and get report which is not
equal with report (simple) got with make. Cmake test-suite version has no features to run custom metrics and generate other report type, right?</span><u></u><u></u></p>
</div>
</div>
</blockquote></div></div></div><div lang="EN-US" link="blue" vlink="purple"><div><div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Are these features of make-version of test-suite planned to be added?</span><u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal">The lit test-suite runner supports arbitrary metrics, it already features codesizes for different segments, compiletime, linktime, executable hash, execution time. I designed it be easily extensible with further metrics. Not all of these
metrics are understood by LNT yet so they may get lost after submission to an LNT database.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We do not use GenerateReport.pl and friends in the cmake/lit version anymore, as the main data collection and reporting mechanism in llvm has mostly shifted to LNT these days. In any case it should be easy to use “lit -o result.json” and
process the resulting json file in your favorite scripting language to generate arbitrary reports.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">As far as the TEST.*.Makefile things go: Many of those variants have no equivalent in cmake/lit. But I have the strong feeling that the majority of those is not really used or even broken so there are no plans to adding them to. On the
other hand I would love to hear from people that actually use any of those besides TEST.simple.Makefile, it should be possible to transition most of them but I’d first like to hear what they are used for.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">- Matthias<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div></div></div>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div>