<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">The new schema definition logic landed in r307222/r307223 I'd be happy to get some post-commit reviews.</div><div class=""><br class=""></div><div class="">- The new mechanisms are documented in the importing-data part of the docu.</div><div class=""><br class=""></div><div class="">- Unfortunately today you have to choose whether you define the schema directly in the database ('nts' and 'compile' still use that) or whether you use the new .yaml schemas in the database. This means you cannot simply change the default 'nts' schema. (I'd be happy to transfer it to a .yaml schema however that means we loose some flexibility in automatically upgrading old databases so I cannot simply do that as an NFC commit).</div><div class=""><br class=""></div><div class="">- I'll convert the 'compile' schema to the new format in the next days (just have to synchronize this with our internal jobs so we have at least 1 real world example in the public repository.</div><div class=""><br class=""></div><div class="">- Matthias</div><br class=""><div><blockquote type="cite" class=""><div class="">On Jul 5, 2017, at 2:09 PM, Chris Matthews <<a href="mailto:chris.matthews@apple.com" class="">chris.matthews@apple.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=us-ascii" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">The test-suite schema is defined in the DB. It is not hard to extend if you have server access. The docs detail the steps to add a new metric:</div><div class=""><br class=""></div><div class=""><a href="http://llvm.org/docs/lnt/importing_data.html#custom-test-suites" class="">http://llvm.org/docs/lnt/importing_data.html#custom-test-suites</a></div><div class=""><br class=""></div><div class="">I have setup several custom suites. It works.</div><div class=""><br class=""></div><div class="">Matthias is working on something to make that even easier by having the test suites self describe their schemas. This will still require server access, but will be less scary than editing the DB directly.</div><div class=""><br class=""></div><div class="">A lot of this boils down to naming, and how the data is later presented. For instance, in some places we have elected to store the new link time metric as a differently named test in the compile_time metric (foo.c vs foo.c.link). When you do this, those are presented side by side in the data listing views, that is handy. Each metric is given a section in the run reports, you can imagine what that might look like with 50 metrics. We might need to do some UI redesign to make the run reports sane.</div><div class=""><br class=""></div><div class="">I think LNT will have a hard time collecting all the stats right now. There are 2722 source files in the llvm test-suite right now, and many hundred stats. Especially pages like the run reports which still do inline calculations are going to be slow. We do cache all the stuff needed to render the pages quickly now, but that pages are not updated to use that.</div><div class=""><br class=""></div><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Jul 5, 2017, at 9:20 AM, Tobias Grosser via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">On Wed, Jul 5, 2017, at 05:48 PM, Matthias Braun via llvm-dev wrote:</span><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class=""><br class=""><br class=""><blockquote type="cite" class="">On Jul 4, 2017, at 2:02 AM, Tobias Grosser <<a href="mailto:tobias.grosser@inf.ethz.ch" class="">tobias.grosser@inf.ethz.ch</a>> wrote:<br class=""><br class=""><blockquote type="cite" class="">On Tue, Jul 4, 2017, at 09:48 AM, Kristof Beyls wrote:<br class="">Hi Tobias,<br class=""><br class="">The metrics that you can collect in LNT are fixed per "test suite".<br class="">There are 2 such "test suite"s defined in LNT at the moment: nts and<br class="">compile.<br class="">For more details on this, see<br class=""><a href="http://llvm.org/docs/lnt/concepts.html#test-suites" class="">http://llvm.org/docs/lnt/concepts.html#test-suites</a>.<br class=""><br class="">AFAIK, If you need to collect different metrics, you'll need to define a<br class="">new "test suite". I'm afraid I don't really know what is needed for that.<br class="">I'm guessing you may need to write some LNT code to do so, but I'm not<br class="">sure. Hopefully Matthias or Chris will be able to explain how to do that.<br class=""><br class="">We probably should investigate how to make it easier to define new<br class="">"test-suite"s more easily. Or at least make it easier to record different<br class="">sets of metrics more easily, without having to change the LNT code or a<br class="">running LNT server instance.<br class="">The question on recording a different set of metrics has come up on this<br class="">list before, so it seems like it's an issue people do run into from time<br class="">to time.<br class=""></blockquote><br class="">Hi Kristof,<br class=""><br class="">thanks for your fast reply. This is a very helpful summary that confirms<br class="">my current understanding in parts. I never run the "compile" test suite,<br class="">so I am not sure how much of the statistics interface is used by it (if<br class="">at all). I somehow had the feeling something else might exist, as the<br class="">cmake test-suite<br class="">runner dumps some of the statistics to stdout. Would be interested to<br class="">read if Chris or Matthias have more insights.<br class=""></blockquote><br class="">I often run the testsuite without LNT. Lit -o dumps the output to a json<br class="">file. If your goal is just some A/B testing (rather than tracking<br class="">continuously with CI systems) then something simple like<br class="">test-suite/utils/compare.py is enough to view and compare lit result<br class="">files.<br class=""></blockquote><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Right. That's what I have been seeing.<span class="Apple-converted-space"> </span></span><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">For future LNT plans:<br class=""><br class="">You also asked at an interesting moment: I am polishing a commit to LNT<br class="">right now that makes it easier to define custom schemas or create new<br class="">ones. Though that is only part of the solution, as even with the new<br class="">schema the runner needs to be adapted to actually collect/transform all<br class="">values.<br class=""></blockquote><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Great. Would be glad to follow the patch review.</span><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">I think we also will not start collecting all the llvm stats by default<br class="">in the current system; with a few thousand runs in the database its<br class="">slightly sluggish already, I don't think adding 10x more metrics to the<br class="">database helps there. Of course once it is easier to modify schemas you<br class="">could setup special instances with extended schemas that maybe track<br class="">fewer instances/runs.<br class=""></blockquote><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">I wonder if the sluggishness comes from trying to display all of these</span><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">or from storing them. If they would be in a separate table that is only</span><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">accessed when needed, maybe storing would not have such a large cost.</span><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Best,</span><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Tobias</span><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">LLVM Developers mailing list</span><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="mailto:llvm-dev@lists.llvm.org" style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">llvm-dev@lists.llvm.org</a><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a></div></blockquote></div><br class=""></div></div></blockquote></div><br class=""></body></html>