<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><br class=""></div><div><blockquote type="cite" class=""><div class="">On Jan 25, 2017, at 11:39 AM, Chris Matthews via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div id="bloop_customfont" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;" class="">There are a lot more things in there than the metadata detection. Off the top of my head:</div><div id="bloop_customfont" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;" class=""><br class=""></div><div id="bloop_customfont" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;" class=""> - xml test reports for the CI systems, and CSV reports</div></div></blockquote><div>Just out of interested, what are the CSV reports used for?</div><div>Both tasks sound like we could drop a ~100 line python script in test-suite/utils to get them done (I would even volunteer to write them if that is the last thing necessary and someone actually gives me a specification/good examples of what should come out of it :)</div><br class=""><blockquote type="cite" class=""><div class=""><div id="bloop_customfont" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;" class=""> - profile data collection</div></div></blockquote>What exactly do you mean here if it is not a PGO build? (see below)<br class=""><div><br class=""></div><blockquote type="cite" class=""><div class=""><div id="bloop_customfont" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;" class=""> - test-suite json is different than LNTs</div></div></blockquote><div>Indeed, though I don't see a strong reason why it has to be that way. Looks like a relatively small tweak to either lit or LNT (or teach one of them a 2nd format) to fix that, because the underlying data model looks compatible.</div><br class=""><blockquote type="cite" class=""><div class=""><div id="bloop_customfont" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;" class=""> - compile/exec multisample</div></div></blockquote><div>It is easy to change the test-suite lit stuff to run a benchmark multiple times. The main question would be whether we invent a data format that allows multiple results for each metric (I am not entirely sure I want that as it complicates the data model) or whether we teach the lit code to do the aggregation (in which case we have to figure out how to do that exactly. The right choice of summary function probably depends on the metric so we would need some extra information on that front).</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div id="bloop_customfont" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;" class=""> - the diagnose stuff</div></div></blockquote><div>Could maybe live outside of LNT as test-suite/utils/diagnose.py?</div><br class=""><blockquote type="cite" class=""><div class=""><div id="bloop_customfont" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;" class=""> - running with PGO</div></div></blockquote><div>Well a PGO build should boil down to:</div><div>cmake -DTEST_SUITE_PROFILE_GENERATE ...</div><div>ninja</div><div>lit</div><div>cmake -DTEST_SUITE_PROFILE_USE ...</div><div>ninja</div><div>lit</div><div><br class=""></div><div>and I would say it is reasonable to expect the 3 extra commands compared to a "normal" build...</div><br class=""><blockquote type="cite" class=""><div class=""><div id="bloop_customfont" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;" class=""> - result filtering (exclude stats)</div></div></blockquote><div>From a general point of view I would expect it to be best to just dump stuff into the LNT database and leave the filtering and aggregation to clients (= the website).</div><div>Of course if it is an unreasonable amount of data some forms of early filtering may be apropriate. In fact you can already control this somewhat today by modifying the lit plugins used (i.e. stats only appear in the output if you do cmake -DTEST_SUITE_COLLECT_STATE, codesize is a litsupport plugin as well so it would take a 3 line patch to add a cmake flag to disable that).</div><br class=""><blockquote type="cite" class=""><div class=""><div id="bloop_customfont" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;" class=""> </div><div id="bloop_customfont" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;" class="">My goal is no client side LNT at all. Workflow: user checks out test-suite, runs, and curls the resulting json to the server. We use this workflow on all of our other LNT based data collection, and it works really well. Running test-suite manually, then some other tools after would be a regression in the complexity.</div><div style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><div style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">For metadata: could we write a tool, that the test-suite runs ad the end, that makes a new json for submission? We could add the metadata and apply the transformations to make the json grockable by LNT, and produce the xml and csv. </div></div></blockquote><div>As for collecting the metadata I think that would fit nicely into the cmake step (and you can write arbitrary files with cmake so it should be perfectly possible to write some .json files).</div><div><br class=""></div><div>As an intermediate step we could indeed write a script in test-suite/utils that takes json output and the metadata file produced by lnt, integrates both and produces lnt output. Though in an ideal world I would see LNT being able to grok the output of lit and lit just passing the metadata along so we can leave out the extra conversion step.</div><div><br class=""></div><div>- Matze</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div id="bloop_sign_1485372112364166912" class="bloop_sign" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"></div><br style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><p class="airmail_on" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">On January 25, 2017 at 11:14:06 AM, Matthias Braun (<a href="mailto:matze@braunis.de" class="">matze@braunis.de</a>) wrote:</p><blockquote type="cite" class="clean_bq" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;"><span class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class=""></div><div class=""><div class="">I did a half-hearted attempt here: <a href="https://reviews.llvm.org/D19949" class="">https://reviews.llvm.org/D19949</a></div><div class=""><br class=""></div><div class="">The main blocker here IMO is that we want some meta information about a build in LNT (compiler version used, hostname of the machine, ...) which the testsuite currently doesn't produce by itself. I see 3 solutions for that:</div><div class="">- We could change the test-suite cmake file to write this information to a separate file</div><div class="">- We could modify lit to produce the metainformation in the output (that functionality feels out of place in lit IMO), or possibly pass the information along from a file produced by cmake</div><div class="">- We could continue using a client side lnt tool that collects the information as part of the submission process.</div><div class=""><br class=""></div><div class="">- Matthias</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class="">On Jan 25, 2017, at 10:59 AM, Chris Matthews via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div id="bloop_customfont" class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;">I agree that dropping all the client side tools would be (REALLY) nice. The proxied test-suite approach adds conceptual and implementation complexity.</div><br class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div id="bloop_sign_1485370178489704192" class="bloop_sign" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"></div><span class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">lnt runtest test-suite is still doing a lot, it is not by any means a thin wrapper. I’d love to see a concrete proposal how we can migrate that into the test suite.</span><div class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class=""></div><div class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">Also at the same time, I think the test-suite would start to need its own testing.</div><div class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><p class="airmail_on">On January 25, 2017 at 1:40:03 AM, Kristof Beyls via Phabricator (<a href="mailto:reviews@reviews.llvm.org" class="">reviews@reviews.llvm.org</a>) wrote:</p><blockquote type="cite" class="clean_bq"><div class=""><div class=""></div><div class=""><span class="">kristof.beyls marked an inline comment as done.<span class="Apple-converted-space"> </span><br class="">kristof.beyls added a comment.<span class="Apple-converted-space"> </span><br class=""><br class=""><br class=""><br class="">>>> As for the rest of the commit: Using a toolchain file looks like just passing a -D flag which is already supported. As for picking up the compiler, couldn't you simply extract that from CMakeCache.txt? And if that doesn't work, shouldn't we rather upstream the changes to CMakeLists.txt that print the compiler instead of having LNT mess around with them?<span class="Apple-converted-space"> </span><br class="">>><span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span><br class="">>> I'm trying to eliminate as many not-needed command line options as possible, e.g. --llvm-arch, --cross-compiling, ... . There are already more than enough command line options to lnt runtest, reducing the amount of them probably would make it simpler to use.<span class="Apple-converted-space"> </span><br class="">>> Indeed, just using a -D flag to enable cross-compiling instead of having a separate lnt runtest command line option for it would be preferable (and then documenting how to use it to do cross-compiling).<span class="Apple-converted-space"> </span><br class="">>> But then, doesn't the same argument apply to dropping lnt runtest -cflags or lnt runtest -cppflags: they also both could be set using -D?<span class="Apple-converted-space"> </span><br class="">>> Anyway, let's not go there as part of this patch; I'll first look into whether I can eliminate the need for<span class="Apple-converted-space"> </span><a href="http://opts.cc/" class="">opts.cc</a><span class="Apple-converted-space"> </span>and opts.cxx also for the non-cross-compiling case. In that case, I indeed no longer see the need for a separate option so that lnt is aware we're cross-compiling, and instead we can just use -D.<span class="Apple-converted-space"> </span><br class="">><span class="Apple-converted-space"> </span><span class="Apple-converted-space"> </span><br class="">> My argument would be to drop the whole "lnt runtest" mode and let users do the cmake/ninja/lit sequence themselfes and teach "lnt submit" to just submit the results. That way you are never limited by missing or confusing options in lnt runtest, have one layer less to debug and we have one thing less to maintain and keep up to date. Though we may need to write more documentation about the cmake/lit test-suite.<span class="Apple-converted-space"> </span><br class=""><br class="">Interesting idea, I like it! When I explain others (non-LLVM-developers) about what lnt is, the first thing I explain is that it is both a tool to run the LLVM test-suite (which mostly isn't interesting to the non-LLVM-developers) and a server/database/analysis engine to keep track of code generation quality of a code generator (which is interesting to non-LLVM code generator developers). At least from this point of view, it'd be nice if lnt was "one thing" rather than "two things".<span class="Apple-converted-space"> </span><br class="">Maybe 'lnt submit' could also be dropped, and users could upload the produced json after a lit run using e.g. curl? On one hand, submitting jsons to the server could be classified as being part of the "server/database/analysis engine" functionality. OTOH, it might remove the need for lnt on the client/test machine side completely?<span class="Apple-converted-space"> </span><br class=""><br class=""><br class=""><a href="https://reviews.llvm.org/D29030" class="">https://reviews.llvm.org/D29030</a><span class="Apple-converted-space"> </span><br class=""><br class=""><br class=""><br class=""></span></div></div></blockquote></div><span class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">_______________________________________________</span><br class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">llvm-commits mailing list</span><br class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><a href="mailto:llvm-commits@lists.llvm.org" class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">llvm-commits@lists.llvm.org</a><br class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a></div></blockquote></div><br class=""></div></div></span></blockquote><span style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">llvm-commits mailing list</span><br style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="mailto:llvm-commits@lists.llvm.org" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">llvm-commits@lists.llvm.org</a><br style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a></div></blockquote></div><br class=""></body></html>