<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
<div>
<blockquote type="cite" class="">
<div class="">On 25 Jan 2017, at 22:39, Matthias Braun via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" class="">
<div class=""><br class="Apple-interchange-newline">
On Jan 25, 2017, at 11:39 AM, Chris Matthews via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div id="bloop_customfont" class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;">
There are a lot more things in there than the metadata detection. Off the top of my head:</div>
<div id="bloop_customfont" class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;">
<br class="">
</div>
<div id="bloop_customfont" class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;">
- xml test reports for the CI systems, and CSV reports</div>
</div>
</blockquote>
<div class="">Just out of interested, what are the CSV reports used for?</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>I use them from time to time to do ad-hoc analysis on test-suite results, often by analyzing them in a spreadsheet.</div>
<div>The LNT webui currently doesn't make it very easy to do ad-hoc analyses.</div>
<div>The CSV tends to be more straightforward to import into other pieces of software, like a spreadsheet software, than the json.</div>
<div>If the LNT webui had a better support for ad-hoc analyses, maybe the need for a CSV format goes away? Although there may still be value in being able to run the test-suite and do some ad-hoc analyses on the results without needing to have an LNT instance.</div>
<div><br class="">
</div>
<blockquote type="cite" class="">
<div class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<div class="">Both tasks sound like we could drop a ~100 line python script in test-suite/utils to get them done (I would even volunteer to write them if that is the last thing necessary and someone actually gives me a specification/good examples of what should
come out of it :)</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div id="bloop_customfont" class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;">
- profile data collection</div>
</div>
</blockquote>
What exactly do you mean here if it is not a PGO build? (see below)<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>If this is the interpreting of linux perf records: yes, that's non-trivial. I think we may want to avoid moving all of that machinery to the test-suite, not sure.</div>
<div>Hopefully James has some ideas on how the linux perf records parsing and encoding logic might be simplified a bit, so it becomes a bit more trivial?</div>
<div>Maybe if instead of being a mix of Python and C code, it would all be C(++) code, it would fit more naturally under test-suite/tools?</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<div class=""><br class="">
</div>
<blockquote type="cite" class="">
<div class="">
<div id="bloop_customfont" class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;">
- test-suite json is different than LNTs</div>
</div>
</blockquote>
<div class="">Indeed, though I don't see a strong reason why it has to be that way. Looks like a relatively small tweak to either lit or LNT (or teach one of them a 2nd format) to fix that, because the underlying data model looks compatible.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div id="bloop_customfont" class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;">
- compile/exec multisample</div>
</div>
</blockquote>
<div class="">It is easy to change the test-suite lit stuff to run a benchmark multiple times. The main question would be whether we invent a data format that allows multiple results for each metric (I am not entirely sure I want that as it complicates the
data model) or whether we teach the lit code to do the aggregation (in which case we have to figure out how to do that exactly. The right choice of summary function probably depends on the metric so we would need some extra information on that front).</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>Maybe the test-suite doesn't need to do aggregation, as that's already interpretation of the data? Just send all collected data to LNT and let LNT do the analysis to come up with the preferred aggregation?</div>
<div>I think it'd be a slippery slope to start doing statistical analysis, however simple, in the test-suite scripts - probably best if that remains part of LNT.</div>
<div>I'd definitely be opposed to only send a single result, instead of multiple results to LNT, as the multiple results are often one of the most obvious ways to classify an apparent regression as noise or not.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<div class=""><br class="">
</div>
<blockquote type="cite" class="">
<div class="">
<div id="bloop_customfont" class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;">
- the diagnose stuff</div>
</div>
</blockquote>
<div class="">Could maybe live outside of LNT as test-suite/utils/diagnose.py?</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div id="bloop_customfont" class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;">
- running with PGO</div>
</div>
</blockquote>
<div class="">Well a PGO build should boil down to:</div>
<div class="">cmake -DTEST_SUITE_PROFILE_GENERATE ...</div>
<div class="">ninja</div>
<div class="">lit</div>
<div class="">cmake -DTEST_SUITE_PROFILE_USE ...</div>
<div class="">ninja</div>
<div class="">lit</div>
<div class=""><br class="">
</div>
<div class="">and I would say it is reasonable to expect the 3 extra commands compared to a "normal" build...</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>Seems reasonable to me - if we make sure we do write enough documentation for it to be easy for newbies to setup and run the test-suite for the different use cases.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
<blockquote type="cite" class="">
<div class="">
<div id="bloop_customfont" class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;">
- result filtering (exclude stats)</div>
</div>
</blockquote>
<div class="">From a general point of view I would expect it to be best to just dump stuff into the LNT database and leave the filtering and aggregation to clients (= the website).</div>
<div class="">Of course if it is an unreasonable amount of data some forms of early filtering may be apropriate. In fact you can already control this somewhat today by modifying the lit plugins used (i.e. stats only appear in the output if you do cmake -DTEST_SUITE_COLLECT_STATE,
codesize is a litsupport plugin as well so it would take a 3 line patch to add a cmake flag to disable that).</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>I think there's still value in not collecting a metric that you know is complete noise (the reason I introduced this functionality was not to collect compile-time numbers for compiling in parallel using all cores in a big.LITTLE system, where depending
on which core your compile job happens to land on, you can easily get a 2x difference in compilation speed).</div>
<div>That being said, I assume that it would be close to trivial to have this functionality live in the test-suite.</div>
<div><br class="">
</div>
<div>All in all, if we'd end up deciding that the responsibility of test-suite is to run stuff & collect data, and the responsibility of LNT is to archive and analyse it, I'd say filtering of results that up-front are known to be meaningless is a responsibility
of test-suite.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
<blockquote type="cite" class="">
<div class="">
<div id="bloop_customfont" class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;">
</div>
<div id="bloop_customfont" class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px;">
My goal is no client side LNT at all. Workflow: user checks out test-suite, runs, and curls the resulting json to the server. We use this workflow on all of our other LNT based data collection, and it works really well. Running test-suite manually, then some
other tools after would be a regression in the complexity.</div>
<div class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<br class="">
</div>
<div class="" style="font-family: Helvetica, Arial; font-size: 13px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
For metadata: could we write a tool, that the test-suite runs ad the end, that makes a new json for submission? We could add the metadata and apply the transformations to make the json grockable by LNT, and produce the xml and csv. </div>
</div>
</blockquote>
<div class="">As for collecting the metadata I think that would fit nicely into the cmake step (and you can write arbitrary files with cmake so it should be perfectly possible to write some .json files).</div>
<div class=""><br class="">
</div>
<div class="">As an intermediate step we could indeed write a script in test-suite/utils that takes json output and the metadata file produced by lnt, integrates both and produces lnt output. Though in an ideal world I would see LNT being able to grok the output
of lit and lit just passing the metadata along so we can leave out the extra conversion step.</div>
<div class=""><br class="">
</div>
<div class="">- Matze</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>One more piece of functionality that I think may be harder to get rid of is the rerun-functionality that exists in 'lnt runtest nt'.</div>
<div>As part of submitting data, the LNT server can send back a request to do a few more runs of specific tests, to get more statistically significant results where needed.</div>
<div>I'm not sure if we'd be able to do this if we'd use e.g. curl for data submission instead of something as part of lnt.</div>
<div><br class="">
</div>
</div>
<br class="">
</body>
</html>