<div dir="ltr">Hi Chris,<div><br></div><div>We (ab)use Orders by setting them to be buildbot build numbers.</div><div><br></div><div>Internally, we have a buildbot master and an LNT instance running. We have the ability to do "try" builds (for A/B testing) and these also submit to LNT. This means two things:</div><div> 1. The try builds shouldn't be mixed with the daily/weekendly builds to avoid polluting the database and making it useless for regression tracking.</div><div> 2. Being able to locate a specific buildbot build result inside LNT is required.</div><div><br></div><div>To handle both of these, we ensure there is a bijective mapping between LNT results and buildbot builds:</div><div> buildbot builder -> LNT machine</div><div> buildbot buildnumber -> LNT order</div><div><br></div><div>This makes it easy to work out which LNT run corresponds to which buildbot run, which is a good thing. It does mean we abuse Orders horribly though, and so date ordering doesn't work sometimes. We have a specific A/B buildbot builder, so these show up in a different LNT machine to the dailies.</div><div><br></div><div>I think for us to use orders properly, we'd need two things to happen:</div><div> 1. The ability to mark a submitted run as "polluting" or somesuch - it shouldn't be compared against by default and it shouldn't show up on performance trend graphs unless explicitly asked to. This handles A/B testing.</div><div> 2. The ability to "tag" an LNT run with some sort of label, and then to search for that label. That allows us to tag LNT runs with the builder name and number, and we can reuse Orders to mean the right thing (SVN revision).</div><div><br></div><div>I think for your changes, that would mean that your order information should also be able to contain information about whether a run was built using a patched compiler, or different command line flags.</div><div><br></div><div>Any other/better ideas about how we can converge our workflows appreciated!</div><div><br></div><div>Cheers,</div><div><br></div><div>James</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, 29 Sep 2015 at 00:49 Chris Matthews via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I’d like some feedback on how people are using Orders in LNT, and how we can improve that. Abstractly, Orders are the version of the thing LNT is testing. On <a href="http://llvm.org/perf" rel="noreferrer" target="_blank">llvm.org/perf</a>, Orders are the SVN revision of the compiler. The Order is used to combine runs together, as well as to provide the x-axis of LNT’s graphs. Other things I have seen used as Orders: git tags, git distance, unix time stamps, letters or numbers.<br>
<br>
What I would really like in LNT is more information about Orders. Data I (personally) would like:<br>
- blame list<br>
- change list with commit messages<br>
- links to more information<br>
<br>
It would be great to be able to show change lists between Orders so we know what the Order is actually capturing, including in the performance graphs and run pages. For SVN based system, that is easy, it is the change list between the two revisions. I’d also like to be able to link to more information (like veiwvc in the case of SVN), I think that can be accomplished with templates URLs. Long term, it would be nice to formally blame performance regressions, so a real blame list is needed for that.<br>
<br>
I think the ideal interface for everything I have been thinking about is just given two Orders, show the changes. Concretely for the SVN case I think we’d need to have LNT keep a mirror of the SVN repo of interest. That might be too much to formally reside in LNT, so maybe a new external service that collects and caches Order information. Does anyone have any input on what and how we should do this?<br>
<br>
<br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div>