<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Questions from this thread that I can help address:</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">The LNT compile suite: we use it a lot here at Apple. It has a metric set that is customized for the analysis of compile time regressions. Given the recent interest in compile time, I hope to be able to setup a public bot to collect data in this suite sometime soon.</div><div class=""><br class=""></div><div class="">On the topic of encoding configurations in or across machines: both work and are used. LNT Compile stores all the optimization levels in the same run, and uses part of the benchmark name to encode that. “compile.benchmark.name (opt and flags).metric”. This kind of flexibility is nice. The tradeoff is that it is harder to compare results from different machines (the web ui make it almost impossible - but you can do it by editing the URLs by hand. I think James made this a bit better recently?).</div><div class=""> </div><div class="">The ill-fated FieldChange table. This was added to LNT ages ago, and there were no consumers of the data. When I went to do the regression tracking feature, I realized there was an error in how the data was being calculated (a missing join was mixing data from other machines). Since sometime people disagree with me, and people blindly update their LNT instances, I decided best thing to do was not DROP the table, but just leave it incase a rollback was needed. That is how FieldChangeV2 came about, and that is why FieldChange still exists. Everyone can feel free to DROP the old table any time, and now that we have not been using it for a while and no one has complained, it is probably safe to remove it with a migration.</div><div class=""><br class=""></div><div class="">The intent of the test-suites as the primary database entity in LNT is manage the schema of the metics. The test-suite abstraction adds gobs of complexity to LNT in the backend. I’d be happy to drop that idea in favor of a more flexible scheme. IMO, the complexity comes from how we dynamically create the database schema from the test-suite definition. I’ve tried to add things to LNT in the past, (JSON api, admin interface, better migration to name three), and most of the third party flask modules require DB schema to be defined up front. That said about the test-suite system, it might help out here. We could implement your proposed changes as a third test-suite kind.</div><div class=""><br class=""></div><div class=""><br class=""></div><br class=""><div><blockquote type="cite" class=""><div class="">On Apr 25, 2016, at 4:33 AM, Elena Lepilkina via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></span></div><div class=""><div style="border-style: solid none none; border-top-color: rgb(225, 225, 225); border-top-width: 1pt; padding: 3pt 0in 0in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><b class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">From:</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="Apple-converted-space"> </span>Elena Lepilkina<span class="Apple-converted-space"> </span><br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Monday, April 25, 2016 2:33 PM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span>'James Molloy' <<a href="mailto:james@jamesmolloy.co.uk" style="color: purple; text-decoration: underline;" class="">james@jamesmolloy.co.uk</a>>; Kristof Beyls <<a href="mailto:Kristof.Beyls@arm.com" style="color: purple; text-decoration: underline;" class="">Kristof.Beyls@arm.com</a>>; Mehdi Amini <<a href="mailto:mehdi.amini@apple.com" style="color: purple; text-decoration: underline;" class="">mehdi.amini@apple.com</a>><br class=""><b class="">Cc:</b><span class="Apple-converted-space"> </span>nd <<a href="mailto:nd@arm.com" style="color: purple; text-decoration: underline;" class="">nd@arm.com</a>>; Matthias Braun <<a href="mailto:matze@braunis.de" style="color: purple; text-decoration: underline;" class="">matze@braunis.de</a>><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>RE: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test parameterization<o:p class=""></o:p></span></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Hi everyone,<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thank you for your answer. BLOB format adds some more actions for working with metrics. We know that<span class="Apple-converted-space"> </span></span>ComparisonResult class makes analysis work. But it gets all metrics by request from database, we will need additional time for work with fields during analysis in ComparisonResult class. May be it will be better to do one Sample table for each testsuite, as it was suggested before. It should be more quickly, shouldn’t it? Moreover, next wished LNT changes will need getting some metrics separately and BLOB format will add some delay in time for queries.<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">As we see now problem of performance is actual, because time for rendering graph page is about 3 minutes.<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span id="cid:image001.png@01D19EFA.6E5F0A70"><image001.png></span><o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">So maybe it will be better to start working with NoSql databases? I made a small prototype with TestSuite, TestSuiteFields, Test, Run and Sample tables for getting time metrics. It works quickly. And using NoSQL helps solve problems with different fields for samples metrics fields. Then it will be possible to store different metrics for different testsuites in one table.<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">What do you think about this proposal?<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">I used MongoDB, but I know that there is NoSQL extension for PostgresSQL with JSONB fields which are more<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">effective than JSON-encoded BLOB, because it can be included in queries very simply and let use indexes.<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">About proposal that not all metrics should be shown. It can be added as a field in JSON in .fields file, which describes fields getted from test-suite. To see other metrics user should choose them with checkboxes in view options. Will be this solution suitable?<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">We can make as you wrote<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">“I'd also suggest that if we're adding many more metrics to a test, we should create a "test sample information" page that the test link goes to instead of just the graph. This page could contain all counter/metric data, historic sparklines, the full graph and profiling links.<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">”<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">But the render time of this page will be too great because of graph render time. In my opinion, some users wouldn’t like to wait so long for see some additional metrics.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thanks for your suggestions,<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Elena.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><b class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">From:</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="Apple-converted-space"> </span>llvm-dev [<a href="mailto:llvm-dev-bounces@lists.llvm.org" style="color: purple; text-decoration: underline;" class="">mailto:llvm-dev-bounces@lists.llvm.org</a>]<span class="Apple-converted-space"> </span><b class="">On Behalf Of<span class="Apple-converted-space"> </span></b>James Molloy via llvm-dev<br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Monday, April 25, 2016 12:43 PM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span>Kristof Beyls <<a href="mailto:Kristof.Beyls@arm.com" style="color: purple; text-decoration: underline;" class="">Kristof.Beyls@arm.com</a>>; Mehdi Amini <<a href="mailto:mehdi.amini@apple.com" style="color: purple; text-decoration: underline;" class="">mehdi.amini@apple.com</a>><br class=""><b class="">Cc:</b><span class="Apple-converted-space"> </span>llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" style="color: purple; text-decoration: underline;" class="">llvm-dev@lists.llvm.org</a>>; nd <<a href="mailto:nd@arm.com" style="color: purple; text-decoration: underline;" class="">nd@arm.com</a>>; Matthias Braun <<a href="mailto:matze@braunis.de" style="color: purple; text-decoration: underline;" class="">matze@braunis.de</a>><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test parameterization<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">Hi Sergey, Elena,<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">Firstly, thanks for this RFC. It's great to see more people actively using and modifying LNT and the test metrics support in general is rather weak currently.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">Metrics<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">-------<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">I agree with Daniel and Kristof that your proposed schema changes have the potential to make many queries extremely slow. Certainly for the metrics enhancements, I don't see a reason why we need such a radical change in schema.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">To add custom metrics on the fly, we need to change the schema for the Sample table. Currently this consists of a column for each metric, but actually we never ever query those metric values. We never query for example for "all failing tests in a run" - when we do analyses we use the ComparisonResult class which reads *all* samples from the database for a run and performs the analysis entirely in Python.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">Therefore, having a semi-structured format where some fields are first-class columns and the rest are in a JSON-encoded BLOB (as Daniel suggests) seems totally acceptable. There is certainly an argument now that we're using the wrong backend storage solution and that a key-value store might be more suitable, but that's a very invasive change and I don't think we've reached the point where we need to force a move from the simplicity of SQLite.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">Adding an extra BLOB column would be easy - there would just need to be logic in testsuitedb.py for reading and writing it - the Sample model class would expose the JSON-encoded fields as normal python fields so the rest of LNT would be isolated from this change.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">But I think this is a small detail compared to the bigger problem of how to effectively *display* all this new data. Currently every new metric gets its own separate table in the report/run views, and this does not scale well at all.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">I think we need some more concepts in the metric system to make it scaleable:<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""> * What "attribute" of the test is this metric measuring? For example, both "exec_time" and "score" measure the same attribute; performance of the generated code. It's superfluous to have them displayed in separate tables. However mem_size and compile_time both measure completely different aspects of the test.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""> * Is this metric useful to display at the top level? or should it only be exposed when more data about a test result is requested?<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""> * An example of this is the pass statistics. I don't want my daily report view cluttered by the time spent in register allocation for every test! OK, this is useful information when debugging a problem, but it should be available when requested rather than by default.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">An example of why we need the above is your screenshots in your google doc. I'm looking at the last screenshot, and it's incredibly difficult to read and get useful information out of.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">I'd also suggest that if we're adding many more metrics to a test, we should create a "test sample information" page that the test link goes to instead of just the graph. This page could contain all counter/metric data, historic sparklines, the full graph and profiling links.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">Cheers,<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">James<o:p class=""></o:p></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">On Fri, 22 Apr 2016 at 10:17 Kristof Beyls via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" style="color: purple; text-decoration: underline;" class="">llvm-dev@lists.llvm.org</a>> wrote:<o:p class=""></o:p></div></div><blockquote style="border-style: none none none solid; border-left-color: rgb(204, 204, 204); border-left-width: 1pt; padding: 0in 0in 0in 6pt; margin: 5pt 0in 5pt 4.8pt;" class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">On 22 Apr 2016, at 11:14, Mehdi Amini <<a href="mailto:mehdi.amini@apple.com" target="_blank" style="color: purple; text-decoration: underline;" class="">mehdi.amini@apple.com</a>> wrote:<o:p class=""></o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt; text-align: start; word-spacing: 0px;" class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class="">On Apr 22, 2016, at 12:45 AM, Kristof Beyls via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" style="color: purple; text-decoration: underline;" class="">llvm-dev@lists.llvm.org</a>> wrote:<o:p class=""></o:p></span></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div><div class=""><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class="">On 21 Apr 2016, at 17:44, Sergey Yakoushkin <<a href="mailto:sergey.yakoushkin@gmail.com" target="_blank" style="color: purple; text-decoration: underline;" class="">sergey.yakoushkin@gmail.com</a>> wrote:<o:p class=""></o:p></span></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Hi Kristof,<o:p class=""></o:p></span></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(80, 0, 80);" class=""> The way we use LNT, we would run different configuration (e.g. -O3 vs -Os) as different "machines" in LNT's model.</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">O2/O3 is indeed bad example. We're also using different machines for Os/O3 - such parameters apply to all tests and we don't propose major changes.<o:p class=""></o:p></span></div></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Elena was only extending LNT interface a bit to ease LLVM-testsuite execution with different compiler or HW flags.<o:p class=""></o:p></span></div></div></div></div></div></blockquote><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Oh I see, this boils down to extending the lnt runtest interface to be able to specify a set of configurations, rather than a single configuration and making<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">sure configurations get submitted under different machine names? We kick off the different configuration runs through a script invoking lnt runtest multiple<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">times. I don't see a big problem with extending the lnt runtest interface to do this, assuming it doesn't break the underlying concepts assumed throughout<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">LNT. Maybe the only downside is that this will add even more command line options to lnt runtest, which already has a lot (too many?) command line<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">options.<o:p class=""></o:p></span></div></div><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: 'Times New Roman', serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></p><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Maybe some changes are required to analyze and compare metrics between "machines": e.g. code size/performance between Os/O2/O3.<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Do you perform such comparisons?<o:p class=""></o:p></span></div></div></div></div></div></blockquote><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">We typically do these kinds of comparisons when we test our patches pre-commit, i.e. comparing for example '-O3' with '-O3 'mllvm -enable-my-new-pass'.<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">To stick with the LNT concepts, tests enabling new passes are stored as a different "machine".<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">The only way I know to be able to do a comparison between runs on 2 different "machine"s is to manually edit the URL for run vs run comparison<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">and fill in the runids of the 2 runs you want to compare.<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">For example, the following URL is a comparison of green-dragon-07-x86_64-O3-flto vs green-dragon-06-x86_64-O0-g on the public <a href="http://llvm.org/perf" target="_blank" style="color: purple; text-decoration: underline;" class="">llvm.org/perf</a> server:<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><a href="http://llvm.org/perf/db_default/v4/nts/70644?compare_to=70634" target="_blank" style="color: purple; text-decoration: underline;" class="">http://llvm.org/perf/db_default/v4/nts/70644?compare_to=70634</a><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">I had to manually look up and fill in the run ids 70644 and 70634.<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">It would be great if there was a better way to be able to do these kind of comparisons - i.e. not having to manually fill in run ids, but having a webui to easily find and pick the runs you want to compare.<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">(As an aside: I find it intriguing that the URL above suggests that there are quite a few cases where "-O0 -g" produces faster code than "-O3 -flto").<o:p class=""></o:p></span></div></div></div></div></blockquote><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Can you be more explicit which ones? I don't see any regression (other than compared to the baseline, or on the compile time).<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">-- <o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Mehdi<o:p class=""></o:p></span></div></div></div></blockquote><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">D'Oh! I was misinterpreting the compile time differences as execution time differences. Indeed, there is no unexpected result in there.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">Sorry for the noise!<o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">Kristof<o:p class=""></o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><o:p class=""> </o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">_______________________________________________<br class="">LLVM Developers mailing list<br class=""><a href="mailto:llvm-dev@lists.llvm.org" target="_blank" style="color: purple; text-decoration: underline;" class="">llvm-dev@lists.llvm.org</a><br class=""><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank" style="color: purple; text-decoration: underline;" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><o:p class=""></o:p></div></blockquote></div></div><span style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">LLVM Developers mailing list</span><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""><a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a></span><br style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a></span></div></blockquote></div><br class=""></body></html>