<div dir="ltr">Hi Elena,<div><br></div><div>Thanks for pushing forward with this. I like the idea of using a NoSQL solution.</div><div><br></div><div>My primary reservation is about adding the new NoSQL stuff as an extra backend. I think the current database backend and its use of SQLAlchemy is extremely complex and is in fact the most complex part of LNT. Adding something more (as opposed to *replacing* it) would just make this worse and make it more likely that contributors wouldn't be able to test LNT very well (having three engines to test: SQLite, PostgreSQL and MongoDB).</div><div><br></div><div>I think it'd be far better all around, if we decide to go with the NoSQL solution, to just bite the bullet and force users who want to run a server to install MongoDB.</div><div><br></div><div>In my experience most of the teams I've seen using LNT have a single LNT server instance and submit results to that, rather than launching small instances to run "lnt viewcomparison".</div><div><br></div><div>Cheers,</div><div><br></div><div>James</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, 26 Apr 2016 at 09:15 Elena Lepilkina via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal">Hi everyone.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks to everyone who took participant in discussion of this proposal.<u></u><u></u></p>
<p class="MsoNormal">After discussion we understood how other users use LNT and how great datasets may be.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">So there is new updated proposal.<u></u><u></u></p>
<p class="MsoNormal">(Google docs version with some images - <a href="https://docs.google.com/document/d/11qHNWRIQ2gc2aWH3gNBeeCUB3-JPe7AoMtn7n9JoCeY/edit?usp=sharing" target="_blank">https://docs.google.com/document/d/11qHNWRIQ2gc2aWH3gNBeeCUB3-JPe7AoMtn7n9JoCeY/edit?usp=sharing</a>)<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Goal is the same.<u></u><u></u></p>
<p class="MsoNormal">Enable LNT support of custom metrics such as: user-defined run-time and static metrics (power, etc.) and LLVM pass statistic counters. Provide integration with LLVM testsuite to automatically collect LLVM statistic counters or custom metrics.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Analysis of current Database<b><u></u><u></u></b></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Limitations<u></u><u></u></p>
<p><u></u><span>1.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>This structure isn’t flexible.<u></u><u></u></p>
<p class="MsoNormal">There is no opportunity to run another test-suite except simple one.<u></u><u></u></p>
<p><u></u><span>2.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Performance is quite bad when database has a lot of records.<u></u><u></u></p>
<p class="MsoNormal">For example, rendering graph is too slow. On green-dragon-07-x86_64-O3-flto:42 SingleSource/Benchmarks/Shootout/objinst compile_time need for rendering 191.8 seconds.<u></u><u></u></p>
<p><u></u><span>3.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u> It’s difficult to add new features which need queries to sample table in database(if we use BLOB field for custom metrics).<u></u><u></u></p>
<p class="MsoNormal">Queries will be needed for more complex analysis. For example, if we would like to add some additional check for tests which compile time is too long, we should get result of query where this metric is greater than some constant.<u></u><u></u></p>
<p class="MsoNormal">Or we would like to compare tests with different run options, so we should get only some tests but not all.<u></u><u></u></p>
<p class="MsoNormal">BLOB field will help to save current structure and make system a bit more flexible. But in the nearest future it will be not enough.
<u></u><u></u></p>
<p class="MsoNormal">Getting all metrics of all tests will make work slow on great datasets. And this way isn’t enough optimal.
<u></u><u></u></p>
<p class="MsoNormal">So we wouldn’t like to do BLOB field, which wouldn’t help to add new features and have flexible system in future.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Proposal<u></u><u></u></p>
<p class="MsoNormal"><b><u></u> <u></u></b></p>
<p class="MsoNormal">We suggest to do third part of LNT (as Chris Matthews suggested). This part will be used for getting custom metrics<span style="color:black"> and running any test-suite.</span><u></u><u></u></p>
<p class="MsoNormal">We suggest to use NoSQL database (for example, MongoDB or JSON/JSONB extension of PostgresSQL, which let use PostgresSQL as NoSQL database) for this part. This part will be enable if there is path to NoSQL database in config file.<u></u><u></u></p>
<p class="MsoNormal">It helps to have one Sample table(collection in NoSQL). If we use schemaless feature in MongoDB, for example, then it’s possible to add new fields when new testsuite is running. Then there would be one table with a lot of fields, some
of which are empty. At any moment of time it will be possible to change schema of table(document).<u></u><u></u></p>
<p class="MsoNormal">A small prototype was made with MongoDB and ORM MongoEngine. This ORM was choosen because MongoAlchemy doesn’t support schemaless features and last MongoKit version has error with last pymongo release.<u></u><u></u></p>
<p class="MsoNormal">I try it on virtual machine and get next results on 5 000 000 records.<u></u><u></u></p>
<p class="MsoNormal">Current scheme - 13.72 seconds<u></u><u></u></p>
<p class="MsoNormal">MongoDB – 1.35 seconds.<u></u><u></u></p>
<p class="MsoNormal">Results of course will be better on real server machine .<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black">For use some test-suite user should describe fields in file with format .fields such way:</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black">{</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> "Fields" : [{</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> "TestSuiteName" : "Bytecode",</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> "Type" : "integer",</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> "BiggerIsBetter" : 0,</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> "Show" : true</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> },</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> {</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> "TestSuiteName" : "GCC",</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> "Type" : "real",</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> "BiggerIsBetter" : 0,</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> "Name" : "GCC time"</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> },</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> {</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> "TestSuiteName" : "JIT",</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> "Type" : "real",</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> "BiggerIsBetter" : 0,</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> "Name" : "JIT Compile time",</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> "Show" : true</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> },</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> {</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> "TestSuiteName" : "GCC/LLC",</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> "Type" : "string",</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> "BiggerIsBetter" : 0</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"> }]</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black">}</span><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black">There was added one field “Show” for describing if this metric should be shown by default on web page (as James Molloy suggested). Other metrics would be added to page if user chooses
them in view options.</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"><u></u> <u></u></span></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black">Conclusion<u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"><u></u> <u></u></span></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black">This change will let user to choose if he wants to use flexible powerful system or use limited version with SQLite database.</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black">If user chooses NoSQL version his data can be copied from its old database to new one. This will help to use new features without losing old data.<u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black"><u></u> <u></u></span></p>
<p class="MsoNormal" style="text-align:justify"><span style="color:black">The actual question is which NoSQL database will be better for LNT. We are interested in opinions of people, who know features of LNT better.</span><u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><u></u> <u></u></p>
<p class="MsoNormal" style="text-align:justify">Thanks,<u></u><u></u></p>
<p class="MsoNormal" style="text-align:justify"><u></u> <u></u></p>
<p class="MsoNormal" style="text-align:justify">Elena.<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> llvm-dev [mailto:<a href="mailto:llvm-dev-bounces@lists.llvm.org" target="_blank">llvm-dev-bounces@lists.llvm.org</a>]
<b>On Behalf Of </b>Elena Lepilkina via llvm-dev<br>
<b>Sent:</b> Tuesday, April 26, 2016 9:07 AM<br>
<b>To:</b> <a href="mailto:chris.matthews@apple.com" target="_blank">chris.matthews@apple.com</a></span></p></div></div></div></div><div lang="EN-US" link="blue" vlink="purple"><div><div><div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
<b>Cc:</b> llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br>
<b>Subject:</b> Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test parameterization<u></u><u></u></span></p></div></div></div></div><div lang="EN-US" link="blue" vlink="purple"><div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Hi, Chris.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thank you for your answer about compile tests. As I understood during looking through code of compile tests they don’t use test suite at all. Am I right? There is lack of information
and examples of running compile tests in LNT documentation.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">We understood that there are two groups of users: users using servers and collecting a lot of data and SQLite users, but these users as I think wouldn’t have about millions
of sample records.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I think that it’s obvious that there is no universal solution for simple installing process and flexible high-loaded system.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I will update proposal and take into consideration your suggestion about third part of test-suite.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Elena.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<a href="mailto:chris.matthews@apple.com" target="_blank">chris.matthews@apple.com</a> [<a href="mailto:chris.matthews@apple.com" target="_blank">mailto:chris.matthews@apple.com</a>]
<br>
<b>Sent:</b> Monday, April 25, 2016 8:06 PM<br>
<b>To:</b> Elena Lepilkina <<a href="mailto:Elena.Lepilkina@synopsys.com" target="_blank">Elena.Lepilkina@synopsys.com</a>><br>
<b>Cc:</b> llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br>
<b>Subject:</b> Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test parameterization<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I am really torn about this.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">When I implemented the regression tracking stuff recently, it really showed me how badly we are scaling. On our production server, the run ingestion can take well over 100s. Time is mostly spent in FieldChange generation and regression
grouping. Both have to access a lot of recent samples. This is not the end of the world, because it runs in a background process. Where this really sucks is when a regression has a lot indicators. The web interface renders these in a graph, and just trying
to pull down 100 graphs worth of data kills the server. I ended up limiting those to a max of 10 datasets, and even that takes 30s to load.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">So I do think we need some improvements to the scalability.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">LNT usage is spread between two groups. Users who setup big servers, with Postgres and apache/Gunicorn. For those uses I think a NoSQL is the way to go. However, our second (and probably more common) user, is the people running little
instance on their own machine to do some local compiler benchmarking. Their setup process needs to be dead simple, and I think requiring a NoSQL database to be setup on their machine first is a no starter. Like we do with sqlite, I think we need a transparent
fall back for people who don’t have a NoSQL database.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Would it be helpful to anyone if I got a dump of the <a href="http://llvm.org" target="_blank">
llvm.org</a> LNT Postgres database? It is a good dataset big dataset to test with, and I assume everyone is okay with it being public, since the LNT server already is.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Apr 25, 2016, at 4:33 AM, Elena Lepilkina via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
</div>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Elena
Lepilkina<span> </span><br>
<b>Sent:</b><span> </span>Monday, April 25, 2016 2:33 PM<br>
<b>To:</b><span> </span>'James Molloy' <<a href="mailto:james@jamesmolloy.co.uk" target="_blank"><span style="color:purple">james@jamesmolloy.co.uk</span></a>>; Kristof Beyls <<a href="mailto:Kristof.Beyls@arm.com" target="_blank"><span style="color:purple">Kristof.Beyls@arm.com</span></a>>;
Mehdi Amini <<a href="mailto:mehdi.amini@apple.com" target="_blank"><span style="color:purple">mehdi.amini@apple.com</span></a>><br>
<b>Cc:</b><span> </span>nd <<a href="mailto:nd@arm.com" target="_blank"><span style="color:purple">nd@arm.com</span></a>>; Matthias Braun <<a href="mailto:matze@braunis.de" target="_blank"><span style="color:purple">matze@braunis.de</span></a>><br>
<b>Subject:</b><span> </span>RE: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test parameterization</span><u></u><u></u></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Hi everyone,</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thank you for your answer. BLOB format adds some more actions for working with metrics. We know that<span> </span></span>ComparisonResult class
makes analysis work. But it gets all metrics by request from database, we will need additional time for work with fields during analysis in ComparisonResult class. May be it will be better to do one Sample table for each testsuite, as it was suggested before.
It should be more quickly, shouldn’t it? Moreover, next wished LNT changes will need getting some metrics separately and BLOB format will add some delay in time for queries.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">As we see now problem of performance is actual, because time for rendering graph page is about 3 minutes.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><image001.png><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">So maybe it will be better to start working with NoSql databases? I made a small prototype with TestSuite, TestSuiteFields, Test, Run and Sample tables for getting time metrics. It works quickly. And using NoSQL helps solve problems with
different fields for samples metrics fields. Then it will be possible to store different metrics for different testsuites in one table.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">What do you think about this proposal?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I used MongoDB, but I know that there is NoSQL extension for PostgresSQL with JSONB fields which are more<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">effective than JSON-encoded BLOB, because it can be included in queries very simply and let use indexes.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">About proposal that not all metrics should be shown. It can be added as a field in JSON in .fields file, which describes fields getted from test-suite. To see other metrics user should choose them with checkboxes in view options. Will be
this solution suitable?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">We can make as you wrote<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">“I'd also suggest that if we're adding many more metrics to a test, we should create a "test sample information" page that the test link goes to instead of just the graph. This page could contain all counter/metric data, historic sparklines,
the full graph and profiling links.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">”<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">But the render time of this page will be too great because of graph render time. In my opinion, some users wouldn’t like to wait so long for see some additional metrics.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks for your suggestions,</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Elena.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">llvm-dev
[<a href="mailto:llvm-dev-bounces@lists.llvm.org" target="_blank"><span style="color:purple">mailto:llvm-dev-bounces@lists.llvm.org</span></a>]<span> </span><b>On Behalf Of<span> </span></b>James Molloy via llvm-dev<br>
<b>Sent:</b><span> </span>Monday, April 25, 2016 12:43 PM<br>
<b>To:</b><span> </span>Kristof Beyls <<a href="mailto:Kristof.Beyls@arm.com" target="_blank"><span style="color:purple">Kristof.Beyls@arm.com</span></a>>; Mehdi Amini <<a href="mailto:mehdi.amini@apple.com" target="_blank"><span style="color:purple">mehdi.amini@apple.com</span></a>><br>
<b>Cc:</b><span> </span>llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank"><span style="color:purple">llvm-dev@lists.llvm.org</span></a>>; nd <<a href="mailto:nd@arm.com" target="_blank"><span style="color:purple">nd@arm.com</span></a>>; Matthias
Braun <<a href="mailto:matze@braunis.de" target="_blank"><span style="color:purple">matze@braunis.de</span></a>><br>
<b>Subject:</b><span> </span>Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test parameterization</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">Hi Sergey, Elena,<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Firstly, thanks for this RFC. It's great to see more people actively using and modifying LNT and the test metrics support in general is rather weak currently.<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Metrics<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">-------<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">I agree with Daniel and Kristof that your proposed schema changes have the potential to make many queries extremely slow. Certainly for the metrics enhancements, I don't see a reason why we need such a radical change in schema.<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">To add custom metrics on the fly, we need to change the schema for the Sample table. Currently this consists of a column for each metric, but actually we never ever query those metric values. We never query for example for "all failing
tests in a run" - when we do analyses we use the ComparisonResult class which reads *all* samples from the database for a run and performs the analysis entirely in Python.<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Therefore, having a semi-structured format where some fields are first-class columns and the rest are in a JSON-encoded BLOB (as Daniel suggests) seems totally acceptable. There is certainly an argument now that we're using the wrong backend
storage solution and that a key-value store might be more suitable, but that's a very invasive change and I don't think we've reached the point where we need to force a move from the simplicity of SQLite.<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Adding an extra BLOB column would be easy - there would just need to be logic in testsuitedb.py for reading and writing it - the Sample model class would expose the JSON-encoded fields as normal python fields so the rest of LNT would be
isolated from this change.<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">But I think this is a small detail compared to the bigger problem of how to effectively *display* all this new data. Currently every new metric gets its own separate table in the report/run views, and this does not scale well at all.<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">I think we need some more concepts in the metric system to make it scaleable:<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> * What "attribute" of the test is this metric measuring? For example, both "exec_time" and "score" measure the same attribute; performance of the generated code. It's superfluous to have them displayed in separate tables. However mem_size
and compile_time both measure completely different aspects of the test.<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> * Is this metric useful to display at the top level? or should it only be exposed when more data about a test result is requested?<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> * An example of this is the pass statistics. I don't want my daily report view cluttered by the time spent in register allocation for every test! OK, this is useful information when debugging a problem, but it should be available when
requested rather than by default.<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">An example of why we need the above is your screenshots in your google doc. I'm looking at the last screenshot, and it's incredibly difficult to read and get useful information out of.<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">I'd also suggest that if we're adding many more metrics to a test, we should create a "test sample information" page that the test link goes to instead of just the graph. This page could contain all counter/metric data, historic sparklines,
the full graph and profiling links.<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Cheers,<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">James<u></u><u></u></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Fri, 22 Apr 2016 at 10:17 Kristof Beyls via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank"><span style="color:purple">llvm-dev@lists.llvm.org</span></a>> wrote:<u></u><u></u></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">On 22 Apr 2016, at 11:14, Mehdi Amini <<a href="mailto:mehdi.amini@apple.com" target="_blank"><span style="color:purple">mehdi.amini@apple.com</span></a>> wrote:<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt;text-align:start;word-spacing:0px">
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
On Apr 22, 2016, at 12:45 AM, Kristof Beyls via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank"><span style="color:purple">llvm-dev@lists.llvm.org</span></a>> wrote:</span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"> </span><u></u><u></u></p>
</div>
<div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
On 21 Apr 2016, at 17:44, Sergey Yakoushkin <<a href="mailto:sergey.yakoushkin@gmail.com" target="_blank"><span style="color:purple">sergey.yakoushkin@gmail.com</span></a>> wrote:</span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"> </span><u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Hi Kristof,</span><u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"> </span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#500050"> The way we use LNT, we would run different configuration (e.g. -O3 vs -Os) as different "machines" in LNT's model.</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"> </span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">O2/O3 is indeed bad example. We're also using different machines for Os/O3 - such parameters apply to all tests and we don't propose major changes.</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Elena was only extending LNT interface a bit to ease LLVM-testsuite execution with different compiler or HW flags.</span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"> </span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Oh I see, this boils down to extending the lnt runtest interface to be able to specify a set of configurations, rather than a single configuration and making</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">sure configurations get submitted under different machine names? We kick off the different configuration runs through a script invoking lnt runtest multiple</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">times. I don't see a big problem with extending the lnt runtest interface to do this, assuming it doesn't break the underlying concepts assumed throughout</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">LNT. Maybe the only downside is that this will add even more command line options to lnt runtest, which already has a lot (too many?) command line</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">options.</span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"> </span><u></u><u></u></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Maybe some changes are required to analyze and compare metrics between "machines": e.g. code size/performance between Os/O2/O3.</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Do you perform such comparisons?</span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"> </span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">We typically do these kinds of comparisons when we test our patches pre-commit, i.e. comparing for example '-O3' with '-O3 'mllvm -enable-my-new-pass'.</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">To stick with the LNT concepts, tests enabling new passes are stored as a different "machine".</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">The only way I know to be able to do a comparison between runs on 2 different "machine"s is to manually edit the URL for run vs run comparison</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">and fill in the runids of the 2 runs you want to compare.</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">For example, the following URL is a comparison of green-dragon-07-x86_64-O3-flto vs green-dragon-06-x86_64-O0-g on the public <a href="http://llvm.org/perf" target="_blank"><span style="color:purple">llvm.org/perf</span></a> server:</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><a href="http://llvm.org/perf/db_default/v4/nts/70644?compare_to=70634" target="_blank"><span style="color:purple">http://llvm.org/perf/db_default/v4/nts/70644?compare_to=70634</span></a></span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">I had to manually look up and fill in the run ids 70644 and 70634.</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">It would be great if there was a better way to be able to do these kind of comparisons - i.e. not having to manually fill in run ids, but having a webui to easily find and
pick the runs you want to compare.</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">(As an aside: I find it intriguing that the URL above suggests that there are quite a few cases where "-O0 -g" produces faster code than "-O3 -flto").</span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"> </span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Can you be more explicit which ones? I don't see any regression (other than compared to the baseline, or on the compile time).</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"> </span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">-- </span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Mehdi</span><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">D'Oh! I was misinterpreting the compile time differences as execution time differences. Indeed, there is no unexpected result in there.<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Sorry for the noise!<u></u><u></u></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Kristof<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal">_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank"><span style="color:purple">llvm-dev@lists.llvm.org</span></a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank"><span style="color:purple">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</span></a><u></u><u></u></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a></span><u></u><u></u></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div>