<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">(changing mailing list from llvm-commits to llvm-dev)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">llvm-commits <llvm-commits-bounces@lists.llvm.org> on behalf of Matthias Braun via llvm-commits <llvm-commits@lists.llvm.org><br>
<b>Reply-To: </b>Matthias Braun <matze@braunis.de><br>
<b>Date: </b>Tuesday, October 31, 2017 at 11:33 AM<br>
<b>To: </b>Rafael Avila de Espindola <rafael.espindola@gmail.com><br>
<b>Cc: </b>llvm-commits <llvm-commits@lists.llvm.org>, James Henderson <jh7370@my.bristol.ac.uk><br>
<b>Subject: </b>Re: Using lnt to track lld performance<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Oct 31, 2017, at 11:08 AM, Rafael Avila de Espindola via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Right now performance tracking in lld is a manual and very laborious<br>
process.<br>
<br>
I decided to take some time off main development to automate it a<br>
bit.<br>
<br>
The current state is that we have a bunch of test in<br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__s3-2Dus-2Dwest-2D2.amazonaws.com_linker-2Dtests_lld-2Dspeed-2Dtest.tar.xz&d=DwMFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=o3kDXzdBUE3ljQXKeTWOMw&m=ZHTxQwyvouSBeubS2n1CjEMcFFYh1w6aBvvkFtJwnQQ&s=7llNHnFLqvPoq6LUmtDRf92EhiQn_xKgpF3wf9ahm9o&e=">https://s3-us-west-2.amazonaws.com/linker-tests/lld-speed-test.tar.xz</a>.
Each<br>
test is a program that lld can link (clang, chrome, firefox, etc).<br>
<br>
Right now there is just a hackish run.sh that links every program with<br>
two versions of lld to compare the performance. Some programs are also<br>
linked in different modes. For example, we compare chrome with and<br>
without icf.<br>
<br>
I want to improve this in two ways:<br>
<br>
* Make the directory structure uniform. What I am currently prototyping<br>
is that each program is its own directory and it can have multiple<br>
response*.txt files. Each response file is an independent test. The<br>
reason for allowing multiple response files is to allow for variants:<br>
* response.txt<br>
* response-icf.txt<br>
* response-gc.txt<br>
<br>
* Instead of just comparing lld's run time, parse and save various<br>
metrics to a database.<br>
<br>
The database hierarchy would be:<br>
<br>
So for each llvm revision there will be multiple benchmarks (chrome,<br>
chrome-icf, clang).<br>
<br>
For each benchmark, there will be multiple metrics (lld runtime, branches,<br>
output size, etc).<br>
<br>
Some output metrics will include multiple measurements. The output size<br>
should always be the same, but we can have multiple runs with slightly<br>
different time for example.<br>
<br>
Not too surprisingly, the above structure is remarkably similar to what lnt<br>
uses:<br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__llvm.org_docs_lnt_importing-5Fdata.html-23importing-2Ddata-2Din-2Da-2Dtext-2Dfile&d=DwMFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=o3kDXzdBUE3ljQXKeTWOMw&m=ZHTxQwyvouSBeubS2n1CjEMcFFYh1w6aBvvkFtJwnQQ&s=UvmQM64VT6LWp9kq5002H_BM9Alwc4cP75cjB1THdqk&e=">http://llvm.org/docs/lnt/importing_data.html#importing-data-in-a-text-file</a><br>
<br>
So my idea is to first write a python script that runs all the<br>
benchmarks in the above structure and submits the results to a lnt<br>
server. This would replace the existing run.sh.<br>
<br>
For comparing just two revisions locally, this will be a bit more work.<br>
<br>
But it should allow us to setup a bot that continually submits lld<br>
performance results.<br>
<br>
For those working on lld, does the above sound like a good idea?<br>
<br>
For those working on lnt, is it a good match for the above? The main use<br>
case I would like is to build graphs of various metrics over llvm<br>
revisions. For example:<br>
<br>
* chrome's output size over the last 1000 llvm revisions.<br>
* firefox link time (with error bars) over that last 1000 llvm revisions.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">This sounds good from the LNT side. Indeed you should be good to go with the default 'nts' schema. And just submit to the server.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">For reference this how a submission can look in python (you have to wrap the data in one more level of records with the 'input_data' thing).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">You should be able to just submit away to <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> # Submit the resulting data to an LNT server.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> jsondata = json.dumps(result, sort_keys=True)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> postdata = {'input_data': json.dumps(result, sort_keys=True)}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> data = urllib.urlencode(postdata)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> try:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> opener = urllib2.build_opener(urllib2.HTTPHandler)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> response = urllib2.urlopen(config.url, data=data)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> responsedata = response.read()<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> try:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> resultdata = json.loads(responsedata)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> result_url = resultdata.get('result_url')<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> if result_url is not None:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> sys.stdout.write("%s\n" % result_url)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> except:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> sys.stderr.write("Unexpected server response:\n" + responsedata)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> except urllib2.HTTPError as e:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> sys.stderr.write("PUT Request failed with code %s\n" % e.code)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> sys.stderr.write("%s\n" % e.reason)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> sys.exit(1)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">With that you should be able to just submit away to <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lnt.llvm.org_db-5Fdefault_v4_nts_submitRun&d=DwMFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=o3kDXzdBUE3ljQXKeTWOMw&m=ZHTxQwyvouSBeubS2n1CjEMcFFYh1w6aBvvkFtJwnQQ&s=GvKzJ3ltYWyj8KHinUNKUkjTDt1-zfL45rQbO76o0j0&e=">http://lnt.llvm.org/db_default/v4/nts/submitRun</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Drawing those two graphs should also just work.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">- Matthias<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>