<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Oct 31, 2017, at 11:08 AM, Rafael Avila de Espindola via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Right now performance tracking in lld is a manual and very laborious<br class="">process.<br class=""><br class="">I decided to take some time off main development to automate it a<br class="">bit.<br class=""><br class="">The current state is that we have a bunch of test in<br class=""><a href="https://s3-us-west-2.amazonaws.com/linker-tests/lld-speed-test.tar.xz" class="">https://s3-us-west-2.amazonaws.com/linker-tests/lld-speed-test.tar.xz</a>. Each<br class="">test is a program that lld can link (clang, chrome, firefox, etc).<br class=""><br class="">Right now there is just a hackish run.sh that links every program with<br class="">two versions of lld to compare the performance. Some programs are also<br class="">linked in different modes. For example, we compare chrome with and<br class="">without icf.<br class=""><br class="">I want to improve this in two ways:<br class=""><br class="">* Make the directory structure uniform. What I am currently prototyping<br class=""> is that each program is its own directory and it can have multiple<br class=""> response*.txt files. Each response file is an independent test. The<br class=""> reason for allowing multiple response files is to allow for variants:<br class=""> * response.txt<br class=""> * response-icf.txt<br class=""> * response-gc.txt<br class=""><br class="">* Instead of just comparing lld's run time, parse and save various<br class=""> metrics to a database.<br class=""><br class="">The database hierarchy would be:<br class=""><br class="">So for each llvm revision there will be multiple benchmarks (chrome,<br class="">chrome-icf, clang).<br class=""><br class="">For each benchmark, there will be multiple metrics (lld runtime, branches,<br class="">output size, etc).<br class=""><br class="">Some output metrics will include multiple measurements. The output size<br class="">should always be the same, but we can have multiple runs with slightly<br class="">different time for example.<br class=""><br class="">Not too surprisingly, the above structure is remarkably similar to what lnt<br class="">uses:<br class=""><a href="http://llvm.org/docs/lnt/importing_data.html#importing-data-in-a-text-file" class="">http://llvm.org/docs/lnt/importing_data.html#importing-data-in-a-text-file</a><br class=""><br class="">So my idea is to first write a python script that runs all the<br class="">benchmarks in the above structure and submits the results to a lnt<br class="">server. This would replace the existing run.sh.<br class=""><br class="">For comparing just two revisions locally, this will be a bit more work.<br class=""><br class="">But it should allow us to setup a bot that continually submits lld<br class="">performance results.<br class=""><br class="">For those working on lld, does the above sound like a good idea?<br class=""><br class="">For those working on lnt, is it a good match for the above? The main use<br class="">case I would like is to build graphs of various metrics over llvm<br class="">revisions. For example:<br class=""><br class="">* chrome's output size over the last 1000 llvm revisions.<br class="">* firefox link time (with error bars) over that last 1000 llvm revisions.<br class=""></div></div></blockquote><div>This sounds good from the LNT side. Indeed you should be good to go with the default 'nts' schema. And just submit to the server.</div><div><br class=""></div><div>For reference this how a submission can look in python (you have to wrap the data in one more level of records with the 'input_data' thing).</div><div>You should be able to just submit away to </div><div><br class=""></div><div><div> # Submit the resulting data to an LNT server.</div><div> jsondata = json.dumps(result, sort_keys=True)</div><div> postdata = {'input_data': json.dumps(result, sort_keys=True)}</div><div> data = urllib.urlencode(postdata)</div><div> try:</div><div> opener = urllib2.build_opener(urllib2.HTTPHandler)</div><div> response = urllib2.urlopen(config.url, data=data)</div><div> responsedata = response.read()</div><div><br class=""></div><div> try:</div><div> resultdata = json.loads(responsedata)</div><div> result_url = resultdata.get('result_url')</div><div> if result_url is not None:</div><div> sys.stdout.write("%s\n" % result_url)</div><div> except:</div><div> sys.stderr.write("Unexpected server response:\n" + responsedata)</div><div> except urllib2.HTTPError as e:</div><div> sys.stderr.write("PUT Request failed with code %s\n" % e.code)</div><div> sys.stderr.write("%s\n" % e.reason)</div><div> sys.exit(1)</div><div><br class=""></div><div>With that you should be able to just submit away to <a href="http://lnt.llvm.org/db_default/v4/nts/submitRun" class="">http://lnt.llvm.org/db_default/v4/nts/submitRun</a></div><div><br class=""></div><div>Drawing those two graphs should also just work.</div><div><br class=""></div><div>- Matthias</div><div><br class=""></div></div></div></body></html>