Using lnt to track lld performance

[Matthias Braun via llvm-commits]

> On Oct 31, 2017, at 11:08 AM, Rafael Avila de Espindola via llvm-commits <llvm-commits at lists.llvm.org> wrote:
> 
> Right now performance tracking in lld is a manual and very laborious
> process.
> 
> I decided to take some time off main development to automate it a
> bit.
> 
> The current state is that we have a bunch of test in
> https://s3-us-west-2.amazonaws.com/linker-tests/lld-speed-test.tar.xz. Each
> test is a program that lld can link (clang, chrome, firefox, etc).
> 
> Right now there is just a hackish run.sh that links every program with
> two versions of lld to compare the performance. Some programs are also
> linked in different modes. For example, we compare chrome with and
> without icf.
> 
> I want to improve this in two ways:
> 
> * Make the directory structure uniform. What I am currently prototyping
>  is that each program is its own directory and it can have multiple
>  response*.txt files. Each response file is an independent test. The
>  reason for allowing multiple response files is to allow for variants:
>  * response.txt
>  * response-icf.txt
>  * response-gc.txt
> 
> * Instead of just comparing lld's run time, parse and save various
>  metrics to a database.
> 
> The database hierarchy would be:
> 
> So for each llvm revision there will be multiple benchmarks (chrome,
> chrome-icf, clang).
> 
> For each benchmark, there will be multiple metrics (lld runtime, branches,
> output size, etc).
> 
> Some output metrics will include multiple measurements. The output size
> should always be the same, but we can have multiple runs with slightly
> different time for example.
> 
> Not too surprisingly, the above structure is remarkably similar to what lnt
> uses:
> http://llvm.org/docs/lnt/importing_data.html#importing-data-in-a-text-file
> 
> So my idea is to first write a python script that runs all the
> benchmarks in the above structure and submits the results to a lnt
> server. This would replace the existing run.sh.
> 
> For comparing just two revisions locally, this will be a bit more work.
> 
> But it should allow us to setup a bot that continually submits lld
> performance results.
> 
> For those working on lld, does the above sound like a good idea?
> 
> For those working on lnt, is it a good match for the above? The main use
> case I would like is to build graphs of various metrics over llvm
> revisions. For example:
> 
> * chrome's output size over the last 1000 llvm revisions.
> * firefox link time (with error bars) over that last 1000 llvm revisions.
This sounds good from the LNT side. Indeed you should be good to go with the default 'nts' schema. And just submit to the server.

For reference this how a submission can look in python (you have to wrap the data in one more level of records with the 'input_data' thing).
You should be able to just submit away to 

    # Submit the resulting data to an LNT server.
    jsondata = json.dumps(result, sort_keys=True)
    postdata = {'input_data': json.dumps(result, sort_keys=True)}
    data = urllib.urlencode(postdata)
    try:
        opener = urllib2.build_opener(urllib2.HTTPHandler)
        response = urllib2.urlopen(config.url, data=data)
        responsedata = response.read()

        try:
            resultdata = json.loads(responsedata)
            result_url = resultdata.get('result_url')
            if result_url is not None:
                sys.stdout.write("%s\n" % result_url)
        except:
            sys.stderr.write("Unexpected server response:\n" + responsedata)
    except urllib2.HTTPError as e:
        sys.stderr.write("PUT Request failed with code %s\n" % e.code)
        sys.stderr.write("%s\n" % e.reason)
        sys.exit(1)

With that you should be able to just submit away to http://lnt.llvm.org/db_default/v4/nts/submitRun <http://lnt.llvm.org/db_default/v4/nts/submitRun>

Drawing those two graphs should also just work.

- Matthias

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20171031/d881dae9/attachment.html>