[LNT][PATCH] extract configuration from nightly tests

Chris Matthews chris.matthews at apple.com
Fri Aug 9 14:29:27 PDT 2013


Hi Renato,

This patch is a little bit simpler than that.  There are two types of rerunning that I want to support.  First when an LNT run is finished and after the initial submission, rerun any benchmarks which the server reports back as regressed. Second, provide a “lnt rerun” command which reruns some/all benchmarks in a sandbox to gather extra data at a later date. This second one will be future work.  Both of these are not intended to control for the environment, just to basically execute the test suite in the same manner as the original run, but on a per test basis.

I have implemented the first type of rerunning already, unfortunately the patch was a pretty messy.  One of the problem is that most of the logic for determining paths, test suite make options etc, is baked right into the run functionality.  Since all of these properties are just derived in some way from the command line options, it made sense to pull the configuration out.  Now that it is out, the rerun code can access it directly.

Turns out that all that is needed for a rerun is a serialized timestamp of the original run start time, and the original lnt command line options. No plans right now to collect system information though.

In terms of validation, I am also working on a separate patch which checks parts of the “Run Info” section of the run reports to make sure they match previous runs on the same machine. I think this would be the ideal place too add in library information in the future.  This would not catch changes in the system during reruns, but it would between runs.

Both Daniel and myself have some scripts for helping with these sort of performance comparisons you are talking about, basically just looking at A/B situations and comparing the performance that way.  Daniel even just committed this: r188100 - [lnttool] Add a view-comparison command, for quickly viewing results. 

Chris Matthews
chris.matthews at apple.com



On Aug 9, 2013, at 4:51 AM, Renato Golin <renato.golin at linaro.org> wrote:

> On 8 August 2013 21:55, Chris Matthews <chris.matthews at apple.com> wrote:
> This is the first of several patches to refactor the LNT nt test, to make it more amenable to single test reruns. 
> 
> This patch extracts common configuration options into a separate stateless object so that they can be accessed again for rerunning.  Common functionality from the run is also extracted so that it can be run more than once.
> 
> Hi Chris,
> 
> My Python skills are seriously lacking, so I'll only comment on the description of the functionality.
> 
> Is the idea to save this context with with the runs themselves, so that you can de-serialize it back to object and run "the same configuration" back again? I'm assuming you're controlling everything on the LLVM side, but not so much on the system side (library and platform compiler versions, architecture, etc).
> 
> In the case you could save the system information, maybe you could warn the user that something had changed, but you can't install/change it yourself.
> 
> For instance, if the users tries to compare runs A and B, both on x86_64+Linux, but A has a different glibc version than B, you could warn saying that some results might be different due to library version mismatch.
> 
> Another usage would be to compare the relative performance difference between two different architectures from the same change in library. 
> 
> So, if in the first case you found a difference in x86_64, but got a warning due to libraries, you can go and check other architectures that had the same library change, to see if there was the same relative difference in performance *just* due to library change.
> 
> cheers,
> --renato

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20130809/8d196261/attachment.html>


More information about the llvm-commits mailing list