[LLVMdev] Making test-suite regression detection easier

[Tanya M. Lattner]

>> Would it be useful for the community to have a website where one can
>> submit the results of a nightly test style run of the test-suite and
>> then have it tell you which tests regressed based upon a recently
>> known "good" run of the nightly test for your target?
>
> The results across machines are rarely comparable. There are just too
> many variables (machine, operating system, compiler version, etc.).  The
> best thing to do is to run a nightly test on your own machine and then
> compare that with the test results you get on your "non-public" version.
> Doing so will give you the exact delta.
>
> So far we've been doing this manually but I think you're right that its
> time to automate it.  To support this better what we need is:
>
> 1. NightlyTest.pl that can use an already checked out tree
> 2. Ability to automate the results delta.

I believe the NightlyTester can already work on a checked out tree, but 
you may have to manually update it. That could easily be added.

On a side note, I've been working on a new implementation of our nightly 
tester results web page. I've designed a new database and I'm in process 
or revamping the php scripts that are used to view the results. Once I'm 
done, it should be pretty easy to see regressions between runs. I'll also 
be adding some supporting stuff so that you can easily create your own 
database and keep everything internal. The nightly tester supports 
redirecting the output to another server, but we've never published the 
database layout.

-Tanya