[LLVMdev] 1 Week Before 2.0 Branch Creation

[Chris Lattner]

On Sat, 5 May 2007, David Greene wrote:
> There's the llvm-testresults list, but I find it less than fully useful
> because it's not immediately obvious from scanning message subjects if
> there's been a test failure.  It's a lot of messages to wade through and
> read to get this information.

Right.

> What about a Tinderbox-like setup where we could consult a web page to
> see the current status of the repository?  Boost has a nice setup:
>
> http://engineering.meta-comm.com/boost-regression/CVS-RC_1_34_0/developer/index_release.html
>
> It's probably more complex than what we need.  Maybe we just need a
> page grouping each test under it's suite and marking the result on
> each architecture.  Something like this:

I think that this is a great idea.  However, instead of picking up yet 
another setting of testing infrastructure, I think we should make what we 
have already (the nightly testers) better.  In particular, the main page 
of the tester:
http://llvm.org/nightlytest/

Already captures a lot of this: it tells you the number of unexpected 
failures, whether or not the build succeeded etc.  You can even drill down 
to a specific machine, e.g.:
http://llvm.org/nightlytest/machine.php?machine=120

I see several problems with this, all of which are solvable:

1. The tester script doesn't update and rebuild the CFE, so often you get
    failures due to an out-of-date CFE.
2. There is no way to group by architecture, target-triple, etc. 
3. Minor stupid stuff: on the per-machine page, you can click on the test
    failures number and go to the list of unexpected failures, but you
    can't do that on the main page.

-Chris

-- 
http://nondot.org/sabre/
http://llvm.org/