[LLVMdev] 1 Week Before 2.0 Branch Creation

[Ralph Corderoy]

Hi Chris,

> I think that this is a great idea.  However, instead of picking up yet
> another setting of testing infrastructure, I think we should make what
> we have already (the nightly testers) better.  In particular, the main
> page of the tester:
> http://llvm.org/nightlytest/
> 
> Already captures a lot of this: it tells you the number of unexpected
> failures, whether or not the build succeeded etc.  You can even drill
> down to a specific machine, e.g.:
> http://llvm.org/nightlytest/machine.php?machine=120
> 
> I see several problems with this, all of which are solvable:
> 
> 1. The tester script doesn't update and rebuild the CFE, so often you
> get failures due to an out-of-date CFE.
> 2. There is no way to group by architecture, target-triple, etc. 
> 3. Minor stupid stuff: on the per-machine page, you can click on the
> test failures number and go to the list of unexpected failures, but
> you can't do that on the main page.

Are you aware of buildbot?  It's quite widely used and flexible.

    http://buildbot.sourceforge.net/

    "By automatically rebuilding and testing the tree each time
    something has changed, build problems are pinpointed quickly, before
    other developers are inconvenienced by the failure. The guilty
    developer can be **identified and harassed without human
    intervention**.  By running the builds on a variety of platforms,
    developers who do not have the facilities to test their changes
    everywhere before checkin will at least know shortly afterwards
    whether they have broken the build or not. Warning counts, lint
    checks, image size, compile time, and other build parameters can be
    tracked over time, are more visible, and are therefore easier to
    improve."

    ** -- my emphasis.

I'd suggest at least one machine given over to always building whenever
anything changes, and have the other nightly volunteers as now.  Using
just nightly volunteers isn't great because breakage is noticed too late
and can affect too many people by then.

An example of the output is http://www.python.org/dev/buildbot/trunk/

Cheers,


Ralph.