[LLVMdev] Validation Buildbot is Up!

[David Greene]

On Thursday 29 January 2009 11:01, David Greene wrote:

> > - The first criteria is "does it compile and run without regressions
> > from the last run?".
> > - The second criteria is "does it run significantly slower than the
> > previous run.
>
> I'm not sure performance regressions should be a requirement for
> validation. Validation is aobut functional correctness.  It gives
> developers confidence that they can update to a particular revision without
> breaking anything.
>
> Performance regression is important, no doubt, but I'd make it a condition
> for release, not validation.

Thinking about this some more, I could make a case that performance 
regressions might fit into another type of validation - performance 
validation.  We could have two types of validation: functional and 
performance.  People could individually decide which they care about.

The key is to keep things simple for buildbot.  We don't want to write a
bunch of complex test code in buildbot to do the performance
checks / statistical analysis Bill is talking about.  Buildbot wants to
see PASS / FAIL / XFAIL, etc.  So I think we'd need a nice DejaGNU 
suite that does performance regression testing.  To my knowledge, we
don't have one yet.  So I'm going to leave it to others to develop such
a suite and if and when one pops up we can look at integrating it into
the buildbot validator.

We're all learning here and I know that as we go along we'll tweak the way 
this works.  I'm not worried about that.  :)

                                       -Dave