[LLVMdev] Validation Buildbot is Up!

[David Greene]

On Thursday 29 January 2009 03:18, Bill Wendling wrote:
> On Jan 28, 2009, at 8:25 PM, Tanya Lattner wrote:
> > On Jan 28, 2009, at 3:46 PM, David Greene wrote:
> >> On Wednesday 28 January 2009 15:59, Tanya Lattner wrote:
> >>> On Jan 28, 2009, at 12:18 PM, David Greene wrote:
> >>>> I have a buildbot operating to do validating builds of llvm up at
> >>>>
> >>>> http://obbligato.org:8080
> >>>>
> >>>> My DSL has been stable enough for the past few months for me to
> >>>> feel comfortable hosting the buildbot there.
> >>>
> >>> We had a discussion in the past on what validate means. Did you ever
> >>> formalize that? It might be good if you posted (on your website?)
> >>> what
> >>> specific criteria you are using to declare a build validated. Or is
> >>> this just a normal build bot?
> >>
> >> We had a long discussion about this.  I'll post some information but
> >> the buildbot essentially does this:
> >>
> >> - Build an LLVM without llvm-gcc
> >> - Run LLVM tests
> >> - Build llvm-gcc pointing to the newly-build LLVM
> >> - Rebuild LLVM pointing to the newly-build llvm-gcc
> >> - Run LLVM tests
> >> - Run llvm-test
> >>
> >> If everything passes for debug, release and paranoid
> >> (--enable-expensive-checks) we'll consider LLVM validated
> >> for that target.
> >
> > As I mentioned before, I'm curious what reference point you are using
> > to determine "pass" for llvm-test.
>
> Here's my idea for this:
>
> - The first criteria is "does it compile and run without regressions
> from the last run?".
> - The second criteria is "does it run significantly slower than the
> previous run.

I'm not sure performance regressions should be a requirement for validation.  
Validation is aobut functional correctness.  It gives developers confidence 
that they can update to a particular revision without breaking anything.

Performance regression is important, no doubt, but I'd make it a condition
for release, not validation.

                                            -Dave