[LLVMdev] Validation Buildbot is Up!

[Bill Wendling]

On Jan 28, 2009, at 8:25 PM, Tanya Lattner wrote:

> On Jan 28, 2009, at 3:46 PM, David Greene wrote:
>
>> On Wednesday 28 January 2009 15:59, Tanya Lattner wrote:
>>> On Jan 28, 2009, at 12:18 PM, David Greene wrote:
>>>> I have a buildbot operating to do validating builds of llvm up at
>>>>
>>>> http://obbligato.org:8080
>>>>
>>>> My DSL has been stable enough for the past few months for me to
>>>> feel comfortable hosting the buildbot there.
>>>
>>> We had a discussion in the past on what validate means. Did you ever
>>> formalize that? It might be good if you posted (on your website?)
>>> what
>>> specific criteria you are using to declare a build validated. Or is
>>> this just a normal build bot?
>>
>> We had a long discussion about this.  I'll post some information but
>> the buildbot essentially does this:
>>
>> - Build an LLVM without llvm-gcc
>> - Run LLVM tests
>> - Build llvm-gcc pointing to the newly-build LLVM
>> - Rebuild LLVM pointing to the newly-build llvm-gcc
>> - Run LLVM tests
>> - Run llvm-test
>>
>> If everything passes for debug, release and paranoid
>> (--enable-expensive-checks) we'll consider LLVM validated
>> for that target.
>>
> As I mentioned before, I'm curious what reference point you are using
> to determine "pass" for llvm-test.
>
Here's my idea for this:

- The first criteria is "does it compile and run without regressions  
from the last run?".
- The second criteria is "does it run significantly slower than the  
previous run.
   - Defining what is "significantly slower" is the hard part here.  
Some statistical analysis needs to be done on the data; more than just  
simple ratios. (I'd like to see these analyses done on our nightly  
tests as well.) They can tell us if: A) the change is significant, and  
B) if we're gradually regressing over time.

The second criteria is *much* more difficult, of course.

>>>> It's not yet sending messages to llvmdev.  I want to do some more
>>>> testing of the setup before I turn it loose on everyone.  But you
>>>> can
>>>> go take a look to see how it operates.
>>>
>>> I don't think llvm-dev is the right place to be sending mail to.
>>> Maybe
>>> the testresults list? What mails do you plan to send and how
>>> frequent?
>>
>> The buildbot kicks off every 100 commits or so.  There are three
>> builds for
>> each target (the only buildslaves we have right now are for x86_64-
>> linux and
>> i686-linux).  Each one of those will send an e-mail.
>> I'm fine sending it to testresults if people pay attention.  I know
>> that I
>> don't read testresults regularly because there are a lot of test
>> runs I
>> don't care about.
>>
>
> I still don't think llvm-dev is the right place. We don't want that
> mailing list to get cluttered with buildbot test results. You can send
> to llvm-test and use filtering if you want to ignore the other  
> results.
>
I think that "testresults" is fine for now. That's where the other  
buildbots send their results.

-bw