[LLVMdev] Using the test suite to benchmark patches

[Tanya M. Lattner]

>> On May 21, 2008, at 8:09 AM, Matthijs Kooijman wrote:
>>> Any thoughts or suggestions on how to do this testing in a
>>> structured manner?
>>
>> I think that if what you're doing is sound, and you get the results
>> you want, say, on compiling something like gcc with it and others
>> review the basic idea (hi evan or chris) and like it, just checking it
>> in and watching the performance numbers for the next day seems
>> reasonable to me.  You can always revert the patch if there are
>> unexpected downsides that make it not worth while.
>
> It depends on the scope of the change.  If it is a relatively minor
> change, getting the code approved, testing it for correctness, and adding
> a regression test is sufficient.  If it is major (adding a new pass,
> significantly changing pass ordering etc) then the bar is much higher.
>
> We don't have a great way of diffing performance runs, other than the
> nightly tester.  Devang has an experimental "opt-beta" mode that can be
> used for experimenting with optimization passes, and we have "llc-beta"
> which is great for measuring the impact of codegen changes.
>
> The usual approach is to decide that the patch is good, check it in, then
> watch for unexpected fallout on the nightly testers.

Btw, this is all spelled out in the developers policy, which I like to 
mention in case people are not aware of it:
http://llvm.org/docs/DeveloperPolicy.html#quality

-Tanya