[LLVMdev] Using the test suite to benchmark patches

[Chris Lattner]

On Wed, 21 May 2008, Mike Stump wrote:
> On May 21, 2008, at 8:09 AM, Matthijs Kooijman wrote:
>> Any thoughts or suggestions on how to do this testing in a
>> structured manner?
>
> I think that if what you're doing is sound, and you get the results
> you want, say, on compiling something like gcc with it and others
> review the basic idea (hi evan or chris) and like it, just checking it
> in and watching the performance numbers for the next day seems
> reasonable to me.  You can always revert the patch if there are
> unexpected downsides that make it not worth while.

It depends on the scope of the change.  If it is a relatively minor 
change, getting the code approved, testing it for correctness, and adding 
a regression test is sufficient.  If it is major (adding a new pass, 
significantly changing pass ordering etc) then the bar is much higher.

We don't have a great way of diffing performance runs, other than the 
nightly tester.  Devang has an experimental "opt-beta" mode that can be 
used for experimenting with optimization passes, and we have "llc-beta" 
which is great for measuring the impact of codegen changes.

The usual approach is to decide that the patch is good, check it in, then 
watch for unexpected fallout on the nightly testers.

-Chris

-- 
http://nondot.org/sabre/
http://llvm.org/