[LLVMdev] Using the test suite to benchmark patches

[Devang Patel]

On May 22, 2008, at 4:30 AM, Matthijs Kooijman wrote:

> Hi Devang,
>
> I've tried the OPTBETA approach, and it now runs with and without my  
> patch
> succesfully. I've found two problems, however:
> 1. The output of the nightly report does not include the figures from
>    opt-beta by default. I've modified the TEST.nightly.report script  
> to add
> 		two columns (OPT-BETA and LLC/OPT-BETA), but committing this  
> change would
> 		mean that most users will be having a lot of useless columns  
> (though they
> 		already have that for the LLC-BETA stuff).

Oh, I forgot to mention this. And such change may disrupt nightly  
tester database also.

> 		Is there a way to access the ENABLE_* variables from the makefile  
> in the
> 		report script? If so, you could only show the columns that will  
> contain
> 		actual output, which will make the report a lot easier to read as  
> well.

I'm not sure.

> 2. The opt-beta output is always processed by llc, so you can only  
> compare
>    llc results with and without a change. It seems that for my  
> change, the
> 		largest degradations are in the cbe test, not llc. I don't think  
> there is
> 		an easy way to solve this, though.


Yes, opt-beta output is always processed by llc. If you want to use  
CBE then most likely you'll need another version of opt-beta. Or yet  
another variable to select cbe when opt-beta is used.

-
Devang