[LLVMdev] Using the test suite to benchmark patches

[Tanya M. Lattner]

> just a quick email. I've been working on a patch to simplifycfg last week and
> want to test its performance. I've ran the test-suite succesfully, both with
> the patched and unpatched versions. However, I could find no easy way to
> compare both results. I see that the web pages of the nightly tester provide
> nice results (changes compared to the day before, together with percentages
> and colors etc). Something like that should be supported for two local test
> runs as well, but I couldn't find how.

Currently, the nightly tester scripts only compare current day to previous 
and do not have the ability to compare between two test runs that are 
more than 1 day/run apart. There is a GSOC student who will be working to 
improve this and add this feature.

However, the nightly tester scripts are PHP based and require the results 
to be in a database. You could just send them to the LLVM server to do 
your comparisions if you wanted. Another option is to do a local set up, 
but this is more work and really only necessary if you have results that 
you don't want out in the public.

> I did a bit of hacking on the HTMLColDiff.pl script that I found lying around,
> the (rough) patch is attached. Is this script a quick hackup that got
> forgotten, or is it still used by people for a purpose that I don't see right
> now?

I do not know if people are using this frequently. If it doesn't work, I 
am sure the answer is no. :) It was probably used before the nightly 
tester was around.

> Any thoughts or suggestions on how to do this testing in a structured manner?
> In particular, it would be useful to have some means of running a test a few
> times and taking mean values for comparison or something...

I would look at the TEST.*.Makefile and TEST.*.report for a way to do your 
multiple run testing. You still have the problem with comparing 2 
different sets of results though.. and that would require a new script.

Of course, someone else may have a better suggestion.

-Tanya