[LLVMdev] FYI: Changing RunSafely.sh to only track user time

[John Criswell]

Daniel Dunbar wrote:
> I am going to change the LLVM test-suite RunSafely.sh to only track
> 'user' time, instead of reporting 'user' + 'sys' time as it currently
> does. This will probably cause a spike in nightly test numbers,
> although hopefully it will be limited to the smaller tests.
>
> The eventual goal is to report all numbers (independently), so that we
> can control for noise better. However, until that happens it is better
> to track the most stable & interesting number.
>   

First, you should be aware that the test-suite infrastructure is used by 
the Automatic Pool Allocation and SAFECode projects for their testing 
infrastructure.  It is also used by our internal research projects and 
may be used by other research projects at other universities.  We use 
this infrastructure for our research, so changes you make can affect us.

Second, why are you only interested in user time?  The reason why we had 
RunSafely.sh measure user + system time is that it gives a more accurate 
depiction of how well an optimization works.  If a program spends most 
of its time in the OS, increasing speed in user-space doesn't gain us 
much.  If a transform decreases user time but increases system time, 
then measuring only user time may show a speedup when measuring 
user+system will show a loss.

If noise is your concern, I think it would make sense to run the tests 
several times and report averages and standard deviations.  System time 
should only be reporting OS CPU time, so delays due to interrupts, I/O, 
etc. should not affect the results.  User+System reported for several 
runs should provide the most accurate evaluation of how well an 
optimization is improving performance.

Third, IIRC, I think you can change the nightly testers to report just 
user time by changing how they grab the time measurements in 
TEST.nightly.Makefile.  I think the .time files created by the test 
suite record user, system, and user+system, and the testing Makefiles 
just grab the number that they want.

Fourth, if you are intent on changing RunSafely.sh to measure just user 
time, why not add a feature that toggles whether to measure user+system 
or user time and *then* change the default behavior to measure user 
time?  That way, I can easily toggle it back.  What you've suggested 
above (more or less) is to remove a feature, add a feature, and then add 
back the feature you removed.  It makes more sense to me to add the one 
feature you want and then to change the default behavior to use the new 
feature instead of the old one.

-- John T.

>  - Daniel
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>