[LLVMdev] Use perf tool for more accurate time measuring on Linux

[Yi Kong]

Hi all,

The LLVM benchmarking system produces very noisy results even on quiet
machines. One of the sources of inaccuracy is the timing tool we are
using. Because it is a user-space tool, the OS can context switch it
and we will get an outlier result. Perf stat uses SW_TASK_CLOCK
counter in kernel to measure time, therefore more accurate. It also
does not get context switched.

I've implemented a wrapper script over perf stat which mimics the
behaviour of timeit tool in test suite, so that nothing else needs to
be modified. The script is not yet feature complete as timeit, but
enough to run nightly tests.

I carried out experiments on several machines and saw different level
of improvements. I am no longer seeing outlier results, and MAD is
considerably lower. The run-by-run changes results over the same
revision shrank from around 10 to only 2-3. The MAD reduced from
around 0.01 to 0.003 on a quiet machine.

I've attached the patch and please experiment with it.

Cheers,
Yi Kong
-------------- next part --------------
Index: tools/timeit.sh
===================================================================
--- tools/timeit.sh	(revision 0)
+++ tools/timeit.sh	(revision 0)
@@ -0,0 +1,25 @@
+#! /bin/bash
+# A wrapper over perf to provide similar functionality to timeit.c
+
+REPORT=/dev/stderr
+OUTPUT=/dev/stdout
+
+# FIXME: Have similar behaviour as timeit.c
+while [[ $1 = -* ]]; do
+	if [ $1 = "--summary" ]; then
+		REPORT=$2
+	elif [ $1 = "--redirect-output" ]; then
+		OUTPUT=$2
+	fi
+	shift 2
+done
+
+perf stat -o stats $@ > $OUTPUT
+
+echo exit $? > $REPORT
+awk -F' ' '{if ($2 == "task-clock") print "user",$1/1000; else if($2 =="seconds") print "real",$1;}' stats >> $REPORT
+
+# FIXME: How to measure sys time?
+echo sys 0.0000 >> $REPORT
+
+rm stats

Property changes on: tools/timeit.sh
___________________________________________________________________
Added: svn:executable
   + *

Index: tools/Makefile
===================================================================
--- tools/Makefile	(revision 208774)
+++ tools/Makefile	(working copy)
@@ -8,8 +8,13 @@
 all:: timeit-target
 endif
 
+ifeq ($(TARGET_OS),Linux)
+timeit: timeit.sh
+	cp -f $< $@
+else
 timeit: timeit.c
 	$(ORIGINAL_CC) $(CFLAGS) -O3 -o $@ $<
+endif
 
 timeit-target: timeit.c
 	$(LD_ENV_OVERRIDES) $(LCC) -o $@ $< $(LDFLAGS) $(CFLAGS) $(TARGET_FLAGS) -O3