[PATCH] D33343: Add some tips on how to benchhmark

[Kristof Beyls via llvm-commits]

On 19 May 2017, at 23:59, Rafael Avila de Espindola <rafael.espindola at gmail.com<mailto:rafael.espindola at gmail.com>> wrote:

================
Comment at: docs/Benchmarking.rst:25
+    mount -t tmpfs -o size=<XX>g none dir_to_mount
+
+Linux
----------------
At http://lnt.llvm.org/quickstart.html#running-tests, there are a few hints under number 3 on how to reduce noise and cope with the remaining noise better when you use LNT for benchmarking. Would it be useful to have a pointer here to there?
Maybe some of the hints there might also make sense to have here?
Or maybe some of them should just be moved here, and there can be a pointer from there to here?

Some are lnt specific. I think that document should refer to this one
for non lnt specific issues.

Makes sense to me.

The non-lnt-specific issues that are mentioned there that I think aren't mentioned here are:

  *   When running under linux: Make lnt use linux perf to get more accurate timing for short-running benchmarks (--use-perf=1)
Making this LNT-independent: make sure you use a high resolution timer, e.g. linux perf under linux.

  *   Make sure each program gets run multiple times, so that LNT has a higher chance of recognizing which programs are inherently noisy (--multisample=5)
Making this LNT-independent: Make sure to run the benchmark multiple times to be able to recognize noise. This may be pretty obvious, but it may be worth spelling it out in the introduction?

  *   Disable frequency scaling / turbo boost. In case of thermal throttling it can skew the results.
You mention this already in the linux-specific part, but maybe it's worthwhile to state this in a OS-independent way in the "general" section?

  *   Disable as many processes or services as possible on the target system.
I don't think this is explicitly suggested yet in the document?


================
Comment at: docs/Benchmarking.rst:29-31
+* Disable address space randomization::
+
+    echo 0 > /proc/sys/kernel/randomize_va_space
----------------
Results will be less noisy with address space randomization turned off, but I continue to think that incorrectly skews experiment results.
If you recommend this, maybe you also need to recommend to align all functions to a reasonably large offset so that small code changes in one function have less of a probability of affecting code layout and the associated performance impact in another function?

Any time we reduce noise, it is possible that we start measuring
something we are not accounting for.

I think the correct way to avoid this is to introduce variance, but in a
reproducible way. For example, having a linker option to randomize the
section layout given a seed would help to find if a patch improves
performance in most layouts. That would be a generalization of
reordering .o files suggested in
https://www.cis.upenn.edu/~cis501/papers/producing-wrong-data.pdf.

In any case, for this patch I would like to stay focused on lowering
the noise.

OK, it makes sense to me to keep this focussed on lowering noise. I still think it'd be worthwhile in the introduction to have 1 or 2 sentences explaining this and e.g. refer to that Mytkowicz paper, so that hopefully the number of times people come to the wrong conclusion on their compiler patch due to measurement bias is reduced. Just to make sure more people understand that even though they have low noise levels, their measurements may still not allow them to come to definite conclusions.

Thanks,

Kristof


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20170520/c531c3c4/attachment.html>