[PATCH] D33343: Add some tips on how to benchhmark

[Rafael Avila de Espindola via llvm-commits]

Kristof Beyls via Phabricator <reviews at reviews.llvm.org> writes:

> ================
> Comment at: docs/Benchmarking.rst:9
> +
> +For benchmarking a patch we want to be in control of all the possible sources of noise. How to do that is very OS dependent.
> +
> ----------------
> nitpick: s/to be in control of all the possible sources of noise/reduce all possible sources of noise as much as possible/?
> I think that's slightly better, but don't feel strongly about this change.

Done.


> ================
> Comment at: docs/Benchmarking.rst:25
> +    mount -t tmpfs -o size=<XX>g none dir_to_mount
> +
> +Linux
> ----------------
> At http://lnt.llvm.org/quickstart.html#running-tests, there are a few hints under number 3 on how to reduce noise and cope with the remaining noise better when you use LNT for benchmarking. Would it be useful to have a pointer here to there?
> Maybe some of the hints there might also make sense to have here?
> Or maybe some of them should just be moved here, and there can be a pointer from there to here?

Some are lnt specific. I think that document should refer to this one
for non lnt specific issues.

> ================
> Comment at: docs/Benchmarking.rst:29-31
> +* Disable address space randomization::
> +
> +    echo 0 > /proc/sys/kernel/randomize_va_space
> ----------------
> Results will be less noisy with address space randomization turned off, but I continue to think that incorrectly skews experiment results.
> If you recommend this, maybe you also need to recommend to align all functions to a reasonably large offset so that small code changes in one function have less of a probability of affecting code layout and the associated performance impact in another function?

Any time we reduce noise, it is possible that we start measuring
something we are not accounting for.

I think the correct way to avoid this is to introduce variance, but in a
reproducible way. For example, having a linker option to randomize the
section layout given a seed would help to find if a patch improves
performance in most layouts. That would be a generalization of
reordering .o files suggested in
https://www.cis.upenn.edu/~cis501/papers/producing-wrong-data.pdf.

In any case, for this patch I would like to stay focused on lowering
the noise.

Cheers,
Rafael