[PATCH] D21590: nbench: Eliminate timing and re-running related code

[Kristof Beyls via llvm-commits]

kristof.beyls added a comment.

I also see nbench consistently being noisy.
The benchmarks in the test-suite should on every run do the same amount of work when invoked in the same way, so indeed, it's broken that nbench changes the number of iterations it runs the way it does at the moment. If such functionality would be needed, it would need to be moved to the test-suite driver level, not be implemented in individual programs.

That being said, while we're fiddling with the number of iterations this benchmark runs, could we reduce this by a factor of about 100, so it runs in between 0.1 and 1 seconds on most systems?
I've started doing analysis on what would be needed to make the test-suite in benchmark mode run an order of magnitude more quickly, so that with the same amount of experimentation time, we could get an order of magnitude more samples to do statistical analysis.
My initial experiments on x86_64 and AArch64 systems indicate that as long as a benchmark runs for at least 0.1 seconds, noise isn't larger than if it runs for much longer.
Making all programs in the test-suite run between 0.1 and 1 second would make the test-suite run probably between 1 and 2 orders of magnitude faster.
In these experiments, I see nbench taking between 0.5% and 3% of the total test-suite running time. So if it's little work to reduce the number of iterations in nbench as part of this work, we at least already have this test-suite speedup in the bank.

Thanks!

Kristof


================
Comment at: MultiSource/Benchmarks/nbench/nbench0.c:68
@@ -67,2 +67,3 @@
 int i;                  /* Index */
+int iter;
 time_t time_and_date;   /* Self-explanatory */
----------------
It seems this iter variable is unused, as you declare another iter variable in for-loop-scope below?


Repository:
  rL LLVM

http://reviews.llvm.org/D21590