[PATCH] D24593: Standford/Bubble sort code restructure

[Chandler Carruth via llvm-commits]

chandlerc added a comment.

In https://reviews.llvm.org/D24593#563106, @evstupac wrote:

> > This is not flaky.
>
> Maybe it is not flaky. But what I'm trying to say that now the test is not ok to measure compiler performance.
>  Exactly the same (same instructions, same addresses) hot loop, but performance is 2 times lower. How compiler can deal with this?


But there is *some* difference in the generated code. You were passing two different flags to the compiler and changing the transformations.

> Branches here are unpredictable, the data is random. In prolog case we are lucky to have more predictable branches. In epilog case no.

Sure. Why shouldn't we have a benchmark with unpredictable branches?

I understand if you *also* want to benchmark predictable branches. But that doesn't mean the current benchmark is invalid.

> We can leave the test code as is, but modify array data filing. This will also help. But leaving the test as is will cause unreasonable performance changes in future.

I'm not sure why you call this unreasonable.

You've not really explained:

- What transform you are changing that causes this benchmark's performance to fluctuate.
- Why that transform is clearly a good thing despite regressing this benchmark

Until you do those two things, IMO this benchmark is doing exactly what is intended: it is identifying a performance regression.

Now, if you present clear and clearly explained evidence that there is a transform you would like to enable which happens to regress this benchmark but has a very good impact on other benchmarks, perhaps we would be OK with accepting that tradeoff. I don't see why that makes this benchmark invalid, it just means that the regression observed is tolerable.

I will also point out that your analysis is completely x86 specific. There are other architectures that this might be a very useful benchmark for even if it does lose all utility for benchmarking changes to the compiler on x86.


https://reviews.llvm.org/D24593