[PATCH] D27826: Regenerate Halide benchmark bitcode files. Resolves TBAA verification failures.

[Kristof Beyls via Phabricator via llvm-commits]

kristof.beyls added a comment.

In https://reviews.llvm.org/D27826#624570, @MatzeB wrote:

> In https://reviews.llvm.org/D27826#624321, @sanjoy wrote:
>
> > In https://reviews.llvm.org/D27826#624246, @asbirlea wrote:
> >
> > > Yes, they're generated straight from Halide and they pass the verification. I'll update a git mirror shortly as additional reference.
> >
> >
> > Given that ^ I'd say they're okay to check in from the perspective of the TBAA verifier.  I have not manually verified the binary files.
> >
> > However, I'm not qualified to discuss if this is going to invalidate previous benchmark results and if people need to be notified about that.  What is the normal protocol here?
>
>
> I am not aware we have any process for this sort of stuff. The usual answer is to try hard not to change the benchmarks in fundamental ways. But this is not the first time such a thing happens and in this specific case we can probably consider the BitCode part as a young emerging part of the test-suite that isn't tracked by that many people yet. I guess Alina knows best who/which bots are tracking the BitCode parts.


We don't track performance of the Halide tests at the moment.
Even if we did, if there is a really good reason to change the benchmark source, we should do it, irrespective of the historical benchmark data becoming incomparable. We've made changes with the same impact before to parts of the test-suite most of us are tracking and I think we should continue to allow making such changes if there's a good reason for them. Otherwise, we'd stop improving the test-suite and how it can be used to do efficient performance tracking -- and I believe there is still scope for quite a lot of improvements to be made.


https://reviews.llvm.org/D27826