[PATCH] D68230: [UpdateCCTestChecks] Allow asm in output with --allow-asm

[Eric Christopher via Phabricator via llvm-commits]

echristo added a comment.

In D68230#2051241 <https://reviews.llvm.org/D68230#2051241>, @greened wrote:

> In D68230#2042289 <https://reviews.llvm.org/D68230#2042289>, @echristo wrote:
>
> > In general I have a pretty strong objection to end to end tests in the clang repository unless there's no other way to test them.
> >
> > Can you give an idea of what you plan on testing with this?
>
>
> The thread has a lot of details but I'll attempt to summarize.  Basically, there is nothing that currently tests that a full run through the clang pipeline actualy generates expected code.  This is important for things that aren't sufficient to test in isolation.  For example, we can test that the prefetching pass produces prefetches in the IR and we can test that IR with prefetches turns into prefetch instructions but we currently can't test that clang/LLVM maintains all of the proper information and conditions needed for prefetches to make it from the prefetch pass all the way through codegen and into prefetch instructions.  There are many reasons there might be problems: clang emitting "unexpected" IR, LLVM doing optimisations that remove/obscure prefetch opportunities and so on.  The problem with individual unit tests is that the tests set an environment that might not be what clang actually produces and when that unit test produces might not make it all the way through to the end.
>
> That said, I have no particular objection to putting such tests somewhere other than clang.  That question is orthogonal to this patch and indeed my initial proposal was to put the tests in a top-level test directory.  This patch only *allows* `update_cc_test_checks.py` to generate asm.  It doesn't mandate its use.  Many people have expressed support for this functionality and we have found it invaluable in our testing, where we definitely see breakage in the clang pipeline not covered by existing unit tests.  How we incorporate such tests upstream is a separate question.
>
> Keep in mind that end-to-end tests are *not* meant to be a first line of defense.  They are meant to catch things that slip through unit testing.  One suggestion is to put such tests in test-suite and I would be open to that.


I think the llvm test-suite would be a good place to put this sort of thing. We did that when we wanted to test some other end to end for C++ simd work and I think it's probably the best way. Would also be interested in figuring out how to try to get the best testing we can without having any end to end tests. Sometimes it happens because we don't have sufficient separation, but that should be in the absolute minority.

Thanks!


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D68230/new/

https://reviews.llvm.org/D68230