[cfe-dev] Testing Best Practices/Goals (in the context of compiler-rt)

[Justin Bogner via cfe-dev]

David Blaikie via cfe-dev <cfe-dev at lists.llvm.org> writes:
> Recently had a bit of a digression in a review thread related to some tests
> going in to compiler-rt (
> http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160208/330759.html
> ) and there seems to be some disconnect at least between my expectations
> and reality. So I figured I'd have a bit of a discussion out here on the
> dev lists where there's a bit more visibility.
>
> My basic expectation is that the lit tests in any LLVM project except the
> test-suite are targeted tests intended to test only the functionality in
> the project. This seems like a pretty well accepted doctrine across most
> LLVM projects - most visibly in Clang, where we make a concerted effort not
> to have tests that execute LLVM optimizations, etc.
>
> There are exceptions/middle ground to this - DIBuilder is in LLVM, but
> essentially tested in Clang rather than writing LLVM unit tests. It's
> somewhat unavoidable that any of the IR building code (IRBuilder,
> DIBuilder, IR asm printing, etc) is 'tested' incidentally in Clang in
> process of testing Clang's IR generation. But these are seen as incidental,
> not intentionally trying to cover LLVM with Clang tests (we don't add a
> Clang test if we add a new feature to IRBuilder just to test the IRBuilder).
>
> Another case with some middle ground are things like linker tests and
> objdump, dwarfdump, etc - in theory to isolate the test we would checkin
> binaries (or the textual object representation lld had for a while, etc) to
> test those tools. Some tests instead checkin assembly and assemble it with
> llvm-mc. Again, not to cover llvm-mc, but on the assumption that llvm-mc is
> tested, and just using it as a tool to make tests easier to maintain.
>
> So I was surprised to find that the compiler-rt lit tests seem to diverge
> from this philosophy & contain more intentional end-to-end tests (eg:
> adding a test there when making a fix to Clang to add a counter to a
> function that was otherwise missing a counter - I'd expect that to be
> tested in Clang and that there would already be coverage in compiler-rt for
> "if a function has a counter, does compiler-rt do the right thing with that
> counter" (testing whatever code in compiler-rt needs to be tested)).
>
> Am I off base here? Are compiler-rt's tests fundamentally different to the
> rest of the LLVM project? Why? Should they continue to be?

I think there's a bit of grey area in terms testing the runtime -
generally it's pretty hard to use the runtime without a fairly
end-to-end test, so tests of the runtime often end up looking pretty
close to an end-to-end test.

That said, I don't think that should be used as an excuse to sneak
arbitrary end-to-end tests into compiler-rt. We should absolutely write
tests in clang and llvm that we're inputting what we expect to the
runtime and try to keep the tests in compiler-rt as focused on just
exercising the runtime code as possible.

IIUC, the correct place for integration tests in general is somewhere
like test-suite, but I think it's a bit intimidating to some people to
add new tests there (Are there docs on this?). I suspect some of the
profiling related tests in compiler-rt are doing a bit much and should
graduate to a spot in the test-suite (but I don't have time to volunteer
to do the work, unfortunately).