[cfe-dev] Testing Best Practices/Goals (in the context of compiler-rt)

[David Blaikie via cfe-dev]

Recently had a bit of a digression in a review thread related to some tests
going in to compiler-rt (
http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160208/330759.html
) and there seems to be some disconnect at least between my expectations
and reality. So I figured I'd have a bit of a discussion out here on the
dev lists where there's a bit more visibility.

My basic expectation is that the lit tests in any LLVM project except the
test-suite are targeted tests intended to test only the functionality in
the project. This seems like a pretty well accepted doctrine across most
LLVM projects - most visibly in Clang, where we make a concerted effort not
to have tests that execute LLVM optimizations, etc.

There are exceptions/middle ground to this - DIBuilder is in LLVM, but
essentially tested in Clang rather than writing LLVM unit tests. It's
somewhat unavoidable that any of the IR building code (IRBuilder,
DIBuilder, IR asm printing, etc) is 'tested' incidentally in Clang in
process of testing Clang's IR generation. But these are seen as incidental,
not intentionally trying to cover LLVM with Clang tests (we don't add a
Clang test if we add a new feature to IRBuilder just to test the IRBuilder).

Another case with some middle ground are things like linker tests and
objdump, dwarfdump, etc - in theory to isolate the test we would checkin
binaries (or the textual object representation lld had for a while, etc) to
test those tools. Some tests instead checkin assembly and assemble it with
llvm-mc. Again, not to cover llvm-mc, but on the assumption that llvm-mc is
tested, and just using it as a tool to make tests easier to maintain.


So I was surprised to find that the compiler-rt lit tests seem to diverge
from this philosophy & contain more intentional end-to-end tests (eg:
adding a test there when making a fix to Clang to add a counter to a
function that was otherwise missing a counter - I'd expect that to be
tested in Clang and that there would already be coverage in compiler-rt for
"if a function has a counter, does compiler-rt do the right thing with that
counter" (testing whatever code in compiler-rt needs to be tested)).

Am I off base here? Are compiler-rt's tests fundamentally different to the
rest of the LLVM project? Why? Should they continue to be?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20160210/bb67d5e0/attachment.html>