<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 10, 2016 at 2:33 PM, Xinliang David Li via cfe-dev <span dir="ltr"><<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Wed, Feb 10, 2016 at 2:11 PM, Justin Bogner via llvm-dev <span dir="ltr"><<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>David Blaikie via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> writes:<br>
> Recently had a bit of a digression in a review thread related to some tests<br>
> going in to compiler-rt (<br>
> <a href="http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160208/330759.html" rel="noreferrer" target="_blank">http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160208/330759.html</a><br>
> ) and there seems to be some disconnect at least between my expectations<br>
> and reality. So I figured I'd have a bit of a discussion out here on the<br>
> dev lists where there's a bit more visibility.<br>
><br>
> My basic expectation is that the lit tests in any LLVM project except the<br>
> test-suite are targeted tests intended to test only the functionality in<br>
> the project. This seems like a pretty well accepted doctrine across most<br>
> LLVM projects - most visibly in Clang, where we make a concerted effort not<br>
> to have tests that execute LLVM optimizations, etc.<br>
><br>
> There are exceptions/middle ground to this - DIBuilder is in LLVM, but<br>
> essentially tested in Clang rather than writing LLVM unit tests. It's<br>
> somewhat unavoidable that any of the IR building code (IRBuilder,<br>
> DIBuilder, IR asm printing, etc) is 'tested' incidentally in Clang in<br>
> process of testing Clang's IR generation. But these are seen as incidental,<br>
> not intentionally trying to cover LLVM with Clang tests (we don't add a<br>
> Clang test if we add a new feature to IRBuilder just to test the IRBuilder).<br>
><br>
> Another case with some middle ground are things like linker tests and<br>
> objdump, dwarfdump, etc - in theory to isolate the test we would checkin<br>
> binaries (or the textual object representation lld had for a while, etc) to<br>
> test those tools. Some tests instead checkin assembly and assemble it with<br>
> llvm-mc. Again, not to cover llvm-mc, but on the assumption that llvm-mc is<br>
> tested, and just using it as a tool to make tests easier to maintain.<br>
><br>
> So I was surprised to find that the compiler-rt lit tests seem to diverge<br>
> from this philosophy & contain more intentional end-to-end tests (eg:<br>
> adding a test there when making a fix to Clang to add a counter to a<br>
> function that was otherwise missing a counter - I'd expect that to be<br>
> tested in Clang and that there would already be coverage in compiler-rt for<br>
> "if a function has a counter, does compiler-rt do the right thing with that<br>
> counter" (testing whatever code in compiler-rt needs to be tested)).<br>
><br>
> Am I off base here? Are compiler-rt's tests fundamentally different to the<br>
> rest of the LLVM project? Why? Should they continue to be?<br>
<br>
</div></div>I think there's a bit of grey area in terms testing the runtime -<br>
generally it's pretty hard to use the runtime without a fairly<br>
end-to-end test, so tests of the runtime often end up looking pretty<br>
close to an end-to-end test.<br>
<br>
That said, I don't think that should be used as an excuse to sneak<br>
arbitrary end-to-end tests into compiler-rt. We should absolutely write<br>
tests in clang and llvm that we're inputting what we expect to the<br>
runtime and try to keep the tests in compiler-rt as focused on just<br>
exercising the runtime code as possible.<br></blockquote><div><br></div></div></div><div>Yes, we should not use compiler-rt tests as an excuse of not adding clang/LLVM test. The latter should always be added if possible -- they are platform independent and is the first level of defense.  runtime test's focus is also more on the runtime lib itself and interaction between  runtime, compiler, binutils and other tools.  </div></div></div></div></blockquote><div><br></div><div>This latter bit is where there's some disagreement.<br><br>The LLVM project generally expects the "make check" style tests to be fast and narrow in scope - LLVM tests test LLVM, Clang tests Clang, and it would seem compiler-rt tests should be just about testing the runtime lib, specifically. Sure, that means making sure it can handle the diversity of input/output, etc. But if two inputs to Clang produce the same output we only write one LLVM test for that output and I would expect the same of compiler-rt - if two inputs to Clang produce the same sort of output (a function with a counter, in the example we started with) I'd expect only one test for compiler-rt, to check that it can handle a function with a counter.<br><br>(generally: if a change wasn't made to compiler-rt, I wouldn't expect a test to be added (except to makeup for cases of missing compiler-rt coverage))</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>David</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<br>
IIUC, the correct place for integration tests in general is somewhere<br>
like test-suite, but I think it's a bit intimidating to some people to<br>
add new tests there (Are there docs on this?). I suspect some of the<br>
profiling related tests in compiler-rt are doing a bit much and should<br>
graduate to a spot in the test-suite (but I don't have time to volunteer<br>
to do the work, unfortunately).<br></span>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div><br></div></div>
<br>_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
<br></blockquote></div><br></div></div>