[cfe-dev] [llvm-dev] Testing Best Practices/Goals (in the context of compiler-rt)

[Sean Silva via cfe-dev]

On Tue, Mar 1, 2016 at 12:18 PM, Xinliang David Li <xinliangli at gmail.com>
wrote:

>
>
> On Tue, Mar 1, 2016 at 11:10 AM, Sean Silva via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>>
>>
>> On Tue, Mar 1, 2016 at 11:05 AM, David Blaikie via cfe-dev <
>> cfe-dev at lists.llvm.org> wrote:
>>
>>>
>>>
>>> On Mon, Feb 29, 2016 at 10:26 AM, Anna Zaks <ganna at apple.com> wrote:
>>>
>>>>
>>>> On Feb 28, 2016, at 7:57 AM, David Blaikie <dblaikie at gmail.com> wrote:
>>>>
>>>>
>>>>
>>>> On Sat, Feb 27, 2016 at 3:13 PM, Anna Zaks <ganna at apple.com> wrote:
>>>>
>>>>>
>>>>> On Feb 26, 2016, at 2:10 PM, Alexey Samsonov via llvm-dev <
>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>
>>>>>
>>>>> On Fri, Feb 26, 2016 at 1:34 PM, David Blaikie <dblaikie at gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 26, 2016 at 1:31 PM, Sean Silva <chisophugis at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Feb 26, 2016 at 1:11 PM, David Blaikie <dblaikie at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Feb 26, 2016 at 1:07 PM, Sean Silva <chisophugis at gmail.com>
>>>>>>>>  wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Feb 17, 2016 at 8:45 AM, David Blaikie via cfe-dev <
>>>>>>>>> cfe-dev at lists.llvm.org> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Feb 12, 2016 at 5:43 PM, Alexey Samsonov <
>>>>>>>>>> vonosmas at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Feb 11, 2016 at 1:50 PM, David Blaikie <
>>>>>>>>>>> dblaikie at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Feb 10, 2016 at 3:55 PM, Alexey Samsonov via cfe-dev <
>>>>>>>>>>>> cfe-dev at lists.llvm.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I mostly agree with what Richard and Justin said. Adding a few
>>>>>>>>>>>>> notes about the general strategy we use:
>>>>>>>>>>>>>
>>>>>>>>>>>>> (1) lit tests which look "end-to-end" proved to be way more
>>>>>>>>>>>>> convenient for testing runtime libraries than unit tests.
>>>>>>>>>>>>>
>>>>>>>>>>>> We do have
>>>>>>>>>>>>> the latter, and use them to provide test coverage for utility
>>>>>>>>>>>>> functions, but we quite often accompany fix to the runtime library with
>>>>>>>>>>>>> "end-to-end" small reproducer extracted from the real-world
>>>>>>>>>>>>> code that exposed the issue.
>>>>>>>>>>>>> Incidentally, this tests a whole lot of other functionality:
>>>>>>>>>>>>> Clang driver, frontend, LLVM passes, etc, but it's not the intent of the
>>>>>>>>>>>>> test.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Indeed - this is analogous to the tests for, say, LLD that use
>>>>>>>>>>>> llvm-mc to produce the inputs rather than checking in object files. That
>>>>>>>>>>>> area is open to some discussion as to just how many tools we should rope
>>>>>>>>>>>> in/how isolated we should make tests (eg: maybe building the json object
>>>>>>>>>>>> file format was going too far towards isolation? Not clear - opinions
>>>>>>>>>>>> differ). But the point of the test is to test the compiler-rt functionality
>>>>>>>>>>>> that was added/removed/modified.
>>>>>>>>>>>>
>>>>>>>>>>>> I think most people are in agreement with that, while
>>>>>>>>>>>> acknowledging the fuzzy line about how isolated we might be.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yes.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> These tests are sometimes platform-specific and poorly
>>>>>>>>>>>>> portable, but they are more reliable (we make the same steps as the
>>>>>>>>>>>>> user of the compiler), and serve the purpose of documentation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> (2) If we change LLVM instrumentation - we add a test to LLVM.
>>>>>>>>>>>>> If we change Clang code generation or driver behavior - we add
>>>>>>>>>>>>> a test to Clang. No excuses here.
>>>>>>>>>>>>>
>>>>>>>>>>>>> (3) Sometimes we still add a compiler-rt test for the change
>>>>>>>>>>>>> in LLVM or Clang: e.g. if we enhance Clang frontend to teach UBSan
>>>>>>>>>>>>> to detecting yet another kind of overflow, it makes sense to
>>>>>>>>>>>>> add a test to UBSan test-suite that demonstrates it, in addition to
>>>>>>>>>>>>> Clang test verifying that we emit a call to UBSan runtime.
>>>>>>>>>>>>> Also, compiler-rt test would allow us to verify that the actual error report
>>>>>>>>>>>>> we present to the user is sane.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This bit ^ is a bit unclear to me. If there was no change to
>>>>>>>>>>>> the UBSan runtime, and the code generated by Clang is equivalent/similar to
>>>>>>>>>>>> an existing use of the UBSan runtime - what is it that the new compiler-rt
>>>>>>>>>>>> test is providing? (perhaps you could give a concrete example you had in
>>>>>>>>>>>> mind to look at?)
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> See r235568 (change to Clang) followed by r235569 (change to
>>>>>>>>>>> compiler-rt test). Now, it's a cheat because I'm fixing test, not adding
>>>>>>>>>>> it. However, I would have definitely added it, if it was missing.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Right, I think the difference here is "if it was missing" - the
>>>>>>>>>> test case itself seems like it could be a reasonable one (are there other
>>>>>>>>>> tests of the same compiler-rt functionality? (I assume the compiler-rt
>>>>>>>>>> functionality is the implementation of sadd/ssub?))
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> In this case, a change to Clang
>>>>>>>>>>> instrumentation (arguments passed to UBSan runtime callbacks)
>>>>>>>>>>> improved the user-facing part of the tool, and compiler-rt test suite is a
>>>>>>>>>>> good place to verify that.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This seems like the problematic part - changes to LLVM improve
>>>>>>>>>> the user-facing part of Clang, but we don't add end-to-end tests of that,
>>>>>>>>>> as a general rule. I'm trying to understand why the difference between that
>>>>>>>>>> and compiler-rt
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In what way do changes in LLVM change the user-facing part of
>>>>>>>>> Clang?
>>>>>>>>> It obviously depends on how broadly one defines user-facing. Is a
>>>>>>>>> 1% performance improvement from a particular optimization user-facing? Is
>>>>>>>>> better debug info accuracy user-facing? I'm not sure. But it seems clear
>>>>>>>>> that "the user sees a diagnostic or not" definitely is.
>>>>>>>>>
>>>>>>>>
>>>>>>>> There's more than just performance in LLVM - ABI features, and yes,
>>>>>>>> I'd argue some pieces of debug info are pretty user facing (as are some
>>>>>>>> optimizations). We also have the remarks system in place now. (also the
>>>>>>>> compiler crashing (or not) is pretty user facing).
>>>>>>>>
>>>>>>>
>>>>>>> I'd argue that we probably should have some sort of integration
>>>>>>> tests for ABI features. I think at the moment we're getting by thanks to
>>>>>>> self-hosting and regularly building lots of real-world programs with
>>>>>>> ToT-ish compilers.
>>>>>>>
>>>>>>
>>>>>> Perhaps so, but I'd argue that they shouldn't be run as part of "make
>>>>>> check" & should be in a separate test grouping (probably mostly run by
>>>>>> buildbots) for the purpose of integration testing.
>>>>>>
>>>>>
>>>>> If you have llvm/clang/compiler-rt/libc++/libc++abi checkout, they are
>>>>> not run as a part of "make check", only "make check-all", which kind of
>>>>> makes sense (run *all* the tests!). You're free to run "make check-clang",
>>>>> "make check-asan" etc.
>>>>> if you're sure your changes are limited in scope. Just to be clear -
>>>>> do you suggest that compiler-rt tests are too heavy for this configuration,
>>>>> and want to introduce extra level - i.e. extract "make check-compiler-rt"
>>>>> out of "make check-all", and introduce "make check-absolutely-everything",
>>>>> that would encompass them?
>>>>>
>>>>>
>>>>>> We've made a pretty conscious, deliberate, and consistent effort to
>>>>>> not do integration testing across the LLVM projects in "make check"-like
>>>>>> testing, and to fix it where we do find it. It seems to me that compiler-rt
>>>>>> diverged from that approach and I'm not really in favor of that divergence.
>>>>>>
>>>>>
>>>>> I don't see why consistency by itself is a good thing. As a sanitizer
>>>>> developer, current situation is convenient for me, but if it harms / slows
>>>>> down / complicates workflow for other developers or LLVM as a whole - sure,
>>>>> let's fix it.
>>>>>
>>>>>
>>>>> +1
>>>>>
>>>>> I find the end-to-end compiler-rt tests very useful and easy to write.
>>>>> They run much faster on my machine than the Unit tests. The LLVM modular
>>>>> test policy does not apply as well to compiler-rt.
>>>>>
>>>>
>>>> Sorry this all seems to have become a bit confused. I am not generally
>>>> objecting to end-to-end tests in compiler-rt or elsewhere. I'm trying to
>>>> understand why, beyond that, in compiler-rt those tests seem to be being
>>>> used to exercise code and fixes in LLVM and Clang. Generally across the
>>>> LLVM project when we fix something in LLVM we test it there, when we fix
>>>> something in Clang we test it there. We don't add a Clang test for every
>>>> codegen/optimization fix we add in LLVM (even putting aside the question of
>>>> the observability of optimizations that Sean raised - consider debug info,
>>>> or correctness issues like noinline/alwaysinline, etc). So I'm trying to
>>>> understand why compiler-rt is being treated differently in this regard.
>>>>
>>>>
>>>>> (David, maybe following up with some specifics such as this test is
>>>>> redundant or we should rewrite these tests in a certain way would help to
>>>>> convey your point across.)
>>>>>
>>>>
>>>> Thanks for chiming in, Anna (I'd really like some more voices on this
>>>> thread - it seems like a pretty fundamental issue about LLVM's project
>>>> philosophy and I'm really confused). I started this conversation in such an
>>>> example (
>>>> http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160208/330759.html
>>>> ) and that's how I ended up here. A more recent example would be
>>>> r262161/r262157.
>>>>
>>>>
>>>> Whenever someone changes clang/llvm, a modular clang/llvm test should
>>>> be added. We want to make sure people do not rely solely on end-to-end
>>>> tests to test modifications to clang/llvm. From what I understand, everyone
>>>> on this thread agrees with that.
>>>>
>>>> So the question is if the end-to-end tests are redundant or not.
>>>> Sanitizers and the rest of compiler-rt builds on a complex interplay of
>>>> many players: the compiler, the runtime, symbolicator, the linker...
>>>> End-to-end tests are an effective way of testing that all these pieces
>>>> align and we get the expected behavior. One way to explain this particular
>>>> commit is that this test is not testing the specific clang change, but was
>>>> added because constructors were not tested sufficiently before.
>>>>
>>>
>>> The way I think about this, much like the comment I made to David (email
>>> I just sent) is that these things still seem to be fairly orthogonal - in
>>> Clang we don't test template type diffing in every diagnostic that could
>>> trigger it. We test the diagnostics and we test a sample of the type
>>> diffing, and we know/rely on them being essentially orthogonal. We have to
>>> do this in any testing strategy or we'd be testing the possible combination
>>> of every feature of C++, which is intractable.
>>>
>>> It seems like compiler-rt is orthogonal to constructors, it just
>>> profiles whatever it profiles - Clang decides not to profile ctors, so as
>>> long as we have a test of compiler-rt of a function with a counter and a
>>> function without a counter, I'm not sure why we'd want another test of a
>>> different way to produce a function without a counter.
>>>
>>
>> Unless there is special handling in compiler-rt for ctors, I agree that
>> we need not entangle ctors with regular functions. A test for regular
>> functions should be sufficient (for the orthogonality reason you
>> expressed). Checking that we don't forget to instrument ctors is best left
>> as a clang lit test.
>>
>
> I only agree with this partially. Making assumptions about test coverage
> in different ways (e.g, it is covered in upstream component, it is covered
> in same path as regular functions in current implementation) may not be the
> best thing to do. We missed in tests in LLVM for ctor probably because of
> similar fallacy (e.g, testing regular function).  For compiler-rt, it is
> best to not depend too much on LLVM tests, so having one representative
> case for special functions make sense to me. It can also make compiler-rt
> test more standalone (for quick testing, or a hypothetical case where we
> need to test a new experimental FE that shares the same runtime).
>

The question is what is the principle guiding which tests are added and
which are not?
An orthogonality criterion based on the current implementation (and
judgment about any potential future implementations) is one way to decide.
What criterion do you propose?

In this particular case (ctors not being instrumented) I think that it is
clear that clang was merely forgetting to instrument them and that the
issue is completely orthogonal to compiler-rt, so under the orthogonality
criterion we would not add a test.

-- Sean Silva


>
> thanks,
>
> David
>
>
>
>>
>> -- Sean Silva
>>
>>
>>>
>>>
>>>> I believe the LLVM policy was designed for testing the compiler
>>>> infrastructure and does not apply to features that heavily depend on
>>>> runtime libraries as much.
>>>>
>>>
>>> I'm really not understanding the large difference here, though. What I'm
>>> picturing is, if the runtime were built into Clang (& we just generated
>>> gobs of static LLVM IR each time we compiled and needed the runtime
>>> support) we wouldn't be compelled to test every way we call into that
>>> library I don't think - we'd test representative cases in each of the
>>> otherwise independent pieces.
>>>
>>>
>>>> The libc++ repo is another example of an LLVM project with a runtime
>>>> library and it does utilize end-to-end tests.
>>>>
>>>
>>> I haven't looked closely at libc++'s testing strategy, but I would
>>> hope/imagine/expect that it would treat it's runtime library in the same
>>> way. Test its features, but not test every possible combination of ways
>>> that might exercise those features.
>>>
>>> - David
>>>
>>>
>>>>
>>>> - Dave
>>>>
>>>>
>>>>>
>>>>>
>>>>>> - David
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> -- Sean Silva
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Also, I think part of this is that in compiler-rt there are
>>>>>>>>> usually more moving parts we don't control. E.g. it isn't just the
>>>>>>>>> interface between LLVM and clang. The information needs to pass through
>>>>>>>>> archivers, linkers, runtime loaders, etc. that all may have issues that
>>>>>>>>> affect whether the user sees the final result. In general the interface
>>>>>>>>> between LLVM and clang has no middlemen so there really isn't anything to
>>>>>>>>> check.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Correctness/behavior of the compiler depends on those things too
>>>>>>>> (linkers, loaders, etc) to produce the final working product the user
>>>>>>>> requested. If we emitted symbols with the wrong linkage we could produce
>>>>>>>> linker errors, drop important entities, etc. But we don't generally test
>>>>>>>> that the output of LLVM/Clang produces the right binary when linked, we
>>>>>>>> test that it produces the right linkages on the resulting entities.
>>>>>>>>
>>>>>>>> - David
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> -- Sean Silva
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> You may argue that Clang test would have been enough (I disagree
>>>>>>>>>>> with that), or that it qualifies as "adding coverage" (maybe).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> (4) True, we're intimidated by test-suite :) I feel that
>>>>>>>>>>>>> current use of compiler-rt test suite to check compiler-rt libs better
>>>>>>>>>>>>> follows
>>>>>>>>>>>>> the doctrine described by David.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Which David? ;) (I guess David Li, not me)
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Nope, paragraph 2 from your original email.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I think maybe what could be worth doing would be separating out
>>>>>>>>>>>> the broader/intentionally "end to end" sort of tests from the ones intended
>>>>>>>>>>>> to test compiler-rt in relative isolation.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It's really hard to draw the line here, even some of compiler-rt
>>>>>>>>>>> unit tests require instrumentation, therefore depend on new features of
>>>>>>>>>>> Clang/LLVM. Unlike builtins, which are
>>>>>>>>>>> trivial to test in isolation, testing sanitizer runtimes in
>>>>>>>>>>> isolation (w/o compiler) is often hard to implement (we tried to do so for
>>>>>>>>>>> TSan, but found unit tests extremely hard to write),
>>>>>>>>>>> and is barely useful - compiler-rt runtimes don't consist of
>>>>>>>>>>> modules (like LLVMCodeGen and LLVMMC for instance), and are never used w/o
>>>>>>>>>>> the compiler anyway.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Most importantly, I'd expect only the latter to run in a "make
>>>>>>>>>>>> check-all" run, as we do for Clang/LLVM, etc.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> And now we're getting to the goals :) Why would such a change be
>>>>>>>>>>> good? Do you worry about the time it takes to execute the test suite?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Also, there's significant complexity in compiler-rt test suite
>>>>>>>>>>>>> that narrows the tests executed
>>>>>>>>>>>>> to those supported by current host.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Feb 10, 2016 at 2:33 PM, Xinliang David Li via cfe-dev
>>>>>>>>>>>>>  <cfe-dev at lists.llvm.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Feb 10, 2016 at 2:11 PM, Justin Bogner via llvm-dev <
>>>>>>>>>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> David Blaikie via cfe-dev <cfe-dev at lists.llvm.org> writes:
>>>>>>>>>>>>>>> > Recently had a bit of a digression in a review thread
>>>>>>>>>>>>>>> related to some tests
>>>>>>>>>>>>>>> > going in to compiler-rt (
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160208/330759.html
>>>>>>>>>>>>>>> > ) and there seems to be some disconnect at least between
>>>>>>>>>>>>>>> my expectations
>>>>>>>>>>>>>>> > and reality. So I figured I'd have a bit of a discussion
>>>>>>>>>>>>>>> out here on the
>>>>>>>>>>>>>>> > dev lists where there's a bit more visibility.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > My basic expectation is that the lit tests in any LLVM
>>>>>>>>>>>>>>> project except the
>>>>>>>>>>>>>>> > test-suite are targeted tests intended to test only the
>>>>>>>>>>>>>>> functionality in
>>>>>>>>>>>>>>> > the project. This seems like a pretty well accepted
>>>>>>>>>>>>>>> doctrine across most
>>>>>>>>>>>>>>> > LLVM projects - most visibly in Clang, where we make a
>>>>>>>>>>>>>>> concerted effort not
>>>>>>>>>>>>>>> > to have tests that execute LLVM optimizations, etc.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > There are exceptions/middle ground to this - DIBuilder is
>>>>>>>>>>>>>>> in LLVM, but
>>>>>>>>>>>>>>> > essentially tested in Clang rather than writing LLVM unit
>>>>>>>>>>>>>>> tests. It's
>>>>>>>>>>>>>>> > somewhat unavoidable that any of the IR building code
>>>>>>>>>>>>>>> (IRBuilder,
>>>>>>>>>>>>>>> > DIBuilder, IR asm printing, etc) is 'tested' incidentally
>>>>>>>>>>>>>>> in Clang in
>>>>>>>>>>>>>>> > process of testing Clang's IR generation. But these are
>>>>>>>>>>>>>>> seen as incidental,
>>>>>>>>>>>>>>> > not intentionally trying to cover LLVM with Clang tests
>>>>>>>>>>>>>>> (we don't add a
>>>>>>>>>>>>>>> > Clang test if we add a new feature to IRBuilder just to
>>>>>>>>>>>>>>> test the IRBuilder).
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Another case with some middle ground are things like
>>>>>>>>>>>>>>> linker tests and
>>>>>>>>>>>>>>> > objdump, dwarfdump, etc - in theory to isolate the test we
>>>>>>>>>>>>>>> would checkin
>>>>>>>>>>>>>>> > binaries (or the textual object representation lld had for
>>>>>>>>>>>>>>> a while, etc) to
>>>>>>>>>>>>>>> > test those tools. Some tests instead checkin assembly and
>>>>>>>>>>>>>>> assemble it with
>>>>>>>>>>>>>>> > llvm-mc. Again, not to cover llvm-mc, but on the
>>>>>>>>>>>>>>> assumption that llvm-mc is
>>>>>>>>>>>>>>> > tested, and just using it as a tool to make tests easier
>>>>>>>>>>>>>>> to maintain.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > So I was surprised to find that the compiler-rt lit tests
>>>>>>>>>>>>>>> seem to diverge
>>>>>>>>>>>>>>> > from this philosophy & contain more intentional end-to-end
>>>>>>>>>>>>>>> tests (eg:
>>>>>>>>>>>>>>> > adding a test there when making a fix to Clang to add a
>>>>>>>>>>>>>>> counter to a
>>>>>>>>>>>>>>> > function that was otherwise missing a counter - I'd expect
>>>>>>>>>>>>>>> that to be
>>>>>>>>>>>>>>> > tested in Clang and that there would already be coverage
>>>>>>>>>>>>>>> in compiler-rt for
>>>>>>>>>>>>>>> > "if a function has a counter, does compiler-rt do the
>>>>>>>>>>>>>>> right thing with that
>>>>>>>>>>>>>>> > counter" (testing whatever code in compiler-rt needs to be
>>>>>>>>>>>>>>> tested)).
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Am I off base here? Are compiler-rt's tests fundamentally
>>>>>>>>>>>>>>> different to the
>>>>>>>>>>>>>>> > rest of the LLVM project? Why? Should they continue to be?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think there's a bit of grey area in terms testing the
>>>>>>>>>>>>>>> runtime -
>>>>>>>>>>>>>>> generally it's pretty hard to use the runtime without a
>>>>>>>>>>>>>>> fairly
>>>>>>>>>>>>>>> end-to-end test, so tests of the runtime often end up
>>>>>>>>>>>>>>> looking pretty
>>>>>>>>>>>>>>> close to an end-to-end test.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That said, I don't think that should be used as an excuse to
>>>>>>>>>>>>>>> sneak
>>>>>>>>>>>>>>> arbitrary end-to-end tests into compiler-rt. We should
>>>>>>>>>>>>>>> absolutely write
>>>>>>>>>>>>>>> tests in clang and llvm that we're inputting what we expect
>>>>>>>>>>>>>>> to the
>>>>>>>>>>>>>>> runtime and try to keep the tests in compiler-rt as focused
>>>>>>>>>>>>>>> on just
>>>>>>>>>>>>>>> exercising the runtime code as possible.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, we should not use compiler-rt tests as an excuse of not
>>>>>>>>>>>>>> adding clang/LLVM test. The latter should always be added if possible --
>>>>>>>>>>>>>> they are platform independent and is the first level of defense.  runtime
>>>>>>>>>>>>>> test's focus is also more on the runtime lib itself and interaction between
>>>>>>>>>>>>>>  runtime, compiler, binutils and other tools.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> IIUC, the correct place for integration tests in general is
>>>>>>>>>>>>>>> somewhere
>>>>>>>>>>>>>>> like test-suite, but I think it's a bit intimidating to some
>>>>>>>>>>>>>>> people to
>>>>>>>>>>>>>>> add new tests there (Are there docs on this?). I suspect
>>>>>>>>>>>>>>> some of the
>>>>>>>>>>>>>>> profiling related tests in compiler-rt are doing a bit much
>>>>>>>>>>>>>>> and should
>>>>>>>>>>>>>>> graduate to a spot in the test-suite (but I don't have time
>>>>>>>>>>>>>>> to volunteer
>>>>>>>>>>>>>>> to do the work, unfortunately).
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> LLVM Developers mailing list
>>>>>>>>>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> cfe-dev mailing list
>>>>>>>>>>>>>> cfe-dev at lists.llvm.org
>>>>>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Alexey Samsonov
>>>>>>>>>>>>> vonosmas at gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> cfe-dev mailing list
>>>>>>>>>>>>> cfe-dev at lists.llvm.org
>>>>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Alexey Samsonov
>>>>>>>>>>> vonosmas at gmail.com
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> cfe-dev mailing list
>>>>>>>>>> cfe-dev at lists.llvm.org
>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alexey Samsonov
>>>>> vonosmas at gmail.com
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> cfe-dev mailing list
>>> cfe-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>
>>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20160301/b4f0b322/attachment.html>