[cfe-dev] [llvm-dev] Testing Best Practices/Goals (in the context of compiler-rt)

[David Blaikie via cfe-dev]

On Sun, Feb 28, 2016 at 10:32 AM, Xinliang David Li <davidxl at google.com>
wrote:

>
>
> On Sun, Feb 28, 2016 at 7:46 AM, David Blaikie <dblaikie at gmail.com> wrote:
>
>>
>>
>> On Fri, Feb 26, 2016 at 4:59 PM, Xinliang David Li <xinliangli at gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Fri, Feb 26, 2016 at 4:28 PM, David Blaikie via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>>
>>>>
>>>> On Fri, Feb 26, 2016 at 3:17 PM, Xinliang David Li <davidxl at google.com>
>>>> wrote:
>>>>
>>>>> Sean and Alexey have said a lot on this topic.  Here is my version of
>>>>> explanation that LLVM testing is not suitable to replace end to end testing.
>>>>>
>>>>>  - The most important reason being that LLVM tests tend to test
>>>>> 'implementation details'. This has many drawbacks:
>>>>>   a) by only writing test cases like this, it is hard to expose bugs
>>>>> inherited in the implementation itself;
>>>>>   b) the connection between the implementation and end-user
>>>>> expectation is not obvious
>>>>>   c) often in times, we will have to throw out the existing
>>>>> implementation for a whole new more efficient implementation -- not only do
>>>>> we need to pay the cost of tedious test case update (from one impl to
>>>>> another) -- but that all the previous implementation specific code coverage
>>>>> become useless -- how are we sure the new implementation does not break the
>>>>> end-end user expectation?
>>>>>
>>>>
>>>> Yep, there are certainly tradeoffs between unit/feature/integration
>>>> testing.
>>>>
>>>>
>>>
>>>
>>>> End-end tests on the other hand have no such drawbacks -- its
>>>>> specification is well defined:
>>>>>   1) it is easy to write test cases based on well defined
>>>>> specifications and cover all corner cases (e.g, new language constructs
>>>>> etc);
>>>>>
>>>>
>>>> I'm not sure that integration/end-to-end testing is more likely to
>>>> exercise corner cases.
>>>>
>>>
>>> I actually meant  various cases covering by public specs. Test writers
>>> do not need to assume internal path coverage (e.g, assuming this construct
>>> is emitted in the same code path as the other construct, so let's skip the
>>> testing of one of them and save some test time) -- the assumption can be
>>> harmful whenever implementation changes.
>>>
>>
>> It seems to me we always make assumptions about internal implementation
>> details with any testing and/or our testing matrix becomes huge (for
>> example we don't test that every optimization preserves/doesn't break
>> profile counters - many optimizations and parts of code are completely
>> orthogonal to other properties of the code - and we certainly don't do that
>> with end-to-end tests)
>>
>> In any case, this is a pretty consistent philosophy of targeted testing
>> across the LLVM project. compiler-rt seems to be an outlier here and I'm
>> trying to understand why it should be so.
>>
>> I don't disagree that some amount of to end-to-end testing can be
>> beneficial, though I perhaps disagree on how much - but mostly it has been
>> a pretty core attitude of the LLVM project across subprojects that they
>> test only themselves as much as possible in their "check"-style regression
>> suite.
>>
>>
>>>
>>>
>>>>
>>>>>   2) it is easy to extract test cases from real failures from very
>>>>> large apps
>>>>>
>>>>
>>>> We seem to have been pretty good, as a community, at producing the
>>>> narrow feature/unit tests for breakages from very large
>>>> applications/scenarios. I wouldn't be happy about not having targeted tests
>>>> because we had broad coverage from some large test - so I don't think this
>>>> removes the need to reduce and isolate test cases.
>>>>
>>>
>>> If the runtime test case is not in form that is self contained and can
>>> be executed, it has probably been reduced too much to a form that the
>>> original problem can no longer be recognized.
>>>
>>
>> Presumably the same is true of the many LLVM IR-level regression tests we
>> add (& even Clang regression tests that test Clang's LLVM IR output, but
>> don't test the ultimate product of that IR and the behavior of that
>> product), no? This is a pretty core choice the LLVM project has made, to
>> test in a fairly narrow/isolated way.
>>
>>
>>>   Cutting down small reproducible (that is runnable) usually  takes huge
>>> amount of effort and it is not something we should throw away easily.
>>>
>>
>> I don't think we're throwing it away when we keep the isolated regression
>> tests (such as a Clang frontend test, etc) - that I see as the product of
>> such reduction.
>>
>> In any case, some amount of end-to-end testing doesn't seem bad (we have
>> the test-suite for a reason, it does lots of end-to-end testing, and should
>> probably do lots more), it just seems out of place to me - based on the
>> philosophy/approach of the LLVM project historically.
>>
>>
>
> your concern is really about where those test should be or whether those
> end-end testing should be run by default ?
>
>
>
>>
>>>
>>>>
>>>>
>>>>>   3) once the test case is written, it almost never needs to be
>>>>> updated as long as the public interfaces (options) are used.
>>>>>
>>>>> End-end testing also cover areas not in compiler proper -- like
>>>>> runtime, linker, other tools, and their subtle interactions.
>>>>>
>>>>
>>>> This is actually a drawback of integration testing - we don't actually
>>>> want to test those products. And we don't want to make our testing
>>>> dependent on them, generally - it makes the tests more expensive in a
>>>> number of ways (more work to setup, costlier to run, more likely to break
>>>> for non-product reasons, more work to debug because they involve more
>>>> moving parts). This is one of the tradeoffs of broader testing strategies.
>>>>
>>>
>>> So you think it is ok to make assumption that everything will just work
>>> and release it to the customer?
>>>
>>>>
>>>>
>>>>> For instance latest darwin i386 profile-rt bug exposed by the improved
>>>>> profile runtime test coverage will be hard to expose via LLVM only testing
>>>>> ..
>>>>>
>>>>
>>>> I'm not familiar with the bug - could you provide more detail?
>>>>
>>>
>>> The runtime assumes the Stop symbol (for a given section) is well
>>> aligned and point to the end of padding, but i386 darwin linker does not
>>> leading to garbage been written out.
>>>
>>
>> OK - sounds like a bug in the Darwin linker, yes? Why would it be
>> compelling for us to test for bugs in external tools? Would we expect the
>> people working on a linker to test for bugs in Clang?
>>
>> But, yes, obviously people care about the overall ecosystem and need to
>> test that it all ultimately works together - generally, the test-suite has
>> been where that happens. (again, not something I'm saying is the
>> necessary/only place that should happen, but historically, that's been the
>> pretty hard line between 'check' target tests and end-to-end execution
>> tests)
>>
>>
>>>
>>>
>>>>
>>>>
>>>>> In short, I think runtime tests do not overlap with LLVM testing --
>>>>> even though they look like testing the same thing. For reference, we
>>>>> currently have < 100 profile rt end-end testing which is alarmingly small.
>>>>> We should push increasing its coverage.
>>>>>
>>>>
>>>> While I don't fundamentally have a problem with different kinds of
>>>> testing (Up to feature/integration testing, etc), I really don't want that
>>>> testing to become what we rely on for correctness,
>>>>
>>>
>>> As I said, there are scenarios we will unfortunately have to rely on it.
>>> For instance, we completely change the way how instrumentation is done
>>> (from compiler to runtime) and all existing unittests are nullified --
>>> though that is not expected to be run every day by every other developers,
>>> the coverage in this space needs to be there.
>>>
>>
>> Then we rewrite those tests - LLVM's IR changes regularly, and that's
>> what we do - we don't throw all the tests out. (unless the feature changes
>> to be unrecognizable, then we work up new tests - such as when the new
>> Windows exception handling representation was added, I assume)
>>
>
> you missed the point here.  Such rewrite does not guarantee to give you
> the right coverage. Many LLVM IR testing is doing checks like: "I am
> expecting this IR pattern, upstream producers don't break it!".  The
> connection and check of the end results (what user sees) from the IR
> pattern is totally absent. When the test is rewritten, you can not simply
> write the newly expected pattern -- they need to be verified to be
> equivalent (producing the same end results) -- doing pure reasoning is not
> enough.
>
>
>
>>
>> "not expected to be run every day by every other developer" - this is the
>> point I'm trying to make. Compiler-rt should not be special and the holder
>> of "all the end-to-end tests". Its "check" target should work like the rest
>> of LLVM and be for targeted tests on compiler-rt. If we want intentional
>> (rather than as a convenient method of access, as talked about earlier)
>> end-to-end testing, it should be separated (either at least into a separate
>> target or possibly a separate repo, like the test-suite)
>>
>
> you think running 'check-all' should not invoke those tests? Having a new
> check target works fine too.
>
>
>>
>>
>>>
>>>
>>>> nor what we expect developers (or even most buildbots) to run for
>>>> correctness. They are slower and problematic in different ways that would
>>>> slow down our velocity.
>>>>
>>>
>>> That is true and  totally fine.
>>>
>>>
>>>>
>>>> That sort of coverage should still be carefully chosen for when it adds
>>>> value, not too scattershot/broad (we have the test-suite for really broad
>>>> "does this work with the real world" sort of stuff - but anything below
>>>> that should still be fairly isolated/targeted, I think).
>>>>
>>>>
>>> For the runtime feature owners and developers, those tests are a must --
>>> not that the burden of running them should be on every other developer.
>>>
>>
>> Why is the runtime unique/different in this regard - all the arguments
>> you've been making sound like they would apply equally to clang, for
>> example? & that Clang should have end-to-end, execution tests? Yet this is
>> something the LLVM project has pushed back /very/ hard on and tried very
>> hard to not do and to remove when it comes up. So I'm trying to understand
>> why this case is so different.
>>
>>
> I feel we are running in circles here. The main point is that we should
> have tests that test things against public specifications (e.g, language
> specs, C++ ABIs, etc) which are invariants. If there is a way to test those
> invariants without resorting to end-to-end, it is the best. For instance,
> both LLVM IR and C++ ABI are well specified, clang tests can choose to test
> against those.
>
> Many runtime features on the other hand only have well defined user level
> specifications. The implementation can and will often change . Having tests
> that only test against internal internal implementations is not enough thus
> end-end testing is needed.
>

I agree this does seem to be going around in circles, though I still don't
really follow.

Even if we consider things without well defined specifications, for
example, the Clang AST->IR mapping (Clang's lib/CodeGen) we don't generally
test things that are independent. Otherwise we would need to test the
combination of, for example, calling a function from a ctor separately from
calling a function from a free function, or we would need to test our
diagnostic generation (this might be a better analogy to Alexey's example)
with a template type versus a non-template type (we have that fancy
template type diffing - but we don't test it in every diagnostic that might
produce it - because it's orthogonal to the diagnostic in question)

- David


>
> Hope this is clearer,
>
> thanks,
>
> David
>
>
> - David
>>
>>
>>>
>>> David
>>>
>>>
>>>
>>>
>>>> - David
>>>>
>>>>
>>>>>
>>>>> thanks,
>>>>>
>>>>> David
>>>>>
>>>>>
>>>>> On Fri, Feb 26, 2016 at 2:15 PM, David Blaikie <dblaikie at gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 26, 2016 at 2:10 PM, Alexey Samsonov <vonosmas at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> On Fri, Feb 26, 2016 at 1:34 PM, David Blaikie <dblaikie at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Feb 26, 2016 at 1:31 PM, Sean Silva <chisophugis at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Feb 26, 2016 at 1:11 PM, David Blaikie <dblaikie at gmail.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Feb 26, 2016 at 1:07 PM, Sean Silva <
>>>>>>>>>> chisophugis at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Feb 17, 2016 at 8:45 AM, David Blaikie via cfe-dev <
>>>>>>>>>>> cfe-dev at lists.llvm.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Feb 12, 2016 at 5:43 PM, Alexey Samsonov <
>>>>>>>>>>>> vonosmas at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Feb 11, 2016 at 1:50 PM, David Blaikie <
>>>>>>>>>>>>> dblaikie at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Feb 10, 2016 at 3:55 PM, Alexey Samsonov via cfe-dev
>>>>>>>>>>>>>> <cfe-dev at lists.llvm.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I mostly agree with what Richard and Justin said. Adding a
>>>>>>>>>>>>>>> few notes about the general strategy we use:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (1) lit tests which look "end-to-end" proved to be way more
>>>>>>>>>>>>>>> convenient for testing runtime libraries than unit tests.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We do have
>>>>>>>>>>>>>>> the latter, and use them to provide test coverage for
>>>>>>>>>>>>>>> utility functions, but we quite often accompany fix to the runtime library
>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>> "end-to-end" small reproducer extracted from the real-world
>>>>>>>>>>>>>>> code that exposed the issue.
>>>>>>>>>>>>>>> Incidentally, this tests a whole lot of other functionality:
>>>>>>>>>>>>>>> Clang driver, frontend, LLVM passes, etc, but it's not the intent of the
>>>>>>>>>>>>>>> test.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Indeed - this is analogous to the tests for, say, LLD that
>>>>>>>>>>>>>> use llvm-mc to produce the inputs rather than checking in object files.
>>>>>>>>>>>>>> That area is open to some discussion as to just how many tools we should
>>>>>>>>>>>>>> rope in/how isolated we should make tests (eg: maybe building the json
>>>>>>>>>>>>>> object file format was going too far towards isolation? Not clear -
>>>>>>>>>>>>>> opinions differ). But the point of the test is to test the compiler-rt
>>>>>>>>>>>>>> functionality that was added/removed/modified.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think most people are in agreement with that, while
>>>>>>>>>>>>>> acknowledging the fuzzy line about how isolated we might be.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> These tests are sometimes platform-specific and poorly
>>>>>>>>>>>>>>> portable, but they are more reliable (we make the same steps as the
>>>>>>>>>>>>>>> user of the compiler), and serve the purpose of
>>>>>>>>>>>>>>> documentation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (2) If we change LLVM instrumentation - we add a test to
>>>>>>>>>>>>>>> LLVM. If we change Clang code generation or driver behavior - we add
>>>>>>>>>>>>>>> a test to Clang. No excuses here.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (3) Sometimes we still add a compiler-rt test for the change
>>>>>>>>>>>>>>> in LLVM or Clang: e.g. if we enhance Clang frontend to teach UBSan
>>>>>>>>>>>>>>> to detecting yet another kind of overflow, it makes sense to
>>>>>>>>>>>>>>> add a test to UBSan test-suite that demonstrates it, in addition to
>>>>>>>>>>>>>>> Clang test verifying that we emit a call to UBSan runtime.
>>>>>>>>>>>>>>> Also, compiler-rt test would allow us to verify that the actual error report
>>>>>>>>>>>>>>> we present to the user is sane.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This bit ^ is a bit unclear to me. If there was no change to
>>>>>>>>>>>>>> the UBSan runtime, and the code generated by Clang is equivalent/similar to
>>>>>>>>>>>>>> an existing use of the UBSan runtime - what is it that the new compiler-rt
>>>>>>>>>>>>>> test is providing? (perhaps you could give a concrete example you had in
>>>>>>>>>>>>>> mind to look at?)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> See r235568 (change to Clang) followed by r235569 (change to
>>>>>>>>>>>>> compiler-rt test). Now, it's a cheat because I'm fixing test, not adding
>>>>>>>>>>>>> it. However, I would have definitely added it, if it was missing.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Right, I think the difference here is "if it was missing" - the
>>>>>>>>>>>> test case itself seems like it could be a reasonable one (are there other
>>>>>>>>>>>> tests of the same compiler-rt functionality? (I assume the compiler-rt
>>>>>>>>>>>> functionality is the implementation of sadd/ssub?))
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> In this case, a change to Clang
>>>>>>>>>>>>> instrumentation (arguments passed to UBSan runtime callbacks)
>>>>>>>>>>>>> improved the user-facing part of the tool, and compiler-rt test suite is a
>>>>>>>>>>>>> good place to verify that.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This seems like the problematic part - changes to LLVM improve
>>>>>>>>>>>> the user-facing part of Clang, but we don't add end-to-end tests of that,
>>>>>>>>>>>> as a general rule. I'm trying to understand why the difference between that
>>>>>>>>>>>> and compiler-rt
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> In what way do changes in LLVM change the user-facing part of
>>>>>>>>>>> Clang?
>>>>>>>>>>> It obviously depends on how broadly one defines user-facing. Is
>>>>>>>>>>> a 1% performance improvement from a particular optimization user-facing? Is
>>>>>>>>>>> better debug info accuracy user-facing? I'm not sure. But it seems clear
>>>>>>>>>>> that "the user sees a diagnostic or not" definitely is.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> There's more than just performance in LLVM - ABI features, and
>>>>>>>>>> yes, I'd argue some pieces of debug info are pretty user facing (as are
>>>>>>>>>> some optimizations). We also have the remarks system in place now. (also
>>>>>>>>>> the compiler crashing (or not) is pretty user facing).
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'd argue that we probably should have some sort of integration
>>>>>>>>> tests for ABI features. I think at the moment we're getting by thanks to
>>>>>>>>> self-hosting and regularly building lots of real-world programs with
>>>>>>>>> ToT-ish compilers.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Perhaps so, but I'd argue that they shouldn't be run as part of
>>>>>>>> "make check" & should be in a separate test grouping (probably mostly run
>>>>>>>> by buildbots) for the purpose of integration testing.
>>>>>>>>
>>>>>>>
>>>>>>> If you have llvm/clang/compiler-rt/libc++/libc++abi checkout, they
>>>>>>> are not run as a part of "make check", only "make check-all", which kind of
>>>>>>> makes sense (run *all* the tests!). You're free to run "make check-clang",
>>>>>>> "make check-asan" etc.
>>>>>>> if you're sure your changes are limited in scope. Just to be clear -
>>>>>>> do you suggest that compiler-rt tests are too heavy for this configuration,
>>>>>>> and want to introduce extra level - i.e. extract "make check-compiler-rt"
>>>>>>> out of "make check-all", and introduce "make check-absolutely-everything",
>>>>>>> that would encompass them?
>>>>>>>
>>>>>>
>>>>>> Fair point, check-all would be/is check all the features, but, yes,
>>>>>> potentially pulling out a separate subgroup for integration testing of any
>>>>>> kind. Yes, I realize this has the "-Wall" problem. But I suppose what I'm
>>>>>> getting at is "check" in the LLVM project parlance is, with the exception
>>>>>> of compiler-rt, the "lightweight, immediate verification tests, not
>>>>>> integration testing" & I would rather like that to be the case across the
>>>>>> board.
>>>>>>
>>>>>> Perhaps using a different verb for cross-project testing (I don't
>>>>>> object to a catch-all target for running all the different kinds of testing
>>>>>> we have). Naming is hard.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> We've made a pretty conscious, deliberate, and consistent effort to
>>>>>>>> not do integration testing across the LLVM projects in "make check"-like
>>>>>>>> testing, and to fix it where we do find it. It seems to me that compiler-rt
>>>>>>>> diverged from that approach and I'm not really in favor of that divergence.
>>>>>>>>
>>>>>>>
>>>>>>> I don't see why consistency by itself is a good thing.
>>>>>>>
>>>>>>
>>>>>> Makes it easier to work between projects. Sets expectations for
>>>>>> developers when they work in one area or another.
>>>>>>
>>>>>>
>>>>>>> As a sanitizer developer, current situation is convenient for me,
>>>>>>> but if it harms / slows down / complicates workflow for other developers or
>>>>>>> LLVM as a whole - sure, let's fix it.
>>>>>>>
>>>>>>
>>>>>> One of the things I'm trying to understand is what the benefit of
>>>>>> this extra testing is - to take the add/sub example again, is adding extra
>>>>>> coverage for printing different text through the same mechanism in
>>>>>> compiler-rt valuable? What sort of regressions/bugs is that catching that
>>>>>> makes it compelling to author, maintain, and incur the ongoing execution
>>>>>> cost of?
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> - David
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> -- Sean Silva
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Also, I think part of this is that in compiler-rt there are
>>>>>>>>>>> usually more moving parts we don't control. E.g. it isn't just the
>>>>>>>>>>> interface between LLVM and clang. The information needs to pass through
>>>>>>>>>>> archivers, linkers, runtime loaders, etc. that all may have issues that
>>>>>>>>>>> affect whether the user sees the final result. In general the interface
>>>>>>>>>>> between LLVM and clang has no middlemen so there really isn't anything to
>>>>>>>>>>> check.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Correctness/behavior of the compiler depends on those things too
>>>>>>>>>> (linkers, loaders, etc) to produce the final working product the user
>>>>>>>>>> requested. If we emitted symbols with the wrong linkage we could produce
>>>>>>>>>> linker errors, drop important entities, etc. But we don't generally test
>>>>>>>>>> that the output of LLVM/Clang produces the right binary when linked, we
>>>>>>>>>> test that it produces the right linkages on the resulting entities.
>>>>>>>>>>
>>>>>>>>>> - David
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -- Sean Silva
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> You may argue that Clang test would have been enough (I
>>>>>>>>>>>>> disagree with that), or that it qualifies as "adding coverage" (maybe).
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (4) True, we're intimidated by test-suite :) I feel that
>>>>>>>>>>>>>>> current use of compiler-rt test suite to check compiler-rt libs better
>>>>>>>>>>>>>>> follows
>>>>>>>>>>>>>>> the doctrine described by David.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Which David? ;) (I guess David Li, not me)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Nope, paragraph 2 from your original email.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think maybe what could be worth doing would be separating
>>>>>>>>>>>>>> out the broader/intentionally "end to end" sort of tests from the ones
>>>>>>>>>>>>>> intended to test compiler-rt in relative isolation.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> It's really hard to draw the line here, even some of
>>>>>>>>>>>>> compiler-rt unit tests require instrumentation, therefore depend on new
>>>>>>>>>>>>> features of Clang/LLVM. Unlike builtins, which are
>>>>>>>>>>>>> trivial to test in isolation, testing sanitizer runtimes in
>>>>>>>>>>>>> isolation (w/o compiler) is often hard to implement (we tried to do so for
>>>>>>>>>>>>> TSan, but found unit tests extremely hard to write),
>>>>>>>>>>>>> and is barely useful - compiler-rt runtimes don't consist of
>>>>>>>>>>>>> modules (like LLVMCodeGen and LLVMMC for instance), and are never used w/o
>>>>>>>>>>>>> the compiler anyway.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Most importantly, I'd expect only the latter to run in a
>>>>>>>>>>>>>> "make check-all" run, as we do for Clang/LLVM, etc.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> And now we're getting to the goals :) Why would such a change
>>>>>>>>>>>>> be good? Do you worry about the time it takes to execute the test suite?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Also, there's significant complexity in compiler-rt test
>>>>>>>>>>>>>>> suite that narrows the tests executed
>>>>>>>>>>>>>>> to those supported by current host.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Feb 10, 2016 at 2:33 PM, Xinliang David Li via
>>>>>>>>>>>>>>> cfe-dev <cfe-dev at lists.llvm.org> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Feb 10, 2016 at 2:11 PM, Justin Bogner via llvm-dev
>>>>>>>>>>>>>>>> <llvm-dev at lists.llvm.org> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> David Blaikie via cfe-dev <cfe-dev at lists.llvm.org> writes:
>>>>>>>>>>>>>>>>> > Recently had a bit of a digression in a review thread
>>>>>>>>>>>>>>>>> related to some tests
>>>>>>>>>>>>>>>>> > going in to compiler-rt (
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160208/330759.html
>>>>>>>>>>>>>>>>> > ) and there seems to be some disconnect at least between
>>>>>>>>>>>>>>>>> my expectations
>>>>>>>>>>>>>>>>> > and reality. So I figured I'd have a bit of a discussion
>>>>>>>>>>>>>>>>> out here on the
>>>>>>>>>>>>>>>>> > dev lists where there's a bit more visibility.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > My basic expectation is that the lit tests in any LLVM
>>>>>>>>>>>>>>>>> project except the
>>>>>>>>>>>>>>>>> > test-suite are targeted tests intended to test only the
>>>>>>>>>>>>>>>>> functionality in
>>>>>>>>>>>>>>>>> > the project. This seems like a pretty well accepted
>>>>>>>>>>>>>>>>> doctrine across most
>>>>>>>>>>>>>>>>> > LLVM projects - most visibly in Clang, where we make a
>>>>>>>>>>>>>>>>> concerted effort not
>>>>>>>>>>>>>>>>> > to have tests that execute LLVM optimizations, etc.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > There are exceptions/middle ground to this - DIBuilder
>>>>>>>>>>>>>>>>> is in LLVM, but
>>>>>>>>>>>>>>>>> > essentially tested in Clang rather than writing LLVM
>>>>>>>>>>>>>>>>> unit tests. It's
>>>>>>>>>>>>>>>>> > somewhat unavoidable that any of the IR building code
>>>>>>>>>>>>>>>>> (IRBuilder,
>>>>>>>>>>>>>>>>> > DIBuilder, IR asm printing, etc) is 'tested'
>>>>>>>>>>>>>>>>> incidentally in Clang in
>>>>>>>>>>>>>>>>> > process of testing Clang's IR generation. But these are
>>>>>>>>>>>>>>>>> seen as incidental,
>>>>>>>>>>>>>>>>> > not intentionally trying to cover LLVM with Clang tests
>>>>>>>>>>>>>>>>> (we don't add a
>>>>>>>>>>>>>>>>> > Clang test if we add a new feature to IRBuilder just to
>>>>>>>>>>>>>>>>> test the IRBuilder).
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Another case with some middle ground are things like
>>>>>>>>>>>>>>>>> linker tests and
>>>>>>>>>>>>>>>>> > objdump, dwarfdump, etc - in theory to isolate the test
>>>>>>>>>>>>>>>>> we would checkin
>>>>>>>>>>>>>>>>> > binaries (or the textual object representation lld had
>>>>>>>>>>>>>>>>> for a while, etc) to
>>>>>>>>>>>>>>>>> > test those tools. Some tests instead checkin assembly
>>>>>>>>>>>>>>>>> and assemble it with
>>>>>>>>>>>>>>>>> > llvm-mc. Again, not to cover llvm-mc, but on the
>>>>>>>>>>>>>>>>> assumption that llvm-mc is
>>>>>>>>>>>>>>>>> > tested, and just using it as a tool to make tests easier
>>>>>>>>>>>>>>>>> to maintain.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > So I was surprised to find that the compiler-rt lit
>>>>>>>>>>>>>>>>> tests seem to diverge
>>>>>>>>>>>>>>>>> > from this philosophy & contain more intentional
>>>>>>>>>>>>>>>>> end-to-end tests (eg:
>>>>>>>>>>>>>>>>> > adding a test there when making a fix to Clang to add a
>>>>>>>>>>>>>>>>> counter to a
>>>>>>>>>>>>>>>>> > function that was otherwise missing a counter - I'd
>>>>>>>>>>>>>>>>> expect that to be
>>>>>>>>>>>>>>>>> > tested in Clang and that there would already be coverage
>>>>>>>>>>>>>>>>> in compiler-rt for
>>>>>>>>>>>>>>>>> > "if a function has a counter, does compiler-rt do the
>>>>>>>>>>>>>>>>> right thing with that
>>>>>>>>>>>>>>>>> > counter" (testing whatever code in compiler-rt needs to
>>>>>>>>>>>>>>>>> be tested)).
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Am I off base here? Are compiler-rt's tests
>>>>>>>>>>>>>>>>> fundamentally different to the
>>>>>>>>>>>>>>>>> > rest of the LLVM project? Why? Should they continue to
>>>>>>>>>>>>>>>>> be?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think there's a bit of grey area in terms testing the
>>>>>>>>>>>>>>>>> runtime -
>>>>>>>>>>>>>>>>> generally it's pretty hard to use the runtime without a
>>>>>>>>>>>>>>>>> fairly
>>>>>>>>>>>>>>>>> end-to-end test, so tests of the runtime often end up
>>>>>>>>>>>>>>>>> looking pretty
>>>>>>>>>>>>>>>>> close to an end-to-end test.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> That said, I don't think that should be used as an excuse
>>>>>>>>>>>>>>>>> to sneak
>>>>>>>>>>>>>>>>> arbitrary end-to-end tests into compiler-rt. We should
>>>>>>>>>>>>>>>>> absolutely write
>>>>>>>>>>>>>>>>> tests in clang and llvm that we're inputting what we
>>>>>>>>>>>>>>>>> expect to the
>>>>>>>>>>>>>>>>> runtime and try to keep the tests in compiler-rt as
>>>>>>>>>>>>>>>>> focused on just
>>>>>>>>>>>>>>>>> exercising the runtime code as possible.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes, we should not use compiler-rt tests as an excuse of
>>>>>>>>>>>>>>>> not adding clang/LLVM test. The latter should always be added if possible
>>>>>>>>>>>>>>>> -- they are platform independent and is the first level of defense.
>>>>>>>>>>>>>>>>  runtime test's focus is also more on the runtime lib itself and
>>>>>>>>>>>>>>>> interaction between  runtime, compiler, binutils and other tools.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> IIUC, the correct place for integration tests in general
>>>>>>>>>>>>>>>>> is somewhere
>>>>>>>>>>>>>>>>> like test-suite, but I think it's a bit intimidating to
>>>>>>>>>>>>>>>>> some people to
>>>>>>>>>>>>>>>>> add new tests there (Are there docs on this?). I suspect
>>>>>>>>>>>>>>>>> some of the
>>>>>>>>>>>>>>>>> profiling related tests in compiler-rt are doing a bit
>>>>>>>>>>>>>>>>> much and should
>>>>>>>>>>>>>>>>> graduate to a spot in the test-suite (but I don't have
>>>>>>>>>>>>>>>>> time to volunteer
>>>>>>>>>>>>>>>>> to do the work, unfortunately).
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> LLVM Developers mailing list
>>>>>>>>>>>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> cfe-dev mailing list
>>>>>>>>>>>>>>>> cfe-dev at lists.llvm.org
>>>>>>>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Alexey Samsonov
>>>>>>>>>>>>>>> vonosmas at gmail.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> cfe-dev mailing list
>>>>>>>>>>>>>>> cfe-dev at lists.llvm.org
>>>>>>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Alexey Samsonov
>>>>>>>>>>>>> vonosmas at gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> cfe-dev mailing list
>>>>>>>>>>>> cfe-dev at lists.llvm.org
>>>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Alexey Samsonov
>>>>>>> vonosmas at gmail.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20160301/69b39fcc/attachment.html>