[llvm-dev] [RFC] Cross-project lit test suite

Tue Jan 26 10:38:18 PST 2021

On Tue, Jan 26, 2021 at 4:50 AM James Henderson <
jh7370.2008 at my.bristol.ac.uk> wrote:

> On Tue, 26 Jan 2021 at 00:28, David Blaikie <dblaikie at gmail.com> wrote:
>
>>
>>
>> On Mon, Jan 25, 2021 at 4:20 PM Mehdi AMINI <joker.eph at gmail.com> wrote:
>>
>>>
>>>
>>> On Mon, Jan 25, 2021 at 12:16 PM David Blaikie via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> On Mon, Jan 25, 2021 at 3:36 AM James Henderson via llvm-dev
>>>> <llvm-dev at lists.llvm.org> wrote:
>>>> >
>>>> > Dear all,
>>>> >
>>>> >
>>>> > Recently, I and a number of my colleagues have run into cases where
>>>> we would like the ability to write tests that involve components from
>>>> multiple LLVM projects, for example using both clang and LLD. Similarly, I
>>>> have seen a few instances recently where tests would ideally make use of
>>>> LLD but only to help generate input objects for testing an LLVM tool, such
>>>> as llvm-symbolizer (see for example https://reviews.llvm.org/D88988).
>>>> Currently, there is no location where lit tests that use both clang and LLD
>>>> can be put, whilst the llvm-symbolizer cases I’ve hit are testing
>>>> llvm-symbolizer (and not LLD), so don’t really fit in the LLD test suite. I
>>>> therefore have prototyped a lit test suite that would be part of the
>>>> monorepo, and which can support tests that use elements from multiple
>>>> projects - see https://reviews.llvm.org/D95339. Tests could be added
>>>> to this suite as needed. The suite is modelled as an additional top-level
>>>> directory, and is enabled by enabling the “cross-project-tests” project in
>>>> CMake. I have initially added support for both LLD and clang. At
>>>> configuration time, the tests that require LLD or clang will only be
>>>> enabled when the respective projects are enabled, so that developers
>>>> continue to benefit from the subset of tests that are applicable for the
>>>> projects they are building. Note that I am not especially familiar with
>>>> CMake or lit, so this code may not be perfect, but it should be sufficient
>>>> to demonstrate what it can do.
>>>> >
>>>> >
>>>> >
>>>> > One could argue that these sorts of tests should belong in the
>>>> external (to the monorepo) test-suite, but this is a) quite distant from
>>>> the existing testing, and therefore easily forgotten, delaying potential
>>>> feedback for any breakages/resulting in potentially duplicate testing etc,
>>>> and b) is not as easy to set up and run (owing to the fact that it isn’t
>>>> part of the monorepo, isn’t connected to check-all etc), therefore making
>>>> it harder for developers to maintain the tests. Back in October 2019, there
>>>> was an extensive discussion on end-to-end testing and how to write them
>>>> (starting from
>>>> https://lists.llvm.org/pipermail/cfe-dev/2019-October/063509.html).
>>>> The suggestion was that these tests would be lit-based and run as part of
>>>> check-all, and would not be inside the clang tree, although there was some
>>>> opposition. This concluded with a round table. Unfortunately, I am unaware
>>>> of what the conclusion of that round table conversation was, so it’s
>>>> possible that what I am proposing is redundant/being worked on by someone
>>>> else. Additionally, I don’t consider all classes of tests that the proposed
>>>> lit suite would be useful for to be “end-to-end” testing. For example,
>>>> llvm-symbolizer is usually used on linked output containing debug
>>>> information. Usually tests that consume objects that have debug data in
>>>> them rely on assembly that has been written by hand or generated by clang
>>>> prior to commit, with a limited set able to make use of yaml2obj to
>>>> generate the debug data instead. However, the output of these approaches is
>>>> typically not a fully linked output (yaml2obj output can be made to look
>>>> like one, but getting all the addresses to match up in a maintainable
>>>> manner makes this approach not particularly viable). Being able to use LLD
>>>> to link the object file produced would make the test significantly more
>>>> readable, much as using llvm-mc and assembly to generate test inputs is
>>>> more preferable to using prebuilt binaries. Such a test is ultimately not
>>>> really any more an end-to-end test than an llvm-symbolizer test that just
>>>> uses the object produced by the assembler directly.
>>>> >
>>>> >
>>>> >
>>>> > What do people think?
>>>>
>>>> Some concerns (the usual: Things should be tested in isolation, things
>>>> should be tested independently - but end to end tests have some value
>>>> too), but generally seems good.
>>>>
>>>
>>> Indeed this is a usual concern: such tests shouldn't be seen
>>> as replacing isolated lit tests ("unit tests").
>>>
>>
> I completely agree. Indeed, the llvm-symbolizer test referenced in the
> review I'd class as an isolated test - LLD was just being used to generate
> the input for the test. The key here is that the thing being tested was the
> llvm-symbolizer behaviour, and not the linker behaviour. As mentioned, this
> isn't really different to how llvm-mc or llc might be used to convert some
> input source (asm/IR etc) into something the tool under test wants to be
> run on. Potentially, changes in those tools might break things, but as long
> as the input is specific enough, this shouldn't happen often.
>
> But I have another question about the cost of maintenance here: are we
>>> gonna revert patches to either project when one of the integration tests
>>> fails?
>>>
>>
>> Possibly, yeah. If they demonstrate a bug.
>>
>
> That would be my intention - these tests should be classed as first-class
> citizens as much as any other lit test. They're just unit tests that use
> other components (LLD, clang etc) to generate inputs.
>
>
>>
>>> What about integration tests that require to be updated manually when
>>> changing another component?
>>>
>>
>> If they need to be updated, because their failure isn't representative of
>> a bug, yes.
>>
>
> Hopefully these sorts of failures would occur only infrequently. As noted,
> my aim here isn't to provide a place in opensource LLVM to do integration
> testing, but rather unit tests that just can't sit in the corresponding lit
> area, so input variability should be minimal.
>
> That being said, the proposed suite could be used for integration testing,
> if the community agreed such testing belonged in the monorepo - indeed, I
> have plans for some downstream integration tests that would make use of
> this if it lands - but that isn't the goal here.
>
>>
>>
>>> I find the cost of maintenance of end-to-end tests is often hard to
>>> carry over, especially since they are only supplementing and not replacing
>>> lit "unit-tests".
>>>
>>
>> One of the nice thing about end to end tests (as with all tests, if
>> designed carefully - eg: don't take some arbitrary code, compile it with
>> optimizations, and expect a very specific backtrace - optimizations might
>> lead to different line table/stack frame details (if some code was merged,
>> or moved, it might lose or gain a specific source file/line)) is that they
>> can be pretty resilient to implementation details, so less likely to need
>> updating due to changes in implementation details. If someone changes the
>> output format of llvm-symbolizer these would require updating and I think
>> it'd be reasonable to expect that to be updated/not left failing.
>>
>
> Right, just as we would change other tests for tools where the output
> format changed.
>
>
>> - Dave
>>
>>
>>>
>>> Best,
>>>
>>> --
>>> Mehdi
>>>
>>>
>>>
>>>>
>>>> Though perhaps debuginfo-tests (this presumably already supports the
>>>> multiple-subproject mechanical isssue you're discussing?) could be
>>>> generalized/renamed to be all our cross-project lit testing
>>>> (Essentially an in-monorepo, lit-only, high-reliability/not-flakey/etc
>>>> version of the test-suite).
>>>>
>>>>
> The existing debug-info test area looks like it could work if it was more
> generalised. It looks like we'd want the ability to have tests that would
> still work without clang/lldb being built, and similarly which could use
> LLD, but I don't think those are insurmountable issues - it would just
> require taking some of the ideas from my existing prototype (or equivalent
> from elsewhere) and merging them in.
>

Sounds good to me - might want some buy-in from other debuginfo-tests
folks, though.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210126/5f6ee80f/attachment.html>