[llvm-dev] [RFC] Cross-project lit test suite

[David Blaikie via llvm-dev]

On Mon, Jan 25, 2021 at 4:20 PM Mehdi AMINI <joker.eph at gmail.com> wrote:

>
>
> On Mon, Jan 25, 2021 at 12:16 PM David Blaikie via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> On Mon, Jan 25, 2021 at 3:36 AM James Henderson via llvm-dev
>> <llvm-dev at lists.llvm.org> wrote:
>> >
>> > Dear all,
>> >
>> >
>> > Recently, I and a number of my colleagues have run into cases where we
>> would like the ability to write tests that involve components from multiple
>> LLVM projects, for example using both clang and LLD. Similarly, I have seen
>> a few instances recently where tests would ideally make use of LLD but only
>> to help generate input objects for testing an LLVM tool, such as
>> llvm-symbolizer (see for example https://reviews.llvm.org/D88988).
>> Currently, there is no location where lit tests that use both clang and LLD
>> can be put, whilst the llvm-symbolizer cases I’ve hit are testing
>> llvm-symbolizer (and not LLD), so don’t really fit in the LLD test suite. I
>> therefore have prototyped a lit test suite that would be part of the
>> monorepo, and which can support tests that use elements from multiple
>> projects - see https://reviews.llvm.org/D95339. Tests could be added to
>> this suite as needed. The suite is modelled as an additional top-level
>> directory, and is enabled by enabling the “cross-project-tests” project in
>> CMake. I have initially added support for both LLD and clang. At
>> configuration time, the tests that require LLD or clang will only be
>> enabled when the respective projects are enabled, so that developers
>> continue to benefit from the subset of tests that are applicable for the
>> projects they are building. Note that I am not especially familiar with
>> CMake or lit, so this code may not be perfect, but it should be sufficient
>> to demonstrate what it can do.
>> >
>> >
>> >
>> > One could argue that these sorts of tests should belong in the external
>> (to the monorepo) test-suite, but this is a) quite distant from the
>> existing testing, and therefore easily forgotten, delaying potential
>> feedback for any breakages/resulting in potentially duplicate testing etc,
>> and b) is not as easy to set up and run (owing to the fact that it isn’t
>> part of the monorepo, isn’t connected to check-all etc), therefore making
>> it harder for developers to maintain the tests. Back in October 2019, there
>> was an extensive discussion on end-to-end testing and how to write them
>> (starting from
>> https://lists.llvm.org/pipermail/cfe-dev/2019-October/063509.html). The
>> suggestion was that these tests would be lit-based and run as part of
>> check-all, and would not be inside the clang tree, although there was some
>> opposition. This concluded with a round table. Unfortunately, I am unaware
>> of what the conclusion of that round table conversation was, so it’s
>> possible that what I am proposing is redundant/being worked on by someone
>> else. Additionally, I don’t consider all classes of tests that the proposed
>> lit suite would be useful for to be “end-to-end” testing. For example,
>> llvm-symbolizer is usually used on linked output containing debug
>> information. Usually tests that consume objects that have debug data in
>> them rely on assembly that has been written by hand or generated by clang
>> prior to commit, with a limited set able to make use of yaml2obj to
>> generate the debug data instead. However, the output of these approaches is
>> typically not a fully linked output (yaml2obj output can be made to look
>> like one, but getting all the addresses to match up in a maintainable
>> manner makes this approach not particularly viable). Being able to use LLD
>> to link the object file produced would make the test significantly more
>> readable, much as using llvm-mc and assembly to generate test inputs is
>> more preferable to using prebuilt binaries. Such a test is ultimately not
>> really any more an end-to-end test than an llvm-symbolizer test that just
>> uses the object produced by the assembler directly.
>> >
>> >
>> >
>> > What do people think?
>>
>> Some concerns (the usual: Things should be tested in isolation, things
>> should be tested independently - but end to end tests have some value
>> too), but generally seems good.
>>
>
> Indeed this is a usual concern: such tests shouldn't be seen as replacing
> isolated lit tests ("unit tests").
> But I have another question about the cost of maintenance here: are we
> gonna revert patches to either project when one of the integration tests
> fails?
>

Possibly, yeah. If they demonstrate a bug.


> What about integration tests that require to be updated manually when
> changing another component?
>

If they need to be updated, because their failure isn't representative of a
bug, yes.


> I find the cost of maintenance of end-to-end tests is often hard to carry
> over, especially since they are only supplementing and not replacing lit
> "unit-tests".
>

One of the nice thing about end to end tests (as with all tests, if
designed carefully - eg: don't take some arbitrary code, compile it with
optimizations, and expect a very specific backtrace - optimizations might
lead to different line table/stack frame details (if some code was merged,
or moved, it might lose or gain a specific source file/line)) is that they
can be pretty resilient to implementation details, so less likely to need
updating due to changes in implementation details. If someone changes the
output format of llvm-symbolizer these would require updating and I think
it'd be reasonable to expect that to be updated/not left failing.

- Dave


>
> Best,
>
> --
> Mehdi
>
>
>
>>
>> Though perhaps debuginfo-tests (this presumably already supports the
>> multiple-subproject mechanical isssue you're discussing?) could be
>> generalized/renamed to be all our cross-project lit testing
>> (Essentially an in-monorepo, lit-only, high-reliability/not-flakey/etc
>> version of the test-suite).
>>
>> - Dave
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210125/21bed3f6/attachment.html>