[llvm-dev] [RFC] Cross-project lit test suite

[James Henderson via llvm-dev]

Dear all,


Recently, I and a number of my colleagues have run into cases where we
would like the ability to write tests that involve components from multiple
LLVM projects, for example using both clang and LLD. Similarly, I have seen
a few instances recently where tests would ideally make use of LLD but only
to help generate input objects for testing an LLVM tool, such as
llvm-symbolizer (see for example https://reviews.llvm.org/D88988).
Currently, there is no location where lit tests that use both clang and LLD
can be put, whilst the llvm-symbolizer cases I’ve hit are testing
llvm-symbolizer (and not LLD), so don’t really fit in the LLD test suite. I
therefore have prototyped a lit test suite that would be part of the
monorepo, and which can support tests that use elements from multiple
projects - see https://reviews.llvm.org/D95339. Tests could be added to
this suite as needed. The suite is modelled as an additional top-level
directory, and is enabled by enabling the “cross-project-tests” project in
CMake. I have initially added support for both LLD and clang. At
configuration time, the tests that require LLD or clang will only be
enabled when the respective projects are enabled, so that developers
continue to benefit from the subset of tests that are applicable for the
projects they are building. Note that I am not especially familiar with
CMake or lit, so this code may not be perfect, but it should be sufficient
to demonstrate what it can do.



One could argue that these sorts of tests should belong in the external (to
the monorepo) test-suite, but this is a) quite distant from the existing
testing, and therefore easily forgotten, delaying potential feedback for
any breakages/resulting in potentially duplicate testing etc, and b) is not
as easy to set up and run (owing to the fact that it isn’t part of the
monorepo, isn’t connected to check-all etc), therefore making it harder for
developers to maintain the tests. Back in October 2019, there was an
extensive discussion on end-to-end testing and how to write them (starting
from https://lists.llvm.org/pipermail/cfe-dev/2019-October/063509.html).
The suggestion was that these tests would be lit-based and run as part of
check-all, and would not be inside the clang tree, although there was some
opposition. This concluded with a round table. Unfortunately, I am unaware
of what the conclusion of that round table conversation was, so it’s
possible that what I am proposing is redundant/being worked on by someone
else. Additionally, I don’t consider all classes of tests that the proposed
lit suite would be useful for to be “end-to-end” testing. For example,
llvm-symbolizer is usually used on linked output containing debug
information. Usually tests that consume objects that have debug data in
them rely on assembly that has been written by hand or generated by clang
prior to commit, with a limited set able to make use of yaml2obj to
generate the debug data instead. However, the output of these approaches is
typically not a fully linked output (yaml2obj output can be made to look
like one, but getting all the addresses to match up in a maintainable
manner makes this approach not particularly viable). Being able to use LLD
to link the object file produced would make the test significantly more
readable, much as using llvm-mc and assembly to generate test inputs is
more preferable to using prebuilt binaries. Such a test is ultimately not
really any more an end-to-end test than an llvm-symbolizer test that just
uses the object produced by the assembler directly.


What do people think?

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210125/dbd7ccfd/attachment-0001.html>