[llvm-dev] [RFC] Cross-project lit test suite

[Thomas Lively via llvm-dev]

This RFC thread came up in discussion with David on
https://reviews.llvm.org/D101684, where I was going to add end-to-end test
testing that the WebAssembly SIMD intrinsics each lower to the correct Wasm
instruction. That's a valuable test to have somewhere because the SIMD
intrinsics have a contract between the source code and the specific
instructions that are emitted; in the context of the intrinsics, the IR is
an irrelevant implementation detail. An end-to-end test is easily able to
show that the contract is upheld, but David correctly pointed out that it
doesn't really belong in the clang directory because it tests LLVM as well.

I would propose that the cross-project-tests directory also be used to host
tests that test for such contracts between the source and the emitted
instructions. The two areas that come to mind are intrinsics and ABIs. Does
that sound like a reasonable use case?

On Thu, Feb 11, 2021 at 7:53 AM James Henderson via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> I've gone ahead with a patch series that starts by moving the existing
> debuginfo-tests into a new top-level directory, with the intent that future
> tests can be written in other directories within the top-level one, without
> needing to be "debuginfo-tests". Please take a look at the patch series and
> see what you think!
>
> James
>
> On Wed, 27 Jan 2021 at 11:10, James Henderson <
> jh7370.2008 at my.bristol.ac.uk> wrote:
>
>> Thanks all for the input! I've culled the thread history to avoid bloat.
>> To summarise the concerns as I see them:
>>
>> 1) We don't want to have to test the world to make sure we haven't broken
>> tests somewhere else. E.g. ideally a change to LLD would only need to run
>> the lld/test tests.
>> 2) Related to 1) - we don't want to have to build the world to run the
>> existing debug-info tests.
>> 3) It's not clear what should be allowed into the cross-project test area.
>>
>> Please let me know if I missed any.
>>
>> For 1), maybe we just don't expect people to run check-all for every
>> change - in practice we don't already anyway, even if we claim we do. Build
>> bots will report (hopefully as a pre-merge check in Phabricator) any
>> problems, and the change can be then fixed/reverted as needed. It's
>> possibly not ideal, but I would expect that tests in this area would be
>> relatively few in number and stable, so the chances of a tool change
>> causing them to fail should be relatively slim. Indeed, in practice, that's
>> what we already do with other tools. For example, the LLVM binutils like
>> llvm-readobj, llvm-objdump and so on are often used to verify other tools'
>> behaviour. However, I wouldn't necessarily expect a developer who has made
>> a small change to them to run the whole set of LLVM tests across all
>> projects, especially if the change shouldn't change the output in 99% of
>> cases. If there are odd tests that need updating later, they are usually
>> easy to fix up.
>>
>> For 2), my prototype adds REQUIRES tags for lld and clang, which are
>> enabled if the relevant projects are enabled. I'd envision changing the
>> existing debug-info tests to make use of these and other tags (lldb etc) in
>> the same manner. These could be configured on a per-directory basis, to
>> avoid needing to modify every test, if desired. If your CMake configuration
>> doesn't enable building LLD for example, this should mean the
>> check-debug-info (or whatever it ends up being) shouldn't build LLD and
>> would mark such tests as UNSUPPORTED. Alternatively/additionally, depending
>> on how we address 3), we could also divide the test directory into separate
>> sub-directories, which allow for
>> check-debug-info/check-lld-llvm-symbolizer/... to target just the specified
>> directories.
>>
>> For 3) I don't have a great answer to this. Whatever we decide, I think
>> it should be documented, so that reviewers can point to the documentation
>> to keep out things that don't belong. Certainly, it seems like tests which
>> are likely to be fragile to changes in tools (especially those merely used
>> to generate inputs) would need to be avoided. I think we'd need to leave
>> this up to those involved in writing/reviewing any new tests to determine
>> whether this is likely to be an issue for any given test. One option could
>> be for now to limit the new tests to specifically llvm-symbolizer tests
>> that make use of LLD, as that is a known hole in test coverage. We could
>> then discuss each new category of tests (e.g. end-to-end testing as
>> discussed in David Blaikie's latest email) as the need for them arises. The
>> doc could then say something like:
>>
>> "Only tests that meet one of the following criteria should be added to
>> this location:
>> 1) <one or more things about the existing debug-info tests, best written
>> by the existing maintainers>.
>> 2) Tests for llvm-symbolizer and llvm-addr2line which require LLD to
>> create a linked object for an input file.
>> Other tests will be considered on a case-by-case basis, and should be
>> discussed on the llvm-dev mailing list. Should other cases be approved,
>> they should be added to this list."
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210503/b2b27ce7/attachment.html>