[llvm-dev] [RFC] Cross-project lit test suite

[James Henderson via llvm-dev]

Thanks all for the input! I've culled the thread history to avoid bloat. To
summarise the concerns as I see them:

1) We don't want to have to test the world to make sure we haven't broken
tests somewhere else. E.g. ideally a change to LLD would only need to run
the lld/test tests.
2) Related to 1) - we don't want to have to build the world to run the
existing debug-info tests.
3) It's not clear what should be allowed into the cross-project test area.

Please let me know if I missed any.

For 1), maybe we just don't expect people to run check-all for every change
- in practice we don't already anyway, even if we claim we do. Build bots
will report (hopefully as a pre-merge check in Phabricator) any problems,
and the change can be then fixed/reverted as needed. It's possibly not
ideal, but I would expect that tests in this area would be relatively few
in number and stable, so the chances of a tool change causing them to fail
should be relatively slim. Indeed, in practice, that's what we already do
with other tools. For example, the LLVM binutils like llvm-readobj,
llvm-objdump and so on are often used to verify other tools' behaviour.
However, I wouldn't necessarily expect a developer who has made a small
change to them to run the whole set of LLVM tests across all projects,
especially if the change shouldn't change the output in 99% of cases. If
there are odd tests that need updating later, they are usually easy to fix
up.

For 2), my prototype adds REQUIRES tags for lld and clang, which are
enabled if the relevant projects are enabled. I'd envision changing the
existing debug-info tests to make use of these and other tags (lldb etc) in
the same manner. These could be configured on a per-directory basis, to
avoid needing to modify every test, if desired. If your CMake configuration
doesn't enable building LLD for example, this should mean the
check-debug-info (or whatever it ends up being) shouldn't build LLD and
would mark such tests as UNSUPPORTED. Alternatively/additionally, depending
on how we address 3), we could also divide the test directory into separate
sub-directories, which allow for
check-debug-info/check-lld-llvm-symbolizer/... to target just the specified
directories.

For 3) I don't have a great answer to this. Whatever we decide, I think it
should be documented, so that reviewers can point to the documentation to
keep out things that don't belong. Certainly, it seems like tests which are
likely to be fragile to changes in tools (especially those merely used to
generate inputs) would need to be avoided. I think we'd need to leave this
up to those involved in writing/reviewing any new tests to determine
whether this is likely to be an issue for any given test. One option could
be for now to limit the new tests to specifically llvm-symbolizer tests
that make use of LLD, as that is a known hole in test coverage. We could
then discuss each new category of tests (e.g. end-to-end testing as
discussed in David Blaikie's latest email) as the need for them arises. The
doc could then say something like:

"Only tests that meet one of the following criteria should be added to this
location:
1) <one or more things about the existing debug-info tests, best written by
the existing maintainers>.
2) Tests for llvm-symbolizer and llvm-addr2line which require LLD to create
a linked object for an input file.
Other tests will be considered on a case-by-case basis, and should be
discussed on the llvm-dev mailing list. Should other cases be approved,
they should be added to this list."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210127/0311c95a/attachment.html>