[llvm-dev] [RFC] Cross-project lit test suite

[James Henderson via llvm-dev]

I've gone ahead with a patch series that starts by moving the existing
debuginfo-tests into a new top-level directory, with the intent that future
tests can be written in other directories within the top-level one, without
needing to be "debuginfo-tests". Please take a look at the patch series and
see what you think!

James

On Wed, 27 Jan 2021 at 11:10, James Henderson <jh7370.2008 at my.bristol.ac.uk>
wrote:

> Thanks all for the input! I've culled the thread history to avoid bloat.
> To summarise the concerns as I see them:
>
> 1) We don't want to have to test the world to make sure we haven't broken
> tests somewhere else. E.g. ideally a change to LLD would only need to run
> the lld/test tests.
> 2) Related to 1) - we don't want to have to build the world to run the
> existing debug-info tests.
> 3) It's not clear what should be allowed into the cross-project test area.
>
> Please let me know if I missed any.
>
> For 1), maybe we just don't expect people to run check-all for every
> change - in practice we don't already anyway, even if we claim we do. Build
> bots will report (hopefully as a pre-merge check in Phabricator) any
> problems, and the change can be then fixed/reverted as needed. It's
> possibly not ideal, but I would expect that tests in this area would be
> relatively few in number and stable, so the chances of a tool change
> causing them to fail should be relatively slim. Indeed, in practice, that's
> what we already do with other tools. For example, the LLVM binutils like
> llvm-readobj, llvm-objdump and so on are often used to verify other tools'
> behaviour. However, I wouldn't necessarily expect a developer who has made
> a small change to them to run the whole set of LLVM tests across all
> projects, especially if the change shouldn't change the output in 99% of
> cases. If there are odd tests that need updating later, they are usually
> easy to fix up.
>
> For 2), my prototype adds REQUIRES tags for lld and clang, which are
> enabled if the relevant projects are enabled. I'd envision changing the
> existing debug-info tests to make use of these and other tags (lldb etc) in
> the same manner. These could be configured on a per-directory basis, to
> avoid needing to modify every test, if desired. If your CMake configuration
> doesn't enable building LLD for example, this should mean the
> check-debug-info (or whatever it ends up being) shouldn't build LLD and
> would mark such tests as UNSUPPORTED. Alternatively/additionally, depending
> on how we address 3), we could also divide the test directory into separate
> sub-directories, which allow for
> check-debug-info/check-lld-llvm-symbolizer/... to target just the specified
> directories.
>
> For 3) I don't have a great answer to this. Whatever we decide, I think it
> should be documented, so that reviewers can point to the documentation to
> keep out things that don't belong. Certainly, it seems like tests which are
> likely to be fragile to changes in tools (especially those merely used to
> generate inputs) would need to be avoided. I think we'd need to leave this
> up to those involved in writing/reviewing any new tests to determine
> whether this is likely to be an issue for any given test. One option could
> be for now to limit the new tests to specifically llvm-symbolizer tests
> that make use of LLD, as that is a known hole in test coverage. We could
> then discuss each new category of tests (e.g. end-to-end testing as
> discussed in David Blaikie's latest email) as the need for them arises. The
> doc could then say something like:
>
> "Only tests that meet one of the following criteria should be added to
> this location:
> 1) <one or more things about the existing debug-info tests, best written
> by the existing maintainers>.
> 2) Tests for llvm-symbolizer and llvm-addr2line which require LLD to
> create a linked object for an input file.
> Other tests will be considered on a case-by-case basis, and should be
> discussed on the llvm-dev mailing list. Should other cases be approved,
> they should be added to this list."
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210211/85872635/attachment-0001.html>