[llvm-dev] [RFC] Cross-project lit test suite

Tue May 4 02:06:24 PDT 2021

It does sound like a good use-case to me. There are a few places we've come
across in the past where this sort of contract simply never gets tested
directly. Often it's the case A implies B is tested and B implies C is
tested, but we actually care that A implies C. Obviously, if A instead
started implying D, and D didn't imply C, the contract would be broken, but
testing wouldn't pick it up.

Sorry, my patch series stalled due to a combination of factors. I'll try to
carve out some time in the next week or two to push it forward a bit more.
Any assistance with the item discussed in
https://reviews.llvm.org/D95339#2640711 (i.e. using installed tools versus
those in the build directory) would of course be welcome.

On Mon, 3 May 2021 at 23:29, Thomas Lively <tlively at google.com> wrote:

> This RFC thread came up in discussion with David on
> https://reviews.llvm.org/D101684, where I was going to add end-to-end
> test testing that the WebAssembly SIMD intrinsics each lower to the correct
> Wasm instruction. That's a valuable test to have somewhere because the SIMD
> intrinsics have a contract between the source code and the specific
> instructions that are emitted; in the context of the intrinsics, the IR is
> an irrelevant implementation detail. An end-to-end test is easily able to
> show that the contract is upheld, but David correctly pointed out that it
> doesn't really belong in the clang directory because it tests LLVM as well.
>
> I would propose that the cross-project-tests directory also be used to
> host tests that test for such contracts between the source and the emitted
> instructions. The two areas that come to mind are intrinsics and ABIs. Does
> that sound like a reasonable use case?
>
> On Thu, Feb 11, 2021 at 7:53 AM James Henderson via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> I've gone ahead with a patch series that starts by moving the existing
>> debuginfo-tests into a new top-level directory, with the intent that future
>> tests can be written in other directories within the top-level one, without
>> needing to be "debuginfo-tests". Please take a look at the patch series and
>> see what you think!
>>
>> James
>>
>> On Wed, 27 Jan 2021 at 11:10, James Henderson <
>> jh7370.2008 at my.bristol.ac.uk> wrote:
>>
>>> Thanks all for the input! I've culled the thread history to avoid bloat.
>>> To summarise the concerns as I see them:
>>>
>>> 1) We don't want to have to test the world to make sure we haven't
>>> broken tests somewhere else. E.g. ideally a change to LLD would only need
>>> to run the lld/test tests.
>>> 2) Related to 1) - we don't want to have to build the world to run the
>>> existing debug-info tests.
>>> 3) It's not clear what should be allowed into the cross-project test
>>> area.
>>>
>>> Please let me know if I missed any.
>>>
>>> For 1), maybe we just don't expect people to run check-all for every
>>> change - in practice we don't already anyway, even if we claim we do. Build
>>> bots will report (hopefully as a pre-merge check in Phabricator) any
>>> problems, and the change can be then fixed/reverted as needed. It's
>>> possibly not ideal, but I would expect that tests in this area would be
>>> relatively few in number and stable, so the chances of a tool change
>>> causing them to fail should be relatively slim. Indeed, in practice, that's
>>> what we already do with other tools. For example, the LLVM binutils like
>>> llvm-readobj, llvm-objdump and so on are often used to verify other tools'
>>> behaviour. However, I wouldn't necessarily expect a developer who has made
>>> a small change to them to run the whole set of LLVM tests across all
>>> projects, especially if the change shouldn't change the output in 99% of
>>> cases. If there are odd tests that need updating later, they are usually
>>> easy to fix up.
>>>
>>> For 2), my prototype adds REQUIRES tags for lld and clang, which are
>>> enabled if the relevant projects are enabled. I'd envision changing the
>>> existing debug-info tests to make use of these and other tags (lldb etc) in
>>> the same manner. These could be configured on a per-directory basis, to
>>> avoid needing to modify every test, if desired. If your CMake configuration
>>> doesn't enable building LLD for example, this should mean the
>>> check-debug-info (or whatever it ends up being) shouldn't build LLD and
>>> would mark such tests as UNSUPPORTED. Alternatively/additionally, depending
>>> on how we address 3), we could also divide the test directory into separate
>>> sub-directories, which allow for
>>> check-debug-info/check-lld-llvm-symbolizer/... to target just the specified
>>> directories.
>>>
>>> For 3) I don't have a great answer to this. Whatever we decide, I think
>>> it should be documented, so that reviewers can point to the documentation
>>> to keep out things that don't belong. Certainly, it seems like tests which
>>> are likely to be fragile to changes in tools (especially those merely used
>>> to generate inputs) would need to be avoided. I think we'd need to leave
>>> this up to those involved in writing/reviewing any new tests to determine
>>> whether this is likely to be an issue for any given test. One option could
>>> be for now to limit the new tests to specifically llvm-symbolizer tests
>>> that make use of LLD, as that is a known hole in test coverage. We could
>>> then discuss each new category of tests (e.g. end-to-end testing as
>>> discussed in David Blaikie's latest email) as the need for them arises. The
>>> doc could then say something like:
>>>
>>> "Only tests that meet one of the following criteria should be added to
>>> this location:
>>> 1) <one or more things about the existing debug-info tests, best written
>>> by the existing maintainers>.
>>> 2) Tests for llvm-symbolizer and llvm-addr2line which require LLD to
>>> create a linked object for an input file.
>>> Other tests will be considered on a case-by-case basis, and should be
>>> discussed on the llvm-dev mailing list. Should other cases be approved,
>>> they should be added to this list."
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210504/66ebb428/attachment.html>