[llvm-dev] [RFC] Cross-project lit test suite

[David Greene via llvm-dev]

James Henderson via llvm-dev <llvm-dev at lists.llvm.org> writes:

> Currently, there is no location where lit tests that use both clang and LLD
> can be put, whilst the llvm-symbolizer cases I’ve hit are testing
> llvm-symbolizer (and not LLD), so don’t really fit in the LLD test suite. I
> therefore have prototyped a lit test suite that would be part of the
> monorepo, and which can support tests that use elements from multiple
> projects - see https://reviews.llvm.org/D95339. Tests could be added to
> this suite as needed. The suite is modelled as an additional top-level
> directory, and is enabled by enabling the “cross-project-tests” project in
> CMake.

This is fantastic!

> Back in October 2019, there was an extensive discussion on end-to-end
> testing and how to write them (starting from
> https://lists.llvm.org/pipermail/cfe-dev/2019-October/063509.html).
> The suggestion was that these tests would be lit-based and run as part
> of check-all, and would not be inside the clang tree, although there
> was some opposition. This concluded with a round table. Unfortunately,
> I am unaware of what the conclusion of that round table conversation
> was, so it’s possible that what I am proposing is redundant/being
> worked on by someone else.

I started that thread and IIRC we ended up with the suggestion that such
tests should live in test-suite.  As you noted having tests separated
from the monorepo is less than ideal.  I haven't done anything with this
conclusion yet, mostly due to lack of time.  If your proposal gains
traction I would like to see if we could build end-to-end testing on top
of it.

> Additionally, I don’t consider all classes of tests that the proposed
> lit suite would be useful for to be “end-to-end” testing.

Agreed.  There are various classes of tests that could make use of your
proposed layout, one of which is "end-to-end."  Your proposal doesn't
provide end-to-end testing per se, but it does make adding end-to-end
tests later on more straightforward.

             -David