[llvm-dev] [RFC] Cross-project lit test suite

Mon Feb 8 09:31:02 PST 2021

I haven't done any Windows work with debuginfo-tests on Windows since the
dexter transition. I had to disable the tests on my bot because they didn't
pass out of the box, and I never found time to follow up. It seems like
other people have more leads as to what exactly is going wrong. I'd love to
re-enable them, but I probably don't have time to debug them.

On Fri, Feb 5, 2021 at 10:50 AM David Blaikie <dblaikie at gmail.com> wrote:

> Reid and Amy might have some context for Windows (though I don't know
> if any Windows folks have done much with this test suite).
>
> On Fri, Feb 5, 2021 at 7:38 AM James Henderson via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> >
> > Given that the debuginfo tests already have cross-project dependencies,
> I figured I'd try adapting them instead. I've updated
> https://reviews.llvm.org/D95339 accordingly. Ideally, I think making the
> existing debug-info tests a subdirectory, and renaming the top-level
> directory, might be a good idea, but I haven't really come to any
> conclusions about that yet.
> >
> > I also found that several of the existing debuginfo-test tests fail for
> me. Are these tests expected to work on Windows? If so, are there any
> slightly more unusual prerequisites that I might be missing?
> >
> > What do people think?
> >
> > James
> >
> > On Wed, 27 Jan 2021 at 15:40, <paul.robinson at sony.com> wrote:
> >>
> >>
> >>
> >> > -----Original Message-----
> >> > From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of David
> Greene
> >> > via llvm-dev
> >> > Sent: Wednesday, January 27, 2021 10:29 AM
> >> > To: jh7370.2008 at my.bristol.ac.uk; llvm-dev <llvm-dev at lists.llvm.org>
> >> > Subject: Re: [llvm-dev] [RFC] Cross-project lit test suite
> >> >
> >> > James Henderson via llvm-dev <llvm-dev at lists.llvm.org> writes:
> >> >
> >> > > Currently, there is no location where lit tests that use both clang
> and
> >> > LLD
> >> > > can be put, whilst the llvm-symbolizer cases I’ve hit are testing
> >> > > llvm-symbolizer (and not LLD), so don’t really fit in the LLD test
> >> > suite. I
> >> > > therefore have prototyped a lit test suite that would be part of the
> >> > > monorepo, and which can support tests that use elements from
> multiple
> >> > > projects - see
> >> >
> https://urldefense.com/v3/__https://reviews.llvm.org/D95339__;!!JmoZiZGBv3
> >> >
> RvKRSx!vWidWrbKJid6-eIKVUT-dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqKCvkqW4A$
> >> > . Tests could be added to
> >> > > this suite as needed. The suite is modelled as an additional
> top-level
> >> > > directory, and is enabled by enabling the “cross-project-tests”
> project
> >> > in
> >> > > CMake.
> >> >
> >> > This is fantastic!
> >> >
> >> > > Back in October 2019, there was an extensive discussion on
> end-to-end
> >> > > testing and how to write them (starting from
> >> > > https://urldefense.com/v3/__https://lists.llvm.org/pipermail/cfe-
> >> > dev/2019-October/063509.html__;!!JmoZiZGBv3RvKRSx!vWidWrbKJid6-eIKVUT-
> >> > dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqJU8k_M_Q$ ).
> >> > > The suggestion was that these tests would be lit-based and run as
> part
> >> > > of check-all, and would not be inside the clang tree, although there
> >> > > was some opposition. This concluded with a round table.
> Unfortunately,
> >> > > I am unaware of what the conclusion of that round table conversation
> >> > > was, so it’s possible that what I am proposing is redundant/being
> >> > > worked on by someone else.
> >> >
> >> > I started that thread and IIRC we ended up with the suggestion that
> such
> >> > tests should live in test-suite.  As you noted having tests separated
> >> > from the monorepo is less than ideal.  I haven't done anything with
> this
> >> > conclusion yet, mostly due to lack of time.  If your proposal gains
> >> > traction I would like to see if we could build end-to-end testing on
> top
> >> > of it.
> >> >
> >> > > Additionally, I don’t consider all classes of tests that the
> proposed
> >> > > lit suite would be useful for to be “end-to-end” testing.
> >> >
> >> > Agreed.  There are various classes of tests that could make use of
> your
> >> > proposed layout, one of which is "end-to-end."  Your proposal doesn't
> >> > provide end-to-end testing per se, but it does make adding end-to-end
> >> > tests later on more straightforward.
> >> >
> >> >              -David
> >>
> >> I think a useful distinction here is that lit tests are generally very
> >> focused on a specific feature/function, where test-suite has a much
> >> broader scope.  Another slice at it would be that lit tests tend to be
> >> "regression" tests, while test-suite is more of an "integration" suite.
> >>
> >> I am not a QA person so I may be abusing some of these terms, but that's
> >> how I look at it.  Sometimes writing that focused lit test ends up
> >> depending on multiple tools, and the cross-project lit suite would be a
> >> good place to drop those; debuginfo-tests is a prime example.
> >> --paulr
> >>
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210208/5335ae89/attachment.html>