[lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Zachary Turner via lldb-dev]

Note that we're going off topic from the original goal, and I just want to
re-iterate that I fully support smaller, incremental changes.  But since I
like talking about lit so much, I can't help but chime in :)

If we *did* want to move to a lit based system for the end to end based
tests, the first step would be to make an LLDBTestFormat and teach it to
literally just call dotest.py in single-process mode with the path to a
specific test file.

That would be a *great* first step, and also very manageable and bite
sized. I think we would see a measurable reduction in flakiness just from
this change.

On the other side though, I still think it is very important to increase
test coverage of *non* end-to-end tests.  When you increase the granularity
of the tests you're able to write, a lot more things become possible.  For
non end-to-end based tests, we can still use a more traditional llvm style
lit test based on lldb-test (which is still new, but the foundation is
there).

On Wed, Jan 17, 2018 at 2:59 PM Vedant Kumar <vsk at apple.com> wrote:

> Lit provides some helpful utilities which make it easier to write portable
> tests. E.g %t, for temporary test directories, and portable replacements
> for things like `diff -r`. This is how compiler-rt's end-to-end tests are
> structured, and we haven't needed any build-system like functionality there.
>
> vedant
>
> > On Jan 17, 2018, at 2:56 PM, Jim Ingham <jingham at apple.com> wrote:
> >
> >
> >> On Jan 17, 2018, at 2:55 PM, Adrian Prantl <aprantl at apple.com> wrote:
> >>
> >>
> >>
> >>> On Jan 17, 2018, at 2:50 PM, Jim Ingham <jingham at apple.com> wrote:
> >>>
> >>>
> >>>
> >>>> On Jan 17, 2018, at 2:49 PM, Adrian Prantl <aprantl at apple.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>>> On Jan 17, 2018, at 2:31 PM, Zachary Turner <zturner at google.com>
> wrote:
> >>>>>
> >>>>> I don't think new test authors really need to add CMake any more so
> than they currently need to understand Make.  Which is to say, not very
> much.  Most Makefiles are currently 1-2 lines of code that simply does
> nothing other than include the common Makefile.
> >>>>>
> >>>>> On the other hand, CMake defines a lot of constructs designed to
> support portable builds, so actually writing and maintaining that common
> CMake build file would be much easier.  The existing Makefile-based system
> already doesn't require you to understand the specific compiler invocations
> you want.  Here's 3 random Makefiles, which is hopefully representative
> given that I pulled them completely at random.
> >>>>>
> >>>>> breakpoint-commands/Makefile:
> >>>>> LEVEL = ../../../make
> >>>>> CXX_SOURCES := nested.cpp
> >>>>> include $(LEVEL)/Makefile.rules
> >>>>>
> >>>>> functionalities/inferior-assert:
> >>>>> LEVEL = ../../make
> >>>>> C_SOURCES := main.c
> >>>>> include $(LEVEL)/Makefile.rules
> >>>>>
> >>>>>
> >>>>> types:
> >>>>> LEVEL = ../make
> >>>>> # Example:
> >>>>> #
> >>>>> # CXX_SOURCES := int.cpp
> >>>>> include $(LEVEL)/Makefile.rules
> >>>>>
> >>>>> None of this is particularly interesting.  There are a very few
> tests that need to do something weird.  I opened 10 other random Makefiles
> and still didn't find any.  I don't believe it would be hard to support
> those cases.
> >>>>>
> >>>>> So now instead of "understand Make" it becomes "understand CMake".
> Whic is already a requirement of building LLVM.
> >>>>
> >>>> Fair point. I would suggest that I'll try to make LLDB's testsuite
> build out-of-tree using the existing Makefile system. That should be a
> generally useful first step. After doing this I will hopefully have a much
> better understanding of the requirements of the Makefiles and then we can
> revisit this idea with me actually knowing what I'm talking about :-)
> >>>>
> >>>>> If our test suite was lit-based where you actually had to write
> compiler invocations into the test files, I would feel differently, but
> that isn't what we have today.  We have something that is almost a direct
> mapping to using CMake.
> >>>>
> >>>> Question: how would you feel about converting the Makefiles to
> LIT-style .test files with very explicit RUN-lines?
> >>>
> >>> I'm not sure what you mean by this.
> >>
> >> Instead of using a build system at all to build the tests, we would
> hard-code the compiler and linker invocations without encoding any
> dependencies. Because we still need this to be configurable, it would
> probably look something like this:
> >>
> >> RUN: %CXX test.cpp -O0 %CXXFLAGS -o test.exe
> >> RUN: %test_driver test.exe mytest.py
> >
> > I'm worried we'd back into building another make system over time.  What
> advantage would we get from this.
> >
> > Jim
> >
> >
> >
> >>
> >> -- adrian
> >>
> >>>
> >>> Jim
> >>>
> >>>
> >>>>
> >>>> -- adrian
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20180117/4943e9ab/attachment-0001.html>