[lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Zachary Turner via lldb-dev]

I don't think new test authors really need to add CMake any more so than
they currently need to understand Make.  Which is to say, not very much.
Most Makefiles are currently 1-2 lines of code that simply does nothing
other than include the common Makefile.

On the other hand, CMake defines a lot of constructs designed to support
portable builds, so actually writing and maintaining that common CMake
build file would be much easier.  The existing Makefile-based system
already doesn't require you to understand the specific compiler invocations
you want.  Here's 3 random Makefiles, which is hopefully representative
given that I pulled them completely at random.

breakpoint-commands/Makefile:
LEVEL = ../../../make
CXX_SOURCES := nested.cpp
include $(LEVEL)/Makefile.rules

functionalities/inferior-assert:
LEVEL = ../../make
C_SOURCES := main.c
include $(LEVEL)/Makefile.rules


types:
LEVEL = ../make
# Example:
#
# CXX_SOURCES := int.cpp
include $(LEVEL)/Makefile.rules

None of this is particularly interesting.  There are a very few tests that
need to do something weird.  I opened 10 other random Makefiles and still
didn't find any.  I don't believe it would be hard to support those cases.

So now instead of "understand Make" it becomes "understand CMake".  Whic is
already a requirement of building LLVM.

If our test suite was lit-based where you actually had to write compiler
invocations into the test files, I would feel differently, but that isn't
what we have today.  We have something that is almost a direct mapping to
using CMake.

On Wed, Jan 17, 2018 at 2:22 PM Adrian Prantl <aprantl at apple.com> wrote:

>
>
> > On Jan 17, 2018, at 1:45 PM, Vedant Kumar <vsk at apple.com> wrote:
> >
> > I would prefer having all of our test dependencies tracked by CMake for
> all the reasons Zach brought up, but I think we should defer that
> undertaking until after the bots are more stable. We have some immediate
> problems caused by stale in-tree test artifacts. As a first milestone, it'd
> be great to not have to run `git clean -fdx` anymore.
>
> I'm pretty sure I do not want to go all the way to CMake right now, but I
> am curious about your motivation: Why do you think that using CMake to
> build the tests in the testsuite is a good idea? In my opinion this adds a
> layer of complexity to the tests that makes it harder to understand what
> exactly is happening and test authors now need to understand CMake *and*
> the compiler invocations they want to execute in their tests. Do you also
> share Zachary's point of view that the tests should be build artifacts that
> should be kept after an incremental build?
>
> -- adrian
>
> >
> >
> >> On Jan 17, 2018, at 1:13 PM, Davide Italiano via lldb-dev <
> lldb-dev at lists.llvm.org> wrote:
> >>
> >> On Wed, Jan 17, 2018 at 1:02 PM, Davide Italiano <dccitaliano at gmail.com>
> wrote:
> >>> On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev
> >>> <lldb-dev at lists.llvm.org> wrote:
> >>>> Hi lldb-dev!
> >>>>
> >>>> I've been investigating some spurious LLDB test suite failures on
> http://green.lab.llvm.org/green/ that had to do with build artifacts from
> previous runs lying around in the test directories and this prompted me to
> ask a couple of general noob questions about the LLDB testsuite.
> >>>>
> >>>> My understanding is that all execution tests are compiled using using
> `make` in-tree. I.e.: the test driver (dotest.py) effectively executes
> something equivalent to `cd $srcdir/packages/.../mytest && make`. And it
> does this in a serial fashion for all configurations (dwarf, dSYM, dwo,
> ...) and relies on the `clean` target to be implemented correctly.
> >>>>
> >>>> I don't understand all the design decisions that went into the LLDB
> testsuite, but my naive intuition tells me that this is sub-optimal
> (because of the serialization of the configurations) and dangerous (because
> it relies on make clean being implemented correctly). It seems to me that a
> better approach would be to create a separate build directory for each test
> variant and then invoke something like `cd $builddir/test/mytest.dwarf &&
> make -C $srcdir/packages/.../mytest`. This way all configurations can build
> in parallel, and we can simply nuke the build directory afterwards and this
> way get rid of all custom implementations of the `clean` target.
> >
> > This sgtm as a starting point.
> >
> > vedant
> >
> >>>>
> >>>> - Is this already possible, and/or am I misunderstanding how it works?
> >>>> - Would this be a goal that is worthwhile to pursue?
> >>>> - Is there a good reason why we are not already doing it this way?
> >>>>
> >>>
> >>> As we're discussing lldb test suite changes, another detail that I
> >>> find a little weird is that every time you execute the test suite you
> >>> get a new build directory named after the time at which you run the
> >>> test.
> >>> It would be much much better IMHO to just have a `log/` generic
> >>> directory where the failures are logged, and those who want to
> >>> override this setting can just pass a flag.
> >>>
> >>
> >> (The logs should also be moved out of tree, FWIW).
> >>
> >> --
> >> Davide
> >> _______________________________________________
> >> lldb-dev mailing list
> >> lldb-dev at lists.llvm.org
> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20180117/38bfb665/attachment-0001.html>