[lldb-dev] Questions about the LLDB testsuite and improving its reliability

Wed Jan 17 15:12:32 PST 2018

I disagree that understanding CMake is required to build LLVM. When I build top-of-tree on Linux (as opposed to a build that is Hexagon only) I make a build directory at the same level as my checkout, and simply run “cmake ../llvm”. I don’t need to know anything.

--

Qualcomm Innovation Center, Inc.

The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project

From: lldb-dev [mailto:lldb-dev-bounces at lists.llvm.org] On Behalf Of Zachary Turner via lldb-dev
Sent: Wednesday, January 17, 2018 4:31 PM
To: Adrian Prantl <aprantl at apple.com>
Cc: LLDB <lldb-dev at lists.llvm.org>
Subject: Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

I don't think new test authors really need to add CMake any more so than they currently need to understand Make.  Which is to say, not very much.  Most Makefiles are currently 1-2 lines of code that simply does nothing other than include the common Makefile.

On the other hand, CMake defines a lot of constructs designed to support portable builds, so actually writing and maintaining that common CMake build file would be much easier.  The existing Makefile-based system already doesn't require you to understand the specific compiler invocations you want.  Here's 3 random Makefiles, which is hopefully representative given that I pulled them completely at random.

breakpoint-commands/Makefile:

LEVEL = ../../../make

CXX_SOURCES := nested.cpp

include $(LEVEL)/Makefile.rules

functionalities/inferior-assert:

LEVEL = ../../make

C_SOURCES := main.c

include $(LEVEL)/Makefile.rules

types:

LEVEL = ../make

# Example:

#

# CXX_SOURCES := int.cpp

include $(LEVEL)/Makefile.rules

None of this is particularly interesting.  There are a very few tests that need to do something weird.  I opened 10 other random Makefiles and still didn't find any.  I don't believe it would be hard to support those cases.

So now instead of "understand Make" it becomes "understand CMake".  Whic is already a requirement of building LLVM.

If our test suite was lit-based where you actually had to write compiler invocations into the test files, I would feel differently, but that isn't what we have today.  We have something that is almost a direct mapping to using CMake.

On Wed, Jan 17, 2018 at 2:22 PM Adrian Prantl <aprantl at apple.com <mailto:aprantl at apple.com> > wrote:

> On Jan 17, 2018, at 1:45 PM, Vedant Kumar <vsk at apple.com <mailto:vsk at apple.com> > wrote:
>
> I would prefer having all of our test dependencies tracked by CMake for all the reasons Zach brought up, but I think we should defer that undertaking until after the bots are more stable. We have some immediate problems caused by stale in-tree test artifacts. As a first milestone, it'd be great to not have to run `git clean -fdx` anymore.

I'm pretty sure I do not want to go all the way to CMake right now, but I am curious about your motivation: Why do you think that using CMake to build the tests in the testsuite is a good idea? In my opinion this adds a layer of complexity to the tests that makes it harder to understand what exactly is happening and test authors now need to understand CMake *and* the compiler invocations they want to execute in their tests. Do you also share Zachary's point of view that the tests should be build artifacts that should be kept after an incremental build?

-- adrian

>
>
>> On Jan 17, 2018, at 1:13 PM, Davide Italiano via lldb-dev <lldb-dev at lists.llvm.org <mailto:lldb-dev at lists.llvm.org> > wrote:
>>
>> On Wed, Jan 17, 2018 at 1:02 PM, Davide Italiano <dccitaliano at gmail.com <mailto:dccitaliano at gmail.com> > wrote:
>>> On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev
>>> <lldb-dev at lists.llvm.org <mailto:lldb-dev at lists.llvm.org> > wrote:
>>>> Hi lldb-dev!
>>>>
>>>> I've been investigating some spurious LLDB test suite failures on http://green.lab.llvm.org/green/ that had to do with build artifacts from previous runs lying around in the test directories and this prompted me to ask a couple of general noob questions about the LLDB testsuite.
>>>>
>>>> My understanding is that all execution tests are compiled using using `make` in-tree. I.e.: the test driver (dotest.py) effectively executes something equivalent to `cd $srcdir/packages/.../mytest && make`. And it does this in a serial fashion for all configurations (dwarf, dSYM, dwo, ...) and relies on the `clean` target to be implemented correctly.
>>>>
>>>> I don't understand all the design decisions that went into the LLDB testsuite, but my naive intuition tells me that this is sub-optimal (because of the serialization of the configurations) and dangerous (because it relies on make clean being implemented correctly). It seems to me that a better approach would be to create a separate build directory for each test variant and then invoke something like `cd $builddir/test/mytest.dwarf && make -C $srcdir/packages/.../mytest`. This way all configurations can build in parallel, and we can simply nuke the build directory afterwards and this way get rid of all custom implementations of the `clean` target.
>
> This sgtm as a starting point.
>
> vedant
>
>>>>
>>>> - Is this already possible, and/or am I misunderstanding how it works?
>>>> - Would this be a goal that is worthwhile to pursue?
>>>> - Is there a good reason why we are not already doing it this way?
>>>>
>>>
>>> As we're discussing lldb test suite changes, another detail that I
>>> find a little weird is that every time you execute the test suite you
>>> get a new build directory named after the time at which you run the
>>> test.
>>> It would be much much better IMHO to just have a `log/` generic
>>> directory where the failures are logged, and those who want to
>>> override this setting can just pass a flag.
>>>
>>
>> (The logs should also be moved out of tree, FWIW).
>>
>> --
>> Davide
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at lists.llvm.org <mailto:lldb-dev at lists.llvm.org> 
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20180117/6c8393f2/attachment.html>