[lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Adrian Prantl via lldb-dev]

> On Jan 17, 2018, at 1:45 PM, Vedant Kumar <vsk at apple.com> wrote:
> 
> I would prefer having all of our test dependencies tracked by CMake for all the reasons Zach brought up, but I think we should defer that undertaking until after the bots are more stable. We have some immediate problems caused by stale in-tree test artifacts. As a first milestone, it'd be great to not have to run `git clean -fdx` anymore.

I'm pretty sure I do not want to go all the way to CMake right now, but I am curious about your motivation: Why do you think that using CMake to build the tests in the testsuite is a good idea? In my opinion this adds a layer of complexity to the tests that makes it harder to understand what exactly is happening and test authors now need to understand CMake *and* the compiler invocations they want to execute in their tests. Do you also share Zachary's point of view that the tests should be build artifacts that should be kept after an incremental build?

-- adrian

> 
> 
>> On Jan 17, 2018, at 1:13 PM, Davide Italiano via lldb-dev <lldb-dev at lists.llvm.org> wrote:
>> 
>> On Wed, Jan 17, 2018 at 1:02 PM, Davide Italiano <dccitaliano at gmail.com> wrote:
>>> On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev
>>> <lldb-dev at lists.llvm.org> wrote:
>>>> Hi lldb-dev!
>>>> 
>>>> I've been investigating some spurious LLDB test suite failures on http://green.lab.llvm.org/green/ that had to do with build artifacts from previous runs lying around in the test directories and this prompted me to ask a couple of general noob questions about the LLDB testsuite.
>>>> 
>>>> My understanding is that all execution tests are compiled using using `make` in-tree. I.e.: the test driver (dotest.py) effectively executes something equivalent to `cd $srcdir/packages/.../mytest && make`. And it does this in a serial fashion for all configurations (dwarf, dSYM, dwo, ...) and relies on the `clean` target to be implemented correctly.
>>>> 
>>>> I don't understand all the design decisions that went into the LLDB testsuite, but my naive intuition tells me that this is sub-optimal (because of the serialization of the configurations) and dangerous (because it relies on make clean being implemented correctly). It seems to me that a better approach would be to create a separate build directory for each test variant and then invoke something like `cd $builddir/test/mytest.dwarf && make -C $srcdir/packages/.../mytest`. This way all configurations can build in parallel, and we can simply nuke the build directory afterwards and this way get rid of all custom implementations of the `clean` target.
> 
> This sgtm as a starting point.
> 
> vedant
> 
>>>> 
>>>> - Is this already possible, and/or am I misunderstanding how it works?
>>>> - Would this be a goal that is worthwhile to pursue?
>>>> - Is there a good reason why we are not already doing it this way?
>>>> 
>>> 
>>> As we're discussing lldb test suite changes, another detail that I
>>> find a little weird is that every time you execute the test suite you
>>> get a new build directory named after the time at which you run the
>>> test.
>>> It would be much much better IMHO to just have a `log/` generic
>>> directory where the failures are logged, and those who want to
>>> override this setting can just pass a flag.
>>> 
>> 
>> (The logs should also be moved out of tree, FWIW).
>> 
>> --
>> Davide
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev