[lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Adrian Prantl via lldb-dev]

> On Jan 17, 2018, at 1:13 PM, Davide Italiano <dccitaliano at gmail.com> wrote:
> 
> On Wed, Jan 17, 2018 at 1:02 PM, Davide Italiano <dccitaliano at gmail.com> wrote:
>> On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev
>> <lldb-dev at lists.llvm.org> wrote:
>>> Hi lldb-dev!
>>> 
>>> I've been investigating some spurious LLDB test suite failures on http://green.lab.llvm.org/green/ that had to do with build artifacts from previous runs lying around in the test directories and this prompted me to ask a couple of general noob questions about the LLDB testsuite.
>>> 
>>> My understanding is that all execution tests are compiled using using `make` in-tree. I.e.: the test driver (dotest.py) effectively executes something equivalent to `cd $srcdir/packages/.../mytest && make`. And it does this in a serial fashion for all configurations (dwarf, dSYM, dwo, ...) and relies on the `clean` target to be implemented correctly.
>>> 
>>> I don't understand all the design decisions that went into the LLDB testsuite, but my naive intuition tells me that this is sub-optimal (because of the serialization of the configurations) and dangerous (because it relies on make clean being implemented correctly). It seems to me that a better approach would be to create a separate build directory for each test variant and then invoke something like `cd $builddir/test/mytest.dwarf && make -C $srcdir/packages/.../mytest`. This way all configurations can build in parallel, and we can simply nuke the build directory afterwards and this way get rid of all custom implementations of the `clean` target.
>>> 
>>> - Is this already possible, and/or am I misunderstanding how it works?
>>> - Would this be a goal that is worthwhile to pursue?
>>> - Is there a good reason why we are not already doing it this way?
>>> 
>> 
>> As we're discussing lldb test suite changes, another detail that I
>> find a little weird is that every time you execute the test suite you
>> get a new build directory named after the time at which you run the
>> test.
>> It would be much much better IMHO to just have a `log/` generic
>> directory where the failures are logged, and those who want to
>> override this setting can just pass a flag.
>> 
> 
> (The logs should also be moved out of tree, FWIW).

If I'm going to move the test build artifacts out-of-source-tree the logs would naturally end up there, too. Let's discuss whether creating a timestamped log directory should be the default or an option in a different thread to keep things simple. This is entirely orthogonal.

-- adrian

> 
> --
> Davide