[lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Jim Ingham via lldb-dev]

> On Jan 17, 2018, at 12:32 PM, Adrian Prantl <aprantl at apple.com> wrote:
> 
> Hi lldb-dev!
> 
> I've been investigating some spurious LLDB test suite failures on http://green.lab.llvm.org/green/ that had to do with build artifacts from previous runs lying around in the test directories and this prompted me to ask a couple of general noob questions about the LLDB testsuite.
> 
> My understanding is that all execution tests are compiled using using `make` in-tree. I.e.: the test driver (dotest.py) effectively executes something equivalent to `cd $srcdir/packages/.../mytest && make`. And it does this in a serial fashion for all configurations (dwarf, dSYM, dwo, ...) and relies on the `clean` target to be implemented correctly.
> 
> I don't understand all the design decisions that went into the LLDB testsuite, but my naive intuition tells me that this is sub-optimal (because of the serialization of the configurations) and dangerous (because it relies on make clean being implemented correctly). It seems to me that a better approach would be to create a separate build directory for each test variant and then invoke something like `cd $builddir/test/mytest.dwarf && make -C $srcdir/packages/.../mytest`. This way all configurations can build in parallel, and we can simply nuke the build directory afterwards and this way get rid of all custom implementations of the `clean` target.
> 
> - Is this already possible, and/or am I misunderstanding how it works?
> - Would this be a goal that is worthwhile to pursue?
> - Is there a good reason why we are not already doing it this way?

It would be really great to get all the binaries that you need for tests building outside of the test directory.  It was done in tree originally for expediency - the tests need to know where their binaries are, and that task is simple if they are in CWD of the test.  But it is annoying, both because it relies on each test to clean up after itself, and because you can't preserve one test run's results while making another, or preserve the debug variants.  Plus, jamming stuff willy-nilly into the source tree is not something you should do.

It shouldn't be that hard to make a parallel hierarchy for the tests in the build directory, and pass that to the test as the root for products.  That would be a valuable project!

Jim