[lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Jim Ingham via lldb-dev]

> On Jan 17, 2018, at 1:02 PM, Davide Italiano via lldb-dev <lldb-dev at lists.llvm.org> wrote:
> 
> On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev
> <lldb-dev at lists.llvm.org> wrote:
>> Hi lldb-dev!
>> 
>> I've been investigating some spurious LLDB test suite failures on http://green.lab.llvm.org/green/ that had to do with build artifacts from previous runs lying around in the test directories and this prompted me to ask a couple of general noob questions about the LLDB testsuite.
>> 
>> My understanding is that all execution tests are compiled using using `make` in-tree. I.e.: the test driver (dotest.py) effectively executes something equivalent to `cd $srcdir/packages/.../mytest && make`. And it does this in a serial fashion for all configurations (dwarf, dSYM, dwo, ...) and relies on the `clean` target to be implemented correctly.
>> 
>> I don't understand all the design decisions that went into the LLDB testsuite, but my naive intuition tells me that this is sub-optimal (because of the serialization of the configurations) and dangerous (because it relies on make clean being implemented correctly). It seems to me that a better approach would be to create a separate build directory for each test variant and then invoke something like `cd $builddir/test/mytest.dwarf && make -C $srcdir/packages/.../mytest`. This way all configurations can build in parallel, and we can simply nuke the build directory afterwards and this way get rid of all custom implementations of the `clean` target.
>> 
>> - Is this already possible, and/or am I misunderstanding how it works?
>> - Would this be a goal that is worthwhile to pursue?
>> - Is there a good reason why we are not already doing it this way?
>> 
> 
> As we're discussing lldb test suite changes, another detail that I
> find a little weird is that every time you execute the test suite you
> get a new build directory named after the time at which you run the
> test.
> It would be much much better IMHO to just have a `log/` generic
> directory where the failures are logged, and those who want to
> override this setting can just pass a flag.

I often use the fact that the past couple run's logs are preserved, and often find this valuable after the fact, so I'd like it to continue to be the default.  

If you want to change it, you can pass "-s log" or whatever you like to dotest.py.

Jim