[lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Adrian Prantl via lldb-dev]

> On Jan 17, 2018, at 12:41 PM, Zachary Turner <zturner at google.com> wrote:
> 
> If we're going to be making any significant changes to the way inferiors are compiled, why not use cmake?  Make clean is already not implemented correctly in many places, leading to lots of remnants left over in the source tree after test runs.  Furthermore, make is run every single time currently, leading to hundreds (if not thousands) of unnecessary compilations.  Seems to me like all the inferiors should be compiled one time, up front, as part of the configure step, and into the build directory.  This is nice because it already integrates perfectly into the existing LLVM "way" of building things.

To be honest, I have not considered the the tests to be part of the build. Doing so is an interesting idea that I haven't thought about so far, here are some thoughts about this:
- CMake is more a replacement for autoconf than for make and I'm not sure if we need a better tool for the "configuration" part of the testsuite.
- Some of the tests purposefully do weird stuff, such as deleting or damaging one .o file to test LLDB's abilities to cope with incomplete debug info. To implement this we will need to micro-manage things at the "make" level that could be hard to express in CMake.
- The LLVM/CFE way of building *tests* is to hard-code the commands to recompile the testcases every time you run the testsuite.
- We probably want to be able to run the LLDB testsuite using many different compilers. It could be possible that CMake is helpful for this, but then again I doubt that we need much more than setting a custom CC/CXX/CFLAGs to support this.

I don't want to immediately shoot this idea down, but I think it is one step further than I would like to go at this time. Thanks for pointing it out though!

-- adrian