[lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Adrian Prantl via lldb-dev]

> On Jan 17, 2018, at 2:31 PM, Zachary Turner <zturner at google.com> wrote:
> 
> I don't think new test authors really need to add CMake any more so than they currently need to understand Make.  Which is to say, not very much.  Most Makefiles are currently 1-2 lines of code that simply does nothing other than include the common Makefile.
> 
> On the other hand, CMake defines a lot of constructs designed to support portable builds, so actually writing and maintaining that common CMake build file would be much easier.  The existing Makefile-based system already doesn't require you to understand the specific compiler invocations you want.  Here's 3 random Makefiles, which is hopefully representative given that I pulled them completely at random.
> 
> breakpoint-commands/Makefile:
> LEVEL = ../../../make
> CXX_SOURCES := nested.cpp
> include $(LEVEL)/Makefile.rules
> 
> functionalities/inferior-assert:
> LEVEL = ../../make
> C_SOURCES := main.c
> include $(LEVEL)/Makefile.rules
> 
> 
> types:
> LEVEL = ../make
> # Example:
> #
> # CXX_SOURCES := int.cpp
> include $(LEVEL)/Makefile.rules
> 
> None of this is particularly interesting.  There are a very few tests that need to do something weird.  I opened 10 other random Makefiles and still didn't find any.  I don't believe it would be hard to support those cases.
> 
> So now instead of "understand Make" it becomes "understand CMake".  Whic is already a requirement of building LLVM.

Fair point. I would suggest that I'll try to make LLDB's testsuite build out-of-tree using the existing Makefile system. That should be a generally useful first step. After doing this I will hopefully have a much better understanding of the requirements of the Makefiles and then we can revisit this idea with me actually knowing what I'm talking about :-)

> If our test suite was lit-based where you actually had to write compiler invocations into the test files, I would feel differently, but that isn't what we have today.  We have something that is almost a direct mapping to using CMake.

Question: how would you feel about converting the Makefiles to LIT-style .test files with very explicit RUN-lines?

-- adrian