[lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Jim Ingham via lldb-dev]

> On Jan 17, 2018, at 2:49 PM, Adrian Prantl <aprantl at apple.com> wrote:
> 
> 
> 
>> On Jan 17, 2018, at 2:31 PM, Zachary Turner <zturner at google.com> wrote:
>> 
>> I don't think new test authors really need to add CMake any more so than they currently need to understand Make.  Which is to say, not very much.  Most Makefiles are currently 1-2 lines of code that simply does nothing other than include the common Makefile.
>> 
>> On the other hand, CMake defines a lot of constructs designed to support portable builds, so actually writing and maintaining that common CMake build file would be much easier.  The existing Makefile-based system already doesn't require you to understand the specific compiler invocations you want.  Here's 3 random Makefiles, which is hopefully representative given that I pulled them completely at random.
>> 
>> breakpoint-commands/Makefile:
>> LEVEL = ../../../make
>> CXX_SOURCES := nested.cpp
>> include $(LEVEL)/Makefile.rules
>> 
>> functionalities/inferior-assert:
>> LEVEL = ../../make
>> C_SOURCES := main.c
>> include $(LEVEL)/Makefile.rules
>> 
>> 
>> types:
>> LEVEL = ../make
>> # Example:
>> #
>> # CXX_SOURCES := int.cpp
>> include $(LEVEL)/Makefile.rules
>> 
>> None of this is particularly interesting.  There are a very few tests that need to do something weird.  I opened 10 other random Makefiles and still didn't find any.  I don't believe it would be hard to support those cases.
>> 
>> So now instead of "understand Make" it becomes "understand CMake".  Whic is already a requirement of building LLVM.
> 
> Fair point. I would suggest that I'll try to make LLDB's testsuite build out-of-tree using the existing Makefile system. That should be a generally useful first step. After doing this I will hopefully have a much better understanding of the requirements of the Makefiles and then we can revisit this idea with me actually knowing what I'm talking about :-)
> 
>> If our test suite was lit-based where you actually had to write compiler invocations into the test files, I would feel differently, but that isn't what we have today.  We have something that is almost a direct mapping to using CMake.
> 
> Question: how would you feel about converting the Makefiles to LIT-style .test files with very explicit RUN-lines?

I'm not sure what you mean by this.

Jim


> 
> -- adrian