[lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Jim Ingham via lldb-dev]

> On Jan 17, 2018, at 2:55 PM, Adrian Prantl <aprantl at apple.com> wrote:
> 
> 
> 
>> On Jan 17, 2018, at 2:50 PM, Jim Ingham <jingham at apple.com> wrote:
>> 
>> 
>> 
>>> On Jan 17, 2018, at 2:49 PM, Adrian Prantl <aprantl at apple.com> wrote:
>>> 
>>> 
>>> 
>>>> On Jan 17, 2018, at 2:31 PM, Zachary Turner <zturner at google.com> wrote:
>>>> 
>>>> I don't think new test authors really need to add CMake any more so than they currently need to understand Make.  Which is to say, not very much.  Most Makefiles are currently 1-2 lines of code that simply does nothing other than include the common Makefile.
>>>> 
>>>> On the other hand, CMake defines a lot of constructs designed to support portable builds, so actually writing and maintaining that common CMake build file would be much easier.  The existing Makefile-based system already doesn't require you to understand the specific compiler invocations you want.  Here's 3 random Makefiles, which is hopefully representative given that I pulled them completely at random.
>>>> 
>>>> breakpoint-commands/Makefile:
>>>> LEVEL = ../../../make
>>>> CXX_SOURCES := nested.cpp
>>>> include $(LEVEL)/Makefile.rules
>>>> 
>>>> functionalities/inferior-assert:
>>>> LEVEL = ../../make
>>>> C_SOURCES := main.c
>>>> include $(LEVEL)/Makefile.rules
>>>> 
>>>> 
>>>> types:
>>>> LEVEL = ../make
>>>> # Example:
>>>> #
>>>> # CXX_SOURCES := int.cpp
>>>> include $(LEVEL)/Makefile.rules
>>>> 
>>>> None of this is particularly interesting.  There are a very few tests that need to do something weird.  I opened 10 other random Makefiles and still didn't find any.  I don't believe it would be hard to support those cases.
>>>> 
>>>> So now instead of "understand Make" it becomes "understand CMake".  Whic is already a requirement of building LLVM.
>>> 
>>> Fair point. I would suggest that I'll try to make LLDB's testsuite build out-of-tree using the existing Makefile system. That should be a generally useful first step. After doing this I will hopefully have a much better understanding of the requirements of the Makefiles and then we can revisit this idea with me actually knowing what I'm talking about :-)
>>> 
>>>> If our test suite was lit-based where you actually had to write compiler invocations into the test files, I would feel differently, but that isn't what we have today.  We have something that is almost a direct mapping to using CMake.
>>> 
>>> Question: how would you feel about converting the Makefiles to LIT-style .test files with very explicit RUN-lines?
>> 
>> I'm not sure what you mean by this.
> 
> Instead of using a build system at all to build the tests, we would hard-code the compiler and linker invocations without encoding any dependencies. Because we still need this to be configurable, it would probably look something like this:
> 
>  RUN: %CXX test.cpp -O0 %CXXFLAGS -o test.exe
>  RUN: %test_driver test.exe mytest.py

I'm worried we'd back into building another make system over time.  What advantage would we get from this.

Jim



> 
> -- adrian
> 
>> 
>> Jim
>> 
>> 
>>> 
>>> -- adrian