<div dir="ltr">Note that we're going off topic from the original goal, and I just want to re-iterate that I fully support smaller, incremental changes. But since I like talking about lit so much, I can't help but chime in :)<div><br></div><div>If we *did* want to move to a lit based system for the end to end based tests, the first step would be to make an LLDBTestFormat and teach it to literally just call dotest.py in single-process mode with the path to a specific test file.</div><div><br></div><div>That would be a *great* first step, and also very manageable and bite sized. I think we would see a measurable reduction in flakiness just from this change.</div><div><br></div><div>On the other side though, I still think it is very important to increase test coverage of *non* end-to-end tests. When you increase the granularity of the tests you're able to write, a lot more things become possible. For non end-to-end based tests, we can still use a more traditional llvm style lit test based on lldb-test (which is still new, but the foundation is there).</div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Jan 17, 2018 at 2:59 PM Vedant Kumar <<a href="mailto:vsk@apple.com">vsk@apple.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Lit provides some helpful utilities which make it easier to write portable tests. E.g %t, for temporary test directories, and portable replacements for things like `diff -r`. This is how compiler-rt's end-to-end tests are structured, and we haven't needed any build-system like functionality there.<br>
<br>
vedant<br>
<br>
> On Jan 17, 2018, at 2:56 PM, Jim Ingham <<a href="mailto:jingham@apple.com" target="_blank">jingham@apple.com</a>> wrote:<br>
><br>
><br>
>> On Jan 17, 2018, at 2:55 PM, Adrian Prantl <<a href="mailto:aprantl@apple.com" target="_blank">aprantl@apple.com</a>> wrote:<br>
>><br>
>><br>
>><br>
>>> On Jan 17, 2018, at 2:50 PM, Jim Ingham <<a href="mailto:jingham@apple.com" target="_blank">jingham@apple.com</a>> wrote:<br>
>>><br>
>>><br>
>>><br>
>>>> On Jan 17, 2018, at 2:49 PM, Adrian Prantl <<a href="mailto:aprantl@apple.com" target="_blank">aprantl@apple.com</a>> wrote:<br>
>>>><br>
>>>><br>
>>>><br>
>>>>> On Jan 17, 2018, at 2:31 PM, Zachary Turner <<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>> wrote:<br>
>>>>><br>
>>>>> I don't think new test authors really need to add CMake any more so than they currently need to understand Make. Which is to say, not very much. Most Makefiles are currently 1-2 lines of code that simply does nothing other than include the common Makefile.<br>
>>>>><br>
>>>>> On the other hand, CMake defines a lot of constructs designed to support portable builds, so actually writing and maintaining that common CMake build file would be much easier. The existing Makefile-based system already doesn't require you to understand the specific compiler invocations you want. Here's 3 random Makefiles, which is hopefully representative given that I pulled them completely at random.<br>
>>>>><br>
>>>>> breakpoint-commands/Makefile:<br>
>>>>> LEVEL = ../../../make<br>
>>>>> CXX_SOURCES := nested.cpp<br>
>>>>> include $(LEVEL)/Makefile.rules<br>
>>>>><br>
>>>>> functionalities/inferior-assert:<br>
>>>>> LEVEL = ../../make<br>
>>>>> C_SOURCES := main.c<br>
>>>>> include $(LEVEL)/Makefile.rules<br>
>>>>><br>
>>>>><br>
>>>>> types:<br>
>>>>> LEVEL = ../make<br>
>>>>> # Example:<br>
>>>>> #<br>
>>>>> # CXX_SOURCES := int.cpp<br>
>>>>> include $(LEVEL)/Makefile.rules<br>
>>>>><br>
>>>>> None of this is particularly interesting. There are a very few tests that need to do something weird. I opened 10 other random Makefiles and still didn't find any. I don't believe it would be hard to support those cases.<br>
>>>>><br>
>>>>> So now instead of "understand Make" it becomes "understand CMake". Whic is already a requirement of building LLVM.<br>
>>>><br>
>>>> Fair point. I would suggest that I'll try to make LLDB's testsuite build out-of-tree using the existing Makefile system. That should be a generally useful first step. After doing this I will hopefully have a much better understanding of the requirements of the Makefiles and then we can revisit this idea with me actually knowing what I'm talking about :-)<br>
>>>><br>
>>>>> If our test suite was lit-based where you actually had to write compiler invocations into the test files, I would feel differently, but that isn't what we have today. We have something that is almost a direct mapping to using CMake.<br>
>>>><br>
>>>> Question: how would you feel about converting the Makefiles to LIT-style .test files with very explicit RUN-lines?<br>
>>><br>
>>> I'm not sure what you mean by this.<br>
>><br>
>> Instead of using a build system at all to build the tests, we would hard-code the compiler and linker invocations without encoding any dependencies. Because we still need this to be configurable, it would probably look something like this:<br>
>><br>
>> RUN: %CXX test.cpp -O0 %CXXFLAGS -o test.exe<br>
>> RUN: %test_driver test.exe mytest.py<br>
><br>
> I'm worried we'd back into building another make system over time. What advantage would we get from this.<br>
><br>
> Jim<br>
><br>
><br>
><br>
>><br>
>> -- adrian<br>
>><br>
>>><br>
>>> Jim<br>
>>><br>
>>><br>
>>>><br>
>>>> -- adrian<br>
><br>
<br>
</blockquote></div>