[lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Greg Clayton via lldb-dev]

Everything sounds good on this thread. My two cents:

We should add some post verification after each test that looks for file that are left over after the "clean" phase. This can help us catch the tests that aren't cleaning up after themselves. This will help us weed out the bad tests and fix this ASAP. This can be done both for in tree and out of tree solutions to verify there is no source polution.

We can easily move build artifacts out of the source tree. Running the test suite remotely via "lldb-server platform" has code that creates directories for each test in the platform working directory. If the test runs fine and passes, it cleans up the entire numbered test directory, else it leaves the numbered directory there so we can debug any issues. Shouldn't be hard to enable this.

I like the current default of having a new directory with the time and data with results inside as it allows comparison of one test suite run to another.

Switching build systems to cmake is fine if someone has the energy to do it, that would be great. I don't see why we would go with a lit based system that manually specifies the arguments. Seems like a pain to get the right compiler flags for creating shared libs on different systems (or executables, frameworks, etc). Seems like cmake is good at that and very simple.


> On Jan 17, 2018, at 3:18 PM, Jim Ingham via lldb-dev <lldb-dev at lists.llvm.org> wrote:
> 
> Yeah, w.r.t. the actual builder part, it seems to me any option is going to be sufficiently simple to use that it would be hard for the incremental benefits to lldb developers to ever amortize the cost of switching.  The only compelling reason to me is if one or the other tool made it much easier to get building the test cases out of tree working, but that seems unlikely.
> 
> Jim
> 
> 
>> On Jan 17, 2018, at 3:07 PM, Zachary Turner <zturner at google.com> wrote:
>> 
>> 
>> 
>> On Wed, Jan 17, 2018 at 3:04 PM Adrian Prantl <aprantl at apple.com> wrote:
>> 
>> On the other hand:
>> - everybody already knows make
>> 
>> I'm not sold on this particular reason.  Make is not the LLVM build system, CMake is.  "I don't know the build system of the project I actually work on, but I do know this other build system" is not a compelling argument. 
>> 
>> (As an aside, not every knows Make that well, but it doesn't actually matter because the amount of actual Make code is negligibly small, i.e. 1-2 lines per test in a vast majority of cases)
> 
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev