[lldb-dev] Questions about the LLDB testsuite and improving its reliability

Wed Jan 17 17:45:51 PST 2018

> On Jan 17, 2018, at 3:26 PM, Greg Clayton via lldb-dev <lldb-dev at lists.llvm.org> wrote:
> 
> Everything sounds good on this thread. My two cents:
> 
> We should add some post verification after each test that looks for file that are left over after the "clean" phase. This can help us catch the tests that aren't cleaning up after themselves. This will help us weed out the bad tests and fix this ASAP. This can be done both for in tree and out of tree solutions to verify there is no source polution.
> 
> We can easily move build artifacts out of the source tree. Running the test suite remotely via "lldb-server platform" has code that creates directories for each test in the platform working directory. If the test runs fine and passes, it cleans up the entire numbered test directory, else it leaves the numbered directory there so we can debug any issues. Shouldn't be hard to enable this.

For completeness, I looked at this and it doesn't look like that is how it works. My understanding (and keep in mind that this is the first time I am looking at this code so I might be misunderstanding something here) is that the remote platform support works by building the test on the host in-tree and then _RemoteProcess.launch() ships over only the binary when called from Base.spawnSubprocess().

That's not a big deal, though. I will introduce the concept of a build directory (which has to be separate from the remote platform working directory) and find a way to pass the source directory to runBuildCommands().

-- adrian

> 
> I like the current default of having a new directory with the time and data with results inside as it allows comparison of one test suite run to another.
> 
> Switching build systems to cmake is fine if someone has the energy to do it, that would be great. I don't see why we would go with a lit based system that manually specifies the arguments. Seems like a pain to get the right compiler flags for creating shared libs on different systems (or executables, frameworks, etc). Seems like cmake is good at that and very simple.
> 
> 
>> On Jan 17, 2018, at 3:18 PM, Jim Ingham via lldb-dev <lldb-dev at lists.llvm.org> wrote:
>> 
>> Yeah, w.r.t. the actual builder part, it seems to me any option is going to be sufficiently simple to use that it would be hard for the incremental benefits to lldb developers to ever amortize the cost of switching.  The only compelling reason to me is if one or the other tool made it much easier to get building the test cases out of tree working, but that seems unlikely.
>> 
>> Jim
>> 
>> 
>>> On Jan 17, 2018, at 3:07 PM, Zachary Turner <zturner at google.com> wrote:
>>> 
>>> 
>>> 
>>> On Wed, Jan 17, 2018 at 3:04 PM Adrian Prantl <aprantl at apple.com> wrote:
>>> 
>>> On the other hand:
>>> - everybody already knows make
>>> 
>>> I'm not sold on this particular reason.  Make is not the LLVM build system, CMake is.  "I don't know the build system of the project I actually work on, but I do know this other build system" is not a compelling argument. 
>>> 
>>> (As an aside, not every knows Make that well, but it doesn't actually matter because the amount of actual Make code is negligibly small, i.e. 1-2 lines per test in a vast majority of cases)
>> 
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> 
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev