[lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Jim Ingham via lldb-dev]

> On Jan 17, 2018, at 3:47 PM, Zachary Turner <zturner at google.com> wrote:
> 
> 
> 
> On Wed, Jan 17, 2018 at 3:39 PM Adrian Prantl <aprantl at apple.com> wrote:
> 
>> On Jan 17, 2018, at 3:25 PM, Zachary Turner <zturner at google.com> wrote:
>> 
>> I don't know what would be involved in getting the tests building out of tree with Make.  But I do know it would be simple with CMake.  I'm sure it's probably not terrible with Make either, I just don't know enough about it to say.
>> 
>> One thing that I do like about CMake is that it can be integrated into the existing LLDB build configuration step, which already uses CMake, to build inferiors up front.  This has the potential to speed up the test suite by an order of magnitude.
> 
> Since the tests in the LLDB testsuite are typically very small and don't import a lot of headers I'm not convinced that an incremental build of the tests will have a very big impact on the running time of the testsuite, but to be honest I also haven't benchmarked it to see where the time is really spent.
> 
> It actually is pretty significant.  Part of this is due to the fact that a single .py file has multiple tests, and the compile happens for every one of those tests, even though it just produces the exact same output every time.

Ack, that doesn't seem necessary, right?  The clean should happen as part of the test case object cleanup, and then make can figure out what needs to be built.  This would have to be done with a little care, since it puts the responsibility on any test that mucks with the built product to clean as a test cleanup, but I bet very few tests do that.

Again, only worth while after we actually measure the time spent in the various parts of the test run.  But this seems like not that hard to fix.

Jim