[lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Pavel Labath via lldb-dev]

On 18 January 2018 at 17:52, Adrian Prantl <aprantl at apple.com> wrote:
>
>
>> On Jan 18, 2018, at 5:07 AM, Pavel Labath <labath at google.com> wrote:
>>
>> Looks like I missed a party. :)
>>
>> I'll try to give my thoughts on some of the things that were said here:
>>
>>> make -C
>> I don't think make -C does what you think it does. "make -C foo" is
>> basically equivalent to "cd foo && make", which is what we are doing
>> now already. Of course, you can make this work, but you would have to
>> pass an extra OUTDIR=... argument to make and then modify the
>> Makefiles to reference $(OUTDIR) for its outputs:
>> $(OUTDIR)/a.out: main.cc
>>  $(CC) -o $(OUTDIR)/a.out main.cc ...
>>
>> The standard way of doing an out-of-tree build with make is to have
>> the Makefile in the build-directory and to set the magic VPATH
>> variable in the Makefile (or as a part of make invocation). VPATH
>> alters make's search path, so when searching for a dependency foo and
>> the foo is not present in the current (build) directory, it will go
>> searching for it in the VPATH (source) directory. You still need to be
>> careful about paths in the command line (generally this means using
>> make variables like $@ and $< instead of bare file names), but our
>> makefiles are generally pretty good at this already. We even have a
>> couple of makefiles using VPATH already (see TestConcurrentEvents) --
>> Todd added this to speed up the build by spreading out tests over
>> different folders while sharing sources (the serial execution
>> problem).
>>
> This is of course correct. Thanks for pointing this out and for outlining all the necessary changes!
>
>> I still fully support being able to build the tests out of tree, I
>> just think it may be a bit more involved than you realise.
>
> Sounds good.
>
>>
>>> cmake
>> I agree that using cmake for building tests would some things simpler.
>> Building fully working executables is fairly tricky, especially when
>> you're cross-compiling. Take test/testcases/Android.rules for example.
>> This is basically a reimplementation of the android cmake toolchain
>> file distributed with the android ndk. If we had cmake for building
>> tests we could delete all of that and replace it with
>> -DCMAKE_TOOLCHAIN_FILE=$(ANDROID_NDK_HOME)/android.toolchain.cmake.
>> However, I only had to write Android.rules just once, so it's not that
>> big of a deal for me.
>>
>>> explicit RUN lines:
>> Yes, it's true that all you need is custom CXXFLAGS (and LDFLAGS), but
>> those CXX flags could be quite complex. I'm not really opposed to
>> that, but I don't see any immediate benefit either (the only impact
>> for me would be that I'd have to translate Android.rules to python).
>>
>>> running clean after every test
>> Currently we must run "make clean" after every test because make does
>> not track the changes in it's arguments. So, if you first run "make
>> MAKE_DWO=NO" and then "make MAKE_DWO=YES", make will happily declare
>> that it has nothing to do without building the DWO binary. (This will
>> go away if we run each test variant in a separate directory).
>>
>> I don't have any data, but I would expect that running make upfront
>> would make a significant performance improvement on windows (spinning
>> up tons of processes and creating/deleting a bunch of files tends to
>> be much slower there), but to have no noticable difference on other
>> platforms.
>>
>
> I have not thought of Windows being a possible bottleneck. That sounds plausible. I'm still wondering how useful incremental builds for the testsuite are going to be. When I'm looking at our bots, almost all incoming commits are not in LLDB, but in LLVM or Clang. If we are modeling dependencies correctly, then each of these commits that changes the compiler should still trigger a full rebuild of all testcases, even with CMake. The only situation where incremental builds would be useful are for a configuration where we are building the LLDB testsuite against a fixed compiler, such as a released/stable version of clang or gcc. That's not to say that that isn't an important use-case, too, of course, but it's not how the bots on green dragon are configured at the moment.
>
> -- adrian

It wouldn't speed up the bots, but it may make the workflow of an lldb
developer (who  changes only lldb code most of the time) faster.
However, that doesn't matter I guess, as that's not the direction
we're going now.