[lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Pavel Labath via lldb-dev]

Looks like I missed a party. :)

I'll try to give my thoughts on some of the things that were said here:

> make -C
I don't think make -C does what you think it does. "make -C foo" is
basically equivalent to "cd foo && make", which is what we are doing
now already. Of course, you can make this work, but you would have to
pass an extra OUTDIR=... argument to make and then modify the
Makefiles to reference $(OUTDIR) for its outputs:
$(OUTDIR)/a.out: main.cc
  $(CC) -o $(OUTDIR)/a.out main.cc ...

The standard way of doing an out-of-tree build with make is to have
the Makefile in the build-directory and to set the magic VPATH
variable in the Makefile (or as a part of make invocation). VPATH
alters make's search path, so when searching for a dependency foo and
the foo is not present in the current (build) directory, it will go
searching for it in the VPATH (source) directory. You still need to be
careful about paths in the command line (generally this means using
make variables like $@ and $< instead of bare file names), but our
makefiles are generally pretty good at this already. We even have a
couple of makefiles using VPATH already (see TestConcurrentEvents) --
Todd added this to speed up the build by spreading out tests over
different folders while sharing sources (the serial execution
problem).

I still fully support being able to build the tests out of tree, I
just think it may be a bit more involved than you realise.

> cmake
I agree that using cmake for building tests would some things simpler.
Building fully working executables is fairly tricky, especially when
you're cross-compiling. Take test/testcases/Android.rules for example.
This is basically a reimplementation of the android cmake toolchain
file distributed with the android ndk. If we had cmake for building
tests we could delete all of that and replace it with
-DCMAKE_TOOLCHAIN_FILE=$(ANDROID_NDK_HOME)/android.toolchain.cmake.
However, I only had to write Android.rules just once, so it's not that
big of a deal for me.

> explicit RUN lines:
Yes, it's true that all you need is custom CXXFLAGS (and LDFLAGS), but
those CXX flags could be quite complex. I'm not really opposed to
that, but I don't see any immediate benefit either (the only impact
for me would be that I'd have to translate Android.rules to python).

> running clean after every test
Currently we must run "make clean" after every test because make does
not track the changes in it's arguments. So, if you first run "make
MAKE_DWO=NO" and then "make MAKE_DWO=YES", make will happily declare
that it has nothing to do without building the DWO binary. (This will
go away if we run each test variant in a separate directory).

I don't have any data, but I would expect that running make upfront
would make a significant performance improvement on windows (spinning
up tons of processes and creating/deleting a bunch of files tends to
be much slower there), but to have no noticable difference on other
platforms.



On 18 January 2018 at 01:45, Adrian Prantl via lldb-dev
<lldb-dev at lists.llvm.org> wrote:
>
>
>> On Jan 17, 2018, at 3:26 PM, Greg Clayton via lldb-dev <lldb-dev at lists.llvm.org> wrote:
>>
>> Everything sounds good on this thread. My two cents:
>>
>> We should add some post verification after each test that looks for file that are left over after the "clean" phase. This can help us catch the tests that aren't cleaning up after themselves. This will help us weed out the bad tests and fix this ASAP. This can be done both for in tree and out of tree solutions to verify there is no source polution.
>>
>> We can easily move build artifacts out of the source tree. Running the test suite remotely via "lldb-server platform" has code that creates directories for each test in the platform working directory. If the test runs fine and passes, it cleans up the entire numbered test directory, else it leaves the numbered directory there so we can debug any issues. Shouldn't be hard to enable this.
>
> For completeness, I looked at this and it doesn't look like that is how it works. My understanding (and keep in mind that this is the first time I am looking at this code so I might be misunderstanding something here) is that the remote platform support works by building the test on the host in-tree and then _RemoteProcess.launch() ships over only the binary when called from Base.spawnSubprocess().
>
> That's not a big deal, though. I will introduce the concept of a build directory (which has to be separate from the remote platform working directory) and find a way to pass the source directory to runBuildCommands().
>
> -- adrian
>
>>
>> I like the current default of having a new directory with the time and data with results inside as it allows comparison of one test suite run to another.
>>
>> Switching build systems to cmake is fine if someone has the energy to do it, that would be great. I don't see why we would go with a lit based system that manually specifies the arguments. Seems like a pain to get the right compiler flags for creating shared libs on different systems (or executables, frameworks, etc). Seems like cmake is good at that and very simple.
>>
>>
>>> On Jan 17, 2018, at 3:18 PM, Jim Ingham via lldb-dev <lldb-dev at lists.llvm.org> wrote:
>>>
>>> Yeah, w.r.t. the actual builder part, it seems to me any option is going to be sufficiently simple to use that it would be hard for the incremental benefits to lldb developers to ever amortize the cost of switching.  The only compelling reason to me is if one or the other tool made it much easier to get building the test cases out of tree working, but that seems unlikely.
>>>
>>> Jim
>>>
>>>
>>>> On Jan 17, 2018, at 3:07 PM, Zachary Turner <zturner at google.com> wrote:
>>>>
>>>>
>>>>
>>>> On Wed, Jan 17, 2018 at 3:04 PM Adrian Prantl <aprantl at apple.com> wrote:
>>>>
>>>> On the other hand:
>>>> - everybody already knows make
>>>>
>>>> I'm not sold on this particular reason.  Make is not the LLVM build system, CMake is.  "I don't know the build system of the project I actually work on, but I do know this other build system" is not a compelling argument.
>>>>
>>>> (As an aside, not every knows Make that well, but it doesn't actually matter because the amount of actual Make code is negligibly small, i.e. 1-2 lines per test in a vast majority of cases)
>>>
>>> _______________________________________________
>>> lldb-dev mailing list
>>> lldb-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev