[lldb-dev] [RFC] Testsuite in lldb & possible future directions

[Davide Italiano via lldb-dev]

On Tue, Feb 6, 2018 at 7:09 AM, Pavel Labath <labath at google.com> wrote:
> On 6 February 2018 at 04:11, Davide Italiano via lldb-dev
> <lldb-dev at lists.llvm.org> wrote:
>> Hi,
>> in the last couple of months a lot of people put a lot of attentions
>> and energy into lldb and we're starting to see the first results. I
>> decided to sit down and write this e-mail to state where we are and
>> what are some possible future directions for the projects, in terms of
>> better quality/higher testability.
>>
>> Current state:
>>
>> 1) We got the testsuite on MacOS to build with zero unexpected
>> successes and zero failures (modulo one change I'm going to push
>> tomorrow). This is a collaborative effort and it's very important
>> because it allows us to push for unexpected successes as failures on
>> the bots, allowing us to discover issues quicker. Other platform I
>> think are improving their state as well, mainly thanks to the work of
>> Pavel and Jan.
>
> I don't mean to belittle this statement, as I think the situation has
> definitely improved a lot lately, but I feel I have to point out
> that's I've never been able to get a clean test suite run on a mac
> (not even the "0 failures" kind of clean). I'm not sure what are these
> caused by, but I guess that's because the tests are still very much
> dependent on the environment. So, I have to ask: what kind of
> environment are you running those tests in?
>
> My machine is not a completely off-the-shelf mac, as it has some
> google-specific customizations. I don't really know what this
> encompasses, but I would be surprised if these impact the result of
> the test suite. If I had to bet, my money would be on your machines
> having some apple-specific stuff which is not available in regular
> macs.
>
> I tried an experiment today. I configured with: cmake ../../llvm
> -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -GNinja. First
> problem I ran into was that I couldn't run check-lldb, as the clang I
> have just built was unable to compile any of the test binaries due to
> missing headers (this could be a manifestation of the SDKROOT issue we
> ran into a couple of weeks ago). So, I tried running with the system
> compiler and I got this output:
>
> FAIL: test_c_global_variables_dwarf
> (lang/c/global_variables/TestGlobalVariables.py)
> FAIL: test_c_global_variables_gmodules
> (lang/c/global_variables/TestGlobalVariables.py)
> FAIL: test_dsym (functionalities/ubsan/basic/TestUbsanBasic.py)
> FAIL: test_dwarf (functionalities/ubsan/basic/TestUbsanBasic.py)
> FAIL: test_gmodules (functionalities/ubsan/basic/TestUbsanBasic.py)
> FAIL: test_with_python_api_dsym (lang/cpp/class_static/TestStaticVariables.py)
> FAIL: test_with_python_api_dwarf (lang/cpp/class_static/TestStaticVariables.py)
> FAIL: test_with_python_api_gmodules
> (lang/cpp/class_static/TestStaticVariables.py)
> ERROR: test_debug_info_for_apple_types_dsym
> (macosx/debug-info/apple_types/TestAppleTypesIsProduced.py)
> ERROR: test_debug_info_for_apple_types_dwarf
> (macosx/debug-info/apple_types/TestAppleTypesIsProduced.py)
> ERROR: test_debug_info_for_apple_types_gmodules
> (macosx/debug-info/apple_types/TestAppleTypesIsProduced.py)
> UNEXPECTED SUCCESS: test_lldbmi_output_grammar
> (tools/lldb-mi/syntax/TestMiSyntax.py)
> UNEXPECTED SUCCESS: test_process_interrupt_dsym
> (functionalities/thread/state/TestThreadStates.py)
> UNEXPECTED SUCCESS: test_process_interrupt_gmodules
> (functionalities/thread/state/TestThreadStates.py)
>

FWIW, I strongly believe we should all agree on a configuration to run
tests and standardize on that.
It's unfortunate that we have two build systems, but there are plans
to move away from manually generating xcodebuild, as many agree it's a
terrible maintenance burden.
So, FWIW, I'll share my conf (I'm on high Sierra):


git clone https://github.com/monorepo
symlink clang -> tools
symlink lldb -> tools
symlink libcxx -> projects (this particular one has caused lots of
trouble for me in the past, and I realized it's undocumented :()

cmake -GNinja -DCMAKE_BUILD_TYPE=Release ../llvm
ninja check-lldb

This *should* work just fine for every developer (and we should error
out if all the dependencies are not in place). If it doesn't, well,
it's a bug.
Can you please try with this and report all the bugs that you find?
I'll work with you to fix them, as I'm particularly interested in
getting the lldb experience for users flawlessly out-the-box (at least
on the platforms I work on :)

--
Davide