[lldb-dev] [RFC] Testsuite in lldb & possible future directions

[Davide Italiano via lldb-dev]

On Tue, Feb 6, 2018 at 7:09 AM, Pavel Labath <labath at google.com> wrote:
> On 6 February 2018 at 04:11, Davide Italiano via lldb-dev
> <lldb-dev at lists.llvm.org> wrote:
>> Hi,
>> in the last couple of months a lot of people put a lot of attentions
>> and energy into lldb and we're starting to see the first results. I
>> decided to sit down and write this e-mail to state where we are and
>> what are some possible future directions for the projects, in terms of
>> better quality/higher testability.
>>
>> Current state:
>>
>> 1) We got the testsuite on MacOS to build with zero unexpected
>> successes and zero failures (modulo one change I'm going to push
>> tomorrow). This is a collaborative effort and it's very important
>> because it allows us to push for unexpected successes as failures on
>> the bots, allowing us to discover issues quicker. Other platform I
>> think are improving their state as well, mainly thanks to the work of
>> Pavel and Jan.
>
> I don't mean to belittle this statement, as I think the situation has
> definitely improved a lot lately, but I feel I have to point out
> that's I've never been able to get a clean test suite run on a mac
> (not even the "0 failures" kind of clean). I'm not sure what are these
> caused by, but I guess that's because the tests are still very much
> dependent on the environment. So, I have to ask: what kind of
> environment are you running those tests in?
>
> My machine is not a completely off-the-shelf mac, as it has some
> google-specific customizations. I don't really know what this
> encompasses, but I would be surprised if these impact the result of
> the test suite. If I had to bet, my money would be on your machines
> having some apple-specific stuff which is not available in regular
> macs.
>
> I tried an experiment today. I configured with: cmake ../../llvm
> -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -GNinja. First
> problem I ran into was that I couldn't run check-lldb, as the clang I
> have just built was unable to compile any of the test binaries due to
> missing headers (this could be a manifestation of the SDKROOT issue we
> ran into a couple of weeks ago). So, I tried running with the system
> compiler and I got this output:
>
> FAIL: test_c_global_variables_dwarf
> (lang/c/global_variables/TestGlobalVariables.py)
> FAIL: test_c_global_variables_gmodules
> (lang/c/global_variables/TestGlobalVariables.py)
> FAIL: test_dsym (functionalities/ubsan/basic/TestUbsanBasic.py)
> FAIL: test_dwarf (functionalities/ubsan/basic/TestUbsanBasic.py)
> FAIL: test_gmodules (functionalities/ubsan/basic/TestUbsanBasic.py)
> FAIL: test_with_python_api_dsym (lang/cpp/class_static/TestStaticVariables.py)
> FAIL: test_with_python_api_dwarf (lang/cpp/class_static/TestStaticVariables.py)
> FAIL: test_with_python_api_gmodules
> (lang/cpp/class_static/TestStaticVariables.py)
> ERROR: test_debug_info_for_apple_types_dsym
> (macosx/debug-info/apple_types/TestAppleTypesIsProduced.py)
> ERROR: test_debug_info_for_apple_types_dwarf
> (macosx/debug-info/apple_types/TestAppleTypesIsProduced.py)
> ERROR: test_debug_info_for_apple_types_gmodules
> (macosx/debug-info/apple_types/TestAppleTypesIsProduced.py)
> UNEXPECTED SUCCESS: test_lldbmi_output_grammar
> (tools/lldb-mi/syntax/TestMiSyntax.py)
> UNEXPECTED SUCCESS: test_process_interrupt_dsym
> (functionalities/thread/state/TestThreadStates.py)
> UNEXPECTED SUCCESS: test_process_interrupt_gmodules
> (functionalities/thread/state/TestThreadStates.py)
>
> So, I guess my question is: are you guys looking into making sure that
> others are also able to reproduce the 0-fail+0-xpass state? I would
> love to run the mac test suite locally, as I tend to touch a lot of
> stuff that impacts all targets, but as it stands now, I have very
> little confidence that the test I am running reflect in any way the
> results you will get when you run the test on your end.
>
> I am ready to supply any test logs or information you need if you want
> to try to tackle this.
>

Yes, I'm definitely interested in making the testusuite
working/reliable on any configuration.
I was afraid there were a lot of latent issues, that's why I sent this
mail in the first place.
It's also the reason why I started thinking about `lldb-test` as a
driver for testing, because I found out the testsuite being a little
inconsistent/brittle depending on the environment it's run on (which,
FWIW, doesn't happen when you run lit/FileCheck or even the unit tests
in lldb). I'm not currently claiming switching to a different method
would improve the situation, but it's worth a shot.

>> 3) In the short term I plan to remove support for unmaintained
>> languages (Java/Go/Swift). This allows us to bring them back again (or
> I hope you meant OCaml instead of Swift. :P

Oh, yes, sigh.