[lldb-dev] [RFC] Testsuite in lldb & possible future directions

Pavel Labath via lldb-dev lldb-dev at lists.llvm.org
Tue Feb 6 07:09:06 PST 2018

On 6 February 2018 at 04:11, Davide Italiano via lldb-dev
<lldb-dev at lists.llvm.org> wrote:
> Hi,
> in the last couple of months a lot of people put a lot of attentions
> and energy into lldb and we're starting to see the first results. I
> decided to sit down and write this e-mail to state where we are and
> what are some possible future directions for the projects, in terms of
> better quality/higher testability.
> Current state:
> 1) We got the testsuite on MacOS to build with zero unexpected
> successes and zero failures (modulo one change I'm going to push
> tomorrow). This is a collaborative effort and it's very important
> because it allows us to push for unexpected successes as failures on
> the bots, allowing us to discover issues quicker. Other platform I
> think are improving their state as well, mainly thanks to the work of
> Pavel and Jan.

I don't mean to belittle this statement, as I think the situation has
definitely improved a lot lately, but I feel I have to point out
that's I've never been able to get a clean test suite run on a mac
(not even the "0 failures" kind of clean). I'm not sure what are these
caused by, but I guess that's because the tests are still very much
dependent on the environment. So, I have to ask: what kind of
environment are you running those tests in?

My machine is not a completely off-the-shelf mac, as it has some
google-specific customizations. I don't really know what this
encompasses, but I would be surprised if these impact the result of
the test suite. If I had to bet, my money would be on your machines
having some apple-specific stuff which is not available in regular

I tried an experiment today. I configured with: cmake ../../llvm
problem I ran into was that I couldn't run check-lldb, as the clang I
have just built was unable to compile any of the test binaries due to
missing headers (this could be a manifestation of the SDKROOT issue we
ran into a couple of weeks ago). So, I tried running with the system
compiler and I got this output:

FAIL: test_c_global_variables_dwarf
FAIL: test_c_global_variables_gmodules
FAIL: test_dsym (functionalities/ubsan/basic/TestUbsanBasic.py)
FAIL: test_dwarf (functionalities/ubsan/basic/TestUbsanBasic.py)
FAIL: test_gmodules (functionalities/ubsan/basic/TestUbsanBasic.py)
FAIL: test_with_python_api_dsym (lang/cpp/class_static/TestStaticVariables.py)
FAIL: test_with_python_api_dwarf (lang/cpp/class_static/TestStaticVariables.py)
FAIL: test_with_python_api_gmodules
ERROR: test_debug_info_for_apple_types_dsym
ERROR: test_debug_info_for_apple_types_dwarf
ERROR: test_debug_info_for_apple_types_gmodules
UNEXPECTED SUCCESS: test_lldbmi_output_grammar
UNEXPECTED SUCCESS: test_process_interrupt_dsym
UNEXPECTED SUCCESS: test_process_interrupt_gmodules

So, I guess my question is: are you guys looking into making sure that
others are also able to reproduce the 0-fail+0-xpass state? I would
love to run the mac test suite locally, as I tend to touch a lot of
stuff that impacts all targets, but as it stands now, I have very
little confidence that the test I am running reflect in any way the
results you will get when you run the test on your end.

I am ready to supply any test logs or information you need if you want
to try to tackle this.

> 3) In the short term I plan to remove support for unmaintained
> languages (Java/Go/Swift). This allows us to bring them back again (or
I hope you meant OCaml instead of Swift. :P

More information about the lldb-dev mailing list