[lldb-dev] LLDB Linux x86_64 local target testing summary

[Todd Fiala]

Hi all,

I just thought I'd send out an update on some work I've been doing to get a
handle on how much of Linux x86_64 LLDB is/isn't tested.  I recently went
through all the Linux x86_64 local-target tests, turning on previously
disabled tests that ran, and marking XFAIL those tests that should work but
are consistently failing to make this more useful.

A high-level summary of the test run results are as follows, running
against lldb TOT r202887:

======
TOTALS
======
pass:   388 (89.40% of tests run)
xfail:   46 (10.60% of tests run)
fail:     0 ( 0.00% of tests run)

===========
UNSUPPORTED
===========
unsupported (total):        480
unsupported (Darwin-only):  421 (87.71% of unsupported tests)
unsupported (skip linux):    13 ( 2.71% of unsupported tests)
unsupported (other):         46 ( 9.58% of unsupported tests)


For the expected failures, I then went ahead and took a rough guess as to
the underlying area that the expected failure tests were exercising (some
of these might be wildly off - they're my best guess for a first pass).  I
used this to get a rough idea of which parts of LLDB on Linux x86_64 would
be best bang-for-the-buck on investigating and fixing.

Here is a summary of the areas where the expected failures were occurring:

System Best GuessCountconcurrent17language features9exit handling4
thread:state2thread:stack trace2dynamic loader2data formatter2watch
variables1trap1register:iteration1misc:stl1expressions:function
call1expressions:floating
point1breakpoints1<unknown>1Grand Total46

Here is the raw XFAIL test data that generated the summary above:
Test ModuleTest NameSystem (best guess)TestRdar12991846.Rdar12991846TestCase
test_with_dwarf<unknown>TestStepAndBreakpoints.TestCStepping
test_with_dwarf_and_python_apibreakpoints
TestExprDoesntBlock.ExprDoesntDeadlockTestCase
test_with_dwarf_and_run_commandconcurrent
TestConcurrentEvents.ConcurrentEventsTestCase
test_breakpoints_delayed_breakpoint_one_watchpoint_dwarfconcurrent
TestConcurrentEvents.ConcurrentEventsTestCasetest_delay_signal_watch_dwarf
concurrentTestConcurrentEvents.ConcurrentEventsTestCase
test_delay_watch_break_dwarfconcurrent
TestConcurrentEvents.ConcurrentEventsTestCasetest_signal_delay_watch_dwarf
concurrentTestConcurrentEvents.ConcurrentEventsTestCase
test_signal_watch_break_dwarfconcurrent
TestConcurrentEvents.ConcurrentEventsTestCasetest_signal_watch_dwarf
concurrentTestConcurrentEvents.ConcurrentEventsTestCase
test_two_breakpoints_one_watchpoint_dwarfconcurrent
TestConcurrentEvents.ConcurrentEventsTestCase
test_two_watchpoint_threads_dwarfconcurrent
TestConcurrentEvents.ConcurrentEventsTestCase
test_two_watchpoints_one_breakpoint_dwarfconcurrent
TestConcurrentEvents.ConcurrentEventsTestCase
test_two_watchpoints_one_delay_breakpoint_dwarfconcurrent
TestConcurrentEvents.ConcurrentEventsTestCase
test_two_watchpoints_one_signal_dwarfconcurrent
TestConcurrentEvents.ConcurrentEventsTestCasetest_watch_break_dwarf
concurrentTestConcurrentEvents.ConcurrentEventsTestCase
test_watch_break_dwarf_delayconcurrent
TestConcurrentEvents.ConcurrentEventsTestCase
test_watchpoint_delay_watchpoint_one_breakpoint_dwarfconcurrent
TestConcurrentEvents.ConcurrentEventsTestCase
test_watchpoint_with_delay_waychpoint_threads_dwarfconcurrent
TestMultithreaded.SBBreakpointCallbackCasetest_breakpoint_callbackconcurrent
TestDataFormatterStdVector.StdVectorDataFormatterTestCase
test_with_dwarf_and_run_commanddata formatter
TestDataFormatterStdVBool.StdVBoolDataFormatterTestCase
test_with_dwarf_and_run_commanddata formatterTestAnonymous.AnonymousTestCase
test_expr_nulllanguage featuresTestBitfields.BitfieldsTestCase
test_with_dwarf_and_python_apilanguage featuresTestBlocks.BlocksTestCase
test_expr_with_dwarflanguage featuresTestNamespace.NamespaceTestCase
test_with_dwarf_and_run_commandlanguage features
TestRvalueReferences.RvalueReferencesTestCasetest_with_dwarf_and_run_commandlanguage
featuresTestCPPStaticMembers.CPPStaticMembersTestCase
test_with_dwarf_and_run_commandlanguage features
TestStaticVariables.StaticVariableTestCasetest_with_dwarf_and_run_commandlanguage
featuresTestCPPThis.CPPThisTestCasetest_with_dwarf_and_run_commandlanguage
featuresTestCppValueCast.CppValueCastTestCase
test_value_cast_with_dwarf_and_virtual_inheritancelanguage features
TestSharedLib.SharedLibTestCasetest_frame_variable_with_dwarfdynamic loader
TestSharedLibStrippedSymbols.SharedLibTestCasetest_frame_variable_with_dwarfdynamic
loaderTestInferiorAssert.AssertingInferiorTestCase
test_inferior_asserting_disassembleexit handling
TestRecursiveInferior.CrashingRecursiveInferiorTestCase
test_recursive_inferior_crashing_expr_step_and_expr_dwarfexit handling
TestInferiorCrashing.CrashingInferiorTestCase
test_inferior_crashing_expr_step_and_expr_dwarfexit handling
TestInferiorCrashing.CrashingInferiorTestCase
test_inferior_crashing_step_after_break_dwarfexit handling
TestExprs.BasicExprCommandsTestCasetest_floating_point_expr_commandsexpressions:floating
pointTestCallStdStringFunction.ExprCommandCallFunctionTestCase
test_with_dwarfexpressions:function callTestSTL.STLTestCasetest_with_dwarf
misc:stlTestRegistersIterator.RegistersIteratorTestCasetest_iter_registers
register:iterationTestPrintStackTraces.ThreadsStackTracesTestCase
test_stack_tracesthread:stack traceTestThreadAPI.ThreadAPITestCase
test_step_out_of_malloc_into_function_b_with_dwarfthread:stack trace
TestThreadStates.ThreadStateTestCasetest_process_interrupt_with_dwarf
thread:stateTestThreadStates.ThreadStateTestCase
test_process_state_with_dwarfthread:stateTestBuiltinTrap.BuiltinTrapTestCase
test_with_dwarf_and_run_commandtrap
TestWatchedVarHitWhenInScope.WatchedVariableHitWhenInScopeTestCase
test_watched_var_should_only_hit_when_in_scope_with_dwarfwatch variables


Note my test runs were done under the following conditions:

   - Host: Ubuntu 12.04 LTS x86_64
   - Packages: swig, python-dev, ncurses-dev
   - Compiler: hand-built gcc 4.8.2 (required libmpc-dev).
   - libstdc++ from gcc 4.8.2
   - Libedit (July 2013) from here <http://thrysoee.dk/editline/>.  Note
   since we grabbed this the link has moved on to a 2014 cut of libedit.
   - Built with cmake/ninja, debug enabled, assertions enabled.
   - Run command: 'ninja check-lldb' within the build directory.

Observations:

   - Concurrent tests (TestConcurrentEvents) seem to expose the "most
   broken" area.
   - Language features (C/C++) are the next largest group of failures
   (could this be related to building with gcc but using clang's AST engine?
    Need to investigate deeper since this is the next most broken area).
   - There are a set of tests, libc++-based data formatters, that I don't
   run because (1) I don't have a working libc++-based solution on Ubuntu yet.
    Those tests are not included above and represents the largest selection of
   tests that are currently disabled on local target Linux x86_64 testing.
   - The areas that are no functioning seem to be fairly narrow - this is
   reasonably good news.
   - The test results do not include running a Linux x86_64 host against a
   Linux x86_64 remote.  I'll generate a summary for that once I (finally) get
   to lldb-gdbserver.

Hopefully this is useful info for the following groups of people:

   - Those wondering what kind of state LLDB is in on Linux (e.g. how many
   tests run successfully on Linux x86_64 vs. OS X).
   - Those wondering what areas of Linux could use the most help to make
   the product better.

Feel free to ping me if you'd like more details on any of this.  I just
figured I'd share these results with the list since I've had a few people
ask me about areas this data would help answer.
-- 
Todd Fiala | Software Engineer | tfiala at google.com | 650-943-3180
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20140304/d10812e3/attachment.html>