[lldb-dev] LLDB Linux x86_64 local target testing summary

[Todd Fiala]

(Yikes - the mailing list archive does not show the tables as "tables"
visually as I see them in GMail - if the rest of you are seeing those as
gigantic clumps of text without tables, I can resend with tables attached
as CSVs).


On Tue, Mar 4, 2014 at 1:45 PM, Todd Fiala <tfiala at google.com> wrote:

> Hi all,
>
> I just thought I'd send out an update on some work I've been doing to get
> a handle on how much of Linux x86_64 LLDB is/isn't tested.  I recently went
> through all the Linux x86_64 local-target tests, turning on previously
> disabled tests that ran, and marking XFAIL those tests that should work but
> are consistently failing to make this more useful.
>
> A high-level summary of the test run results are as follows, running
> against lldb TOT r202887:
>
> ======
> TOTALS
> ======
> pass:   388 (89.40% of tests run)
> xfail:   46 (10.60% of tests run)
> fail:     0 ( 0.00% of tests run)
>
> ===========
> UNSUPPORTED
> ===========
> unsupported (total):        480
> unsupported (Darwin-only):  421 (87.71% of unsupported tests)
> unsupported (skip linux):    13 ( 2.71% of unsupported tests)
> unsupported (other):         46 ( 9.58% of unsupported tests)
>
>
> For the expected failures, I then went ahead and took a rough guess as to
> the underlying area that the expected failure tests were exercising (some
> of these might be wildly off - they're my best guess for a first pass).  I
> used this to get a rough idea of which parts of LLDB on Linux x86_64 would
> be best bang-for-the-buck on investigating and fixing.
>
> Here is a summary of the areas where the expected failures were occurring:
>
>  System Best GuessCount concurrent17 language features9 exit handling4thread:state
> 2 thread:stack trace2 dynamic loader2 data formatter2 watch variables1trap
> 1 register:iteration1 misc:stl1 expressions:function call1expressions:floating point
> 1 breakpoints1 <unknown> 1 Grand Total 46
>
> Here is the raw XFAIL test data that generated the summary above:
> Test Module Test NameSystem (best guess)
> TestRdar12991846.Rdar12991846TestCase test_with_dwarf<unknown>
> TestStepAndBreakpoints.TestCStepping test_with_dwarf_and_python_api
> breakpoints TestExprDoesntBlock.ExprDoesntDeadlockTestCase
> test_with_dwarf_and_run_commandconcurrent
> TestConcurrentEvents.ConcurrentEventsTestCase
> test_breakpoints_delayed_breakpoint_one_watchpoint_dwarf concurrent
> TestConcurrentEvents.ConcurrentEventsTestCasetest_delay_signal_watch_dwarf
> concurrentTestConcurrentEvents.ConcurrentEventsTestCasetest_delay_watch_break_dwarf
> concurrentTestConcurrentEvents.ConcurrentEventsTestCase
> test_signal_delay_watch_dwarfconcurrentTestConcurrentEvents.ConcurrentEventsTestCase
> test_signal_watch_break_dwarfconcurrent
> TestConcurrentEvents.ConcurrentEventsTestCasetest_signal_watch_dwarfconcurrent
> TestConcurrentEvents.ConcurrentEventsTestCase
> test_two_breakpoints_one_watchpoint_dwarf concurrent
> TestConcurrentEvents.ConcurrentEventsTestCasetest_two_watchpoint_threads_dwarf
> concurrentTestConcurrentEvents.ConcurrentEventsTestCase
> test_two_watchpoints_one_breakpoint_dwarfconcurrentTestConcurrentEvents.ConcurrentEventsTestCase
> test_two_watchpoints_one_delay_breakpoint_dwarfconcurrent
> TestConcurrentEvents.ConcurrentEventsTestCase
> test_two_watchpoints_one_signal_dwarf concurrent
> TestConcurrentEvents.ConcurrentEventsTestCasetest_watch_break_dwarf
> concurrentTestConcurrentEvents.ConcurrentEventsTestCasetest_watch_break_dwarf_delay
> concurrentTestConcurrentEvents.ConcurrentEventsTestCase
> test_watchpoint_delay_watchpoint_one_breakpoint_dwarfconcurrentTestConcurrentEvents.ConcurrentEventsTestCase
> test_watchpoint_with_delay_waychpoint_threads_dwarfconcurrent
> TestMultithreaded.SBBreakpointCallbackCasetest_breakpoint_callbackconcurrent
> TestDataFormatterStdVector.StdVectorDataFormatterTestCase
> test_with_dwarf_and_run_command data formatter
> TestDataFormatterStdVBool.StdVBoolDataFormatterTestCasetest_with_dwarf_and_run_commanddata
> formatterTestAnonymous.AnonymousTestCase test_expr_nulllanguage features
> TestBitfields.BitfieldsTestCase test_with_dwarf_and_python_apilanguage
> featuresTestBlocks.BlocksTestCase test_expr_with_dwarflanguage features
> TestNamespace.NamespaceTestCase test_with_dwarf_and_run_commandlanguage
> featuresTestRvalueReferences.RvalueReferencesTestCase
> test_with_dwarf_and_run_commandlanguage featuresTestCPPStaticMembers.CPPStaticMembersTestCase
> test_with_dwarf_and_run_commandlanguage features
> TestStaticVariables.StaticVariableTestCasetest_with_dwarf_and_run_commandlanguage features
> TestCPPThis.CPPThisTestCasetest_with_dwarf_and_run_command language
> featuresTestCppValueCast.CppValueCastTestCase
> test_value_cast_with_dwarf_and_virtual_inheritance language features
> TestSharedLib.SharedLibTestCase test_frame_variable_with_dwarfdynamic
> loaderTestSharedLibStrippedSymbols.SharedLibTestCase
> test_frame_variable_with_dwarfdynamic loaderTestInferiorAssert.AssertingInferiorTestCase
> test_inferior_asserting_disassembleexit handling
> TestRecursiveInferior.CrashingRecursiveInferiorTestCase
> test_recursive_inferior_crashing_expr_step_and_expr_dwarf exit handling
> TestInferiorCrashing.CrashingInferiorTestCasetest_inferior_crashing_expr_step_and_expr_dwarfexit
> handlingTestInferiorCrashing.CrashingInferiorTestCase
> test_inferior_crashing_step_after_break_dwarfexit handlingTestExprs.BasicExprCommandsTestCase
> test_floating_point_expr_commandsexpressions:floating point
> TestCallStdStringFunction.ExprCommandCallFunctionTestCasetest_with_dwarfexpressions:function call
> TestSTL.STLTestCasetest_with_dwarf misc:stl
> TestRegistersIterator.RegistersIteratorTestCasetest_iter_registersregister:iteration
> TestPrintStackTraces.ThreadsStackTracesTestCasetest_stack_traces thread:stack
> traceTestThreadAPI.ThreadAPITestCasetest_step_out_of_malloc_into_function_b_with_dwarfthread:stack
> traceTestThreadStates.ThreadStateTestCase
> test_process_interrupt_with_dwarfthread:stateTestThreadStates.ThreadStateTestCase
> test_process_state_with_dwarfthread:state
> TestBuiltinTrap.BuiltinTrapTestCasetest_with_dwarf_and_run_command trap
> TestWatchedVarHitWhenInScope.WatchedVariableHitWhenInScopeTestCase
> test_watched_var_should_only_hit_when_in_scope_with_dwarf watch variables
>
>
> Note my test runs were done under the following conditions:
>
>    - Host: Ubuntu 12.04 LTS x86_64
>    - Packages: swig, python-dev, ncurses-dev
>    - Compiler: hand-built gcc 4.8.2 (required libmpc-dev).
>    - libstdc++ from gcc 4.8.2
>    - Libedit (July 2013) from here <http://thrysoee.dk/editline/>.  Note
>    since we grabbed this the link has moved on to a 2014 cut of libedit.
>    - Built with cmake/ninja, debug enabled, assertions enabled.
>    - Run command: 'ninja check-lldb' within the build directory.
>
> Observations:
>
>    - Concurrent tests (TestConcurrentEvents) seem to expose the "most
>    broken" area.
>    - Language features (C/C++) are the next largest group of failures
>    (could this be related to building with gcc but using clang's AST engine?
>     Need to investigate deeper since this is the next most broken area).
>    - There are a set of tests, libc++-based data formatters, that I don't
>    run because (1) I don't have a working libc++-based solution on Ubuntu yet.
>     Those tests are not included above and represents the largest selection of
>    tests that are currently disabled on local target Linux x86_64 testing.
>    - The areas that are no functioning seem to be fairly narrow - this is
>    reasonably good news.
>    - The test results do not include running a Linux x86_64 host against
>    a Linux x86_64 remote.  I'll generate a summary for that once I (finally)
>    get to lldb-gdbserver.
>
> Hopefully this is useful info for the following groups of people:
>
>    - Those wondering what kind of state LLDB is in on Linux (e.g. how
>    many tests run successfully on Linux x86_64 vs. OS X).
>    - Those wondering what areas of Linux could use the most help to make
>    the product better.
>
> Feel free to ping me if you'd like more details on any of this.  I just
> figured I'd share these results with the list since I've had a few people
> ask me about areas this data would help answer.
> --
> Todd Fiala | Software Engineer |  tfiala at google.com |  650-943-3180
>



-- 
Todd Fiala | Software Engineer | tfiala at google.com | 650-943-3180
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20140304/7b9ae286/attachment.html>