[lldb-dev] LLDB Linux x86_64 local target testing summary

Tue Mar 4 13:56:46 PST 2014

Attaching CSVs for tabular content.

Note the filenames indicate r202726, but the data remained the same up
through the reported r202887.

On Tue, Mar 4, 2014 at 1:51 PM, Todd Fiala <tfiala at google.com> wrote:

> (Yikes - the mailing list archive does not show the tables as "tables"
> visually as I see them in GMail - if the rest of you are seeing those as
> gigantic clumps of text without tables, I can resend with tables attached
> as CSVs).
>
>
> On Tue, Mar 4, 2014 at 1:45 PM, Todd Fiala <tfiala at google.com> wrote:
>
>> Hi all,
>>
>> I just thought I'd send out an update on some work I've been doing to get
>> a handle on how much of Linux x86_64 LLDB is/isn't tested.  I recently went
>> through all the Linux x86_64 local-target tests, turning on previously
>> disabled tests that ran, and marking XFAIL those tests that should work but
>> are consistently failing to make this more useful.
>>
>> A high-level summary of the test run results are as follows, running
>> against lldb TOT r202887:
>>
>> ======
>> TOTALS
>> ======
>> pass:   388 (89.40% of tests run)
>> xfail:   46 (10.60% of tests run)
>> fail:     0 ( 0.00% of tests run)
>>
>> ===========
>> UNSUPPORTED
>> ===========
>> unsupported (total):        480
>> unsupported (Darwin-only):  421 (87.71% of unsupported tests)
>> unsupported (skip linux):    13 ( 2.71% of unsupported tests)
>> unsupported (other):         46 ( 9.58% of unsupported tests)
>>
>>
>> For the expected failures, I then went ahead and took a rough guess as to
>> the underlying area that the expected failure tests were exercising (some
>> of these might be wildly off - they're my best guess for a first pass).  I
>> used this to get a rough idea of which parts of LLDB on Linux x86_64 would
>> be best bang-for-the-buck on investigating and fixing.
>>
>> Here is a summary of the areas where the expected failures were occurring:
>>
>>  System Best GuessCount concurrent17 language features9 exit handling4thread:state
>> 2 thread:stack trace2 dynamic loader2 data formatter2 watch variables1trap
>> 1 register:iteration1 misc:stl1 expressions:function call1expressions:floating point
>> 1 breakpoints1 <unknown> 1 Grand Total 46
>>
>> Here is the raw XFAIL test data that generated the summary above:
>> Test Module Test NameSystem (best guess)
>> TestRdar12991846.Rdar12991846TestCase test_with_dwarf<unknown>
>> TestStepAndBreakpoints.TestCStepping test_with_dwarf_and_python_api
>> breakpoints TestExprDoesntBlock.ExprDoesntDeadlockTestCase
>> test_with_dwarf_and_run_commandconcurrent
>> TestConcurrentEvents.ConcurrentEventsTestCase
>> test_breakpoints_delayed_breakpoint_one_watchpoint_dwarf concurrent
>> TestConcurrentEvents.ConcurrentEventsTestCase
>> test_delay_signal_watch_dwarf concurrent
>> TestConcurrentEvents.ConcurrentEventsTestCasetest_delay_watch_break_dwarf
>> concurrentTestConcurrentEvents.ConcurrentEventsTestCase
>> test_signal_delay_watch_dwarfconcurrentTestConcurrentEvents.ConcurrentEventsTestCase
>> test_signal_watch_break_dwarfconcurrent
>> TestConcurrentEvents.ConcurrentEventsTestCasetest_signal_watch_dwarfconcurrent
>> TestConcurrentEvents.ConcurrentEventsTestCase
>> test_two_breakpoints_one_watchpoint_dwarf concurrent
>> TestConcurrentEvents.ConcurrentEventsTestCasetest_two_watchpoint_threads_dwarf
>> concurrentTestConcurrentEvents.ConcurrentEventsTestCase
>> test_two_watchpoints_one_breakpoint_dwarfconcurrentTestConcurrentEvents.ConcurrentEventsTestCase
>> test_two_watchpoints_one_delay_breakpoint_dwarfconcurrent
>> TestConcurrentEvents.ConcurrentEventsTestCase
>> test_two_watchpoints_one_signal_dwarf concurrent
>> TestConcurrentEvents.ConcurrentEventsTestCasetest_watch_break_dwarf
>> concurrentTestConcurrentEvents.ConcurrentEventsTestCasetest_watch_break_dwarf_delay
>> concurrentTestConcurrentEvents.ConcurrentEventsTestCase
>> test_watchpoint_delay_watchpoint_one_breakpoint_dwarfconcurrentTestConcurrentEvents.ConcurrentEventsTestCase
>> test_watchpoint_with_delay_waychpoint_threads_dwarfconcurrent
>> TestMultithreaded.SBBreakpointCallbackCasetest_breakpoint_callbackconcurrent
>> TestDataFormatterStdVector.StdVectorDataFormatterTestCase
>> test_with_dwarf_and_run_command data formatter
>> TestDataFormatterStdVBool.StdVBoolDataFormatterTestCasetest_with_dwarf_and_run_commanddata
>> formatterTestAnonymous.AnonymousTestCase test_expr_nulllanguage features
>> TestBitfields.BitfieldsTestCase test_with_dwarf_and_python_apilanguage
>> featuresTestBlocks.BlocksTestCase test_expr_with_dwarflanguage features
>> TestNamespace.NamespaceTestCase test_with_dwarf_and_run_commandlanguage
>> featuresTestRvalueReferences.RvalueReferencesTestCase
>> test_with_dwarf_and_run_commandlanguage featuresTestCPPStaticMembers.CPPStaticMembersTestCase
>> test_with_dwarf_and_run_commandlanguage features
>> TestStaticVariables.StaticVariableTestCasetest_with_dwarf_and_run_commandlanguage features
>> TestCPPThis.CPPThisTestCasetest_with_dwarf_and_run_command language
>> featuresTestCppValueCast.CppValueCastTestCase
>> test_value_cast_with_dwarf_and_virtual_inheritance language features
>> TestSharedLib.SharedLibTestCase test_frame_variable_with_dwarfdynamic
>> loaderTestSharedLibStrippedSymbols.SharedLibTestCase
>> test_frame_variable_with_dwarfdynamic loaderTestInferiorAssert.AssertingInferiorTestCase
>> test_inferior_asserting_disassembleexit handling
>> TestRecursiveInferior.CrashingRecursiveInferiorTestCase
>> test_recursive_inferior_crashing_expr_step_and_expr_dwarf exit handling
>> TestInferiorCrashing.CrashingInferiorTestCasetest_inferior_crashing_expr_step_and_expr_dwarfexit
>> handlingTestInferiorCrashing.CrashingInferiorTestCase
>> test_inferior_crashing_step_after_break_dwarfexit handlingTestExprs.BasicExprCommandsTestCase
>> test_floating_point_expr_commandsexpressions:floating point
>> TestCallStdStringFunction.ExprCommandCallFunctionTestCasetest_with_dwarfexpressions:function call
>> TestSTL.STLTestCasetest_with_dwarf misc:stl
>> TestRegistersIterator.RegistersIteratorTestCasetest_iter_registersregister:iteration
>> TestPrintStackTraces.ThreadsStackTracesTestCasetest_stack_traces thread:stack
>> traceTestThreadAPI.ThreadAPITestCasetest_step_out_of_malloc_into_function_b_with_dwarfthread:stack
>> traceTestThreadStates.ThreadStateTestCase
>> test_process_interrupt_with_dwarfthread:stateTestThreadStates.ThreadStateTestCase
>> test_process_state_with_dwarfthread:state
>> TestBuiltinTrap.BuiltinTrapTestCasetest_with_dwarf_and_run_command trap
>> TestWatchedVarHitWhenInScope.WatchedVariableHitWhenInScopeTestCase
>> test_watched_var_should_only_hit_when_in_scope_with_dwarf watch variables
>>
>>
>> Note my test runs were done under the following conditions:
>>
>>    - Host: Ubuntu 12.04 LTS x86_64
>>    - Packages: swig, python-dev, ncurses-dev
>>    - Compiler: hand-built gcc 4.8.2 (required libmpc-dev).
>>    - libstdc++ from gcc 4.8.2
>>    - Libedit (July 2013) from here <http://thrysoee.dk/editline/>.  Note
>>    since we grabbed this the link has moved on to a 2014 cut of libedit.
>>    - Built with cmake/ninja, debug enabled, assertions enabled.
>>    - Run command: 'ninja check-lldb' within the build directory.
>>
>> Observations:
>>
>>    - Concurrent tests (TestConcurrentEvents) seem to expose the "most
>>    broken" area.
>>    - Language features (C/C++) are the next largest group of failures
>>    (could this be related to building with gcc but using clang's AST engine?
>>     Need to investigate deeper since this is the next most broken area).
>>    - There are a set of tests, libc++-based data formatters, that I
>>    don't run because (1) I don't have a working libc++-based solution on
>>    Ubuntu yet.  Those tests are not included above and represents the largest
>>    selection of tests that are currently disabled on local target Linux x86_64
>>    testing.
>>    - The areas that are no functioning seem to be fairly narrow - this
>>    is reasonably good news.
>>    - The test results do not include running a Linux x86_64 host against
>>    a Linux x86_64 remote.  I'll generate a summary for that once I (finally)
>>    get to lldb-gdbserver.
>>
>> Hopefully this is useful info for the following groups of people:
>>
>>    - Those wondering what kind of state LLDB is in on Linux (e.g. how
>>    many tests run successfully on Linux x86_64 vs. OS X).
>>    - Those wondering what areas of Linux could use the most help to make
>>    the product better.
>>
>> Feel free to ping me if you'd like more details on any of this.  I just
>> figured I'd share these results with the list since I've had a few people
>> ask me about areas this data would help answer.
>>  --
>>  Todd Fiala | Software Engineer |  tfiala at google.com |  650-943-3180
>>
>
>
>
> --
> Todd Fiala | Software Engineer |  tfiala at google.com |  650-943-3180
>

-- 
Todd Fiala | Software Engineer | tfiala at google.com | 650-943-3180
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20140304/a3b4e10b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: XFAIL Tests - r202726_system_summary.csv
Type: text/csv
Size: 298 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20140304/a3b4e10b/attachment.csv>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: XFAIL Tests - r202726_xfail_2014-03-03.csv
Type: text/csv
Size: 4244 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20140304/a3b4e10b/attachment-0001.csv>