[lldb-dev] LLDB Linux x86_64 local target testing summary
tfiala at google.com
Tue Mar 4 13:51:00 PST 2014
(Yikes - the mailing list archive does not show the tables as "tables"
visually as I see them in GMail - if the rest of you are seeing those as
gigantic clumps of text without tables, I can resend with tables attached
On Tue, Mar 4, 2014 at 1:45 PM, Todd Fiala <tfiala at google.com> wrote:
> Hi all,
> I just thought I'd send out an update on some work I've been doing to get
> a handle on how much of Linux x86_64 LLDB is/isn't tested. I recently went
> through all the Linux x86_64 local-target tests, turning on previously
> disabled tests that ran, and marking XFAIL those tests that should work but
> are consistently failing to make this more useful.
> A high-level summary of the test run results are as follows, running
> against lldb TOT r202887:
> pass: 388 (89.40% of tests run)
> xfail: 46 (10.60% of tests run)
> fail: 0 ( 0.00% of tests run)
> unsupported (total): 480
> unsupported (Darwin-only): 421 (87.71% of unsupported tests)
> unsupported (skip linux): 13 ( 2.71% of unsupported tests)
> unsupported (other): 46 ( 9.58% of unsupported tests)
> For the expected failures, I then went ahead and took a rough guess as to
> the underlying area that the expected failure tests were exercising (some
> of these might be wildly off - they're my best guess for a first pass). I
> used this to get a rough idea of which parts of LLDB on Linux x86_64 would
> be best bang-for-the-buck on investigating and fixing.
> Here is a summary of the areas where the expected failures were occurring:
> System Best GuessCount concurrent17 language features9 exit handling4thread:state
> 2 thread:stack trace2 dynamic loader2 data formatter2 watch variables1trap
> 1 register:iteration1 misc:stl1 expressions:function call1expressions:floating point
> 1 breakpoints1 <unknown> 1 Grand Total 46
> Here is the raw XFAIL test data that generated the summary above:
> Test Module Test NameSystem (best guess)
> TestRdar12991846.Rdar12991846TestCase test_with_dwarf<unknown>
> TestStepAndBreakpoints.TestCStepping test_with_dwarf_and_python_api
> breakpoints TestExprDoesntBlock.ExprDoesntDeadlockTestCase
> test_breakpoints_delayed_breakpoint_one_watchpoint_dwarf concurrent
> test_two_breakpoints_one_watchpoint_dwarf concurrent
> test_two_watchpoints_one_signal_dwarf concurrent
> test_with_dwarf_and_run_command data formatter
> formatterTestAnonymous.AnonymousTestCase test_expr_nulllanguage features
> TestBitfields.BitfieldsTestCase test_with_dwarf_and_python_apilanguage
> featuresTestBlocks.BlocksTestCase test_expr_with_dwarflanguage features
> TestNamespace.NamespaceTestCase test_with_dwarf_and_run_commandlanguage
> test_with_dwarf_and_run_commandlanguage featuresTestCPPStaticMembers.CPPStaticMembersTestCase
> test_with_dwarf_and_run_commandlanguage features
> TestStaticVariables.StaticVariableTestCasetest_with_dwarf_and_run_commandlanguage features
> TestCPPThis.CPPThisTestCasetest_with_dwarf_and_run_command language
> test_value_cast_with_dwarf_and_virtual_inheritance language features
> TestSharedLib.SharedLibTestCase test_frame_variable_with_dwarfdynamic
> test_frame_variable_with_dwarfdynamic loaderTestInferiorAssert.AssertingInferiorTestCase
> test_inferior_asserting_disassembleexit handling
> test_recursive_inferior_crashing_expr_step_and_expr_dwarf exit handling
> test_inferior_crashing_step_after_break_dwarfexit handlingTestExprs.BasicExprCommandsTestCase
> test_floating_point_expr_commandsexpressions:floating point
> TestCallStdStringFunction.ExprCommandCallFunctionTestCasetest_with_dwarfexpressions:function call
> TestSTL.STLTestCasetest_with_dwarf misc:stl
> TestPrintStackTraces.ThreadsStackTracesTestCasetest_stack_traces thread:stack
> TestBuiltinTrap.BuiltinTrapTestCasetest_with_dwarf_and_run_command trap
> test_watched_var_should_only_hit_when_in_scope_with_dwarf watch variables
> Note my test runs were done under the following conditions:
> - Host: Ubuntu 12.04 LTS x86_64
> - Packages: swig, python-dev, ncurses-dev
> - Compiler: hand-built gcc 4.8.2 (required libmpc-dev).
> - libstdc++ from gcc 4.8.2
> - Libedit (July 2013) from here <http://thrysoee.dk/editline/>. Note
> since we grabbed this the link has moved on to a 2014 cut of libedit.
> - Built with cmake/ninja, debug enabled, assertions enabled.
> - Run command: 'ninja check-lldb' within the build directory.
> - Concurrent tests (TestConcurrentEvents) seem to expose the "most
> broken" area.
> - Language features (C/C++) are the next largest group of failures
> (could this be related to building with gcc but using clang's AST engine?
> Need to investigate deeper since this is the next most broken area).
> - There are a set of tests, libc++-based data formatters, that I don't
> run because (1) I don't have a working libc++-based solution on Ubuntu yet.
> Those tests are not included above and represents the largest selection of
> tests that are currently disabled on local target Linux x86_64 testing.
> - The areas that are no functioning seem to be fairly narrow - this is
> reasonably good news.
> - The test results do not include running a Linux x86_64 host against
> a Linux x86_64 remote. I'll generate a summary for that once I (finally)
> get to lldb-gdbserver.
> Hopefully this is useful info for the following groups of people:
> - Those wondering what kind of state LLDB is in on Linux (e.g. how
> many tests run successfully on Linux x86_64 vs. OS X).
> - Those wondering what areas of Linux could use the most help to make
> the product better.
> Feel free to ping me if you'd like more details on any of this. I just
> figured I'd share these results with the list since I've had a few people
> ask me about areas this data would help answer.
> Todd Fiala | Software Engineer | tfiala at google.com | 650-943-3180
Todd Fiala | Software Engineer | tfiala at google.com | 650-943-3180
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the lldb-dev