[lldb-dev] Some API test failures are really opaque/could be improved

Raphael Isemann via lldb-dev lldb-dev at lists.llvm.org
Mon Oct 18 09:45:08 PDT 2021 

I think https://reviews.llvm.org/D111978 ,
https://reviews.llvm.org/D111981 and the other patches Pavel & me put
up today should improve this situation IIUC.

- Raphael

Am Mo., 18. Okt. 2021 um 05:54 Uhr schrieb David Blaikie via lldb-dev
<lldb-dev at lists.llvm.org>:
>
> Wondering if anyone else has encountered/dealt with debugging lldb test failures like the one shown at the end of this email ("AssertionError: 10 != 5" in "test.assertEqual(process.GetState(), lldb.eStateStopped)" while checking that a breakpoint was reached)
>
> Is there anything that could be done to improve the debuggability of such failures? Logging standard output/error from the lldb process or any other logging it might have? At least for one of these I managed to figure it out by running lldb directly on the binary and finding that the binary couldn't find libc++.so (that's a story for another bug/email thread, looks like maybe lldb libc++ pretty printer tests are using the system installed libc++, not the just-built libc++ (& thus not running if there is no system installed libc++)). But my current failure like this seems a bit more inscrutible and I'm still looking into it.
>
> Looks like dotest.py has some sense of logging (it has a --log-success option which says preserves the logs even on failure - though the output of dotest.py, at least for me, has no mention of logs, log files, or where they might be located, and looking at the source points to some sort of ".log" files... ah, found some)
>
> So, yeah, there do seem to be some Failure.log, SkippedTest.log, etc - should dotest print something about the paths to those files when it exits with failure, maybe?
>
> ********************
>
> FAIL: lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py (23 of 23)
>
> ******************** TEST 'lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py' FAILED ********************
>
> Script:
>
> --
>
> /usr/bin/python3 /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/dotest.py -u CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/bin/ar --env OBJCOPY=/usr/bin/objcopy --env LLVM_LIBS_DIR=/usr/local/google/home/blaikie/dev/llvm/build/release/./lib --arch x86_64 --build-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex --lldb-module-cache-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-lldb/lldb-api --clang-module-cache-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-clang/lldb-api --executable /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/lldb --compiler /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/clang --dsymutil /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/dsymutil --llvm-tools-dir /usr/local/google/home/blaikie/dev/llvm/build/release/./bin --lldb-libs-dir /usr/local/google/home/blaikie/dev/llvm/build/release/./lib /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set -p TestDataFormatterLibcxxSet.py
>
> --
>
> Exit Code: 1
>
>
> Command Output (stdout):
>
> --
>
> lldb version 14.0.0 (git at github.com:llvm/llvm-project.git revision 6176fda3f992b5086302b3826aa0636135cc4cc0)
>
>   clang revision 6176fda3f992b5086302b3826aa0636135cc4cc0
>
>   llvm revision 6176fda3f992b5086302b3826aa0636135cc4cc0
>
> Skipping the following test categories: ['dsym', 'gmodules', 'debugserver', 'objc']
>
>
> --
>
> Command Output (stderr):
>
> --
>
> UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dsym (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run)
>
> FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
>
> FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
>
> UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_gmodules (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run)
>
> UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dsym (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run)
>
> FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
>
> FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
>
> UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_gmodules (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run)
>
> ======================================================================
>
> FAIL: test_ref_and_ptr_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
>
>    Test that the data formatters work on ref and ptr.
>
> ----------------------------------------------------------------------
>
> Traceback (most recent call last):
>
>   File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
>
>     return attrvalue(self)
>
>   File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 128, in test_ref_and_ptr
>
>     (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
>
>   File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
>
>     return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
>
>   File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
>
>     test.assertEqual(process.GetState(), lldb.eStateStopped)
>
> AssertionError: 10 != 5
>
> Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
>
> ======================================================================
>
> FAIL: test_ref_and_ptr_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
>
>    Test that the data formatters work on ref and ptr.
>
> ----------------------------------------------------------------------
>
> Traceback (most recent call last):
>
>   File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
>
>     return attrvalue(self)
>
>   File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 128, in test_ref_and_ptr
>
>     (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
>
>   File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
>
>     return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
>
>   File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
>
>     test.assertEqual(process.GetState(), lldb.eStateStopped)
>
> AssertionError: 10 != 5
>
> Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
>
> ======================================================================
>
> FAIL: test_with_run_command_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
>
>    Test that that file and class static variables display correctly.
>
> ----------------------------------------------------------------------
>
> Traceback (most recent call last):
>
>   File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
>
>     return attrvalue(self)
>
>   File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
>
>     (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
>
>   File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
>
>     return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
>
>   File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
>
>     test.assertEqual(process.GetState(), lldb.eStateStopped)
>
> AssertionError: 10 != 5
>
> Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
>
> ======================================================================
>
> FAIL: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
>
>    Test that that file and class static variables display correctly.
>
> ----------------------------------------------------------------------
>
> Traceback (most recent call last):
>
>   File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
>
>     return attrvalue(self)
>
>   File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
>
>     (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
>
>   File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
>
>     return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
>
>   File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
>
>     test.assertEqual(process.GetState(), lldb.eStateStopped)
>
> AssertionError: 10 != 5
>
> Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
>
> ----------------------------------------------------------------------
>
> Ran 8 tests in 3.035s
>
>
> RESULT: FAILED (0 passes, 4 failures, 0 errors, 4 skipped, 0 expected failures, 0 unexpected successes)
>
>
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

More information about the lldb-dev mailing list