[lldb-dev] Some API test failures are really opaque/could be improved
Raphael Isemann via lldb-dev
lldb-dev at lists.llvm.org
Tue Oct 19 09:02:13 PDT 2021
I just saw in your review comment that this is using
LLVM_ENABLE_RUNTIMES and not LLVM_ENABLE_PROJECTS for libcxx, so the
failure just comes from us setting the wrong RPATH due to the
different runtimes library directory (at least from what I can see).
Would it be possible to put libcxx in LLVM_ENABLE_PROJECTS for now? I
think this shouldn't be too hard to fix (famous last words?).
Am Mo., 18. Okt. 2021 um 22:30 Uhr schrieb David Blaikie <dblaikie at gmail.com>:
>
> On Mon, Oct 18, 2021 at 9:45 AM Raphael Isemann <teemperor at gmail.com> wrote:
>>
>> I think https://reviews.llvm.org/D111978 ,
>> https://reviews.llvm.org/D111981 and the other patches Pavel & me put
>> up today should improve this situation IIUC.
>
>
> Thanks Raphael - really appreciate you & looking into this!
>
> With https://reviews.llvm.org/D111981 I still seem to not have that cxx dependency (building/running the test, then building libcxx, then running the test again goes from unsupported -> failing) - didn't seem to work for me?
>
> The diagnostic improvement sounds good to me (pity about whatever limitations it has re: remote or windows execution gathering the stdout). I guess gathering the logs in the remote execution case has the same problem (that the log is made on the remote machine/non-trivial to retrieve?)?
>
> & yeah, have you got any patches/ideas about how to make the libcxx tests use the just-built libcxx library (using LD_LIBRARY_PATH, rpath, etc)? For now, even with both these fixes I'll just be seeing (consistent, which is nice) failures, rather than being able to run these tests successfully. I'll either have to get used to ignoring certain failures, or disable the tests by not building libcxx in that build tree, which would also be unfortunate. (or maybe there's some other workarounds?) Any idea how this works for other folks?
>
> - Dave
>
>> - Raphael
>>
>> Am Mo., 18. Okt. 2021 um 05:54 Uhr schrieb David Blaikie via lldb-dev
>> <lldb-dev at lists.llvm.org>:
>> >
>> > Wondering if anyone else has encountered/dealt with debugging lldb test failures like the one shown at the end of this email ("AssertionError: 10 != 5" in "test.assertEqual(process.GetState(), lldb.eStateStopped)" while checking that a breakpoint was reached)
>> >
>> > Is there anything that could be done to improve the debuggability of such failures? Logging standard output/error from the lldb process or any other logging it might have? At least for one of these I managed to figure it out by running lldb directly on the binary and finding that the binary couldn't find libc++.so (that's a story for another bug/email thread, looks like maybe lldb libc++ pretty printer tests are using the system installed libc++, not the just-built libc++ (& thus not running if there is no system installed libc++)). But my current failure like this seems a bit more inscrutible and I'm still looking into it.
>> >
>> > Looks like dotest.py has some sense of logging (it has a --log-success option which says preserves the logs even on failure - though the output of dotest.py, at least for me, has no mention of logs, log files, or where they might be located, and looking at the source points to some sort of ".log" files... ah, found some)
>> >
>> > So, yeah, there do seem to be some Failure.log, SkippedTest.log, etc - should dotest print something about the paths to those files when it exits with failure, maybe?
>> >
>> > ********************
>> >
>> > FAIL: lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py (23 of 23)
>> >
>> > ******************** TEST 'lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py' FAILED ********************
>> >
>> > Script:
>> >
>> > --
>> >
>> > /usr/bin/python3 /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/dotest.py -u CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/bin/ar --env OBJCOPY=/usr/bin/objcopy --env LLVM_LIBS_DIR=/usr/local/google/home/blaikie/dev/llvm/build/release/./lib --arch x86_64 --build-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex --lldb-module-cache-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-lldb/lldb-api --clang-module-cache-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-clang/lldb-api --executable /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/lldb --compiler /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/clang --dsymutil /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/dsymutil --llvm-tools-dir /usr/local/google/home/blaikie/dev/llvm/build/release/./bin --lldb-libs-dir /usr/local/google/home/blaikie/dev/llvm/build/release/./lib /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set -p TestDataFormatterLibcxxSet.py
>> >
>> > --
>> >
>> > Exit Code: 1
>> >
>> >
>> > Command Output (stdout):
>> >
>> > --
>> >
>> > lldb version 14.0.0 (git at github.com:llvm/llvm-project.git revision 6176fda3f992b5086302b3826aa0636135cc4cc0)
>> >
>> > clang revision 6176fda3f992b5086302b3826aa0636135cc4cc0
>> >
>> > llvm revision 6176fda3f992b5086302b3826aa0636135cc4cc0
>> >
>> > Skipping the following test categories: ['dsym', 'gmodules', 'debugserver', 'objc']
>> >
>> >
>> > --
>> >
>> > Command Output (stderr):
>> >
>> > --
>> >
>> > UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dsym (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run)
>> >
>> > FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
>> >
>> > FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
>> >
>> > UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_gmodules (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run)
>> >
>> > UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dsym (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run)
>> >
>> > FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
>> >
>> > FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
>> >
>> > UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_gmodules (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run)
>> >
>> > ======================================================================
>> >
>> > FAIL: test_ref_and_ptr_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
>> >
>> > Test that the data formatters work on ref and ptr.
>> >
>> > ----------------------------------------------------------------------
>> >
>> > Traceback (most recent call last):
>> >
>> > File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
>> >
>> > return attrvalue(self)
>> >
>> > File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 128, in test_ref_and_ptr
>> >
>> > (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
>> >
>> > File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
>> >
>> > return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
>> >
>> > File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
>> >
>> > test.assertEqual(process.GetState(), lldb.eStateStopped)
>> >
>> > AssertionError: 10 != 5
>> >
>> > Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
>> >
>> > ======================================================================
>> >
>> > FAIL: test_ref_and_ptr_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
>> >
>> > Test that the data formatters work on ref and ptr.
>> >
>> > ----------------------------------------------------------------------
>> >
>> > Traceback (most recent call last):
>> >
>> > File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
>> >
>> > return attrvalue(self)
>> >
>> > File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 128, in test_ref_and_ptr
>> >
>> > (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
>> >
>> > File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
>> >
>> > return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
>> >
>> > File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
>> >
>> > test.assertEqual(process.GetState(), lldb.eStateStopped)
>> >
>> > AssertionError: 10 != 5
>> >
>> > Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
>> >
>> > ======================================================================
>> >
>> > FAIL: test_with_run_command_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
>> >
>> > Test that that file and class static variables display correctly.
>> >
>> > ----------------------------------------------------------------------
>> >
>> > Traceback (most recent call last):
>> >
>> > File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
>> >
>> > return attrvalue(self)
>> >
>> > File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
>> >
>> > (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
>> >
>> > File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
>> >
>> > return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
>> >
>> > File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
>> >
>> > test.assertEqual(process.GetState(), lldb.eStateStopped)
>> >
>> > AssertionError: 10 != 5
>> >
>> > Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
>> >
>> > ======================================================================
>> >
>> > FAIL: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
>> >
>> > Test that that file and class static variables display correctly.
>> >
>> > ----------------------------------------------------------------------
>> >
>> > Traceback (most recent call last):
>> >
>> > File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
>> >
>> > return attrvalue(self)
>> >
>> > File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
>> >
>> > (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
>> >
>> > File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
>> >
>> > return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
>> >
>> > File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
>> >
>> > test.assertEqual(process.GetState(), lldb.eStateStopped)
>> >
>> > AssertionError: 10 != 5
>> >
>> > Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
>> >
>> > ----------------------------------------------------------------------
>> >
>> > Ran 8 tests in 3.035s
>> >
>> >
>> > RESULT: FAILED (0 passes, 4 failures, 0 errors, 4 skipped, 0 expected failures, 0 unexpected successes)
>> >
>> >
>> > _______________________________________________
>> > lldb-dev mailing list
>> > lldb-dev at lists.llvm.org
>> > https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
More information about the lldb-dev
mailing list