[lldb-dev] Some API test failures are really opaque/could be improved
David Blaikie via lldb-dev
lldb-dev at lists.llvm.org
Mon Oct 18 13:30:28 PDT 2021
On Mon, Oct 18, 2021 at 9:45 AM Raphael Isemann <teemperor at gmail.com> wrote:
> I think https://reviews.llvm.org/D111978 ,
> https://reviews.llvm.org/D111981 and the other patches Pavel & me put
> up today should improve this situation IIUC.
>
Thanks Raphael - really appreciate you & looking into this!
With https://reviews.llvm.org/D111981 I still seem to not have that cxx
dependency (building/running the test, then building libcxx, then running
the test again goes from unsupported -> failing) - didn't seem to work for
me?
The diagnostic improvement sounds good to me (pity about whatever
limitations it has re: remote or windows execution gathering the stdout). I
guess gathering the logs in the remote execution case has the same problem
(that the log is made on the remote machine/non-trivial to retrieve?)?
& yeah, have you got any patches/ideas about how to make the libcxx tests
use the just-built libcxx library (using LD_LIBRARY_PATH, rpath, etc)? For
now, even with both these fixes I'll just be seeing (consistent, which is
nice) failures, rather than being able to run these tests successfully.
I'll either have to get used to ignoring certain failures, or disable the
tests by not building libcxx in that build tree, which would also be
unfortunate. (or maybe there's some other workarounds?) Any idea how this
works for other folks?
- Dave
- Raphael
>
> Am Mo., 18. Okt. 2021 um 05:54 Uhr schrieb David Blaikie via lldb-dev
> <lldb-dev at lists.llvm.org>:
> >
> > Wondering if anyone else has encountered/dealt with debugging lldb test
> failures like the one shown at the end of this email ("AssertionError: 10
> != 5" in "test.assertEqual(process.GetState(), lldb.eStateStopped)" while
> checking that a breakpoint was reached)
> >
> > Is there anything that could be done to improve the debuggability of
> such failures? Logging standard output/error from the lldb process or any
> other logging it might have? At least for one of these I managed to figure
> it out by running lldb directly on the binary and finding that the binary
> couldn't find libc++.so (that's a story for another bug/email thread, looks
> like maybe lldb libc++ pretty printer tests are using the system installed
> libc++, not the just-built libc++ (& thus not running if there is no system
> installed libc++)). But my current failure like this seems a bit more
> inscrutible and I'm still looking into it.
> >
> > Looks like dotest.py has some sense of logging (it has a --log-success
> option which says preserves the logs even on failure - though the output of
> dotest.py, at least for me, has no mention of logs, log files, or where
> they might be located, and looking at the source points to some sort of
> ".log" files... ah, found some)
> >
> > So, yeah, there do seem to be some Failure.log, SkippedTest.log, etc -
> should dotest print something about the paths to those files when it exits
> with failure, maybe?
> >
> > ********************
> >
> > FAIL: lldb-api ::
> functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py
> (23 of 23)
> >
> > ******************** TEST 'lldb-api ::
> functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py'
> FAILED ********************
> >
> > Script:
> >
> > --
> >
> > /usr/bin/python3
> /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/dotest.py -u
> CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/bin/ar --env
> OBJCOPY=/usr/bin/objcopy --env
> LLVM_LIBS_DIR=/usr/local/google/home/blaikie/dev/llvm/build/release/./lib
> --arch x86_64 --build-dir
> /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex
> --lldb-module-cache-dir
> /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-lldb/lldb-api
> --clang-module-cache-dir
> /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-clang/lldb-api
> --executable
> /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/lldb --compiler
> /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/clang
> --dsymutil
> /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/dsymutil
> --llvm-tools-dir
> /usr/local/google/home/blaikie/dev/llvm/build/release/./bin --lldb-libs-dir
> /usr/local/google/home/blaikie/dev/llvm/build/release/./lib
> /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set
> -p TestDataFormatterLibcxxSet.py
> >
> > --
> >
> > Exit Code: 1
> >
> >
> > Command Output (stdout):
> >
> > --
> >
> > lldb version 14.0.0 (git at github.com:llvm/llvm-project.git revision
> 6176fda3f992b5086302b3826aa0636135cc4cc0)
> >
> > clang revision 6176fda3f992b5086302b3826aa0636135cc4cc0
> >
> > llvm revision 6176fda3f992b5086302b3826aa0636135cc4cc0
> >
> > Skipping the following test categories: ['dsym', 'gmodules',
> 'debugserver', 'objc']
> >
> >
> > --
> >
> > Command Output (stderr):
> >
> > --
> >
> > UNSUPPORTED: LLDB
> (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
> test_ref_and_ptr_dsym
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does
> not fall in any category of interest for this run)
> >
> > FAIL: LLDB
> (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
> test_ref_and_ptr_dwarf
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
> >
> > FAIL: LLDB
> (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
> test_ref_and_ptr_dwo
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
> >
> > UNSUPPORTED: LLDB
> (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
> test_ref_and_ptr_gmodules
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does
> not fall in any category of interest for this run)
> >
> > UNSUPPORTED: LLDB
> (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
> test_with_run_command_dsym
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does
> not fall in any category of interest for this run)
> >
> > FAIL: LLDB
> (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
> test_with_run_command_dwarf
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
> >
> > FAIL: LLDB
> (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
> test_with_run_command_dwo
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
> >
> > UNSUPPORTED: LLDB
> (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
> test_with_run_command_gmodules
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does
> not fall in any category of interest for this run)
> >
> > ======================================================================
> >
> > FAIL: test_ref_and_ptr_dwarf
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
> >
> > Test that the data formatters work on ref and ptr.
> >
> > ----------------------------------------------------------------------
> >
> > Traceback (most recent call last):
> >
> > File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py",
> line 1823, in test_method
> >
> > return attrvalue(self)
> >
> > File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py",
> line 128, in test_ref_and_ptr
> >
> > (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
> >
> > File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py",
> line 970, in run_to_source_breakpoint
> >
> > return run_to_breakpoint_do_run(test, target, breakpoint,
> launch_info,
> >
> > File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py",
> line 892, in run_to_breakpoint_do_run
> >
> > test.assertEqual(process.GetState(), lldb.eStateStopped)
> >
> > AssertionError: 10 != 5
> >
> >
> Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
> >
> > ======================================================================
> >
> > FAIL: test_ref_and_ptr_dwo
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
> >
> > Test that the data formatters work on ref and ptr.
> >
> > ----------------------------------------------------------------------
> >
> > Traceback (most recent call last):
> >
> > File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py",
> line 1823, in test_method
> >
> > return attrvalue(self)
> >
> > File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py",
> line 128, in test_ref_and_ptr
> >
> > (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
> >
> > File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py",
> line 970, in run_to_source_breakpoint
> >
> > return run_to_breakpoint_do_run(test, target, breakpoint,
> launch_info,
> >
> > File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py",
> line 892, in run_to_breakpoint_do_run
> >
> > test.assertEqual(process.GetState(), lldb.eStateStopped)
> >
> > AssertionError: 10 != 5
> >
> >
> Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
> >
> > ======================================================================
> >
> > FAIL: test_with_run_command_dwarf
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
> >
> > Test that that file and class static variables display correctly.
> >
> > ----------------------------------------------------------------------
> >
> > Traceback (most recent call last):
> >
> > File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py",
> line 1823, in test_method
> >
> > return attrvalue(self)
> >
> > File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py",
> line 49, in test_with_run_command
> >
> > (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
> >
> > File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py",
> line 970, in run_to_source_breakpoint
> >
> > return run_to_breakpoint_do_run(test, target, breakpoint,
> launch_info,
> >
> > File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py",
> line 892, in run_to_breakpoint_do_run
> >
> > test.assertEqual(process.GetState(), lldb.eStateStopped)
> >
> > AssertionError: 10 != 5
> >
> >
> Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
> >
> > ======================================================================
> >
> > FAIL: test_with_run_command_dwo
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
> >
> > Test that that file and class static variables display correctly.
> >
> > ----------------------------------------------------------------------
> >
> > Traceback (most recent call last):
> >
> > File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py",
> line 1823, in test_method
> >
> > return attrvalue(self)
> >
> > File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py",
> line 49, in test_with_run_command
> >
> > (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
> >
> > File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py",
> line 970, in run_to_source_breakpoint
> >
> > return run_to_breakpoint_do_run(test, target, breakpoint,
> launch_info,
> >
> > File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py",
> line 892, in run_to_breakpoint_do_run
> >
> > test.assertEqual(process.GetState(), lldb.eStateStopped)
> >
> > AssertionError: 10 != 5
> >
> >
> Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
> >
> > ----------------------------------------------------------------------
> >
> > Ran 8 tests in 3.035s
> >
> >
> > RESULT: FAILED (0 passes, 4 failures, 0 errors, 4 skipped, 0 expected
> failures, 0 unexpected successes)
> >
> >
> > _______________________________________________
> > lldb-dev mailing list
> > lldb-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20211018/6ef05a3f/attachment-0001.html>
More information about the lldb-dev
mailing list