[lldb-dev] Some API test failures are really opaque/could be improved

David Blaikie via lldb-dev lldb-dev at lists.llvm.org
Tue Oct 19 16:55:43 PDT 2021 

On Tue, Oct 19, 2021 at 9:08 AM Raphael Isemann <teemperor at gmail.com> wrote:

> Actually the RPATH theory is wrong, but the LLVM_ENABLE_PROJECT
> workaround *should* still work.
>

I'll give that a go (it's running at the moment) though I guess this is
inconsistent with the direction libcxx is moving in for building, re:
https://groups.google.com/g/llvm-dev/c/tpuLxk_ipLw


>
> Am Di., 19. Okt. 2021 um 18:02 Uhr schrieb Raphael Isemann
> <teemperor at gmail.com>:
> >
> > I just saw in your review comment that this is using
> > LLVM_ENABLE_RUNTIMES and not LLVM_ENABLE_PROJECTS for libcxx, so the
> > failure just comes from us setting the wrong RPATH due to the
> > different runtimes library directory (at least from what I can see).
> > Would it be possible to put libcxx in LLVM_ENABLE_PROJECTS for now? I
> > think this shouldn't be too hard to fix (famous last words?).
> >
> > Am Mo., 18. Okt. 2021 um 22:30 Uhr schrieb David Blaikie <
> dblaikie at gmail.com>:
> > >
> > > On Mon, Oct 18, 2021 at 9:45 AM Raphael Isemann <teemperor at gmail.com>
> wrote:
> > >>
> > >> I think https://reviews.llvm.org/D111978 ,
> > >> https://reviews.llvm.org/D111981 and the other patches Pavel & me put
> > >> up today should improve this situation IIUC.
> > >
> > >
> > > Thanks Raphael - really appreciate you & looking into this!
> > >
> > > With https://reviews.llvm.org/D111981 I still seem to not have that
> cxx dependency (building/running the test, then building libcxx, then
> running the test again goes from unsupported -> failing) - didn't seem to
> work for me?
> > >
> > > The diagnostic improvement sounds good to me (pity about whatever
> limitations it has re: remote or windows execution gathering the stdout). I
> guess gathering the logs in the remote execution case has the same problem
> (that the log is made on the remote machine/non-trivial to retrieve?)?
> > >
> > > & yeah, have you got any patches/ideas about how to make the libcxx
> tests use the just-built libcxx library (using LD_LIBRARY_PATH, rpath,
> etc)? For now, even with both these fixes I'll just be seeing (consistent,
> which is nice) failures, rather than being able to run these tests
> successfully. I'll either have to get used to ignoring certain failures, or
> disable the tests by not building libcxx in that build tree, which would
> also be unfortunate. (or maybe there's some other workarounds?) Any idea
> how this works for other folks?
> > >
> > > - Dave
> > >
> > >> - Raphael
> > >>
> > >> Am Mo., 18. Okt. 2021 um 05:54 Uhr schrieb David Blaikie via lldb-dev
> > >> <lldb-dev at lists.llvm.org>:
> > >> >
> > >> > Wondering if anyone else has encountered/dealt with debugging lldb
> test failures like the one shown at the end of this email ("AssertionError:
> 10 != 5" in "test.assertEqual(process.GetState(), lldb.eStateStopped)"
> while checking that a breakpoint was reached)
> > >> >
> > >> > Is there anything that could be done to improve the debuggability
> of such failures? Logging standard output/error from the lldb process or
> any other logging it might have? At least for one of these I managed to
> figure it out by running lldb directly on the binary and finding that the
> binary couldn't find libc++.so (that's a story for another bug/email
> thread, looks like maybe lldb libc++ pretty printer tests are using the
> system installed libc++, not the just-built libc++ (& thus not running if
> there is no system installed libc++)). But my current failure like this
> seems a bit more inscrutible and I'm still looking into it.
> > >> >
> > >> > Looks like dotest.py has some sense of logging (it has a
> --log-success option which says preserves the logs even on failure - though
> the output of dotest.py, at least for me, has no mention of logs, log
> files, or where they might be located, and looking at the source points to
> some sort of ".log" files... ah, found some)
> > >> >
> > >> > So, yeah, there do seem to be some Failure.log, SkippedTest.log,
> etc - should dotest print something about the paths to those files when it
> exits with failure, maybe?
> > >> >
> > >> > ********************
> > >> >
> > >> > FAIL: lldb-api ::
> functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py
> (23 of 23)
> > >> >
> > >> > ******************** TEST 'lldb-api ::
> functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py'
> FAILED ********************
> > >> >
> > >> > Script:
> > >> >
> > >> > --
> > >> >
> > >> > /usr/bin/python3
> /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/dotest.py -u
> CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/bin/ar --env
> OBJCOPY=/usr/bin/objcopy --env
> LLVM_LIBS_DIR=/usr/local/google/home/blaikie/dev/llvm/build/release/./lib
> --arch x86_64 --build-dir
> /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex
> --lldb-module-cache-dir
> /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-lldb/lldb-api
> --clang-module-cache-dir
> /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-clang/lldb-api
> --executable
> /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/lldb --compiler
> /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/clang
> --dsymutil
> /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/dsymutil
> --llvm-tools-dir
> /usr/local/google/home/blaikie/dev/llvm/build/release/./bin --lldb-libs-dir
> /usr/local/google/home/blaikie/dev/llvm/build/release/./lib
> /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set
> -p TestDataFormatterLibcxxSet.py
> > >> >
> > >> > --
> > >> >
> > >> > Exit Code: 1
> > >> >
> > >> >
> > >> > Command Output (stdout):
> > >> >
> > >> > --
> > >> >
> > >> > lldb version 14.0.0 (git at github.com:llvm/llvm-project.git revision
> 6176fda3f992b5086302b3826aa0636135cc4cc0)
> > >> >
> > >> >   clang revision 6176fda3f992b5086302b3826aa0636135cc4cc0
> > >> >
> > >> >   llvm revision 6176fda3f992b5086302b3826aa0636135cc4cc0
> > >> >
> > >> > Skipping the following test categories: ['dsym', 'gmodules',
> 'debugserver', 'objc']
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> > Command Output (stderr):
> > >> >
> > >> > --
> > >> >
> > >> > UNSUPPORTED: LLDB
> (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
> test_ref_and_ptr_dsym
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does
> not fall in any category of interest for this run)
> > >> >
> > >> > FAIL: LLDB
> (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
> test_ref_and_ptr_dwarf
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
> > >> >
> > >> > FAIL: LLDB
> (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
> test_ref_and_ptr_dwo
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
> > >> >
> > >> > UNSUPPORTED: LLDB
> (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
> test_ref_and_ptr_gmodules
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does
> not fall in any category of interest for this run)
> > >> >
> > >> > UNSUPPORTED: LLDB
> (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
> test_with_run_command_dsym
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does
> not fall in any category of interest for this run)
> > >> >
> > >> > FAIL: LLDB
> (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
> test_with_run_command_dwarf
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
> > >> >
> > >> > FAIL: LLDB
> (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
> test_with_run_command_dwo
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
> > >> >
> > >> > UNSUPPORTED: LLDB
> (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) ::
> test_with_run_command_gmodules
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does
> not fall in any category of interest for this run)
> > >> >
> > >> >
> ======================================================================
> > >> >
> > >> > FAIL: test_ref_and_ptr_dwarf
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
> > >> >
> > >> >    Test that the data formatters work on ref and ptr.
> > >> >
> > >> >
> ----------------------------------------------------------------------
> > >> >
> > >> > Traceback (most recent call last):
> > >> >
> > >> >   File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py",
> line 1823, in test_method
> > >> >
> > >> >     return attrvalue(self)
> > >> >
> > >> >   File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py",
> line 128, in test_ref_and_ptr
> > >> >
> > >> >     (self.target, process, _, bkpt) =
> lldbutil.run_to_source_breakpoint(
> > >> >
> > >> >   File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py",
> line 970, in run_to_source_breakpoint
> > >> >
> > >> >     return run_to_breakpoint_do_run(test, target, breakpoint,
> launch_info,
> > >> >
> > >> >   File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py",
> line 892, in run_to_breakpoint_do_run
> > >> >
> > >> >     test.assertEqual(process.GetState(), lldb.eStateStopped)
> > >> >
> > >> > AssertionError: 10 != 5
> > >> >
> > >> >
> Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
> > >> >
> > >> >
> ======================================================================
> > >> >
> > >> > FAIL: test_ref_and_ptr_dwo
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
> > >> >
> > >> >    Test that the data formatters work on ref and ptr.
> > >> >
> > >> >
> ----------------------------------------------------------------------
> > >> >
> > >> > Traceback (most recent call last):
> > >> >
> > >> >   File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py",
> line 1823, in test_method
> > >> >
> > >> >     return attrvalue(self)
> > >> >
> > >> >   File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py",
> line 128, in test_ref_and_ptr
> > >> >
> > >> >     (self.target, process, _, bkpt) =
> lldbutil.run_to_source_breakpoint(
> > >> >
> > >> >   File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py",
> line 970, in run_to_source_breakpoint
> > >> >
> > >> >     return run_to_breakpoint_do_run(test, target, breakpoint,
> launch_info,
> > >> >
> > >> >   File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py",
> line 892, in run_to_breakpoint_do_run
> > >> >
> > >> >     test.assertEqual(process.GetState(), lldb.eStateStopped)
> > >> >
> > >> > AssertionError: 10 != 5
> > >> >
> > >> >
> Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
> > >> >
> > >> >
> ======================================================================
> > >> >
> > >> > FAIL: test_with_run_command_dwarf
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
> > >> >
> > >> >    Test that that file and class static variables display correctly.
> > >> >
> > >> >
> ----------------------------------------------------------------------
> > >> >
> > >> > Traceback (most recent call last):
> > >> >
> > >> >   File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py",
> line 1823, in test_method
> > >> >
> > >> >     return attrvalue(self)
> > >> >
> > >> >   File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py",
> line 49, in test_with_run_command
> > >> >
> > >> >     (self.target, process, _, bkpt) =
> lldbutil.run_to_source_breakpoint(
> > >> >
> > >> >   File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py",
> line 970, in run_to_source_breakpoint
> > >> >
> > >> >     return run_to_breakpoint_do_run(test, target, breakpoint,
> launch_info,
> > >> >
> > >> >   File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py",
> line 892, in run_to_breakpoint_do_run
> > >> >
> > >> >     test.assertEqual(process.GetState(), lldb.eStateStopped)
> > >> >
> > >> > AssertionError: 10 != 5
> > >> >
> > >> >
> Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
> > >> >
> > >> >
> ======================================================================
> > >> >
> > >> > FAIL: test_with_run_command_dwo
> (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
> > >> >
> > >> >    Test that that file and class static variables display correctly.
> > >> >
> > >> >
> ----------------------------------------------------------------------
> > >> >
> > >> > Traceback (most recent call last):
> > >> >
> > >> >   File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py",
> line 1823, in test_method
> > >> >
> > >> >     return attrvalue(self)
> > >> >
> > >> >   File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py",
> line 49, in test_with_run_command
> > >> >
> > >> >     (self.target, process, _, bkpt) =
> lldbutil.run_to_source_breakpoint(
> > >> >
> > >> >   File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py",
> line 970, in run_to_source_breakpoint
> > >> >
> > >> >     return run_to_breakpoint_do_run(test, target, breakpoint,
> launch_info,
> > >> >
> > >> >   File
> "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py",
> line 892, in run_to_breakpoint_do_run
> > >> >
> > >> >     test.assertEqual(process.GetState(), lldb.eStateStopped)
> > >> >
> > >> > AssertionError: 10 != 5
> > >> >
> > >> >
> Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
> > >> >
> > >> >
> ----------------------------------------------------------------------
> > >> >
> > >> > Ran 8 tests in 3.035s
> > >> >
> > >> >
> > >> > RESULT: FAILED (0 passes, 4 failures, 0 errors, 4 skipped, 0
> expected failures, 0 unexpected successes)
> > >> >
> > >> >
> > >> > _______________________________________________
> > >> > lldb-dev mailing list
> > >> > lldb-dev at lists.llvm.org
> > >> > https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20211019/ae84799c/attachment-0001.html>

More information about the lldb-dev mailing list