[llvm-dev] Many LLDB test failures in 13.0.0-rc1 on macOS

Thu Sep 9 01:01:46 PDT 2021

Stefan,

Just picked this up again (sorry for the delay) but I wanted to go
back and build 12.0.1 since I never got that out for macOS before -
and I am unsure what's happening now because even these tests on lldb
fails and it completely stops at some point in
StandardStartupTest.TestStopReplyContainsThreadPcs.

I am pretty sure at this point that it's something with my system
since some release has worked and other hasn't, but I need help to
figure it out since right now I am not able to put up binaries for
macOS at all, here is the final rows of check-lldb:

********************
FAIL: lldb-unit ::
tools/lldb-server/tests/./LLDBServerTests/StandardStartupTest.TestStopReplyContainsThreadPcs
(2222 of 2224)
******************** TEST 'lldb-unit ::
tools/lldb-server/tests/./LLDBServerTests/StandardStartupTest.TestStopReplyContainsThreadPcs'
FAILED
********************
Note: Google Test filter = StandardStartupTest.TestStopReplyContainsThreadPcs
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from StandardStartupTest
[ RUN ] StandardStartupTest.TestStopReplyContainsThreadPcs
/Users/tobias/code/llvm-releases/12.0.1/final/llvm-project/lldb/unittests/tools/lldb-server/tests/ThreadIdsInJstopinfoTest.cpp:35:
Failure
Value of: llvm::detail::TakeError(Client->SetInferior({getInferiorPath("thread_inferior"),
"4"}))
Expected: succeeded
Actual: failed (Error sending message `qProcessInfo`: ErrorDisconnected)
[ FAILED ] StandardStartupTest.TestStopReplyContainsThreadPcs (2413 ms)
[----------] 1 test from StandardStartupTest (2413 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (2413 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] StandardStartupTest.TestStopReplyContainsThreadPcs

1 FAILED TEST

[ INFO ] /Users/tobias/code/llvm-releases/12.0.1/final/llvm-project/lldb/unittests/tools/lldb-server/tests/TestClient.cpp:195::
Send Packet:
QStartNoAckMode

[ INFO ] /Users/tobias/code/llvm-releases/12.0.1/final/llvm-project/lldb/unittests/tools/lldb-server/tests/TestClient.cpp:198::
Read Packet:
OK

[ INFO ] /Users/tobias/code/llvm-releases/12.0.1/final/llvm-project/lldb/unittests/tools/lldb-server/tests/TestClient.cpp:195::
Send Packet:
A286,0,2F55736572732F746F626961732F636F64652F6C6C766D2D72656C65617365732F31322E302E312F66696E616C2F5068617365332F52656C656173652F6C6C766D436
F72652D31322E302E312D66696E616C2E6F626A2F746F6F6C732F6C6C64622F756E697474657374732F746F6F6C732F6C6C64622D7365727665722F2E2F7468726561645F696E
666572696F72,2,1,34

[ INFO ] /Users/tobias/code/llvm-releases/12.0.1/final/llvm-project/lldb/unittests/tools/lldb-server/tests/TestClient.cpp:198::
Read Packet:
OK

[ INFO ] /Users/tobias/code/llvm-releases/12.0.1/final/llvm-project/lldb/unittests/tools/lldb-server/tests/TestClient.cpp:195::
Send Packet:
qLaunchSuccess

[ INFO ] /Users/tobias/code/llvm-releases/12.0.1/final/llvm-project/lldb/unittests/tools/lldb-server/tests/TestClient.cpp:198::
Read Packet:
Efailed to get the task for process 5550

[ INFO ] /Users/tobias/code/llvm-releases/12.0.1/final/llvm-project/lldb/unittests/tools/lldb-server/tests/TestClient.cpp:195::
Send Packet:
qProcessInfo

[ INFO ] /Users/tobias/code/llvm-releases/12.0.1/final/llvm-project/lldb/unittests/tools/lldb-server/tests/TestClient.cpp:198::
Read Packet:

********************
^C interrupted by user, skipping remaining tests
********************
Failed Tests (38):
lldb-shell :: Breakpoint/jitbp_elf.test
lldb-shell :: Commands/command-thread-select.test
lldb-shell :: ExecControl/StopHook/stop-hook.test
lldb-shell :: Expr/TestIRMemoryMap.test
lldb-shell :: Host/TestCustomShell.test
lldb-shell :: Process/Optimization.test
lldb-shell :: Process/TestEnvironment.test
lldb-shell :: Process/UnsupportedLanguage.test
lldb-shell :: Register/x86-64-gp-read.test
lldb-shell :: Register/x86-64-read.test
lldb-shell :: Register/x86-64-ymm-read.test
lldb-shell :: Register/x86-multithread-read.test
lldb-shell :: Register/x86-multithread-write.test
lldb-shell :: Reproducer/Functionalities/TestDataFormatter.test
lldb-shell :: Reproducer/Functionalities/TestExpressionEvaluation.test
lldb-shell :: Reproducer/Functionalities/TestImageList.test
lldb-shell :: Reproducer/Functionalities/TestStepping.test
lldb-shell :: Reproducer/Modules/TestModuleCXX.test
lldb-shell :: Reproducer/TestDSYM.test
lldb-shell :: Reproducer/TestDump.test
lldb-shell :: Reproducer/TestFileRepro.test
lldb-shell :: Reproducer/TestGDBRemoteRepro.test
lldb-shell :: Reproducer/TestMultipleTargets.test
lldb-shell :: Reproducer/TestRelativePath.test
lldb-shell :: Reproducer/TestReuseDirectory.test
lldb-shell :: Reproducer/TestVerify.test
lldb-shell :: Reproducer/TestVersionCheck.test
lldb-shell :: Reproducer/TestWorkingDir.test
lldb-shell :: Settings/TestFrameFormatColor.test
lldb-shell :: Settings/TestFrameFormatMangling.test
lldb-shell :: Settings/TestFrameFormatNoColor.test
lldb-shell :: Settings/TestLineMarkerColor.test
lldb-shell :: SymbolFile/DWARF/deterministic-build.cpp
lldb-shell :: Unwind/eh-frame-dwarf-unwind.test
lldb-shell :: Unwind/thread-step-out-ret-addr-check.test
lldb-shell :: Unwind/trap_frame_sym_ctx.test
lldb-shell :: Watchpoint/SetErrorCases.test
lldb-unit :: tools/lldb-server/tests/./LLDBServerTests/StandardStartupTest.TestStopReplyContainsThreadPcs

Testing Time: 87.78s
Skipped : 2
Unsupported : 1009
Passed : 1165
Expectedly Failed: 10
Failed : 38

2 warning(s) in tests
ninja: build stopped: interrupted by user.

This same problem happens in the 13.0.0-rc2 branch.

I am on latest stable version of macOS 11.5.2 with Xcode 12.5.1 installed.

Thanks,
Tobias

On Mon, Aug 23, 2021 at 5:29 PM Stefan Gränitz
<stefan.graenitz at gmail.com> wrote:
>
> Ok thanks. FYI: As in the subject line, I used the version tagged as
> llvmorg-13.0.0-rc1
>
> On 23/08/2021 12:24, Tobias Hieta wrote:
> > Hello Stefan,
> >
> > I haven't tried to rebuild the release branch recently. I will try
> > soon and report back.
> >
> > -- Tobias
> >
> > On Sun, Aug 22, 2021 at 4:11 PM Stefan Gränitz
> > <stefan.graenitz at gmail.com> wrote:
> >> Hi Tobias
> >>
> >> What's the latest status here? I just made a local build from the tagged
> >> version on macOS 11.5.2 and CommandLineTools 12.5.1. Running the entire
> >> test suite, I do see the typical flakiness but nothing surprising:
> >>
> >> Failed Tests (8):
> >>   lldb-api :: commands/expression/diagnostics/TestExprDiagnostics.py
> >>   lldb-api :: commands/platform/sdk/TestPlatformSDK.py
> >>   lldb-api :: lang/c/modules/TestCModules.py
> >>   lldb-api :: lang/objc/modules-auto-import/TestModulesAutoImport.py
> >>   lldb-api ::
> >> lang/objc/modules-non-objc-target/TestObjCModulesNonObjCTarget.py
> >>   lldb-api :: lang/objc/modules-objc-property/TestModulesObjCProperty.py
> >>   lldb-api :: lang/objc/modules/TestObjCModules.py
> >>   lldb-api :: macosx/posix_spawn/TestLaunchProcessPosixSpawn.py
> >>
> >> Unexpectedly Passed Tests (1):
> >>   lldb-api :: lang/objc/foundation/TestRuntimeTypes.py
> >>
> >> Am I missing anything here? My major concern is that the
> >> Shell/Breakpoint/jit-loader tests keep working in upcoming releases.
> >>
> >> Thanks
> >> Stefan
> >>
> >> On 16/08/2021 11:12, Tobias Hieta via llvm-dev wrote:
> >>> Hello,
> >>>
> >>> While testing LLVM 13.0.0-rc1 on macOS I noticed many lldb test
> >>> failures that are regressions from 12.0.1.
> >>>
> >>> I was going to post this to lldb-dev - but I was not subscribed and
> >>> the process seems manual still. So forgive me for posting it here. Let
> >>> me know if I can share anymore information about these test failures
> >>> with the team.
> >>>
> >>> List of failing tests:
> >>> FAIL: lldb-shell :: Commands/command-thread-select.test (94817 of 96071)
> >>> FAIL: lldb-shell :: Breakpoint/jit-loader_jitlink_elf.test (94818 of 96071)
> >>> FAIL: lldb-shell :: Breakpoint/jit-loader_rtdyld_elf.test (94819 of 96071)
> >>> FAIL: lldb-shell :: ExecControl/StopHook/stop-hook.test (94823 of 96071)
> >>> FAIL: lldb-shell :: Host/TestCustomShell.test (94842 of 96071)
> >>> FAIL: lldb-shell :: Expr/TestIRMemoryMap.test (94849 of 96071)
> >>> FAIL: lldb-shell :: Process/Optimization.test (94852 of 96071)
> >>> FAIL: lldb-shell :: Process/UnsupportedLanguage.test (94853 of 96071)
> >>> FAIL: lldb-shell :: Process/TestEnvironment.test (94854 of 96071)
> >>> FAIL: lldb-shell :: Register/x86-64-gp-read.test (94855 of 96071)
> >>> FAIL: lldb-shell :: Register/x86-64-read.test (94862 of 96071)
> >>> FAIL: lldb-shell :: Register/x86-64-ymm-read.test (94866 of 96071)
> >>> FAIL: lldb-shell :: Register/x86-multithread-read.test (94873 of 96071)
> >>> FAIL: lldb-shell :: Register/x86-multithread-write.test (94876 of 96071)
> >>> FAIL: lldb-shell :: Reproducer/Functionalities/TestImageList.test
> >>> (94879 of 96071)
> >>> FAIL: lldb-shell :: Reproducer/Functionalities/TestDataFormatter.test
> >>> (94880 of 96071)
> >>> FAIL: lldb-shell ::
> >>> Reproducer/Functionalities/TestExpressionEvaluation.test (94882 of
> >>> 96071)
> >>> FAIL: lldb-shell :: Reproducer/Functionalities/TestStepping.test
> >>> (94883 of 96071)
> >>> FAIL: lldb-shell :: Reproducer/Modules/TestModuleCXX.test (94884 of 96071)
> >>> FAIL: lldb-shell :: Reproducer/TestDSYM.test (94902 of 96071)
> >>> FAIL: lldb-shell :: Reproducer/TestDump.test (94922 of 96071)
> >>> FAIL: lldb-shell :: Reproducer/TestFileRepro.test (94924 of 96071)
> >>> FAIL: lldb-shell :: Reproducer/TestGDBRemoteRepro.test (94931 of 96071)
> >>> FAIL: lldb-shell :: Reproducer/TestMultipleTargets.test (94932 of 96071)
> >>> FAIL: lldb-shell :: Reproducer/TestRelativePath.test (94934 of 96071)
> >>> FAIL: lldb-shell :: Reproducer/TestReuseDirectory.test (94946 of 96071)
> >>> FAIL: lldb-shell :: Reproducer/TestVersionCheck.test (94947 of 96071)
> >>> FAIL: lldb-shell :: Reproducer/TestVerify.test (94948 of 96071)
> >>> FAIL: lldb-shell :: Reproducer/TestWorkingDir.test (94950 of 96071)
> >>> FAIL: lldb-shell :: Settings/TestFrameFormatColor.test (94957 of 96071)
> >>> FAIL: lldb-shell :: Settings/TestFrameFormatMangling.test (94961 of 96071)
> >>> FAIL: lldb-shell :: Settings/TestFrameFormatNoColor.test (94969 of 96071)
> >>> FAIL: lldb-shell :: Settings/TestLineMarkerColor.test (94975 of 96071)
> >>> FAIL: lldb-shell :: Subprocess/fork-follow-parent-wp.test (95010 of 96071)
> >>> FAIL: lldb-shell :: Subprocess/fork-follow-parent.test (95032 of 96071)
> >>> FAIL: lldb-shell :: Subprocess/vfork-follow-parent.test (95045 of 96071)
> >>> FAIL: lldb-shell :: SymbolFile/DWARF/deterministic-build.cpp (95046 of 96071)
> >>> FAIL: lldb-shell :: Unwind/eh-frame-dwarf-unwind.test (95219 of 96071)
> >>> FAIL: lldb-shell :: Unwind/thread-step-out-ret-addr-check.test (96063 of 96071)
> >>> FAIL: lldb-shell :: Unwind/trap_frame_sym_ctx.test (96066 of 96071)
> >>> FAIL: lldb-shell :: Watchpoint/SetErrorCases.test (96067 of 96071)
> >>> FAIL: lldb-unit ::
> >>> tools/lldb-server/tests/./LLDBServerTests/StandardStartupTest.TestStopReplyContainsThreadPcs
> >>> (96068 of 96071)
> >>>
> >>> -- Tobias
> >>> _______________________________________________
> >>> LLVM Developers mailing list
> >>> llvm-dev at lists.llvm.org
> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >> --
> >> https://weliveindetail.github.io/blog/about/
> >>
> >>
> --
> https://weliveindetail.github.io/blog/about/
>