[compiler-rt] [RTSan][Apple] Disable AccessingALargeAtomicVariableDiesWhenRealtime (PR #129309)
via llvm-commits
llvm-commits at lists.llvm.org
Wed Mar 12 08:44:17 PDT 2025
davidtrevelyan wrote:
> @davidtrevelyan you said you can only step to cxx_atomic_load. But in the other stack trace I see __atomic_load is what calls OSSpinLock. Can you try to set a break point on __atomic_load and confirm it is never hit? can you disassemble from that frame? I'd really appreciate if we could get a comparison between working device vs non-working device on that symbol.
@thetruestblue thanks so much for the insightful questions - your prompting has led me to what I believe could be a suitable fix. I'll answer your questions directly first, and then outline my idea for the fix. Reading between the lines of your questions, I suspect you may already be planning something along the same lines.
> Can you try to set a break point on __atomic_load and confirm it is never hit?
Great question, thanks for asking me to revisit this. I had not previously appreciated that lldb doesn't automatically attach to forked child processes, and this is why I thought it wasn't being hit. I'll add some additional background info quickly here as to why this is relevant: as you're probably aware already, the unit tests in question here check that invoking some real-time unsafe operations do two things:
1. trigger an RTSan error when invoked in a real-time context, as defined by the `[[clang::nonblocking]]` attribute, and
2. *do not* trigger an RTSan error when invoked in a non-real-time context.
Asserting that 2. is true is, of course, trivial; we just invoke the function in the main test process and if it returns then we're good. But asserting that 1. is true is a bit more involved: we currently make use of googletest's `EXPECT_DEATH` macro, which forks the process and checks the return code. We have two helper methods that, given some function, make this testing easy:
1. `ExpectRealtimeDeath`, and
2. `ExpectNonRealtimeSurvival`.
Both accept a function as an argument and invoke it. [`ExpectRealtimeDeath`](https://github.com/llvm/llvm-project/blob/8be1d1235d58d5b2711295dbd9b36abe4b2401d0/compiler-rt/lib/rtsan/tests/rtsan_test_utilities.h#L26) is the function that invokes another `[[clang::nonblocking]]` function named `RealtimeInvoke` in a child process. What I found was:
- `__atomic_load` was not hit during `ExpectRealtimeDeath`, but
- `__atomic_load` *was* hit during `ExpectNonRealtimeSurvival`.
This makes sense! My debugger was not attaching automatically to the child process, so wasn't breaking in `__atomic_load` during `ExpectRealtimeDeath`. I temporarily switched to running `RealtimeInvoke` in the main process for exploration and, with my new-found ability to trap `__atomic_load` in the debugger, along with your next question, this led me to some new info.
> can you disassemble from that frame?
Excellent idea - this was the key insight - thanks again. Yes. Here's what I found (backtrace first):
```txt
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 2.1
* frame #0: 0x00000001ac241b18 libcompiler_rt.dylib`__atomic_load
frame #1: 0x00000001000087a8 Rtsan-arm64-Test`TestRtsan_AccessingALargeAtomicVariableDiesWhenRealtime_Test::TestBody() [inlined] std::__1::array<float, 2048ul> std::__1::__cxx_atomic_load[abi:ne180100]<std::__1::array<float, 2048ul>>(__a=0x000000016fdfa8d8, __order=memory_order_seq_cst) at cxx_atomic_impl.h:349:10 [opt]
frame #2: 0x0000000100008790 Rtsan-arm64-Test`TestRtsan_AccessingALargeAtomicVariableDiesWhenRealtime_Test::TestBody() [inlined] std::__1::__atomic_base<std::__1::array<float, 2048ul>, false>::load[abi:ne180100](this=0x000000016fdfa8d8, __m=memory_order_seq_cst) const at atomic_base.h:60:12 [opt]
frame #3: 0x0000000100008790 Rtsan-arm64-Test`TestRtsan_AccessingALargeAtomicVariableDiesWhenRealtime_Test::TestBody() [inlined] TestRtsan_AccessingALargeAtomicVariableDiesWhenRealtime_Test::TestBody()::$_1::operator()(this=<unavailable>) const at rtsan_test_functional.cpp:184:46 [opt]
frame #4: 0x0000000100008790 Rtsan-arm64-Test`TestRtsan_AccessingALargeAtomicVariableDiesWhenRealtime_Test::TestBody() [inlined] void rtsan_testing::RealtimeInvoke<TestRtsan_AccessingALargeAtomicVariableDiesWhenRealtime_Test::TestBody()::$_1&>(Func=<unavailable>) at rtsan_test_utilities.h:21:3 [opt]
frame #5: 0x000000010000878c Rtsan-arm64-Test`TestRtsan_AccessingALargeAtomicVariableDiesWhenRealtime_Test::TestBody(this=<unavailable>) at rtsan_test_functional.cpp:187:3 [opt]
frame #6: 0x0000000100066f68 Rtsan-arm64-Test`void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) [inlined] void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(object=<unavailable>, method=0x00000000000000010000000000000020, location="the test body") at gtest.cc:2612:10 [opt]
frame #7: 0x0000000100066f58 Rtsan-arm64-Test`void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(object=0x000060000298c980, method=0x00000000000000000000000000000020, location="the test body") at gtest.cc:2648:14 [opt]
frame #8: 0x0000000100066e44 Rtsan-arm64-Test`testing::Test::Run(this=0x000060000298c980) at gtest.cc:2687:5 [opt]
frame #9: 0x000000010006842c Rtsan-arm64-Test`testing::TestInfo::Run(this=0x00000001278052e0) at gtest.cc:2836:11 [opt]
frame #10: 0x0000000100069308 Rtsan-arm64-Test`testing::TestSuite::Run(this=0x00000001278043f0) at gtest.cc:3015:30 [opt]
frame #11: 0x0000000100079ff0 Rtsan-arm64-Test`testing::internal::UnitTestImpl::RunAllTests(this=0x0000000127804190) at gtest.cc:5920:44 [opt]
frame #12: 0x0000000100079a04 Rtsan-arm64-Test`bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) [inlined] bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(object=<unavailable>, method=(Rtsan-arm64-Test`testing::internal::UnitTestImpl::RunAllTests() at gtest.cc:5797), location="auxiliary test code (environments or event listeners)") at gtest.cc:2612:10 [opt]
frame #13: 0x00000001000799f4 Rtsan-arm64-Test`bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(object=0x0000000127804190, method=(Rtsan-arm64-Test`testing::internal::UnitTestImpl::RunAllTests() at gtest.cc:5797), location="auxiliary test code (environments or event listeners)") at gtest.cc:2648:14 [opt]
frame #14: 0x000000010007995c Rtsan-arm64-Test`testing::UnitTest::Run(this=0x00000001000b8668) at gtest.cc:5484:10 [opt]
frame #15: 0x000000010004cdd4 Rtsan-arm64-Test`main [inlined] RUN_ALL_TESTS() at gtest.h:2317:73 [opt]
frame #16: 0x000000010004cdcc Rtsan-arm64-Test`main(argc=1, argv=<unavailable>) at rtsan_test_main.cpp:36:10 [opt]
frame #17: 0x000000019e300274 dyld`start + 2840
(lldb) disassemble --frame
libcompiler_rt.dylib`__atomic_load:
-> 0x1ac241b18 <+0>: pacibsp
0x1ac241b1c <+4>: stp x22, x21, [sp, #-0x30]!
0x1ac241b20 <+8>: stp x20, x19, [sp, #0x10]
0x1ac241b24 <+12>: stp x29, x30, [sp, #0x20]
0x1ac241b28 <+16>: mov x19, x2
0x1ac241b2c <+20>: mov x20, x1
0x1ac241b30 <+24>: mov x21, x0
0x1ac241b34 <+28>: sub w16, w0, #0x1
0x1ac241b38 <+32>: cmp w16, #0x7
0x1ac241b3c <+36>: b.hi 0x1ac241bb0 ; <+152>
0x1ac241b40 <+40>: cmp x16, #0x7
0x1ac241b44 <+44>: csel x16, x16, xzr, ls
0x1ac241b48 <+48>: adrp x17, 0
0x1ac241b4c <+52>: add x17, x17, #0xc94 ; ___lldb_unnamed_symbol91
0x1ac241b50 <+56>: ldrsw x16, [x17, x16, lsl #2]
0x1ac241b54 <+60>: adr x17, 0x1ac241b54 ; <+60>
0x1ac241b58 <+64>: add x16, x17, x16
0x1ac241b5c <+68>: br x16
0x1ac241b60 <+72>: sub w8, w3, #0x1
0x1ac241b64 <+76>: cmp w8, #0x2
0x1ac241b68 <+80>: b.hs 0x1ac241c04 ; <+236>
0x1ac241b6c <+84>: ldaprb w8, [x20]
0x1ac241b70 <+88>: b 0x1ac241c3c ; <+292>
0x1ac241b74 <+92>: tbnz w20, #0x0, 0x1ac241bb0 ; <+152>
0x1ac241b78 <+96>: sub w8, w3, #0x1
0x1ac241b7c <+100>: cmp w8, #0x2
0x1ac241b80 <+104>: b.hs 0x1ac241c28 ; <+272>
0x1ac241b84 <+108>: ldaprh w8, [x20]
0x1ac241b88 <+112>: b 0x1ac241c68 ; <+336>
0x1ac241b8c <+116>: tst x20, #0x3
0x1ac241b90 <+120>: b.ne 0x1ac241bb0 ; <+152>
0x1ac241b94 <+124>: sub w8, w3, #0x1
0x1ac241b98 <+128>: cmp w8, #0x2
0x1ac241b9c <+132>: b.hs 0x1ac241c44 ; <+300>
0x1ac241ba0 <+136>: ldapr w8, [x20]
0x1ac241ba4 <+140>: b 0x1ac241c74 ; <+348>
0x1ac241ba8 <+144>: tst x20, #0x7
0x1ac241bac <+148>: b.eq 0x1ac241c14 ; <+252>
0x1ac241bb0 <+152>: ubfx x8, x20, #20, #12
0x1ac241bb4 <+156>: eor w8, w8, w20, lsr #4
0x1ac241bb8 <+160>: and x8, x8, #0x3ff
0x1ac241bbc <+164>: adrp x9, 367688
0x1ac241bc0 <+168>: add x9, x9, #0xdd8 ; locks
0x1ac241bc4 <+172>: add x22, x9, x8, lsl #2
0x1ac241bc8 <+176>: mov x0, x22
0x1ac241bcc <+180>: bl 0x1ac2448ac ; symbol stub for: _os_nospin_lock_lock
0x1ac241bd0 <+184>: sxtw x2, w21
0x1ac241bd4 <+188>: mov x0, x19
0x1ac241bd8 <+192>: mov x1, x20
0x1ac241bdc <+196>: bl 0x1ac2448dc ; symbol stub for: memcpy
0x1ac241be0 <+200>: mov x0, x22
0x1ac241be4 <+204>: ldp x29, x30, [sp, #0x20]
0x1ac241be8 <+208>: ldp x20, x19, [sp, #0x10]
0x1ac241bec <+212>: ldp x22, x21, [sp], #0x30
0x1ac241bf0 <+216>: autibsp
0x1ac241bf4 <+220>: eor x16, x30, x30, lsl #1
0x1ac241bf8 <+224>: tbz x16, #0x3e, 0x1ac241c00 ; <+232>
0x1ac241bfc <+228>: brk #0xc471
0x1ac241c00 <+232>: b 0x1ac2448bc ; symbol stub for: _os_nospin_lock_unlock
0x1ac241c04 <+236>: cmp w3, #0x5
0x1ac241c08 <+240>: b.ne 0x1ac241c38 ; <+288>
0x1ac241c0c <+244>: ldarb w8, [x20]
0x1ac241c10 <+248>: b 0x1ac241c3c ; <+292>
0x1ac241c14 <+252>: sub w8, w3, #0x1
0x1ac241c18 <+256>: cmp w8, #0x2
0x1ac241c1c <+260>: b.hs 0x1ac241c54 ; <+316>
0x1ac241c20 <+264>: ldapr x8, [x20]
0x1ac241c24 <+268>: b 0x1ac241c80 ; <+360>
0x1ac241c28 <+272>: cmp w3, #0x5
0x1ac241c2c <+276>: b.ne 0x1ac241c64 ; <+332>
0x1ac241c30 <+280>: ldarh w8, [x20]
0x1ac241c34 <+284>: b 0x1ac241c68 ; <+336>
0x1ac241c38 <+288>: ldrb w8, [x20]
0x1ac241c3c <+292>: strb w8, [x19]
0x1ac241c40 <+296>: b 0x1ac241c84 ; <+364>
0x1ac241c44 <+300>: cmp w3, #0x5
0x1ac241c48 <+304>: b.ne 0x1ac241c70 ; <+344>
0x1ac241c4c <+308>: ldar w8, [x20]
0x1ac241c50 <+312>: b 0x1ac241c74 ; <+348>
0x1ac241c54 <+316>: cmp w3, #0x5
0x1ac241c58 <+320>: b.ne 0x1ac241c7c ; <+356>
0x1ac241c5c <+324>: ldar x8, [x20]
0x1ac241c60 <+328>: b 0x1ac241c80 ; <+360>
0x1ac241c64 <+332>: ldrh w8, [x20]
0x1ac241c68 <+336>: strh w8, [x19]
0x1ac241c6c <+340>: b 0x1ac241c84 ; <+364>
0x1ac241c70 <+344>: ldr w8, [x20]
0x1ac241c74 <+348>: str w8, [x19]
0x1ac241c78 <+352>: b 0x1ac241c84 ; <+364>
0x1ac241c7c <+356>: ldr x8, [x20]
0x1ac241c80 <+360>: str x8, [x19]
0x1ac241c84 <+364>: ldp x29, x30, [sp, #0x20]
0x1ac241c88 <+368>: ldp x20, x19, [sp, #0x10]
0x1ac241c8c <+372>: ldp x22, x21, [sp], #0x30
0x1ac241c90 <+376>: retab
```
Check that out! The implementation is using `_os_nospin_lock_lock` as its locking mechanism. I've `git grep`ped the string `os_nospin` in the llvm-project repo and it's not present, so I can only presume that this is coming from the MacOS SDK.
So - here are my thoughts about a potential fix. I added an interceptor for `_os_nospin_lock_lock` (I had to forward-declare its argument type, because it's private to Darwin's [`os/lock.c` implementation](https://github.com/apple/darwin-libplatform/blob/215b09856ab5765b7462a91be7076183076600df/src/os/lock.c#L848)). Now the test passes on my machine. I'll raise a PR later today to show you and @cjappl, but here's the interceptor implementation for illustration in the meantime:
```cpp
struct _os_nospin_lock_t;
INTERCEPTOR(void, _os_nospin_lock_lock, _os_nospin_lock_t *lock) {
__rtsan_notify_intercepted_call("_os_nospin_lock_lock");
return REAL(_os_nospin_lock_lock)(lock);
}
```
Let me know what you think. I'm not sure whether this solution is particularly sustainable - what if the Darwin implementation switches to a different locking mechanism in future, for example? I guess that's perhaps just something we have to deal with. Anyway, I'm keen to hear your thoughts.
https://github.com/llvm/llvm-project/pull/129309
More information about the llvm-commits
mailing list