[compiler-rt] [asan] Print diagnostic if unlimited stack size detected (PR #133170)

[Thurston Dang via llvm-commits]

https://github.com/thurstond created https://github.com/llvm/llvm-project/pull/133170

This adds a diagnostic message if the stack size is unlimited. This would have simplified the diagnosis of
https://github.com/google/sanitizers/issues/856#issuecomment-2747076811; we anticipate this may help diagnose future issues too.

>From 1b64b0e80f5cbc4aea359deead2d864382978f20 Mon Sep 17 00:00:00 2001
From: Thurston Dang <thurston at google.com>
Date: Wed, 26 Mar 2025 22:10:20 +0000
Subject: [PATCH 1/2] [asan] Print diagnostic if unlimited stack size detected

This adds a diagnostic message if the stack size is unlimited. This
would have simplified the diagnosis of
https://github.com/google/sanitizers/issues/856#issuecomment-2747076811;
we anticipate this may help diagnose future issues too.
---
 compiler-rt/lib/asan/asan_rtl.cpp | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/compiler-rt/lib/asan/asan_rtl.cpp b/compiler-rt/lib/asan/asan_rtl.cpp
index 19c6c210b564c..36a6e38ae8eda 100644
--- a/compiler-rt/lib/asan/asan_rtl.cpp
+++ b/compiler-rt/lib/asan/asan_rtl.cpp
@@ -451,6 +451,17 @@ static bool AsanInitInternal() {
 
   DisableCoreDumperIfNecessary();
 
+#if SANITIZER_POSIX
+  if (StackSizeIsUnlimited()) {
+    VPrintf(1, "WARNING: Unlimited stack size detected. This may affect "
+               "compatibility with the shadow mappings.\n");
+    // MSan and TSan re-exec with a fixed size stack. We don't do that because
+    // it may break the program. InitializeShadowMemory() will, if needed,
+    // re-exec without ASLR, which solves most shadow mapping compatibility
+    // issues.
+  }
+#endif
+
   InitializeShadowMemory();
 
   AsanTSDInit(PlatformTSDDtor);

>From 9340645e899393cceeeb5afbe5f096c5cce64057 Mon Sep 17 00:00:00 2001
From: Thurston Dang <thurston at google.com>
Date: Wed, 26 Mar 2025 22:12:51 +0000
Subject: [PATCH 2/2] clang-format

---
 compiler-rt/lib/asan/asan_rtl.cpp | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/compiler-rt/lib/asan/asan_rtl.cpp b/compiler-rt/lib/asan/asan_rtl.cpp
index 36a6e38ae8eda..d98b350828448 100644
--- a/compiler-rt/lib/asan/asan_rtl.cpp
+++ b/compiler-rt/lib/asan/asan_rtl.cpp
@@ -453,8 +453,9 @@ static bool AsanInitInternal() {
 
 #if SANITIZER_POSIX
   if (StackSizeIsUnlimited()) {
-    VPrintf(1, "WARNING: Unlimited stack size detected. This may affect "
-               "compatibility with the shadow mappings.\n");
+    VPrintf(1,
+            "WARNING: Unlimited stack size detected. This may affect "
+            "compatibility with the shadow mappings.\n");
     // MSan and TSan re-exec with a fixed size stack. We don't do that because
     // it may break the program. InitializeShadowMemory() will, if needed,
     // re-exec without ASLR, which solves most shadow mapping compatibility