[compiler-rt] [tsan] Lazily call 'personality' to minimize sandbox violations (PR #79334)

[Thurston Dang via llvm-commits]

https://github.com/thurstond updated https://github.com/llvm/llvm-project/pull/79334

>From a323748346a3e25918761eaaa785399a008be3ea Mon Sep 17 00:00:00 2001
From: Thurston Dang <thurston at google.com>
Date: Wed, 24 Jan 2024 17:27:20 +0000
Subject: [PATCH 1/3] [tsan] Lazily call 'personality' to minimize sandbox
 violations

My previous patch, "Re-exec TSan with no ASLR if memory layout is incompatible on Linux (#78351)" (0784b1eefa36d4acbb0dacd2d18796e26313b6c5) hoisted the 'personality' call, to
share the code between Android and non-Android Linux. Unfortunately, this eager call to
'personality' may trigger sandbox violations on non-Android Linux.

This patch fixes the issue by only calling 'personality' on non-Android Linux if the
memory mapping is incompatible. This may still cause a sandbox violation, but only if it
was going to abort anyway due to an incompatible memory mapping.

(The behavior on Android Linux is unchanged by this patch or the previous patch.)
---
 compiler-rt/lib/tsan/rtl/tsan_platform_linux.cpp | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/compiler-rt/lib/tsan/rtl/tsan_platform_linux.cpp b/compiler-rt/lib/tsan/rtl/tsan_platform_linux.cpp
index 0d0b1aba1f852a5..c723dba556ed2f7 100644
--- a/compiler-rt/lib/tsan/rtl/tsan_platform_linux.cpp
+++ b/compiler-rt/lib/tsan/rtl/tsan_platform_linux.cpp
@@ -244,12 +244,12 @@ static void ReExecIfNeeded() {
   }
 
 #    if SANITIZER_LINUX
+#      if SANITIZER_ANDROID && (defined(__aarch64__) || defined(__x86_64__))
   // ASLR personality check.
   int old_personality = personality(0xffffffff);
   bool aslr_on =
       (old_personality != -1) && ((old_personality & ADDR_NO_RANDOMIZE) == 0);
 
-#      if SANITIZER_ANDROID && (defined(__aarch64__) || defined(__x86_64__))
   // After patch "arm64: mm: support ARCH_MMAP_RND_BITS." is introduced in
   // linux kernel, the random gap between stack and mapped area is increased
   // from 128M to 36G on 39-bit aarch64. As it is almost impossible to cover
@@ -267,6 +267,14 @@ static void ReExecIfNeeded() {
   if (reexec) {
     // Don't check the address space since we're going to re-exec anyway.
   } else if (!CheckAndProtect(false, false, false)) {
+    // ASLR personality check.
+    // N.B. 'personality' is sometimes forbidden by sandboxes, so we only call
+    // this as a last resort (when the memory mapping is incompatible and TSan
+    // would fail anyway).
+    int old_personality = personality(0xffffffff);
+    bool aslr_on =
+        (old_personality != -1) && ((old_personality & ADDR_NO_RANDOMIZE) == 0);
+
     if (aslr_on) {
       // Disable ASLR if the memory layout was incompatible.
       // Alternatively, we could just keep re-execing until we get lucky

>From 112d328f8b02ac5019e25af62ab01e7f39ae811b Mon Sep 17 00:00:00 2001
From: Thurston Dang <thurston at google.com>
Date: Wed, 24 Jan 2024 18:47:23 +0000
Subject: [PATCH 2/3] Add sandbox_forbidden_functions.cpp test

---
 .../Linux/sandbox_forbidden_functions.cpp     | 21 +++++++++++++++++++
 1 file changed, 21 insertions(+)
 create mode 100644 compiler-rt/test/sanitizer_common/TestCases/Linux/sandbox_forbidden_functions.cpp

diff --git a/compiler-rt/test/sanitizer_common/TestCases/Linux/sandbox_forbidden_functions.cpp b/compiler-rt/test/sanitizer_common/TestCases/Linux/sandbox_forbidden_functions.cpp
new file mode 100644
index 000000000000000..64c82fdf30ba5ce
--- /dev/null
+++ b/compiler-rt/test/sanitizer_common/TestCases/Linux/sandbox_forbidden_functions.cpp
@@ -0,0 +1,21 @@
+// RUN: %clangxx %s -o %t && %run %t 2>&1 | FileCheck %s
+
+// REQUIRES: target-x86 || target-x86_64
+
+#include <stdio.h>
+#include <stdlib.h>
+
+// Functions that sandboxes don't like. If a sanitizer calls it, this test will
+// likely fail. (There will be a false negative if the sanitizer only calls it
+// during an obscure code path that is not exercised by this test.)
+//
+// Known false positive: TSan with high-entropy ASLR (in a non-sandboxed
+//                       environment)
+extern "C" int personality(unsigned long) { abort(); }
+
+int main(int argc, char **argv) {
+  printf("Hello World!\n");
+  return 0;
+}
+
+// CHECK: Hello World!

>From 0b4c8464b5045e80eccd9acc11e7312d6caa3c88 Mon Sep 17 00:00:00 2001
From: Thurston Dang <thurston at google.com>
Date: Thu, 25 Jan 2024 18:26:14 +0000
Subject: [PATCH 3/3] Undo 'sandbox_forbidden_functions.cpp'

---
 .../Linux/sandbox_forbidden_functions.cpp     | 21 -------------------
 1 file changed, 21 deletions(-)
 delete mode 100644 compiler-rt/test/sanitizer_common/TestCases/Linux/sandbox_forbidden_functions.cpp

diff --git a/compiler-rt/test/sanitizer_common/TestCases/Linux/sandbox_forbidden_functions.cpp b/compiler-rt/test/sanitizer_common/TestCases/Linux/sandbox_forbidden_functions.cpp
deleted file mode 100644
index 64c82fdf30ba5ce..000000000000000
--- a/compiler-rt/test/sanitizer_common/TestCases/Linux/sandbox_forbidden_functions.cpp
+++ /dev/null
@@ -1,21 +0,0 @@
-// RUN: %clangxx %s -o %t && %run %t 2>&1 | FileCheck %s
-
-// REQUIRES: target-x86 || target-x86_64
-
-#include <stdio.h>
-#include <stdlib.h>
-
-// Functions that sandboxes don't like. If a sanitizer calls it, this test will
-// likely fail. (There will be a false negative if the sanitizer only calls it
-// during an obscure code path that is not exercised by this test.)
-//
-// Known false positive: TSan with high-entropy ASLR (in a non-sandboxed
-//                       environment)
-extern "C" int personality(unsigned long) { abort(); }
-
-int main(int argc, char **argv) {
-  printf("Hello World!\n");
-  return 0;
-}
-
-// CHECK: Hello World!