[compiler-rt] [compiler-rt] Avoid generating coredumps when piped to a tool (PR #83701)

[Alexander Richardson via llvm-commits]

https://github.com/arichardson created https://github.com/llvm/llvm-project/pull/83701

I was trying to debug why `ninja check-compiler-rt` was taking so long
to run on my system and after some debugging it turned out that most of
the time was being spent generating core dumps.

On many current Linux systems, coredumps are no longer dumped in the CWD
but instead piped to a utility such as systemd-coredumpd that stores
them in a deterministic location. This can be done by setting the
kernel.core_pattern sysctl to start with a '|'. However, when using such
a setup the kernel ignores a coredump limit of 0 (since there is no file
being written) and we can end up piping many gigabytes of data to
systemd-coredumpd which causes the test suite to freeze for a long time.

Fortunately there is a workaround: the kernel recognizes the magic value
of 1 for RLIMIT_CORE to disable coredumps when piping. One byte is also
too small to generate any coredump, so it effectively behaves as if we
had set the value to zero.

Fixes: https://github.com/llvm/llvm-project/issues/45797


>From 8aabfa9dc6cbd38ad53e06f992e557ef9540a447 Mon Sep 17 00:00:00 2001
From: Alex Richardson <alexrichardson at google.com>
Date: Sat, 2 Mar 2024 19:05:55 -0800
Subject: [PATCH] =?UTF-8?q?[=F0=9D=98=80=F0=9D=97=BD=F0=9D=97=BF]=20initia?=
 =?UTF-8?q?l=20version?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Created using spr 1.3.4
---
 .../sanitizer_posix_libcdep.cpp               | 21 ++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp b/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp
index e88e654eec5a1c..7472a256870c16 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp
@@ -103,9 +103,28 @@ static void setlim(int res, rlim_t lim) {
 }
 
 void DisableCoreDumperIfNecessary() {
+  rlimit rlim;
+  CHECK_EQ(0, getrlimit(RLIMIT_CORE, &rlim));
   if (common_flags()->disable_coredump) {
-    setlim(RLIMIT_CORE, 0);
+#  ifdef __linux__
+    // On Linux, if the kernel.core_pattern sysctl starts with a '|' (i.e. it
+    // is being piped to a coredump handler such as systemd-coredumpd), the
+    // kernel ignores RLIMIT_CORE (since we aren't creating a file in the file
+    // system) except for the magic value of 1, that disables coredumps when
+    // piping. 1 byte is also too small for any kind of valid core dump, so it
+    // also disables coredumps if kernel.core_pattern creates files directly.
+    rlim.rlim_cur = Min<rlim_t>(1, rlim.rlim_max);
+#  else
+    rlim.rlim_cur = 0;
+#  endif
+  } else {
+    // Set the limit to 1GB to avoid blocking the system for multiple minutes
+    // while dumping all (potentially huge) mappings.
+    // We are only setting the soft limit, so if it's really wanted, users can
+    // override it to generate a massive core dump.
+    rlim.rlim_cur = Min<rlim_t>(1024 * 1024 * 1024, rlim.rlim_max);
   }
+  CHECK_EQ(0, setrlimit(RLIMIT_CORE, &rlim));
 }
 
 bool StackSizeIsUnlimited() {