[compiler-rt] [compiler-rt] Avoid generating coredumps when piped to a tool (PR #83701)

[Vitaly Buka via llvm-commits]

================
@@ -103,9 +103,28 @@ static void setlim(int res, rlim_t lim) {
 }
 
 void DisableCoreDumperIfNecessary() {
+  rlimit rlim;
+  CHECK_EQ(0, getrlimit(RLIMIT_CORE, &rlim));
   if (common_flags()->disable_coredump) {
-    setlim(RLIMIT_CORE, 0);
+#  ifdef __linux__
+    // On Linux, if the kernel.core_pattern sysctl starts with a '|' (i.e. it
+    // is being piped to a coredump handler such as systemd-coredumpd), the
+    // kernel ignores RLIMIT_CORE (since we aren't creating a file in the file
+    // system) except for the magic value of 1, that disables coredumps when
+    // piping. 1 byte is also too small for any kind of valid core dump, so it
+    // also disables coredumps if kernel.core_pattern creates files directly.
+    rlim.rlim_cur = Min<rlim_t>(1, rlim.rlim_max);
+#  else
+    rlim.rlim_cur = 0;
+#  endif
+  } else {
+    // Set the limit to 1GB to avoid blocking the system for multiple minutes
+    // while dumping all (potentially huge) mappings.
+    // We are only setting the soft limit, so if it's really wanted, users can
+    // override it to generate a massive core dump.
+    rlim.rlim_cur = Min<rlim_t>(1024 * 1024 * 1024, rlim.rlim_max);
----------------
vitalybuka wrote:

I don't like this hidden change of behavior.
Is this a real problem?


https://github.com/llvm/llvm-project/pull/83701