[compiler-rt] [sanitizer] Suggest checking ulimit -d in addition to ulimit -v (PR #98625)

[Ilya Leoshkevich via llvm-commits]

https://github.com/iii-i created https://github.com/llvm/llvm-project/pull/98625

Since Linux 4.7, RLIMIT_DATA may result in mmap() returning ENOMEM. Example:

    $ clang -fsanitize=address -o hello hello.c
    $ ulimit -d 100000
    $ ./hello
    ==3349007==ERROR: AddressSanitizer failed to allocate 0x10000000 (268435456) bytes at address 7fff7000 (errno: 12)
    ==3349007==ReserveShadowMemoryRange failed while trying to map 0x10000000 bytes. Perhaps you're using ulimit -v

Suggest checking ulimit -d in addition to ulimit -v.

>From 069eb2c8d482a81fce7ce724dfcdc4be8999cb41 Mon Sep 17 00:00:00 2001
From: Ilya Leoshkevich <iii at linux.ibm.com>
Date: Fri, 12 Jul 2024 14:11:52 +0200
Subject: [PATCH] [sanitizer] Suggest checking ulimit -d in addition to ulimit
 -v

Since Linux 4.7, RLIMIT_DATA may result in mmap() returning ENOMEM.
Example:

    $ clang -fsanitize=address -o hello hello.c
    $ ulimit -d 100000
    $ ./hello
    ==3349007==ERROR: AddressSanitizer failed to allocate 0x10000000 (268435456) bytes at address 7fff7000 (errno: 12)
    ==3349007==ReserveShadowMemoryRange failed while trying to map 0x10000000 bytes. Perhaps you're using ulimit -v

Suggest checking ulimit -d in addition to ulimit -v.
---
 compiler-rt/lib/sanitizer_common/sanitizer_common_libcdep.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_common_libcdep.cpp b/compiler-rt/lib/sanitizer_common/sanitizer_common_libcdep.cpp
index a174ae7be991d..3966d82d51ee9 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_common_libcdep.cpp
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_common_libcdep.cpp
@@ -169,7 +169,7 @@ void ReserveShadowMemoryRange(uptr beg, uptr end, const char *name,
                      : !MmapFixedNoReserve(beg, size, name)) {
     Report(
         "ReserveShadowMemoryRange failed while trying to map 0x%zx bytes. "
-        "Perhaps you're using ulimit -v\n",
+        "Perhaps you're using ulimit -v or ulimit -d\n",
         size);
     Abort();
   }