[llvm] [llvm][Support][Memory] Add memfd based fallback for strict W^X Linux systems (PR #98538)

[via llvm-commits]

================
@@ -251,5 +320,50 @@ void Memory::InvalidateInstructionCache(const void *Addr, size_t Len) {
   ValgrindDiscardTranslations(Addr, Len);
 }
 
+static inline bool isPermissionError(int err) {
+  // PaX uses EPERM, SELinux uses EACCES
+  return err == EPERM || err == EACCES;
+}
+
+bool Memory::execProtectionChangeNeedsNewMapping() {
+#if defined(__linux__)
+  static int status = -1;
+
+  if (status != -1)
+    return status;
+
+  // Try to get the status from /proc/self/status, looking for PaX flags.
+  if (auto file = MemoryBuffer::getFileAsStream("/proc/self/status")) {
+    auto pax_flags =
+        (*file)->getBuffer().rsplit("PaX:").second.split('\n').first.trim();
+    if (!pax_flags.empty())
+      // 'M' indicates MPROTECT is enabled
+      return (status = pax_flags.find('M') != StringRef::npos);
+  }
+
+  // Create a temporary writable mapping and try to make it executable.  If
+  // this fails, test 'errno' to ensure it failed because we were not allowed
+  // to create such a mapping and not because of some transient error.
+  size_t size = Process::getPageSizeEstimate();
+  void *addr = ::mmap(NULL, size, PROT_READ | PROT_WRITE,
+                      MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+  if (addr == MAP_FAILED) {
+    // Must be low on memory or have too many mappings already, not much we can
+    // do here.
+    status = 0;
+  } else {
+    if (::mprotect(addr, size, PROT_READ | PROT_EXEC) < 0)
+      status = isPermissionError(errno);
----------------
minipli-oss wrote:

Oh, I wasn't aware that there's a mode where an "execmem" denial would kill the involved process. During my tests on Fedora I only observed failing syscalls returning `-EACCESS` (this very `mprotect()` call).

While I'd like to have a better detection of SELinux's "deny_execmem", there are basically two reasons against it:

1. If this sequence of syscalls (`addr = mmap(RW); mprotect(addr, RX)`) ends up killing the process instead of only making the second syscall fail, it probably shouldn't attempt to make use of a JIT anyways -- with a host configuration that strict.
2. Linking against `libselinux` by default just for this test is a deal breaker as we (a) shouldn't require all users of `libLLVM` to link against `libselinux` (well, technically we could but) and (b) overriding library internal state -- even if only briefly -- will mess with the application also trying to make use of `libselinux` and that'll lead to unintentional misbehavior for applications linking against `libLLVM`. A library shouldn't do that.

We can, however, add a mode (build time option) where we do the specific checks for distributions that do care and want to have such a narrow check. It should, however, try to make use of `libselinux` without changing any of its internal state.

https://github.com/llvm/llvm-project/pull/98538