[llvm-branch-commits] [clang] [release/23.x][clang][analyzer] Fix false positive in StdLibraryFunctionsChecker for mmap with MAP_ANON (PR #220333)

[Daniel M. Zimmerman via llvm-branch-commits]

https://github.com/dmzimmerman created https://github.com/llvm/llvm-project/pull/220333

Cherry-pick of #219568 (commit af6ae639c296e3ab141b0ee755eb669945411810) to `release/23.x`. This fixes a false positive in the `unix.StdCLibraryFunctions` checker where `mmap` calls using `MAP_ANON` with `VM_MAKE_TAG` on Darwin are incorrectly flagged as having invalid `fd` values. It is a bug fix with no regression risk; it relaxes an `fd` range check from the static analyzer on Darwin and leaves all other platforms unchanged.

>From 4a347bd863e11eaac6117ada66c2275371c0ea92 Mon Sep 17 00:00:00 2001
From: "Daniel M. Zimmerman" <dmz at acm.org>
Date: Tue, 1 Sep 2026 02:52:11 -0700
Subject: [PATCH] [clang][analyzer] Fix false positive in
 StdLibraryFunctionsChecker for mmap with MAP_ANON (#219568)

On Darwin,`mmap(2)` acccepts a Mach VM tag encoded in the fd argument
when `MAP_ANON` is set; the encoding is done using `VM_MAKE_TAG(tag)`,
which expands to `(tag << 24)` and is therefore a large negative signed
value for tags >= 128.

This causes a false positive with the existing constraint, which
restricts `fd` to be >= 1, when code uses a `VM_MAKE_TAG` value as the
`fd` argument. The fix here is to eliminate the false positive by
omitting the `fd` constraint on Darwin targets, because it can't be
expressed as a simple range. This does give rise to false negatives (any
`fd` < -1 on Darwin when `MAP_ANON` is not set), but any code with such
a false negative would crash immediately when trying to use the invalid
file descriptor, rather than exhibiting some more subtle dangerous
behavior.

AI disclosure: I used Claude Sonnet 4.6 to help diagnose the original
false positive and suggest possible fixes.

rdar://185124909
---
 .../Checkers/StdLibraryFunctionsChecker.cpp   | 28 ++++++++++++-----
 .../test/Analysis/stdlibraryfunction-darwin.c | 31 +++++++++++++++++++
 2 files changed, 52 insertions(+), 7 deletions(-)
 create mode 100644 clang/test/Analysis/stdlibraryfunction-darwin.c

diff --git a/clang/lib/StaticAnalyzer/Checkers/StdLibraryFunctionsChecker.cpp b/clang/lib/StaticAnalyzer/Checkers/StdLibraryFunctionsChecker.cpp
index 4fe3e1f7623f6..d2c6f432f08a9 100644
--- a/clang/lib/StaticAnalyzer/Checkers/StdLibraryFunctionsChecker.cpp
+++ b/clang/lib/StaticAnalyzer/Checkers/StdLibraryFunctionsChecker.cpp
@@ -41,6 +41,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "ErrnoModeling.h"
+#include "clang/Basic/TargetInfo.h"
 #include "clang/StaticAnalyzer/Checkers/BuiltinCheckerRegistration.h"
 #include "clang/StaticAnalyzer/Core/BugReporter/BugType.h"
 #include "clang/StaticAnalyzer/Core/Checker.h"
@@ -2954,15 +2955,28 @@ void StdLibraryFunctionsChecker::initFunctionSummaries(
     // void *mmap(void *addr, size_t length, int prot, int flags, int fd,
     // off_t offset);
     // FIXME: Improve for errno modeling.
-    addToFunctionSummaryMap(
-        "mmap",
-        Signature(
-            ArgTypes{VoidPtrTy, SizeTyCanonTy, IntTy, IntTy, IntTy, Off_tTy},
-            RetType{VoidPtrTy}),
+    auto MmapSignature = Signature(
+        ArgTypes{VoidPtrTy, SizeTyCanonTy, IntTy, IntTy, IntTy, Off_tTy},
+        RetType{VoidPtrTy});
+    auto MmapSummaryWithLengthConstraint =
         Summary(NoEvalCall)
-            .ArgConstraint(ArgumentCondition(1, WithinRange, Range(1, SizeMax)))
             .ArgConstraint(
-                ArgumentCondition(4, WithinRange, Range(-1, IntMax))));
+                ArgumentCondition(1, WithinRange, Range(1, SizeMax)));
+
+    if (ACtx.getTargetInfo().getTriple().isOSDarwin()) {
+      // On Darwin, MAP_ANON + VM_MAKE_TAG(tag) uses argument 4 (len) for
+      // the tag, which looks like a large negative signed integer.
+      // The valid range for fd is not expressible as a simple union of
+      // ranges so we only constrain the length parameter.
+      addToFunctionSummaryMap("mmap", MmapSignature,
+                              MmapSummaryWithLengthConstraint);
+    } else {
+      // On other platforms, we also constrain the fd parameter (-1 <= fd).
+      addToFunctionSummaryMap(
+          "mmap", MmapSignature,
+          MmapSummaryWithLengthConstraint.ArgConstraint(
+              ArgumentCondition(4, WithinRange, Range(-1, IntMax))));
+    }
 
     std::optional<QualType> Off64_tTy = lookupTy("off64_t");
     // void *mmap64(void *addr, size_t length, int prot, int flags, int fd,
diff --git a/clang/test/Analysis/stdlibraryfunction-darwin.c b/clang/test/Analysis/stdlibraryfunction-darwin.c
new file mode 100644
index 0000000000000..50ed68e6325de
--- /dev/null
+++ b/clang/test/Analysis/stdlibraryfunction-darwin.c
@@ -0,0 +1,31 @@
+// DEFINE: %{analyze} = %clang_analyze_cc1 \
+// DEFINE:   -analyzer-checker=core,unix.StdCLibraryFunctions \
+// DEFINE:   -analyzer-config unix.StdCLibraryFunctions:ModelPOSIX=true
+
+// RUN: %{analyze} -triple arm64-apple-darwin -verify=darwin %s
+// RUN: %{analyze} -triple x86_64-unknown-linux-gnu -verify=linux %s
+
+typedef unsigned long size_t;
+typedef long off_t;
+void *mmap(void *, size_t, int, int, int, off_t);
+
+#define MAP_PRIVATE 0x0002
+#define MAP_ANON    0x1000
+
+// VM_MAKE_TAG on Darwin encodes a Mach VM memory tag in the top 8 bits.
+// For tags >= 128 the result is a large negative signed integer.
+#define VM_MAKE_TAG(tag) ((int)((unsigned)(tag) << 24))
+#define VM_MEMORY_APPLICATION_SPECIFIC_1 240
+
+void test_mmap_vm_make_tag(void) {
+  // darwin-no-warning: no bound restriction on fd parameter on Darwin
+  // linux-warning at +1 {{The 5th argument to 'mmap' is -268435456 but should be >= -1}}
+  void *p = mmap(0, 4096, 0, MAP_ANON | MAP_PRIVATE,
+                 VM_MAKE_TAG(VM_MEMORY_APPLICATION_SPECIFIC_1), 0);
+}
+
+void test_mmap_size_constraint(void) {
+  void *p = mmap(0, 0, 0, MAP_ANON | MAP_PRIVATE, -1, 0);
+  // darwin-warning at -1 {{The 2nd argument to 'mmap' is 0 but should be > 0}}
+  // linux-warning at -2  {{The 2nd argument to 'mmap' is 0 but should be > 0}}
+}