[clang] [NFC][analyzer] Refactor pointer escape in MallocChecker part 1 (PR #238368)
DonĂ¡t Nagy via cfe-commits
cfe-commits at lists.llvm.org
Tue Nov 3 06:49:00 PST 2026
https://github.com/NagyDonat updated https://github.com/llvm/llvm-project/pull/238368
>From 495c9adeb87c681f3cbea4989ae41a8124cb7cde Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Don=C3=A1t=20Nagy?= <donat.nagy at ericsson.com>
Date: Thu, 29 Oct 2026 15:57:09 +0100
Subject: [PATCH 1/8] Correct a lie in a comment
---
clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp b/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
index 11e5b5337f5d4..653ebf8211532 100644
--- a/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
+++ b/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
@@ -819,7 +819,7 @@ class MallocChecker
/// "interesting" and should be modeled explicitly.
///
/// \param [out] EscapingSymbol A function might not free memory in general,
- /// but could be known to free a particular symbol. In this case, false is
+ /// but could be known to free a particular symbol. In this case, true is
/// returned and the single escaping symbol is returned through the out
/// parameter.
///
@@ -3778,7 +3778,7 @@ bool MallocChecker::mayFreeAnyEscapedMemoryOrIsModeledExplicitly(
// after the call.
if (Msg->getMethodFamily() == OMF_init) {
EscapingSymbol = Msg->getReceiverSVal().getAsSymbol();
- return true;
+ return true; // (temporary comment to show this line in diff)
}
// Otherwise, assume that the method does not free memory.
>From 00dee0e65a1d5fb62d7a4107fda3b8f271b4cb65 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Don=C3=A1t=20Nagy?= <donat.nagy at ericsson.com>
Date: Thu, 29 Oct 2026 14:28:20 +0100
Subject: [PATCH 2/8] Rename and negate
mayFreeAnyEscapedMemoryOrIsModeledExplicitly
`MallocChecker` had a helper method with the terribly long name
`mayFreeAnyEscapedMemoryOrIsModeledExplicitly`, which was also
incorrect: the accurate (but terribly long) name would have been
`mayFreeAnyEscapedMemoryAndIsNotModeledExplicitly`.
The only call site of this method negated its return value, which
probably contributed to the misunderstandings.
This commit cleans up the situation by bringing the negtion into the
function (swapping `return true;` and `return false;`) and choosing a
shorter name that fits the role of this method.
---
.../StaticAnalyzer/Checkers/MallocChecker.cpp | 74 +++++++++----------
1 file changed, 36 insertions(+), 38 deletions(-)
diff --git a/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp b/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
index 653ebf8211532..68d7dcfae5c41 100644
--- a/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
+++ b/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
@@ -815,19 +815,20 @@ class MallocChecker
void checkUseZeroAllocated(SymbolRef Sym, CheckerContext &C,
const Stmt *S) const;
- /// Check if the function is known to free memory, or if it is
- /// "interesting" and should be modeled explicitly.
+ /// Return true if this checker can continue to follow the state of pointers
+ /// that are passed directly (as arguments) to the opaque call \p Call.
+ /// By default a pointer passed to an opaque call would be marked as
+ /// "escaped" (i.e. using it, releasing it and not releasing it are all
+ /// accepted as potentially valid) but for many system functions either:
+ /// (1) we can assume that it can't release the memory; or
+ /// (2) it will be explicitly modeled, so the generic "escape" is unwanted.
///
/// \param [out] EscapingSymbol A function might not free memory in general,
- /// but could be known to free a particular symbol. In this case, true is
+ /// but could be known to free a particular symbol. In this case, false is
/// returned and the single escaping symbol is returned through the out
/// parameter.
- ///
- /// We assume that pointers do not escape through calls to system functions
- /// not handled by this checker.
- bool mayFreeAnyEscapedMemoryOrIsModeledExplicitly(const CallEvent *Call,
- ProgramStateRef State,
- SymbolRef &EscapingSymbol) const;
+ bool ignoreDirectEscapeTo(const CallEvent *Call, ProgramStateRef State,
+ SymbolRef &EscapingSymbol) const;
/// Implementation of the checkPointerEscape callbacks.
[[nodiscard]] ProgramStateRef
@@ -3721,10 +3722,9 @@ ProgramStateRef MallocChecker::evalAssume(ProgramStateRef state,
return state;
}
-bool MallocChecker::mayFreeAnyEscapedMemoryOrIsModeledExplicitly(
- const CallEvent *Call,
- ProgramStateRef State,
- SymbolRef &EscapingSymbol) const {
+bool MallocChecker::ignoreDirectEscapeTo(const CallEvent *Call,
+ ProgramStateRef State,
+ SymbolRef &EscapingSymbol) const {
assert(Call);
EscapingSymbol = nullptr;
@@ -3733,26 +3733,26 @@ bool MallocChecker::mayFreeAnyEscapedMemoryOrIsModeledExplicitly(
// regions escape to C++ containers. They seem to do that even now, but for
// mysterious reasons.
if (!isa<SimpleFunctionCall, ObjCMethodCall>(Call))
- return true;
+ return false;
// Check Objective-C messages by selector name.
if (const ObjCMethodCall *Msg = dyn_cast<ObjCMethodCall>(Call)) {
// If it's not a framework call, or if it takes a callback, assume it
// can free memory.
if (!Call->isInSystemHeader() || Call->argumentsMayEscape())
- return true;
+ return false;
// If it's a method we know about, handle it explicitly post-call.
// This should happen before the "freeWhenDone" check below.
if (isKnownDeallocObjCMethodName(*Msg))
- return false;
+ return true;
// If there's a "freeWhenDone" parameter, but the method isn't one we know
// about, we can't be sure that the object will use free() to deallocate the
// memory, so we can't model it explicitly. The best we can do is use it to
// decide whether the pointer escapes.
if (std::optional<bool> FreeWhenDone = getFreeWhenDoneArg(*Msg))
- return *FreeWhenDone;
+ return !*FreeWhenDone;
// If the first selector piece ends with "NoCopy", and there is no
// "freeWhenDone" parameter set to zero, we know ownership is being
@@ -3760,7 +3760,7 @@ bool MallocChecker::mayFreeAnyEscapedMemoryOrIsModeledExplicitly(
// free() to deallocate the memory, so we can't model it explicitly.
StringRef FirstSlot = Msg->getSelector().getNameForSlot(0);
if (FirstSlot.ends_with("NoCopy"))
- return true;
+ return false;
// If the first selector starts with addPointer, insertPointer,
// or replacePointer, assume we are dealing with NSPointerArray or similar.
@@ -3770,7 +3770,7 @@ bool MallocChecker::mayFreeAnyEscapedMemoryOrIsModeledExplicitly(
FirstSlot.starts_with("insertPointer") ||
FirstSlot.starts_with("replacePointer") ||
FirstSlot == "valueWithPointer") {
- return true;
+ return false;
}
// We should escape receiver on call to 'init'. This is especially relevant
@@ -3778,32 +3778,32 @@ bool MallocChecker::mayFreeAnyEscapedMemoryOrIsModeledExplicitly(
// after the call.
if (Msg->getMethodFamily() == OMF_init) {
EscapingSymbol = Msg->getReceiverSVal().getAsSymbol();
- return true; // (temporary comment to show this line in diff)
+ return false;
}
// Otherwise, assume that the method does not free memory.
// Most framework methods do not free memory.
- return false;
+ return true;
}
// At this point the only thing left to handle is straight function calls.
const FunctionDecl *FD = cast<SimpleFunctionCall>(Call)->getDecl();
if (!FD)
- return true;
+ return false;
// If it's one of the allocation functions we can reason about, we model
// its behavior explicitly.
if (isMemCall(*Call))
- return false;
+ return true;
// If it's not a system call, assume it frees memory.
if (!Call->isInSystemHeader())
- return true;
+ return false;
// White list the system functions whose arguments escape.
const IdentifierInfo *II = FD->getIdentifier();
if (!II)
- return true;
+ return false;
StringRef FName = II->getName();
// White list the 'XXXNoCopy' CoreFoundation functions.
@@ -3817,10 +3817,10 @@ bool MallocChecker::mayFreeAnyEscapedMemoryOrIsModeledExplicitly(
if (const DeclRefExpr *DE = dyn_cast<DeclRefExpr>(ArgE)) {
StringRef DeallocatorName = DE->getFoundDecl()->getName();
if (DeallocatorName == "kCFAllocatorNull")
- return false;
+ return true;
}
}
- return true;
+ return false;
}
// Associating streams with malloced buffers. The pointer can escape if
@@ -3841,7 +3841,7 @@ bool MallocChecker::mayFreeAnyEscapedMemoryOrIsModeledExplicitly(
if (const DeclRefExpr *ArgDRE = dyn_cast<DeclRefExpr>(ArgE))
if (const VarDecl *D = dyn_cast<VarDecl>(ArgDRE->getDecl()))
if (D->getCanonicalDecl()->getName().contains("std"))
- return true;
+ return false;
}
}
@@ -3855,22 +3855,22 @@ bool MallocChecker::mayFreeAnyEscapedMemoryOrIsModeledExplicitly(
FName == "CVPixelBufferCreateWithBytes" ||
FName == "CVPixelBufferCreateWithPlanarBytes" ||
FName == "OSAtomicEnqueue") {
- return true;
+ return false;
}
if (FName == "postEvent" &&
FD->getQualifiedNameAsString() == "QCoreApplication::postEvent") {
- return true;
+ return false;
}
if (FName == "connectImpl" &&
FD->getQualifiedNameAsString() == "QObject::connectImpl") {
- return true;
+ return false;
}
if (FName == "singleShotImpl" &&
FD->getQualifiedNameAsString() == "QTimer::singleShotImpl") {
- return true;
+ return false;
}
// Protobuf function declared in `generated_message_util.h` that takes
@@ -3879,7 +3879,7 @@ bool MallocChecker::mayFreeAnyEscapedMemoryOrIsModeledExplicitly(
// to set `EscapingSymbol`. (Also, this is an implementation detail of
// Protobuf, so it's better to be a bit more permissive.)
if (FName == "GetOwnedMessageInternal") {
- return true;
+ return false;
}
// Handle cases where we know a buffer's /address/ can escape.
@@ -3887,11 +3887,11 @@ bool MallocChecker::mayFreeAnyEscapedMemoryOrIsModeledExplicitly(
// even though the address escapes, it's still our responsibility to free the
// buffer.
if (Call->argumentsMayEscape())
- return true;
+ return false;
// Otherwise, assume that the function does not free memory.
// Most system calls do not free the memory.
- return false;
+ return true;
}
ProgramStateRef MallocChecker::checkPointerEscape(ProgramStateRef State,
@@ -3924,9 +3924,7 @@ ProgramStateRef MallocChecker::checkPointerEscapeAux(
// call later, keep tracking the top level arguments.
SymbolRef EscapingSymbol = nullptr;
if (Kind == PSK_DirectEscapeOnCall &&
- !mayFreeAnyEscapedMemoryOrIsModeledExplicitly(Call, State,
- EscapingSymbol) &&
- !EscapingSymbol) {
+ ignoreDirectEscapeTo(Call, State, EscapingSymbol) && !EscapingSymbol) {
return State;
}
>From ba4c93c4238cd6af684d794cfacbe61663749df7 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Don=C3=A1t=20Nagy?= <donat.nagy at ericsson.com>
Date: Thu, 29 Oct 2026 16:02:49 +0100
Subject: [PATCH 3/8] Remove redundant check for '!EscapingSymbol'
`EscapingSymbol` is usually `nullptr` and then `&& true` does not change
anything; in the only case when `EscapingSymbol` is non-null, the method
`ignoreDirectEscapeTo` returns false, so this `!EscapingSymbol` is not
reached at all.
---
clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp b/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
index 68d7dcfae5c41..f1010e246a7b9 100644
--- a/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
+++ b/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
@@ -3924,7 +3924,7 @@ ProgramStateRef MallocChecker::checkPointerEscapeAux(
// call later, keep tracking the top level arguments.
SymbolRef EscapingSymbol = nullptr;
if (Kind == PSK_DirectEscapeOnCall &&
- ignoreDirectEscapeTo(Call, State, EscapingSymbol) && !EscapingSymbol) {
+ ignoreDirectEscapeTo(Call, State, EscapingSymbol)) {
return State;
}
>From 30165a07cb39a952f8429b1bac009e9923936a55 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Don=C3=A1t=20Nagy?= <donat.nagy at ericsson.com>
Date: Fri, 30 Oct 2026 22:45:55 +0100
Subject: [PATCH 4/8] Test `isMemCall(*Call)` before nullness of getDecl()
This does not change behavior because `isMemCall` always returns false
when the virtual method `getDecl()` returns null. (There is an early
return in `CallDescription::matches` and another check directly in
`isMemCall`.)
This is one step in a gradual untangling of this logic.
---
clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp b/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
index f1010e246a7b9..2c8639c965769 100644
--- a/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
+++ b/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
@@ -3786,16 +3786,16 @@ bool MallocChecker::ignoreDirectEscapeTo(const CallEvent *Call,
return true;
}
- // At this point the only thing left to handle is straight function calls.
- const FunctionDecl *FD = cast<SimpleFunctionCall>(Call)->getDecl();
- if (!FD)
- return false;
-
// If it's one of the allocation functions we can reason about, we model
// its behavior explicitly.
if (isMemCall(*Call))
return true;
+ // At this point the only thing left to handle is straight function calls.
+ const FunctionDecl *FD = cast<SimpleFunctionCall>(Call)->getDecl();
+ if (!FD)
+ return false;
+
// If it's not a system call, assume it frees memory.
if (!Call->isInSystemHeader())
return false;
>From 606cfa6e31cf2eb2300330b7fac59242799a8d32 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Don=C3=A1t=20Nagy?= <donat.nagy at ericsson.com>
Date: Fri, 30 Oct 2026 23:10:28 +0100
Subject: [PATCH 5/8] Hoist the isMemCall check from ignoreDirectEscapeTo
This was an early "return true;" that returned `true` for a different
reason than the rest of the (very long) function. Handling it separately
makes the code easier to understand.
This also prepares the ground for hoisting more stuff from this method.
---
.../StaticAnalyzer/Checkers/MallocChecker.cpp | 24 +++++++++----------
1 file changed, 11 insertions(+), 13 deletions(-)
diff --git a/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp b/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
index 2c8639c965769..a1747a98128f0 100644
--- a/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
+++ b/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
@@ -819,9 +819,8 @@ class MallocChecker
/// that are passed directly (as arguments) to the opaque call \p Call.
/// By default a pointer passed to an opaque call would be marked as
/// "escaped" (i.e. using it, releasing it and not releasing it are all
- /// accepted as potentially valid) but for many system functions either:
- /// (1) we can assume that it can't release the memory; or
- /// (2) it will be explicitly modeled, so the generic "escape" is unwanted.
+ /// accepted as potentially valid) but if the called function is from a
+ /// system header we can often see that it wont't release the memory.
///
/// \param [out] EscapingSymbol A function might not free memory in general,
/// but could be known to free a particular symbol. In this case, false is
@@ -3786,11 +3785,6 @@ bool MallocChecker::ignoreDirectEscapeTo(const CallEvent *Call,
return true;
}
- // If it's one of the allocation functions we can reason about, we model
- // its behavior explicitly.
- if (isMemCall(*Call))
- return true;
-
// At this point the only thing left to handle is straight function calls.
const FunctionDecl *FD = cast<SimpleFunctionCall>(Call)->getDecl();
if (!FD)
@@ -3920,12 +3914,16 @@ ProgramStateRef MallocChecker::checkPointerEscapeAux(
ProgramStateRef State, const InvalidatedSymbols &Escaped,
const CallEvent *Call, PointerEscapeKind Kind,
bool IsConstPointerEscape) const {
- // If we know that the call does not free memory, or we want to process the
- // call later, keep tracking the top level arguments.
SymbolRef EscapingSymbol = nullptr;
- if (Kind == PSK_DirectEscapeOnCall &&
- ignoreDirectEscapeTo(Call, State, EscapingSymbol)) {
- return State;
+ if (Kind == PSK_DirectEscapeOnCall) {
+ // If we want to model this call explicitly, don't discard state.
+ // FIXME: Maybe extend this to non-direct escape and/or non-simple calls
+ // (e.g. 'isMemCall' is true for a C++ method with ownership attributes)?
+ if (isa<SimpleFunctionCall>(Call) && isMemCall(*Call))
+ return State;
+ // If we know that the call cannot release memory, don't discard state.
+ if (ignoreDirectEscapeTo(Call, State, EscapingSymbol))
+ return State;
}
for (SymbolRef sym : Escaped) {
>From 35002cc96ca35d2a075408e43060a51191d68da8 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Don=C3=A1t=20Nagy?= <donat.nagy at ericsson.com>
Date: Tue, 3 Nov 2026 11:51:06 +0100
Subject: [PATCH 6/8] Remove unused parameter 'State'
---
clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp b/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
index a1747a98128f0..2a2fdd599f866 100644
--- a/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
+++ b/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
@@ -826,7 +826,7 @@ class MallocChecker
/// but could be known to free a particular symbol. In this case, false is
/// returned and the single escaping symbol is returned through the out
/// parameter.
- bool ignoreDirectEscapeTo(const CallEvent *Call, ProgramStateRef State,
+ bool ignoreDirectEscapeTo(const CallEvent *Call,
SymbolRef &EscapingSymbol) const;
/// Implementation of the checkPointerEscape callbacks.
@@ -3722,7 +3722,6 @@ ProgramStateRef MallocChecker::evalAssume(ProgramStateRef state,
}
bool MallocChecker::ignoreDirectEscapeTo(const CallEvent *Call,
- ProgramStateRef State,
SymbolRef &EscapingSymbol) const {
assert(Call);
EscapingSymbol = nullptr;
@@ -3922,7 +3921,7 @@ ProgramStateRef MallocChecker::checkPointerEscapeAux(
if (isa<SimpleFunctionCall>(Call) && isMemCall(*Call))
return State;
// If we know that the call cannot release memory, don't discard state.
- if (ignoreDirectEscapeTo(Call, State, EscapingSymbol))
+ if (ignoreDirectEscapeTo(Call, EscapingSymbol))
return State;
}
>From ba66db30dc2efe94964367e1353b0d9aea817c41 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Don=C3=A1t=20Nagy?= <donat.nagy at ericsson.com>
Date: Tue, 3 Nov 2026 15:16:58 +0100
Subject: [PATCH 7/8] Tweak a comment
---
clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp b/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
index 2a2fdd599f866..7795f3d6ed26a 100644
--- a/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
+++ b/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
@@ -3916,8 +3916,8 @@ ProgramStateRef MallocChecker::checkPointerEscapeAux(
SymbolRef EscapingSymbol = nullptr;
if (Kind == PSK_DirectEscapeOnCall) {
// If we want to model this call explicitly, don't discard state.
- // FIXME: Maybe extend this to non-direct escape and/or non-simple calls
- // (e.g. 'isMemCall' is true for a C++ method with ownership attributes)?
+ // FIXME: Maybe extend this to non-direct escape and/or non-simple calls?
+ // ('isMemCall' is true for e.g. a C++ method with ownership attributes.)
if (isa<SimpleFunctionCall>(Call) && isMemCall(*Call))
return State;
// If we know that the call cannot release memory, don't discard state.
>From 6a4359cbd375b88528159b0ff4e2c6f4a6442192 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Don=C3=A1t=20Nagy?= <donat.nagy at ericsson.com>
Date: Tue, 3 Nov 2026 15:47:12 +0100
Subject: [PATCH 8/8] Fix the special case for funopen
I accidentally messed up the return value in this case when I rebased
the commits of this PR to a more recent `main` (where another commit
also changed this block).
---
clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp b/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
index 7795f3d6ed26a..4998174b02755 100644
--- a/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
+++ b/clang/lib/StaticAnalyzer/Checkers/MallocChecker.cpp
@@ -3822,7 +3822,7 @@ bool MallocChecker::ignoreDirectEscapeTo(const CallEvent *Call,
// Currently, we do not inspect the 'closefn' function (PR12101).
if (FName == "funopen")
if (Call->getNumArgs() >= 5 && Call->getArgSVal(4).isConstant(0))
- return false;
+ return true;
// Do not warn on pointers passed to 'setbuf' when used with std streams,
// these leaks might be intentional when setting the buffer for stdio.
More information about the cfe-commits
mailing list