[clang] [Clang] Forward incoming Indirect parameters across musttail calls (PR #199351)
Xavier Roche via cfe-commits
cfe-commits at lists.llvm.org
Thu Aug 6 01:51:58 PDT 2026
https://github.com/xroche updated https://github.com/llvm/llvm-project/pull/199351
>From cf30821b4da56f53854ceb81903dd276ffbf68fe Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Sat, 23 May 2026 15:37:12 +0200
Subject: [PATCH 01/21] [Clang] Forward incoming Indirect parameters across
musttail calls
When a C function passes a struct by value to a musttail callee,
Clang's frontend implements "by value" with a local
alloca + memcpy + pass-pointer pattern (the byval-temp). The alloca
lives in the caller's frame, which the tail call deallocates before
the callee dereferences the pointer. The callee reads freed stack
memory. Reproduces on RV64, AArch64, ARM, LoongArch64, and SystemZ at
every optimization level.
This is the argument-side analog of the SRet forwarding fix in
a96c14eeb8fc ("[Clang] Always forward sret parameters to musttail
calls", Kiran 2024-08-19). For musttail calls in the
ABIArgInfo::Indirect case, when the call argument's source LValue
resolves to a forwarded incoming Indirect parameter of the current
function with a matching ABI shape, forward the incoming llvm::Argument
directly instead of creating a byval-temp. Falls through to the
existing byval-temp path for any other source.
Three safety guards in the helper:
- ABI-attribute match (Verifier V7). Refuse to forward when the
incoming parameter and the call slot disagree on byval.
- noalias deduplication. If the user writes
`musttail callee(a, a)` with `a` a noalias Indirect parameter,
do not forward the same Argument to both slots; pre-fix gave two
distinct allocas and aliasing must not regress.
- AddrSpaceCast peek-through. EmitParmDecl wraps incoming Indirect
parameters in addrspacecast on NVPTX/AMDGPU/SPIR. Peek through one
cast; do NOT unwrap loads (a load through a local alloca means the
source is a local and the fix must not engage).
Scope: this PR fixes the C source case. C++ source for the same
construct routes through CXXConstructExpr + EmitAnyExprToTemp which
materializes an agg.tmp before EmitCall runs. The fix correctly falls
through in that case (the source is the local alloca, not the
Argument) but does not eliminate the dangle. A follow-up PR will plumb
IsMustTail through EmitCallArg to cover the C++ case.
Test: clang/test/CodeGen/musttail-indirect-arg.c covers plain forward,
two-arg forward, swapped args, mixed direct+indirect, modify-then-
forward, and negative cases (local source, computed copy, non-
musttail). Runs on riscv64, aarch64, loongarch64, s390x.
Fixes #56908. Helps #116568 #157814 #46402 #190429 #56435 #72555.
Complement of the backend fix in #185094 (RISC-V) and 0be65bac6907
(LoongArch).
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply at anthropic.com>
---
clang/lib/CodeGen/CGCall.cpp | 86 +++++++++++++++++++++
clang/test/CodeGen/musttail-indirect-arg.c | 90 ++++++++++++++++++++++
2 files changed, 176 insertions(+)
create mode 100644 clang/test/CodeGen/musttail-indirect-arg.c
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index 40cc275d40273..89aa89bfb26a4 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5448,6 +5448,69 @@ static unsigned getMaxVectorWidth(const llvm::Type *Ty) {
return MaxVectorWidth;
}
+/// For a musttail call argument lowered as ABIArgInfo::Indirect, returns the
+/// incoming llvm::Argument of the current function when the call argument's
+/// source is a forwarded incoming Indirect parameter with a matching ABI
+/// shape. Returns nullptr to fall through to the normal byval-temp path.
+///
+/// Forwarding is safe under musttail's prototype-match invariant: the
+/// incoming pointer points into the caller's caller's frame and stays valid
+/// across the tail call, whereas a local alloca would dangle. This mirrors
+/// the SRet forwarding in the return path (see commit a96c14eeb8fc,
+/// "Always forward sret parameters to musttail calls").
+///
+/// Guards:
+/// - The source LValue must be the IR-level Argument of CurFn (peek through
+/// one AddrSpaceCastInst for non-default alloca address spaces; do NOT
+/// unwrap loads, since a load through a local alloca means the source
+/// IS a local).
+/// - The incoming parameter must be passed indirectly with byval-ness
+/// matching the call slot (Verifier V7).
+/// - The Argument must not already have been forwarded by a sibling call
+/// argument in this same call (noalias deduplication).
+static llvm::Argument *getForwardableIncomingMustTailArg(
+ CodeGenFunction &CGF, const CallArg &CallArgument,
+ const ABIArgInfo &CallSlotInfo,
+ llvm::SmallPtrSetImpl<llvm::Argument *> &AlreadyForwarded) {
+ // The call argument can be either an LValue (DeclRefExpr to a parameter)
+ // or an RValue aggregate (typical for struct args lowered by CGCall). Both
+ // expose the underlying address; we just need the IR-level pointer.
+ Address SrcAddr = Address::invalid();
+ if (CallArgument.hasLValue())
+ SrcAddr = CallArgument.getKnownLValue().getAddress();
+ else if (CallArgument.getKnownRValue().isAggregate())
+ SrcAddr = CallArgument.getKnownRValue().getAggregateAddress();
+ else
+ return nullptr;
+ llvm::Value *SrcPtr = SrcAddr.emitRawPointer(CGF);
+
+ // Peek through one AddrSpaceCastInst. EmitParmDecl wraps incoming Indirect
+ // parameters in addrspacecast on targets whose alloca address space differs
+ // from the parameter's pointer address space (NVPTX / AMDGPU / SPIR).
+ if (auto *ASC = llvm::dyn_cast<llvm::AddrSpaceCastInst>(SrcPtr))
+ SrcPtr = ASC->getOperand(0);
+
+ auto *IncomingArg = llvm::dyn_cast<llvm::Argument>(SrcPtr);
+ if (!IncomingArg || IncomingArg->getParent() != CGF.CurFn)
+ return nullptr;
+
+ // byval-ness must match between the incoming parameter and the call slot.
+ // The Verifier rejects musttail across an ABI-attribute mismatch (V7), so
+ // producing IR with a mismatch is a verification failure. Falling through
+ // to byval-temp is the safe behavior.
+ if (IncomingArg->hasByValAttr() != CallSlotInfo.getIndirectByVal())
+ return nullptr;
+
+ // noalias deduplication: a noalias incoming parameter must not be
+ // forwarded to two slots in the same call. Pre-fix, each slot got its
+ // own byval-temp; we must not regress that aliasing guarantee.
+ if (IncomingArg->hasNoAliasAttr() &&
+ !AlreadyForwarded.insert(IncomingArg).second)
+ return nullptr;
+
+ return IncomingArg;
+}
+
RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
const CGCallee &Callee,
ReturnValueSlot ReturnValue,
@@ -5571,6 +5634,12 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
// markers that need to be ended right after the call.
SmallVector<CallLifetimeEnd, 2> CallLifetimeEndAfterCall;
+ // For musttail calls forwarding Indirect parameters: tracks incoming
+ // Arguments already forwarded to a slot in this call, so a noalias
+ // incoming Argument is not forwarded to two slots (see
+ // getForwardableIncomingMustTailArg).
+ llvm::SmallPtrSet<llvm::Argument *, 4> ForwardedMustTailArgs;
+
// Translate all of the arguments as necessary to match the IR lowering.
assert(CallInfo.arg_size() == CallArgs.size() &&
"Mismatch between function signature & arguments.");
@@ -5643,6 +5712,23 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
case ABIArgInfo::Indirect:
case ABIArgInfo::IndirectAliased: {
assert(NumIRArgs == 1);
+
+ // For musttail calls, forward an incoming Indirect parameter directly
+ // instead of creating a byval-temp. A local alloca would be deallocated
+ // by the tail call before the callee dereferences the pointer. The
+ // incoming pointer points into the caller's caller's frame, which
+ // remains valid. Mirrors the SRet forwarding above (a96c14eeb8fc).
+ if (IsMustTail) {
+ if (llvm::Argument *FwdArg = getForwardableIncomingMustTailArg(
+ *this, *I, ArgInfo, ForwardedMustTailArgs)) {
+ llvm::Value *Val = FwdArg;
+ if (ArgHasMaybeUndefAttr)
+ Val = Builder.CreateFreeze(Val);
+ IRCallArgs[FirstIRArg] = Val;
+ break;
+ }
+ }
+
if (I->isAggregate()) {
// We want to avoid creating an unnecessary temporary+copy here;
// however, we need one in three cases:
diff --git a/clang/test/CodeGen/musttail-indirect-arg.c b/clang/test/CodeGen/musttail-indirect-arg.c
new file mode 100644
index 0000000000000..70ef193493e27
--- /dev/null
+++ b/clang/test/CodeGen/musttail-indirect-arg.c
@@ -0,0 +1,90 @@
+// Test that Clang forwards incoming Indirect parameters across musttail calls
+// instead of creating a byval-temp alloca that would dangle after the tail call
+// deallocates the caller's frame.
+//
+// Companion to musttail-sret.cpp (commit a96c14eeb8fc): same idea, applied to
+// incoming arguments rather than the sret return slot.
+
+// RUN: %clang_cc1 -triple=riscv64-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
+// RUN: %clang_cc1 -triple=aarch64-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
+// RUN: %clang_cc1 -triple=loongarch64-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
+// RUN: %clang_cc1 -triple=s390x-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
+
+// A struct large enough to land on the indirect-arg path on RV64 (>2*XLEN=16
+// bytes), AArch64 (>16 bytes), LoongArch64, SystemZ.
+struct Big {
+ unsigned long long a, b, c, d;
+};
+
+// Plain forward: caller(B) musttails callee(B). The fix should emit no
+// byval-temp alloca; the call should forward the incoming parameter %a.
+struct Big C1(struct Big a);
+struct Big P1(struct Big a) {
+ __attribute__((musttail)) return C1(a);
+}
+// COMMON-LABEL: define {{.*}} @P1(
+// COMMON-NOT: alloca %struct.Big
+// COMMON: musttail call {{.*}} @C1({{.*}} %a
+
+// Two indirect args, same forwarding: each forwards its own incoming param.
+struct Big C2(struct Big a, struct Big b);
+struct Big P2(struct Big a, struct Big b) {
+ __attribute__((musttail)) return C2(a, b);
+}
+// COMMON-LABEL: define {{.*}} @P2(
+// COMMON-NOT: alloca %struct.Big
+// COMMON: musttail call {{.*}} @C2({{.*}} %a, {{.*}} %b
+
+// Swapped args: caller(a, b) musttails callee(b, a). Each forwarded slot
+// must resolve to the correct incoming Argument, not by position.
+struct Big C3(struct Big x, struct Big y);
+struct Big P3(struct Big a, struct Big b) {
+ __attribute__((musttail)) return C3(b, a);
+}
+// COMMON-LABEL: define {{.*}} @P3(
+// COMMON-NOT: alloca %struct.Big
+// COMMON: musttail call {{.*}} @C3({{.*}} %b, {{.*}} %a
+
+// Mixed direct + indirect: only the indirect arg is affected by the fix.
+struct Big C4(int n, struct Big a);
+struct Big P4(int n, struct Big a) {
+ __attribute__((musttail)) return C4(n, a);
+}
+// COMMON-LABEL: define {{.*}} @P4(
+// COMMON-NOT: alloca %struct.Big
+// COMMON: musttail call {{.*}} @C4({{.*}} %n, {{.*}} %a
+
+// Negative: local source. Caller takes Big a, but musttails with a LOCAL
+// Big initialized in caller's frame. The byval-temp must remain because the
+// source lives in caller's frame and would dangle if forwarded. The fix
+// must NOT engage in this case.
+struct Big C5(struct Big a);
+struct Big P5(struct Big a) {
+ struct Big local = {1, 2, 3, 4};
+ __attribute__((musttail)) return C5(local);
+}
+// COMMON-LABEL: define {{.*}} @P5(
+// COMMON: alloca
+// COMMON: musttail call {{.*}} @C5(
+
+// Negative: computed value (caller modifies the parameter then musttails).
+// The IR will use %a directly (Clang lowers writes through the incoming
+// pointer for Indirect params) so the fix does engage on the formal param,
+// but a fresh alloca is not created either way -- existing behavior.
+struct Big C6(struct Big a);
+struct Big P6(struct Big a) {
+ a.a += 1;
+ __attribute__((musttail)) return C6(a);
+}
+// COMMON-LABEL: define {{.*}} @P6(
+// COMMON-NOT: alloca %struct.Big
+// COMMON: musttail call {{.*}} @C6({{.*}} %a
+
+// Non-musttail tail call: the fix must NOT engage. Existing path emits
+// the byval-temp as before.
+struct Big C7(struct Big a);
+struct Big P7(struct Big a) {
+ return C7(a);
+}
+// COMMON-LABEL: define {{.*}} @P7(
+// COMMON-NOT: musttail
>From a1c3fd3e175cd6c26c3868460c03b931382099f9 Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Sat, 23 May 2026 16:18:46 +0200
Subject: [PATCH 02/21] [Clang][NFC] Trim comments on musttail Indirect
forwarding helper
Reduce comment volume on getForwardableIncomingMustTailArg and the
call site to match the SRet precedent (a96c14eeb8fc). Same code,
shorter comments.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply at anthropic.com>
---
clang/lib/CodeGen/CGCall.cpp | 56 +++++++++---------------------------
1 file changed, 14 insertions(+), 42 deletions(-)
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index 89aa89bfb26a4..b130df61c18b8 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5448,33 +5448,14 @@ static unsigned getMaxVectorWidth(const llvm::Type *Ty) {
return MaxVectorWidth;
}
-/// For a musttail call argument lowered as ABIArgInfo::Indirect, returns the
-/// incoming llvm::Argument of the current function when the call argument's
-/// source is a forwarded incoming Indirect parameter with a matching ABI
-/// shape. Returns nullptr to fall through to the normal byval-temp path.
-///
-/// Forwarding is safe under musttail's prototype-match invariant: the
-/// incoming pointer points into the caller's caller's frame and stays valid
-/// across the tail call, whereas a local alloca would dangle. This mirrors
-/// the SRet forwarding in the return path (see commit a96c14eeb8fc,
-/// "Always forward sret parameters to musttail calls").
-///
-/// Guards:
-/// - The source LValue must be the IR-level Argument of CurFn (peek through
-/// one AddrSpaceCastInst for non-default alloca address spaces; do NOT
-/// unwrap loads, since a load through a local alloca means the source
-/// IS a local).
-/// - The incoming parameter must be passed indirectly with byval-ness
-/// matching the call slot (Verifier V7).
-/// - The Argument must not already have been forwarded by a sibling call
-/// argument in this same call (noalias deduplication).
+/// Returns the incoming llvm::Argument of the current function if this
+/// musttail call argument forwards an incoming Indirect parameter with a
+/// matching ABI shape; nullptr to fall through to the byval-temp path.
+/// Argument-side analog of a96c14eeb8fc (SRet musttail forwarding).
static llvm::Argument *getForwardableIncomingMustTailArg(
CodeGenFunction &CGF, const CallArg &CallArgument,
const ABIArgInfo &CallSlotInfo,
llvm::SmallPtrSetImpl<llvm::Argument *> &AlreadyForwarded) {
- // The call argument can be either an LValue (DeclRefExpr to a parameter)
- // or an RValue aggregate (typical for struct args lowered by CGCall). Both
- // expose the underlying address; we just need the IR-level pointer.
Address SrcAddr = Address::invalid();
if (CallArgument.hasLValue())
SrcAddr = CallArgument.getKnownLValue().getAddress();
@@ -5484,9 +5465,9 @@ static llvm::Argument *getForwardableIncomingMustTailArg(
return nullptr;
llvm::Value *SrcPtr = SrcAddr.emitRawPointer(CGF);
- // Peek through one AddrSpaceCastInst. EmitParmDecl wraps incoming Indirect
- // parameters in addrspacecast on targets whose alloca address space differs
- // from the parameter's pointer address space (NVPTX / AMDGPU / SPIR).
+ // Peek through one AddrSpaceCastInst (NVPTX / AMDGPU / SPIR wrap incoming
+ // Indirect params via EmitParmDecl). Do not unwrap loads: a load through
+ // a local alloca means the source is a local.
if (auto *ASC = llvm::dyn_cast<llvm::AddrSpaceCastInst>(SrcPtr))
SrcPtr = ASC->getOperand(0);
@@ -5494,16 +5475,11 @@ static llvm::Argument *getForwardableIncomingMustTailArg(
if (!IncomingArg || IncomingArg->getParent() != CGF.CurFn)
return nullptr;
- // byval-ness must match between the incoming parameter and the call slot.
- // The Verifier rejects musttail across an ABI-attribute mismatch (V7), so
- // producing IR with a mismatch is a verification failure. Falling through
- // to byval-temp is the safe behavior.
+ // Verifier V7 requires matching ABI attributes across musttail.
if (IncomingArg->hasByValAttr() != CallSlotInfo.getIndirectByVal())
return nullptr;
- // noalias deduplication: a noalias incoming parameter must not be
- // forwarded to two slots in the same call. Pre-fix, each slot got its
- // own byval-temp; we must not regress that aliasing guarantee.
+ // Do not forward the same noalias Argument to two slots in one call.
if (IncomingArg->hasNoAliasAttr() &&
!AlreadyForwarded.insert(IncomingArg).second)
return nullptr;
@@ -5634,10 +5610,8 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
// markers that need to be ended right after the call.
SmallVector<CallLifetimeEnd, 2> CallLifetimeEndAfterCall;
- // For musttail calls forwarding Indirect parameters: tracks incoming
- // Arguments already forwarded to a slot in this call, so a noalias
- // incoming Argument is not forwarded to two slots (see
- // getForwardableIncomingMustTailArg).
+ // Tracks incoming Arguments already forwarded by a musttail Indirect arg,
+ // for noalias deduplication in getForwardableIncomingMustTailArg.
llvm::SmallPtrSet<llvm::Argument *, 4> ForwardedMustTailArgs;
// Translate all of the arguments as necessary to match the IR lowering.
@@ -5713,11 +5687,9 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
case ABIArgInfo::IndirectAliased: {
assert(NumIRArgs == 1);
- // For musttail calls, forward an incoming Indirect parameter directly
- // instead of creating a byval-temp. A local alloca would be deallocated
- // by the tail call before the callee dereferences the pointer. The
- // incoming pointer points into the caller's caller's frame, which
- // remains valid. Mirrors the SRet forwarding above (a96c14eeb8fc).
+ // For musttail, forward an incoming Indirect parameter directly. A
+ // local alloca would dangle after the tail call. Mirrors the SRet
+ // forwarding above (a96c14eeb8fc).
if (IsMustTail) {
if (llvm::Argument *FwdArg = getForwardableIncomingMustTailArg(
*this, *I, ArgInfo, ForwardedMustTailArgs)) {
>From 11cdbbffa6ae25864aafafedc745cac47c8872ab Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Sat, 23 May 2026 18:16:50 +0200
Subject: [PATCH 03/21] [Clang] Address review on musttail Indirect forwarding
- Restrict the helper to ABIArgInfo::Indirect (not IndirectAliased).
CallSlotInfo.getIndirectByVal() asserts on IndirectAliased; falling
through to the byval-temp path is the safe behavior for that case.
- Tighten CHECK patterns: anchor the forwarded operand by IR name
(e.g. `@C1({{.*}} %a)`) and exclude `alloca [32 x i8]` in addition
to `alloca %struct.Big` so a future ABI change that picks a
different temp type cannot mask a regression.
- Add P5 cross-BB musttail (musttail behind a branch) and P7 same-
Argument-to-two-slots (pins the noalias dedup behavior under the
Linux C ABI where incoming Indirect params are not noalias).
- Reword P6 (caller modifies the parameter then musttails): this is
a positive case, not a negative one. The fix correctly engages and
eliminates the byval-temp.
No functional change to the fix shape; reviewer-flagged hardening.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply at anthropic.com>
---
clang/lib/CodeGen/CGCall.cpp | 6 +-
clang/test/CodeGen/musttail-indirect-arg.c | 92 ++++++++++++++--------
2 files changed, 64 insertions(+), 34 deletions(-)
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index b130df61c18b8..ad402c459b892 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5689,8 +5689,10 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
// For musttail, forward an incoming Indirect parameter directly. A
// local alloca would dangle after the tail call. Mirrors the SRet
- // forwarding above (a96c14eeb8fc).
- if (IsMustTail) {
+ // forwarding above (a96c14eeb8fc). Limited to Indirect (not
+ // IndirectAliased) so the byval-ness check in the helper does not
+ // assert on getIndirectByVal().
+ if (IsMustTail && ArgInfo.isIndirect()) {
if (llvm::Argument *FwdArg = getForwardableIncomingMustTailArg(
*this, *I, ArgInfo, ForwardedMustTailArgs)) {
llvm::Value *Val = FwdArg;
diff --git a/clang/test/CodeGen/musttail-indirect-arg.c b/clang/test/CodeGen/musttail-indirect-arg.c
index 70ef193493e27..0e7493f9e38b7 100644
--- a/clang/test/CodeGen/musttail-indirect-arg.c
+++ b/clang/test/CodeGen/musttail-indirect-arg.c
@@ -17,14 +17,16 @@ struct Big {
};
// Plain forward: caller(B) musttails callee(B). The fix should emit no
-// byval-temp alloca; the call should forward the incoming parameter %a.
+// alloca for the forwarded arg; the call should forward the incoming
+// parameter %a.
struct Big C1(struct Big a);
struct Big P1(struct Big a) {
__attribute__((musttail)) return C1(a);
}
// COMMON-LABEL: define {{.*}} @P1(
-// COMMON-NOT: alloca %struct.Big
-// COMMON: musttail call {{.*}} @C1({{.*}} %a
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON-NOT: = alloca [32 x i8]
+// COMMON: musttail call {{.*}} @C1({{.*}} %a)
// Two indirect args, same forwarding: each forwards its own incoming param.
struct Big C2(struct Big a, struct Big b);
@@ -32,8 +34,8 @@ struct Big P2(struct Big a, struct Big b) {
__attribute__((musttail)) return C2(a, b);
}
// COMMON-LABEL: define {{.*}} @P2(
-// COMMON-NOT: alloca %struct.Big
-// COMMON: musttail call {{.*}} @C2({{.*}} %a, {{.*}} %b
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @C2({{.*}} %a, {{.*}} %b)
// Swapped args: caller(a, b) musttails callee(b, a). Each forwarded slot
// must resolve to the correct incoming Argument, not by position.
@@ -42,8 +44,8 @@ struct Big P3(struct Big a, struct Big b) {
__attribute__((musttail)) return C3(b, a);
}
// COMMON-LABEL: define {{.*}} @P3(
-// COMMON-NOT: alloca %struct.Big
-// COMMON: musttail call {{.*}} @C3({{.*}} %b, {{.*}} %a
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @C3({{.*}} %b, {{.*}} %a)
// Mixed direct + indirect: only the indirect arg is affected by the fix.
struct Big C4(int n, struct Big a);
@@ -51,40 +53,66 @@ struct Big P4(int n, struct Big a) {
__attribute__((musttail)) return C4(n, a);
}
// COMMON-LABEL: define {{.*}} @P4(
-// COMMON-NOT: alloca %struct.Big
-// COMMON: musttail call {{.*}} @C4({{.*}} %n, {{.*}} %a
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @C4({{.*}} %n, {{.*}} %a)
-// Negative: local source. Caller takes Big a, but musttails with a LOCAL
-// Big initialized in caller's frame. The byval-temp must remain because the
-// source lives in caller's frame and would dangle if forwarded. The fix
-// must NOT engage in this case.
+// Caller modifies the parameter before the musttail. Clang lowers the
+// write through the incoming pointer, and the fix forwards the same
+// pointer to the callee. No byval-temp.
struct Big C5(struct Big a);
struct Big P5(struct Big a) {
- struct Big local = {1, 2, 3, 4};
- __attribute__((musttail)) return C5(local);
+ a.a += 1;
+ __attribute__((musttail)) return C5(a);
}
// COMMON-LABEL: define {{.*}} @P5(
-// COMMON: alloca
-// COMMON: musttail call {{.*}} @C5(
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @C5({{.*}} %a)
-// Negative: computed value (caller modifies the parameter then musttails).
-// The IR will use %a directly (Clang lowers writes through the incoming
-// pointer for Indirect params) so the fix does engage on the formal param,
-// but a fresh alloca is not created either way -- existing behavior.
-struct Big C6(struct Big a);
-struct Big P6(struct Big a) {
- a.a += 1;
- __attribute__((musttail)) return C6(a);
+// musttail behind a branch: the forwarded pointer must remain live across
+// the basic block transition. Tests that the helper does not assume the
+// musttail is in the entry block.
+struct Big C6(struct Big a, int cond);
+struct Big P6(struct Big a, int cond) {
+ if (cond)
+ __attribute__((musttail)) return C6(a, cond);
+ return a;
}
// COMMON-LABEL: define {{.*}} @P6(
-// COMMON-NOT: alloca %struct.Big
-// COMMON: musttail call {{.*}} @C6({{.*}} %a
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @C6({{.*}} %a,
-// Non-musttail tail call: the fix must NOT engage. Existing path emits
-// the byval-temp as before.
-struct Big C7(struct Big a);
-struct Big P7(struct Big a) {
- return C7(a);
+// Same Argument forwarded to two slots: the helper engages for both. The
+// noalias deduplication, if it ever fired, would force the second slot
+// back to a byval-temp; but incoming Indirect params under the Linux C
+// ABI are not noalias, so both slots forward %a directly. This pins the
+// behavior so a future change introducing noalias on Indirect params
+// would surface here. (musttail requires matching prototypes, so caller
+// and callee both take two Big args.)
+struct Big C7(struct Big x, struct Big y);
+struct Big P7(struct Big a, struct Big b) {
+ __attribute__((musttail)) return C7(a, a);
}
// COMMON-LABEL: define {{.*}} @P7(
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @C7({{.*}} %a, {{.*}} %a)
+
+// Negative: local source. Caller takes Big a, but musttails with a LOCAL
+// Big initialized in caller's frame. The byval-temp must remain because the
+// source lives in caller's frame and would dangle if forwarded.
+struct Big C8(struct Big a);
+struct Big P8(struct Big a) {
+ struct Big local = {1, 2, 3, 4};
+ __attribute__((musttail)) return C8(local);
+}
+// COMMON-LABEL: define {{.*}} @P8(
+// COMMON: = alloca
+// COMMON: musttail call {{.*}} @C8(
+
+// Non-musttail tail call: the fix must NOT engage. Existing path emits
+// the byval-temp as before, no musttail in the IR.
+struct Big C9(struct Big a);
+struct Big P9(struct Big a) {
+ return C9(a);
+}
+// COMMON-LABEL: define {{.*}} @P9(
// COMMON-NOT: musttail
>From 43590837d6a8bf2d8120be835dc0cfd2817d9822 Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Sat, 23 May 2026 21:20:58 +0200
Subject: [PATCH 04/21] [Clang] Extend musttail Indirect forwarding to C++
trivial-copy args
For C++ struct-by-value arguments, the call argument is a
CXXConstructExpr invoking the implicit copy constructor, not the
LValueToRValue cast that EmitCallArg uses for C. The trivial copy
materializes an agg.tmp before EmitCall runs, so the helper in
EmitCall (getForwardableIncomingMustTailArg) sees a local alloca and
falls through.
Detect the trivial-copy CXXConstructExpr whose source is a
ParmVarDecl of the current function inside a musttail call (signaled
by CodeGenFunction::MustTailCall being non-null) and take the
addUncopiedAggregate path so the LValue of the original parameter
reaches EmitCall. The forwarding helper then forwards the incoming
Argument, no agg.tmp.
Limited to trivially-copyable types so we never elide observable
copy constructors or skip destruction. Non-trivial copy ctors / dtors
still take the existing materialize-then-copy path.
Test: clang/test/CodeGen/musttail-indirect-arg.cpp (companion to the
C test) covers plain forward, two-arg, swapped, and verifies the
trivial-copy elision does NOT engage for non-trivial copy ctors or
non-musttail tail calls.
clang/test/CodeGen + CodeGenCXX (7307 tests) clean. Runtime cross-arch
sweep on a C++ minimal repro passes on x86_64, riscv64, aarch64, arm,
loongarch64, s390x; the same repro fails on RV64/AArch64 without this
commit, confirming the new path catches the C++ bug shape.
Fixes the C++ scope hole called out in the prior commit
(11cdbbffa6ae).
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply at anthropic.com>
---
clang/lib/CodeGen/CGCall.cpp | 24 +++++++
clang/test/CodeGen/musttail-indirect-arg.cpp | 73 ++++++++++++++++++++
2 files changed, 97 insertions(+)
create mode 100644 clang/test/CodeGen/musttail-indirect-arg.cpp
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index ad402c459b892..c34c44571b33d 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5139,6 +5139,30 @@ void CodeGenFunction::EmitCallArg(CallArgList &args, const Expr *E,
return;
}
+ // For musttail calls in C++, skip the agg.tmp copy when the source is a
+ // trivially-copyable parameter of the current function. The copy would
+ // live in our frame, which the tail call deallocates before the callee
+ // dereferences it. The companion fix in EmitCall (getForwardableIncoming
+ // MustTailArg) forwards the incoming Indirect Argument; for that to fire
+ // here we must hand it the original parameter LValue, not a fresh temp.
+ if (HasAggregateEvalKind && MustTailCall && type->isRecordType() &&
+ type.isTriviallyCopyableType(getContext())) {
+ if (const auto *CCE = dyn_cast<CXXConstructExpr>(E)) {
+ if (CCE->getConstructor()->isCopyOrMoveConstructor() &&
+ CCE->getConstructor()->isTrivial() && CCE->getNumArgs() == 1) {
+ const Expr *Source = CCE->getArg(0)->IgnoreParenImpCasts();
+ if (const auto *DRE = dyn_cast<DeclRefExpr>(Source))
+ if (const auto *PVD = dyn_cast<ParmVarDecl>(DRE->getDecl()))
+ if (PVD->getDeclContext() == dyn_cast<DeclContext>(CurCodeDecl)) {
+ LValue L = EmitLValue(DRE);
+ assert(L.isSimple());
+ args.addUncopiedAggregate(L, type);
+ return;
+ }
+ }
+ }
+ }
+
args.add(EmitAnyExprToTemp(E), type);
}
diff --git a/clang/test/CodeGen/musttail-indirect-arg.cpp b/clang/test/CodeGen/musttail-indirect-arg.cpp
new file mode 100644
index 0000000000000..a161f979b5a4c
--- /dev/null
+++ b/clang/test/CodeGen/musttail-indirect-arg.cpp
@@ -0,0 +1,73 @@
+// Test that Clang forwards incoming Indirect parameters across musttail calls
+// for C++ struct-by-value arguments with trivially-copyable types. Companion to
+// musttail-indirect-arg.c (the C side of the same fix) and musttail-sret.cpp
+// (the SRet precedent in a96c14eeb8fc).
+//
+// C++ goes through a different EmitCallArg path than C: the call argument is a
+// CXXConstructExpr invoking the implicit copy constructor, which would
+// otherwise materialize an agg.tmp before EmitCall. For musttail with a
+// trivially-copyable parameter forwarded directly, the copy is elided so the
+// helper in EmitCall can forward the incoming llvm::Argument.
+
+// RUN: %clang_cc1 -triple=riscv64-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
+// RUN: %clang_cc1 -triple=aarch64-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
+// RUN: %clang_cc1 -triple=loongarch64-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
+// RUN: %clang_cc1 -triple=s390x-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
+
+// A trivially-copyable struct large enough to land on the indirect-arg path
+// on RV64, AArch64, LoongArch64, SystemZ.
+struct Big {
+ unsigned long long a, b, c, d;
+};
+
+// Plain forward: caller(B) musttails callee(B). No agg.tmp copy, no
+// byval-temp; the incoming parameter %a is forwarded directly.
+struct Big C1(struct Big a);
+struct Big P1(struct Big a) {
+ [[clang::musttail]] return C1(a);
+}
+// COMMON-LABEL: define {{.*}} @_Z2P13Big(
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @_Z2C13Big({{.*}} %a)
+
+// Two args, same forwarding.
+struct Big C2(struct Big a, struct Big b);
+struct Big P2(struct Big a, struct Big b) {
+ [[clang::musttail]] return C2(a, b);
+}
+// COMMON-LABEL: define {{.*}} @_Z2P23BigS_(
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @_Z2C23BigS_({{.*}} %a, {{.*}} %b)
+
+// Swapped args.
+struct Big C3(struct Big x, struct Big y);
+struct Big P3(struct Big a, struct Big b) {
+ [[clang::musttail]] return C3(b, a);
+}
+// COMMON-LABEL: define {{.*}} @_Z2P33BigS_(
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @_Z2C33BigS_({{.*}} %b, {{.*}} %a)
+
+// Non-trivial copy constructor: the trivial-copy elision must NOT engage.
+// Existing path materializes the agg.tmp (the user-defined copy ctor has
+// observable behavior).
+struct NonTrivial {
+ unsigned long long parts[4];
+ NonTrivial(const NonTrivial &);
+};
+NonTrivial C4(NonTrivial a);
+NonTrivial P4(NonTrivial a) {
+ [[clang::musttail]] return C4(a);
+}
+// COMMON-LABEL: define {{.*}} @_Z2P410NonTrivial(
+// The user-defined copy ctor IS called (the agg.tmp pattern still happens):
+// COMMON: call {{.*}} @_ZN10NonTrivialC1ERKS_
+
+// Non-musttail tail call: trivial-copy elision must NOT engage; the regular
+// agg.tmp copy is still emitted.
+struct Big C5(struct Big a);
+struct Big P5(struct Big a) {
+ return C5(a);
+}
+// COMMON-LABEL: define {{.*}} @_Z2P53Big(
+// COMMON-NOT: musttail
>From 9b1aa8b1228237096222c71ecaa4ea73c5b5b96c Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Sat, 23 May 2026 21:37:39 +0200
Subject: [PATCH 05/21] [Clang][test] Mirror C test cases in
musttail-indirect-arg.cpp
Add the C++ analogs of P5/P6/P7/P8 from the C test that the C++ test
was missing per code review:
- P5: modify-then-musttail (trivial-copy elision still engages).
- P6: cross-BB musttail (elision works across basic blocks).
- P7: same Argument forwarded to two slots (pins noalias dedup
behavior; incoming Indirect params are not noalias under Linux C++).
- P8: source is a non-parameter local (elision must NOT engage,
byval-temp pattern remains).
Existing non-musttail test renumbered P5 -> P9.
Closes the test-coverage gap flagged by the second review on commit
43590837d6a8.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply at anthropic.com>
---
clang/test/CodeGen/musttail-indirect-arg.cpp | 52 ++++++++++++++++++--
1 file changed, 49 insertions(+), 3 deletions(-)
diff --git a/clang/test/CodeGen/musttail-indirect-arg.cpp b/clang/test/CodeGen/musttail-indirect-arg.cpp
index a161f979b5a4c..4c3203ca12cf2 100644
--- a/clang/test/CodeGen/musttail-indirect-arg.cpp
+++ b/clang/test/CodeGen/musttail-indirect-arg.cpp
@@ -63,11 +63,57 @@ NonTrivial P4(NonTrivial a) {
// The user-defined copy ctor IS called (the agg.tmp pattern still happens):
// COMMON: call {{.*}} @_ZN10NonTrivialC1ERKS_
-// Non-musttail tail call: trivial-copy elision must NOT engage; the regular
-// agg.tmp copy is still emitted.
+// Caller modifies the parameter before the musttail. The trivial-copy
+// elision still engages because the source LValue is still the parameter;
+// any mutation flowed through the incoming pointer is observed by the
+// forwarded call.
struct Big C5(struct Big a);
struct Big P5(struct Big a) {
- return C5(a);
+ a.a += 1;
+ [[clang::musttail]] return C5(a);
}
// COMMON-LABEL: define {{.*}} @_Z2P53Big(
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @_Z2C53Big({{.*}} %a)
+
+// musttail behind a branch: trivial-copy elision must work across BBs.
+struct Big C6(struct Big a, int cond);
+struct Big P6(struct Big a, int cond) {
+ if (cond)
+ [[clang::musttail]] return C6(a, cond);
+ return a;
+}
+// COMMON-LABEL: define {{.*}} @_Z2P63Bigi(
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @_Z2C63Bigi({{.*}} %a,
+
+// Same Argument forwarded to two slots: both engage. Incoming Indirect
+// params are not noalias under the Linux C++ ABI so the dedup in the
+// helper does not fire and both slots forward %a directly.
+struct Big C7(struct Big x, struct Big y);
+struct Big P7(struct Big a, struct Big b) {
+ [[clang::musttail]] return C7(a, a);
+}
+// COMMON-LABEL: define {{.*}} @_Z2P73BigS_(
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @_Z2C73BigS_({{.*}} %a, {{.*}} %a)
+
+// Negative: source is a local, not a parameter. The trivial-copy elision
+// must NOT engage; the byval-temp pattern remains.
+struct Big C8(struct Big a);
+struct Big P8(struct Big a) {
+ struct Big local = {1, 2, 3, 4};
+ [[clang::musttail]] return C8(local);
+}
+// COMMON-LABEL: define {{.*}} @_Z2P83Big(
+// COMMON: = alloca
+// COMMON: musttail call {{.*}} @_Z2C83Big(
+
+// Non-musttail tail call: trivial-copy elision must NOT engage; the regular
+// agg.tmp copy is still emitted.
+struct Big C9(struct Big a);
+struct Big P9(struct Big a) {
+ return C9(a);
+}
+// COMMON-LABEL: define {{.*}} @_Z2P93Big(
// COMMON-NOT: musttail
>From 99dfa79d0d3f5424b36bf4ea01460f76dec4f98a Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Tue, 26 May 2026 10:11:03 +0200
Subject: [PATCH 06/21] [Clang] Switch musttail Indirect to a two-phase general
algorithm
Addresses review on commit 9b1aa8b12282 (efriedma-quic): the prior
code optimized only the "source is incoming Indirect param" case,
and test P7 asserted a miscompile (forwarding the same incoming
pointer to two call slots aliases two by-value parameters, which
violates the C ABI rule of distinct storage per slot).
For each musttail Indirect slot i, the i-th incoming Indirect
parameter's pointer (CurFn->arg_begin()+FirstIRArg, distinct and
well-defined by musttail prototype-match) is the destination. If
the source LValue already equals that pointer, pass it. Otherwise
the value is copied into the incoming-param storage and that
pointer is passed.
A single-phase implementation is unsound for permutations like
C(b, a) where slot-by-slot in-place writes clobber sources later
slots need. Two phases:
- Phase 1, in the per-arg loop: when source != destination,
capture the source value into a scratch alloca in this frame.
- Phase 2, after the per-arg loop: write each scratch into its
matching incoming-param destination.
Scratches live in this frame and die at tail-call teardown; they
are consumed before the tail jump so this is safe. Destinations
live in the caller's caller's frame and survive the tail call.
EmitCallArg broadens the C++ trivial-copy gate from "ParmVarDecl
of this function" to "param or local of this function" so the
source LValue reaches EmitCall in both cases.
Tests rewritten in spec-driven form: forward, distinct, swap,
modify-then-forward, cross-BB, same-arg-twice/thrice, local source,
mixed direct+indirect, many-args with stack spill, over-aligned
struct, C++ member function, and non-trivial copy ctor as a
negative.
Local validation: check-clang 50997/52167. Runtime probe under
QEMU on aarch64, riscv64, arm-linux-gnueabihf, and s390x passes;
the same probe fails on aarch64 and riscv64 with pre-patch clang,
exposing the alias and swap miscompiles.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.6 <noreply at anthropic.com>
---
clang/lib/CodeGen/CGCall.cpp | 127 +++++++++--------
clang/test/CodeGen/musttail-indirect-arg.c | 135 +++++++++++--------
clang/test/CodeGen/musttail-indirect-arg.cpp | 118 +++++++++-------
3 files changed, 223 insertions(+), 157 deletions(-)
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index c34c44571b33d..04774cc967071 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5139,26 +5139,30 @@ void CodeGenFunction::EmitCallArg(CallArgList &args, const Expr *E,
return;
}
- // For musttail calls in C++, skip the agg.tmp copy when the source is a
- // trivially-copyable parameter of the current function. The copy would
- // live in our frame, which the tail call deallocates before the callee
- // dereferences it. The companion fix in EmitCall (getForwardableIncoming
- // MustTailArg) forwards the incoming Indirect Argument; for that to fire
- // here we must hand it the original parameter LValue, not a fresh temp.
+ // Under musttail, hand a trivially-copyable record source's LValue to
+ // EmitCall rather than materializing an agg.tmp. EmitCall's Indirect path
+ // routes it via the matching incoming parameter, which survives the tail
+ // call. Limited to params and locals: globals and captures don't have the
+ // dangle issue and the existing path may be more efficient for them.
if (HasAggregateEvalKind && MustTailCall && type->isRecordType() &&
type.isTriviallyCopyableType(getContext())) {
if (const auto *CCE = dyn_cast<CXXConstructExpr>(E)) {
if (CCE->getConstructor()->isCopyOrMoveConstructor() &&
CCE->getConstructor()->isTrivial() && CCE->getNumArgs() == 1) {
const Expr *Source = CCE->getArg(0)->IgnoreParenImpCasts();
- if (const auto *DRE = dyn_cast<DeclRefExpr>(Source))
- if (const auto *PVD = dyn_cast<ParmVarDecl>(DRE->getDecl()))
- if (PVD->getDeclContext() == dyn_cast<DeclContext>(CurCodeDecl)) {
+ if (const auto *DRE = dyn_cast<DeclRefExpr>(Source)) {
+ if (const auto *VD = dyn_cast<VarDecl>(DRE->getDecl())) {
+ if (VD->hasLocalStorage() ||
+ (isa<ParmVarDecl>(VD) &&
+ VD->getDeclContext() ==
+ dyn_cast<DeclContext>(CurCodeDecl))) {
LValue L = EmitLValue(DRE);
assert(L.isSimple());
args.addUncopiedAggregate(L, type);
return;
}
+ }
+ }
}
}
}
@@ -5472,43 +5476,15 @@ static unsigned getMaxVectorWidth(const llvm::Type *Ty) {
return MaxVectorWidth;
}
-/// Returns the incoming llvm::Argument of the current function if this
-/// musttail call argument forwards an incoming Indirect parameter with a
-/// matching ABI shape; nullptr to fall through to the byval-temp path.
-/// Argument-side analog of a96c14eeb8fc (SRet musttail forwarding).
-static llvm::Argument *getForwardableIncomingMustTailArg(
- CodeGenFunction &CGF, const CallArg &CallArgument,
- const ABIArgInfo &CallSlotInfo,
- llvm::SmallPtrSetImpl<llvm::Argument *> &AlreadyForwarded) {
- Address SrcAddr = Address::invalid();
- if (CallArgument.hasLValue())
- SrcAddr = CallArgument.getKnownLValue().getAddress();
- else if (CallArgument.getKnownRValue().isAggregate())
- SrcAddr = CallArgument.getKnownRValue().getAggregateAddress();
- else
- return nullptr;
- llvm::Value *SrcPtr = SrcAddr.emitRawPointer(CGF);
-
- // Peek through one AddrSpaceCastInst (NVPTX / AMDGPU / SPIR wrap incoming
- // Indirect params via EmitParmDecl). Do not unwrap loads: a load through
- // a local alloca means the source is a local.
+/// Peel one AddrSpaceCastInst from \p SrcPtr. EmitParmDecl wraps incoming
+/// Indirect params via address-space cast on NVPTX/AMDGPU/SPIR, so peeling
+/// exposes the underlying llvm::Argument when the source IS a forwarded
+/// incoming parameter. Loads are NOT unwrapped: a load through a local
+/// alloca means the source is a local.
+static llvm::Value *peelAddrSpaceCast(llvm::Value *SrcPtr) {
if (auto *ASC = llvm::dyn_cast<llvm::AddrSpaceCastInst>(SrcPtr))
- SrcPtr = ASC->getOperand(0);
-
- auto *IncomingArg = llvm::dyn_cast<llvm::Argument>(SrcPtr);
- if (!IncomingArg || IncomingArg->getParent() != CGF.CurFn)
- return nullptr;
-
- // Verifier V7 requires matching ABI attributes across musttail.
- if (IncomingArg->hasByValAttr() != CallSlotInfo.getIndirectByVal())
- return nullptr;
-
- // Do not forward the same noalias Argument to two slots in one call.
- if (IncomingArg->hasNoAliasAttr() &&
- !AlreadyForwarded.insert(IncomingArg).second)
- return nullptr;
-
- return IncomingArg;
+ return ASC->getOperand(0);
+ return SrcPtr;
}
RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
@@ -5634,9 +5610,16 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
// markers that need to be ended right after the call.
SmallVector<CallLifetimeEnd, 2> CallLifetimeEndAfterCall;
- // Tracks incoming Arguments already forwarded by a musttail Indirect arg,
- // for noalias deduplication in getForwardableIncomingMustTailArg.
- llvm::SmallPtrSet<llvm::Argument *, 4> ForwardedMustTailArgs;
+ // Deferred Phase-2 writes for musttail Indirect args. Splitting reads
+ // (in the per-arg loop) from writes (after the loop) lets permutations
+ // like C(b, a) land correctly: all sources are captured into scratches
+ // before any incoming-param destination is overwritten.
+ struct MustTailIndirectCopy {
+ LValue Scratch;
+ LValue Dst;
+ QualType Ty;
+ };
+ llvm::SmallVector<MustTailIndirectCopy, 4> MustTailIndirectCopies;
// Translate all of the arguments as necessary to match the IR lowering.
assert(CallInfo.arg_size() == CallArgs.size() &&
@@ -5711,20 +5694,45 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
case ABIArgInfo::IndirectAliased: {
assert(NumIRArgs == 1);
- // For musttail, forward an incoming Indirect parameter directly. A
- // local alloca would dangle after the tail call. Mirrors the SRet
- // forwarding above (a96c14eeb8fc). Limited to Indirect (not
- // IndirectAliased) so the byval-ness check in the helper does not
- // assert on getIndirectByVal().
+ // Musttail Indirect: route via the matching incoming parameter.
+ // Prototype-match (Verifier V5/V6/V7) makes CurFn->arg_begin()+
+ // FirstIRArg a distinct destination for this slot that lives in the
+ // caller's caller's frame and survives the tail call. To handle
+ // permutations safely, the source value is captured into a scratch
+ // alloca here (Phase 1); the write to the incoming-param destination
+ // is deferred until after all sources have been read (Phase 2 below).
+ // IndirectAliased uses the existing fallback (different source-AS).
if (IsMustTail && ArgInfo.isIndirect()) {
- if (llvm::Argument *FwdArg = getForwardableIncomingMustTailArg(
- *this, *I, ArgInfo, ForwardedMustTailArgs)) {
- llvm::Value *Val = FwdArg;
+ llvm::Argument *IncomingArg = CurFn->arg_begin() + FirstIRArg;
+ llvm::Value *Dst = IncomingArg;
+ Address SrcAddr = Address::invalid();
+ if (I->hasLValue())
+ SrcAddr = I->getKnownLValue().getAddress();
+ else if (I->getKnownRValue().isAggregate())
+ SrcAddr = I->getKnownRValue().getAggregateAddress();
+ if (SrcAddr.isValid()) {
+ llvm::Value *Src = peelAddrSpaceCast(SrcAddr.emitRawPointer(*this));
+ if (Src != Dst) {
+ CharUnits Align = ArgInfo.getIndirectAlign();
+ QualType Ty = I->Ty;
+ llvm::Type *ElemTy = ConvertTypeForMem(Ty);
+ RawAddress Scratch =
+ CreateMemTempWithoutCast(Ty, Align, "musttail.copy");
+ LValue ScratchLV = MakeAddrLValue(Scratch, Ty);
+ LValue SrcLV = MakeAddrLValue(SrcAddr, Ty);
+ EmitAggregateCopy(ScratchLV, SrcLV, Ty,
+ AggValueSlot::DoesNotOverlap);
+ LValue DstLV =
+ MakeAddrLValue(Address(Dst, ElemTy, Align), Ty);
+ MustTailIndirectCopies.push_back({ScratchLV, DstLV, Ty});
+ }
+ llvm::Value *Val = Dst;
if (ArgHasMaybeUndefAttr)
Val = Builder.CreateFreeze(Val);
IRCallArgs[FirstIRArg] = Val;
break;
}
+ // No clean source address (rare): fall through to byval-temp.
}
if (I->isAggregate()) {
@@ -6053,6 +6061,13 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
}
}
+ // Phase 2 of the musttail Indirect-arg copy: flush each captured scratch
+ // into its incoming-param destination. Phase 1 has read every source, so
+ // permutations like C(b, a) land the right value in each slot.
+ for (const auto &Copy : MustTailIndirectCopies)
+ EmitAggregateCopy(Copy.Dst, Copy.Scratch, Copy.Ty,
+ AggValueSlot::DoesNotOverlap);
+
const CGCallee &ConcreteCallee = Callee.prepareConcreteCallee(*this);
llvm::Value *CalleePtr = ConcreteCallee.getFunctionPointer();
diff --git a/clang/test/CodeGen/musttail-indirect-arg.c b/clang/test/CodeGen/musttail-indirect-arg.c
index 0e7493f9e38b7..fd82e9cf900c1 100644
--- a/clang/test/CodeGen/musttail-indirect-arg.c
+++ b/clang/test/CodeGen/musttail-indirect-arg.c
@@ -1,76 +1,60 @@
-// Test that Clang forwards incoming Indirect parameters across musttail calls
-// instead of creating a byval-temp alloca that would dangle after the tail call
-// deallocates the caller's frame.
-//
-// Companion to musttail-sret.cpp (commit a96c14eeb8fc): same idea, applied to
-// incoming arguments rather than the sret return slot.
-
// RUN: %clang_cc1 -triple=riscv64-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
// RUN: %clang_cc1 -triple=aarch64-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
// RUN: %clang_cc1 -triple=loongarch64-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
// RUN: %clang_cc1 -triple=s390x-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
-// A struct large enough to land on the indirect-arg path on RV64 (>2*XLEN=16
-// bytes), AArch64 (>16 bytes), LoongArch64, SystemZ.
+// Musttail calls with struct-by-value args must route each value through the
+// matching incoming Indirect parameter's storage, not a local alloca that
+// dangles past the tail-call frame teardown. For each Indirect slot i: if
+// the source IS the i-th incoming pointer, pass it; otherwise memcpy the
+// source into the i-th incoming pointer and pass that. Each call slot ends
+// up with a distinct pointer in the caller's caller's frame.
+
+// Plain Indirect-ABI struct on the targets above.
struct Big {
unsigned long long a, b, c, d;
};
-// Plain forward: caller(B) musttails callee(B). The fix should emit no
-// alloca for the forwarded arg; the call should forward the incoming
-// parameter %a.
+// P1: simple forward.
struct Big C1(struct Big a);
struct Big P1(struct Big a) {
__attribute__((musttail)) return C1(a);
}
// COMMON-LABEL: define {{.*}} @P1(
// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON-NOT: = alloca [32 x i8]
-// COMMON: musttail call {{.*}} @C1({{.*}} %a)
+// COMMON: musttail call {{.*}} @C1({{.*}}, ptr {{.*}} %a)
-// Two indirect args, same forwarding: each forwards its own incoming param.
+// P2: two distinct incoming sources.
struct Big C2(struct Big a, struct Big b);
struct Big P2(struct Big a, struct Big b) {
__attribute__((musttail)) return C2(a, b);
}
// COMMON-LABEL: define {{.*}} @P2(
// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON: musttail call {{.*}} @C2({{.*}} %a, {{.*}} %b)
+// COMMON-NOT: llvm.memcpy
+// COMMON: musttail call {{.*}} @C2({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
-// Swapped args: caller(a, b) musttails callee(b, a). Each forwarded slot
-// must resolve to the correct incoming Argument, not by position.
+// P3: swap. Slot 0's dst is %a (positional), so the value of %b is copied
+// into %a; symmetrically for slot 1. Two-phase emit captures both sources
+// before either destination is overwritten.
struct Big C3(struct Big x, struct Big y);
struct Big P3(struct Big a, struct Big b) {
__attribute__((musttail)) return C3(b, a);
}
// COMMON-LABEL: define {{.*}} @P3(
-// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON: musttail call {{.*}} @C3({{.*}} %b, {{.*}} %a)
-
-// Mixed direct + indirect: only the indirect arg is affected by the fix.
-struct Big C4(int n, struct Big a);
-struct Big P4(int n, struct Big a) {
- __attribute__((musttail)) return C4(n, a);
-}
-// COMMON-LABEL: define {{.*}} @P4(
-// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON: musttail call {{.*}} @C4({{.*}} %n, {{.*}} %a)
+// COMMON: musttail call {{.*}} @C3({{.*}}, ptr {{.*}}, ptr {{.*}})
-// Caller modifies the parameter before the musttail. Clang lowers the
-// write through the incoming pointer, and the fix forwards the same
-// pointer to the callee. No byval-temp.
+// P5: caller mutates the parameter before the musttail. The mutation lands
+// at the incoming pointer the callee receives.
struct Big C5(struct Big a);
struct Big P5(struct Big a) {
a.a += 1;
__attribute__((musttail)) return C5(a);
}
// COMMON-LABEL: define {{.*}} @P5(
-// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON: musttail call {{.*}} @C5({{.*}} %a)
+// COMMON: musttail call {{.*}} @C5({{.*}}, ptr {{.*}} %a)
-// musttail behind a branch: the forwarded pointer must remain live across
-// the basic block transition. Tests that the helper does not assume the
-// musttail is in the entry block.
+// P6: musttail in a non-entry block.
struct Big C6(struct Big a, int cond);
struct Big P6(struct Big a, int cond) {
if (cond)
@@ -78,41 +62,82 @@ struct Big P6(struct Big a, int cond) {
return a;
}
// COMMON-LABEL: define {{.*}} @P6(
-// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON: musttail call {{.*}} @C6({{.*}} %a,
+// COMMON: musttail call {{.*}} @C6({{.*}}, ptr {{.*}} %a,
-// Same Argument forwarded to two slots: the helper engages for both. The
-// noalias deduplication, if it ever fired, would force the second slot
-// back to a byval-temp; but incoming Indirect params under the Linux C
-// ABI are not noalias, so both slots forward %a directly. This pins the
-// behavior so a future change introducing noalias on Indirect params
-// would surface here. (musttail requires matching prototypes, so caller
-// and callee both take two Big args.)
+// P7: same arg to two slots. C ABI requires distinct storage per by-value
+// param, so slot 1 cannot share %a's pointer. Slot 0 forwards %a; slot 1
+// memcpys *%a into the i=1 incoming pointer %b and forwards %b.
struct Big C7(struct Big x, struct Big y);
struct Big P7(struct Big a, struct Big b) {
__attribute__((musttail)) return C7(a, a);
}
// COMMON-LABEL: define {{.*}} @P7(
-// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON: musttail call {{.*}} @C7({{.*}} %a, {{.*}} %a)
+// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %b, ptr {{.*}} %a,
+// COMMON: musttail call {{.*}} @C7({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
-// Negative: local source. Caller takes Big a, but musttails with a LOCAL
-// Big initialized in caller's frame. The byval-temp must remain because the
-// source lives in caller's frame and would dangle if forwarded.
+// P8: local source. The local lives in our frame; copy it into %a, forward %a.
struct Big C8(struct Big a);
struct Big P8(struct Big a) {
struct Big local = {1, 2, 3, 4};
__attribute__((musttail)) return C8(local);
}
// COMMON-LABEL: define {{.*}} @P8(
-// COMMON: = alloca
-// COMMON: musttail call {{.*}} @C8(
+// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %a, ptr {{.*}}
+// COMMON: musttail call {{.*}} @C8({{.*}}, ptr {{.*}} %a)
-// Non-musttail tail call: the fix must NOT engage. Existing path emits
-// the byval-temp as before, no musttail in the IR.
+// P9: non-musttail tail call (existing path).
struct Big C9(struct Big a);
struct Big P9(struct Big a) {
return C9(a);
}
// COMMON-LABEL: define {{.*}} @P9(
// COMMON-NOT: musttail
+
+// P10: mixed direct + indirect.
+struct Big C10(int x1, struct Big s1, int x2, struct Big s2);
+struct Big P10(int x1, struct Big s1, int x2, struct Big s2) {
+ __attribute__((musttail)) return C10(x1, s1, x2, s2);
+}
+// COMMON-LABEL: define {{.*}} @P10(
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @C10({{.*}}, i32 {{.*}} %x1, ptr {{.*}} %s1, i32 {{.*}} %x2, ptr {{.*}} %s2)
+
+// P11: many args, including stack-spilled ones on the target ABIs above.
+struct Big C11(struct Big s1, struct Big s2, struct Big s3, struct Big s4,
+ struct Big s5, struct Big s6, struct Big s7, struct Big s8,
+ struct Big s9, struct Big s10);
+struct Big P11(struct Big a1, struct Big a2, struct Big a3, struct Big a4,
+ struct Big a5, struct Big a6, struct Big a7, struct Big a8,
+ struct Big a9, struct Big a10) {
+ __attribute__((musttail)) return C11(a1, a2, a3, a4, a5, a6, a7, a8, a9, a10);
+}
+// COMMON-LABEL: define {{.*}} @P11(
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @C11(
+// COMMON-SAME: ptr {{.*}} %a1, ptr {{.*}} %a2, ptr {{.*}} %a3, ptr {{.*}} %a4
+// COMMON-SAME: ptr {{.*}} %a5, ptr {{.*}} %a6, ptr {{.*}} %a7, ptr {{.*}} %a8
+// COMMON-SAME: ptr {{.*}} %a9, ptr {{.*}} %a10
+
+// P12: over-aligned struct.
+struct __attribute__((aligned(32))) AlignedBig {
+ unsigned long long a, b, c, d;
+};
+struct AlignedBig C12(struct AlignedBig a);
+struct AlignedBig P12(struct AlignedBig a) {
+ __attribute__((musttail)) return C12(a);
+}
+// COMMON-LABEL: define {{.*}} @P12(
+// COMMON: musttail call {{.*}} @C12({{.*}}, ptr {{.*}} %a)
+
+// P17: same arg to three slots (generalization of P7).
+struct Big C17(struct Big x, struct Big y, struct Big z);
+struct Big P17(struct Big a, struct Big b, struct Big c) {
+ __attribute__((musttail)) return C17(a, a, a);
+}
+// COMMON-LABEL: define {{.*}} @P17(
+// At -O1 the optimizer may fold the per-slot reads, so only require that
+// each destination (%b, %c) gets a memcpy/memmove and that the musttail
+// call passes three distinct incoming pointers.
+// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %b,
+// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %c,
+// COMMON: musttail call {{.*}} @C17({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b, ptr {{.*}} %c)
diff --git a/clang/test/CodeGen/musttail-indirect-arg.cpp b/clang/test/CodeGen/musttail-indirect-arg.cpp
index 4c3203ca12cf2..3c8d037dd7d3b 100644
--- a/clang/test/CodeGen/musttail-indirect-arg.cpp
+++ b/clang/test/CodeGen/musttail-indirect-arg.cpp
@@ -1,56 +1,46 @@
-// Test that Clang forwards incoming Indirect parameters across musttail calls
-// for C++ struct-by-value arguments with trivially-copyable types. Companion to
-// musttail-indirect-arg.c (the C side of the same fix) and musttail-sret.cpp
-// (the SRet precedent in a96c14eeb8fc).
-//
-// C++ goes through a different EmitCallArg path than C: the call argument is a
-// CXXConstructExpr invoking the implicit copy constructor, which would
-// otherwise materialize an agg.tmp before EmitCall. For musttail with a
-// trivially-copyable parameter forwarded directly, the copy is elided so the
-// helper in EmitCall can forward the incoming llvm::Argument.
-
// RUN: %clang_cc1 -triple=riscv64-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
// RUN: %clang_cc1 -triple=aarch64-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
// RUN: %clang_cc1 -triple=loongarch64-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
// RUN: %clang_cc1 -triple=s390x-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
-// A trivially-copyable struct large enough to land on the indirect-arg path
-// on RV64, AArch64, LoongArch64, SystemZ.
+// C++ side of the musttail Indirect-arg fix. The call argument is typically
+// a CXXConstructExpr invoking the trivial copy constructor; EmitCallArg
+// detects the trivial-copy-from-DeclRefExpr case under musttail and hands
+// the source LValue to EmitCall so the general path engages. Non-trivial
+// copy or move constructors keep the existing agg.tmp path.
+
struct Big {
unsigned long long a, b, c, d;
};
-// Plain forward: caller(B) musttails callee(B). No agg.tmp copy, no
-// byval-temp; the incoming parameter %a is forwarded directly.
+// P1: simple forward.
struct Big C1(struct Big a);
struct Big P1(struct Big a) {
[[clang::musttail]] return C1(a);
}
// COMMON-LABEL: define {{.*}} @_Z2P13Big(
// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON: musttail call {{.*}} @_Z2C13Big({{.*}} %a)
+// COMMON: musttail call {{.*}} @_Z2C13Big({{.*}}, ptr {{.*}} %a)
-// Two args, same forwarding.
+// P2: two distinct args.
struct Big C2(struct Big a, struct Big b);
struct Big P2(struct Big a, struct Big b) {
[[clang::musttail]] return C2(a, b);
}
// COMMON-LABEL: define {{.*}} @_Z2P23BigS_(
-// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON: musttail call {{.*}} @_Z2C23BigS_({{.*}} %a, {{.*}} %b)
+// COMMON-NOT: llvm.memcpy
+// COMMON: musttail call {{.*}} @_Z2C23BigS_({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
-// Swapped args.
+// P3: swapped args.
struct Big C3(struct Big x, struct Big y);
struct Big P3(struct Big a, struct Big b) {
[[clang::musttail]] return C3(b, a);
}
// COMMON-LABEL: define {{.*}} @_Z2P33BigS_(
-// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON: musttail call {{.*}} @_Z2C33BigS_({{.*}} %b, {{.*}} %a)
-// Non-trivial copy constructor: the trivial-copy elision must NOT engage.
-// Existing path materializes the agg.tmp (the user-defined copy ctor has
-// observable behavior).
+// P4: non-trivial copy constructor. The trivial-copy gate must NOT engage;
+// the user-defined copy ctor IS called. Dangling-stack bug in this corner
+// remains (out of scope).
struct NonTrivial {
unsigned long long parts[4];
NonTrivial(const NonTrivial &);
@@ -60,23 +50,18 @@ NonTrivial P4(NonTrivial a) {
[[clang::musttail]] return C4(a);
}
// COMMON-LABEL: define {{.*}} @_Z2P410NonTrivial(
-// The user-defined copy ctor IS called (the agg.tmp pattern still happens):
// COMMON: call {{.*}} @_ZN10NonTrivialC1ERKS_
-// Caller modifies the parameter before the musttail. The trivial-copy
-// elision still engages because the source LValue is still the parameter;
-// any mutation flowed through the incoming pointer is observed by the
-// forwarded call.
+// P5: modify-then-forward.
struct Big C5(struct Big a);
struct Big P5(struct Big a) {
a.a += 1;
[[clang::musttail]] return C5(a);
}
// COMMON-LABEL: define {{.*}} @_Z2P53Big(
-// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON: musttail call {{.*}} @_Z2C53Big({{.*}} %a)
+// COMMON: musttail call {{.*}} @_Z2C53Big({{.*}}, ptr {{.*}} %a)
-// musttail behind a branch: trivial-copy elision must work across BBs.
+// P6: musttail behind a branch.
struct Big C6(struct Big a, int cond);
struct Big P6(struct Big a, int cond) {
if (cond)
@@ -84,36 +69,77 @@ struct Big P6(struct Big a, int cond) {
return a;
}
// COMMON-LABEL: define {{.*}} @_Z2P63Bigi(
-// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON: musttail call {{.*}} @_Z2C63Bigi({{.*}} %a,
+// COMMON: musttail call {{.*}} @_Z2C63Bigi({{.*}}, ptr {{.*}} %a,
-// Same Argument forwarded to two slots: both engage. Incoming Indirect
-// params are not noalias under the Linux C++ ABI so the dedup in the
-// helper does not fire and both slots forward %a directly.
+// P7: same arg to two slots. Slot 0 forwards %a; slot 1 memcpys *%a into the
+// i=1 incoming pointer %b and forwards %b.
struct Big C7(struct Big x, struct Big y);
struct Big P7(struct Big a, struct Big b) {
[[clang::musttail]] return C7(a, a);
}
// COMMON-LABEL: define {{.*}} @_Z2P73BigS_(
-// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON: musttail call {{.*}} @_Z2C73BigS_({{.*}} %a, {{.*}} %a)
+// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %b, ptr {{.*}} %a,
+// COMMON: musttail call {{.*}} @_Z2C73BigS_({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
-// Negative: source is a local, not a parameter. The trivial-copy elision
-// must NOT engage; the byval-temp pattern remains.
+// P8: local source. Copied into the incoming %a, then %a forwarded.
struct Big C8(struct Big a);
struct Big P8(struct Big a) {
struct Big local = {1, 2, 3, 4};
[[clang::musttail]] return C8(local);
}
// COMMON-LABEL: define {{.*}} @_Z2P83Big(
-// COMMON: = alloca
-// COMMON: musttail call {{.*}} @_Z2C83Big(
+// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %a, ptr {{.*}}
+// COMMON: musttail call {{.*}} @_Z2C83Big({{.*}}, ptr {{.*}} %a)
-// Non-musttail tail call: trivial-copy elision must NOT engage; the regular
-// agg.tmp copy is still emitted.
+// P9: non-musttail tail call (existing path).
struct Big C9(struct Big a);
struct Big P9(struct Big a) {
return C9(a);
}
// COMMON-LABEL: define {{.*}} @_Z2P93Big(
// COMMON-NOT: musttail
+
+// P10: mixed direct + indirect.
+struct Big C10(int x1, struct Big s1, int x2, struct Big s2);
+struct Big P10(int x1, struct Big s1, int x2, struct Big s2) {
+ [[clang::musttail]] return C10(x1, s1, x2, s2);
+}
+// COMMON-LABEL: define {{.*}} @_Z3P10i3BigiS_(
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @_Z3C10i3BigiS_({{.*}}, i32 {{.*}} %x1, ptr {{.*}} %s1, i32 {{.*}} %x2, ptr {{.*}} %s2)
+
+// P11: many args (stack spill on the target ABIs above).
+struct Big C11(struct Big s1, struct Big s2, struct Big s3, struct Big s4,
+ struct Big s5, struct Big s6, struct Big s7, struct Big s8,
+ struct Big s9, struct Big s10);
+struct Big P11(struct Big a1, struct Big a2, struct Big a3, struct Big a4,
+ struct Big a5, struct Big a6, struct Big a7, struct Big a8,
+ struct Big a9, struct Big a10) {
+ [[clang::musttail]] return C11(a1, a2, a3, a4, a5, a6, a7, a8, a9, a10);
+}
+// COMMON-LABEL: define {{.*}} @_Z3P113BigS_S_S_S_S_S_S_S_S_(
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @_Z3C113BigS_S_S_S_S_S_S_S_S_(
+
+// P16: member function. (P15 lambda case skipped: Sema currently rejects
+// musttail from a lambda's operator() to a non-member function, #119152.)
+struct S {
+ struct Big f(struct Big a);
+ struct Big P16(struct Big a);
+};
+struct Big S::P16(struct Big a) {
+ [[clang::musttail]] return f(a);
+}
+// COMMON-LABEL: define {{.*}} @_ZN1S3P16E3Big(
+// COMMON-NOT: = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @_ZN1S1fE3Big({{.*}}, ptr {{.*}}, ptr {{.*}} %a)
+
+// P17: same arg to three slots (generalization of P7).
+struct Big C17(struct Big x, struct Big y, struct Big z);
+struct Big P17(struct Big a, struct Big b, struct Big c) {
+ [[clang::musttail]] return C17(a, a, a);
+}
+// COMMON-LABEL: define {{.*}} @_Z3P173BigS_S_(
+// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %b,
+// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %c,
+// COMMON: musttail call {{.*}} @_Z3C173BigS_S_({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b, ptr {{.*}} %c)
>From 88607e81e6b5121ef54d8a2c27acd867274e3e63 Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Tue, 26 May 2026 12:45:47 +0200
Subject: [PATCH 07/21] [Clang][test] Strengthen P3 and P17 to catch
in-place-write regression
The previous P3 and P17 CHECK patterns were too permissive: they would
pass for the older single-phase emit that wrote in place during the
per-arg loop and would have miscompiled C(b, a) (both slots end up with
orig_b) and C(a, a, a) (callee sees aliased addresses).
Asserts the `%musttail.copy = alloca` scratch buffer in both tests.
The two-phase emit allocates at least one scratch when any non-identity
copy is needed; the single-phase emit allocates none. Verified the
scratch survives at -O0 through -O3 on RV64/AArch64/LoongArch64/s390x.
Runtime coverage of these cases lives in llvm-test-suite (#410); the
IR-level assertion catches the regression without depending on that
project.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.6 <noreply at anthropic.com>
---
clang/lib/CodeGen/CGCall.cpp | 6 ++--
clang/test/CodeGen/musttail-indirect-arg.c | 31 ++++++++++++++++----
clang/test/CodeGen/musttail-indirect-arg.cpp | 17 ++++++++++-
3 files changed, 43 insertions(+), 11 deletions(-)
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index 04774cc967071..77a5949d62020 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5154,8 +5154,7 @@ void CodeGenFunction::EmitCallArg(CallArgList &args, const Expr *E,
if (const auto *VD = dyn_cast<VarDecl>(DRE->getDecl())) {
if (VD->hasLocalStorage() ||
(isa<ParmVarDecl>(VD) &&
- VD->getDeclContext() ==
- dyn_cast<DeclContext>(CurCodeDecl))) {
+ VD->getDeclContext() == dyn_cast<DeclContext>(CurCodeDecl))) {
LValue L = EmitLValue(DRE);
assert(L.isSimple());
args.addUncopiedAggregate(L, type);
@@ -5722,8 +5721,7 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
LValue SrcLV = MakeAddrLValue(SrcAddr, Ty);
EmitAggregateCopy(ScratchLV, SrcLV, Ty,
AggValueSlot::DoesNotOverlap);
- LValue DstLV =
- MakeAddrLValue(Address(Dst, ElemTy, Align), Ty);
+ LValue DstLV = MakeAddrLValue(Address(Dst, ElemTy, Align), Ty);
MustTailIndirectCopies.push_back({ScratchLV, DstLV, Ty});
}
llvm::Value *Val = Dst;
diff --git a/clang/test/CodeGen/musttail-indirect-arg.c b/clang/test/CodeGen/musttail-indirect-arg.c
index fd82e9cf900c1..60dc28672ca74 100644
--- a/clang/test/CodeGen/musttail-indirect-arg.c
+++ b/clang/test/CodeGen/musttail-indirect-arg.c
@@ -35,14 +35,17 @@ struct Big P2(struct Big a, struct Big b) {
// COMMON: musttail call {{.*}} @C2({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
// P3: swap. Slot 0's dst is %a (positional), so the value of %b is copied
-// into %a; symmetrically for slot 1. Two-phase emit captures both sources
-// before either destination is overwritten.
+// into %a; symmetrically for slot 1. The scratch alloca catches the in-
+// place-write regression: v1 would emit memcpy(%a, %b); memcpy(%b, %a) and
+// both slots would end up with orig_b. v2 captures at least one source
+// into a scratch first; we assert the scratch is present.
struct Big C3(struct Big x, struct Big y);
struct Big P3(struct Big a, struct Big b) {
__attribute__((musttail)) return C3(b, a);
}
// COMMON-LABEL: define {{.*}} @P3(
-// COMMON: musttail call {{.*}} @C3({{.*}}, ptr {{.*}}, ptr {{.*}})
+// COMMON: %musttail.copy{{[0-9]*}} = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @C3({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
// P5: caller mutates the parameter before the musttail. The mutation lands
// at the incoming pointer the callee receives.
@@ -129,15 +132,31 @@ struct AlignedBig P12(struct AlignedBig a) {
// COMMON-LABEL: define {{.*}} @P12(
// COMMON: musttail call {{.*}} @C12({{.*}}, ptr {{.*}} %a)
+// P13: mixed source kinds within Indirect slots. Slot 0's source is a local;
+// slot 1's source is an incoming parameter. Both routes engage in the same
+// call: local-source case (would have dangled under v1's byval-temp path)
+// and forward case must coexist with two-phase ordering.
+struct Big C13(struct Big x, struct Big y);
+struct Big P13(struct Big a, struct Big b) {
+ struct Big local = {1, 2, 3, 4};
+ __attribute__((musttail)) return C13(local, a);
+}
+// COMMON-LABEL: define {{.*}} @P13(
+// COMMON-NOT: byval-temp
+// COMMON: %musttail.copy{{[0-9]*}} = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @C13({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
+
// P17: same arg to three slots (generalization of P7).
struct Big C17(struct Big x, struct Big y, struct Big z);
struct Big P17(struct Big a, struct Big b, struct Big c) {
__attribute__((musttail)) return C17(a, a, a);
}
// COMMON-LABEL: define {{.*}} @P17(
-// At -O1 the optimizer may fold the per-slot reads, so only require that
-// each destination (%b, %c) gets a memcpy/memmove and that the musttail
-// call passes three distinct incoming pointers.
+// The scratch alloca catches the in-place-write regression: v1 would emit
+// memcpy(%b, %a); memcpy(%c, %a) directly and pass a v1 check too. The
+// scratch presence asserts the two-phase emit is in use; the per-dst
+// memcpy/memmove and call-arg list pin the rest.
+// COMMON: %musttail.copy{{[0-9]*}} = alloca {{.*}}struct.Big
// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %b,
// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %c,
// COMMON: musttail call {{.*}} @C17({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b, ptr {{.*}} %c)
diff --git a/clang/test/CodeGen/musttail-indirect-arg.cpp b/clang/test/CodeGen/musttail-indirect-arg.cpp
index 3c8d037dd7d3b..979a4d1dd3a86 100644
--- a/clang/test/CodeGen/musttail-indirect-arg.cpp
+++ b/clang/test/CodeGen/musttail-indirect-arg.cpp
@@ -31,12 +31,15 @@ struct Big P2(struct Big a, struct Big b) {
// COMMON-NOT: llvm.memcpy
// COMMON: musttail call {{.*}} @_Z2C23BigS_({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
-// P3: swapped args.
+// P3: swap. Asserts the scratch alloca to catch the in-place-write
+// regression (see musttail-indirect-arg.c).
struct Big C3(struct Big x, struct Big y);
struct Big P3(struct Big a, struct Big b) {
[[clang::musttail]] return C3(b, a);
}
// COMMON-LABEL: define {{.*}} @_Z2P33BigS_(
+// COMMON: %musttail.copy{{[0-9]*}} = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @_Z2C33BigS_({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
// P4: non-trivial copy constructor. The trivial-copy gate must NOT engage;
// the user-defined copy ctor IS called. Dangling-stack bug in this corner
@@ -134,12 +137,24 @@ struct Big S::P16(struct Big a) {
// COMMON-NOT: = alloca {{.*}}struct.Big
// COMMON: musttail call {{.*}} @_ZN1S1fE3Big({{.*}}, ptr {{.*}}, ptr {{.*}} %a)
+// P13: mixed source kinds (local + incoming parameter).
+struct Big C13(struct Big x, struct Big y);
+struct Big P13(struct Big a, struct Big b) {
+ struct Big local = {1, 2, 3, 4};
+ [[clang::musttail]] return C13(local, a);
+}
+// COMMON-LABEL: define {{.*}} @_Z3P133BigS_(
+// COMMON-NOT: byval-temp
+// COMMON: %musttail.copy{{[0-9]*}} = alloca {{.*}}struct.Big
+// COMMON: musttail call {{.*}} @_Z3C133BigS_({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
+
// P17: same arg to three slots (generalization of P7).
struct Big C17(struct Big x, struct Big y, struct Big z);
struct Big P17(struct Big a, struct Big b, struct Big c) {
[[clang::musttail]] return C17(a, a, a);
}
// COMMON-LABEL: define {{.*}} @_Z3P173BigS_S_(
+// COMMON: %musttail.copy{{[0-9]*}} = alloca {{.*}}struct.Big
// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %b,
// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %c,
// COMMON: musttail call {{.*}} @_Z3C173BigS_S_({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b, ptr {{.*}} %c)
>From d12920ee70bfff9b0cbca9dc126923ae41e591ef Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Fri, 5 Jun 2026 20:07:34 +0200
Subject: [PATCH 08/21] [Clang] Diagnose musttail Indirect args with no
addressable source
The previous fall-through under musttail + Indirect silently re-entered
the byval-temp path when the source had no LValue and no aggregate
RValue address (rare; e.g. a scalar RValue the ABI classifies as
Indirect). That re-introduced the dangling-alloca bug this PR fixes.
Refuse the case cleanly with an error. Codegen still completes so the
diagnostic surfaces normally; the queued error prevents the broken IR
from reaching the backend, matching the err_musttail_noexcept_mismatch
pattern earlier in the function.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.6 <noreply at anthropic.com>
---
clang/include/clang/Basic/DiagnosticCommonKinds.td | 3 +++
clang/lib/CodeGen/CGCall.cpp | 8 +++++++-
2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/clang/include/clang/Basic/DiagnosticCommonKinds.td b/clang/include/clang/Basic/DiagnosticCommonKinds.td
index bdbbaffe0a6e1..f18a3c7927ec9 100644
--- a/clang/include/clang/Basic/DiagnosticCommonKinds.td
+++ b/clang/include/clang/Basic/DiagnosticCommonKinds.td
@@ -388,6 +388,9 @@ def err_aix_musttail_unsupported: Error<
"'musttail' attribute is not supported on AIX">;
def err_musttail_noexcept_mismatch: Error<
"'musttail' in a noexcept function requires a noexcept callee">;
+def err_musttail_unsupported_indirect_arg: Error<
+ "'musttail' call requires passing an argument by reference, but the source "
+ "does not have an addressable storage and would alias the caller's frame">;
// Source manager
def err_cannot_open_file : Error<"cannot open file '%0': %1">, DefaultFatal;
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index 77a5949d62020..1704f3d456881 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5730,7 +5730,13 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
IRCallArgs[FirstIRArg] = Val;
break;
}
- // No clean source address (rare): fall through to byval-temp.
+ // No addressable source for this Indirect arg (rare; e.g. a scalar
+ // RValue the ABI classifies as Indirect). The fall-through below
+ // would create a current-frame byval-temp that dangles past the
+ // tail-call teardown. Refuse cleanly; codegen continues producing
+ // IR but the error prevents it from reaching the backend.
+ CGM.getDiags().Report(MustTailCall->getBeginLoc(),
+ diag::err_musttail_unsupported_indirect_arg);
}
if (I->isAggregate()) {
>From f008bf562b7bdc59a0d4dd8479b982c1ec8299af Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Sat, 6 Jun 2026 09:24:18 +0200
Subject: [PATCH 09/21] [Clang][test] Make musttail-indirect-arg scratch checks
SROA-robust
The CHECK lines that pinned the scratch as 'alloca struct.Big' fail when
SROA promotes the alloca into an SSA load/store of a vector type
(observed in CI builds against current main). The scratch variable name
%musttail.copy<N> survives in either shape, so match the name with any
suffix and drop the per-slot memcpy/memmove asserts for P17 that SROA
also collapses. The scratch presence is the load-bearing check; the
one-pass-no-scratch regression still fails it.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.6 <noreply at anthropic.com>
---
clang/test/CodeGen/musttail-indirect-arg.c | 14 +++++---------
clang/test/CodeGen/musttail-indirect-arg.cpp | 8 +++-----
2 files changed, 8 insertions(+), 14 deletions(-)
diff --git a/clang/test/CodeGen/musttail-indirect-arg.c b/clang/test/CodeGen/musttail-indirect-arg.c
index 60dc28672ca74..b2916d75adde2 100644
--- a/clang/test/CodeGen/musttail-indirect-arg.c
+++ b/clang/test/CodeGen/musttail-indirect-arg.c
@@ -44,7 +44,7 @@ struct Big P3(struct Big a, struct Big b) {
__attribute__((musttail)) return C3(b, a);
}
// COMMON-LABEL: define {{.*}} @P3(
-// COMMON: %musttail.copy{{[0-9]*}} = alloca {{.*}}struct.Big
+// COMMON: %musttail.copy{{[0-9.a-z]*}} =
// COMMON: musttail call {{.*}} @C3({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
// P5: caller mutates the parameter before the musttail. The mutation lands
@@ -143,7 +143,7 @@ struct Big P13(struct Big a, struct Big b) {
}
// COMMON-LABEL: define {{.*}} @P13(
// COMMON-NOT: byval-temp
-// COMMON: %musttail.copy{{[0-9]*}} = alloca {{.*}}struct.Big
+// COMMON: %musttail.copy{{[0-9.a-z]*}} =
// COMMON: musttail call {{.*}} @C13({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
// P17: same arg to three slots (generalization of P7).
@@ -152,11 +152,7 @@ struct Big P17(struct Big a, struct Big b, struct Big c) {
__attribute__((musttail)) return C17(a, a, a);
}
// COMMON-LABEL: define {{.*}} @P17(
-// The scratch alloca catches the in-place-write regression: v1 would emit
-// memcpy(%b, %a); memcpy(%c, %a) directly and pass a v1 check too. The
-// scratch presence asserts the two-phase emit is in use; the per-dst
-// memcpy/memmove and call-arg list pin the rest.
-// COMMON: %musttail.copy{{[0-9]*}} = alloca {{.*}}struct.Big
-// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %b,
-// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %c,
+// The scratch presence catches the in-place-write regression: a one-pass
+// emit would memcpy(%b, %a); memcpy(%c, %a) directly with no scratch.
+// COMMON: %musttail.copy{{[0-9.a-z]*}} =
// COMMON: musttail call {{.*}} @C17({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b, ptr {{.*}} %c)
diff --git a/clang/test/CodeGen/musttail-indirect-arg.cpp b/clang/test/CodeGen/musttail-indirect-arg.cpp
index 979a4d1dd3a86..a381dff101af2 100644
--- a/clang/test/CodeGen/musttail-indirect-arg.cpp
+++ b/clang/test/CodeGen/musttail-indirect-arg.cpp
@@ -38,7 +38,7 @@ struct Big P3(struct Big a, struct Big b) {
[[clang::musttail]] return C3(b, a);
}
// COMMON-LABEL: define {{.*}} @_Z2P33BigS_(
-// COMMON: %musttail.copy{{[0-9]*}} = alloca {{.*}}struct.Big
+// COMMON: %musttail.copy{{[0-9.a-z]*}} =
// COMMON: musttail call {{.*}} @_Z2C33BigS_({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
// P4: non-trivial copy constructor. The trivial-copy gate must NOT engage;
@@ -145,7 +145,7 @@ struct Big P13(struct Big a, struct Big b) {
}
// COMMON-LABEL: define {{.*}} @_Z3P133BigS_(
// COMMON-NOT: byval-temp
-// COMMON: %musttail.copy{{[0-9]*}} = alloca {{.*}}struct.Big
+// COMMON: %musttail.copy{{[0-9.a-z]*}} =
// COMMON: musttail call {{.*}} @_Z3C133BigS_({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
// P17: same arg to three slots (generalization of P7).
@@ -154,7 +154,5 @@ struct Big P17(struct Big a, struct Big b, struct Big c) {
[[clang::musttail]] return C17(a, a, a);
}
// COMMON-LABEL: define {{.*}} @_Z3P173BigS_S_(
-// COMMON: %musttail.copy{{[0-9]*}} = alloca {{.*}}struct.Big
-// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %b,
-// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %c,
+// COMMON: %musttail.copy{{[0-9.a-z]*}} =
// COMMON: musttail call {{.*}} @_Z3C173BigS_S_({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b, ptr {{.*}} %c)
>From 3a588362fe78741a6158d4cc4b78d62fb140b325 Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Fri, 26 Jun 2026 09:21:18 +0200
Subject: [PATCH 10/21] [Clang][test] Pin slot operands and swap data-flow in
musttail-indirect-arg
Replace each per-operand `{{.*}}` with `{{[^,]*}}` so a wildcard cannot
span a comma and hide an aliased slot or a smuggled scratch pointer.
Pin the P3 swap and P17 triplicate data-flow with a FileCheck variable:
the value loaded from %a is the one stored into the destination slot, and
the memmove direction is fixed. This catches an in-place clobber the prior
"a scratch exists" check accepted. IR is identical across all four triples;
each new directive was confirmed to fail on the corresponding bug shape.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
clang/test/CodeGen/musttail-indirect-arg.c | 52 ++++++++++----------
clang/test/CodeGen/musttail-indirect-arg.cpp | 41 ++++++++-------
2 files changed, 50 insertions(+), 43 deletions(-)
diff --git a/clang/test/CodeGen/musttail-indirect-arg.c b/clang/test/CodeGen/musttail-indirect-arg.c
index b2916d75adde2..b6073bd956ba0 100644
--- a/clang/test/CodeGen/musttail-indirect-arg.c
+++ b/clang/test/CodeGen/musttail-indirect-arg.c
@@ -22,7 +22,7 @@ struct Big P1(struct Big a) {
}
// COMMON-LABEL: define {{.*}} @P1(
// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON: musttail call {{.*}} @C1({{.*}}, ptr {{.*}} %a)
+// COMMON: musttail call {{.*}} @C1({{.*}}, ptr {{[^,]*}} %a)
// P2: two distinct incoming sources.
struct Big C2(struct Big a, struct Big b);
@@ -32,20 +32,20 @@ struct Big P2(struct Big a, struct Big b) {
// COMMON-LABEL: define {{.*}} @P2(
// COMMON-NOT: = alloca {{.*}}struct.Big
// COMMON-NOT: llvm.memcpy
-// COMMON: musttail call {{.*}} @C2({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
+// COMMON: musttail call {{.*}} @C2({{.*}}, ptr {{[^,]*}} %a, ptr {{[^,]*}} %b)
-// P3: swap. Slot 0's dst is %a (positional), so the value of %b is copied
-// into %a; symmetrically for slot 1. The scratch alloca catches the in-
-// place-write regression: v1 would emit memcpy(%a, %b); memcpy(%b, %a) and
-// both slots would end up with orig_b. v2 captures at least one source
-// into a scratch first; we assert the scratch is present.
+// P3: swap. Pin the data flow: %a is captured before %b overwrites it, and
+// the saved %a lands in %b. An in-place memmove(%a,%b);memmove(%b,%a) would
+// drop orig_a, so both slots would read orig_b.
struct Big C3(struct Big x, struct Big y);
struct Big P3(struct Big a, struct Big b) {
__attribute__((musttail)) return C3(b, a);
}
// COMMON-LABEL: define {{.*}} @P3(
-// COMMON: %musttail.copy{{[0-9.a-z]*}} =
-// COMMON: musttail call {{.*}} @C3({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
+// COMMON: [[SAVED:%musttail.copy[0-9.a-z]*]] = load {{.*}}, ptr %a,
+// COMMON: @llvm.mem{{(cpy|move)}}{{.*}}(ptr {{[^,]*}} %a, ptr {{[^,]*}} %b,
+// COMMON: store {{.*}} [[SAVED]], ptr %b,
+// COMMON: musttail call {{.*}} @C3({{.*}}, ptr {{[^,]*}} %a, ptr {{[^,]*}} %b)
// P5: caller mutates the parameter before the musttail. The mutation lands
// at the incoming pointer the callee receives.
@@ -55,7 +55,7 @@ struct Big P5(struct Big a) {
__attribute__((musttail)) return C5(a);
}
// COMMON-LABEL: define {{.*}} @P5(
-// COMMON: musttail call {{.*}} @C5({{.*}}, ptr {{.*}} %a)
+// COMMON: musttail call {{.*}} @C5({{.*}}, ptr {{[^,]*}} %a)
// P6: musttail in a non-entry block.
struct Big C6(struct Big a, int cond);
@@ -65,7 +65,7 @@ struct Big P6(struct Big a, int cond) {
return a;
}
// COMMON-LABEL: define {{.*}} @P6(
-// COMMON: musttail call {{.*}} @C6({{.*}}, ptr {{.*}} %a,
+// COMMON: musttail call {{.*}} @C6({{.*}}, ptr {{[^,]*}} %a,
// P7: same arg to two slots. C ABI requires distinct storage per by-value
// param, so slot 1 cannot share %a's pointer. Slot 0 forwards %a; slot 1
@@ -75,8 +75,8 @@ struct Big P7(struct Big a, struct Big b) {
__attribute__((musttail)) return C7(a, a);
}
// COMMON-LABEL: define {{.*}} @P7(
-// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %b, ptr {{.*}} %a,
-// COMMON: musttail call {{.*}} @C7({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
+// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{[^,]*}} %b, ptr {{[^,]*}} %a,
+// COMMON: musttail call {{.*}} @C7({{.*}}, ptr {{[^,]*}} %a, ptr {{[^,]*}} %b)
// P8: local source. The local lives in our frame; copy it into %a, forward %a.
struct Big C8(struct Big a);
@@ -85,8 +85,8 @@ struct Big P8(struct Big a) {
__attribute__((musttail)) return C8(local);
}
// COMMON-LABEL: define {{.*}} @P8(
-// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %a, ptr {{.*}}
-// COMMON: musttail call {{.*}} @C8({{.*}}, ptr {{.*}} %a)
+// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{[^,]*}} %a, ptr {{.*}}
+// COMMON: musttail call {{.*}} @C8({{.*}}, ptr {{[^,]*}} %a)
// P9: non-musttail tail call (existing path).
struct Big C9(struct Big a);
@@ -103,7 +103,7 @@ struct Big P10(int x1, struct Big s1, int x2, struct Big s2) {
}
// COMMON-LABEL: define {{.*}} @P10(
// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON: musttail call {{.*}} @C10({{.*}}, i32 {{.*}} %x1, ptr {{.*}} %s1, i32 {{.*}} %x2, ptr {{.*}} %s2)
+// COMMON: musttail call {{.*}} @C10({{.*}}, i32 {{.*}} %x1, ptr {{[^,]*}} %s1, i32 {{.*}} %x2, ptr {{[^,]*}} %s2)
// P11: many args, including stack-spilled ones on the target ABIs above.
struct Big C11(struct Big s1, struct Big s2, struct Big s3, struct Big s4,
@@ -117,9 +117,9 @@ struct Big P11(struct Big a1, struct Big a2, struct Big a3, struct Big a4,
// COMMON-LABEL: define {{.*}} @P11(
// COMMON-NOT: = alloca {{.*}}struct.Big
// COMMON: musttail call {{.*}} @C11(
-// COMMON-SAME: ptr {{.*}} %a1, ptr {{.*}} %a2, ptr {{.*}} %a3, ptr {{.*}} %a4
-// COMMON-SAME: ptr {{.*}} %a5, ptr {{.*}} %a6, ptr {{.*}} %a7, ptr {{.*}} %a8
-// COMMON-SAME: ptr {{.*}} %a9, ptr {{.*}} %a10
+// COMMON-SAME: ptr {{[^,]*}} %a1, ptr {{[^,]*}} %a2, ptr {{[^,]*}} %a3, ptr {{[^,]*}} %a4
+// COMMON-SAME: ptr {{[^,]*}} %a5, ptr {{[^,]*}} %a6, ptr {{[^,]*}} %a7, ptr {{[^,]*}} %a8
+// COMMON-SAME: ptr {{[^,]*}} %a9, ptr {{[^,]*}} %a10
// P12: over-aligned struct.
struct __attribute__((aligned(32))) AlignedBig {
@@ -130,7 +130,7 @@ struct AlignedBig P12(struct AlignedBig a) {
__attribute__((musttail)) return C12(a);
}
// COMMON-LABEL: define {{.*}} @P12(
-// COMMON: musttail call {{.*}} @C12({{.*}}, ptr {{.*}} %a)
+// COMMON: musttail call {{.*}} @C12({{.*}}, ptr {{[^,]*}} %a)
// P13: mixed source kinds within Indirect slots. Slot 0's source is a local;
// slot 1's source is an incoming parameter. Both routes engage in the same
@@ -144,7 +144,7 @@ struct Big P13(struct Big a, struct Big b) {
// COMMON-LABEL: define {{.*}} @P13(
// COMMON-NOT: byval-temp
// COMMON: %musttail.copy{{[0-9.a-z]*}} =
-// COMMON: musttail call {{.*}} @C13({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
+// COMMON: musttail call {{.*}} @C13({{.*}}, ptr {{[^,]*}} %a, ptr {{[^,]*}} %b)
// P17: same arg to three slots (generalization of P7).
struct Big C17(struct Big x, struct Big y, struct Big z);
@@ -152,7 +152,9 @@ struct Big P17(struct Big a, struct Big b, struct Big c) {
__attribute__((musttail)) return C17(a, a, a);
}
// COMMON-LABEL: define {{.*}} @P17(
-// The scratch presence catches the in-place-write regression: a one-pass
-// emit would memcpy(%b, %a); memcpy(%c, %a) directly with no scratch.
-// COMMON: %musttail.copy{{[0-9.a-z]*}} =
-// COMMON: musttail call {{.*}} @C17({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b, ptr {{.*}} %c)
+// Both copied slots take their value from %a: %b via the memmove, %c via the
+// captured load. Neither sources from the other copied slot.
+// COMMON: [[SAVED:%musttail.copy[0-9.a-z]*]] = load {{.*}}, ptr %a,
+// COMMON: @llvm.mem{{(cpy|move)}}{{.*}}(ptr {{[^,]*}} %b, ptr {{[^,]*}} %a,
+// COMMON: store {{.*}} [[SAVED]], ptr %c,
+// COMMON: musttail call {{.*}} @C17({{.*}}, ptr {{[^,]*}} %a, ptr {{[^,]*}} %b, ptr {{[^,]*}} %c)
diff --git a/clang/test/CodeGen/musttail-indirect-arg.cpp b/clang/test/CodeGen/musttail-indirect-arg.cpp
index a381dff101af2..2c5989a81d24c 100644
--- a/clang/test/CodeGen/musttail-indirect-arg.cpp
+++ b/clang/test/CodeGen/musttail-indirect-arg.cpp
@@ -20,7 +20,7 @@ struct Big P1(struct Big a) {
}
// COMMON-LABEL: define {{.*}} @_Z2P13Big(
// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON: musttail call {{.*}} @_Z2C13Big({{.*}}, ptr {{.*}} %a)
+// COMMON: musttail call {{.*}} @_Z2C13Big({{.*}}, ptr {{[^,]*}} %a)
// P2: two distinct args.
struct Big C2(struct Big a, struct Big b);
@@ -29,17 +29,19 @@ struct Big P2(struct Big a, struct Big b) {
}
// COMMON-LABEL: define {{.*}} @_Z2P23BigS_(
// COMMON-NOT: llvm.memcpy
-// COMMON: musttail call {{.*}} @_Z2C23BigS_({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
+// COMMON: musttail call {{.*}} @_Z2C23BigS_({{.*}}, ptr {{[^,]*}} %a, ptr {{[^,]*}} %b)
-// P3: swap. Asserts the scratch alloca to catch the in-place-write
-// regression (see musttail-indirect-arg.c).
+// P3: swap. Pin the data flow (see musttail-indirect-arg.c): %a is captured
+// before %b overwrites it, and the saved %a lands in %b.
struct Big C3(struct Big x, struct Big y);
struct Big P3(struct Big a, struct Big b) {
[[clang::musttail]] return C3(b, a);
}
// COMMON-LABEL: define {{.*}} @_Z2P33BigS_(
-// COMMON: %musttail.copy{{[0-9.a-z]*}} =
-// COMMON: musttail call {{.*}} @_Z2C33BigS_({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
+// COMMON: [[SAVED:%musttail.copy[0-9.a-z]*]] = load {{.*}}, ptr %a,
+// COMMON: @llvm.mem{{(cpy|move)}}{{.*}}(ptr {{[^,]*}} %a, ptr {{[^,]*}} %b,
+// COMMON: store {{.*}} [[SAVED]], ptr %b,
+// COMMON: musttail call {{.*}} @_Z2C33BigS_({{.*}}, ptr {{[^,]*}} %a, ptr {{[^,]*}} %b)
// P4: non-trivial copy constructor. The trivial-copy gate must NOT engage;
// the user-defined copy ctor IS called. Dangling-stack bug in this corner
@@ -62,7 +64,7 @@ struct Big P5(struct Big a) {
[[clang::musttail]] return C5(a);
}
// COMMON-LABEL: define {{.*}} @_Z2P53Big(
-// COMMON: musttail call {{.*}} @_Z2C53Big({{.*}}, ptr {{.*}} %a)
+// COMMON: musttail call {{.*}} @_Z2C53Big({{.*}}, ptr {{[^,]*}} %a)
// P6: musttail behind a branch.
struct Big C6(struct Big a, int cond);
@@ -72,7 +74,7 @@ struct Big P6(struct Big a, int cond) {
return a;
}
// COMMON-LABEL: define {{.*}} @_Z2P63Bigi(
-// COMMON: musttail call {{.*}} @_Z2C63Bigi({{.*}}, ptr {{.*}} %a,
+// COMMON: musttail call {{.*}} @_Z2C63Bigi({{.*}}, ptr {{[^,]*}} %a,
// P7: same arg to two slots. Slot 0 forwards %a; slot 1 memcpys *%a into the
// i=1 incoming pointer %b and forwards %b.
@@ -81,8 +83,8 @@ struct Big P7(struct Big a, struct Big b) {
[[clang::musttail]] return C7(a, a);
}
// COMMON-LABEL: define {{.*}} @_Z2P73BigS_(
-// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %b, ptr {{.*}} %a,
-// COMMON: musttail call {{.*}} @_Z2C73BigS_({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
+// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{[^,]*}} %b, ptr {{[^,]*}} %a,
+// COMMON: musttail call {{.*}} @_Z2C73BigS_({{.*}}, ptr {{[^,]*}} %a, ptr {{[^,]*}} %b)
// P8: local source. Copied into the incoming %a, then %a forwarded.
struct Big C8(struct Big a);
@@ -91,8 +93,8 @@ struct Big P8(struct Big a) {
[[clang::musttail]] return C8(local);
}
// COMMON-LABEL: define {{.*}} @_Z2P83Big(
-// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{.*}} %a, ptr {{.*}}
-// COMMON: musttail call {{.*}} @_Z2C83Big({{.*}}, ptr {{.*}} %a)
+// COMMON: llvm.mem{{(cpy|move)}}{{.*}}(ptr {{[^,]*}} %a, ptr {{.*}}
+// COMMON: musttail call {{.*}} @_Z2C83Big({{.*}}, ptr {{[^,]*}} %a)
// P9: non-musttail tail call (existing path).
struct Big C9(struct Big a);
@@ -109,7 +111,7 @@ struct Big P10(int x1, struct Big s1, int x2, struct Big s2) {
}
// COMMON-LABEL: define {{.*}} @_Z3P10i3BigiS_(
// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON: musttail call {{.*}} @_Z3C10i3BigiS_({{.*}}, i32 {{.*}} %x1, ptr {{.*}} %s1, i32 {{.*}} %x2, ptr {{.*}} %s2)
+// COMMON: musttail call {{.*}} @_Z3C10i3BigiS_({{.*}}, i32 {{.*}} %x1, ptr {{[^,]*}} %s1, i32 {{.*}} %x2, ptr {{[^,]*}} %s2)
// P11: many args (stack spill on the target ABIs above).
struct Big C11(struct Big s1, struct Big s2, struct Big s3, struct Big s4,
@@ -135,7 +137,7 @@ struct Big S::P16(struct Big a) {
}
// COMMON-LABEL: define {{.*}} @_ZN1S3P16E3Big(
// COMMON-NOT: = alloca {{.*}}struct.Big
-// COMMON: musttail call {{.*}} @_ZN1S1fE3Big({{.*}}, ptr {{.*}}, ptr {{.*}} %a)
+// COMMON: musttail call {{.*}} @_ZN1S1fE3Big({{.*}}, ptr {{.*}}, ptr {{[^,]*}} %a)
// P13: mixed source kinds (local + incoming parameter).
struct Big C13(struct Big x, struct Big y);
@@ -146,13 +148,16 @@ struct Big P13(struct Big a, struct Big b) {
// COMMON-LABEL: define {{.*}} @_Z3P133BigS_(
// COMMON-NOT: byval-temp
// COMMON: %musttail.copy{{[0-9.a-z]*}} =
-// COMMON: musttail call {{.*}} @_Z3C133BigS_({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b)
+// COMMON: musttail call {{.*}} @_Z3C133BigS_({{.*}}, ptr {{[^,]*}} %a, ptr {{[^,]*}} %b)
-// P17: same arg to three slots (generalization of P7).
+// P17: same arg to three slots (generalization of P7). Both copied slots
+// take their value from %a: %b via the memmove, %c via the captured load.
struct Big C17(struct Big x, struct Big y, struct Big z);
struct Big P17(struct Big a, struct Big b, struct Big c) {
[[clang::musttail]] return C17(a, a, a);
}
// COMMON-LABEL: define {{.*}} @_Z3P173BigS_S_(
-// COMMON: %musttail.copy{{[0-9.a-z]*}} =
-// COMMON: musttail call {{.*}} @_Z3C173BigS_S_({{.*}}, ptr {{.*}} %a, ptr {{.*}} %b, ptr {{.*}} %c)
+// COMMON: [[SAVED:%musttail.copy[0-9.a-z]*]] = load {{.*}}, ptr %a,
+// COMMON: @llvm.mem{{(cpy|move)}}{{.*}}(ptr {{[^,]*}} %b, ptr {{[^,]*}} %a,
+// COMMON: store {{.*}} [[SAVED]], ptr %c,
+// COMMON: musttail call {{.*}} @_Z3C173BigS_S_({{.*}}, ptr {{[^,]*}} %a, ptr {{[^,]*}} %b, ptr {{[^,]*}} %c)
>From 6f56dba0020a926c7455847f786cae1aac57e820 Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Sat, 27 Jun 2026 11:45:50 +0200
Subject: [PATCH 11/21] [Clang] Drop redundant freeze of forwarded musttail
Indirect pointer
The musttail Indirect-arg path forwards the i-th incoming parameter
pointer (CurFn->arg_begin()+FirstIRArg) as the call slot. That pointer
is an SSA Argument, never poison, so the maybe_undef freeze on it was
dead defensiveness. Pass it directly.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.6 <noreply at anthropic.com>
---
clang/lib/CodeGen/CGCall.cpp | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index 39729a3cd31f6..7c43580777a94 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5840,10 +5840,8 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
LValue DstLV = MakeAddrLValue(Address(Dst, ElemTy, Align), Ty);
MustTailIndirectCopies.push_back({ScratchLV, DstLV, Ty});
}
- llvm::Value *Val = Dst;
- if (ArgHasMaybeUndefAttr)
- Val = Builder.CreateFreeze(Val);
- IRCallArgs[FirstIRArg] = Val;
+ // No freeze: Dst is an incoming parameter pointer, never poison.
+ IRCallArgs[FirstIRArg] = Dst;
break;
}
// No addressable source for this Indirect arg (rare; e.g. a scalar
>From cb7efcbc3799c686acf08c1c1f597adb9abb90e5 Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Sat, 27 Jun 2026 11:46:19 +0200
Subject: [PATCH 12/21] [Clang] Forward trivially-copyable by-value args beyond
musttail
The musttail Indirect-arg fix forwarded a trivially-copyable record
argument's source LValue to EmitCall instead of materializing an
agg.tmp, but gated it on MustTailCall. The C path right above it (the
CK_LValueToRValue case) already forwards unconditionally, so the gate
was the only thing keeping the C++ trivial copy/move case from doing
the same.
Drop the MustTailCall and storage gates. A trivial copy/move
constructor whose source is a variable now forwards that variable's
LValue for any call; EmitCall makes the real copy at the Indirect/byval
boundary, dropping the redundant agg.tmp. A same-type guard rejects a
stripped derived-to-base cast that would otherwise slice at a wrong
offset.
The source is kept to a direct variable reference (the common case);
broader lvalue sources stay on the existing path. Test updates reflect
the elided agg.tmp/copy.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.6 <noreply at anthropic.com>
---
clang/lib/CodeGen/CGCall.cpp | 38 +++++++-------
.../AArch64/struct-coerce-using-ptr.cpp | 40 ++++++---------
clang/test/CodeGen/aapcs64-align.cpp | 2 +-
clang/test/CodeGenCXX/amdgcn-func-arg.cpp | 16 +++---
.../CodeGenCXX/assign-construct-memcpy.cpp | 8 ++-
clang/test/CodeGenCXX/cxx2b-deducing-this.cpp | 2 -
.../pr40771-ctad-with-lambda-copy-capture.cpp | 2 -
.../CodeGenCXX/trivial-copy-arg-forward.cpp | 50 +++++++++++++++++++
clang/test/CodeGenCXX/varargs.cpp | 4 +-
9 files changed, 98 insertions(+), 64 deletions(-)
create mode 100644 clang/test/CodeGenCXX/trivial-copy-arg-forward.cpp
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index 7c43580777a94..f699c3498e14e 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5255,27 +5255,29 @@ void CodeGenFunction::EmitCallArg(CallArgList &args, const Expr *E,
}
}
- // Under musttail, hand a trivially-copyable record source's LValue to
- // EmitCall rather than materializing an agg.tmp. EmitCall's Indirect path
- // routes it via the matching incoming parameter, which survives the tail
- // call. Limited to params and locals: globals and captures don't have the
- // dangle issue and the existing path may be more efficient for them.
- if (HasAggregateEvalKind && MustTailCall && type->isRecordType() &&
+ // C++ analog of the CK_LValueToRValue case above: a trivial copy/move
+ // constructor from a variable forwards the source LValue instead of
+ // materializing an agg.tmp. EmitCall makes the real copy at the
+ // Indirect/byval boundary. Under musttail this also keeps the value in
+ // storage that survives the tail call.
+ if (HasAggregateEvalKind && type->isRecordType() &&
type.isTriviallyCopyableType(getContext())) {
if (const auto *CCE = dyn_cast<CXXConstructExpr>(E)) {
- if (CCE->getConstructor()->isCopyOrMoveConstructor() &&
- CCE->getConstructor()->isTrivial() && CCE->getNumArgs() == 1) {
+ const CXXConstructorDecl *Ctor = CCE->getConstructor();
+ if (Ctor->isCopyOrMoveConstructor() && Ctor->isTrivial() &&
+ CCE->getNumArgs() == 1) {
const Expr *Source = CCE->getArg(0)->IgnoreParenImpCasts();
- if (const auto *DRE = dyn_cast<DeclRefExpr>(Source)) {
- if (const auto *VD = dyn_cast<VarDecl>(DRE->getDecl())) {
- if (VD->hasLocalStorage() ||
- (isa<ParmVarDecl>(VD) &&
- VD->getDeclContext() == dyn_cast<DeclContext>(CurCodeDecl))) {
- LValue L = EmitLValue(DRE);
- assert(L.isSimple());
- args.addUncopiedAggregate(L, type);
- return;
- }
+ // Same-type guard: a stripped derived-to-base cast would forward a
+ // derived lvalue into a base-typed slot and slice at a wrong offset.
+ // Exclude hlsl_constant sources, as the CK_LValueToRValue path does.
+ if (const auto *DRE = dyn_cast<DeclRefExpr>(Source);
+ DRE && isa<VarDecl>(DRE->getDecl()) &&
+ Source->getType().getAddressSpace() != LangAS::hlsl_constant &&
+ getContext().hasSameUnqualifiedType(Source->getType(), type)) {
+ LValue L = EmitLValue(DRE);
+ if (L.isSimple()) {
+ args.addUncopiedAggregate(L, type);
+ return;
}
}
}
diff --git a/clang/test/CodeGen/AArch64/struct-coerce-using-ptr.cpp b/clang/test/CodeGen/AArch64/struct-coerce-using-ptr.cpp
index 42128c18412a5..64fc65628d453 100644
--- a/clang/test/CodeGen/AArch64/struct-coerce-using-ptr.cpp
+++ b/clang/test/CodeGen/AArch64/struct-coerce-using-ptr.cpp
@@ -139,7 +139,7 @@ struct Srp {
// CHECK-A64-NEXT: [[S:%.*]] = alloca [[STRUCT_SRP:%.*]], align 8
// CHECK-A64-NEXT: store [2 x ptr] [[S_COERCE]], ptr [[S]], align 8
// CHECK-A64-NEXT: [[X:%.*]] = getelementptr inbounds nuw [[STRUCT_SRP]], ptr [[S]], i32 0, i32 0
-// CHECK-A64-NEXT: [[TMP0:%.*]] = load ptr, ptr [[X]], align 8, !nonnull [[META2:![0-9]+]], !align [[META3:![0-9]+]]
+// CHECK-A64-NEXT: [[TMP0:%.*]] = load ptr, ptr [[X]], align 8, !nonnull [[META1:![0-9]+]], !align [[META2:![0-9]+]]
// CHECK-A64-NEXT: store i32 1, ptr [[TMP0]], align 4
// CHECK-A64-NEXT: ret void
//
@@ -149,7 +149,7 @@ struct Srp {
// CHECK-A64_32-NEXT: [[S:%.*]] = alloca [[STRUCT_SRP:%.*]], align 4
// CHECK-A64_32-NEXT: store i64 [[S_COERCE]], ptr [[S]], align 4
// CHECK-A64_32-NEXT: [[X:%.*]] = getelementptr inbounds nuw [[STRUCT_SRP]], ptr [[S]], i32 0, i32 0
-// CHECK-A64_32-NEXT: [[TMP0:%.*]] = load ptr, ptr [[X]], align 4, !nonnull [[META2:![0-9]+]], !align [[META3:![0-9]+]]
+// CHECK-A64_32-NEXT: [[TMP0:%.*]] = load ptr, ptr [[X]], align 4, !nonnull [[META1:![0-9]+]], !align [[META2:![0-9]+]]
// CHECK-A64_32-NEXT: store i32 1, ptr [[TMP0]], align 4
// CHECK-A64_32-NEXT: ret void
//
@@ -654,9 +654,7 @@ void Tpaddrspace(Spaddrspace s) { *s.x = 1; }
// CHECK-A64-SAME: ) #[[ATTR0]] {
// CHECK-A64-NEXT: [[ENTRY:.*:]]
// CHECK-A64-NEXT: [[S:%.*]] = alloca [[STRUCT_SPADDRSPACE:%.*]], align 8
-// CHECK-A64-NEXT: [[AGG_TMP:%.*]] = alloca [[STRUCT_SPADDRSPACE]], align 8
-// CHECK-A64-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[AGG_TMP]], ptr align 8 [[S]], i64 8, i1 false)
-// CHECK-A64-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SPADDRSPACE]], ptr [[AGG_TMP]], i32 0, i32 0
+// CHECK-A64-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SPADDRSPACE]], ptr [[S]], i32 0, i32 0
// CHECK-A64-NEXT: [[TMP0:%.*]] = load ptr addrspace(100), ptr [[COERCE_DIVE]], align 8
// CHECK-A64-NEXT: [[COERCE_VAL_PI:%.*]] = ptrtoint ptr addrspace(100) [[TMP0]] to i64
// CHECK-A64-NEXT: call void @_Z11Tpaddrspace11Spaddrspace(i64 [[COERCE_VAL_PI]])
@@ -666,9 +664,7 @@ void Tpaddrspace(Spaddrspace s) { *s.x = 1; }
// CHECK-A64_32-SAME: ) #[[ATTR0]] {
// CHECK-A64_32-NEXT: [[ENTRY:.*:]]
// CHECK-A64_32-NEXT: [[S:%.*]] = alloca [[STRUCT_SPADDRSPACE:%.*]], align 4
-// CHECK-A64_32-NEXT: [[AGG_TMP:%.*]] = alloca [[STRUCT_SPADDRSPACE]], align 4
-// CHECK-A64_32-NEXT: call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[AGG_TMP]], ptr align 4 [[S]], i32 4, i1 false)
-// CHECK-A64_32-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SPADDRSPACE]], ptr [[AGG_TMP]], i32 0, i32 0
+// CHECK-A64_32-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SPADDRSPACE]], ptr [[S]], i32 0, i32 0
// CHECK-A64_32-NEXT: [[TMP0:%.*]] = load ptr addrspace(100), ptr [[COERCE_DIVE]], align 4
// CHECK-A64_32-NEXT: [[COERCE_VAL_PI:%.*]] = ptrtoint ptr addrspace(100) [[TMP0]] to i32
// CHECK-A64_32-NEXT: [[COERCE_VAL_II:%.*]] = zext i32 [[COERCE_VAL_PI]] to i64
@@ -709,9 +705,7 @@ void Tp2addrspace(Sp2addrspace s) { *s.x[0] = 1; }
// CHECK-A64-SAME: ) #[[ATTR0]] {
// CHECK-A64-NEXT: [[ENTRY:.*:]]
// CHECK-A64-NEXT: [[S:%.*]] = alloca [[STRUCT_SP2ADDRSPACE:%.*]], align 8
-// CHECK-A64-NEXT: [[AGG_TMP:%.*]] = alloca [[STRUCT_SP2ADDRSPACE]], align 8
-// CHECK-A64-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[AGG_TMP]], ptr align 8 [[S]], i64 16, i1 false)
-// CHECK-A64-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SP2ADDRSPACE]], ptr [[AGG_TMP]], i32 0, i32 0
+// CHECK-A64-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SP2ADDRSPACE]], ptr [[S]], i32 0, i32 0
// CHECK-A64-NEXT: [[TMP0:%.*]] = load [2 x i64], ptr [[COERCE_DIVE]], align 8
// CHECK-A64-NEXT: call void @_Z12Tp2addrspace12Sp2addrspace([2 x i64] [[TMP0]])
// CHECK-A64-NEXT: ret void
@@ -720,9 +714,7 @@ void Tp2addrspace(Sp2addrspace s) { *s.x[0] = 1; }
// CHECK-A64_32-SAME: ) #[[ATTR0]] {
// CHECK-A64_32-NEXT: [[ENTRY:.*:]]
// CHECK-A64_32-NEXT: [[S:%.*]] = alloca [[STRUCT_SP2ADDRSPACE:%.*]], align 4
-// CHECK-A64_32-NEXT: [[AGG_TMP:%.*]] = alloca [[STRUCT_SP2ADDRSPACE]], align 4
-// CHECK-A64_32-NEXT: call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[AGG_TMP]], ptr align 4 [[S]], i32 8, i1 false)
-// CHECK-A64_32-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SP2ADDRSPACE]], ptr [[AGG_TMP]], i32 0, i32 0
+// CHECK-A64_32-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SP2ADDRSPACE]], ptr [[S]], i32 0, i32 0
// CHECK-A64_32-NEXT: [[TMP0:%.*]] = load i64, ptr [[COERCE_DIVE]], align 4
// CHECK-A64_32-NEXT: call void @_Z12Tp2addrspace12Sp2addrspace(i64 [[TMP0]])
// CHECK-A64_32-NEXT: ret void
@@ -740,7 +732,7 @@ struct Sraddrspace {
// CHECK-A64-NEXT: [[COERCE_VAL_IP:%.*]] = inttoptr i64 [[S_COERCE]] to ptr addrspace(100)
// CHECK-A64-NEXT: store ptr addrspace(100) [[COERCE_VAL_IP]], ptr [[COERCE_DIVE]], align 8
// CHECK-A64-NEXT: [[X:%.*]] = getelementptr inbounds nuw [[STRUCT_SRADDRSPACE]], ptr [[S]], i32 0, i32 0
-// CHECK-A64-NEXT: [[TMP0:%.*]] = load ptr addrspace(100), ptr [[X]], align 8, !align [[META3]]
+// CHECK-A64-NEXT: [[TMP0:%.*]] = load ptr addrspace(100), ptr [[X]], align 8, !align [[META2]]
// CHECK-A64-NEXT: store i32 1, ptr addrspace(100) [[TMP0]], align 4
// CHECK-A64-NEXT: ret void
//
@@ -752,7 +744,7 @@ struct Sraddrspace {
// CHECK-A64_32-NEXT: [[COERCE_VAL_II:%.*]] = trunc i64 [[S_COERCE]] to i32
// CHECK-A64_32-NEXT: store i32 [[COERCE_VAL_II]], ptr [[COERCE_DIVE]], align 4
// CHECK-A64_32-NEXT: [[X:%.*]] = getelementptr inbounds nuw [[STRUCT_SRADDRSPACE]], ptr [[S]], i32 0, i32 0
-// CHECK-A64_32-NEXT: [[TMP0:%.*]] = load ptr addrspace(100), ptr [[X]], align 4, !align [[META3]]
+// CHECK-A64_32-NEXT: [[TMP0:%.*]] = load ptr addrspace(100), ptr [[X]], align 4, !align [[META2]]
// CHECK-A64_32-NEXT: store i32 1, ptr addrspace(100) [[TMP0]], align 4
// CHECK-A64_32-NEXT: ret void
//
@@ -761,12 +753,10 @@ void Traddrspace(Sraddrspace s) { s.x = 1; }
// CHECK-A64-SAME: i64 [[S_COERCE:%.*]]) #[[ATTR0]] {
// CHECK-A64-NEXT: [[ENTRY:.*:]]
// CHECK-A64-NEXT: [[S:%.*]] = alloca [[STRUCT_SRADDRSPACE:%.*]], align 8
-// CHECK-A64-NEXT: [[AGG_TMP:%.*]] = alloca [[STRUCT_SRADDRSPACE]], align 8
// CHECK-A64-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SRADDRSPACE]], ptr [[S]], i32 0, i32 0
// CHECK-A64-NEXT: [[COERCE_VAL_IP:%.*]] = inttoptr i64 [[S_COERCE]] to ptr addrspace(100)
// CHECK-A64-NEXT: store ptr addrspace(100) [[COERCE_VAL_IP]], ptr [[COERCE_DIVE]], align 8
-// CHECK-A64-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[AGG_TMP]], ptr align 8 [[S]], i64 8, i1 false)
-// CHECK-A64-NEXT: [[COERCE_DIVE1:%.*]] = getelementptr inbounds nuw [[STRUCT_SRADDRSPACE]], ptr [[AGG_TMP]], i32 0, i32 0
+// CHECK-A64-NEXT: [[COERCE_DIVE1:%.*]] = getelementptr inbounds nuw [[STRUCT_SRADDRSPACE]], ptr [[S]], i32 0, i32 0
// CHECK-A64-NEXT: [[TMP0:%.*]] = load ptr addrspace(100), ptr [[COERCE_DIVE1]], align 8
// CHECK-A64-NEXT: [[COERCE_VAL_PI:%.*]] = ptrtoint ptr addrspace(100) [[TMP0]] to i64
// CHECK-A64-NEXT: call void @_Z11Traddrspace11Sraddrspace(i64 [[COERCE_VAL_PI]])
@@ -776,12 +766,10 @@ void Traddrspace(Sraddrspace s) { s.x = 1; }
// CHECK-A64_32-SAME: i64 [[S_COERCE:%.*]]) #[[ATTR0]] {
// CHECK-A64_32-NEXT: [[ENTRY:.*:]]
// CHECK-A64_32-NEXT: [[S:%.*]] = alloca [[STRUCT_SRADDRSPACE:%.*]], align 4
-// CHECK-A64_32-NEXT: [[AGG_TMP:%.*]] = alloca [[STRUCT_SRADDRSPACE]], align 4
// CHECK-A64_32-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SRADDRSPACE]], ptr [[S]], i32 0, i32 0
// CHECK-A64_32-NEXT: [[COERCE_VAL_II:%.*]] = trunc i64 [[S_COERCE]] to i32
// CHECK-A64_32-NEXT: store i32 [[COERCE_VAL_II]], ptr [[COERCE_DIVE]], align 4
-// CHECK-A64_32-NEXT: call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[AGG_TMP]], ptr align 4 [[S]], i32 4, i1 false)
-// CHECK-A64_32-NEXT: [[COERCE_DIVE1:%.*]] = getelementptr inbounds nuw [[STRUCT_SRADDRSPACE]], ptr [[AGG_TMP]], i32 0, i32 0
+// CHECK-A64_32-NEXT: [[COERCE_DIVE1:%.*]] = getelementptr inbounds nuw [[STRUCT_SRADDRSPACE]], ptr [[S]], i32 0, i32 0
// CHECK-A64_32-NEXT: [[TMP0:%.*]] = load ptr addrspace(100), ptr [[COERCE_DIVE1]], align 4
// CHECK-A64_32-NEXT: [[COERCE_VAL_PI:%.*]] = ptrtoint ptr addrspace(100) [[TMP0]] to i32
// CHECK-A64_32-NEXT: [[COERCE_VAL_II2:%.*]] = zext i32 [[COERCE_VAL_PI]] to i64
@@ -791,9 +779,9 @@ void Traddrspace(Sraddrspace s) { s.x = 1; }
void Craddrspace(Sraddrspace s) { Traddrspace(s); }
//.
-// CHECK-A64: [[META2]] = !{}
-// CHECK-A64: [[META3]] = !{i64 4}
+// CHECK-A64: [[META1]] = !{}
+// CHECK-A64: [[META2]] = !{i64 4}
//.
-// CHECK-A64_32: [[META2]] = !{}
-// CHECK-A64_32: [[META3]] = !{i64 4}
+// CHECK-A64_32: [[META1]] = !{}
+// CHECK-A64_32: [[META2]] = !{i64 4}
//.
diff --git a/clang/test/CodeGen/aapcs64-align.cpp b/clang/test/CodeGen/aapcs64-align.cpp
index dd7e0cd0e1807..6192e09cbe869 100644
--- a/clang/test/CodeGen/aapcs64-align.cpp
+++ b/clang/test/CodeGen/aapcs64-align.cpp
@@ -122,7 +122,7 @@ unsigned sizeof_RidiculouslyOverSizedBitfield = sizeof(RidiculouslyOverSizedBitf
unsigned alignof_RidiculouslyOverSizedBitfield = alignof(RidiculouslyOverSizedBitfield);
// CHECK: define{{.*}} void @g9
-// CHECK: call void @f9(i32 noundef 1, ptr noundef nonnull dead_on_return %agg.tmp)
+// CHECK: call void @f9(i32 noundef 1, ptr noundef nonnull dead_on_return %byval-temp)
// CHECK: declare void @f9(i32 noundef, ptr noundef dead_on_return)
void f9(int a, RidiculouslyOverSizedBitfield b);
void g9() {
diff --git a/clang/test/CodeGenCXX/amdgcn-func-arg.cpp b/clang/test/CodeGenCXX/amdgcn-func-arg.cpp
index eec8bbc146cda..d93da70cdb164 100644
--- a/clang/test/CodeGenCXX/amdgcn-func-arg.cpp
+++ b/clang/test/CodeGenCXX/amdgcn-func-arg.cpp
@@ -111,12 +111,10 @@ void func_with_byval_arg(B b) {
// CHECK-SAME: ) #[[ATTR1]] {
// CHECK-NEXT: [[ENTRY:.*:]]
// CHECK-NEXT: [[B:%.*]] = alloca [[CLASS_B:%.*]], align 4, addrspace(5)
-// CHECK-NEXT: [[AGG_TMP:%.*]] = alloca [[CLASS_B]], align 4, addrspace(5)
+// CHECK-NEXT: [[BYVAL_TEMP:%.*]] = alloca [[CLASS_B]], align 4, addrspace(5)
// CHECK-NEXT: [[B_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[B]] to ptr
-// CHECK-NEXT: [[AGG_TMP_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[AGG_TMP]] to ptr
-// CHECK-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[AGG_TMP_ASCAST]], ptr align 4 [[B_ASCAST]], i64 400, i1 false)
-// CHECK-NEXT: [[AGG_TMP_ASCAST_ASCAST:%.*]] = addrspacecast ptr [[AGG_TMP_ASCAST]] to ptr addrspace(5)
-// CHECK-NEXT: call void @_Z19func_with_byval_arg1B(ptr addrspace(5) noundef byref([[CLASS_B]]) align 4 [[AGG_TMP_ASCAST_ASCAST]]) #[[ATTR5]]
+// CHECK-NEXT: call void @llvm.memcpy.p5.p0.i64(ptr addrspace(5) align 4 [[BYVAL_TEMP]], ptr align 4 [[B_ASCAST]], i64 400, i1 false)
+// CHECK-NEXT: call void @_Z19func_with_byval_arg1B(ptr addrspace(5) noundef byref([[CLASS_B]]) align 4 [[BYVAL_TEMP]]) #[[ATTR5]]
// CHECK-NEXT: call void @_Z17func_with_ref_argR1B(ptr noundef nonnull align 4 dereferenceable(400) [[B_ASCAST]]) #[[ATTR5]]
// CHECK-NEXT: ret void
//
@@ -129,11 +127,9 @@ void test_byval_arg_auto() {
// CHECK-LABEL: define dso_local void @_Z21test_byval_arg_globalv(
// CHECK-SAME: ) #[[ATTR1]] {
// CHECK-NEXT: [[ENTRY:.*:]]
-// CHECK-NEXT: [[AGG_TMP:%.*]] = alloca [[CLASS_B:%.*]], align 4, addrspace(5)
-// CHECK-NEXT: [[AGG_TMP_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[AGG_TMP]] to ptr
-// CHECK-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[AGG_TMP_ASCAST]], ptr align 4 addrspacecast (ptr addrspace(1) @g_b to ptr), i64 400, i1 false)
-// CHECK-NEXT: [[AGG_TMP_ASCAST_ASCAST:%.*]] = addrspacecast ptr [[AGG_TMP_ASCAST]] to ptr addrspace(5)
-// CHECK-NEXT: call void @_Z19func_with_byval_arg1B(ptr addrspace(5) noundef byref([[CLASS_B]]) align 4 [[AGG_TMP_ASCAST_ASCAST]]) #[[ATTR5]]
+// CHECK-NEXT: [[BYVAL_TEMP:%.*]] = alloca [[CLASS_B:%.*]], align 4, addrspace(5)
+// CHECK-NEXT: call void @llvm.memcpy.p5.p0.i64(ptr addrspace(5) align 4 [[BYVAL_TEMP]], ptr align 4 addrspacecast (ptr addrspace(1) @g_b to ptr), i64 400, i1 false)
+// CHECK-NEXT: call void @_Z19func_with_byval_arg1B(ptr addrspace(5) noundef byref([[CLASS_B]]) align 4 [[BYVAL_TEMP]]) #[[ATTR5]]
// CHECK-NEXT: call void @_Z17func_with_ref_argR1B(ptr noundef nonnull align 4 dereferenceable(400) addrspacecast (ptr addrspace(1) @g_b to ptr)) #[[ATTR5]]
// CHECK-NEXT: ret void
//
diff --git a/clang/test/CodeGenCXX/assign-construct-memcpy.cpp b/clang/test/CodeGenCXX/assign-construct-memcpy.cpp
index 7d6c5d76107da..609dc31ce5829 100644
--- a/clang/test/CodeGenCXX/assign-construct-memcpy.cpp
+++ b/clang/test/CodeGenCXX/assign-construct-memcpy.cpp
@@ -81,9 +81,13 @@ void byval(foo f);
void test7(const foo &x) {
byval(x);
+// A trivially-copyable byval arg forwards the source; the byval boundary
+// makes the copy, so no separate memcpy is emitted.
// CHECK-POD: test7
-// CHECK-POD: call void @llvm.memcpy.p0.p0.i64({{.*}} align 8 {{.*}} align 8 {{.*}}i64 24
+// CHECK-POD-NOT: call void @llvm.memcpy
+// CHECK-POD: call void @_Z5byval3foo(ptr noundef byval(%struct.foo) align 8
// CHECK-NONPOD: test7
-// CHECK-NONPOD: call void @llvm.memcpy.p0.p0.i64({{.*}} align 8 {{.*}} align 8 {{.*}}i64 24
+// CHECK-NONPOD-NOT: call void @llvm.memcpy
+// CHECK-NONPOD: call void @_Z5byval3foo(ptr noundef byval(%struct.foo) align 8
}
diff --git a/clang/test/CodeGenCXX/cxx2b-deducing-this.cpp b/clang/test/CodeGenCXX/cxx2b-deducing-this.cpp
index 9664a866376ae..d5dddf10af7a9 100644
--- a/clang/test/CodeGenCXX/cxx2b-deducing-this.cpp
+++ b/clang/test/CodeGenCXX/cxx2b-deducing-this.cpp
@@ -10,7 +10,6 @@ void test() {
// CHECK: define {{.*}}test{{.*}}
// CHECK-NEXT: entry:
// CHECK: {{.*}} = alloca %struct.TrivialStruct, align 1
-// CHECK: {{.*}} = alloca %struct.TrivialStruct, align 1
// CHECK: call void {{.*}}explicit_object_function{{.*}}
// CHECK-NEXT: ret void
// CHECK-NEXT: }
@@ -38,7 +37,6 @@ void test_lambda() {
//CHECK: define internal noundef i32 @"_ZZ11test_lambdavENH3$_0clIS_EEiT_"() #0 align 2 {
//CHECK: entry:
//CHECK: %This = alloca %class.anon, align 1
-//CHECK: %agg.tmp = alloca %class.anon, align 1
//CHECK: %call = call noundef i32 @"_ZZ11test_lambdavENH3$_0clIS_EEiT_"()
//CHECK: ret i32 %call
//CHECK: }
diff --git a/clang/test/CodeGenCXX/pr40771-ctad-with-lambda-copy-capture.cpp b/clang/test/CodeGenCXX/pr40771-ctad-with-lambda-copy-capture.cpp
index 74e1f17652e17..10fce3d05870c 100644
--- a/clang/test/CodeGenCXX/pr40771-ctad-with-lambda-copy-capture.cpp
+++ b/clang/test/CodeGenCXX/pr40771-ctad-with-lambda-copy-capture.cpp
@@ -10,9 +10,7 @@ T t { R{q}, S{q} };
// CHECK-LABEL: define internal void @__cxx_global_var_init.1() {{.*}} {
// CHECK-NEXT: [[TMP_R:%[a-z0-9.]+]] = alloca %struct.R, align 1
-// CHECK-NEXT: [[TMP_Q1:%[a-z0-9.]+]] = alloca %struct.Q, align 1
// CHECK-NEXT: [[TMP_S:%[a-z0-9.]+]] = alloca %struct.S, align 1
-// CHECK-NEXT: [[TMP_Q2:%[a-z0-9.]+]] = alloca %struct.Q, align 1
// CHECK-NEXT: [[XPT:%[a-z0-9.]+]] = alloca ptr
// CHECK-NEXT: [[SLOT:%[a-z0-9.]+]] = alloca i32
// CHECK-NEXT: [[ACTIVE:%[a-z0-9.]+]] = alloca i1, align 1
diff --git a/clang/test/CodeGenCXX/trivial-copy-arg-forward.cpp b/clang/test/CodeGenCXX/trivial-copy-arg-forward.cpp
new file mode 100644
index 0000000000000..1dabf5a7c1f3e
--- /dev/null
+++ b/clang/test/CodeGenCXX/trivial-copy-arg-forward.cpp
@@ -0,0 +1,50 @@
+// RUN: %clang_cc1 -triple x86_64-linux-gnu %s -emit-llvm -o - | FileCheck %s
+
+// A by-value argument built by a trivial copy/move constructor from a
+// variable forwards the variable's storage to the call; the byval boundary
+// makes the copy, so no agg.tmp is materialized. This is the C++ analog of
+// the LValueToRValue forwarding the C path already does.
+
+struct Triv {
+ unsigned long a, b, c, d;
+};
+void sink(Triv);
+
+// Same-type source: forward, no temporary.
+void from_local() {
+ Triv t;
+ sink(t);
+}
+// CHECK-LABEL: define {{.*}}@_Z10from_localv(
+// CHECK: [[T:%.*]] = alloca %struct.Triv
+// CHECK-NOT: = alloca %struct.Triv
+// CHECK: call void @_Z4sink4Triv(ptr noundef byval(%struct.Triv) align 8 [[T]])
+
+void from_param(Triv t) {
+ sink(t);
+}
+// CHECK-LABEL: define {{.*}}@_Z10from_param4Triv(
+// CHECK-NOT: = alloca %struct.Triv
+// CHECK: call void @_Z4sink4Triv(ptr noundef byval(%struct.Triv) align 8 %t)
+
+// Derived-to-base: forwarding the derived lvalue into a base-typed slot would
+// slice at the wrong offset, so the same-type guard keeps the temp and copies
+// from the adjusted base offset.
+struct Base1 {
+ long a, b;
+};
+struct Base2 {
+ long c, d;
+};
+struct Derived : Base1, Base2 {
+ long e;
+};
+void sink_base(Base2);
+void slice_to_base(Derived d) {
+ sink_base(d);
+}
+// CHECK-LABEL: define {{.*}}@_Z13slice_to_base7Derived(
+// CHECK: [[TMP:%.*]] = alloca %struct.Base2
+// CHECK: [[ADJ:%.*]] = getelementptr inbounds i8, ptr %d, i64 16
+// CHECK: call void @llvm.memcpy{{.*}}(ptr {{.*}} [[TMP]], ptr {{.*}} [[ADJ]],
+// CHECK: call void @_Z9sink_base5Base2(
diff --git a/clang/test/CodeGenCXX/varargs.cpp b/clang/test/CodeGenCXX/varargs.cpp
index afffaf5554deb..830faaaf4bda0 100644
--- a/clang/test/CodeGenCXX/varargs.cpp
+++ b/clang/test/CodeGenCXX/varargs.cpp
@@ -31,9 +31,7 @@ namespace test1 {
}
// CHECK-LABEL: define{{.*}} void @_ZN5test14testEv()
// CHECK: [[X:%.*]] = alloca [[A:%.*]], align 4
- // CHECK-NEXT: [[TMP:%.*]] = alloca [[A]], align 4
- // CHECK-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[TMP]], ptr align 4 [[X]], i64 8, i1 false)
- // CHECK-NEXT: [[T1:%.*]] = load i64, ptr [[TMP]], align 4
+ // CHECK-NEXT: [[T1:%.*]] = load i64, ptr [[X]], align 4
// CHECK-NEXT: call void (...) @_ZN5test13fooEz(i64 [[T1]])
// CHECK-NEXT: ret void
}
>From 3a8db70f065f4688d5986de0db7517e43c4a1fdd Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Sat, 27 Jun 2026 11:46:30 +0200
Subject: [PATCH 13/21] [Clang][test] Cover musttail Indirect arg with no
in-memory source
A musttail Indirect argument is forwarded through the matching incoming
parameter, which needs an in-memory source. A wide _BitInt has scalar
evaluation kind, so the argument is a scalar value with no source
storage to forward, and the call is rejected. Document the diagnostic.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.6 <noreply at anthropic.com>
---
.../musttail-indirect-arg-unsupported.c | 25 +++++++++++++++++++
1 file changed, 25 insertions(+)
create mode 100644 clang/test/CodeGen/musttail-indirect-arg-unsupported.c
diff --git a/clang/test/CodeGen/musttail-indirect-arg-unsupported.c b/clang/test/CodeGen/musttail-indirect-arg-unsupported.c
new file mode 100644
index 0000000000000..bf938406aa7ba
--- /dev/null
+++ b/clang/test/CodeGen/musttail-indirect-arg-unsupported.c
@@ -0,0 +1,25 @@
+// RUN: %clang_cc1 -triple=x86_64-linux-gnu -verify -emit-llvm-only %s
+// RUN: %clang_cc1 -triple=riscv64-linux-gnu -verify -emit-llvm-only %s
+
+// A musttail Indirect argument is forwarded through the matching incoming
+// parameter, which requires an in-memory source. A wide _BitInt has scalar
+// evaluation kind, so the argument is a scalar value with no source storage
+// to forward, regardless of value category. Such a call is rejected rather
+// than routed through a caller-frame temp that dangles past the tail call.
+
+typedef _BitInt(256) BI;
+BI cee(BI x);
+BI pee(BI a) {
+ // expected-error at +1 {{'musttail' call requires passing an argument by reference, but the source does not have an addressable storage and would alias the caller's frame}}
+ __attribute__((musttail)) return cee(a);
+}
+
+// An aggregate lvalue has addressable storage to forward, so it is accepted.
+// Confirms the diagnostic is specific to the no-source case.
+struct Big {
+ unsigned long long a, b, c, d;
+};
+struct Big cee_ok(struct Big x);
+struct Big pee_ok(struct Big a) {
+ __attribute__((musttail)) return cee_ok(a);
+}
>From 2a770e50b6449b95dc35f09e42869a486e7cc9a0 Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Sat, 27 Jun 2026 19:42:30 +0200
Subject: [PATCH 14/21] [Clang] Exclude CUDA surface/texture types from
by-value arg forwarding
A CUDA device_builtin_surface_type/texture_type is trivially copyable,
so the generalized forwarding would hand its global LValue to the call.
On device these types are coerced to an i64 handle, and the global is
lowered to that handle in EmitAggregateCopy via the NVPTX target hook.
Forwarding bypasses that copy, so the direct-coerce read loads the raw
global bytes instead of the handle, dropping the texsurf.handle.internal
intrinsic. Exclude both types, mirroring the EmitAggregateCopy special
case, so they keep the temp-materialization path.
Also update CodeGenSYCL/kernel-caller-entry-point.cpp: the kernel-launch
wrappers forward their trivially-copyable functor argument on the Linux
host ABI, dropping the agg.tmp copy. The lambdas with a non-trivial
destructor are not trivially copyable and keep the copy.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.6 <noreply at anthropic.com>
---
clang/lib/CodeGen/CGCall.cpp | 8 +++++--
.../CodeGenSYCL/kernel-caller-entry-point.cpp | 22 +++++--------------
2 files changed, 11 insertions(+), 19 deletions(-)
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index f699c3498e14e..e165484f5ed7c 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5259,9 +5259,13 @@ void CodeGenFunction::EmitCallArg(CallArgList &args, const Expr *E,
// constructor from a variable forwards the source LValue instead of
// materializing an agg.tmp. EmitCall makes the real copy at the
// Indirect/byval boundary. Under musttail this also keeps the value in
- // storage that survives the tail call.
+ // storage that survives the tail call. CUDA surface/texture types are
+ // excluded: their copy lowers a global to a handle in EmitAggregateCopy,
+ // which forwarding would bypass.
if (HasAggregateEvalKind && type->isRecordType() &&
- type.isTriviallyCopyableType(getContext())) {
+ type.isTriviallyCopyableType(getContext()) &&
+ !type->isCUDADeviceBuiltinSurfaceType() &&
+ !type->isCUDADeviceBuiltinTextureType()) {
if (const auto *CCE = dyn_cast<CXXConstructExpr>(E)) {
const CXXConstructorDecl *Ctor = CCE->getConstructor();
if (Ctor->isCopyOrMoveConstructor() && Ctor->isTrivial() &&
diff --git a/clang/test/CodeGenSYCL/kernel-caller-entry-point.cpp b/clang/test/CodeGenSYCL/kernel-caller-entry-point.cpp
index 270671284c3d1..c18b0f677548d 100644
--- a/clang/test/CodeGenSYCL/kernel-caller-entry-point.cpp
+++ b/clang/test/CodeGenSYCL/kernel-caller-entry-point.cpp
@@ -160,7 +160,6 @@ int main() {
// CHECK-HOST-LINUX: define dso_local void @_Z26single_purpose_kernel_task21single_purpose_kernel() #{{[0-9]+}} {
// CHECK-HOST-LINUX-NEXT: entry:
// CHECK-HOST-LINUX-NEXT: %kernelFunc = alloca %struct.single_purpose_kernel, align 1
-// CHECK-HOST-LINUX-NEXT: %agg.tmp = alloca %struct.single_purpose_kernel, align 1
// CHECK-HOST-LINUX-NEXT: call void @_Z18sycl_kernel_launchI26single_purpose_kernel_nameJ21single_purpose_kernelEEvPKcDpT0_(ptr noundef @.str)
// CHECK-HOST-LINUX-NEXT: ret void
// CHECK-HOST-LINUX-NEXT: }
@@ -168,11 +167,9 @@ int main() {
// CHECK-HOST-LINUX: define internal void @_Z18kernel_single_taskIZ4mainEUlT_E_S1_EvT0_(i32 %kernelFunc.coerce) #{{[0-9]+}} {
// CHECK-HOST-LINUX-NEXT: entry:
// CHECK-HOST-LINUX-NEXT: %kernelFunc = alloca %class.anon, align 4
-// CHECK-HOST-LINUX-NEXT: %agg.tmp = alloca %class.anon, align 4
// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon, ptr %kernelFunc, i32 0, i32 0
// CHECK-HOST-LINUX-NEXT: store i32 %kernelFunc.coerce, ptr %coerce.dive, align 4
-// CHECK-HOST-LINUX-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 4 %agg.tmp, ptr align 4 %kernelFunc, i64 4, i1 false)
-// CHECK-HOST-LINUX-NEXT: %coerce.dive1 = getelementptr inbounds nuw %class.anon, ptr %agg.tmp, i32 0, i32 0
+// CHECK-HOST-LINUX-NEXT: %coerce.dive1 = getelementptr inbounds nuw %class.anon, ptr %kernelFunc, i32 0, i32 0
// CHECK-HOST-LINUX-NEXT: %0 = load i32, ptr %coerce.dive1, align 4
// CHECK-HOST-LINUX-NEXT: call void @_Z18sycl_kernel_launchIZ4mainEUlT_E_JS1_EEvPKcDpT0_(ptr noundef @.str.1, i32 %0)
// CHECK-HOST-LINUX-NEXT: ret void
@@ -181,7 +178,6 @@ int main() {
// CHECK-HOST-LINUX: define internal void @"_Z18kernel_single_taskI6\CE\B4\CF\84\CF\87Z4mainEUliE_EvT0_"() #{{[0-9]+}} {
// CHECK-HOST-LINUX-NEXT: entry:
// CHECK-HOST-LINUX-NEXT: %kernelFunc = alloca %class.anon.0, align 1
-// CHECK-HOST-LINUX-NEXT: %agg.tmp = alloca %class.anon.0, align 1
// CHECK-HOST-LINUX-NEXT: call void @"_Z18sycl_kernel_launchI6\CE\B4\CF\84\CF\87JZ4mainEUliE_EEvPKcDpT0_"(ptr noundef @.str.2)
// CHECK-HOST-LINUX-NEXT: ret void
// CHECK-HOST-LINUX-NEXT: }
@@ -210,11 +206,9 @@ int main() {
// CHECK-HOST-LINUX: define internal void @_Z14ref_arg_kernelI19ref_arg_kernel_nameZ4mainEUlT_E_EvRKT0_(ptr noundef nonnull align 4 dereferenceable(4) %ref) #{{[0-9]+}} {
// CHECK-HOST-LINUX-NEXT: entry:
// CHECK-HOST-LINUX-NEXT: %ref.addr = alloca ptr, align 8
-// CHECK-HOST-LINUX-NEXT: %agg.tmp = alloca %class.anon, align 4
// CHECK-HOST-LINUX-NEXT: store ptr %ref, ptr %ref.addr, align 8
// CHECK-HOST-LINUX-NEXT: %0 = load ptr, ptr %ref.addr, align 8
-// CHECK-HOST-LINUX-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 4 %agg.tmp, ptr align 4 %0, i64 4, i1 false)
-// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon, ptr %agg.tmp, i32 0, i32 0
+// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon, ptr %0, i32 0, i32 0
// CHECK-HOST-LINUX-NEXT: %1 = load i32, ptr %coerce.dive, align 4
// CHECK-HOST-LINUX-NEXT: call void @_Z18sycl_kernel_launchI19ref_arg_kernel_nameJZ4mainEUlT_E_EEvPKcDpT0_(ptr noundef @.str.4, i32 %1)
// CHECK-HOST-LINUX-NEXT: ret void
@@ -223,11 +217,9 @@ int main() {
// CHECK-HOST-LINUX: define internal void @_Z18fwd_ref_arg_kernelI23fwd_ref_arg_kernel_nameRZ4mainEUlT_E_EvOT0_(ptr noundef nonnull align 4 dereferenceable(4) %ref) #{{[0-9]+}} {
// CHECK-HOST-LINUX-NEXT: entry:
// CHECK-HOST-LINUX-NEXT: %ref.addr = alloca ptr, align 8
-// CHECK-HOST-LINUX-NEXT: %agg.tmp = alloca %class.anon, align 4
// CHECK-HOST-LINUX-NEXT: store ptr %ref, ptr %ref.addr, align 8
// CHECK-HOST-LINUX-NEXT: %0 = load ptr, ptr %ref.addr, align 8
-// CHECK-HOST-LINUX-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 4 %agg.tmp, ptr align 4 %0, i64 4, i1 false)
-// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon, ptr %agg.tmp, i32 0, i32 0
+// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon, ptr %0, i32 0, i32 0
// CHECK-HOST-LINUX-NEXT: %1 = load i32, ptr %coerce.dive, align 4
// CHECK-HOST-LINUX-NEXT: call void @_Z18sycl_kernel_launchI23fwd_ref_arg_kernel_nameJZ4mainEUlT_E_EEvPKcDpT0_(ptr noundef @.str.5, i32 %1)
// CHECK-HOST-LINUX-NEXT: ret void
@@ -236,11 +228,9 @@ int main() {
// CHECK-HOST-LINUX: define internal void @_Z18fwd_ref_arg_kernelI28fwd_ref_arg_kernel_name_moveZ4mainEUlT_E_EvOT0_(ptr noundef nonnull align 4 dereferenceable(4) %ref) #{{[0-9]+}} {
// CHECK-HOST-LINUX-NEXT: entry:
// CHECK-HOST-LINUX-NEXT: %ref.addr = alloca ptr, align 8
-// CHECK-HOST-LINUX-NEXT: %agg.tmp = alloca %class.anon, align 4
// CHECK-HOST-LINUX-NEXT: store ptr %ref, ptr %ref.addr, align 8
// CHECK-HOST-LINUX-NEXT: %0 = load ptr, ptr %ref.addr, align 8
-// CHECK-HOST-LINUX-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 4 %agg.tmp, ptr align 4 %0, i64 4, i1 false)
-// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon, ptr %agg.tmp, i32 0, i32 0
+// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon, ptr %0, i32 0, i32 0
// CHECK-HOST-LINUX-NEXT: %1 = load i32, ptr %coerce.dive, align 4
// CHECK-HOST-LINUX-NEXT: call void @_Z18sycl_kernel_launchI28fwd_ref_arg_kernel_name_moveJZ4mainEUlT_E_EEvPKcDpT0_(ptr noundef @.str.6, i32 %1)
// CHECK-HOST-LINUX-NEXT: ret void
@@ -249,11 +239,9 @@ int main() {
// CHECK-HOST-LINUX: define internal void @_Z21rvalue_ref_arg_kernelI26rvalue_ref_arg_kernel_nameZ4mainEUlT_E0_EvONSt13type_identityIT0_E4typeE(ptr noundef nonnull align 4 dereferenceable(4) %ref) #{{[0-9]+}} {
// CHECK-HOST-LINUX-NEXT: entry:
// CHECK-HOST-LINUX-NEXT: %ref.addr = alloca ptr, align 8
-// CHECK-HOST-LINUX-NEXT: %agg.tmp = alloca %class.anon.2, align 4
// CHECK-HOST-LINUX-NEXT: store ptr %ref, ptr %ref.addr, align 8
// CHECK-HOST-LINUX-NEXT: %0 = load ptr, ptr %ref.addr, align 8
-// CHECK-HOST-LINUX-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 4 %agg.tmp, ptr align 4 %0, i64 4, i1 false)
-// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon.2, ptr %agg.tmp, i32 0, i32 0
+// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon.2, ptr %0, i32 0, i32 0
// CHECK-HOST-LINUX-NEXT: %1 = load i32, ptr %coerce.dive, align 4
// CHECK-HOST-LINUX-NEXT: call void @_Z18sycl_kernel_launchI26rvalue_ref_arg_kernel_nameJZ4mainEUlT_E0_EEvPKcDpT0_(ptr noundef @.str.7, i32 %1)
// CHECK-HOST-LINUX-NEXT: ret void
>From 5be3d32db09f2a27c6e5feda9efa57c062de3a8e Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Sat, 27 Jun 2026 20:40:04 +0200
Subject: [PATCH 15/21] [Clang] Exclude ObjC GC object-member records from
by-value arg forwarding
Under -fobjc-gc, a trivially-copyable struct with an object member needs
the objc_memmove_collectable write barrier when copied by value, emitted
in EmitAggregateCopy. Such a record is still isTriviallyCopyableType, so
the generalized forwarding handed its source LValue to the call and the
byval boundary passed it without a copy, dropping the barrier. Exclude
records with an object member under GC, mirroring the EmitAggregateCopy
special case alongside CUDA surface/texture.
Regression test in CodeGenObjCXX/gc.mm: removing the guard drops the
barrier on the forwarded path.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.6 <noreply at anthropic.com>
---
clang/lib/CodeGen/CGCall.cpp | 11 +++++++----
clang/test/CodeGenObjCXX/gc.mm | 12 ++++++++++++
2 files changed, 19 insertions(+), 4 deletions(-)
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index e165484f5ed7c..e32668702ac1b 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5259,13 +5259,16 @@ void CodeGenFunction::EmitCallArg(CallArgList &args, const Expr *E,
// constructor from a variable forwards the source LValue instead of
// materializing an agg.tmp. EmitCall makes the real copy at the
// Indirect/byval boundary. Under musttail this also keeps the value in
- // storage that survives the tail call. CUDA surface/texture types are
- // excluded: their copy lowers a global to a handle in EmitAggregateCopy,
- // which forwarding would bypass.
+ // storage that survives the tail call. Types whose copy is not a plain
+ // memcpy in EmitAggregateCopy are excluded, since forwarding would bypass
+ // that special copy: CUDA surface/texture (lowered to a handle) and, under
+ // ObjC GC, records with object members (need a write barrier).
if (HasAggregateEvalKind && type->isRecordType() &&
type.isTriviallyCopyableType(getContext()) &&
!type->isCUDADeviceBuiltinSurfaceType() &&
- !type->isCUDADeviceBuiltinTextureType()) {
+ !type->isCUDADeviceBuiltinTextureType() &&
+ !(getLangOpts().getGC() != LangOptions::NonGC &&
+ type->getAsRecordDecl()->hasObjectMember())) {
if (const auto *CCE = dyn_cast<CXXConstructExpr>(E)) {
const CXXConstructorDecl *Ctor = CCE->getConstructor();
if (Ctor->isCopyOrMoveConstructor() && Ctor->isTrivial() &&
diff --git a/clang/test/CodeGenObjCXX/gc.mm b/clang/test/CodeGenObjCXX/gc.mm
index 2a34f1f288e89..e67470120362a 100644
--- a/clang/test/CodeGenObjCXX/gc.mm
+++ b/clang/test/CodeGenObjCXX/gc.mm
@@ -18,3 +18,15 @@
// CHECK-NEXT: call ptr @objc_assign_strongCast(ptr [[T2]], ptr [[T1]])
// CHECK-NEXT: ret void
}
+
+namespace test1 {
+ // A trivially-copyable struct with an object member, passed by value, needs
+ // the GC write barrier. Forwarding the source variable must not bypass it.
+ struct S { id a; };
+ void sink(S);
+ void pass(S s) { sink(s); }
+
+// CHECK-LABEL: define{{.*}} void @_ZN5test14passENS_1SE(
+// CHECK: call ptr @objc_memmove_collectable(
+// CHECK: call void @_ZN5test14sinkENS_1SE(
+}
>From 8c5c3be74dbd2e0fff676ff380d4da550eeed133 Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Fri, 3 Jul 2026 08:57:45 +0200
Subject: [PATCH 16/21] [Clang] Re-gate trivial-copy arg forwarding to musttail
calls
Reverts the generalization of the EmitCallArg trivial-copy LValue
forwarding to all calls (cb7efcbc3799) and the ObjC GC object-member
exclusion it required (5be3d32db09f), restoring the MustTailCall gate.
Requested in review: the all-calls behavior change is easier to bisect
as its own patch, and the exclusion-list question belongs with it.
Under the musttail gate the GC exclusion is not load-bearing: the
Indirect-path copies go through EmitAggregateCopy, which emits the
write barrier, and a Direct-classified load never writes a collectable
slot.
The CUDA surface/texture exclusion (2a770e50b644) is kept, narrowed to
device compilation: on NVPTX those types classify as Direct, so a
forwarded lvalue would load raw record bytes instead of the handle
that EmitAggregateCopy materializes.
The generalization moves to a follow-up PR together with its test
updates and trivial-copy-arg-forward.cpp.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.6 <noreply at anthropic.com>
---
clang/lib/CodeGen/CGCall.cpp | 51 +++++++++----------
.../AArch64/struct-coerce-using-ptr.cpp | 40 ++++++++++-----
clang/test/CodeGen/aapcs64-align.cpp | 2 +-
clang/test/CodeGenCXX/amdgcn-func-arg.cpp | 16 +++---
.../CodeGenCXX/assign-construct-memcpy.cpp | 8 +--
clang/test/CodeGenCXX/cxx2b-deducing-this.cpp | 2 +
.../pr40771-ctad-with-lambda-copy-capture.cpp | 2 +
.../CodeGenCXX/trivial-copy-arg-forward.cpp | 50 ------------------
clang/test/CodeGenCXX/varargs.cpp | 4 +-
clang/test/CodeGenObjCXX/gc.mm | 12 -----
.../CodeGenSYCL/kernel-caller-entry-point.cpp | 22 ++++++--
11 files changed, 87 insertions(+), 122 deletions(-)
delete mode 100644 clang/test/CodeGenCXX/trivial-copy-arg-forward.cpp
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index 70b809234279b..af5477c69a8d8 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5253,36 +5253,33 @@ void CodeGenFunction::EmitCallArg(CallArgList &args, const Expr *E,
}
}
- // C++ analog of the CK_LValueToRValue case above: a trivial copy/move
- // constructor from a variable forwards the source LValue instead of
- // materializing an agg.tmp. EmitCall makes the real copy at the
- // Indirect/byval boundary. Under musttail this also keeps the value in
- // storage that survives the tail call. Types whose copy is not a plain
- // memcpy in EmitAggregateCopy are excluded, since forwarding would bypass
- // that special copy: CUDA surface/texture (lowered to a handle) and, under
- // ObjC GC, records with object members (need a write barrier).
- if (HasAggregateEvalKind && type->isRecordType() &&
+ // Under musttail, hand a trivially-copyable record source's LValue to
+ // EmitCall rather than materializing an agg.tmp. EmitCall's Indirect path
+ // routes it via the matching incoming parameter, which survives the tail
+ // call. Limited to params and locals: globals and captures don't have the
+ // dangle issue and the existing path may be more efficient for them.
+ // On the device side CUDA surface/texture types are excluded: they
+ // classify as Direct and forwarding would load raw record bytes instead
+ // of the handle that EmitAggregateCopy materializes.
+ if (HasAggregateEvalKind && MustTailCall && type->isRecordType() &&
type.isTriviallyCopyableType(getContext()) &&
- !type->isCUDADeviceBuiltinSurfaceType() &&
- !type->isCUDADeviceBuiltinTextureType() &&
- !(getLangOpts().getGC() != LangOptions::NonGC &&
- type->getAsRecordDecl()->hasObjectMember())) {
+ !(getLangOpts().CUDAIsDevice &&
+ (type->isCUDADeviceBuiltinSurfaceType() ||
+ type->isCUDADeviceBuiltinTextureType()))) {
if (const auto *CCE = dyn_cast<CXXConstructExpr>(E)) {
- const CXXConstructorDecl *Ctor = CCE->getConstructor();
- if (Ctor->isCopyOrMoveConstructor() && Ctor->isTrivial() &&
- CCE->getNumArgs() == 1) {
+ if (CCE->getConstructor()->isCopyOrMoveConstructor() &&
+ CCE->getConstructor()->isTrivial() && CCE->getNumArgs() == 1) {
const Expr *Source = CCE->getArg(0)->IgnoreParenImpCasts();
- // Same-type guard: a stripped derived-to-base cast would forward a
- // derived lvalue into a base-typed slot and slice at a wrong offset.
- // Exclude hlsl_constant sources, as the CK_LValueToRValue path does.
- if (const auto *DRE = dyn_cast<DeclRefExpr>(Source);
- DRE && isa<VarDecl>(DRE->getDecl()) &&
- Source->getType().getAddressSpace() != LangAS::hlsl_constant &&
- getContext().hasSameUnqualifiedType(Source->getType(), type)) {
- LValue L = EmitLValue(DRE);
- if (L.isSimple()) {
- args.addUncopiedAggregate(L, type);
- return;
+ if (const auto *DRE = dyn_cast<DeclRefExpr>(Source)) {
+ if (const auto *VD = dyn_cast<VarDecl>(DRE->getDecl())) {
+ if (VD->hasLocalStorage() ||
+ (isa<ParmVarDecl>(VD) &&
+ VD->getDeclContext() == dyn_cast<DeclContext>(CurCodeDecl))) {
+ LValue L = EmitLValue(DRE);
+ assert(L.isSimple());
+ args.addUncopiedAggregate(L, type);
+ return;
+ }
}
}
}
diff --git a/clang/test/CodeGen/AArch64/struct-coerce-using-ptr.cpp b/clang/test/CodeGen/AArch64/struct-coerce-using-ptr.cpp
index b6392e5c0b1f2..a7c4f8e401529 100644
--- a/clang/test/CodeGen/AArch64/struct-coerce-using-ptr.cpp
+++ b/clang/test/CodeGen/AArch64/struct-coerce-using-ptr.cpp
@@ -139,7 +139,7 @@ struct Srp {
// CHECK-A64-NEXT: [[S:%.*]] = alloca [[STRUCT_SRP:%.*]], align 8
// CHECK-A64-NEXT: store [2 x ptr] [[S_COERCE]], ptr [[S]], align 8
// CHECK-A64-NEXT: [[X:%.*]] = getelementptr inbounds nuw [[STRUCT_SRP]], ptr [[S]], i32 0, i32 0
-// CHECK-A64-NEXT: [[TMP0:%.*]] = load ptr, ptr [[X]], align 8, !nonnull [[META1:![0-9]+]], !align [[META2:![0-9]+]]
+// CHECK-A64-NEXT: [[TMP0:%.*]] = load ptr, ptr [[X]], align 8, !nonnull [[META2:![0-9]+]], !align [[META3:![0-9]+]]
// CHECK-A64-NEXT: store i32 1, ptr [[TMP0]], align 4
// CHECK-A64-NEXT: ret void
//
@@ -149,7 +149,7 @@ struct Srp {
// CHECK-A64_32-NEXT: [[S:%.*]] = alloca [[STRUCT_SRP:%.*]], align 4
// CHECK-A64_32-NEXT: store i64 [[S_COERCE]], ptr [[S]], align 4
// CHECK-A64_32-NEXT: [[X:%.*]] = getelementptr inbounds nuw [[STRUCT_SRP]], ptr [[S]], i32 0, i32 0
-// CHECK-A64_32-NEXT: [[TMP0:%.*]] = load ptr, ptr [[X]], align 4, !nonnull [[META1:![0-9]+]], !align [[META2:![0-9]+]]
+// CHECK-A64_32-NEXT: [[TMP0:%.*]] = load ptr, ptr [[X]], align 4, !nonnull [[META2:![0-9]+]], !align [[META3:![0-9]+]]
// CHECK-A64_32-NEXT: store i32 1, ptr [[TMP0]], align 4
// CHECK-A64_32-NEXT: ret void
//
@@ -654,7 +654,9 @@ void Tpaddrspace(Spaddrspace s) { *s.x = 1; }
// CHECK-A64-SAME: ) #[[ATTR0]] {
// CHECK-A64-NEXT: [[ENTRY:.*:]]
// CHECK-A64-NEXT: [[S:%.*]] = alloca [[STRUCT_SPADDRSPACE:%.*]], align 8
-// CHECK-A64-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SPADDRSPACE]], ptr [[S]], i32 0, i32 0
+// CHECK-A64-NEXT: [[AGG_TMP:%.*]] = alloca [[STRUCT_SPADDRSPACE]], align 8
+// CHECK-A64-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[AGG_TMP]], ptr align 8 [[S]], i64 8, i1 false)
+// CHECK-A64-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SPADDRSPACE]], ptr [[AGG_TMP]], i32 0, i32 0
// CHECK-A64-NEXT: [[TMP0:%.*]] = load ptr addrspace(100), ptr [[COERCE_DIVE]], align 8
// CHECK-A64-NEXT: [[COERCE_VAL_PI:%.*]] = ptrtoint ptr addrspace(100) [[TMP0]] to i64
// CHECK-A64-NEXT: call void @_Z11Tpaddrspace11Spaddrspace(i64 [[COERCE_VAL_PI]])
@@ -664,7 +666,9 @@ void Tpaddrspace(Spaddrspace s) { *s.x = 1; }
// CHECK-A64_32-SAME: ) #[[ATTR0]] {
// CHECK-A64_32-NEXT: [[ENTRY:.*:]]
// CHECK-A64_32-NEXT: [[S:%.*]] = alloca [[STRUCT_SPADDRSPACE:%.*]], align 4
-// CHECK-A64_32-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SPADDRSPACE]], ptr [[S]], i32 0, i32 0
+// CHECK-A64_32-NEXT: [[AGG_TMP:%.*]] = alloca [[STRUCT_SPADDRSPACE]], align 4
+// CHECK-A64_32-NEXT: call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[AGG_TMP]], ptr align 4 [[S]], i32 4, i1 false)
+// CHECK-A64_32-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SPADDRSPACE]], ptr [[AGG_TMP]], i32 0, i32 0
// CHECK-A64_32-NEXT: [[TMP0:%.*]] = load ptr addrspace(100), ptr [[COERCE_DIVE]], align 4
// CHECK-A64_32-NEXT: [[COERCE_VAL_PI:%.*]] = ptrtoint ptr addrspace(100) [[TMP0]] to i32
// CHECK-A64_32-NEXT: [[COERCE_VAL_II:%.*]] = zext i32 [[COERCE_VAL_PI]] to i64
@@ -705,7 +709,9 @@ void Tp2addrspace(Sp2addrspace s) { *s.x[0] = 1; }
// CHECK-A64-SAME: ) #[[ATTR0]] {
// CHECK-A64-NEXT: [[ENTRY:.*:]]
// CHECK-A64-NEXT: [[S:%.*]] = alloca [[STRUCT_SP2ADDRSPACE:%.*]], align 8
-// CHECK-A64-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SP2ADDRSPACE]], ptr [[S]], i32 0, i32 0
+// CHECK-A64-NEXT: [[AGG_TMP:%.*]] = alloca [[STRUCT_SP2ADDRSPACE]], align 8
+// CHECK-A64-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[AGG_TMP]], ptr align 8 [[S]], i64 16, i1 false)
+// CHECK-A64-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SP2ADDRSPACE]], ptr [[AGG_TMP]], i32 0, i32 0
// CHECK-A64-NEXT: [[TMP0:%.*]] = load [2 x i64], ptr [[COERCE_DIVE]], align 8
// CHECK-A64-NEXT: call void @_Z12Tp2addrspace12Sp2addrspace([2 x i64] [[TMP0]])
// CHECK-A64-NEXT: ret void
@@ -714,7 +720,9 @@ void Tp2addrspace(Sp2addrspace s) { *s.x[0] = 1; }
// CHECK-A64_32-SAME: ) #[[ATTR0]] {
// CHECK-A64_32-NEXT: [[ENTRY:.*:]]
// CHECK-A64_32-NEXT: [[S:%.*]] = alloca [[STRUCT_SP2ADDRSPACE:%.*]], align 4
-// CHECK-A64_32-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SP2ADDRSPACE]], ptr [[S]], i32 0, i32 0
+// CHECK-A64_32-NEXT: [[AGG_TMP:%.*]] = alloca [[STRUCT_SP2ADDRSPACE]], align 4
+// CHECK-A64_32-NEXT: call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[AGG_TMP]], ptr align 4 [[S]], i32 8, i1 false)
+// CHECK-A64_32-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SP2ADDRSPACE]], ptr [[AGG_TMP]], i32 0, i32 0
// CHECK-A64_32-NEXT: [[TMP0:%.*]] = load i64, ptr [[COERCE_DIVE]], align 4
// CHECK-A64_32-NEXT: call void @_Z12Tp2addrspace12Sp2addrspace(i64 [[TMP0]])
// CHECK-A64_32-NEXT: ret void
@@ -732,7 +740,7 @@ struct Sraddrspace {
// CHECK-A64-NEXT: [[COERCE_VAL_IP:%.*]] = inttoptr i64 [[S_COERCE]] to ptr addrspace(100)
// CHECK-A64-NEXT: store ptr addrspace(100) [[COERCE_VAL_IP]], ptr [[COERCE_DIVE]], align 8
// CHECK-A64-NEXT: [[X:%.*]] = getelementptr inbounds nuw [[STRUCT_SRADDRSPACE]], ptr [[S]], i32 0, i32 0
-// CHECK-A64-NEXT: [[TMP0:%.*]] = load ptr addrspace(100), ptr [[X]], align 8, !align [[META2]]
+// CHECK-A64-NEXT: [[TMP0:%.*]] = load ptr addrspace(100), ptr [[X]], align 8, !align [[META3]]
// CHECK-A64-NEXT: store i32 1, ptr addrspace(100) [[TMP0]], align 4
// CHECK-A64-NEXT: ret void
//
@@ -744,7 +752,7 @@ struct Sraddrspace {
// CHECK-A64_32-NEXT: [[COERCE_VAL_II:%.*]] = trunc i64 [[S_COERCE]] to i32
// CHECK-A64_32-NEXT: store i32 [[COERCE_VAL_II]], ptr [[COERCE_DIVE]], align 4
// CHECK-A64_32-NEXT: [[X:%.*]] = getelementptr inbounds nuw [[STRUCT_SRADDRSPACE]], ptr [[S]], i32 0, i32 0
-// CHECK-A64_32-NEXT: [[TMP0:%.*]] = load ptr addrspace(100), ptr [[X]], align 4, !align [[META2]]
+// CHECK-A64_32-NEXT: [[TMP0:%.*]] = load ptr addrspace(100), ptr [[X]], align 4, !align [[META3]]
// CHECK-A64_32-NEXT: store i32 1, ptr addrspace(100) [[TMP0]], align 4
// CHECK-A64_32-NEXT: ret void
//
@@ -753,10 +761,12 @@ void Traddrspace(Sraddrspace s) { s.x = 1; }
// CHECK-A64-SAME: i64 [[S_COERCE:%.*]]) #[[ATTR0]] {
// CHECK-A64-NEXT: [[ENTRY:.*:]]
// CHECK-A64-NEXT: [[S:%.*]] = alloca [[STRUCT_SRADDRSPACE:%.*]], align 8
+// CHECK-A64-NEXT: [[AGG_TMP:%.*]] = alloca [[STRUCT_SRADDRSPACE]], align 8
// CHECK-A64-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SRADDRSPACE]], ptr [[S]], i32 0, i32 0
// CHECK-A64-NEXT: [[COERCE_VAL_IP:%.*]] = inttoptr i64 [[S_COERCE]] to ptr addrspace(100)
// CHECK-A64-NEXT: store ptr addrspace(100) [[COERCE_VAL_IP]], ptr [[COERCE_DIVE]], align 8
-// CHECK-A64-NEXT: [[COERCE_DIVE1:%.*]] = getelementptr inbounds nuw [[STRUCT_SRADDRSPACE]], ptr [[S]], i32 0, i32 0
+// CHECK-A64-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 8 [[AGG_TMP]], ptr align 8 [[S]], i64 8, i1 false)
+// CHECK-A64-NEXT: [[COERCE_DIVE1:%.*]] = getelementptr inbounds nuw [[STRUCT_SRADDRSPACE]], ptr [[AGG_TMP]], i32 0, i32 0
// CHECK-A64-NEXT: [[TMP0:%.*]] = load ptr addrspace(100), ptr [[COERCE_DIVE1]], align 8
// CHECK-A64-NEXT: [[COERCE_VAL_PI:%.*]] = ptrtoint ptr addrspace(100) [[TMP0]] to i64
// CHECK-A64-NEXT: call void @_Z11Traddrspace11Sraddrspace(i64 [[COERCE_VAL_PI]])
@@ -766,10 +776,12 @@ void Traddrspace(Sraddrspace s) { s.x = 1; }
// CHECK-A64_32-SAME: i64 [[S_COERCE:%.*]]) #[[ATTR0]] {
// CHECK-A64_32-NEXT: [[ENTRY:.*:]]
// CHECK-A64_32-NEXT: [[S:%.*]] = alloca [[STRUCT_SRADDRSPACE:%.*]], align 4
+// CHECK-A64_32-NEXT: [[AGG_TMP:%.*]] = alloca [[STRUCT_SRADDRSPACE]], align 4
// CHECK-A64_32-NEXT: [[COERCE_DIVE:%.*]] = getelementptr inbounds nuw [[STRUCT_SRADDRSPACE]], ptr [[S]], i32 0, i32 0
// CHECK-A64_32-NEXT: [[COERCE_VAL_II:%.*]] = trunc i64 [[S_COERCE]] to i32
// CHECK-A64_32-NEXT: store i32 [[COERCE_VAL_II]], ptr [[COERCE_DIVE]], align 4
-// CHECK-A64_32-NEXT: [[COERCE_DIVE1:%.*]] = getelementptr inbounds nuw [[STRUCT_SRADDRSPACE]], ptr [[S]], i32 0, i32 0
+// CHECK-A64_32-NEXT: call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[AGG_TMP]], ptr align 4 [[S]], i32 4, i1 false)
+// CHECK-A64_32-NEXT: [[COERCE_DIVE1:%.*]] = getelementptr inbounds nuw [[STRUCT_SRADDRSPACE]], ptr [[AGG_TMP]], i32 0, i32 0
// CHECK-A64_32-NEXT: [[TMP0:%.*]] = load ptr addrspace(100), ptr [[COERCE_DIVE1]], align 4
// CHECK-A64_32-NEXT: [[COERCE_VAL_PI:%.*]] = ptrtoint ptr addrspace(100) [[TMP0]] to i32
// CHECK-A64_32-NEXT: [[COERCE_VAL_II2:%.*]] = zext i32 [[COERCE_VAL_PI]] to i64
@@ -779,9 +791,9 @@ void Traddrspace(Sraddrspace s) { s.x = 1; }
void Craddrspace(Sraddrspace s) { Traddrspace(s); }
//.
-// CHECK-A64: [[META1]] = !{}
-// CHECK-A64: [[META2]] = !{i64 4}
+// CHECK-A64: [[META2]] = !{}
+// CHECK-A64: [[META3]] = !{i64 4}
//.
-// CHECK-A64_32: [[META1]] = !{}
-// CHECK-A64_32: [[META2]] = !{i64 4}
+// CHECK-A64_32: [[META2]] = !{}
+// CHECK-A64_32: [[META3]] = !{i64 4}
//.
diff --git a/clang/test/CodeGen/aapcs64-align.cpp b/clang/test/CodeGen/aapcs64-align.cpp
index 67c1feca6ded1..f37fc5566c3e5 100644
--- a/clang/test/CodeGen/aapcs64-align.cpp
+++ b/clang/test/CodeGen/aapcs64-align.cpp
@@ -122,7 +122,7 @@ unsigned sizeof_RidiculouslyOverSizedBitfield = sizeof(RidiculouslyOverSizedBitf
unsigned alignof_RidiculouslyOverSizedBitfield = alignof(RidiculouslyOverSizedBitfield);
// CHECK: define{{.*}} void @g9
-// CHECK: call void @f9(i32 noundef 1, ptr noundef nonnull align 16 dead_on_return %byval-temp)
+// CHECK: call void @f9(i32 noundef 1, ptr noundef nonnull align 16 dead_on_return %agg.tmp)
// CHECK: declare void @f9(i32 noundef, ptr noundef align 16 dead_on_return)
void f9(int a, RidiculouslyOverSizedBitfield b);
void g9() {
diff --git a/clang/test/CodeGenCXX/amdgcn-func-arg.cpp b/clang/test/CodeGenCXX/amdgcn-func-arg.cpp
index fed391bd5f9c2..175af8a97b2da 100644
--- a/clang/test/CodeGenCXX/amdgcn-func-arg.cpp
+++ b/clang/test/CodeGenCXX/amdgcn-func-arg.cpp
@@ -111,10 +111,12 @@ void func_with_byval_arg(B b) {
// CHECK-SAME: ) #[[ATTR1]] {
// CHECK-NEXT: [[ENTRY:.*:]]
// CHECK-NEXT: [[B:%.*]] = alloca [[CLASS_B:%.*]], align 4, addrspace(5)
-// CHECK-NEXT: [[BYVAL_TEMP:%.*]] = alloca [[CLASS_B]], align 4, addrspace(5)
+// CHECK-NEXT: [[AGG_TMP:%.*]] = alloca [[CLASS_B]], align 4, addrspace(5)
// CHECK-NEXT: [[B_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[B]] to ptr
-// CHECK-NEXT: call void @llvm.memcpy.p5.p0.i64(ptr addrspace(5) align 4 [[BYVAL_TEMP]], ptr align 4 [[B_ASCAST]], i64 400, i1 false)
-// CHECK-NEXT: call void @_Z19func_with_byval_arg1B(ptr addrspace(5) noundef byref([[CLASS_B]]) align 4 [[BYVAL_TEMP]]) #[[ATTR5]]
+// CHECK-NEXT: [[AGG_TMP_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[AGG_TMP]] to ptr
+// CHECK-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[AGG_TMP_ASCAST]], ptr align 4 [[B_ASCAST]], i64 400, i1 false)
+// CHECK-NEXT: [[AGG_TMP_ASCAST_ASCAST:%.*]] = addrspacecast ptr [[AGG_TMP_ASCAST]] to ptr addrspace(5)
+// CHECK-NEXT: call void @_Z19func_with_byval_arg1B(ptr addrspace(5) noundef byref([[CLASS_B]]) align 4 [[AGG_TMP_ASCAST_ASCAST]]) #[[ATTR5]]
// CHECK-NEXT: call void @_Z17func_with_ref_argR1B(ptr noundef nonnull align 4 dereferenceable(400) [[B_ASCAST]]) #[[ATTR5]]
// CHECK-NEXT: ret void
//
@@ -127,9 +129,11 @@ void test_byval_arg_auto() {
// CHECK-LABEL: define dso_local void @_Z21test_byval_arg_globalv(
// CHECK-SAME: ) #[[ATTR1]] {
// CHECK-NEXT: [[ENTRY:.*:]]
-// CHECK-NEXT: [[BYVAL_TEMP:%.*]] = alloca [[CLASS_B:%.*]], align 4, addrspace(5)
-// CHECK-NEXT: call void @llvm.memcpy.p5.p0.i64(ptr addrspace(5) align 4 [[BYVAL_TEMP]], ptr align 4 addrspacecast (ptr addrspace(1) @g_b to ptr), i64 400, i1 false)
-// CHECK-NEXT: call void @_Z19func_with_byval_arg1B(ptr addrspace(5) noundef byref([[CLASS_B]]) align 4 [[BYVAL_TEMP]]) #[[ATTR5]]
+// CHECK-NEXT: [[AGG_TMP:%.*]] = alloca [[CLASS_B:%.*]], align 4, addrspace(5)
+// CHECK-NEXT: [[AGG_TMP_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[AGG_TMP]] to ptr
+// CHECK-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[AGG_TMP_ASCAST]], ptr align 4 addrspacecast (ptr addrspace(1) @g_b to ptr), i64 400, i1 false)
+// CHECK-NEXT: [[AGG_TMP_ASCAST_ASCAST:%.*]] = addrspacecast ptr [[AGG_TMP_ASCAST]] to ptr addrspace(5)
+// CHECK-NEXT: call void @_Z19func_with_byval_arg1B(ptr addrspace(5) noundef byref([[CLASS_B]]) align 4 [[AGG_TMP_ASCAST_ASCAST]]) #[[ATTR5]]
// CHECK-NEXT: call void @_Z17func_with_ref_argR1B(ptr noundef nonnull align 4 dereferenceable(400) addrspacecast (ptr addrspace(1) @g_b to ptr)) #[[ATTR5]]
// CHECK-NEXT: ret void
//
diff --git a/clang/test/CodeGenCXX/assign-construct-memcpy.cpp b/clang/test/CodeGenCXX/assign-construct-memcpy.cpp
index 609dc31ce5829..7d6c5d76107da 100644
--- a/clang/test/CodeGenCXX/assign-construct-memcpy.cpp
+++ b/clang/test/CodeGenCXX/assign-construct-memcpy.cpp
@@ -81,13 +81,9 @@ void byval(foo f);
void test7(const foo &x) {
byval(x);
-// A trivially-copyable byval arg forwards the source; the byval boundary
-// makes the copy, so no separate memcpy is emitted.
// CHECK-POD: test7
-// CHECK-POD-NOT: call void @llvm.memcpy
-// CHECK-POD: call void @_Z5byval3foo(ptr noundef byval(%struct.foo) align 8
+// CHECK-POD: call void @llvm.memcpy.p0.p0.i64({{.*}} align 8 {{.*}} align 8 {{.*}}i64 24
// CHECK-NONPOD: test7
-// CHECK-NONPOD-NOT: call void @llvm.memcpy
-// CHECK-NONPOD: call void @_Z5byval3foo(ptr noundef byval(%struct.foo) align 8
+// CHECK-NONPOD: call void @llvm.memcpy.p0.p0.i64({{.*}} align 8 {{.*}} align 8 {{.*}}i64 24
}
diff --git a/clang/test/CodeGenCXX/cxx2b-deducing-this.cpp b/clang/test/CodeGenCXX/cxx2b-deducing-this.cpp
index d5dddf10af7a9..9664a866376ae 100644
--- a/clang/test/CodeGenCXX/cxx2b-deducing-this.cpp
+++ b/clang/test/CodeGenCXX/cxx2b-deducing-this.cpp
@@ -10,6 +10,7 @@ void test() {
// CHECK: define {{.*}}test{{.*}}
// CHECK-NEXT: entry:
// CHECK: {{.*}} = alloca %struct.TrivialStruct, align 1
+// CHECK: {{.*}} = alloca %struct.TrivialStruct, align 1
// CHECK: call void {{.*}}explicit_object_function{{.*}}
// CHECK-NEXT: ret void
// CHECK-NEXT: }
@@ -37,6 +38,7 @@ void test_lambda() {
//CHECK: define internal noundef i32 @"_ZZ11test_lambdavENH3$_0clIS_EEiT_"() #0 align 2 {
//CHECK: entry:
//CHECK: %This = alloca %class.anon, align 1
+//CHECK: %agg.tmp = alloca %class.anon, align 1
//CHECK: %call = call noundef i32 @"_ZZ11test_lambdavENH3$_0clIS_EEiT_"()
//CHECK: ret i32 %call
//CHECK: }
diff --git a/clang/test/CodeGenCXX/pr40771-ctad-with-lambda-copy-capture.cpp b/clang/test/CodeGenCXX/pr40771-ctad-with-lambda-copy-capture.cpp
index 10fce3d05870c..74e1f17652e17 100644
--- a/clang/test/CodeGenCXX/pr40771-ctad-with-lambda-copy-capture.cpp
+++ b/clang/test/CodeGenCXX/pr40771-ctad-with-lambda-copy-capture.cpp
@@ -10,7 +10,9 @@ T t { R{q}, S{q} };
// CHECK-LABEL: define internal void @__cxx_global_var_init.1() {{.*}} {
// CHECK-NEXT: [[TMP_R:%[a-z0-9.]+]] = alloca %struct.R, align 1
+// CHECK-NEXT: [[TMP_Q1:%[a-z0-9.]+]] = alloca %struct.Q, align 1
// CHECK-NEXT: [[TMP_S:%[a-z0-9.]+]] = alloca %struct.S, align 1
+// CHECK-NEXT: [[TMP_Q2:%[a-z0-9.]+]] = alloca %struct.Q, align 1
// CHECK-NEXT: [[XPT:%[a-z0-9.]+]] = alloca ptr
// CHECK-NEXT: [[SLOT:%[a-z0-9.]+]] = alloca i32
// CHECK-NEXT: [[ACTIVE:%[a-z0-9.]+]] = alloca i1, align 1
diff --git a/clang/test/CodeGenCXX/trivial-copy-arg-forward.cpp b/clang/test/CodeGenCXX/trivial-copy-arg-forward.cpp
deleted file mode 100644
index 1dabf5a7c1f3e..0000000000000
--- a/clang/test/CodeGenCXX/trivial-copy-arg-forward.cpp
+++ /dev/null
@@ -1,50 +0,0 @@
-// RUN: %clang_cc1 -triple x86_64-linux-gnu %s -emit-llvm -o - | FileCheck %s
-
-// A by-value argument built by a trivial copy/move constructor from a
-// variable forwards the variable's storage to the call; the byval boundary
-// makes the copy, so no agg.tmp is materialized. This is the C++ analog of
-// the LValueToRValue forwarding the C path already does.
-
-struct Triv {
- unsigned long a, b, c, d;
-};
-void sink(Triv);
-
-// Same-type source: forward, no temporary.
-void from_local() {
- Triv t;
- sink(t);
-}
-// CHECK-LABEL: define {{.*}}@_Z10from_localv(
-// CHECK: [[T:%.*]] = alloca %struct.Triv
-// CHECK-NOT: = alloca %struct.Triv
-// CHECK: call void @_Z4sink4Triv(ptr noundef byval(%struct.Triv) align 8 [[T]])
-
-void from_param(Triv t) {
- sink(t);
-}
-// CHECK-LABEL: define {{.*}}@_Z10from_param4Triv(
-// CHECK-NOT: = alloca %struct.Triv
-// CHECK: call void @_Z4sink4Triv(ptr noundef byval(%struct.Triv) align 8 %t)
-
-// Derived-to-base: forwarding the derived lvalue into a base-typed slot would
-// slice at the wrong offset, so the same-type guard keeps the temp and copies
-// from the adjusted base offset.
-struct Base1 {
- long a, b;
-};
-struct Base2 {
- long c, d;
-};
-struct Derived : Base1, Base2 {
- long e;
-};
-void sink_base(Base2);
-void slice_to_base(Derived d) {
- sink_base(d);
-}
-// CHECK-LABEL: define {{.*}}@_Z13slice_to_base7Derived(
-// CHECK: [[TMP:%.*]] = alloca %struct.Base2
-// CHECK: [[ADJ:%.*]] = getelementptr inbounds i8, ptr %d, i64 16
-// CHECK: call void @llvm.memcpy{{.*}}(ptr {{.*}} [[TMP]], ptr {{.*}} [[ADJ]],
-// CHECK: call void @_Z9sink_base5Base2(
diff --git a/clang/test/CodeGenCXX/varargs.cpp b/clang/test/CodeGenCXX/varargs.cpp
index 830faaaf4bda0..afffaf5554deb 100644
--- a/clang/test/CodeGenCXX/varargs.cpp
+++ b/clang/test/CodeGenCXX/varargs.cpp
@@ -31,7 +31,9 @@ namespace test1 {
}
// CHECK-LABEL: define{{.*}} void @_ZN5test14testEv()
// CHECK: [[X:%.*]] = alloca [[A:%.*]], align 4
- // CHECK-NEXT: [[T1:%.*]] = load i64, ptr [[X]], align 4
+ // CHECK-NEXT: [[TMP:%.*]] = alloca [[A]], align 4
+ // CHECK-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 4 [[TMP]], ptr align 4 [[X]], i64 8, i1 false)
+ // CHECK-NEXT: [[T1:%.*]] = load i64, ptr [[TMP]], align 4
// CHECK-NEXT: call void (...) @_ZN5test13fooEz(i64 [[T1]])
// CHECK-NEXT: ret void
}
diff --git a/clang/test/CodeGenObjCXX/gc.mm b/clang/test/CodeGenObjCXX/gc.mm
index e67470120362a..2a34f1f288e89 100644
--- a/clang/test/CodeGenObjCXX/gc.mm
+++ b/clang/test/CodeGenObjCXX/gc.mm
@@ -18,15 +18,3 @@
// CHECK-NEXT: call ptr @objc_assign_strongCast(ptr [[T2]], ptr [[T1]])
// CHECK-NEXT: ret void
}
-
-namespace test1 {
- // A trivially-copyable struct with an object member, passed by value, needs
- // the GC write barrier. Forwarding the source variable must not bypass it.
- struct S { id a; };
- void sink(S);
- void pass(S s) { sink(s); }
-
-// CHECK-LABEL: define{{.*}} void @_ZN5test14passENS_1SE(
-// CHECK: call ptr @objc_memmove_collectable(
-// CHECK: call void @_ZN5test14sinkENS_1SE(
-}
diff --git a/clang/test/CodeGenSYCL/kernel-caller-entry-point.cpp b/clang/test/CodeGenSYCL/kernel-caller-entry-point.cpp
index 3fa91e527621d..56558cec82c56 100644
--- a/clang/test/CodeGenSYCL/kernel-caller-entry-point.cpp
+++ b/clang/test/CodeGenSYCL/kernel-caller-entry-point.cpp
@@ -160,6 +160,7 @@ int main() {
// CHECK-HOST-LINUX: define dso_local void @_Z26single_purpose_kernel_task21single_purpose_kernel() #{{[0-9]+}} {
// CHECK-HOST-LINUX-NEXT: entry:
// CHECK-HOST-LINUX-NEXT: %kernelFunc = alloca %struct.single_purpose_kernel, align 1
+// CHECK-HOST-LINUX-NEXT: %agg.tmp = alloca %struct.single_purpose_kernel, align 1
// CHECK-HOST-LINUX-NEXT: call void @_Z18sycl_kernel_launchI26single_purpose_kernel_nameJ21single_purpose_kernelEEvPKcDpT0_(ptr noundef @.str)
// CHECK-HOST-LINUX-NEXT: ret void
// CHECK-HOST-LINUX-NEXT: }
@@ -167,9 +168,11 @@ int main() {
// CHECK-HOST-LINUX: define internal void @_Z18kernel_single_taskIZ4mainEUlT_E_S1_EvT0_(i32 %kernelFunc.coerce) #{{[0-9]+}} {
// CHECK-HOST-LINUX-NEXT: entry:
// CHECK-HOST-LINUX-NEXT: %kernelFunc = alloca %class.anon, align 4
+// CHECK-HOST-LINUX-NEXT: %agg.tmp = alloca %class.anon, align 4
// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon, ptr %kernelFunc, i32 0, i32 0
// CHECK-HOST-LINUX-NEXT: store i32 %kernelFunc.coerce, ptr %coerce.dive, align 4
-// CHECK-HOST-LINUX-NEXT: %coerce.dive1 = getelementptr inbounds nuw %class.anon, ptr %kernelFunc, i32 0, i32 0
+// CHECK-HOST-LINUX-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 4 %agg.tmp, ptr align 4 %kernelFunc, i64 4, i1 false)
+// CHECK-HOST-LINUX-NEXT: %coerce.dive1 = getelementptr inbounds nuw %class.anon, ptr %agg.tmp, i32 0, i32 0
// CHECK-HOST-LINUX-NEXT: %0 = load i32, ptr %coerce.dive1, align 4
// CHECK-HOST-LINUX-NEXT: call void @_Z18sycl_kernel_launchIZ4mainEUlT_E_JS1_EEvPKcDpT0_(ptr noundef @.str.1, i32 %0)
// CHECK-HOST-LINUX-NEXT: ret void
@@ -178,6 +181,7 @@ int main() {
// CHECK-HOST-LINUX: define internal void @"_Z18kernel_single_taskI6\CE\B4\CF\84\CF\87Z4mainEUliE_EvT0_"() #{{[0-9]+}} {
// CHECK-HOST-LINUX-NEXT: entry:
// CHECK-HOST-LINUX-NEXT: %kernelFunc = alloca %class.anon.0, align 1
+// CHECK-HOST-LINUX-NEXT: %agg.tmp = alloca %class.anon.0, align 1
// CHECK-HOST-LINUX-NEXT: call void @"_Z18sycl_kernel_launchI6\CE\B4\CF\84\CF\87JZ4mainEUliE_EEvPKcDpT0_"(ptr noundef @.str.2)
// CHECK-HOST-LINUX-NEXT: ret void
// CHECK-HOST-LINUX-NEXT: }
@@ -206,9 +210,11 @@ int main() {
// CHECK-HOST-LINUX: define internal void @_Z14ref_arg_kernelI19ref_arg_kernel_nameZ4mainEUlT_E_EvRKT0_(ptr noundef nonnull align 4 dereferenceable(4) %ref) #{{[0-9]+}} {
// CHECK-HOST-LINUX-NEXT: entry:
// CHECK-HOST-LINUX-NEXT: %ref.addr = alloca ptr, align 8
+// CHECK-HOST-LINUX-NEXT: %agg.tmp = alloca %class.anon, align 4
// CHECK-HOST-LINUX-NEXT: store ptr %ref, ptr %ref.addr, align 8
// CHECK-HOST-LINUX-NEXT: %0 = load ptr, ptr %ref.addr, align 8
-// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon, ptr %0, i32 0, i32 0
+// CHECK-HOST-LINUX-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 4 %agg.tmp, ptr align 4 %0, i64 4, i1 false)
+// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon, ptr %agg.tmp, i32 0, i32 0
// CHECK-HOST-LINUX-NEXT: %1 = load i32, ptr %coerce.dive, align 4
// CHECK-HOST-LINUX-NEXT: call void @_Z18sycl_kernel_launchI19ref_arg_kernel_nameJZ4mainEUlT_E_EEvPKcDpT0_(ptr noundef @.str.4, i32 %1)
// CHECK-HOST-LINUX-NEXT: ret void
@@ -217,9 +223,11 @@ int main() {
// CHECK-HOST-LINUX: define internal void @_Z18fwd_ref_arg_kernelI23fwd_ref_arg_kernel_nameRZ4mainEUlT_E_EvOT0_(ptr noundef nonnull align 4 dereferenceable(4) %ref) #{{[0-9]+}} {
// CHECK-HOST-LINUX-NEXT: entry:
// CHECK-HOST-LINUX-NEXT: %ref.addr = alloca ptr, align 8
+// CHECK-HOST-LINUX-NEXT: %agg.tmp = alloca %class.anon, align 4
// CHECK-HOST-LINUX-NEXT: store ptr %ref, ptr %ref.addr, align 8
// CHECK-HOST-LINUX-NEXT: %0 = load ptr, ptr %ref.addr, align 8
-// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon, ptr %0, i32 0, i32 0
+// CHECK-HOST-LINUX-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 4 %agg.tmp, ptr align 4 %0, i64 4, i1 false)
+// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon, ptr %agg.tmp, i32 0, i32 0
// CHECK-HOST-LINUX-NEXT: %1 = load i32, ptr %coerce.dive, align 4
// CHECK-HOST-LINUX-NEXT: call void @_Z18sycl_kernel_launchI23fwd_ref_arg_kernel_nameJZ4mainEUlT_E_EEvPKcDpT0_(ptr noundef @.str.5, i32 %1)
// CHECK-HOST-LINUX-NEXT: ret void
@@ -228,9 +236,11 @@ int main() {
// CHECK-HOST-LINUX: define internal void @_Z18fwd_ref_arg_kernelI28fwd_ref_arg_kernel_name_moveZ4mainEUlT_E_EvOT0_(ptr noundef nonnull align 4 dereferenceable(4) %ref) #{{[0-9]+}} {
// CHECK-HOST-LINUX-NEXT: entry:
// CHECK-HOST-LINUX-NEXT: %ref.addr = alloca ptr, align 8
+// CHECK-HOST-LINUX-NEXT: %agg.tmp = alloca %class.anon, align 4
// CHECK-HOST-LINUX-NEXT: store ptr %ref, ptr %ref.addr, align 8
// CHECK-HOST-LINUX-NEXT: %0 = load ptr, ptr %ref.addr, align 8
-// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon, ptr %0, i32 0, i32 0
+// CHECK-HOST-LINUX-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 4 %agg.tmp, ptr align 4 %0, i64 4, i1 false)
+// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon, ptr %agg.tmp, i32 0, i32 0
// CHECK-HOST-LINUX-NEXT: %1 = load i32, ptr %coerce.dive, align 4
// CHECK-HOST-LINUX-NEXT: call void @_Z18sycl_kernel_launchI28fwd_ref_arg_kernel_name_moveJZ4mainEUlT_E_EEvPKcDpT0_(ptr noundef @.str.6, i32 %1)
// CHECK-HOST-LINUX-NEXT: ret void
@@ -239,9 +249,11 @@ int main() {
// CHECK-HOST-LINUX: define internal void @_Z21rvalue_ref_arg_kernelI26rvalue_ref_arg_kernel_nameZ4mainEUlT_E0_EvONSt13type_identityIT0_E4typeE(ptr noundef nonnull align 4 dereferenceable(4) %ref) #{{[0-9]+}} {
// CHECK-HOST-LINUX-NEXT: entry:
// CHECK-HOST-LINUX-NEXT: %ref.addr = alloca ptr, align 8
+// CHECK-HOST-LINUX-NEXT: %agg.tmp = alloca %class.anon.2, align 4
// CHECK-HOST-LINUX-NEXT: store ptr %ref, ptr %ref.addr, align 8
// CHECK-HOST-LINUX-NEXT: %0 = load ptr, ptr %ref.addr, align 8
-// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon.2, ptr %0, i32 0, i32 0
+// CHECK-HOST-LINUX-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr align 4 %agg.tmp, ptr align 4 %0, i64 4, i1 false)
+// CHECK-HOST-LINUX-NEXT: %coerce.dive = getelementptr inbounds nuw %class.anon.2, ptr %agg.tmp, i32 0, i32 0
// CHECK-HOST-LINUX-NEXT: %1 = load i32, ptr %coerce.dive, align 4
// CHECK-HOST-LINUX-NEXT: call void @_Z18sycl_kernel_launchI26rvalue_ref_arg_kernel_nameJZ4mainEUlT_E0_EEvPKcDpT0_(ptr noundef @.str.7, i32 %1)
// CHECK-HOST-LINUX-NEXT: ret void
>From beda9ba015ab1495ced89016212768e0e4adb256 Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Fri, 3 Jul 2026 08:58:26 +0200
Subject: [PATCH 17/21] [Clang] Accept any same-type glvalue as musttail
forwarding source
The trivial-copy forwarding in EmitCallArg required the source to be a
DeclRefExpr to a local or parameter, after stripping implicit casts;
the stripping is what forced that narrow match, since it could drop a
derived-to-base adjustment. Per review, match any same-type glvalue
without stripping casts: member accesses, dereferences, and globals now
forward too, and a derived-to-base source keeps its adjusted base
subobject address.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.6 <noreply at anthropic.com>
---
clang/lib/CodeGen/CGCall.cpp | 36 +++++++-------
clang/test/CodeGen/musttail-indirect-arg.cpp | 49 ++++++++++++++++++--
clang/test/CodeGenCUDA/surface.cu | 11 +++++
3 files changed, 74 insertions(+), 22 deletions(-)
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index af5477c69a8d8..b6a0b4dd088e9 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5255,31 +5255,29 @@ void CodeGenFunction::EmitCallArg(CallArgList &args, const Expr *E,
// Under musttail, hand a trivially-copyable record source's LValue to
// EmitCall rather than materializing an agg.tmp. EmitCall's Indirect path
- // routes it via the matching incoming parameter, which survives the tail
- // call. Limited to params and locals: globals and captures don't have the
- // dangle issue and the existing path may be more efficient for them.
- // On the device side CUDA surface/texture types are excluded: they
- // classify as Direct and forwarding would load raw record bytes instead
- // of the handle that EmitAggregateCopy materializes.
+ // copies it into the matching incoming parameter, which survives the tail
+ // call. Any same-type glvalue works as the source; casts are not stripped,
+ // so a derived-to-base source keeps its adjusted address. On the device
+ // side CUDA surface/texture types are excluded: they classify as Direct
+ // and forwarding would load raw record bytes instead of the handle that
+ // EmitAggregateCopy materializes.
if (HasAggregateEvalKind && MustTailCall && type->isRecordType() &&
type.isTriviallyCopyableType(getContext()) &&
!(getLangOpts().CUDAIsDevice &&
(type->isCUDADeviceBuiltinSurfaceType() ||
type->isCUDADeviceBuiltinTextureType()))) {
if (const auto *CCE = dyn_cast<CXXConstructExpr>(E)) {
- if (CCE->getConstructor()->isCopyOrMoveConstructor() &&
- CCE->getConstructor()->isTrivial() && CCE->getNumArgs() == 1) {
- const Expr *Source = CCE->getArg(0)->IgnoreParenImpCasts();
- if (const auto *DRE = dyn_cast<DeclRefExpr>(Source)) {
- if (const auto *VD = dyn_cast<VarDecl>(DRE->getDecl())) {
- if (VD->hasLocalStorage() ||
- (isa<ParmVarDecl>(VD) &&
- VD->getDeclContext() == dyn_cast<DeclContext>(CurCodeDecl))) {
- LValue L = EmitLValue(DRE);
- assert(L.isSimple());
- args.addUncopiedAggregate(L, type);
- return;
- }
+ const CXXConstructorDecl *Ctor = CCE->getConstructor();
+ if (Ctor->isCopyOrMoveConstructor() && Ctor->isTrivial() &&
+ CCE->getNumArgs() == 1) {
+ const Expr *Source = CCE->getArg(0);
+ if (Source->isGLValue() &&
+ Source->getType().getAddressSpace() != LangAS::hlsl_constant &&
+ getContext().hasSameUnqualifiedType(Source->getType(), type)) {
+ LValue L = EmitLValue(Source);
+ if (L.isSimple()) {
+ args.addUncopiedAggregate(L, type);
+ return;
}
}
}
diff --git a/clang/test/CodeGen/musttail-indirect-arg.cpp b/clang/test/CodeGen/musttail-indirect-arg.cpp
index 2c5989a81d24c..dd65e73bddbef 100644
--- a/clang/test/CodeGen/musttail-indirect-arg.cpp
+++ b/clang/test/CodeGen/musttail-indirect-arg.cpp
@@ -5,9 +5,9 @@
// C++ side of the musttail Indirect-arg fix. The call argument is typically
// a CXXConstructExpr invoking the trivial copy constructor; EmitCallArg
-// detects the trivial-copy-from-DeclRefExpr case under musttail and hands
-// the source LValue to EmitCall so the general path engages. Non-trivial
-// copy or move constructors keep the existing agg.tmp path.
+// hands the same-type glvalue source's LValue to EmitCall so the general
+// path engages. Non-trivial copy or move constructors keep the existing
+// agg.tmp path.
struct Big {
unsigned long long a, b, c, d;
@@ -161,3 +161,46 @@ struct Big P17(struct Big a, struct Big b, struct Big c) {
// COMMON: @llvm.mem{{(cpy|move)}}{{.*}}(ptr {{[^,]*}} %b, ptr {{[^,]*}} %a,
// COMMON: store {{.*}} [[SAVED]], ptr %c,
// COMMON: musttail call {{.*}} @_Z3C173BigS_S_({{.*}}, ptr {{[^,]*}} %a, ptr {{[^,]*}} %b, ptr {{[^,]*}} %c)
+
+// P18: member of a global as the source. Forwarded with no agg.tmp; the copy
+// lands directly in the incoming %a.
+struct Wrap {
+ struct Big inner;
+};
+extern Wrap gw;
+struct Big C18(struct Big a);
+struct Big P18(struct Big a) {
+ [[clang::musttail]] return C18(gw.inner);
+}
+// COMMON-LABEL: define {{.*}} @_Z3P183Big(
+// COMMON-NOT: %agg.tmp
+// COMMON: @llvm.mem{{(cpy|move)}}{{.*}}(ptr {{[^,]*}} %a, ptr {{[^,]*}} @gw, i64 32
+// COMMON: musttail call {{.*}} @_Z3C183Big({{.*}}, ptr {{[^,]*}} %a)
+
+// P19: deref of a global pointer as the source.
+extern struct Big *gp;
+struct Big C19(struct Big a);
+struct Big P19(struct Big a) {
+ [[clang::musttail]] return C19(*gp);
+}
+// COMMON-LABEL: define {{.*}} @_Z3P193Big(
+// COMMON-NOT: %agg.tmp
+// COMMON: [[SRC:%[0-9a-z.]+]] = load ptr, ptr @gp
+// COMMON: @llvm.mem{{(cpy|move)}}{{.*}}(ptr {{[^,]*}} %a, ptr {{[^,]*}} [[SRC]], i64 32
+// COMMON: musttail call {{.*}} @_Z3C193Big({{.*}}, ptr {{[^,]*}} %a)
+
+// P20: derived-to-base source. The base subobject sits at offset 8 in Der;
+// the forwarded address must carry that adjustment.
+struct Pad {
+ unsigned long long p;
+};
+struct Der : Pad, Big {};
+extern Der gd;
+struct Big C20(struct Big a);
+struct Big P20(struct Big a) {
+ [[clang::musttail]] return C20(gd);
+}
+// COMMON-LABEL: define {{.*}} @_Z3P203Big(
+// COMMON-NOT: %agg.tmp
+// COMMON: @llvm.mem{{(cpy|move)}}{{.*}}(ptr {{[^,]*}} %a, ptr {{[^,]*}} getelementptr inbounds {{(nuw )?}}(i8, ptr @gd, i64 8), i64 32
+// COMMON: musttail call {{.*}} @_Z3C203Big({{.*}}, ptr {{[^,]*}} %a)
diff --git a/clang/test/CodeGenCUDA/surface.cu b/clang/test/CodeGenCUDA/surface.cu
index 4106673f3138a..883ff78ac5601 100644
--- a/clang/test/CodeGenCUDA/surface.cu
+++ b/clang/test/CodeGenCUDA/surface.cu
@@ -34,6 +34,17 @@ __attribute__((device)) int foo(int x, int y) {
return suld_2d_zero(surf, x, y);
}
+__attribute__((device)) int musttail_callee(surface<void, 2> s, int x);
+
+// A surface passed to a musttail call must keep the handle materialization;
+// the argument is not forwarded as a raw lvalue.
+// DEVICE-LABEL: @_Z15musttail_caller7surfaceIvLi2EEi(
+// DEVICE: call i64 @llvm.nvvm.texsurf.handle.internal.p1(ptr addrspace(1) @surf)
+// DEVICE: musttail call noundef i32 @_Z15musttail_callee7surfaceIvLi2EEi(i64
+__attribute__((device)) int musttail_caller(surface<void, 2> s, int x) {
+ [[clang::musttail]] return musttail_callee(surf, x);
+}
+
// HOST: define internal void @[[PREFIX:__cuda]]_register_globals
// Texture references need registering with correct arguments.
// HOST: call void @[[PREFIX]]RegisterSurface(ptr %0, ptr @surf, ptr @0, ptr @0, i32 2, i32 0)
>From d6fe44f9b4041bad966b3ab82060f88f84f4dfb5 Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Fri, 3 Jul 2026 08:58:27 +0200
Subject: [PATCH 18/21] [Clang][test] Note the unsupported musttail Indirect
case is liftable
Per review, make clear the no-addressable-source rejection is an
implementation limit that could be supported, not a fundamental one.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.6 <noreply at anthropic.com>
---
clang/test/CodeGen/musttail-indirect-arg-unsupported.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/clang/test/CodeGen/musttail-indirect-arg-unsupported.c b/clang/test/CodeGen/musttail-indirect-arg-unsupported.c
index bf938406aa7ba..108968ffe7784 100644
--- a/clang/test/CodeGen/musttail-indirect-arg-unsupported.c
+++ b/clang/test/CodeGen/musttail-indirect-arg-unsupported.c
@@ -6,6 +6,8 @@
// evaluation kind, so the argument is a scalar value with no source storage
// to forward, regardless of value category. Such a call is rejected rather
// than routed through a caller-frame temp that dangles past the tail call.
+// This is an implementation limit, not a fundamental one: it could be lifted
+// by storing the value through the incoming parameter's own slot.
typedef _BitInt(256) BI;
BI cee(BI x);
>From c3ba863a776de1aa5b66f4ee747502b478621e84 Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Fri, 3 Jul 2026 09:32:37 +0200
Subject: [PATCH 19/21] [Clang] Restrict musttail forwarding sources to pure
lvalue chains
Forwarding defers the source byte read to the call boundary, after the
other arguments have been evaluated. For a source whose address
computation has side effects or reads mutable state (a dereference or a
call), that splits the argument's evaluation around the other
arguments', an interleaving C++17 [expr.call]/8 forbids. For a pure
chain (a variable, dot-member access, derived-to-base adjustment) the
deferral is equivalent to initializing that argument last, which is a
legal ordering.
Restrict the match to pure chains; impure sources fall back to the temp
path, which reads at argument position and still routes through the
incoming parameter without dangling. This also keeps the sanitizer
checks the temp path emits for dereferenced sources.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 4.6 <noreply at anthropic.com>
---
clang/lib/CodeGen/CGCall.cpp | 33 ++++++++++++++++----
clang/test/CodeGen/musttail-indirect-arg.cpp | 20 +++++++++++-
2 files changed, 46 insertions(+), 7 deletions(-)
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index b6a0b4dd088e9..5b23e84fc17ad 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5173,6 +5173,26 @@ void CodeGenFunction::EmitWritebacks(const CallArgList &args) {
emitWriteback(*this, I);
}
+/// Whether emitting this glvalue neither has side effects nor reads mutable
+/// state, so deferring its byte read to the call boundary is equivalent to
+/// initializing the argument last, a sequencing C++17 [expr.call]/8 allows.
+/// A dereference or call in the address computation would instead split the
+/// argument's evaluation around the other arguments'.
+static bool isPureForwardableLValue(const Expr *E) {
+ E = E->IgnoreParens();
+ if (const auto *DRE = dyn_cast<DeclRefExpr>(E))
+ return isa<VarDecl>(DRE->getDecl());
+ if (const auto *ME = dyn_cast<MemberExpr>(E))
+ return !ME->isArrow() && isa<FieldDecl>(ME->getMemberDecl()) &&
+ isPureForwardableLValue(ME->getBase());
+ if (const auto *ICE = dyn_cast<ImplicitCastExpr>(E))
+ if (ICE->getCastKind() == CK_DerivedToBase ||
+ ICE->getCastKind() == CK_UncheckedDerivedToBase ||
+ ICE->getCastKind() == CK_NoOp)
+ return isPureForwardableLValue(ICE->getSubExpr());
+ return false;
+}
+
void CodeGenFunction::EmitCallArg(CallArgList &args, const Expr *E,
QualType type) {
std::optional<DisableDebugLocationUpdates> Dis;
@@ -5256,11 +5276,12 @@ void CodeGenFunction::EmitCallArg(CallArgList &args, const Expr *E,
// Under musttail, hand a trivially-copyable record source's LValue to
// EmitCall rather than materializing an agg.tmp. EmitCall's Indirect path
// copies it into the matching incoming parameter, which survives the tail
- // call. Any same-type glvalue works as the source; casts are not stripped,
- // so a derived-to-base source keeps its adjusted address. On the device
- // side CUDA surface/texture types are excluded: they classify as Direct
- // and forwarding would load raw record bytes instead of the handle that
- // EmitAggregateCopy materializes.
+ // call. The byte read is deferred to the call boundary, so the source is
+ // restricted to pure lvalue chains (see isPureForwardableLValue); casts
+ // are not stripped, so a derived-to-base source keeps its adjusted
+ // address. On the device side CUDA surface/texture types are excluded:
+ // they classify as Direct and forwarding would load raw record bytes
+ // instead of the handle that EmitAggregateCopy materializes.
if (HasAggregateEvalKind && MustTailCall && type->isRecordType() &&
type.isTriviallyCopyableType(getContext()) &&
!(getLangOpts().CUDAIsDevice &&
@@ -5271,7 +5292,7 @@ void CodeGenFunction::EmitCallArg(CallArgList &args, const Expr *E,
if (Ctor->isCopyOrMoveConstructor() && Ctor->isTrivial() &&
CCE->getNumArgs() == 1) {
const Expr *Source = CCE->getArg(0);
- if (Source->isGLValue() &&
+ if (Source->isGLValue() && isPureForwardableLValue(Source) &&
Source->getType().getAddressSpace() != LangAS::hlsl_constant &&
getContext().hasSameUnqualifiedType(Source->getType(), type)) {
LValue L = EmitLValue(Source);
diff --git a/clang/test/CodeGen/musttail-indirect-arg.cpp b/clang/test/CodeGen/musttail-indirect-arg.cpp
index dd65e73bddbef..dd05eab868843 100644
--- a/clang/test/CodeGen/musttail-indirect-arg.cpp
+++ b/clang/test/CodeGen/musttail-indirect-arg.cpp
@@ -177,7 +177,9 @@ struct Big P18(struct Big a) {
// COMMON: @llvm.mem{{(cpy|move)}}{{.*}}(ptr {{[^,]*}} %a, ptr {{[^,]*}} @gw, i64 32
// COMMON: musttail call {{.*}} @_Z3C183Big({{.*}}, ptr {{[^,]*}} %a)
-// P19: deref of a global pointer as the source.
+// P19: deref of a global pointer. The address computation reads mutable
+// state, so the bytes are captured at argument position (no forwarding);
+// the temp still routes through the incoming parameter.
extern struct Big *gp;
struct Big C19(struct Big a);
struct Big P19(struct Big a) {
@@ -204,3 +206,19 @@ struct Big P20(struct Big a) {
// COMMON-NOT: %agg.tmp
// COMMON: @llvm.mem{{(cpy|move)}}{{.*}}(ptr {{[^,]*}} %a, ptr {{[^,]*}} getelementptr inbounds {{(nuw )?}}(i8, ptr @gd, i64 8), i64 32
// COMMON: musttail call {{.*}} @_Z3C203Big({{.*}}, ptr {{[^,]*}} %a)
+
+// P21: impure source with a side-effecting second argument. The source bytes
+// are read at argument position, before bump() runs, so the argument's
+// evaluation is not interleaved with the other argument's ([expr.call]/8).
+extern int bump();
+struct Big C21(struct Big x, int y);
+struct Big P21(struct Big a, int b) {
+ [[clang::musttail]] return C21(*gp, bump());
+}
+// COMMON-LABEL: define {{.*}} @_Z3P213Bigi(
+// COMMON: [[SRC:%[0-9a-z.]+]] = load ptr, ptr @gp
+// COMMON-NOT: @_Z4bumpv
+// COMMON: [[VAL:%[0-9a-z.]+]] = load {{.*}}, ptr [[SRC]]
+// COMMON: call {{.*}} @_Z4bumpv()
+// COMMON: store {{.*}} [[VAL]], ptr %a
+// COMMON: musttail call {{.*}} @_Z3C213Bigi({{.*}}, ptr {{[^,]*}} %a,
>From feb571352f5a606752a64cb6f4e6e22b656c06c1 Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Thu, 6 Aug 2026 10:49:28 +0200
Subject: [PATCH 20/21] [Clang] Do not defer a musttail argument read when the
operand order is fixed
An overloaded operator keeps the built-in operand sequencing
([over.match.oper]/2), so handing EmitCall the source lvalue and reading it at
the call boundary can move that read past another operand's side effect. Both
directions are affected: the left-to-right operators, and assignment, where
[expr.assign]/1 sequences the right operand first.
EmitCallArgs already receives an EvaluationOrder. Pass it into EmitCallArg and
decline to forward when the order is fixed. The switch that computes it omits
OO_Subscript and OO_Call, so a C++23 explicit-object operator[] leaves the
order looking unspecified while [expr.sub]/1 still fixes it. EmitCall reports
any overloaded operator call separately rather than relying on that switch.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 5 <noreply at anthropic.com>
---
clang/lib/CodeGen/CGCall.cpp | 21 ++--
clang/lib/CodeGen/CGExpr.cpp | 5 +-
clang/lib/CodeGen/CodeGenFunction.h | 12 ++-
clang/test/CodeGen/musttail-indirect-arg.cpp | 103 +++++++++++++++++++
4 files changed, 129 insertions(+), 12 deletions(-)
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index 509675d0deaef..18a38672c2503 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5033,7 +5033,8 @@ static bool isObjCMethodWithTypeParams(const ObjCMethodDecl *method) {
void CodeGenFunction::EmitCallArgs(
CallArgList &Args, PrototypeWrapper Prototype,
llvm::iterator_range<CallExpr::const_arg_iterator> ArgRange,
- AbstractCallee AC, unsigned ParamsToSkip, EvaluationOrder Order) {
+ AbstractCallee AC, unsigned ParamsToSkip, EvaluationOrder Order,
+ bool OperandOrderFixed) {
SmallVector<QualType, 16> ArgTypes;
assert((ParamsToSkip == 0 || Prototype.P) &&
@@ -5106,6 +5107,10 @@ void CodeGenFunction::EmitCallArgs(
? Order == EvaluationOrder::ForceLeftToRight
: Order != EvaluationOrder::ForceRightToLeft;
+ // Order tracks only the emission direction. OperandOrderFixed additionally
+ // marks operators whose built-in form fixes the operand order.
+ bool PrescribedOrder = Order != EvaluationOrder::Default || OperandOrderFixed;
+
auto MaybeEmitImplicitObjectSize = [&](unsigned I, const Expr *Arg,
RValue EmittedArg) {
if (!AC.hasFunctionDecl() || I >= AC.getNumParams())
@@ -5148,7 +5153,7 @@ void CodeGenFunction::EmitCallArgs(
(isa<ObjCMethodDecl>(AC.getDecl()) &&
isObjCMethodWithTypeParams(cast<ObjCMethodDecl>(AC.getDecl())))) &&
"Argument and parameter types don't match");
- EmitCallArg(Args, *Arg, ArgTypes[Idx]);
+ EmitCallArg(Args, *Arg, ArgTypes[Idx], PrescribedOrder);
// In particular, we depend on it being the last arg in Args, and the
// objectsize bits depend on there only being one arg if !LeftToRight.
assert(InitialArgSize + 1 == Args.size() &&
@@ -5234,9 +5239,9 @@ void CodeGenFunction::EmitWritebacks(const CallArgList &args) {
/// Whether emitting this glvalue neither has side effects nor reads mutable
/// state, so deferring its byte read to the call boundary is equivalent to
-/// initializing the argument last, a sequencing C++17 [expr.call]/8 allows.
-/// A dereference or call in the address computation would instead split the
-/// argument's evaluation around the other arguments'.
+/// evaluating the argument last. A dereference or call in the address
+/// computation would instead split the argument's evaluation around the other
+/// arguments'.
static bool isPureForwardableLValue(const Expr *E) {
E = E->IgnoreParens();
if (const auto *DRE = dyn_cast<DeclRefExpr>(E))
@@ -5253,7 +5258,7 @@ static bool isPureForwardableLValue(const Expr *E) {
}
void CodeGenFunction::EmitCallArg(CallArgList &args, const Expr *E,
- QualType type) {
+ QualType type, bool PrescribedOrder) {
std::optional<DisableDebugLocationUpdates> Dis;
if (isa<CXXDefaultArgExpr>(E))
Dis.emplace(*this);
@@ -5341,8 +5346,8 @@ void CodeGenFunction::EmitCallArg(CallArgList &args, const Expr *E,
// address. On the device side CUDA surface/texture types are excluded:
// they classify as Direct and forwarding would load raw record bytes
// instead of the handle that EmitAggregateCopy materializes.
- if (HasAggregateEvalKind && MustTailCall && type->isRecordType() &&
- type.isTriviallyCopyableType(getContext()) &&
+ if (HasAggregateEvalKind && MustTailCall && !PrescribedOrder &&
+ type->isRecordType() && type.isTriviallyCopyableType(getContext()) &&
!(getLangOpts().CUDAIsDevice &&
(type->isCUDADeviceBuiltinSurfaceType() ||
type->isCUDADeviceBuiltinTextureType()))) {
diff --git a/clang/lib/CodeGen/CGExpr.cpp b/clang/lib/CodeGen/CGExpr.cpp
index 9201e40bc13a1..569c136977aa5 100644
--- a/clang/lib/CodeGen/CGExpr.cpp
+++ b/clang/lib/CodeGen/CGExpr.cpp
@@ -7137,8 +7137,11 @@ RValue CodeGenFunction::EmitCall(QualType CalleeType,
EmitIgnoredExpr(E->getArg(0));
Arguments = drop_begin(Arguments, 1);
}
+ // Every overloaded operator keeps its built-in operand sequencing
+ // ([over.match.oper]/2), including the ones missing from the switch above.
EmitCallArgs(Args, dyn_cast<FunctionProtoType>(FnType), Arguments,
- E->getDirectCallee(), /*ParamsToSkip=*/0, Order);
+ E->getDirectCallee(), /*ParamsToSkip=*/0, Order,
+ /*OperandOrderFixed=*/isa<CXXOperatorCallExpr>(E));
const CGFunctionInfo &FnInfo = CGM.getTypes().arrangeFreeFunctionCall(
Args, FnType, /*ChainCall=*/Chain, getCurrentFunctionDecl());
diff --git a/clang/lib/CodeGen/CodeGenFunction.h b/clang/lib/CodeGen/CodeGenFunction.h
index 783f97dc354eb..ded4ec77c590d 100644
--- a/clang/lib/CodeGen/CodeGenFunction.h
+++ b/clang/lib/CodeGen/CodeGenFunction.h
@@ -5439,8 +5439,11 @@ class CodeGenFunction : public CodeGenTypeCache {
/// EmitWriteback - Emit callbacks for function.
void EmitWritebacks(const CallArgList &Args);
- /// EmitCallArg - Emit a single call argument.
- void EmitCallArg(CallArgList &args, const Expr *E, QualType ArgType);
+ /// EmitCallArg - Emit a single call argument. \p PrescribedOrder is set when
+ /// the language fixes this call's argument evaluation order, which forbids
+ /// deferring an argument's evaluation past the other arguments'.
+ void EmitCallArg(CallArgList &args, const Expr *E, QualType ArgType,
+ bool PrescribedOrder);
/// EmitDelegateCallArg - We are performing a delegate call; that
/// is, the current function is delegating to another one. Produce
@@ -5577,11 +5580,14 @@ class CodeGenFunction : public CodeGenTypeCache {
PrototypeWrapper(const ObjCMethodDecl *MD) : P(MD) {}
};
+ /// \p OperandOrderFixed marks a language-fixed argument order that
+ /// \p Order alone does not capture.
void EmitCallArgs(CallArgList &Args, PrototypeWrapper Prototype,
llvm::iterator_range<CallExpr::const_arg_iterator> ArgRange,
AbstractCallee AC = AbstractCallee(),
unsigned ParamsToSkip = 0,
- EvaluationOrder Order = EvaluationOrder::Default);
+ EvaluationOrder Order = EvaluationOrder::Default,
+ bool OperandOrderFixed = false);
/// EmitPointerWithAlignment - Given an expression with a pointer type,
/// emit the value and compute our best estimate of the alignment of the
diff --git a/clang/test/CodeGen/musttail-indirect-arg.cpp b/clang/test/CodeGen/musttail-indirect-arg.cpp
index dd05eab868843..6ca7143694b63 100644
--- a/clang/test/CodeGen/musttail-indirect-arg.cpp
+++ b/clang/test/CodeGen/musttail-indirect-arg.cpp
@@ -2,6 +2,7 @@
// RUN: %clang_cc1 -triple=aarch64-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
// RUN: %clang_cc1 -triple=loongarch64-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
// RUN: %clang_cc1 -triple=s390x-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
+// RUN: %clang_cc1 -std=c++23 -triple=aarch64-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=CXX23
// C++ side of the musttail Indirect-arg fix. The call argument is typically
// a CXXConstructExpr invoking the trivial copy constructor; EmitCallArg
@@ -222,3 +223,105 @@ struct Big P21(struct Big a, int b) {
// COMMON: call {{.*}} @_Z4bumpv()
// COMMON: store {{.*}} [[VAL]], ptr %a
// COMMON: musttail call {{.*}} @_Z3C213Bigi({{.*}}, ptr {{[^,]*}} %a,
+
+// P22: an overloaded operator keeps the built-in operand order
+// ([over.match.oper]/2), so forwarding must not defer the left operand's read
+// past the right operand's side effect.
+struct Big operator<<(struct Big x, struct Big y);
+struct Big P22(struct Big a, struct Big b) {
+ [[clang::musttail]] return a << Big{a.a = 10};
+}
+// COMMON-LABEL: define {{.*}} @_Z3P223BigS_(
+// COMMON: [[SNAP:%[0-9a-z.]+]] = load <4 x i64>, ptr %a
+// COMMON: store i64 10, ptr %a
+// COMMON: store <4 x i64> [[SNAP]], ptr %a
+// COMMON: musttail call {{.*}} @_Zls3BigS_({{.*}}, ptr {{[^,]*}} %a, ptr {{[^,]*}} %b)
+
+// P24: [expr.assign]/1 sequences an assignment's right operand first, so the
+// deferred read of the right operand is the one that must not move.
+struct Big operator+=(struct Big x, struct Big y);
+struct Big mutate(struct Big *p);
+struct Big P24(struct Big a, struct Big b) {
+ [[clang::musttail]] return mutate(&b) += b;
+}
+// COMMON-LABEL: define {{.*}} @_Z3P243BigS_(
+// COMMON: [[SNAP:%[0-9a-z.]+]] = load <4 x i64>, ptr %b
+// COMMON: call {{.*}} @_Z6mutateP3Big(
+// COMMON: store <4 x i64> [[SNAP]], ptr %b
+// COMMON: musttail call {{.*}} @_ZpL3BigS_({{.*}}, ptr {{[^,]*}} %a, ptr {{[^,]*}} %b)
+
+// P25: the other operand mutates the source through an opaque callee, so
+// nothing in the expression names it. Whether the source is reachable from the
+// other operand cannot decide this.
+struct Big bumpg();
+struct Big P25(struct Big a, struct Big b) {
+ [[clang::musttail]] return a << bumpg();
+}
+// COMMON-LABEL: define {{.*}} @_Z3P253BigS_(
+// COMMON: [[SNAP:%[0-9a-z.]+]] = load <4 x i64>, ptr %a
+// COMMON: call {{.*}} @_Z5bumpgv(
+// COMMON: store <4 x i64> [[SNAP]], ptr %a
+// COMMON: musttail call {{.*}} @_Zls3BigS_({{.*}}, ptr {{[^,]*}} %a, ptr {{[^,]*}} %b)
+
+// P26: the right-operand-first rule of [expr.assign]/1 through a member
+// operator+=, so the read of %rhs must precede bumpacc().
+struct Acc {
+ unsigned long long v;
+ Acc &operator+=(struct Big rhs);
+ Acc &tailadd(struct Big rhs);
+};
+Acc &bumpacc(struct Big *p);
+Acc &Acc::tailadd(struct Big rhs) {
+ [[clang::musttail]] return bumpacc(&rhs) += rhs;
+}
+// COMMON-LABEL: define {{.*}} @_ZN3Acc7tailaddE3Big(
+// COMMON: [[SNAP:%[0-9a-z.]+]] = load <4 x i64>, ptr %rhs
+// COMMON: call {{.*}} @_Z7bumpaccP3Big(
+// COMMON: store <4 x i64> [[SNAP]], ptr %rhs
+// COMMON: musttail call {{.*}} @_ZN3AccpLE3Big({{.*}}, ptr {{[^,]*}} %rhs)
+
+// P27: the prescribed-order call is an argument inside the musttail return,
+// not the tail call itself.
+struct Big bumpbig();
+int use27(struct Big v);
+int C27(int x);
+int P27(int x) {
+ [[clang::musttail]] return C27(use27(gw.inner << bumpbig()));
+}
+// COMMON-LABEL: define {{.*}} @_Z3P27i(
+// COMMON: @llvm.mem{{(cpy|move)}}{{.*}}(ptr {{[^,]*}} [[LHS:%agg.tmp[0-9]*]], ptr {{[^,]*}} @gw, i64 32
+// COMMON: call {{.*}} @_Z7bumpbigv(
+// COMMON: call {{.*}} @_Zls3BigS_({{.*}}, ptr {{[^,]*}} [[LHS]],
+// COMMON: musttail call {{.*}} @_Z3C27i(
+
+// P28: prescribed order with the source in the second incoming slot, so both
+// argument slots are relocated and the read still precedes the mutation.
+struct Big operator>>(struct Big x, struct Big y);
+struct Big P28(struct Big a, struct Big b) {
+ [[clang::musttail]] return b >> Big{b.a = 10};
+}
+// COMMON-LABEL: define {{.*}} @_Z3P283BigS_(
+// COMMON: [[SNAP:%[0-9a-z.]+]] = load <4 x i64>, ptr %b
+// COMMON: store i64 10, ptr %b
+// COMMON: store <4 x i64> [[SNAP]], ptr %a
+// COMMON: musttail call {{.*}} @_Zrs3BigS_({{.*}}, ptr {{[^,]*}} %a, ptr {{[^,]*}} %b)
+
+#if __cplusplus >= 202302L
+// P23: the same rule for an operator with no explicit EvaluationOrder case.
+// A subscript operator's object parameter is sequenced before the index
+// ([expr.sub]/1).
+struct Sub {
+ unsigned long long a, b, c, d;
+ Sub operator[](this Sub self, int i);
+ Sub tail(this Sub self, int i);
+};
+int bump(Sub *p);
+Sub Sub::tail(this Sub self, int i) {
+ [[clang::musttail]] return self[bump(&self)];
+}
+// CXX23-LABEL: define {{.*}} @_ZNH3Sub4tailES_i(
+// CXX23: [[SNAP:%[0-9a-z.]+]] = load <4 x i64>, ptr %self
+// CXX23: call {{.*}} @_Z4bumpP3Sub(
+// CXX23: store <4 x i64> [[SNAP]], ptr %self
+// CXX23: musttail call {{.*}} @_ZNH3SubixES_i({{.*}}, ptr {{[^,]*}} %self, i32 {{.*}})
+#endif
>From 9a45e6787875203a39f7c45056da5101498c0920 Mon Sep 17 00:00:00 2001
From: Xavier Roche <xavier.roche at algolia.com>
Date: Thu, 6 Aug 2026 10:50:22 +0200
Subject: [PATCH 21/21] [Clang] Only relocate a musttail argument when the type
is byte-copyable
EmitCall copied every Indirect musttail argument into the matching incoming
parameter, including one that arrived as a copy-constructed temporary because
the forwarding gate had already rejected it. A class with no trivial copy or
move operation then hit the EmitAggregateCopy assert, and a class with a
non-trivial copy constructor was relocated bytewise, which breaks a copy
constructor that records the object's own address and an address-discriminated
__ptrauth member.
Restrict the relocation to trivially copyable types, plus trivial_abi, which
opts in despite a non-trivial copy. Everything else keeps the pre-existing
path.
Assisted-by: Claude (Anthropic)
Co-Authored-By: Claude Opus 5 <noreply at anthropic.com>
---
clang/lib/CodeGen/CGCall.cpp | 9 +-
clang/test/CodeGen/musttail-indirect-arg.cpp | 89 +++++++++++++++++++-
2 files changed, 92 insertions(+), 6 deletions(-)
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index 18a38672c2503..bc55e7d8e1a61 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5903,8 +5903,13 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
// permutations safely, the source value is captured into a scratch
// alloca here (Phase 1); the write to the incoming-param destination
// is deferred until after all sources have been read (Phase 2 below).
- // IndirectAliased uses the existing fallback (different source-AS).
- if (IsMustTail && ArgInfo.isIndirect()) {
+ // IndirectAliased (different source-AS) and non-relocatable types fall
+ // back to the pre-existing path, which still allows a dangling temporary.
+ // trivial_abi opts in despite having a non-trivial copy constructor.
+ const auto *ArgRD = I->Ty->getAsCXXRecordDecl();
+ bool ByteRelocatable = I->Ty.isTriviallyCopyableType(getContext()) ||
+ (ArgRD && ArgRD->hasAttr<TrivialABIAttr>());
+ if (IsMustTail && ArgInfo.isIndirect() && ByteRelocatable) {
llvm::Argument *IncomingArg = CurFn->arg_begin() + FirstIRArg;
llvm::Value *Dst = IncomingArg;
Address SrcAddr = Address::invalid();
diff --git a/clang/test/CodeGen/musttail-indirect-arg.cpp b/clang/test/CodeGen/musttail-indirect-arg.cpp
index 6ca7143694b63..05ed68324c231 100644
--- a/clang/test/CodeGen/musttail-indirect-arg.cpp
+++ b/clang/test/CodeGen/musttail-indirect-arg.cpp
@@ -3,6 +3,7 @@
// RUN: %clang_cc1 -triple=loongarch64-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
// RUN: %clang_cc1 -triple=s390x-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=COMMON
// RUN: %clang_cc1 -std=c++23 -triple=aarch64-linux-gnu %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=CXX23
+// RUN: %clang_cc1 -triple=arm64e-apple-ios -fptrauth-calls -fptrauth-intrinsics %s -emit-llvm -O1 -o - | FileCheck %s --check-prefix=PTRAUTH
// C++ side of the musttail Indirect-arg fix. The call argument is typically
// a CXXConstructExpr invoking the trivial copy constructor; EmitCallArg
@@ -44,19 +45,99 @@ struct Big P3(struct Big a, struct Big b) {
// COMMON: store {{.*}} [[SAVED]], ptr %b,
// COMMON: musttail call {{.*}} @_Z2C33BigS_({{.*}}, ptr {{[^,]*}} %a, ptr {{[^,]*}} %b)
-// P4: non-trivial copy constructor. The trivial-copy gate must NOT engage;
-// the user-defined copy ctor IS called. Dangling-stack bug in this corner
-// remains (out of scope).
+// P4: no trivial copy or move operation, so the argument is not relocated.
+// Keep operator= declared: without it the implicit copy assignment stays
+// trivial and EmitAggregateCopy would accept a bytewise copy.
struct NonTrivial {
unsigned long long parts[4];
NonTrivial(const NonTrivial &);
+ NonTrivial &operator=(const NonTrivial &);
};
NonTrivial C4(NonTrivial a);
NonTrivial P4(NonTrivial a) {
[[clang::musttail]] return C4(a);
}
// COMMON-LABEL: define {{.*}} @_Z2P410NonTrivial(
-// COMMON: call {{.*}} @_ZN10NonTrivialC1ERKS_
+// COMMON: call {{.*}} @_ZN10NonTrivialC1ERKS_(ptr {{[^,]*}} [[TMP:%agg.tmp[0-9]*]], ptr {{[^,]*}} %a
+// COMMON-NOT: musttail.copy
+// COMMON: musttail call {{.*}} @_Z2C410NonTrivial({{.*}}, ptr {{[^,]*}} [[TMP]])
+
+// P4b: only a copy ctor, so the implicit copy assignment is trivial. Still not
+// relocatable: a copy ctor can record the object's own address.
+struct CtorOnly {
+ unsigned long long parts[4];
+ CtorOnly(const CtorOnly &);
+};
+CtorOnly C4b(CtorOnly x, CtorOnly y);
+CtorOnly P4b(CtorOnly a, CtorOnly b) {
+ [[clang::musttail]] return C4b(b, a);
+}
+// COMMON-LABEL: define {{.*}} @_Z3P4b8CtorOnlyS_(
+// COMMON-NOT: musttail.copy
+// COMMON: musttail call {{.*}} @_Z3C4b8CtorOnlyS_(
+
+// P4c: trivial_abi marks the type ABI-trivial, so relocation is allowed
+// despite the non-trivial copy ctor.
+struct __attribute__((trivial_abi)) TrivialAbi {
+ unsigned long long parts[4];
+ TrivialAbi(const TrivialAbi &);
+};
+TrivialAbi C4c(TrivialAbi x, TrivialAbi y);
+TrivialAbi P4c(TrivialAbi a, TrivialAbi b) {
+ [[clang::musttail]] return C4c(b, a);
+}
+// COMMON-LABEL: define {{.*}} @_Z3P4c10TrivialAbiS_(
+// COMMON: [[SLOT0:%agg.tmp[0-9]*]] = alloca
+// COMMON: [[SLOT1:%musttail.copy[0-9a-z.]*]] = alloca
+// COMMON: @llvm.mem{{(cpy|move)}}{{.*}}(ptr {{[^,]*}} %a, ptr {{[^,]*}} [[SLOT0]], i64 32
+// COMMON: @llvm.mem{{(cpy|move)}}{{.*}}(ptr {{[^,]*}} %b, ptr {{[^,]*}} [[SLOT1]], i64 32
+// COMMON: musttail call {{.*}} @_Z3C4c10TrivialAbiS_({{.*}}, ptr {{[^,]*}} %a, ptr {{[^,]*}} %b)
+
+// P4d: a virtual function makes the type non-trivially copyable. Pins the
+// boundary so widening the relocation predicate has to update this.
+struct Poly {
+ unsigned long long parts[4];
+ virtual void f();
+};
+Poly C4d(Poly x, Poly y);
+Poly P4d(Poly a, Poly b) {
+ [[clang::musttail]] return C4d(b, a);
+}
+// COMMON-LABEL: define {{.*}} @_Z3P4d4PolyS_(
+// COMMON-NOT: musttail.copy
+// COMMON: musttail call {{.*}} @_Z3C4d4PolyS_(
+
+// P4e: a union is enough for EmitAggregateCopy, but the user copy operations
+// still rule out a relocation.
+union UnionCopy {
+ unsigned long long parts[4];
+ UnionCopy(const UnionCopy &);
+ UnionCopy &operator=(const UnionCopy &);
+};
+UnionCopy C4e(UnionCopy x, UnionCopy y);
+UnionCopy P4e(UnionCopy a, UnionCopy b) {
+ [[clang::musttail]] return C4e(b, a);
+}
+// COMMON-LABEL: define {{.*}} @_Z3P4e9UnionCopyS_(
+// COMMON-NOT: musttail.copy
+// COMMON: musttail call {{.*}} @_Z3C4e9UnionCopyS_(
+
+#ifdef __PTRAUTH__
+// P4f: an address-discriminated __ptrauth member is signed with the object's
+// own address, so relocating the bytes would leave the signature bound to the
+// old one.
+struct Signed {
+ int *__ptrauth(2, 1, 42) p;
+ unsigned long long a, b, c;
+};
+Signed C4f(Signed x, Signed y);
+Signed P4f(Signed a, Signed b) {
+ [[clang::musttail]] return C4f(b, a);
+}
+// PTRAUTH-LABEL: define {{.*}} @_Z3P4f6SignedS_(
+// PTRAUTH-NOT: musttail.copy
+// PTRAUTH: musttail call {{.*}} @_Z3C4f6SignedS_(
+#endif
// P5: modify-then-forward.
struct Big C5(struct Big a);
More information about the cfe-commits
mailing list