[llvm] [NVPTX] Add Custom ABI (abi_preserve) support via function attributes (PR #227499)
Akshay Deodhar via llvm-commits
llvm-commits at lists.llvm.org
Wed Sep 30 20:08:47 PDT 2026
https://github.com/akshayrdeodhar updated https://github.com/llvm/llvm-project/pull/227499
>From 9a688b35b69243a0b267852dd34141e84953278c Mon Sep 17 00:00:00 2001
From: Akshay Deodhar <adeodhar at nvidia.com>
Date: Tue, 29 Sep 2026 22:23:53 +0000
Subject: [PATCH 1/2] [NVPTX] Add Custom ABI (abi_preserve) support via
function attributes
Adds support for the abi_preserve and abi_preserve_control performance tuning directives, which specify how many general purpose and control registers the callers of a function are responsible for preserving.
These are modeled as function and callsite attributes:
"nvvm.preserve_n_data"="8"
"nvvm.preserve_n_control"="3"
On a function they describe that function's contract and are emitted on its definition and declaration. On a callsite they are appended to the generated .callprototype, which is how an indirect call -- having no callee to consult -- states the contract. The attributes are looked up on the callsite only, never inherited from the callee.
PTX ISA documentation: https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#performance-tuning-directives-abi-preserve, https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#performance-tuning-directives-abi-preserve-control
---
llvm/docs/NVPTXUsage.md | 28 ++++
llvm/include/llvm/Support/NVVMAttributes.h | 8 +
llvm/lib/IR/Verifier.cpp | 14 +-
llvm/lib/Target/NVPTX/NVPTXAsmPrinter.cpp | 29 ++++
llvm/lib/Target/NVPTX/NVPTXSubtarget.h | 3 +
llvm/lib/Target/NVPTX/NVVMProperties.cpp | 54 ++++++
llvm/lib/Target/NVPTX/NVVMProperties.h | 11 ++
.../CodeGen/NVPTX/custom-abi-preserve-reg.ll | 156 ++++++++++++++++++
.../SimplifyCFG/nvvm-preserve-reg-abi-sink.ll | 73 ++++++++
llvm/test/Verifier/nvvm-preserve-reg-abi.ll | 44 +++++
10 files changed, 419 insertions(+), 1 deletion(-)
create mode 100644 llvm/test/CodeGen/NVPTX/custom-abi-preserve-reg.ll
create mode 100644 llvm/test/Transforms/SimplifyCFG/nvvm-preserve-reg-abi-sink.ll
create mode 100644 llvm/test/Verifier/nvvm-preserve-reg-abi.ll
diff --git a/llvm/docs/NVPTXUsage.md b/llvm/docs/NVPTXUsage.md
index 088035ee83f1f..6d943da764603 100644
--- a/llvm/docs/NVPTXUsage.md
+++ b/llvm/docs/NVPTXUsage.md
@@ -104,6 +104,34 @@ When compiled, the PTX kernel functions are callable by host-side code.
of the number of thread blocks. This attribute is only allowed for kernel
functions and requires `nvvm.reqntid` and `nvvm.cluster_dim` attributes.
+`"nvvm.preserve_n_data"="<n>"`
+
+: This attribute specifies the number of general purpose registers that the
+ callers of this function are responsible for preserving. The backend bounds
+ the number of live data variables the callers may keep in callee-save
+ registers accordingly. It is lowered to the PTX
+ [`.abi_preserve`](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#performance-tuning-directives-abi-preserve)
+ directive, which requires PTX ISA 8.3 and `sm_80` or higher. The directives
+ are omitted on targets below that.
+
+`"nvvm.preserve_n_control"="<n>"`
+
+: This attribute specifies the number of control registers that the callers of
+ this function are responsible for preserving, corresponding to the number of
+ divergent program points in the call tree leading to this call. It is lowered
+ to the PTX
+ [`.abi_preserve_control`](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#performance-tuning-directives-abi-preserve-control)
+ directive, which requires PTX ISA 8.3 and `sm_80` or higher. The directives
+ are omitted on targets below that.
+
+Both `nvvm.preserve_n_*` attributes are valid on function definitions,
+declarations and call sites. PTX permits these directives only on device
+functions, between the `.func` directive and the function body. On a call site
+they are appended to the generated `.callprototype`; this is how an indirect
+call, which has no callee to consult, states the contract. A call site is never
+given the contract of its callee: the attributes are looked up on the call site
+only.
+
(address-spaces)=
### Address Spaces
diff --git a/llvm/include/llvm/Support/NVVMAttributes.h b/llvm/include/llvm/Support/NVVMAttributes.h
index cff3f8514bbfb..19e5912ac880a 100644
--- a/llvm/include/llvm/Support/NVVMAttributes.h
+++ b/llvm/include/llvm/Support/NVVMAttributes.h
@@ -27,6 +27,14 @@ constexpr StringLiteral MaxNReg("nvvm.maxnreg");
constexpr StringLiteral BlocksAreClusters("nvvm.blocksareclusters");
constexpr StringLiteral GridConstant("nvvm.grid_constant");
+// Custom ABI attributes: the number of registers a function preserves across a
+// call. Valid on function definitions and declarations, and on callsites for
+// indirect calls.
+// Ref:
+// https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#performance-tuning-directives-abi-preserve
+constexpr StringLiteral PreserveNData("nvvm.preserve_n_data");
+constexpr StringLiteral PreserveNControl("nvvm.preserve_n_control");
+
} // namespace NVVMAttr
} // namespace llvm
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 2502d574dd992..e08f5fe3ef79f 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -2711,6 +2711,8 @@ void Verifier::verifyFunctionAttrs(FunctionType *FT, AttributeList Attrs,
.empty(),
"\"patchable-function-entry-section\" must not be empty");
checkUnsignedBaseTenFuncAttr(Attrs, "warn-stack-size", V);
+ checkUnsignedBaseTenFuncAttr(Attrs, "nvvm.preserve_n_data", V);
+ checkUnsignedBaseTenFuncAttr(Attrs, "nvvm.preserve_n_control", V);
if (auto A = Attrs.getFnAttr("sign-return-address"); A.isValid()) {
StringRef S = A.getValueAsString();
@@ -3304,13 +3306,23 @@ void Verifier::visitFunction(const Function &F) {
case CallingConv::Fast:
case CallingConv::Cold:
case CallingConv::Intel_OCL_BI:
- case CallingConv::PTX_Kernel:
case CallingConv::PTX_Device:
Check(!F.isVarArg(),
"Calling convention does not support varargs or "
"perfect forwarding!",
&F);
break;
+ case CallingConv::PTX_Kernel:
+ // PTX permits the abi_preserve directives only between a .func directive
+ // and its body, so a kernel cannot express them.
+ for (StringRef Attr : {"nvvm.preserve_n_data", "nvvm.preserve_n_control"})
+ Check(!F.hasFnAttribute(Attr),
+ "'" + Attr + "' is not allowed on kernel functions", &F);
+ Check(!F.isVarArg(),
+ "Calling convention does not support varargs or "
+ "perfect forwarding!",
+ &F);
+ break;
case CallingConv::AMDGPU_Gfx_WholeWave:
Check(!F.arg_empty() && F.arg_begin()->getType()->isIntegerTy(1),
"Calling convention requires first argument to be i1", &F);
diff --git a/llvm/lib/Target/NVPTX/NVPTXAsmPrinter.cpp b/llvm/lib/Target/NVPTX/NVPTXAsmPrinter.cpp
index ab90036722ca4..10c4bc2cfa112 100644
--- a/llvm/lib/Target/NVPTX/NVPTXAsmPrinter.cpp
+++ b/llvm/lib/Target/NVPTX/NVPTXAsmPrinter.cpp
@@ -248,6 +248,7 @@ class NVPTXAsmPrinter : public AsmPrinter {
void emitGlobalAlias(const Module &M, const GlobalAlias &GA) override;
void emitHeader(Module &M, const NVPTXSubtarget &STI);
void emitKernelFunctionDirectives(const Function &F, raw_ostream &O) const;
+ void emitFunctionDirectives(const Function &F, raw_ostream &O) const;
void emitFunctionParamList(const Function *, raw_ostream &O);
void setAndEmitFunctionVirtualRegisters(const MachineFunction &MF);
void encodeDebugInfoRegisterNumbers(const MachineFunction &MF);
@@ -756,6 +757,13 @@ void NVPTXAsmPrinter::emitCallPrototype(const CallBase &CB,
O << ")";
if (shouldEmitPTXNoReturn(CB))
O << " .noreturn";
+ // Print the abi_preserve directives carried by the callsite. A prototype sits
+ // in the caller's body, so only the target gate applies here.
+ if (STI.hasABIPreserve())
+ if (const std::string ABI =
+ getABIPreserveDirectives(CB, /*Multiline=*/false);
+ !ABI.empty())
+ O << " " << ABI;
O << ";\n";
OutStreamer->emitRawText(O.str());
@@ -844,6 +852,8 @@ void NVPTXAsmPrinter::emitFunctionEntryLabel() {
if (isKernelFunction(*F))
emitKernelFunctionDirectives(*F, O);
+ emitFunctionDirectives(*F, O);
+
if (shouldEmitPTXNoReturn(*F))
O << ".noreturn";
@@ -888,6 +898,17 @@ void NVPTXAsmPrinter::emitFunctionBodyEnd() {
VRegMapping.clear();
}
+void NVPTXAsmPrinter::emitFunctionDirectives(const Function &F,
+ raw_ostream &O) const {
+ // A parse error on a .entry, and unassemblable on targets that cannot
+ // express them, so the attributes are ignored in both cases.
+ if (!isKernelFunction(F) &&
+ TM.getSubtarget<NVPTXSubtarget>(F).hasABIPreserve())
+ if (const std::string ABI = getABIPreserveDirectives(F, /*Multiline=*/true);
+ !ABI.empty())
+ O << ABI << "\n";
+}
+
const MCSymbol *NVPTXAsmPrinter::getFunctionFrameSymbol() const {
return OutContext.getOrCreateSymbol(DEPOTNAME + Twine(getFunctionNumber()));
}
@@ -1013,6 +1034,14 @@ void NVPTXAsmPrinter::emitDeclarationWithName(const Function *F, MCSymbol *S,
O << "\n";
if (shouldEmitPTXNoReturn(*F))
O << ".noreturn";
+ // Print abi_preserve directives regardless of whether F is a definition or a
+ // declaration, subject to the same restrictions as above.
+ if (!isKernelFunction(*F) &&
+ TM.getSubtarget<NVPTXSubtarget>(*F).hasABIPreserve())
+ if (const std::string ABI =
+ getABIPreserveDirectives(*F, /*Multiline=*/false);
+ !ABI.empty())
+ O << ABI;
O << ";\n";
}
diff --git a/llvm/lib/Target/NVPTX/NVPTXSubtarget.h b/llvm/lib/Target/NVPTX/NVPTXSubtarget.h
index ee8adec2da060..31a40a9fb8a45 100644
--- a/llvm/lib/Target/NVPTX/NVPTXSubtarget.h
+++ b/llvm/lib/Target/NVPTX/NVPTXSubtarget.h
@@ -85,6 +85,9 @@ class NVPTXSubtarget : public NVPTXGenSubtargetInfo {
bool hasUsedBytesMaskPragma() const {
return hasFeature(NVPTX::SM50) && hasFeature(NVPTX::PTX83);
}
+ bool hasABIPreserve() const {
+ return hasFeature(NVPTX::SM80) && hasFeature(NVPTX::PTX83);
+ }
bool hasAtomAddF64() const { return hasFeature(NVPTX::SM60); }
bool hasAtomScope() const { return hasFeature(NVPTX::SM60); }
bool hasAtomBitwise64() const { return hasFeature(NVPTX::SM32); }
diff --git a/llvm/lib/Target/NVPTX/NVVMProperties.cpp b/llvm/lib/Target/NVPTX/NVVMProperties.cpp
index ee268096c96d2..cb8725feafdc9 100644
--- a/llvm/lib/Target/NVPTX/NVVMProperties.cpp
+++ b/llvm/lib/Target/NVPTX/NVVMProperties.cpp
@@ -25,6 +25,7 @@
#include "llvm/Support/ModRef.h"
#include "llvm/Support/Mutex.h"
#include "llvm/Support/NVVMAttributes.h"
+#include <array>
#include <functional>
#include <map>
#include <mutex>
@@ -295,6 +296,59 @@ bool llvm::hasBlocksAreClusters(const Function &F) {
return F.hasFnAttribute(NVVMAttr::BlocksAreClusters);
}
+// PTX register-preservation ("custom ABI") directives. Each entry maps an NVVM
+// function attribute to the prototype directive it selects. This table is the
+// single point of extension for further preservation directives.
+static constexpr auto PreserveRegABIDirectives = std::array{
+ std::pair{NVVMAttr::PreserveNData, StringLiteral(".abi_preserve")},
+ std::pair{NVVMAttr::PreserveNControl,
+ StringLiteral(".abi_preserve_control")},
+};
+
+// Look the attribute up on the callsite only.
+static std::optional<unsigned> getFnAttrParsedInt(const CallBase &CB,
+ StringRef Attr) {
+ const Attribute A = CB.getAttributes().getFnAttr(Attr);
+ if (!A.isValid())
+ return std::nullopt;
+
+ unsigned Value;
+ if (A.getValueAsString().getAsInteger(10, Value)) {
+ CB.getContext().emitError("can't parse integer attribute " +
+ A.getValueAsString() + " in " + Attr);
+ return std::nullopt;
+ }
+ return Value;
+}
+
+static std::string formatABIPreserveDirectives(
+ bool Multiline, function_ref<std::optional<unsigned>(StringRef)> GetValue) {
+ std::string PrototypeAttributes;
+ for (const auto &[Attr, Directive] : PreserveRegABIDirectives) {
+ const std::optional<unsigned> Count = GetValue(Attr);
+ if (!Count)
+ continue;
+
+ if (!PrototypeAttributes.empty())
+ PrototypeAttributes += Multiline ? '\n' : ' ';
+ PrototypeAttributes += Directive;
+ PrototypeAttributes += " ";
+ PrototypeAttributes += std::to_string(*Count);
+ }
+ return PrototypeAttributes;
+}
+
+std::string llvm::getABIPreserveDirectives(const Function &F, bool Multiline) {
+ return formatABIPreserveDirectives(
+ Multiline, [&F](StringRef Attr) { return getFnAttrParsedInt(F, Attr); });
+}
+
+std::string llvm::getABIPreserveDirectives(const CallBase &CB, bool Multiline) {
+ return formatABIPreserveDirectives(Multiline, [&CB](StringRef Attr) {
+ return getFnAttrParsedInt(CB, Attr);
+ });
+}
+
bool llvm::isParamGridConstant(const Argument &Arg) {
assert(isKernelFunction(*Arg.getParent()) &&
"only kernel arguments can be grid_constant");
diff --git a/llvm/lib/Target/NVPTX/NVVMProperties.h b/llvm/lib/Target/NVPTX/NVVMProperties.h
index 7187b18d3cbf7..bc081e4e27bb4 100644
--- a/llvm/lib/Target/NVPTX/NVVMProperties.h
+++ b/llvm/lib/Target/NVPTX/NVVMProperties.h
@@ -20,6 +20,7 @@
#include "llvm/Support/Alignment.h"
#include <cstdint>
#include <optional>
+#include <string>
namespace llvm {
@@ -59,6 +60,16 @@ bool hasBlocksAreClusters(const Function &);
bool isParamGridConstant(const Argument &);
+/// Extract the PTX abi_preserve directive string implied by
+/// the NVVM preserve_n_* attributes, e.g. ".abi_preserve 8". Returns an empty
+/// string when none are present. Entries are separated by '\n' when
+/// \p Multiline is set, and by ' ' otherwise.
+///
+/// On a function, the attributes are looked up on the function definition or
+/// declaration. On a callsite, the attributes are looked up on the call.
+std::string getABIPreserveDirectives(const Function &, bool Multiline);
+std::string getABIPreserveDirectives(const CallBase &, bool Multiline);
+
inline MaybeAlign getStackAlign(const Function &F, unsigned Index) {
return F.getAttributes().getAttributes(Index).getStackAlignment();
}
diff --git a/llvm/test/CodeGen/NVPTX/custom-abi-preserve-reg.ll b/llvm/test/CodeGen/NVPTX/custom-abi-preserve-reg.ll
new file mode 100644
index 0000000000000..f8dad60740d8c
--- /dev/null
+++ b/llvm/test/CodeGen/NVPTX/custom-abi-preserve-reg.ll
@@ -0,0 +1,156 @@
+; RUN: llc < %s -mtriple=nvptx64-nvidia-cuda -mcpu=sm_80 -mattr=+ptx83 | FileCheck %s
+; RUN: %if ptxas %{ llc < %s -mtriple=nvptx64-nvidia-cuda -mcpu=sm_80 -mattr=+ptx83 | %ptxas-verify -arch=sm_80 %}
+
+; .abi_preserve and .abi_preserve_control require PTX ISA 8.3 and sm_80, and
+; ptxas rejects them on a .entry. Both restrictions are checked below.
+
+target datalayout = "e-p:64:64:64-p3:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-i128:128:128-f32:32:32-f64:64:64-f128:128:128-v16:16:16-v32:32:32-v64:64:64-v128:128:128-n16:32:64"
+target triple = "nvptx64-nvidia-cuda"
+
+ at llvm.used = appending global [1 x ptr] [ptr @kernel], section "llvm.metadata"
+
+declare i32 @only_data(i32, i32) "nvvm.preserve_n_data"="4"
+; CHECK-LABEL: only_data
+; CHECK: (
+; CHECK: .param
+; CHECK: )
+; CHECK: .abi_preserve 4;
+
+declare void @only_control(i32, i32) "nvvm.preserve_n_control"="2"
+; CHECK-LABEL: only_control
+; CHECK: (
+; CHECK: .param
+; CHECK: )
+; CHECK: .abi_preserve_control 2;
+
+; def_data_and_control is defined below and has its address taken, so it is
+; emitted both as a declaration here and as a definition later. One attribute
+; set must yield the single-line form here and the multiline form there.
+; CHECK-LABEL: def_data_and_control
+; CHECK: (
+; CHECK: .param
+; CHECK: )
+; CHECK: .abi_preserve 8 .abi_preserve_control 2;
+
+; Taking a function's address forces a declaration to be emitted for it, and
+; that declaration carries the directives.
+; CHECK: .visible .func (.param .b32 func_retval0) fn
+; CHECK: (
+; CHECK: .param .b32 fn_param_0
+; CHECK: )
+; CHECK: .abi_preserve 6;
+; CHECK: .visible .global .align 8 .u64 dc_addr = def_data_and_control;
+; CHECK: .visible .global .align 8 .u64 fn_addr = fn;
+ at dc_addr = addrspace(1) global ptr @def_data_and_control, align 8
+ at fn_addr = addrspace(1) global ptr @fn, align 8
+
+; An internal function emits plain .func rather than .visible .func, and still
+; carries the directives.
+define internal fastcc i32 @internal_linkage(i32 %a, i32 %b) unnamed_addr #0 {
+; CHECK-LABEL: internal_linkage(
+; CHECK: .abi_preserve 8
+; CHECK-NEXT: .abi_preserve_control 4
+; CHECK-NEXT: {
+ %add = add nsw i32 %b, %a
+ ret i32 %add
+}
+
+define i32 @def_data_and_control(i32 %a, i32 %b) "nvvm.preserve_n_data"="8" "nvvm.preserve_n_control"="2" {
+; CHECK-LABEL: def_data_and_control(
+; CHECK: .abi_preserve 8
+; CHECK-NEXT: .abi_preserve_control 2
+; CHECK-NEXT: {
+ %r = add i32 %a, %b
+ ret i32 %r
+}
+
+; The calls below are what force the declarations above to be emitted -- do not
+; remove them.
+define ptx_kernel void @kernel(ptr nocapture %a, ptr nocapture readonly %b) #1 {
+; CHECK-LABEL: kernel(
+; CHECK: .maxntid 128
+ %1 = addrspacecast ptr %b to ptr addrspace(1)
+ %2 = addrspacecast ptr %a to ptr addrspace(1)
+ %3 = tail call i32 @llvm.nvvm.read.ptx.sreg.tid.x()
+ %4 = sext i32 %3 to i64
+ %getElem = getelementptr inbounds i32, ptr addrspace(1) %2, i64 %4
+ %tmp3 = load i32, ptr addrspace(1) %getElem, align 4
+ %getElem1 = getelementptr inbounds i32, ptr addrspace(1) %1, i64 %4
+ %tmp7 = load i32, ptr addrspace(1) %getElem1, align 4
+ %call = tail call fastcc i32 @internal_linkage(i32 %tmp3, i32 %tmp7)
+ %call3 = tail call i32 @only_data(i32 %tmp3, i32 %tmp7)
+ %add = add nsw i32 %call3, %call
+ call void @only_control(i32 %tmp3, i32 %tmp7)
+ %call4 = tail call i32 @def_data_and_control(i32 %tmp3, i32 %tmp7)
+ store i32 %add, ptr addrspace(1) %getElem, align 4
+ ret void
+}
+
+; An indirect call has no callee to consult, so the contract lives on the call
+; site and is appended to the generated .callprototype, single-line. The
+; prototype label is captured and matched against the call's final operand.
+define internal fastcc i32 @indirect_call(i32 %a, ptr %b) unnamed_addr {
+; CHECK-LABEL: indirect_call(
+; CHECK: $L__[[P1:prototype_[0-9]+]]:
+; CHECK-NEXT: .callprototype (.param .b32 _) _ (.param .b32 _) .abi_preserve 8 .abi_preserve_control 2;
+; The callee pointer may be loaded either side of the call sequence brace
+; depending on scheduling, so the setup is matched order-independently.
+; CHECK-DAG: ld.param{{(::(func|entry))?}}.{{u|b}}32 [[R1:%.*]], [indirect_call_param_0];
+; CHECK-DAG: ld.param{{(::(func|entry))?}}.{{u|b}}64 [[RD1:%.*]], [indirect_call_param_1];
+; CHECK-DAG: .param {{.*}} param0;
+; CHECK-DAG: .param {{.*}} retval0;
+; CHECK-DAG: st.param{{(::(func|entry))?}}.{{.*}} [param0[[_:(\+0)?]]], [[R1]];
+; CHECK: call (retval0),
+; CHECK: [[RD1]],
+; CHECK: (
+; CHECK: param0
+; CHECK: )
+; CHECK: , $L__[[P1]];
+; CHECK: ld.param{{(::(func|entry))?}}.{{.*}} [[R2:%.*]], [retval0[[_:(\+0)?]]];
+; CHECK: }
+; CHECK: st.param{{(::(func|entry))?}}.{{.*}} [func_retval0[[_:(\+0)?]]], [[R2]];
+; CHECK: ret;
+ %retval = call i32 %b(i32 %a) "nvvm.preserve_n_data"="8" "nvvm.preserve_n_control"="2"
+ ret i32 %retval
+}
+
+; Negative control: an indirect call with no attributes gets a bare prototype,
+; with nothing appended before the semicolon.
+define internal fastcc i32 @indirect_call_no_abi(i32 %a, ptr %b) unnamed_addr {
+; CHECK-LABEL: indirect_call_no_abi(
+; CHECK: $L__{{prototype_[0-9]+}}:
+; CHECK-NEXT: .callprototype (.param .b32 _) _ (.param .b32 _);
+; CHECK-NOT: abi_preserve
+ %retval = call i32 %b(i32 %a)
+ ret i32 %retval
+}
+
+; The callee reached through this pointer is @fn, which carries
+; nvvm.preserve_n_data. The contract is not inherited from the callee: only the
+; attributes on the callsite itself are emitted, so the prototype stays bare.
+define internal fastcc i32 @indirect_call_to_known_callee(i32 %a) unnamed_addr {
+; CHECK-LABEL: indirect_call_to_known_callee(
+; CHECK: $L__{{prototype_[0-9]+}}:
+; CHECK-NEXT: .callprototype (.param .b32 _) _ (.param .b32 _);
+; CHECK-NOT: abi_preserve
+ %fp = load ptr, ptr addrspace(1) @fn_addr, align 8
+ %retval = call i32 %fp(i32 %a)
+ ret i32 %retval
+}
+
+; The definition half of the address-taken pair above. A single directive must
+; not pick up a stray separator, so nothing may sit between it and the body.
+define i32 @fn(i32 %pp) "nvvm.preserve_n_data"="6" {
+; CHECK-LABEL: fn(
+; CHECK: .abi_preserve 6
+; CHECK-NEXT: {
+ %add = add nsw i32 %pp, 1
+ ret i32 %add
+}
+
+; Function Attrs: nounwind readnone speculatable
+declare i32 @llvm.nvvm.read.ptx.sreg.tid.x() #2
+
+attributes #0 = { noinline norecurse nounwind readnone "nvvm.preserve_n_data"="8" "nvvm.preserve_n_control"="4" }
+attributes #1 = { alwaysinline nounwind "nvvm.maxntid"="128" }
+attributes #2 = { nounwind readnone speculatable }
diff --git a/llvm/test/Transforms/SimplifyCFG/nvvm-preserve-reg-abi-sink.ll b/llvm/test/Transforms/SimplifyCFG/nvvm-preserve-reg-abi-sink.ll
new file mode 100644
index 0000000000000..d8369cea9241d
--- /dev/null
+++ b/llvm/test/Transforms/SimplifyCFG/nvvm-preserve-reg-abi-sink.ll
@@ -0,0 +1,73 @@
+; RUN: opt -S -passes='simplifycfg<sink-common-insts>' < %s | FileCheck %s
+
+; The custom-ABI attributes are string attributes, so AttributeSet::intersectWith
+; requires them to match exactly before two calls can be sunk into a common
+; successor. Sinking calls with different register-preservation contracts would
+; silently drop one of them.
+
+; Differing values: both calls must survive in their own blocks.
+; CHECK-LABEL: @differing_values(
+; CHECK: if:
+; CHECK-NEXT: call void %fp() #[[#]]
+; CHECK: else:
+; CHECK-NEXT: call void %fp() #[[#]]
+define void @differing_values(i1 %c, ptr %fp) {
+entry:
+ br i1 %c, label %if, label %else
+
+if:
+ call void %fp() "nvvm.preserve_n_data"="8"
+ br label %end
+
+else:
+ call void %fp() "nvvm.preserve_n_data"="2"
+ br label %end
+
+end:
+ ret void
+}
+
+; Present on one side only: still must not be sunk.
+; CHECK-LABEL: @one_sided(
+; CHECK: if:
+; CHECK-NEXT: call void %fp() #[[#]]
+; CHECK: else:
+; CHECK-NEXT: call void %fp(){{$}}
+define void @one_sided(i1 %c, ptr %fp) {
+entry:
+ br i1 %c, label %if, label %else
+
+if:
+ call void %fp() "nvvm.preserve_n_data"="8"
+ br label %end
+
+else:
+ call void %fp()
+ br label %end
+
+end:
+ ret void
+}
+
+; Identical values: sinking is legal, and still happens. This is the control
+; that shows the checks above are testing the attribute and not merely that
+; sinking is disabled.
+; CHECK-LABEL: @identical_values(
+; CHECK: entry:
+; CHECK: call void %fp() #[[#]]
+; CHECK-NOT: call void %fp()
+define void @identical_values(i1 %c, ptr %fp) {
+entry:
+ br i1 %c, label %if, label %else
+
+if:
+ call void %fp() "nvvm.preserve_n_data"="8"
+ br label %end
+
+else:
+ call void %fp() "nvvm.preserve_n_data"="8"
+ br label %end
+
+end:
+ ret void
+}
diff --git a/llvm/test/Verifier/nvvm-preserve-reg-abi.ll b/llvm/test/Verifier/nvvm-preserve-reg-abi.ll
new file mode 100644
index 0000000000000..bbd472000ccec
--- /dev/null
+++ b/llvm/test/Verifier/nvvm-preserve-reg-abi.ll
@@ -0,0 +1,44 @@
+; RUN: not llvm-as %s -o /dev/null 2>&1 | FileCheck %s
+
+; The custom-ABI preserve counts must parse as unsigned base-ten integers.
+
+; CHECK: "nvvm.preserve_n_data" takes an unsigned integer: foobar
+define void @not_an_integer() "nvvm.preserve_n_data"="foobar" {
+ ret void
+}
+
+; CHECK: "nvvm.preserve_n_control" takes an unsigned integer: -1
+define void @negative() "nvvm.preserve_n_control"="-1" {
+ ret void
+}
+
+; CHECK: "nvvm.preserve_n_data" takes an unsigned integer: 8,2
+define void @not_a_vector() "nvvm.preserve_n_data"="8,2" {
+ ret void
+}
+
+; The same check applies at call sites, which is where an indirect call carries
+; its contract. verifyFunctionAttrs() is reached from visitCallBase() as well as
+; from the function walk.
+; CHECK: "nvvm.preserve_n_control" takes an unsigned integer: bad
+define void @bad_call_site(ptr %fp) {
+ call void %fp() "nvvm.preserve_n_control"="bad"
+ ret void
+}
+
+; PTX only permits the directives between a .func directive and its body, so a
+; kernel cannot express them at all.
+; CHECK: 'nvvm.preserve_n_data' is not allowed on kernel functions
+define ptx_kernel void @kernel_data() "nvvm.preserve_n_data"="8" {
+ ret void
+}
+
+; CHECK: 'nvvm.preserve_n_control' is not allowed on kernel functions
+define ptx_kernel void @kernel_control() "nvvm.preserve_n_control"="2" {
+ ret void
+}
+
+; A device function may carry them; only kernels are rejected.
+define void @device_fn_is_fine() "nvvm.preserve_n_data"="8" "nvvm.preserve_n_control"="2" {
+ ret void
+}
>From db62633f1d5fd59924a77d0bcce8a91969d77f2f Mon Sep 17 00:00:00 2001
From: Akshay Deodhar <adeodhar at nvidia.com>
Date: Thu, 1 Oct 2026 03:08:21 +0000
Subject: [PATCH 2/2] address review comments, cleanup
---
llvm/docs/NVPTXUsage.md | 6 +-
llvm/include/llvm/Support/NVVMAttributes.h | 8 ++-
llvm/lib/IR/Verifier.cpp | 7 ++-
llvm/lib/Target/NVPTX/NVPTXAsmPrinter.cpp | 63 ++++++++++++-------
llvm/lib/Target/NVPTX/NVVMProperties.cpp | 47 ++++----------
llvm/lib/Target/NVPTX/NVVMProperties.h | 19 +++---
.../CodeGen/NVPTX/custom-abi-preserve-reg.ll | 59 +++++++++--------
.../SimplifyCFG/nvvm-preserve-reg-abi-sink.ll | 20 ++----
llvm/test/Verifier/nvvm-preserve-reg-abi.ll | 34 ++++------
9 files changed, 127 insertions(+), 136 deletions(-)
diff --git a/llvm/docs/NVPTXUsage.md b/llvm/docs/NVPTXUsage.md
index 6d943da764603..095325b04c5d9 100644
--- a/llvm/docs/NVPTXUsage.md
+++ b/llvm/docs/NVPTXUsage.md
@@ -104,7 +104,7 @@ When compiled, the PTX kernel functions are callable by host-side code.
of the number of thread blocks. This attribute is only allowed for kernel
functions and requires `nvvm.reqntid` and `nvvm.cluster_dim` attributes.
-`"nvvm.preserve_n_data"="<n>"`
+`"nvvm.abi_preserve"="<n>"`
: This attribute specifies the number of general purpose registers that the
callers of this function are responsible for preserving. The backend bounds
@@ -114,7 +114,7 @@ When compiled, the PTX kernel functions are callable by host-side code.
directive, which requires PTX ISA 8.3 and `sm_80` or higher. The directives
are omitted on targets below that.
-`"nvvm.preserve_n_control"="<n>"`
+`"nvvm.abi_preserve_control"="<n>"`
: This attribute specifies the number of control registers that the callers of
this function are responsible for preserving, corresponding to the number of
@@ -124,7 +124,7 @@ When compiled, the PTX kernel functions are callable by host-side code.
directive, which requires PTX ISA 8.3 and `sm_80` or higher. The directives
are omitted on targets below that.
-Both `nvvm.preserve_n_*` attributes are valid on function definitions,
+Both `nvvm.abi_preserve*` attributes are valid on function definitions,
declarations and call sites. PTX permits these directives only on device
functions, between the `.func` directive and the function body. On a call site
they are appended to the generated `.callprototype`; this is how an indirect
diff --git a/llvm/include/llvm/Support/NVVMAttributes.h b/llvm/include/llvm/Support/NVVMAttributes.h
index 19e5912ac880a..28c03c1119962 100644
--- a/llvm/include/llvm/Support/NVVMAttributes.h
+++ b/llvm/include/llvm/Support/NVVMAttributes.h
@@ -32,8 +32,12 @@ constexpr StringLiteral GridConstant("nvvm.grid_constant");
// indirect calls.
// Ref:
// https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#performance-tuning-directives-abi-preserve
-constexpr StringLiteral PreserveNData("nvvm.preserve_n_data");
-constexpr StringLiteral PreserveNControl("nvvm.preserve_n_control");
+constexpr StringLiteral AbiPreserve("nvvm.abi_preserve");
+constexpr StringLiteral AbiPreserveControl("nvvm.abi_preserve_control");
+
+// All custom ABI attributes, in PTX emission order.
+inline constexpr StringLiteral AbiPreserveAttrs[] = {AbiPreserve,
+ AbiPreserveControl};
} // namespace NVVMAttr
} // namespace llvm
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index e08f5fe3ef79f..2cbf6ff73f435 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -125,6 +125,7 @@
#include "llvm/Support/FormatVariadic.h"
#include "llvm/Support/MathExtras.h"
#include "llvm/Support/ModRef.h"
+#include "llvm/Support/NVVMAttributes.h"
#include "llvm/Support/TimeProfiler.h"
#include "llvm/Support/raw_ostream.h"
#include "llvm/TargetParser/RISCVTargetParser.h"
@@ -2711,8 +2712,8 @@ void Verifier::verifyFunctionAttrs(FunctionType *FT, AttributeList Attrs,
.empty(),
"\"patchable-function-entry-section\" must not be empty");
checkUnsignedBaseTenFuncAttr(Attrs, "warn-stack-size", V);
- checkUnsignedBaseTenFuncAttr(Attrs, "nvvm.preserve_n_data", V);
- checkUnsignedBaseTenFuncAttr(Attrs, "nvvm.preserve_n_control", V);
+ for (StringLiteral Attr : NVVMAttr::AbiPreserveAttrs)
+ checkUnsignedBaseTenFuncAttr(Attrs, Attr, V);
if (auto A = Attrs.getFnAttr("sign-return-address"); A.isValid()) {
StringRef S = A.getValueAsString();
@@ -3315,7 +3316,7 @@ void Verifier::visitFunction(const Function &F) {
case CallingConv::PTX_Kernel:
// PTX permits the abi_preserve directives only between a .func directive
// and its body, so a kernel cannot express them.
- for (StringRef Attr : {"nvvm.preserve_n_data", "nvvm.preserve_n_control"})
+ for (StringLiteral Attr : NVVMAttr::AbiPreserveAttrs)
Check(!F.hasFnAttribute(Attr),
"'" + Attr + "' is not allowed on kernel functions", &F);
Check(!F.isVarArg(),
diff --git a/llvm/lib/Target/NVPTX/NVPTXAsmPrinter.cpp b/llvm/lib/Target/NVPTX/NVPTXAsmPrinter.cpp
index 10c4bc2cfa112..7d4c36a7f1d0c 100644
--- a/llvm/lib/Target/NVPTX/NVPTXAsmPrinter.cpp
+++ b/llvm/lib/Target/NVPTX/NVPTXAsmPrinter.cpp
@@ -274,6 +274,14 @@ class NVPTXAsmPrinter : public AsmPrinter {
return true;
}
+ /// Should the abi_preserve directives of \p F be emitted? They are a parse
+ /// error on a .entry, and unassemblable on targets that cannot express them,
+ /// so the attributes are ignored in both cases.
+ bool shouldEmitABIPreserve(const Function &F) const {
+ return !isKernelFunction(F) &&
+ TM.getSubtarget<NVPTXSubtarget>(F).hasABIPreserve();
+ }
+
bool PrintAsmOperand(const MachineInstr *MI, unsigned OpNo,
const char *ExtraCode, raw_ostream &) override;
void printOperand(const MachineInstr *MI, unsigned OpNum, raw_ostream &O);
@@ -716,6 +724,25 @@ static void printReturnValClause(const OwnerT *Owner, StringRef Name,
O << ") ";
}
+// Print the abi_preserve directives of \p ABI as Prefix, the directives joined
+// by Sep, then Suffix. Prints nothing when \p ABI has no directives.
+static void printABIPreserve(const ABIPreserve &ABI, raw_ostream &O,
+ StringRef Prefix, StringRef Sep,
+ StringRef Suffix) {
+ if (ABI.empty())
+ return;
+
+ O << Prefix;
+ ListSeparator LS(Sep);
+ for (const auto &[Attr, RegCount] : ABI) {
+ // The directive is the attribute name without its "nvvm" namespace, e.g.
+ // "nvvm.abi_preserve" selects ".abi_preserve".
+ assert(Attr.contains('.') && "expected a namespaced NVVM attribute");
+ O << LS << Attr.substr(Attr.find('.')) << ' ' << RegCount;
+ }
+ O << Suffix;
+}
+
void NVPTXAsmPrinter::emitCallPrototype(const CallBase &CB,
MCSymbol *PrototypeSymbol) const {
const DataLayout &DL = getDataLayout();
@@ -760,10 +787,8 @@ void NVPTXAsmPrinter::emitCallPrototype(const CallBase &CB,
// Print the abi_preserve directives carried by the callsite. A prototype sits
// in the caller's body, so only the target gate applies here.
if (STI.hasABIPreserve())
- if (const std::string ABI =
- getABIPreserveDirectives(CB, /*Multiline=*/false);
- !ABI.empty())
- O << " " << ABI;
+ printABIPreserve(getABIPreserve(CB), O, /*Prefix=*/" ", /*Sep=*/" ",
+ /*Suffix=*/"");
O << ";\n";
OutStreamer->emitRawText(O.str());
@@ -854,9 +879,6 @@ void NVPTXAsmPrinter::emitFunctionEntryLabel() {
emitFunctionDirectives(*F, O);
- if (shouldEmitPTXNoReturn(*F))
- O << ".noreturn";
-
OutStreamer->emitRawText(O.str());
VRegMapping.clear();
@@ -900,13 +922,12 @@ void NVPTXAsmPrinter::emitFunctionBodyEnd() {
void NVPTXAsmPrinter::emitFunctionDirectives(const Function &F,
raw_ostream &O) const {
- // A parse error on a .entry, and unassemblable on targets that cannot
- // express them, so the attributes are ignored in both cases.
- if (!isKernelFunction(F) &&
- TM.getSubtarget<NVPTXSubtarget>(F).hasABIPreserve())
- if (const std::string ABI = getABIPreserveDirectives(F, /*Multiline=*/true);
- !ABI.empty())
- O << ABI << "\n";
+ if (shouldEmitPTXNoReturn(F))
+ O << ".noreturn\n";
+
+ if (shouldEmitABIPreserve(F))
+ printABIPreserve(getABIPreserve(F), O, /*Prefix=*/"", /*Sep=*/"\n",
+ /*Suffix=*/"\n");
}
const MCSymbol *NVPTXAsmPrinter::getFunctionFrameSymbol() const {
@@ -1032,16 +1053,14 @@ void NVPTXAsmPrinter::emitDeclarationWithName(const Function *F, MCSymbol *S,
O << "\n";
emitFunctionParamList(F, O);
O << "\n";
- if (shouldEmitPTXNoReturn(*F))
+ const bool NoReturn = shouldEmitPTXNoReturn(*F);
+ if (NoReturn)
O << ".noreturn";
// Print abi_preserve directives regardless of whether F is a definition or a
- // declaration, subject to the same restrictions as above.
- if (!isKernelFunction(*F) &&
- TM.getSubtarget<NVPTXSubtarget>(*F).hasABIPreserve())
- if (const std::string ABI =
- getABIPreserveDirectives(*F, /*Multiline=*/false);
- !ABI.empty())
- O << ABI;
+ // declaration. They must not run into a preceding .noreturn.
+ if (shouldEmitABIPreserve(*F))
+ printABIPreserve(getABIPreserve(*F), O, /*Prefix=*/NoReturn ? " " : "",
+ /*Sep=*/" ", /*Suffix=*/"");
O << ";\n";
}
diff --git a/llvm/lib/Target/NVPTX/NVVMProperties.cpp b/llvm/lib/Target/NVPTX/NVVMProperties.cpp
index cb8725feafdc9..c68033d041231 100644
--- a/llvm/lib/Target/NVPTX/NVVMProperties.cpp
+++ b/llvm/lib/Target/NVPTX/NVVMProperties.cpp
@@ -25,8 +25,6 @@
#include "llvm/Support/ModRef.h"
#include "llvm/Support/Mutex.h"
#include "llvm/Support/NVVMAttributes.h"
-#include <array>
-#include <functional>
#include <map>
#include <mutex>
#include <numeric>
@@ -296,15 +294,6 @@ bool llvm::hasBlocksAreClusters(const Function &F) {
return F.hasFnAttribute(NVVMAttr::BlocksAreClusters);
}
-// PTX register-preservation ("custom ABI") directives. Each entry maps an NVVM
-// function attribute to the prototype directive it selects. This table is the
-// single point of extension for further preservation directives.
-static constexpr auto PreserveRegABIDirectives = std::array{
- std::pair{NVVMAttr::PreserveNData, StringLiteral(".abi_preserve")},
- std::pair{NVVMAttr::PreserveNControl,
- StringLiteral(".abi_preserve_control")},
-};
-
// Look the attribute up on the callsite only.
static std::optional<unsigned> getFnAttrParsedInt(const CallBase &CB,
StringRef Attr) {
@@ -321,32 +310,20 @@ static std::optional<unsigned> getFnAttrParsedInt(const CallBase &CB,
return Value;
}
-static std::string formatABIPreserveDirectives(
- bool Multiline, function_ref<std::optional<unsigned>(StringRef)> GetValue) {
- std::string PrototypeAttributes;
- for (const auto &[Attr, Directive] : PreserveRegABIDirectives) {
- const std::optional<unsigned> Count = GetValue(Attr);
- if (!Count)
- continue;
-
- if (!PrototypeAttributes.empty())
- PrototypeAttributes += Multiline ? '\n' : ' ';
- PrototypeAttributes += Directive;
- PrototypeAttributes += " ";
- PrototypeAttributes += std::to_string(*Count);
- }
- return PrototypeAttributes;
-}
-
-std::string llvm::getABIPreserveDirectives(const Function &F, bool Multiline) {
- return formatABIPreserveDirectives(
- Multiline, [&F](StringRef Attr) { return getFnAttrParsedInt(F, Attr); });
+ABIPreserve llvm::getABIPreserve(const Function &F) {
+ ABIPreserve ABI;
+ for (StringLiteral Attr : NVVMAttr::AbiPreserveAttrs)
+ if (const std::optional<unsigned> RegCount = getFnAttrParsedInt(F, Attr))
+ ABI[Attr] = *RegCount;
+ return ABI;
}
-std::string llvm::getABIPreserveDirectives(const CallBase &CB, bool Multiline) {
- return formatABIPreserveDirectives(Multiline, [&CB](StringRef Attr) {
- return getFnAttrParsedInt(CB, Attr);
- });
+ABIPreserve llvm::getABIPreserve(const CallBase &CB) {
+ ABIPreserve ABI;
+ for (StringLiteral Attr : NVVMAttr::AbiPreserveAttrs)
+ if (const std::optional<unsigned> RegCount = getFnAttrParsedInt(CB, Attr))
+ ABI[Attr] = *RegCount;
+ return ABI;
}
bool llvm::isParamGridConstant(const Argument &Arg) {
diff --git a/llvm/lib/Target/NVPTX/NVVMProperties.h b/llvm/lib/Target/NVPTX/NVVMProperties.h
index bc081e4e27bb4..235fd37a2e3b0 100644
--- a/llvm/lib/Target/NVPTX/NVVMProperties.h
+++ b/llvm/lib/Target/NVPTX/NVVMProperties.h
@@ -14,13 +14,14 @@
#ifndef LLVM_LIB_TARGET_NVPTX_NVVMPROPERTIES_H
#define LLVM_LIB_TARGET_NVPTX_NVVMPROPERTIES_H
+#include "llvm/ADT/MapVector.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
#include "llvm/IR/CallingConv.h"
#include "llvm/IR/Function.h"
#include "llvm/Support/Alignment.h"
#include <cstdint>
#include <optional>
-#include <string>
namespace llvm {
@@ -60,15 +61,15 @@ bool hasBlocksAreClusters(const Function &);
bool isParamGridConstant(const Argument &);
-/// Extract the PTX abi_preserve directive string implied by
-/// the NVVM preserve_n_* attributes, e.g. ".abi_preserve 8". Returns an empty
-/// string when none are present. Entries are separated by '\n' when
-/// \p Multiline is set, and by ' ' otherwise.
-///
+/// Maps the name of each nvvm.abi_preserve* attribute that is present to its
+/// register count, in PTX emission order. An absent attribute has no entry.
+using ABIPreserve = SmallMapVector<StringRef, unsigned, 2>;
+
/// On a function, the attributes are looked up on the function definition or
-/// declaration. On a callsite, the attributes are looked up on the call.
-std::string getABIPreserveDirectives(const Function &, bool Multiline);
-std::string getABIPreserveDirectives(const CallBase &, bool Multiline);
+/// declaration. On a callsite, the attributes are looked up on the call only;
+/// they are not inherited from the callee.
+ABIPreserve getABIPreserve(const Function &);
+ABIPreserve getABIPreserve(const CallBase &);
inline MaybeAlign getStackAlign(const Function &F, unsigned Index) {
return F.getAttributes().getAttributes(Index).getStackAlignment();
diff --git a/llvm/test/CodeGen/NVPTX/custom-abi-preserve-reg.ll b/llvm/test/CodeGen/NVPTX/custom-abi-preserve-reg.ll
index f8dad60740d8c..68611e8791349 100644
--- a/llvm/test/CodeGen/NVPTX/custom-abi-preserve-reg.ll
+++ b/llvm/test/CodeGen/NVPTX/custom-abi-preserve-reg.ll
@@ -1,39 +1,39 @@
; RUN: llc < %s -mtriple=nvptx64-nvidia-cuda -mcpu=sm_80 -mattr=+ptx83 | FileCheck %s
; RUN: %if ptxas %{ llc < %s -mtriple=nvptx64-nvidia-cuda -mcpu=sm_80 -mattr=+ptx83 | %ptxas-verify -arch=sm_80 %}
-; .abi_preserve and .abi_preserve_control require PTX ISA 8.3 and sm_80, and
-; ptxas rejects them on a .entry. Both restrictions are checked below.
-
target datalayout = "e-p:64:64:64-p3:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-i128:128:128-f32:32:32-f64:64:64-f128:128:128-v16:16:16-v32:32:32-v64:64:64-v128:128:128-n16:32:64"
target triple = "nvptx64-nvidia-cuda"
@llvm.used = appending global [1 x ptr] [ptr @kernel], section "llvm.metadata"
-declare i32 @only_data(i32, i32) "nvvm.preserve_n_data"="4"
+declare i32 @only_data(i32, i32) "nvvm.abi_preserve"="4"
; CHECK-LABEL: only_data
; CHECK: (
; CHECK: .param
; CHECK: )
; CHECK: .abi_preserve 4;
-declare void @only_control(i32, i32) "nvvm.preserve_n_control"="2"
+declare void @only_control(i32, i32) "nvvm.abi_preserve_control"="2"
; CHECK-LABEL: only_control
; CHECK: (
; CHECK: .param
; CHECK: )
; CHECK: .abi_preserve_control 2;
-; def_data_and_control is defined below and has its address taken, so it is
-; emitted both as a declaration here and as a definition later. One attribute
-; set must yield the single-line form here and the multiline form there.
+; .noreturn comes first and must stay separated from the directives.
+declare void @noreturn_and_abi(i32) noreturn "nvvm.abi_preserve"="4" "nvvm.abi_preserve_control"="2"
+; CHECK-LABEL: noreturn_and_abi
+; CHECK: (
+; CHECK: .param
+; CHECK: )
+; CHECK: .noreturn .abi_preserve 4 .abi_preserve_control 2;
+
; CHECK-LABEL: def_data_and_control
; CHECK: (
; CHECK: .param
; CHECK: )
; CHECK: .abi_preserve 8 .abi_preserve_control 2;
-; Taking a function's address forces a declaration to be emitted for it, and
-; that declaration carries the directives.
; CHECK: .visible .func (.param .b32 func_retval0) fn
; CHECK: (
; CHECK: .param .b32 fn_param_0
@@ -44,8 +44,6 @@ declare void @only_control(i32, i32) "nvvm.preserve_n_control"="2"
@dc_addr = addrspace(1) global ptr @def_data_and_control, align 8
@fn_addr = addrspace(1) global ptr @fn, align 8
-; An internal function emits plain .func rather than .visible .func, and still
-; carries the directives.
define internal fastcc i32 @internal_linkage(i32 %a, i32 %b) unnamed_addr #0 {
; CHECK-LABEL: internal_linkage(
; CHECK: .abi_preserve 8
@@ -55,7 +53,7 @@ define internal fastcc i32 @internal_linkage(i32 %a, i32 %b) unnamed_addr #0 {
ret i32 %add
}
-define i32 @def_data_and_control(i32 %a, i32 %b) "nvvm.preserve_n_data"="8" "nvvm.preserve_n_control"="2" {
+define i32 @def_data_and_control(i32 %a, i32 %b) "nvvm.abi_preserve"="8" "nvvm.abi_preserve_control"="2" {
; CHECK-LABEL: def_data_and_control(
; CHECK: .abi_preserve 8
; CHECK-NEXT: .abi_preserve_control 2
@@ -64,6 +62,11 @@ define i32 @def_data_and_control(i32 %a, i32 %b) "nvvm.preserve_n_data"="8" "nvv
ret i32 %r
}
+define void @calls_noreturn(i32 %x) {
+ call void @noreturn_and_abi(i32 %x)
+ unreachable
+}
+
; The calls below are what force the declarations above to be emitted -- do not
; remove them.
define ptx_kernel void @kernel(ptr nocapture %a, ptr nocapture readonly %b) #1 {
@@ -86,15 +89,10 @@ define ptx_kernel void @kernel(ptr nocapture %a, ptr nocapture readonly %b) #1 {
ret void
}
-; An indirect call has no callee to consult, so the contract lives on the call
-; site and is appended to the generated .callprototype, single-line. The
-; prototype label is captured and matched against the call's final operand.
define internal fastcc i32 @indirect_call(i32 %a, ptr %b) unnamed_addr {
; CHECK-LABEL: indirect_call(
; CHECK: $L__[[P1:prototype_[0-9]+]]:
; CHECK-NEXT: .callprototype (.param .b32 _) _ (.param .b32 _) .abi_preserve 8 .abi_preserve_control 2;
-; The callee pointer may be loaded either side of the call sequence brace
-; depending on scheduling, so the setup is matched order-independently.
; CHECK-DAG: ld.param{{(::(func|entry))?}}.{{u|b}}32 [[R1:%.*]], [indirect_call_param_0];
; CHECK-DAG: ld.param{{(::(func|entry))?}}.{{u|b}}64 [[RD1:%.*]], [indirect_call_param_1];
; CHECK-DAG: .param {{.*}} param0;
@@ -110,12 +108,10 @@ define internal fastcc i32 @indirect_call(i32 %a, ptr %b) unnamed_addr {
; CHECK: }
; CHECK: st.param{{(::(func|entry))?}}.{{.*}} [func_retval0[[_:(\+0)?]]], [[R2]];
; CHECK: ret;
- %retval = call i32 %b(i32 %a) "nvvm.preserve_n_data"="8" "nvvm.preserve_n_control"="2"
+ %retval = call i32 %b(i32 %a) "nvvm.abi_preserve"="8" "nvvm.abi_preserve_control"="2"
ret i32 %retval
}
-; Negative control: an indirect call with no attributes gets a bare prototype,
-; with nothing appended before the semicolon.
define internal fastcc i32 @indirect_call_no_abi(i32 %a, ptr %b) unnamed_addr {
; CHECK-LABEL: indirect_call_no_abi(
; CHECK: $L__{{prototype_[0-9]+}}:
@@ -125,9 +121,6 @@ define internal fastcc i32 @indirect_call_no_abi(i32 %a, ptr %b) unnamed_addr {
ret i32 %retval
}
-; The callee reached through this pointer is @fn, which carries
-; nvvm.preserve_n_data. The contract is not inherited from the callee: only the
-; attributes on the callsite itself are emitted, so the prototype stays bare.
define internal fastcc i32 @indirect_call_to_known_callee(i32 %a) unnamed_addr {
; CHECK-LABEL: indirect_call_to_known_callee(
; CHECK: $L__{{prototype_[0-9]+}}:
@@ -138,9 +131,23 @@ define internal fastcc i32 @indirect_call_to_known_callee(i32 %a) unnamed_addr {
ret i32 %retval
}
+; A direct call whose function type differs from the callee's is lowered as an
+; indirect call and gets a prototype, yet its called operand is still the
+; Function @fn. Looking the attribute up through CallBase::getFnAttr would fall
+; back to @fn's "nvvm.abi_preserve"; the contract must come from the callsite
+; alone, so the prototype stays bare.
+define internal fastcc i32 @type_mismatched_call(i32 %a) unnamed_addr {
+; CHECK-LABEL: type_mismatched_call(
+; CHECK: $L__{{prototype_[0-9]+}}:
+; CHECK-NEXT: .callprototype (.param .b32 _) _ (.param .b32 _, .param .b32 _);
+; CHECK-NOT: abi_preserve
+ %retval = call i32 @fn(i32 %a, i32 %a)
+ ret i32 %retval
+}
+
; The definition half of the address-taken pair above. A single directive must
; not pick up a stray separator, so nothing may sit between it and the body.
-define i32 @fn(i32 %pp) "nvvm.preserve_n_data"="6" {
+define i32 @fn(i32 %pp) "nvvm.abi_preserve"="6" {
; CHECK-LABEL: fn(
; CHECK: .abi_preserve 6
; CHECK-NEXT: {
@@ -151,6 +158,6 @@ define i32 @fn(i32 %pp) "nvvm.preserve_n_data"="6" {
; Function Attrs: nounwind readnone speculatable
declare i32 @llvm.nvvm.read.ptx.sreg.tid.x() #2
-attributes #0 = { noinline norecurse nounwind readnone "nvvm.preserve_n_data"="8" "nvvm.preserve_n_control"="4" }
+attributes #0 = { noinline norecurse nounwind readnone "nvvm.abi_preserve"="8" "nvvm.abi_preserve_control"="4" }
attributes #1 = { alwaysinline nounwind "nvvm.maxntid"="128" }
attributes #2 = { nounwind readnone speculatable }
diff --git a/llvm/test/Transforms/SimplifyCFG/nvvm-preserve-reg-abi-sink.ll b/llvm/test/Transforms/SimplifyCFG/nvvm-preserve-reg-abi-sink.ll
index d8369cea9241d..1c33ca9debde7 100644
--- a/llvm/test/Transforms/SimplifyCFG/nvvm-preserve-reg-abi-sink.ll
+++ b/llvm/test/Transforms/SimplifyCFG/nvvm-preserve-reg-abi-sink.ll
@@ -1,11 +1,5 @@
; RUN: opt -S -passes='simplifycfg<sink-common-insts>' < %s | FileCheck %s
-; The custom-ABI attributes are string attributes, so AttributeSet::intersectWith
-; requires them to match exactly before two calls can be sunk into a common
-; successor. Sinking calls with different register-preservation contracts would
-; silently drop one of them.
-
-; Differing values: both calls must survive in their own blocks.
; CHECK-LABEL: @differing_values(
; CHECK: if:
; CHECK-NEXT: call void %fp() #[[#]]
@@ -16,18 +10,17 @@ entry:
br i1 %c, label %if, label %else
if:
- call void %fp() "nvvm.preserve_n_data"="8"
+ call void %fp() "nvvm.abi_preserve"="8"
br label %end
else:
- call void %fp() "nvvm.preserve_n_data"="2"
+ call void %fp() "nvvm.abi_preserve"="2"
br label %end
end:
ret void
}
-; Present on one side only: still must not be sunk.
; CHECK-LABEL: @one_sided(
; CHECK: if:
; CHECK-NEXT: call void %fp() #[[#]]
@@ -38,7 +31,7 @@ entry:
br i1 %c, label %if, label %else
if:
- call void %fp() "nvvm.preserve_n_data"="8"
+ call void %fp() "nvvm.abi_preserve"="8"
br label %end
else:
@@ -49,9 +42,6 @@ end:
ret void
}
-; Identical values: sinking is legal, and still happens. This is the control
-; that shows the checks above are testing the attribute and not merely that
-; sinking is disabled.
; CHECK-LABEL: @identical_values(
; CHECK: entry:
; CHECK: call void %fp() #[[#]]
@@ -61,11 +51,11 @@ entry:
br i1 %c, label %if, label %else
if:
- call void %fp() "nvvm.preserve_n_data"="8"
+ call void %fp() "nvvm.abi_preserve"="8"
br label %end
else:
- call void %fp() "nvvm.preserve_n_data"="8"
+ call void %fp() "nvvm.abi_preserve"="8"
br label %end
end:
diff --git a/llvm/test/Verifier/nvvm-preserve-reg-abi.ll b/llvm/test/Verifier/nvvm-preserve-reg-abi.ll
index bbd472000ccec..bd28e333a33e7 100644
--- a/llvm/test/Verifier/nvvm-preserve-reg-abi.ll
+++ b/llvm/test/Verifier/nvvm-preserve-reg-abi.ll
@@ -1,44 +1,36 @@
; RUN: not llvm-as %s -o /dev/null 2>&1 | FileCheck %s
-; The custom-ABI preserve counts must parse as unsigned base-ten integers.
-
-; CHECK: "nvvm.preserve_n_data" takes an unsigned integer: foobar
-define void @not_an_integer() "nvvm.preserve_n_data"="foobar" {
+; CHECK: "nvvm.abi_preserve" takes an unsigned integer: foobar
+define void @not_an_integer() "nvvm.abi_preserve"="foobar" {
ret void
}
-; CHECK: "nvvm.preserve_n_control" takes an unsigned integer: -1
-define void @negative() "nvvm.preserve_n_control"="-1" {
+; CHECK: "nvvm.abi_preserve_control" takes an unsigned integer: -1
+define void @negative() "nvvm.abi_preserve_control"="-1" {
ret void
}
-; CHECK: "nvvm.preserve_n_data" takes an unsigned integer: 8,2
-define void @not_a_vector() "nvvm.preserve_n_data"="8,2" {
+; CHECK: "nvvm.abi_preserve" takes an unsigned integer: 8,2
+define void @not_a_vector() "nvvm.abi_preserve"="8,2" {
ret void
}
-; The same check applies at call sites, which is where an indirect call carries
-; its contract. verifyFunctionAttrs() is reached from visitCallBase() as well as
-; from the function walk.
-; CHECK: "nvvm.preserve_n_control" takes an unsigned integer: bad
+; CHECK: "nvvm.abi_preserve_control" takes an unsigned integer: bad
define void @bad_call_site(ptr %fp) {
- call void %fp() "nvvm.preserve_n_control"="bad"
+ call void %fp() "nvvm.abi_preserve_control"="bad"
ret void
}
-; PTX only permits the directives between a .func directive and its body, so a
-; kernel cannot express them at all.
-; CHECK: 'nvvm.preserve_n_data' is not allowed on kernel functions
-define ptx_kernel void @kernel_data() "nvvm.preserve_n_data"="8" {
+; CHECK: 'nvvm.abi_preserve' is not allowed on kernel functions
+define ptx_kernel void @kernel_data() "nvvm.abi_preserve"="8" {
ret void
}
-; CHECK: 'nvvm.preserve_n_control' is not allowed on kernel functions
-define ptx_kernel void @kernel_control() "nvvm.preserve_n_control"="2" {
+; CHECK: 'nvvm.abi_preserve_control' is not allowed on kernel functions
+define ptx_kernel void @kernel_control() "nvvm.abi_preserve_control"="2" {
ret void
}
-; A device function may carry them; only kernels are rejected.
-define void @device_fn_is_fine() "nvvm.preserve_n_data"="8" "nvvm.preserve_n_control"="2" {
+define void @device_fn_is_fine() "nvvm.abi_preserve"="8" "nvvm.abi_preserve_control"="2" {
ret void
}
More information about the llvm-commits
mailing list