[clang] [llvm] [KCFI] Add support for out-of-line traps (PR #242488)
Chandler Carruth via cfe-commits
cfe-commits at lists.llvm.org
Fri Nov 13 23:12:31 PST 2026
https://github.com/chandlerc updated https://github.com/llvm/llvm-project/pull/242488
>From 1a87a7e8e9f439bab80bbe1f776bee32ac1b74c8 Mon Sep 17 00:00:00 2001
From: Chandler Carruth <chandlerc at gmail.com>
Date: Fri, 13 Nov 2026 18:57:44 +0000
Subject: [PATCH 1/2] [KCFI] Add support for out-of-line traps
The forced inline trap is _very_ useful for patching, tracing, etc. But
it is very hostile to modern CPU branch predictors.
Add a flag (`-fsanitize-kcfi-out-of-line-traps`) for moving traps out of
line for code (userspace, or perf sensitive) that would rather the
opposite tradeoff of the fastest code path possible when not hitting a
check failure.
This is also the pattern that full CFI and UBSan already use.
We continue to emit the trap late to preserve all the optimizations on
indirect calls (ICP, etc), and keep a distinct trap for each check with
the debug location of the call. This shouldn't worsen backtraces when
debugging, etc. The MI level KCFI pass does the block splitting, etc.,
and each tagret has to opt-in to supporting it. This PR adds support for
x86-64 and AArch64.
Assisted-by: Antigravity with Gemini
---
clang/docs/ControlFlowIntegrity.md | 17 ++
clang/docs/ReleaseNotes.md | 6 +
clang/docs/UsersManual.md | 8 +
clang/include/clang/Basic/CodeGenOptions.def | 1 +
clang/include/clang/Driver/SanitizerArgs.h | 1 +
clang/include/clang/Options/Options.td | 6 +
clang/lib/CodeGen/CodeGenModule.cpp | 3 +
clang/lib/Driver/SanitizerArgs.cpp | 12 +
clang/test/CodeGen/kcfi-out-of-line-traps.c | 5 +
.../Driver/fsanitize-kcfi-out-of-line-traps.c | 15 +
llvm/include/llvm/CodeGen/TargetLowering.h | 13 +
llvm/lib/CodeGen/KCFI.cpp | 46 +++
llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp | 32 +-
.../Target/AArch64/AArch64ISelLowering.cpp | 33 ++
llvm/lib/Target/AArch64/AArch64ISelLowering.h | 6 +
llvm/lib/Target/AArch64/AArch64InstrInfo.cpp | 5 +
llvm/lib/Target/AArch64/AArch64InstrInfo.td | 15 +
llvm/lib/Target/X86/X86AsmPrinter.h | 1 +
llvm/lib/Target/X86/X86ISelLowering.cpp | 18 ++
llvm/lib/Target/X86/X86ISelLowering.h | 6 +
llvm/lib/Target/X86/X86InstrCompiler.td | 7 +
llvm/lib/Target/X86/X86MCInstLower.cpp | 21 +-
.../AArch64/kcfi-out-of-line-traps-xzr.mir | 47 +++
.../CodeGen/AArch64/kcfi-out-of-line-traps.ll | 286 ++++++++++++++++++
.../CodeGen/X86/kcfi-out-of-line-traps.ll | 217 +++++++++++++
25 files changed, 824 insertions(+), 3 deletions(-)
create mode 100644 clang/test/CodeGen/kcfi-out-of-line-traps.c
create mode 100644 clang/test/Driver/fsanitize-kcfi-out-of-line-traps.c
create mode 100644 llvm/test/CodeGen/AArch64/kcfi-out-of-line-traps-xzr.mir
create mode 100644 llvm/test/CodeGen/AArch64/kcfi-out-of-line-traps.ll
create mode 100644 llvm/test/CodeGen/X86/kcfi-out-of-line-traps.ll
diff --git a/clang/docs/ControlFlowIntegrity.md b/clang/docs/ControlFlowIntegrity.md
index c25a27fd910c51..e06afc00ed78cf 100644
--- a/clang/docs/ControlFlowIntegrity.md
+++ b/clang/docs/ControlFlowIntegrity.md
@@ -329,6 +329,23 @@ number of arguments passed in registers) into the binary. Some kernel CFI
techniques, such as FineIBT, may be able to use this information to provide
enhanced security.
+### `-fsanitize-kcfi-out-of-line-traps`
+
+By default (`-fno-sanitize-kcfi-out-of-line-traps`), KCFI emits the trap
+instruction inline immediately before each checked indirect call, and a check
+that passes takes a conditional branch over the trap. Consumers such as the
+Linux kernel rely on this inline layout to patch checks at boot and decode trap
+addresses.
+
+For user-space or other software that does not require inline traps,
+`-fsanitize-kcfi-out-of-line-traps` (supported on x86-64 and AArch64) moves
+each check's trap into a separate cold basic block so that a passing check
+falls through directly to the indirect call and only branches on failure. Each
+check still receives a distinct trap instruction with the call's debug location
+(and `.kcfi_traps` entry on x86-64, or register-encoding `brk` immediate on
+AArch64). Calls that are already bundled or in a basic block with an exception
+handling pad successor keep an inline trap.
+
## Member Function Pointer Call Checking
This scheme checks that indirect calls via a member function pointer
diff --git a/clang/docs/ReleaseNotes.md b/clang/docs/ReleaseNotes.md
index 14382a06db34c5..5e414ae49a40d8 100644
--- a/clang/docs/ReleaseNotes.md
+++ b/clang/docs/ReleaseNotes.md
@@ -377,6 +377,12 @@ documentation](CIR/index.md) for more information regarding ClangIR's status and
### New Compiler Flags
+- Added `-fsanitize-kcfi-out-of-line-traps` (and
+ `-fno-sanitize-kcfi-out-of-line-traps`, the default) on x86-64 and AArch64 to
+ emit `-fsanitize=kcfi` failure traps in separate cold blocks instead of
+ inline before each indirect call, avoiding a taken branch on every passing
+ check when inline trap layout is not required.
+
- Added `-fpch-base-dir-is-cwd` to serialize precompiled header input paths
relative to the current working directory. Paths resolve against the
consumer's working directory when loading the PCH, allowing a build tree
diff --git a/clang/docs/UsersManual.md b/clang/docs/UsersManual.md
index a784bacd810e91..07e0b5fe4af4f3 100644
--- a/clang/docs/UsersManual.md
+++ b/clang/docs/UsersManual.md
@@ -2463,6 +2463,14 @@ additional function arity information (for supported targets). See
{doc}`ControlFlowIntegrity` for more details.
:::
+:::{option} -f[no-]sanitize-kcfi-out-of-line-traps
+
+Move the trap instructions of `-fsanitize=kcfi` indirect call checks out of
+line into cold blocks instead of emitting them inline before each call (on
+x86-64 and AArch64). Disabled by default. See {doc}`ControlFlowIntegrity` for
+more details.
+:::
+
:::{option} -fsanitize-trap-loop
When a UBSan or CFI check fails in trapping mode, spin in an infinite
diff --git a/clang/include/clang/Basic/CodeGenOptions.def b/clang/include/clang/Basic/CodeGenOptions.def
index 8c7974ae1664a1..97282fd603cef8 100644
--- a/clang/include/clang/Basic/CodeGenOptions.def
+++ b/clang/include/clang/Basic/CodeGenOptions.def
@@ -302,6 +302,7 @@ CODEGENOPT(SanitizeCfiICallNormalizeIntegers, 1, 0, Benign) ///< Normalize integ
CODEGENOPT(SanitizeCfiCanonicalJumpTables, 1, 0, Benign) ///< Make jump table symbols canonical
///< instead of creating a local jump table.
CODEGENOPT(SanitizeKcfiArity, 1, 0, Benign) ///< Embed arity in KCFI patchable function prefix
+CODEGENOPT(SanitizeKcfiOutOfLineTraps, 1, 0, Benign) ///< Move the traps of KCFI checks out of line
CODEGENOPT(CopyProf, 1, 0, Benign) ///< Enable CopyProf instrumentation
VALUE_CODEGENOPT(CopyProfStaticSizeThreshold, 32, 16, Benign) ///< Set threshold for CopyProf instrumentation
CODEGENOPT(SanitizeCoverageType, 2, 0, Benign) ///< Type of sanitizer coverage
diff --git a/clang/include/clang/Driver/SanitizerArgs.h b/clang/include/clang/Driver/SanitizerArgs.h
index 145df2d94c61f3..1ebbf45fdfd74c 100644
--- a/clang/include/clang/Driver/SanitizerArgs.h
+++ b/clang/include/clang/Driver/SanitizerArgs.h
@@ -51,6 +51,7 @@ class SanitizerArgs {
bool CfiCanonicalJumpTables = false;
bool KcfiArity = false;
std::optional<std::string> KcfiHash;
+ bool KcfiOutOfLineTraps = false;
int AsanFieldPadding = 0;
bool SharedRuntime = false;
bool StableABI = false;
diff --git a/clang/include/clang/Options/Options.td b/clang/include/clang/Options/Options.td
index 4fc924f848507e..5327fa2e9b4661 100644
--- a/clang/include/clang/Options/Options.td
+++ b/clang/include/clang/Options/Options.td
@@ -3031,6 +3031,12 @@ def fsanitize_kcfi_arity : Flag<["-"], "fsanitize-kcfi-arity">,
Group<f_clang_Group>,
HelpText<"Embed function arity information into the KCFI patchable function prefix">,
MarshallingInfoFlag<CodeGenOpts<"SanitizeKcfiArity">>;
+defm sanitize_kcfi_out_of_line_traps : BoolOption<"f", "sanitize-kcfi-out-of-line-traps",
+ CodeGenOpts<"SanitizeKcfiOutOfLineTraps">, DefaultFalse,
+ PosFlag<SetTrue, [], [ClangOption, CC1Option], "Move">,
+ NegFlag<SetFalse, [], [ClangOption], "Do not move">,
+ BothFlags<[], [ClangOption], " the traps of KCFI indirect call checks out of line">>,
+ Group<f_clang_Group>;
def fsanitize_kcfi_hash_EQ
: Joined<["-"], "fsanitize-kcfi-hash=">,
HelpText<"Select hash algorithm for KCFI type IDs (xxHash64, FNV-1a)">,
diff --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index 3b8ec02a94e542..5b55d0dcd01476 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -1633,6 +1633,9 @@ void CodeGenModule::Release() {
CodeGenOpts.PatchableFunctionEntryOffset);
if (CodeGenOpts.SanitizeKcfiArity)
getModule().addModuleFlag(llvm::Module::Override, "kcfi-arity", 1);
+ if (CodeGenOpts.SanitizeKcfiOutOfLineTraps)
+ getModule().addModuleFlag(llvm::Module::Override,
+ "kcfi-out-of-line-traps", 1);
// Store the hash algorithm choice for use in LLVM passes
getModule().addModuleFlag(
llvm::Module::Override, "kcfi-hash",
diff --git a/clang/lib/Driver/SanitizerArgs.cpp b/clang/lib/Driver/SanitizerArgs.cpp
index 7b46553d035b5b..40f9c9b7714e10 100644
--- a/clang/lib/Driver/SanitizerArgs.cpp
+++ b/clang/lib/Driver/SanitizerArgs.cpp
@@ -1012,6 +1012,10 @@ SanitizerArgs::SanitizerArgs(const ToolChain &TC,
KcfiArity = Args.hasArg(options::OPT_fsanitize_kcfi_arity);
+ KcfiOutOfLineTraps =
+ Args.hasFlag(options::OPT_fsanitize_kcfi_out_of_line_traps,
+ options::OPT_fno_sanitize_kcfi_out_of_line_traps, false);
+
if (const Arg *A = Args.getLastArg(options::OPT_fsanitize_kcfi_hash_EQ))
KcfiHash = A->getValue();
@@ -1627,6 +1631,14 @@ void SanitizerArgs::addArgs(const ToolChain &TC, const llvm::opt::ArgList &Args,
CmdArgs.push_back("-fsanitize-kcfi-arity");
}
+ if (KcfiOutOfLineTraps) {
+ if (TC.getTriple().getArch() != llvm::Triple::x86_64 &&
+ !TC.getTriple().isAArch64(64))
+ TC.getDriver().Diag(clang::diag::err_drv_unsupported_opt_for_target)
+ << "-fsanitize-kcfi-out-of-line-traps" << TC.getTriple().str();
+ CmdArgs.push_back("-fsanitize-kcfi-out-of-line-traps");
+ }
+
if (KcfiHash)
CmdArgs.push_back(Args.MakeArgString("-fsanitize-kcfi-hash=" + *KcfiHash));
diff --git a/clang/test/CodeGen/kcfi-out-of-line-traps.c b/clang/test/CodeGen/kcfi-out-of-line-traps.c
new file mode 100644
index 00000000000000..ae178f44bd5178
--- /dev/null
+++ b/clang/test/CodeGen/kcfi-out-of-line-traps.c
@@ -0,0 +1,5 @@
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm -fsanitize=kcfi -fsanitize-kcfi-out-of-line-traps -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm -fsanitize=kcfi -o - %s | FileCheck %s --check-prefix=INLINE
+
+// CHECK: ![[#]] = !{i32 4, !"kcfi-out-of-line-traps", i32 1}
+// INLINE-NOT: !"kcfi-out-of-line-traps"
diff --git a/clang/test/Driver/fsanitize-kcfi-out-of-line-traps.c b/clang/test/Driver/fsanitize-kcfi-out-of-line-traps.c
new file mode 100644
index 00000000000000..bf3dc437a30e94
--- /dev/null
+++ b/clang/test/Driver/fsanitize-kcfi-out-of-line-traps.c
@@ -0,0 +1,15 @@
+// RUN: %clang --target=x86_64-linux-gnu -fsanitize=kcfi -fsanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s
+// RUN: %clang --target=aarch64-linux-gnu -fsanitize=kcfi -fsanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s
+// RUN: %clang --target=x86_64-linux-gnu -fsanitize=kcfi -fno-sanitize-kcfi-out-of-line-traps -fsanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s
+// CHECK: "-fsanitize-kcfi-out-of-line-traps"
+
+// RUN: %clang --target=x86_64-linux-gnu -fsanitize=kcfi %s -### 2>&1 | FileCheck %s --check-prefix=INLINE
+// RUN: %clang --target=x86_64-linux-gnu -fsanitize=kcfi -fsanitize-kcfi-out-of-line-traps -fno-sanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s --check-prefix=INLINE
+// RUN: %clang --target=riscv64-linux-gnu -fsanitize=kcfi -fno-sanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s --check-prefix=INLINE
+// INLINE-NOT: "-fsanitize-kcfi-out-of-line-traps"
+
+// RUN: not %clang --target=riscv64-linux-gnu -fsanitize=kcfi -fsanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s --check-prefix=TARGET
+// TARGET: error: unsupported option '-fsanitize-kcfi-out-of-line-traps' for target 'riscv64-unknown-linux-gnu'
+
+// RUN: %clang --target=x86_64-linux-gnu -fsanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s --check-prefix=UNUSED
+// UNUSED: warning: argument unused during compilation: '-fsanitize-kcfi-out-of-line-traps'
diff --git a/llvm/include/llvm/CodeGen/TargetLowering.h b/llvm/include/llvm/CodeGen/TargetLowering.h
index 858a90e6c4888e..456d912413c60e 100644
--- a/llvm/include/llvm/CodeGen/TargetLowering.h
+++ b/llvm/include/llvm/CodeGen/TargetLowering.h
@@ -2486,6 +2486,19 @@ class LLVM_ABI TargetLoweringBase {
llvm_unreachable("KCFI is not supported on this target");
}
+ /// Return true if the target implements EmitKCFIOutOfLineTrap.
+ virtual bool supportKCFIOutOfLineTraps() const { return false; }
+
+ /// Moves the trap of \p Check, which EmitKCFICheck emitted, out of line:
+ /// rewrites \p Check to not trap, emits a branch after it that goes to
+ /// \p TrapMBB if the check fails, and emits the trap into \p TrapMBB.
+ /// \returns the branch, after which the caller splits the block.
+ virtual MachineInstr *
+ EmitKCFIOutOfLineTrap(MachineInstr &Check, MachineBasicBlock &TrapMBB,
+ const TargetInstrInfo *TII) const {
+ llvm_unreachable("Out-of-line KCFI traps are not supported on this target");
+ }
+
/// @}
//===--------------------------------------------------------------------===//
diff --git a/llvm/lib/CodeGen/KCFI.cpp b/llvm/lib/CodeGen/KCFI.cpp
index 4f940a44b41b43..7c07c5b05cb7e0 100644
--- a/llvm/lib/CodeGen/KCFI.cpp
+++ b/llvm/lib/CodeGen/KCFI.cpp
@@ -44,6 +44,10 @@ class KCFI {
/// Target lowering for arch-specific parts.
const TargetLowering *TLI = nullptr;
+ /// Whether a failed check branches to a trap in a block of its own, rather
+ /// than a passed check branching over a trap that's right before the call.
+ bool OutOfLineTraps = false;
+
/// Emits a KCFI check before an indirect call.
/// \returns true if the check was added and false otherwise.
bool emitCheck(MachineBasicBlock &MBB,
@@ -80,6 +84,9 @@ PreservedAnalyses MachineKCFIPass::run(MachineFunction &MF,
return PreservedAnalyses::all();
PreservedAnalyses PA = getMachineFunctionPassPreservedAnalyses();
+ // Moving the traps out of line adds blocks.
+ if (MF.getFunction().getParent()->getModuleFlag("kcfi-out-of-line-traps"))
+ return PA;
PA.preserveSet<CFGAnalyses>();
return PA;
}
@@ -102,6 +109,29 @@ bool KCFI::emitCheck(MachineBasicBlock &MBB,
assert(MBBI->isCall() && "Unexpected instruction type");
MBBI->setCFIType(*MBB.getParent(), 0);
+ // Move the trap out of line: the check ends in a branch to a new block that
+ // only has the trap, and the call starts the block that the check falls
+ // through to. That branch is never taken unless the check fails, whereas a
+ // branch over an inline trap is taken every time, and the trap is laid out
+ // with the rest of the cold code.
+ //
+ // A call that's already bundled has to stay bundled with its check, and one
+ // in a block with an EH pad successor can be an invoke, which has to stay in
+ // the block of the EH label that begins its try-range. Those keep an inline
+ // trap.
+ if (OutOfLineTraps && !MBBI->isBundled() && !MBB.hasEHPadSuccessor()) {
+ MachineFunction &MF = *MBB.getParent();
+ MachineBasicBlock *TrapMBB =
+ MF.CreateMachineBasicBlock(MBB.getBasicBlock());
+ MF.push_back(TrapMBB);
+ MachineInstr *Branch = TLI->EmitKCFIOutOfLineTrap(*Check, *TrapMBB, TII);
+ MBB.splitAt(*Branch, /*UpdateLiveIns=*/true);
+ MBB.setSuccProbability(MBB.succ_begin(), BranchProbability::getOne());
+ MBB.addSuccessor(TrapMBB, BranchProbability::getZero());
+ ++NumKCFIChecksAdded;
+ return true;
+ }
+
// If not already bundled, bundle the check and the call to prevent
// further changes.
if (!MBBI->isBundled())
@@ -120,6 +150,22 @@ bool KCFI::run(MachineFunction &MF) {
TII = SubTarget.getInstrInfo();
TLI = SubTarget.getTargetLowering();
+ if (M->getModuleFlag("kcfi-out-of-line-traps") &&
+ TLI->supportKCFIOutOfLineTraps()) {
+ OutOfLineTraps = true;
+ // Moving a trap out of line splits the block of its call, so find the
+ // calls first, and go through them last to first so that each split only
+ // moves the instructions up to the previous one.
+ SmallVector<MachineInstr *, 8> Calls;
+ for (MachineBasicBlock &MBB : MF)
+ for (MachineInstr &MI : MBB.instrs())
+ if (MI.isCall() && MI.getCFIType())
+ Calls.push_back(&MI);
+ for (MachineInstr *Call : reverse(Calls))
+ emitCheck(*Call->getParent(), Call->getIterator());
+ return !Calls.empty();
+ }
+
bool Changed = false;
for (MachineBasicBlock &MBB : MF) {
// Use instr_iterator because we don't want to skip bundles.
diff --git a/llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp b/llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp
index 0df6c2cae06303..6aa57d38152003 100644
--- a/llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp
+++ b/llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp
@@ -156,6 +156,7 @@ class AArch64AsmPrinter : public AsmPrinter {
HwasanMemaccessTuple;
std::map<HwasanMemaccessTuple, MCSymbol *> HwasanMemaccessSymbols;
void LowerKCFI_CHECK(const MachineInstr &MI);
+ void LowerKCFI_TRAP(const MachineInstr &MI);
void LowerHWASAN_CHECK_MEMACCESS(const MachineInstr &MI);
void emitHwasanMemaccessSymbols(Module &M);
@@ -696,9 +697,13 @@ void AArch64AsmPrinter::LowerPATCHABLE_EVENT_CALL(const MachineInstr &MI,
void AArch64AsmPrinter::LowerKCFI_CHECK(const MachineInstr &MI) {
Register AddrReg = MI.getOperand(0).getReg();
- assert(std::next(MI.getIterator())->isCall() &&
+ // KCFI_CMP is KCFI_CHECK without the trap: it's followed by the branch to
+ // its KCFI_TRAP, which falls through to the call.
+ const bool IsCmp = MI.getOpcode() == AArch64::KCFI_CMP;
+ assert((IsCmp || std::next(MI.getIterator())->isCall()) &&
"KCFI_CHECK not followed by a call instruction");
- assert(std::next(MI.getIterator())->getOperand(0).getReg() == AddrReg &&
+ assert((IsCmp ||
+ std::next(MI.getIterator())->getOperand(0).getReg() == AddrReg) &&
"KCFI_CHECK call target doesn't match call operand");
// Default to using the intra-procedure-call temporary registers for
@@ -748,6 +753,9 @@ void AArch64AsmPrinter::LowerKCFI_CHECK(const MachineInstr &MI) {
.addReg(ScratchRegs[1])
.addImm(0));
+ if (IsCmp)
+ return;
+
MCSymbol *Pass = OutContext.createTempSymbol();
EmitToStreamer(*OutStreamer,
MCInstBuilder(AArch64::Bcc)
@@ -780,6 +788,18 @@ void AArch64AsmPrinter::LowerKCFI_CHECK(const MachineInstr &MI) {
OutStreamer->emitLabel(Pass);
}
+void AArch64AsmPrinter::LowerKCFI_TRAP(const MachineInstr &MI) {
+ // The ESR that the trap of a KCFI_CHECK has, for the registers that the
+ // KCFI_CMP that branches here left the target and the expected type in.
+ const MCRegisterInfo *RI = OutContext.getRegisterInfo();
+ const unsigned AddrIndex = RI->getEncodingValue(MI.getOperand(0).getReg());
+ const unsigned TypeIndex = RI->getEncodingValue(MI.getOperand(1).getReg());
+ assert(AddrIndex < 31 && TypeIndex < 31);
+
+ const unsigned ESR = 0x8000 | (TypeIndex << 5) | AddrIndex;
+ EmitToStreamer(*OutStreamer, MCInstBuilder(AArch64::BRK).addImm(ESR));
+}
+
void AArch64AsmPrinter::LowerHWASAN_CHECK_MEMACCESS(const MachineInstr &MI) {
Register Reg = MI.getOperand(0).getReg();
@@ -3975,6 +3995,14 @@ void AArch64AsmPrinter::emitInstruction(const MachineInstr *MI) {
LowerKCFI_CHECK(*MI);
return;
+ case AArch64::KCFI_CMP:
+ LowerKCFI_CHECK(*MI);
+ return;
+
+ case AArch64::KCFI_TRAP:
+ LowerKCFI_TRAP(*MI);
+ return;
+
case AArch64::HWASAN_CHECK_MEMACCESS:
case AArch64::HWASAN_CHECK_MEMACCESS_SHORTGRANULES:
case AArch64::HWASAN_CHECK_MEMACCESS_FIXEDSHADOW:
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
index 181f800f4dff88..5a148867cc2983 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
@@ -34626,6 +34626,39 @@ AArch64TargetLowering::EmitKCFICheck(MachineBasicBlock &MBB,
.getInstr();
}
+MachineInstr *
+AArch64TargetLowering::EmitKCFIOutOfLineTrap(MachineInstr &Check,
+ MachineBasicBlock &TrapMBB,
+ const TargetInstrInfo *TII) const {
+ assert(Check.getOpcode() == AArch64::KCFI_CHECK && "Unexpected KCFI check");
+ const MIMetadata MIMD(Check);
+ Check.setDesc(TII->get(AArch64::KCFI_CMP));
+
+ // The trap reports the registers that the check leaves the target and the
+ // expected type in. AArch64AsmPrinter::LowerKCFI_CHECK picks them: the
+ // expected type is in W17, or in W9 if the target is in X17, and a target of
+ // XZR is in X16. They're live into the trap's block, so that a branch to it
+ // that has to be relaxed doesn't clobber them.
+ Register AddrReg = Check.getOperand(0).getReg();
+ const Register TypeReg = AddrReg == AArch64::X17 ? AArch64::W9 : AArch64::W17;
+ if (AddrReg == AArch64::XZR)
+ AddrReg = AArch64::X16;
+ TrapMBB.addLiveIn(AddrReg);
+ TrapMBB.addLiveIn(getXRegFromWReg(TypeReg));
+
+ // Don't let the traps get merged, so that the address of one still says
+ // which check failed, as it does when they're inline.
+ BuildMI(&TrapMBB, MIMD, TII->get(AArch64::KCFI_TRAP))
+ .addReg(AddrReg)
+ .addReg(TypeReg)
+ .setMIFlag(MachineInstr::NoMerge);
+ return BuildMI(*Check.getParent(), std::next(Check.getIterator()), MIMD,
+ TII->get(AArch64::Bcc))
+ .addImm(AArch64CC::NE)
+ .addMBB(&TrapMBB)
+ .getInstr();
+}
+
bool AArch64TargetLowering::shallExtractConstSplatVectorElementToStore(
Type *VectorTy, unsigned ElemSizeInBits, unsigned &Index) const {
// On AArch64, we can efficiently extract a scalar from a splat vector using
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.h b/llvm/lib/Target/AArch64/AArch64ISelLowering.h
index b14fe552da4fd0..09a278445cdf88 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.h
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.h
@@ -519,6 +519,12 @@ class AArch64TargetLowering : public TargetLowering {
MachineBasicBlock::instr_iterator &MBBI,
const TargetInstrInfo *TII) const override;
+ bool supportKCFIOutOfLineTraps() const override { return true; }
+
+ MachineInstr *
+ EmitKCFIOutOfLineTrap(MachineInstr &Check, MachineBasicBlock &TrapMBB,
+ const TargetInstrInfo *TII) const override;
+
bool shallExtractConstSplatVectorElementToStore(
Type *VectorTy, unsigned ElemSizeInBits, unsigned &Index) const override;
diff --git a/llvm/lib/Target/AArch64/AArch64InstrInfo.cpp b/llvm/lib/Target/AArch64/AArch64InstrInfo.cpp
index 40f5d0ef2383eb..2371254f4f108f 100644
--- a/llvm/lib/Target/AArch64/AArch64InstrInfo.cpp
+++ b/llvm/lib/Target/AArch64/AArch64InstrInfo.cpp
@@ -11937,6 +11937,11 @@ bool AArch64InstrInfo::isMBBSafeToSplitToCold(
case AArch64::JumpTableDest16:
case AArch64::JumpTableDest8:
return false;
+ // A conditional branch can't go to another section, so splitting the trap
+ // of a KCFI check from its check would have every check that passes take
+ // a branch over an unconditional one, to save one cold instruction.
+ case AArch64::KCFI_TRAP:
+ return false;
default:
continue;
}
diff --git a/llvm/lib/Target/AArch64/AArch64InstrInfo.td b/llvm/lib/Target/AArch64/AArch64InstrInfo.td
index bb0b5abf440be0..c53936e4be6854 100644
--- a/llvm/lib/Target/AArch64/AArch64InstrInfo.td
+++ b/llvm/lib/Target/AArch64/AArch64InstrInfo.td
@@ -2593,6 +2593,21 @@ def KCFI_CHECK : Pseudo<
(outs), (ins GPR64:$ptr, i32imm:$type), []>, Sched<[]>;
}
+// KCFI_CHECK without the trap: it only sets Z if the type of $ptr is $type.
+// This gets lowered into a 16-byte instruction sequence.
+let Defs = [ X9, X16, X17, NZCV ], Size = 16 in {
+def KCFI_CMP : Pseudo<
+ (outs), (ins GPR64:$ptr, i32imm:$type), []>, Sched<[]>;
+}
+
+// The trap that the branch after a KCFI_CMP goes to if Z isn't set. It's a BRK
+// that says that the KCFI_CMP left the target of the call in $ptr and the
+// expected type in $type.
+let isTrap = 1, hasSideEffects = 1, Size = 4 in {
+def KCFI_TRAP : Pseudo<
+ (outs), (ins GPR64:$ptr, GPR32:$type), []>, Sched<[]>;
+}
+
let Uses = [ X9 ], Defs = [ X16, X17, LR, NZCV ] in {
def HWASAN_CHECK_MEMACCESS : Pseudo<
(outs), (ins GPR64noip:$ptr, i32imm:$accessinfo),
diff --git a/llvm/lib/Target/X86/X86AsmPrinter.h b/llvm/lib/Target/X86/X86AsmPrinter.h
index 2f06ed9ed11a1a..a1dd9a63599cfd 100644
--- a/llvm/lib/Target/X86/X86AsmPrinter.h
+++ b/llvm/lib/Target/X86/X86AsmPrinter.h
@@ -128,6 +128,7 @@ class LLVM_LIBRARY_VISIBILITY X86AsmPrinter : public AsmPrinter {
uint32_t MaskKCFIType(uint32_t Value);
void EmitKCFITypePadding(const MachineFunction &MF, bool HasType = true);
void LowerKCFI_CHECK(const MachineInstr &MI);
+ void LowerKCFI_TRAP(const MachineInstr &MI);
// Address sanitizer specific lowering for X86.
void LowerASAN_CHECK_MEMACCESS(const MachineInstr &MI);
diff --git a/llvm/lib/Target/X86/X86ISelLowering.cpp b/llvm/lib/Target/X86/X86ISelLowering.cpp
index 10d1841a69681a..03734d362a9e65 100644
--- a/llvm/lib/Target/X86/X86ISelLowering.cpp
+++ b/llvm/lib/Target/X86/X86ISelLowering.cpp
@@ -65945,6 +65945,24 @@ X86TargetLowering::EmitKCFICheck(MachineBasicBlock &MBB,
.getInstr();
}
+MachineInstr *
+X86TargetLowering::EmitKCFIOutOfLineTrap(MachineInstr &Check,
+ MachineBasicBlock &TrapMBB,
+ const TargetInstrInfo *TII) const {
+ assert(Check.getOpcode() == X86::KCFI_CHECK && "Unexpected KCFI check");
+ const MIMetadata MIMD(Check);
+ Check.setDesc(TII->get(X86::KCFI_CMP));
+ // Don't let the traps get merged, so that the address of one still says
+ // which check failed, as it does when they're inline.
+ BuildMI(&TrapMBB, MIMD, TII->get(X86::KCFI_TRAP))
+ .setMIFlag(MachineInstr::NoMerge);
+ return BuildMI(*Check.getParent(), std::next(Check.getIterator()), MIMD,
+ TII->get(X86::JCC_1))
+ .addMBB(&TrapMBB)
+ .addImm(X86::COND_NE)
+ .getInstr();
+}
+
/// Returns true if stack probing through a function call is requested.
bool X86TargetLowering::hasStackProbeSymbol(const MachineFunction &MF) const {
return !getStackProbeSymbolName(MF).empty();
diff --git a/llvm/lib/Target/X86/X86ISelLowering.h b/llvm/lib/Target/X86/X86ISelLowering.h
index 8565fedbbb3089..7b313ce4473a17 100644
--- a/llvm/lib/Target/X86/X86ISelLowering.h
+++ b/llvm/lib/Target/X86/X86ISelLowering.h
@@ -694,6 +694,12 @@ namespace llvm {
MachineBasicBlock::instr_iterator &MBBI,
const TargetInstrInfo *TII) const override;
+ bool supportKCFIOutOfLineTraps() const override { return true; }
+
+ MachineInstr *
+ EmitKCFIOutOfLineTrap(MachineInstr &Check, MachineBasicBlock &TrapMBB,
+ const TargetInstrInfo *TII) const override;
+
bool hasStackProbeSymbol(const MachineFunction &MF) const override;
bool hasInlineStackProbe(const MachineFunction &MF) const override;
StringRef getStackProbeSymbolName(const MachineFunction &MF) const override;
diff --git a/llvm/lib/Target/X86/X86InstrCompiler.td b/llvm/lib/Target/X86/X86InstrCompiler.td
index 196c53141d10d6..2b509f6325d8ad 100644
--- a/llvm/lib/Target/X86/X86InstrCompiler.td
+++ b/llvm/lib/Target/X86/X86InstrCompiler.td
@@ -281,8 +281,15 @@ let
Defs = [R10, R11, EFLAGS] in {
def KCFI_CHECK : PseudoI<
(outs), (ins GR64:$ptr, i32imm:$type), []>, Sched<[]>;
+// KCFI_CHECK without the trap: it only sets ZF if the type of $ptr is $type.
+def KCFI_CMP : PseudoI<
+ (outs), (ins GR64:$ptr, i32imm:$type), []>, Sched<[]>;
}
+// The trap that the branch after a KCFI_CMP goes to if ZF isn't set.
+let isTrap = 1, hasSideEffects = 1 in
+def KCFI_TRAP : PseudoI<(outs), (ins), []>, Sched<[]>;
+
//===----------------------------------------------------------------------===//
// Pseudo instructions used by address sanitizer.
//===----------------------------------------------------------------------===//
diff --git a/llvm/lib/Target/X86/X86MCInstLower.cpp b/llvm/lib/Target/X86/X86MCInstLower.cpp
index 00a752a81bc772..58bf56550bea30 100644
--- a/llvm/lib/Target/X86/X86MCInstLower.cpp
+++ b/llvm/lib/Target/X86/X86MCInstLower.cpp
@@ -914,7 +914,10 @@ void X86AsmPrinter::LowerFENTRY_CALL(const MachineInstr &MI,
}
void X86AsmPrinter::LowerKCFI_CHECK(const MachineInstr &MI) {
- assert(std::next(MI.getIterator())->isCall() &&
+ // KCFI_CMP is KCFI_CHECK without the trap: it's followed by the branch to
+ // its KCFI_TRAP, which falls through to the call.
+ const bool IsCmp = MI.getOpcode() == X86::KCFI_CMP;
+ assert((IsCmp || std::next(MI.getIterator())->isCall()) &&
"KCFI_CHECK not followed by a call instruction");
// Adjust the offset for patchable-function-prefix. X86InstrInfo::getNop()
@@ -946,6 +949,9 @@ void X86AsmPrinter::LowerKCFI_CHECK(const MachineInstr &MI) {
.addImm(-(PrefixNops + 4))
.addReg(X86::NoRegister));
+ if (IsCmp)
+ return;
+
MCSymbol *Pass = OutContext.createTempSymbol();
EmitAndCountInstruction(
MCInstBuilder(X86::JCC_1)
@@ -959,6 +965,13 @@ void X86AsmPrinter::LowerKCFI_CHECK(const MachineInstr &MI) {
OutStreamer->emitLabel(Pass);
}
+void X86AsmPrinter::LowerKCFI_TRAP(const MachineInstr &MI) {
+ MCSymbol *Trap = OutContext.createTempSymbol();
+ OutStreamer->emitLabel(Trap);
+ EmitAndCountInstruction(MCInstBuilder(X86::TRAP));
+ emitKCFITrapEntry(*MI.getMF(), Trap);
+}
+
void X86AsmPrinter::LowerASAN_CHECK_MEMACCESS(const MachineInstr &MI) {
// FIXME: Make this work on non-ELF.
if (!TM.getTargetTriple().isOSBinFormatELF()) {
@@ -2563,6 +2576,12 @@ void X86AsmPrinter::emitInstruction(const MachineInstr *MI) {
case X86::KCFI_CHECK:
return LowerKCFI_CHECK(*MI);
+ case X86::KCFI_CMP:
+ return LowerKCFI_CHECK(*MI);
+
+ case X86::KCFI_TRAP:
+ return LowerKCFI_TRAP(*MI);
+
case X86::ASAN_CHECK_MEMACCESS:
return LowerASAN_CHECK_MEMACCESS(*MI);
diff --git a/llvm/test/CodeGen/AArch64/kcfi-out-of-line-traps-xzr.mir b/llvm/test/CodeGen/AArch64/kcfi-out-of-line-traps-xzr.mir
new file mode 100644
index 00000000000000..ee71526d3a4a82
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/kcfi-out-of-line-traps-xzr.mir
@@ -0,0 +1,47 @@
+# RUN: llc -mtriple=aarch64-- -start-before=kcfi -verify-machineinstrs %s -o - | FileCheck %s --check-prefix=ASM
+# RUN: llc -mtriple=aarch64-- -run-pass=kcfi -verify-machineinstrs %s -o - | FileCheck %s --check-prefix=MIR
+
+## A check of a call to xzr compares the expected type with zero, which it has
+## in x16. The out-of-line trap says that the target is in x16, as an inline
+## one does.
+
+# ASM-LABEL: f:
+# ASM: mov x16, xzr
+# ASM-NEXT: movk w17, #24910
+# ASM-NEXT: movk w17, #188, lsl #16
+# ASM-NEXT: cmp w16, w17
+# ASM-NEXT: b.ne [[TRAP:.LBB[0-9_]+]]
+# ASM-NEXT: // %bb.{{[0-9]+}}:
+# ASM-NEXT: blr xzr
+# ASM-NEXT: ret
+# ASM-NEXT: [[TRAP]]:
+# ASM-NEXT: brk #0x8230
+
+# MIR-LABEL: name: f
+# MIR: KCFI_CMP $xzr, 12345678, implicit-def $x9, implicit-def $x16, implicit-def $x17, implicit-def $nzcv
+# MIR-NEXT: Bcc 1, %bb.1, implicit $nzcv
+# MIR: bb.2:
+# MIR-NEXT: BLR $xzr, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
+# MIR: bb.1:
+# MIR-NEXT: liveins: $x16, $x17
+# MIR: nomerge KCFI_TRAP $x16, $w17
+
+--- |
+ define void @f() {
+ ret void
+ }
+
+ !llvm.module.flags = !{!0, !1}
+ !0 = !{i32 4, !"kcfi", i32 1}
+ !1 = !{i32 4, !"kcfi-out-of-line-traps", i32 1}
+...
+---
+name: f
+tracksRegLiveness: true
+body: |
+ bb.0:
+ liveins: $lr
+
+ BLR $xzr, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp, cfi-type 12345678
+ RET undef $lr
+...
diff --git a/llvm/test/CodeGen/AArch64/kcfi-out-of-line-traps.ll b/llvm/test/CodeGen/AArch64/kcfi-out-of-line-traps.ll
new file mode 100644
index 00000000000000..097537a6663bd4
--- /dev/null
+++ b/llvm/test/CodeGen/AArch64/kcfi-out-of-line-traps.ll
@@ -0,0 +1,286 @@
+; RUN: llc -mtriple=aarch64-- -verify-machineinstrs < %s | FileCheck %s --check-prefix=ASM
+; RUN: llc -mtriple=aarch64-- -verify-machineinstrs -stop-after=kcfi < %s | FileCheck %s --check-prefix=MIR
+; RUN: llc -mtriple=aarch64-- -verify-machineinstrs -function-splitting=all < %s | FileCheck %s --check-prefix=SPLIT
+; RUN: llc -mtriple=aarch64-- -verify-machineinstrs -basic-block-sections=all < %s | FileCheck %s --check-prefix=BBS
+
+;; With the kcfi-out-of-line-traps module flag, a KCFI check ends in a branch
+;; that's taken if the check fails, to a block that only has the trap, and the
+;; call starts the block that the check falls through to. The trap has the ESR
+;; that an inline one has.
+
+define void @f1(ptr noundef %x) {
+; ASM-LABEL: f1:
+; ASM: ldur w16, [x0, #-4]
+; ASM-NEXT: movk w17, #24910
+; ASM-NEXT: movk w17, #188, lsl #16
+; ASM-NEXT: cmp w16, w17
+; ASM-NEXT: b.ne [[TRAP:.LBB[0-9_]+]]
+; ASM-NEXT: // %bb.{{[0-9]+}}:
+; ASM-NEXT: blr x0
+; ASM: ret
+; ASM-NEXT: [[TRAP]]:
+; ASM-NEXT: brk #0x8220
+
+; MIR-LABEL: name: f1
+; MIR: bb.0 (%ir-block.0):
+; MIR-NEXT: successors: %bb.2(0x80000000), %bb.1(0x00000000)
+; MIR: KCFI_CMP $x0, 12345678, implicit-def $x9, implicit-def $x16, implicit-def $x17, implicit-def $nzcv
+; MIR-NEXT: Bcc 1, %bb.1, implicit $nzcv
+; MIR: bb.2 (%ir-block.0):
+; MIR-NEXT: liveins: {{.*}}$x0
+; MIR: BLR killed $x0, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
+; MIR: bb.1 (%ir-block.0):
+; MIR-NEXT: liveins: $x0, $x17
+; MIR: nomerge KCFI_TRAP $x0, $w17
+ call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+define void @f2(ptr noundef %x) {
+; ASM-LABEL: f2:
+; ASM: cmp w16, w17
+; ASM-NEXT: b.ne [[TRAP:.LBB[0-9_]+]]
+; ASM-NEXT: // %bb.{{[0-9]+}}:
+; ASM-NEXT: br x0
+; ASM-NEXT: [[TRAP]]:
+; ASM-NEXT: brk #0x8220
+
+; MIR-LABEL: name: f2
+; MIR: KCFI_CMP $x0, 12345678, implicit-def $x9, implicit-def $x16, implicit-def $x17, implicit-def $nzcv
+; MIR-NEXT: Bcc 1, %bb.1, implicit $nzcv
+; MIR: bb.2 (%ir-block.0):
+; MIR: TCRETURNri killed $x0, 0, csr_aarch64_aapcs, implicit $sp
+; MIR: bb.1 (%ir-block.0):
+; MIR-NEXT: liveins: $x0, $x17
+; MIR: nomerge KCFI_TRAP $x0, $w17
+ tail call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+;; If the target is in one of the registers that a check compares the types
+;; in, the check uses w9 instead, and the trap says so.
+define void @f3(ptr noundef %x) #1 {
+; ASM-LABEL: f3:
+; ASM: ldur w9, [x16, #-4]
+; ASM-NEXT: movk w17, #24910
+; ASM-NEXT: movk w17, #188, lsl #16
+; ASM-NEXT: cmp w9, w17
+; ASM-NEXT: b.ne [[TRAP:.LBB[0-9_]+]]
+; ASM-NEXT: // %bb.{{[0-9]+}}:
+; ASM-NEXT: br x16
+; ASM-NEXT: [[TRAP]]:
+; ASM-NEXT: brk #0x8230
+
+; MIR-LABEL: name: f3
+; MIR: KCFI_CMP $x16, 12345678, implicit-def $x9, implicit-def $x16, implicit-def $x17, implicit-def $nzcv
+; MIR-NEXT: Bcc 1, %bb.1, implicit $nzcv
+; MIR: bb.2 (%ir-block.0):
+; MIR-NEXT: liveins: {{.*}}$x16
+; MIR: TCRETURNrix16x17 killed $x16, 0, csr_aarch64_aapcs, implicit $sp
+; MIR: bb.1 (%ir-block.0):
+; MIR-NEXT: liveins: $x16, $x17
+; MIR: nomerge KCFI_TRAP $x16, $w17
+
+;; The registers that the trap reports are live into its block. A branch to
+;; another section can get a veneer from the linker, which clobbers x16, so
+;; one to the trap saves x16 if the target is in it.
+; BBS-LABEL: f3:
+; BBS: cmp w9, w17
+; BBS-NEXT: b.eq [[PASS:.LBB[0-9_]+]]
+; BBS-NEXT: // %bb.{{[0-9]+}}:
+; BBS-NEXT: str x16, [sp, #-16]!
+; BBS-NEXT: b [[TRAP:f3.__part.[0-9]+]]
+; BBS: [[TRAP]]:
+; BBS-NEXT: .cfi_startproc
+; BBS-NEXT: ldr x16, [sp], #16
+; BBS-NEXT: // %bb.{{[0-9]+}}:
+; BBS-NEXT: brk #0x8230
+ tail call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+define void @f4(ptr noundef %x) #2 {
+; ASM-LABEL: f4:
+; ASM: ldur w16, [x17, #-4]
+; ASM-NEXT: movk w9, #24910
+; ASM-NEXT: movk w9, #188, lsl #16
+; ASM-NEXT: cmp w16, w9
+; ASM-NEXT: b.ne [[TRAP:.LBB[0-9_]+]]
+; ASM-NEXT: // %bb.{{[0-9]+}}:
+; ASM-NEXT: br x17
+; ASM-NEXT: [[TRAP]]:
+; ASM-NEXT: brk #0x8131
+
+; MIR-LABEL: name: f4
+; MIR: KCFI_CMP $x17, 12345678, implicit-def $x9, implicit-def $x16, implicit-def $x17, implicit-def $nzcv
+; MIR-NEXT: Bcc 1, %bb.1, implicit $nzcv
+; MIR: bb.2 (%ir-block.0):
+; MIR-NEXT: liveins: {{.*}}$x17
+; MIR: TCRETURNrix17 killed $x17, 0, csr_aarch64_aapcs, implicit $sp
+; MIR: bb.1 (%ir-block.0):
+; MIR-NEXT: liveins: $x17, $x9
+; MIR: nomerge KCFI_TRAP $x17, $w9
+ tail call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+;; Each check has a trap of its own.
+define void @f5(ptr noundef %x, ptr noundef %y) {
+; ASM-LABEL: f5:
+; ASM: cmp w16, w17
+; ASM-NEXT: b.ne [[TRAP1:.LBB[0-9_]+]]
+; ASM-NEXT: // %bb.{{[0-9]+}}:
+; ASM-NEXT: blr x0
+; ASM-NEXT: ldur w16, [x19, #-4]
+; ASM-NEXT: movk w17, #24910
+; ASM-NEXT: movk w17, #188, lsl #16
+; ASM-NEXT: cmp w16, w17
+; ASM-NEXT: b.ne [[TRAP2:.LBB[0-9_]+]]
+; ASM-NEXT: // %bb.{{[0-9]+}}:
+; ASM-NEXT: blr x19
+; ASM: ret
+; ASM-NEXT: [[TRAP1]]:
+; ASM-NEXT: brk #0x8220
+; ASM-NEXT: [[TRAP2]]:
+; ASM-NEXT: brk #0x8233
+
+; MIR-LABEL: name: f5
+; MIR: bb.0 (%ir-block.0):
+; MIR-NEXT: successors: %bb.4(0x80000000), %bb.3(0x00000000)
+; MIR: KCFI_CMP $x0, 12345678, implicit-def $x9, implicit-def $x16, implicit-def $x17, implicit-def $nzcv
+; MIR-NEXT: Bcc 1, %bb.3, implicit $nzcv
+; MIR: bb.4 (%ir-block.0):
+; MIR-NEXT: successors: %bb.2(0x80000000), %bb.1(0x00000000)
+; MIR: BLR killed $x0, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
+; MIR-NEXT: KCFI_CMP $x19, 12345678, implicit-def $x9, implicit-def $x16, implicit-def $x17, implicit-def $nzcv
+; MIR-NEXT: Bcc 1, %bb.1, implicit $nzcv
+; MIR: bb.2 (%ir-block.0):
+; MIR: BLR killed $x19, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
+; MIR: bb.1 (%ir-block.0):
+; MIR: nomerge KCFI_TRAP $x19, $w17
+; MIR: bb.3 (%ir-block.0):
+; MIR: nomerge KCFI_TRAP $x0, $w17
+ call void %x() [ "kcfi"(i32 12345678) ]
+ call void %y() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+;; A call that's already bundled, here with the BTI that follows it, stays
+;; bundled with its check, and keeps an inline trap.
+define void @f6(ptr noundef %x) #1 {
+; ASM-LABEL: f6:
+; ASM: cmp w16, w17
+; ASM-NEXT: b.eq [[PASS:.Ltmp[0-9]+]]
+; ASM-NEXT: brk #0x8220
+; ASM-NEXT: [[PASS]]:
+; ASM-NEXT: blr x0
+; ASM-NEXT: hint #36
+
+; MIR-LABEL: name: f6
+; MIR: BUNDLE{{.*}} {
+; MIR-NEXT: KCFI_CHECK $x0, 12345678, implicit-def $x9, implicit-def $x16, implicit-def $x17, implicit-def $nzcv
+; MIR-NEXT: BLR killed $x0, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
+; MIR-NEXT: HINT 36
+; MIR-NEXT: }
+ call void %x() #0 [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+declare i32 @__gxx_personality_v0(...)
+declare void @cleanup()
+
+;; A call in a block that has an EH pad successor can be an invoke, which has
+;; to stay next to the EH label that starts its try-range. It keeps an inline
+;; trap.
+define void @f7(ptr noundef %x) personality ptr @__gxx_personality_v0 {
+; ASM-LABEL: f7:
+; ASM: cmp w16, w17
+; ASM-NEXT: b.eq [[PASS:.Ltmp[0-9]+]]
+; ASM-NEXT: brk #0x8220
+; ASM-NEXT: [[PASS]]:
+; ASM-NEXT: blr x0
+
+; MIR-LABEL: name: f7
+; MIR: BUNDLE{{.*}} {
+; MIR-NEXT: KCFI_CHECK $x0, 12345678, implicit-def $x9, implicit-def $x16, implicit-def $x17, implicit-def $nzcv
+; MIR-NEXT: BLR killed $x0, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
+; MIR-NEXT: }
+; MIR-NEXT: EH_LABEL
+ call void %x() [ "kcfi"(i32 12345678) ]
+ invoke void @cleanup()
+ to label %cont
+ unwind label %err
+cont:
+ ret void
+err:
+ %exn = landingpad { ptr, i32 }
+ cleanup
+ call void @cleanup()
+ resume { ptr, i32 } %exn
+}
+
+;; A conditional branch can't go to another section, so the trap isn't split
+;; from the rest of the function with the cold blocks. That would have every
+;; check that passes take a branch over an unconditional one.
+define void @f8(ptr noundef %x, i1 %c) !prof !17 {
+; SPLIT-LABEL: f8:
+; SPLIT: cmp w16, w17
+; SPLIT-NEXT: b.ne [[TRAP:.LBB[0-9_]+]]
+; SPLIT-NEXT: // %bb.{{[0-9]+}}:
+; SPLIT-NEXT: blr x0
+; SPLIT: ret
+; SPLIT-NEXT: [[TRAP]]:
+; SPLIT-NEXT: brk #0x8220
+; SPLIT: .section .text.split.f8,"ax", at progbits
+; SPLIT-NEXT: f8.cold:
+; SPLIT: bl cleanup
+; SPLIT-NOT: brk
+; SPLIT: .size f8.cold
+ br i1 %c, label %cold, label %hot, !prof !18
+cold:
+ call void @cleanup()
+ br label %hot
+hot:
+ call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+;; The trap has the CFA of the block that branches to it.
+define void @f9(ptr noundef %x) uwtable(async) {
+; ASM-LABEL: f9:
+; ASM: .cfi_remember_state
+; ASM: cmp w16, w17
+; ASM-NEXT: b.ne [[TRAP:.LBB[0-9_]+]]
+; ASM: .cfi_def_cfa_offset 0
+; ASM: ret
+; ASM-NEXT: [[TRAP]]:
+; ASM-NEXT: .cfi_restore_state
+; ASM-NEXT: brk #0x8220
+ call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+attributes #0 = { returns_twice }
+attributes #1 = { "branch-target-enforcement" }
+attributes #2 = { "branch-target-enforcement" "branch-protection-pauth-lr" "sign-return-address"="all" "sign-return-address-key"="a_key" }
+
+!llvm.module.flags = !{!0, !1, !2, !3}
+!0 = !{i32 4, !"kcfi", i32 1}
+!1 = !{i32 4, !"kcfi-out-of-line-traps", i32 1}
+!2 = !{i32 8, !"branch-target-enforcement", i32 1}
+!3 = !{i32 1, !"ProfileSummary", !4}
+!4 = !{!5, !6, !7, !8, !9, !10, !11, !12}
+!5 = !{!"ProfileFormat", !"InstrProf"}
+!6 = !{!"TotalCount", i64 10000}
+!7 = !{!"MaxCount", i64 10}
+!8 = !{!"MaxInternalCount", i64 1}
+!9 = !{!"MaxFunctionCount", i64 1000}
+!10 = !{!"NumCounts", i64 3}
+!11 = !{!"NumFunctions", i64 5}
+!12 = !{!"DetailedSummary", !13}
+!13 = !{!14, !15, !16}
+!14 = !{i32 10000, i64 100, i32 1}
+!15 = !{i32 999900, i64 100, i32 1}
+!16 = !{i32 999999, i64 1, i32 2}
+!17 = !{!"function_entry_count", i64 7000}
+!18 = !{!"branch_weights", i32 0, i32 7000}
diff --git a/llvm/test/CodeGen/X86/kcfi-out-of-line-traps.ll b/llvm/test/CodeGen/X86/kcfi-out-of-line-traps.ll
new file mode 100644
index 00000000000000..2798f7689d5eb6
--- /dev/null
+++ b/llvm/test/CodeGen/X86/kcfi-out-of-line-traps.ll
@@ -0,0 +1,217 @@
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -verify-machineinstrs < %s | FileCheck %s --check-prefix=ASM
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -verify-machineinstrs -stop-after=kcfi < %s | FileCheck %s --check-prefix=MIR
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -stop-after=kcfi -enable-new-pm < %s | FileCheck %s --check-prefix=MIR
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -verify-machineinstrs -function-splitting=all < %s | FileCheck %s --check-prefix=SPLIT
+
+;; With the kcfi-out-of-line-traps module flag, a KCFI check ends in a branch
+;; that's taken if the check fails, to a block that only has the trap, and the
+;; call starts the block that the check falls through to.
+
+define void @f1(ptr noundef %x) {
+; ASM-LABEL: f1:
+; ASM: movl $4282621618, %r10d # imm = 0xFF439EB2
+; ASM-NEXT: addl -4(%rdi), %r10d
+; ASM-NEXT: jne [[TRAP:.LBB[0-9_]+]]
+; ASM-NEXT: # %bb.{{[0-9]+}}:
+; ASM-NEXT: callq *%rdi
+; ASM: retq
+;; The trap is laid out with the cold blocks, and has the right CFA.
+; ASM-NEXT: [[TRAP]]:
+; ASM-NEXT: .cfi_def_cfa_offset 16
+; ASM-NEXT: [[UD2:.Ltmp[0-9]+]]:
+; ASM-NEXT: ud2
+; ASM-NEXT: .section .kcfi_traps,"ao", at progbits,.text
+; ASM-NEXT: [[ENTRY:.Ltmp[0-9]+]]:
+; ASM-NEXT: .long [[UD2]]-[[ENTRY]]
+; ASM-NEXT: .text
+
+; MIR-LABEL: name: f1
+; MIR: bb.0 (%ir-block.0):
+; MIR-NEXT: successors: %bb.2(0x80000000), %bb.1(0x00000000)
+; MIR: KCFI_CMP $rdi, 12345678, implicit-def $r10, implicit-def $r11, implicit-def $eflags
+; MIR-NEXT: JCC_1 %bb.1, 5, implicit $eflags
+; MIR: bb.2 (%ir-block.0):
+; MIR-NEXT: liveins: {{.*}}$rdi
+; MIR: CALL64r killed $rdi, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp
+; MIR: bb.1 (%ir-block.0):
+; MIR-NEXT: nomerge KCFI_TRAP
+ call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+define void @f2(ptr noundef %x) {
+; ASM-LABEL: f2:
+; ASM: addl -4(%rdi), %r10d
+; ASM-NEXT: jne [[TRAP:.LBB[0-9_]+]]
+; ASM-NEXT: # %bb.{{[0-9]+}}:
+; ASM-NEXT: jmpq *%rdi # TAILCALL
+; ASM-NEXT: [[TRAP]]:
+; ASM-NEXT: .Ltmp{{[0-9]+}}:
+; ASM-NEXT: ud2
+
+; MIR-LABEL: name: f2
+; MIR: KCFI_CMP $rdi, 12345678, implicit-def $r10, implicit-def $r11, implicit-def $eflags
+; MIR-NEXT: JCC_1 %bb.1, 5, implicit $eflags
+; MIR: bb.2 (%ir-block.0):
+; MIR: TAILJMPr64 killed $rdi, csr_64, implicit $rsp, implicit $ssp, implicit $rsp, implicit $ssp
+; MIR: bb.1 (%ir-block.0):
+; MIR-NEXT: nomerge KCFI_TRAP
+ tail call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+;; A call through an indirect thunk has its target in r11.
+define void @f3(ptr noundef %x) #0 {
+; ASM-LABEL: f3:
+; ASM: addl -4(%r11), %r10d
+; ASM-NEXT: jne [[TRAP:.LBB[0-9_]+]]
+; ASM-NEXT: # %bb.{{[0-9]+}}:
+; ASM-NEXT: jmp __llvm_retpoline_r11 # TAILCALL
+; ASM-NEXT: [[TRAP]]:
+; ASM-NEXT: .Ltmp{{[0-9]+}}:
+; ASM-NEXT: ud2
+
+; MIR-LABEL: name: f3
+; MIR: KCFI_CMP $r11, 12345678, implicit-def $r10, implicit-def $r11, implicit-def $eflags
+; MIR-NEXT: JCC_1 %bb.1, 5, implicit $eflags
+; MIR: bb.2 (%ir-block.0):
+; MIR-NEXT: liveins: {{.*}}$r11
+; MIR: TAILJMPd64 &__llvm_retpoline_r11, csr_64, implicit $rsp, implicit $ssp, implicit $rsp, implicit $ssp, implicit killed $r11
+; MIR: bb.1 (%ir-block.0):
+; MIR-NEXT: nomerge KCFI_TRAP
+ tail call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+ at g = external local_unnamed_addr global ptr, align 8
+
+;; So does one that had its memory operand unfolded.
+define void @f4() {
+; MIR-LABEL: name: f4
+; MIR: $r11 = MOV64rm killed renamable $rax, 1, $noreg, 0, $noreg
+; MIR-NEXT: KCFI_CMP $r11, 12345678, implicit-def $r10, implicit-def $r11, implicit-def $eflags
+; MIR-NEXT: JCC_1 %bb.1, 5, implicit $eflags
+; MIR: bb.2 (%ir-block.0):
+; MIR-NEXT: liveins: {{.*}}$r11
+; MIR: CALL64r $r11, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp
+; MIR: bb.1 (%ir-block.0):
+; MIR-NEXT: nomerge KCFI_TRAP
+ %1 = load ptr, ptr @g, align 8
+ call void %1() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+;; Each check has a trap of its own.
+define void @f5(ptr noundef %x, ptr noundef %y) {
+; ASM-LABEL: f5:
+; ASM: addl -4(%rdi), %r10d
+; ASM-NEXT: jne [[TRAP1:.LBB[0-9_]+]]
+; ASM-NEXT: # %bb.{{[0-9]+}}:
+; ASM-NEXT: callq *%rdi
+; ASM-NEXT: movl $4282621618, %r10d # imm = 0xFF439EB2
+; ASM-NEXT: addl -4(%rbx), %r10d
+; ASM-NEXT: jne [[TRAP2:.LBB[0-9_]+]]
+; ASM-NEXT: # %bb.{{[0-9]+}}:
+; ASM-NEXT: callq *%rbx
+; ASM: retq
+; ASM-NEXT: [[TRAP1]]:
+; ASM-NEXT: .cfi_def_cfa_offset 16
+; ASM-NEXT: .Ltmp{{[0-9]+}}:
+; ASM-NEXT: ud2
+; ASM: [[TRAP2]]:
+; ASM-NEXT: .Ltmp{{[0-9]+}}:
+; ASM-NEXT: ud2
+
+; MIR-LABEL: name: f5
+; MIR: bb.0 (%ir-block.0):
+; MIR-NEXT: successors: %bb.4(0x80000000), %bb.3(0x00000000)
+; MIR: KCFI_CMP $rdi, 12345678, implicit-def $r10, implicit-def $r11, implicit-def $eflags
+; MIR-NEXT: JCC_1 %bb.3, 5, implicit $eflags
+; MIR: bb.4 (%ir-block.0):
+; MIR-NEXT: successors: %bb.2(0x80000000), %bb.1(0x00000000)
+; MIR: CALL64r killed $rdi, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp
+; MIR-NEXT: KCFI_CMP $rbx, 12345678, implicit-def $r10, implicit-def $r11, implicit-def $eflags
+; MIR-NEXT: JCC_1 %bb.1, 5, implicit $eflags
+; MIR: bb.2 (%ir-block.0):
+; MIR: CALL64r killed $rbx, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp
+; MIR: bb.1 (%ir-block.0):
+; MIR: nomerge KCFI_TRAP
+; MIR: bb.3 (%ir-block.0):
+; MIR-NEXT: nomerge KCFI_TRAP
+ call void %x() [ "kcfi"(i32 12345678) ]
+ call void %y() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+declare i32 @__gxx_personality_v0(...)
+declare void @cleanup()
+
+;; A call in a block that has an EH pad successor can be an invoke, which has
+;; to stay next to the EH label that starts its try-range. It keeps an inline
+;; trap.
+define void @f6(ptr noundef %x) personality ptr @__gxx_personality_v0 {
+; ASM-LABEL: f6:
+; ASM: addl -4(%rdi), %r10d
+; ASM-NEXT: je [[PASS:.Ltmp[0-9]+]]
+; ASM-NEXT: .Ltmp{{[0-9]+}}:
+; ASM-NEXT: ud2
+; ASM: [[PASS]]:
+; ASM-NEXT: callq *%rdi
+
+; MIR-LABEL: name: f6
+; MIR: BUNDLE{{.*}} {
+; MIR-NEXT: KCFI_CHECK $rdi, 12345678, implicit-def $r10, implicit-def $r11, implicit-def $eflags
+; MIR-NEXT: CALL64r killed $rdi, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp
+; MIR-NEXT: }
+; MIR-NEXT: EH_LABEL
+ call void %x() [ "kcfi"(i32 12345678) ]
+ invoke void @cleanup()
+ to label %cont
+ unwind label %err
+cont:
+ ret void
+err:
+ %exn = landingpad { ptr, i32 }
+ cleanup
+ call void @cleanup()
+ resume { ptr, i32 } %exn
+}
+
+;; With a profile, the trap is cold, so it's split from the rest of the
+;; function.
+define void @f7(ptr noundef %x) !prof !16 {
+; SPLIT-LABEL: f7:
+; SPLIT: addl -4(%rdi), %r10d
+; SPLIT-NEXT: jne f7.cold
+; SPLIT-NEXT: # %bb.{{[0-9]+}}:
+; SPLIT-NEXT: callq *%rdi
+; SPLIT: .section .text.split.f7,"ax", at progbits
+; SPLIT-NEXT: f7.cold:
+; SPLIT-NEXT: .cfi_startproc
+; SPLIT-NEXT: .cfi_def_cfa %rsp, 16
+; SPLIT-NEXT: .Ltmp{{[0-9]+}}:
+; SPLIT-NEXT: ud2
+ call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+attributes #0 = { "target-features"="+retpoline-indirect-branches,+retpoline-indirect-calls" }
+
+!llvm.module.flags = !{!0, !1, !2}
+!0 = !{i32 4, !"kcfi", i32 1}
+!1 = !{i32 4, !"kcfi-out-of-line-traps", i32 1}
+!2 = !{i32 1, !"ProfileSummary", !3}
+!3 = !{!4, !5, !6, !7, !8, !9, !10, !11}
+!4 = !{!"ProfileFormat", !"InstrProf"}
+!5 = !{!"TotalCount", i64 10000}
+!6 = !{!"MaxCount", i64 10}
+!7 = !{!"MaxInternalCount", i64 1}
+!8 = !{!"MaxFunctionCount", i64 1000}
+!9 = !{!"NumCounts", i64 3}
+!10 = !{!"NumFunctions", i64 5}
+!11 = !{!"DetailedSummary", !12}
+!12 = !{!13, !14, !15}
+!13 = !{i32 10000, i64 100, i32 1}
+!14 = !{i32 999900, i64 100, i32 1}
+!15 = !{i32 999999, i64 1, i32 2}
+!16 = !{!"function_entry_count", i64 7000}
>From 08ff6505834a2fbbf6864a6b6633067f6ac36182 Mon Sep 17 00:00:00 2001
From: Chandler Carruth <chandlerc at gmail.com>
Date: Fri, 13 Nov 2026 21:59:41 +0000
Subject: [PATCH 2/2] [KCFI] Fix out-of-line traps with EH, AArch64, LTO, and
MIR
Address correctness, safety, and target interactions for
`-fsanitize-kcfi-out-of-line-traps` while falling back cleanly to inline
checks in rare or complex edge cases:
1. Exception handling and KCFI pass splitting (`KCFI.cpp`):
- Split out-of-line traps for trailing indirect `invoke`s in EH blocks
via a stateless local scan that moves the invoke's single begin
`EH_LABEL` and `DW_CFA_GNU_args_size` before the call while
requiring no following calls/bundles, no intervening calls/bundles,
and no earlier `EH_LABEL`s.
- Fall back cleanly to inline traps under `"eh-asynch"`, Windows EH
(`getWinEHFuncInfo()`), asynchronous EH personalities, `EHPad`
blocks, and non-`invoke` calls or extra `EH_LABEL`s in `invoke`
blocks.
- Treat zero-valued `"kcfi-out-of-line-traps"` module flags as disabled
and only invalidate `CFGAnalyses` in `MachineKCFIPass` when blocks are
actually split.
2. AArch64 and X86 lowering (`AArch64`, `X86`, `TargetLowering`):
- Pass `MachineFunction` to `supportKCFIOutOfLineTraps` and fall back
to inline traps on non-64-bit X86 subtargets and on AArch64 when
`SpeculativeLoadHardening`, XRay instrumentation, basic block
sections (TODO), or callee-pop stack-argument conventions (TODO) are
present.
- Keep `KCFI_TRAP`, `KCFI_CMP`, and `KCFI_CMP`'s fallthrough call block
out of cold sections in `AArch64InstrInfo::isMBBSafeToSplitToCold`.
- Mark `KCFI_TRAP` register operands `undef` with no `TrapMBB`
live-ins on AArch64, share `AArch64::getKCFICheckRegs` and
`emitKCFIBrk`/`emitKCFITrap`, mark `KCFI_CMP` a scheduling boundary
on X86 and AArch64, and assert that `KCFI_CMP` is followed by a
conditional branch.
- Fix AArch64 `PAUTH_EPILOGUE` + inline bundled tail-call checks by
using `getFirstTerminator()` in `AArch64PointerAuth` and unwrapping
bundles in `AArch64FrameLowering::getArgumentStackToRestore`.
3. Duplication, outlining, LTO, MIR round-trip, and docs:
- Mark `KCFI_CMP` and `KCFI_TRAP` `isNotDuplicable = 1` and
`Illegal` to outline in `MachineOutliner` on both X86 and AArch64.
- Emit `"kcfi-out-of-line-traps"` with `Module::Min` in Clang so
mixed LTO links conservatively use inline traps.
- Parse the `nomerge` instruction flag in MIRParser so post-KCFI MIR
with `nomerge KCFI_TRAP` round-trips cleanly.
- Update `clang/docs/ControlFlowIntegrity.md` with LTO, FineIBT/arity,
ICF, >1 MiB branch relaxation, and inline fallback details.
Assisted-by: Antigravity with Gemini
---
clang/docs/ControlFlowIntegrity.md | 43 ++-
clang/lib/CodeGen/CodeGenModule.cpp | 3 +-
clang/lib/Driver/SanitizerArgs.cpp | 3 +-
clang/test/CodeGen/kcfi-out-of-line-traps.c | 6 +-
.../Driver/fsanitize-kcfi-out-of-line-traps.c | 23 +-
llvm/include/llvm/CodeGen/TargetLowering.h | 8 +-
llvm/lib/CodeGen/KCFI.cpp | 157 +++++++---
llvm/lib/CodeGen/MIRParser/MILexer.cpp | 1 +
llvm/lib/CodeGen/MIRParser/MILexer.h | 1 +
llvm/lib/CodeGen/MIRParser/MIParser.cpp | 3 +
llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp | 102 +++----
.../Target/AArch64/AArch64FrameLowering.cpp | 10 +-
.../Target/AArch64/AArch64ISelLowering.cpp | 48 ++-
llvm/lib/Target/AArch64/AArch64ISelLowering.h | 2 +-
llvm/lib/Target/AArch64/AArch64InstrInfo.cpp | 34 ++-
llvm/lib/Target/AArch64/AArch64InstrInfo.h | 16 +
llvm/lib/Target/AArch64/AArch64InstrInfo.td | 6 +-
.../lib/Target/AArch64/AArch64PointerAuth.cpp | 2 +-
llvm/lib/Target/X86/X86AsmPrinter.h | 1 +
llvm/lib/Target/X86/X86ISelLowering.cpp | 7 +-
llvm/lib/Target/X86/X86ISelLowering.h | 2 +-
llvm/lib/Target/X86/X86InstrCompiler.td | 5 +-
llvm/lib/Target/X86/X86InstrInfo.cpp | 9 +-
llvm/lib/Target/X86/X86MCInstLower.cpp | 32 +-
.../AArch64/kcfi-out-of-line-traps-xzr.mir | 23 +-
.../CodeGen/AArch64/kcfi-out-of-line-traps.ll | 205 +++++++++++--
.../X86/kcfi-out-of-line-traps-invoke.ll | 284 ++++++++++++++++++
.../CodeGen/X86/kcfi-out-of-line-traps-lto.ll | 58 ++++
.../X86/kcfi-out-of-line-traps-outliner.ll | 52 ++++
.../X86/kcfi-out-of-line-traps-wineh.ll | 67 +++++
.../CodeGen/X86/kcfi-out-of-line-traps.ll | 25 +-
31 files changed, 1007 insertions(+), 231 deletions(-)
create mode 100644 llvm/test/CodeGen/X86/kcfi-out-of-line-traps-invoke.ll
create mode 100644 llvm/test/CodeGen/X86/kcfi-out-of-line-traps-lto.ll
create mode 100644 llvm/test/CodeGen/X86/kcfi-out-of-line-traps-outliner.ll
create mode 100644 llvm/test/CodeGen/X86/kcfi-out-of-line-traps-wineh.ll
diff --git a/clang/docs/ControlFlowIntegrity.md b/clang/docs/ControlFlowIntegrity.md
index e06afc00ed78cf..b43870db458344 100644
--- a/clang/docs/ControlFlowIntegrity.md
+++ b/clang/docs/ControlFlowIntegrity.md
@@ -332,19 +332,36 @@ enhanced security.
### `-fsanitize-kcfi-out-of-line-traps`
By default (`-fno-sanitize-kcfi-out-of-line-traps`), KCFI emits the trap
-instruction inline immediately before each checked indirect call, and a check
-that passes takes a conditional branch over the trap. Consumers such as the
-Linux kernel rely on this inline layout to patch checks at boot and decode trap
-addresses.
-
-For user-space or other software that does not require inline traps,
-`-fsanitize-kcfi-out-of-line-traps` (supported on x86-64 and AArch64) moves
-each check's trap into a separate cold basic block so that a passing check
-falls through directly to the indirect call and only branches on failure. Each
-check still receives a distinct trap instruction with the call's debug location
-(and `.kcfi_traps` entry on x86-64, or register-encoding `brk` immediate on
-AArch64). Calls that are already bundled or in a basic block with an exception
-handling pad successor keep an inline trap.
+instruction inline immediately before each checked indirect call, and a passing
+check takes a conditional branch over the trap. Consumers such as the Linux
+kernel (including FineIBT paired with `-fsanitize-kcfi-arity`) rely on this
+inline layout to patch checks at boot and decode trap addresses, and should keep
+the default `-fno-sanitize-kcfi-out-of-line-traps`.
+
+On x86-64 and AArch64, `-fsanitize-kcfi-out-of-line-traps` moves each check's
+trap into a separate cold basic block so that passing checks fall through
+directly to the indirect call and only failing checks branch. When using LTO,
+out-of-line traps are only enabled if all merged bitcode translation units
+opt in.
+
+Each out-of-line trap retains the call's debug location (and an ELF
+`.kcfi_traps` entry on x86-64, or the register-encoding `brk` immediate on
+AArch64):
+
+- **Automatic inline fallback:** Functions or calls that fall outside the
+ common out-of-line lowering — including Windows / asynchronous EH
+ functions, calls inside landing-pad blocks, non-`invoke` calls preceding an
+ `invoke` in the same block, multi-`invoke` blocks, calls whose `invoke` lowering inserts helper calls, pre-bundled calls (e.g., Objective-C
+ ARC attached-call markers or `returns_twice` calls with BTI), and (on
+ AArch64) basic block sections, Speculative Load Hardening, XRay-instrumented
+ functions, or callee-pop stack-argument conventions — automatically keep
+ inline traps.
+- **Linker ICF with basic block sections:** When `-fbasic-block-sections=` and
+ linker ICF are both enabled, identical cold trap sections across separate
+ functions may be folded together.
+- **AArch64 large-function branch relaxation:** If a function exceeds the 1 MiB
+ conditional-branch range between a check and its cold trap block,
+ intra-section branch relaxation may clobber `x16` before `brk` executes.
## Member Function Pointer Call Checking
diff --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index 79946e079d3c48..920c97927a9fc5 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -1634,8 +1634,7 @@ void CodeGenModule::Release() {
if (CodeGenOpts.SanitizeKcfiArity)
getModule().addModuleFlag(llvm::Module::Override, "kcfi-arity", 1);
if (CodeGenOpts.SanitizeKcfiOutOfLineTraps)
- getModule().addModuleFlag(llvm::Module::Override,
- "kcfi-out-of-line-traps", 1);
+ getModule().addModuleFlag(llvm::Module::Min, "kcfi-out-of-line-traps", 1);
// Store the hash algorithm choice for use in LLVM passes
getModule().addModuleFlag(
llvm::Module::Override, "kcfi-hash",
diff --git a/clang/lib/Driver/SanitizerArgs.cpp b/clang/lib/Driver/SanitizerArgs.cpp
index 16cd3a7647ac0f..6aafc6831ae19d 100644
--- a/clang/lib/Driver/SanitizerArgs.cpp
+++ b/clang/lib/Driver/SanitizerArgs.cpp
@@ -1633,9 +1633,10 @@ void SanitizerArgs::addArgs(const ToolChain &TC, const llvm::opt::ArgList &Args,
if (KcfiOutOfLineTraps) {
if (TC.getTriple().getArch() != llvm::Triple::x86_64 &&
- !TC.getTriple().isAArch64(64))
+ !TC.getTriple().isAArch64(64)) {
TC.getDriver().Diag(clang::diag::err_drv_unsupported_opt_for_target)
<< "-fsanitize-kcfi-out-of-line-traps" << TC.getTriple().str();
+ }
CmdArgs.push_back("-fsanitize-kcfi-out-of-line-traps");
}
diff --git a/clang/test/CodeGen/kcfi-out-of-line-traps.c b/clang/test/CodeGen/kcfi-out-of-line-traps.c
index ae178f44bd5178..384cc553f2ebc0 100644
--- a/clang/test/CodeGen/kcfi-out-of-line-traps.c
+++ b/clang/test/CodeGen/kcfi-out-of-line-traps.c
@@ -1,5 +1,5 @@
// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm -fsanitize=kcfi -fsanitize-kcfi-out-of-line-traps -o - %s | FileCheck %s
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm -fsanitize=kcfi -o - %s | FileCheck %s --check-prefix=INLINE
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm -fsanitize=kcfi -o - %s | FileCheck %s --check-prefix=INLINE --implicit-check-not='"kcfi-out-of-line-traps"'
-// CHECK: ![[#]] = !{i32 4, !"kcfi-out-of-line-traps", i32 1}
-// INLINE-NOT: !"kcfi-out-of-line-traps"
+// CHECK: ![[#]] = !{i32 8, !"kcfi-out-of-line-traps", i32 1}
+// INLINE: ![[#]] = !{i32 4, !"kcfi", i32 1}
diff --git a/clang/test/Driver/fsanitize-kcfi-out-of-line-traps.c b/clang/test/Driver/fsanitize-kcfi-out-of-line-traps.c
index bf3dc437a30e94..28913aa9fc1978 100644
--- a/clang/test/Driver/fsanitize-kcfi-out-of-line-traps.c
+++ b/clang/test/Driver/fsanitize-kcfi-out-of-line-traps.c
@@ -1,15 +1,18 @@
-// RUN: %clang --target=x86_64-linux-gnu -fsanitize=kcfi -fsanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s
-// RUN: %clang --target=aarch64-linux-gnu -fsanitize=kcfi -fsanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s
-// RUN: %clang --target=x86_64-linux-gnu -fsanitize=kcfi -fno-sanitize-kcfi-out-of-line-traps -fsanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s
+// RUN: %clang -c --target=x86_64-linux-gnu -fsanitize=kcfi -fsanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s
+// RUN: %clang -c --target=aarch64-linux-gnu -fsanitize=kcfi -fsanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s
+// RUN: %clang -c --target=x86_64-linux-gnu -fsanitize=kcfi -fno-sanitize-kcfi-out-of-line-traps -fsanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s
// CHECK: "-fsanitize-kcfi-out-of-line-traps"
-// RUN: %clang --target=x86_64-linux-gnu -fsanitize=kcfi %s -### 2>&1 | FileCheck %s --check-prefix=INLINE
-// RUN: %clang --target=x86_64-linux-gnu -fsanitize=kcfi -fsanitize-kcfi-out-of-line-traps -fno-sanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s --check-prefix=INLINE
-// RUN: %clang --target=riscv64-linux-gnu -fsanitize=kcfi -fno-sanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s --check-prefix=INLINE
-// INLINE-NOT: "-fsanitize-kcfi-out-of-line-traps"
+// RUN: %clang -c --target=x86_64-linux-gnu -fsanitize=kcfi %s -### 2>&1 | FileCheck %s --check-prefix=INLINE --implicit-check-not="-fsanitize-kcfi-out-of-line-traps"
+// RUN: %clang -c --target=x86_64-linux-gnu -fsanitize=kcfi -fsanitize-kcfi-out-of-line-traps -fno-sanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s --check-prefix=INLINE --implicit-check-not="-fsanitize-kcfi-out-of-line-traps"
+// RUN: %clang -c --target=riscv64-linux-gnu -fsanitize=kcfi -fno-sanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s --check-prefix=INLINE --implicit-check-not="-fsanitize-kcfi-out-of-line-traps"
+// INLINE: "-cc1"{{.*}}"-fsanitize=kcfi"
-// RUN: not %clang --target=riscv64-linux-gnu -fsanitize=kcfi -fsanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s --check-prefix=TARGET
-// TARGET: error: unsupported option '-fsanitize-kcfi-out-of-line-traps' for target 'riscv64-unknown-linux-gnu'
+// RUN: not %clang -c --target=riscv64-linux-gnu -fsanitize=kcfi -fsanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s --check-prefix=TARGET-RISCV
+// TARGET-RISCV: error: unsupported option '-fsanitize-kcfi-out-of-line-traps' for target 'riscv64-unknown-linux-gnu'
+// RUN: not %clang -c --target=i386-linux-gnu -fsanitize=kcfi -fsanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s --check-prefix=TARGET-I386
+// TARGET-I386: error: unsupported option '-fsanitize-kcfi-out-of-line-traps' for target 'i386-unknown-linux-gnu'
-// RUN: %clang --target=x86_64-linux-gnu -fsanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s --check-prefix=UNUSED
+// RUN: %clang -c --target=x86_64-linux-gnu -fsanitize-kcfi-out-of-line-traps %s -### 2>&1 | FileCheck %s --check-prefix=UNUSED
// UNUSED: warning: argument unused during compilation: '-fsanitize-kcfi-out-of-line-traps'
+// UNUSED-NOT: "-fsanitize-kcfi-out-of-line-traps"
diff --git a/llvm/include/llvm/CodeGen/TargetLowering.h b/llvm/include/llvm/CodeGen/TargetLowering.h
index d4058b78f50036..e40636a5ed806b 100644
--- a/llvm/include/llvm/CodeGen/TargetLowering.h
+++ b/llvm/include/llvm/CodeGen/TargetLowering.h
@@ -2486,12 +2486,16 @@ class LLVM_ABI TargetLoweringBase {
llvm_unreachable("KCFI is not supported on this target");
}
- /// Return true if the target implements EmitKCFIOutOfLineTrap.
- virtual bool supportKCFIOutOfLineTraps() const { return false; }
+ /// Return true if the target implements EmitKCFIOutOfLineTrap for \p MF.
+ virtual bool supportKCFIOutOfLineTraps(const MachineFunction &MF) const {
+ return false;
+ }
/// Moves the trap of \p Check, which EmitKCFICheck emitted, out of line:
/// rewrites \p Check to not trap, emits a branch after it that goes to
/// \p TrapMBB if the check fails, and emits the trap into \p TrapMBB.
+ /// \p Check must already be inserted in a basic block and \p TrapMBB must be
+ /// a newly created empty block.
/// \returns the branch, after which the caller splits the block.
virtual MachineInstr *
EmitKCFIOutOfLineTrap(MachineInstr &Check, MachineBasicBlock &TrapMBB,
diff --git a/llvm/lib/CodeGen/KCFI.cpp b/llvm/lib/CodeGen/KCFI.cpp
index 7c07c5b05cb7e0..0caf7c01af6b9b 100644
--- a/llvm/lib/CodeGen/KCFI.cpp
+++ b/llvm/lib/CodeGen/KCFI.cpp
@@ -8,22 +8,27 @@
//
// This pass implements Kernel Control-Flow Integrity (KCFI) indirect call
// check lowering. For each call instruction with a cfi-type attribute, it
-// emits an arch-specific check before the call, and bundles the check and
-// the call to prevent unintentional modifications.
+// emits an arch-specific check before the call and either bundles the check
+// with the call (keeping the trap inline) or splits the block to move the trap
+// out of line when enabled and supported.
//
//===----------------------------------------------------------------------===//
#include "llvm/CodeGen/KCFI.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SmallVector.h"
#include "llvm/ADT/Statistic.h"
#include "llvm/CodeGen/MachineFunctionPass.h"
#include "llvm/CodeGen/MachineInstrBundle.h"
-#include "llvm/CodeGen/MachineModuleInfo.h"
#include "llvm/CodeGen/RegisterClassInfo.h"
#include "llvm/CodeGen/TargetInstrInfo.h"
#include "llvm/CodeGen/TargetLowering.h"
#include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/IR/Constants.h"
+#include "llvm/IR/EHPersonalities.h"
#include "llvm/IR/Module.h"
#include "llvm/InitializePasses.h"
+#include "llvm/MC/MCDwarf.h"
using namespace llvm;
@@ -32,10 +37,16 @@ using namespace llvm;
STATISTIC(NumKCFIChecksAdded, "Number of indirect call checks added");
+static bool hasNonZeroModuleFlag(const Module &M, StringRef Name) {
+ auto *Val = mdconst::extract_or_null<ConstantInt>(M.getModuleFlag(Name));
+ return Val && !Val->isZero();
+}
+
namespace {
class KCFI {
public:
bool run(MachineFunction &MF);
+ bool didSplitBlocks() const { return SplitBlocks; }
private:
/// Machine instruction info used throughout the class.
@@ -48,10 +59,17 @@ class KCFI {
/// than a passed check branching over a trap that's right before the call.
bool OutOfLineTraps = false;
+ /// Whether run() actually split any block (creating a TrapMBB/SplitBB).
+ bool SplitBlocks = false;
+
+ /// Moves \p Check's trap out of line if safe for \p MBBI, returning true if
+ /// the block was split.
+ bool tryEmitOutOfLineTrap(MachineBasicBlock &MBB,
+ MachineBasicBlock::instr_iterator MBBI,
+ MachineInstr &Check);
+
/// Emits a KCFI check before an indirect call.
- /// \returns true if the check was added and false otherwise.
- bool emitCheck(MachineBasicBlock &MBB,
- MachineBasicBlock::instr_iterator I) const;
+ void emitCheck(MachineBasicBlock &MBB, MachineBasicBlock::instr_iterator I);
};
class MachineKCFILegacy : public MachineFunctionPass {
@@ -80,19 +98,80 @@ FunctionPass *llvm::createKCFIPass() { return new MachineKCFILegacy(); }
PreservedAnalyses MachineKCFIPass::run(MachineFunction &MF,
MachineFunctionAnalysisManager &MFAM) {
- if (!KCFI().run(MF))
+ KCFI Impl;
+ if (!Impl.run(MF))
return PreservedAnalyses::all();
PreservedAnalyses PA = getMachineFunctionPassPreservedAnalyses();
- // Moving the traps out of line adds blocks.
- if (MF.getFunction().getParent()->getModuleFlag("kcfi-out-of-line-traps"))
- return PA;
- PA.preserveSet<CFGAnalyses>();
+ if (!Impl.didSplitBlocks())
+ PA.preserveSet<CFGAnalyses>();
return PA;
}
-bool KCFI::emitCheck(MachineBasicBlock &MBB,
- MachineBasicBlock::instr_iterator MBBI) const {
+bool KCFI::tryEmitOutOfLineTrap(MachineBasicBlock &MBB,
+ MachineBasicBlock::instr_iterator MBBI,
+ MachineInstr &Check) {
+ if (!OutOfLineTraps || MBBI->isBundled() || MBB.isEHPad())
+ return false;
+
+ MachineFunction &MF = *MBB.getParent();
+ SmallVector<MachineInstr *, 2> MoveBeforeCall;
+ if (MBB.hasEHPadSuccessor()) {
+ // Only split the trailing invoke in an EH block: require no following
+ // call/bundle and a single end EH_LABEL after MBBI, plus a single
+ // begin EH_LABEL before Check with no intervening call/bundle and no
+ // earlier EH_LABEL in MBB.
+ unsigned EndLabels = 0;
+ for (auto It = std::next(MBBI); It != MBB.instr_end(); ++It) {
+ if (It->isCall() || It->isBundled())
+ return false;
+ if (It->isEHLabel() && ++EndLabels > 1)
+ return false;
+ }
+ if (EndLabels != 1)
+ return false;
+
+ MachineInstr *BeginLabel = nullptr;
+ for (auto It = Check.getIterator(); It != MBB.instr_begin();) {
+ --It;
+ if (!BeginLabel) {
+ if (It->isCall() || It->isBundled())
+ return false;
+ if (It->isEHLabel()) {
+ BeginLabel = &*It;
+ MoveBeforeCall.push_back(&*It);
+ } else if (It->isCFIInstruction() &&
+ MF.getFrameInstructions()[It->getOperand(0).getCFIIndex()]
+ .getOperation() == MCCFIInstruction::OpGnuArgsSize) {
+ MoveBeforeCall.push_back(&*It);
+ }
+ } else if (It->isEHLabel()) {
+ return false;
+ }
+ }
+ if (!BeginLabel)
+ return false;
+ }
+
+ MachineBasicBlock *TrapMBB = MF.CreateMachineBasicBlock(MBB.getBasicBlock());
+ MF.push_back(TrapMBB);
+ MachineInstr *Branch = TLI->EmitKCFIOutOfLineTrap(Check, *TrapMBB, TII);
+
+ for (MachineInstr *MI : reverse(MoveBeforeCall))
+ MBB.splice(MBBI->getIterator(), &MBB, MI->getIterator());
+
+ MachineBasicBlock *SplitBB = MBB.splitAt(*Branch, /*UpdateLiveIns=*/true);
+ assert(SplitBB != &MBB && "Expected a split block before the call");
+ (void)SplitBB;
+ MBB.setSuccProbability(MBB.succ_begin(), BranchProbability::getOne());
+ MBB.addSuccessor(TrapMBB, BranchProbability::getZero());
+
+ SplitBlocks = true;
+ return true;
+}
+
+void KCFI::emitCheck(MachineBasicBlock &MBB,
+ MachineBasicBlock::instr_iterator MBBI) {
assert(TII && "Target instruction info was not initialized");
assert(TLI && "Target lowering was not initialized");
@@ -109,36 +188,12 @@ bool KCFI::emitCheck(MachineBasicBlock &MBB,
assert(MBBI->isCall() && "Unexpected instruction type");
MBBI->setCFIType(*MBB.getParent(), 0);
- // Move the trap out of line: the check ends in a branch to a new block that
- // only has the trap, and the call starts the block that the check falls
- // through to. That branch is never taken unless the check fails, whereas a
- // branch over an inline trap is taken every time, and the trap is laid out
- // with the rest of the cold code.
- //
- // A call that's already bundled has to stay bundled with its check, and one
- // in a block with an EH pad successor can be an invoke, which has to stay in
- // the block of the EH label that begins its try-range. Those keep an inline
- // trap.
- if (OutOfLineTraps && !MBBI->isBundled() && !MBB.hasEHPadSuccessor()) {
- MachineFunction &MF = *MBB.getParent();
- MachineBasicBlock *TrapMBB =
- MF.CreateMachineBasicBlock(MBB.getBasicBlock());
- MF.push_back(TrapMBB);
- MachineInstr *Branch = TLI->EmitKCFIOutOfLineTrap(*Check, *TrapMBB, TII);
- MBB.splitAt(*Branch, /*UpdateLiveIns=*/true);
- MBB.setSuccProbability(MBB.succ_begin(), BranchProbability::getOne());
- MBB.addSuccessor(TrapMBB, BranchProbability::getZero());
- ++NumKCFIChecksAdded;
- return true;
- }
-
- // If not already bundled, bundle the check and the call to prevent
- // further changes.
- if (!MBBI->isBundled())
+ // Move the trap out of line if supported and safe; otherwise bundle the
+ // check and call to prevent further changes.
+ if (!tryEmitOutOfLineTrap(MBB, MBBI, *Check) && !MBBI->isBundled())
finalizeBundle(MBB, Check->getIterator(), std::next(MBBI->getIterator()));
++NumKCFIChecksAdded;
- return true;
}
bool KCFI::run(MachineFunction &MF) {
@@ -150,9 +205,13 @@ bool KCFI::run(MachineFunction &MF) {
TII = SubTarget.getInstrInfo();
TLI = SubTarget.getTargetLowering();
- if (M->getModuleFlag("kcfi-out-of-line-traps") &&
- TLI->supportKCFIOutOfLineTraps()) {
- OutOfLineTraps = true;
+ const Function &F = MF.getFunction();
+ if (hasNonZeroModuleFlag(*M, "kcfi-out-of-line-traps") &&
+ !hasNonZeroModuleFlag(*M, "eh-asynch") && !MF.getWinEHFuncInfo() &&
+ (!F.hasPersonalityFn() ||
+ !isAsynchronousEHPersonality(
+ classifyEHPersonality(F.getPersonalityFn()))) &&
+ TLI->supportKCFIOutOfLineTraps(MF)) {
// Moving a trap out of line splits the block of its call, so find the
// calls first, and go through them last to first so that each split only
// moves the instructions up to the previous one.
@@ -161,9 +220,13 @@ bool KCFI::run(MachineFunction &MF) {
for (MachineInstr &MI : MBB.instrs())
if (MI.isCall() && MI.getCFIType())
Calls.push_back(&MI);
+ if (Calls.empty())
+ return false;
+
+ OutOfLineTraps = true;
for (MachineInstr *Call : reverse(Calls))
emitCheck(*Call->getParent(), Call->getIterator());
- return !Calls.empty();
+ return true;
}
bool Changed = false;
@@ -172,8 +235,10 @@ bool KCFI::run(MachineFunction &MF) {
for (MachineBasicBlock::instr_iterator MII = MBB.instr_begin(),
MIE = MBB.instr_end();
MII != MIE; ++MII) {
- if (MII->isCall() && MII->getCFIType())
- Changed |= emitCheck(MBB, MII);
+ if (MII->isCall() && MII->getCFIType()) {
+ emitCheck(MBB, MII);
+ Changed = true;
+ }
}
}
diff --git a/llvm/lib/CodeGen/MIRParser/MILexer.cpp b/llvm/lib/CodeGen/MIRParser/MILexer.cpp
index ed8b0f7e06f04b..5c58a511bc61dc 100644
--- a/llvm/lib/CodeGen/MIRParser/MILexer.cpp
+++ b/llvm/lib/CodeGen/MIRParser/MILexer.cpp
@@ -220,6 +220,7 @@ static MIToken::TokenKind getIdentifierKind(StringRef Identifier) {
.Case("inbounds", MIToken::kw_inbounds)
.Case("nonnull", MIToken::kw_nonnull)
.Case("nofpexcept", MIToken::kw_nofpexcept)
+ .Case("nomerge", MIToken::kw_nomerge)
.Case("unpredictable", MIToken::kw_unpredictable)
.Case("debug-location", MIToken::kw_debug_location)
.Case("debug-instr-number", MIToken::kw_debug_instr_number)
diff --git a/llvm/lib/CodeGen/MIRParser/MILexer.h b/llvm/lib/CodeGen/MIRParser/MILexer.h
index 555cc948ea2ba2..406d202856f3a3 100644
--- a/llvm/lib/CodeGen/MIRParser/MILexer.h
+++ b/llvm/lib/CodeGen/MIRParser/MILexer.h
@@ -74,6 +74,7 @@ struct MIToken {
kw_nsw,
kw_exact,
kw_nofpexcept,
+ kw_nomerge,
kw_unpredictable,
kw_nneg,
kw_disjoint,
diff --git a/llvm/lib/CodeGen/MIRParser/MIParser.cpp b/llvm/lib/CodeGen/MIRParser/MIParser.cpp
index 1aaf3015c9d4c9..aacbb8dccc7dce 100644
--- a/llvm/lib/CodeGen/MIRParser/MIParser.cpp
+++ b/llvm/lib/CodeGen/MIRParser/MIParser.cpp
@@ -1425,6 +1425,7 @@ bool MIParser::parseInstruction(unsigned &OpCode, unsigned &Flags) {
Token.is(MIToken::kw_nsw) ||
Token.is(MIToken::kw_exact) ||
Token.is(MIToken::kw_nofpexcept) ||
+ Token.is(MIToken::kw_nomerge) ||
Token.is(MIToken::kw_noconvergent) ||
Token.is(MIToken::kw_unpredictable) ||
Token.is(MIToken::kw_nneg) ||
@@ -1462,6 +1463,8 @@ bool MIParser::parseInstruction(unsigned &OpCode, unsigned &Flags) {
Flags |= MachineInstr::IsExact;
if (Token.is(MIToken::kw_nofpexcept))
Flags |= MachineInstr::NoFPExcept;
+ if (Token.is(MIToken::kw_nomerge))
+ Flags |= MachineInstr::NoMerge;
if (Token.is(MIToken::kw_unpredictable))
Flags |= MachineInstr::Unpredictable;
if (Token.is(MIToken::kw_noconvergent))
diff --git a/llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp b/llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp
index 6aa57d38152003..f490e38104a9ff 100644
--- a/llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp
+++ b/llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp
@@ -157,6 +157,7 @@ class AArch64AsmPrinter : public AsmPrinter {
std::map<HwasanMemaccessTuple, MCSymbol *> HwasanMemaccessSymbols;
void LowerKCFI_CHECK(const MachineInstr &MI);
void LowerKCFI_TRAP(const MachineInstr &MI);
+ void emitKCFIBrk(Register AddrReg, Register TypeReg);
void LowerHWASAN_CHECK_MEMACCESS(const MachineInstr &MI);
void emitHwasanMemaccessSymbols(Module &M);
@@ -699,33 +700,28 @@ void AArch64AsmPrinter::LowerKCFI_CHECK(const MachineInstr &MI) {
Register AddrReg = MI.getOperand(0).getReg();
// KCFI_CMP is KCFI_CHECK without the trap: it's followed by the branch to
// its KCFI_TRAP, which falls through to the call.
- const bool IsCmp = MI.getOpcode() == AArch64::KCFI_CMP;
- assert((IsCmp || std::next(MI.getIterator())->isCall()) &&
- "KCFI_CHECK not followed by a call instruction");
- assert((IsCmp ||
- std::next(MI.getIterator())->getOperand(0).getReg() == AddrReg) &&
- "KCFI_CHECK call target doesn't match call operand");
-
- // Default to using the intra-procedure-call temporary registers for
- // comparing the hashes.
- unsigned ScratchRegs[] = {AArch64::W16, AArch64::W17};
+ bool IsCmp = MI.getOpcode() == AArch64::KCFI_CMP;
+ if (IsCmp) {
+ MachineBasicBlock::const_instr_iterator Next =
+ next_nodbg(MI.getIterator(), MI.getParent()->instr_end());
+ assert(Next != MI.getParent()->instr_end() && Next->isConditionalBranch() &&
+ "KCFI_CMP not followed by a conditional branch");
+ } else {
+ assert(std::next(MI.getIterator())->isCall() &&
+ "KCFI_CHECK not followed by a call instruction");
+ assert(std::next(MI.getIterator())->getOperand(0).getReg() == AddrReg &&
+ "KCFI_CHECK call target doesn't match call operand");
+ }
+
+ // Pick intra-procedure-call temporary registers via getKCFICheckRegs so
+ // EmitKCFIOutOfLineTrap and LowerKCFI_CHECK agree on the trap registers.
+ AArch64::KCFICheckRegs Regs = AArch64::getKCFICheckRegs(AddrReg);
if (AddrReg == AArch64::XZR) {
// Checking XZR makes no sense. Instead of emitting a load, zero
- // ScratchRegs[0] and use it for the ESR AddrIndex below.
- AddrReg = getXRegFromWReg(ScratchRegs[0]);
- emitMovXReg(AddrReg, AArch64::XZR);
+ // Regs.AddrReg so the trap below still encodes a valid GPR.
+ emitMovXReg(Regs.AddrReg, AArch64::XZR);
} else {
- // If one of the scratch registers is used for the call target (e.g.
- // with AArch64::TCRETURNriBTI), we can clobber another caller-saved
- // temporary register instead (in this case, AArch64::W9) as the check
- // is immediately followed by the call instruction.
- for (auto &Reg : ScratchRegs) {
- if (Reg == getWRegFromXReg(AddrReg)) {
- Reg = AArch64::W9;
- break;
- }
- }
- assert(ScratchRegs[0] != AddrReg && ScratchRegs[1] != AddrReg &&
+ assert(Regs.TargetHashReg != AddrReg && Regs.ExpectedTypeReg != AddrReg &&
"Invalid scratch registers for KCFI_CHECK");
// Adjust the offset for patchable-function-prefix. This assumes that
@@ -736,21 +732,21 @@ void AArch64AsmPrinter::LowerKCFI_CHECK(const MachineInstr &MI) {
// Load the target function type hash.
EmitToStreamer(*OutStreamer, MCInstBuilder(AArch64::LDURWi)
- .addReg(ScratchRegs[0])
- .addReg(AddrReg)
+ .addReg(Regs.TargetHashReg)
+ .addReg(Regs.AddrReg)
.addImm(-(PrefixNops * 4 + 4)));
}
// Load the expected type hash.
- const int64_t Type = MI.getOperand(1).getImm();
- emitMOVK(ScratchRegs[1], Type & 0xFFFF, 0);
- emitMOVK(ScratchRegs[1], (Type >> 16) & 0xFFFF, 16);
+ int64_t Type = MI.getOperand(1).getImm();
+ emitMOVK(Regs.ExpectedTypeReg, Type & 0xFFFF, 0);
+ emitMOVK(Regs.ExpectedTypeReg, (Type >> 16) & 0xFFFF, 16);
// Compare the hashes and trap if there's a mismatch.
EmitToStreamer(*OutStreamer, MCInstBuilder(AArch64::SUBSWrs)
.addReg(AArch64::WZR)
- .addReg(ScratchRegs[0])
- .addReg(ScratchRegs[1])
+ .addReg(Regs.TargetHashReg)
+ .addReg(Regs.ExpectedTypeReg)
.addImm(0));
if (IsCmp)
@@ -762,42 +758,27 @@ void AArch64AsmPrinter::LowerKCFI_CHECK(const MachineInstr &MI) {
.addImm(AArch64CC::EQ)
.addExpr(MCSymbolRefExpr::create(Pass, OutContext)));
+ emitKCFIBrk(Regs.AddrReg, Regs.ExpectedTypeReg);
+ OutStreamer->emitLabel(Pass);
+}
+
+void AArch64AsmPrinter::emitKCFIBrk(Register AddrReg, Register TypeReg) {
// The base ESR is 0x8000 and the register information is encoded in bits
// 0-9 as follows:
- // - 0-4: n, where the register Xn contains the target address
- // - 5-9: m, where the register Wm contains the expected type hash
+ // - 0-4: n, where register Xn contains the target address
+ // - 5-9: m, where register Wm contains the expected type hash
// Where n, m are in [0, 30].
- unsigned TypeIndex = ScratchRegs[1] - AArch64::W0;
- unsigned AddrIndex;
- switch (AddrReg) {
- default:
- AddrIndex = AddrReg - AArch64::X0;
- break;
- case AArch64::FP:
- AddrIndex = 29;
- break;
- case AArch64::LR:
- AddrIndex = 30;
- break;
- }
-
- assert(AddrIndex < 31 && TypeIndex < 31);
+ const MCRegisterInfo *RI = OutContext.getRegisterInfo();
+ unsigned AddrIndex = RI->getEncodingValue(AddrReg);
+ unsigned TypeIndex = RI->getEncodingValue(TypeReg);
+ assert(AddrIndex < 31 && TypeIndex < 31 && "Invalid KCFI trap registers");
- unsigned ESR = 0x8000 | ((TypeIndex & 31) << 5) | (AddrIndex & 31);
+ unsigned ESR = 0x8000 | (TypeIndex << 5) | AddrIndex;
EmitToStreamer(*OutStreamer, MCInstBuilder(AArch64::BRK).addImm(ESR));
- OutStreamer->emitLabel(Pass);
}
void AArch64AsmPrinter::LowerKCFI_TRAP(const MachineInstr &MI) {
- // The ESR that the trap of a KCFI_CHECK has, for the registers that the
- // KCFI_CMP that branches here left the target and the expected type in.
- const MCRegisterInfo *RI = OutContext.getRegisterInfo();
- const unsigned AddrIndex = RI->getEncodingValue(MI.getOperand(0).getReg());
- const unsigned TypeIndex = RI->getEncodingValue(MI.getOperand(1).getReg());
- assert(AddrIndex < 31 && TypeIndex < 31);
-
- const unsigned ESR = 0x8000 | (TypeIndex << 5) | AddrIndex;
- EmitToStreamer(*OutStreamer, MCInstBuilder(AArch64::BRK).addImm(ESR));
+ emitKCFIBrk(MI.getOperand(0).getReg(), MI.getOperand(1).getReg());
}
void AArch64AsmPrinter::LowerHWASAN_CHECK_MEMACCESS(const MachineInstr &MI) {
@@ -3992,9 +3973,6 @@ void AArch64AsmPrinter::emitInstruction(const MachineInstr *MI) {
return LowerPATCHABLE_EVENT_CALL(*MI, true);
case AArch64::KCFI_CHECK:
- LowerKCFI_CHECK(*MI);
- return;
-
case AArch64::KCFI_CMP:
LowerKCFI_CHECK(*MI);
return;
diff --git a/llvm/lib/Target/AArch64/AArch64FrameLowering.cpp b/llvm/lib/Target/AArch64/AArch64FrameLowering.cpp
index 195a4ec48c78b7..e710f83813df4b 100644
--- a/llvm/lib/Target/AArch64/AArch64FrameLowering.cpp
+++ b/llvm/lib/Target/AArch64/AArch64FrameLowering.cpp
@@ -270,13 +270,15 @@ AArch64FrameLowering::getArgumentStackToRestore(MachineFunction &MF,
MachineBasicBlock &MBB) const {
MachineBasicBlock::iterator MBBI = MBB.getLastNonDebugInstr();
AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
- bool IsTailCallReturn = (MBB.end() != MBBI)
- ? AArch64InstrInfo::isTailCallReturnInst(*MBBI)
- : false;
+ const MachineInstr *LastMI =
+ MBBI != MBB.end() ? &*std::prev(std::next(MBBI).getInstrIterator())
+ : nullptr;
+ bool IsTailCallReturn =
+ LastMI ? AArch64InstrInfo::isTailCallReturnInst(*LastMI) : false;
int64_t ArgumentPopSize = 0;
if (IsTailCallReturn) {
- MachineOperand &StackAdjust = MBBI->getOperand(1);
+ const MachineOperand &StackAdjust = LastMI->getOperand(1);
// For a tail-call in a callee-pops-arguments environment, some or all of
// the stack may actually be in use for the call's arguments, this is
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
index 5a148867cc2983..cbaba979575b0f 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
@@ -34626,31 +34626,51 @@ AArch64TargetLowering::EmitKCFICheck(MachineBasicBlock &MBB,
.getInstr();
}
+bool AArch64TargetLowering::supportKCFIOutOfLineTraps(
+ const MachineFunction &MF) const {
+ const Function &F = MF.getFunction();
+ // Keep checks bundled inline when SpeculativeLoadHardening (uses X16 as its
+ // taint register) or XRay (PATCHABLE_TAIL_CALL precedes tail calls) is used.
+ if (F.hasFnAttribute(Attribute::SpeculativeLoadHardening) ||
+ F.hasFnAttribute("function-instrument") ||
+ F.hasFnAttribute("xray-instruction-threshold"))
+ return false;
+ // TODO: Support -fbasic-block-sections by keeping KCFI_CMP, its fallthrough
+ // call block, and KCFI_TRAP in the same section so cross-section relaxation
+ // cannot insert veneers or spill X16.
+ if (MF.hasBBSections() ||
+ MF.getTarget().getBBSectionsType() != BasicBlockSection::None)
+ return false;
+ // TODO: Support callee-pop stack-argument conventions by teaching
+ // AArch64FrameLowering::getArgumentStackToRestore to look past KCFI_CMP to
+ // the split tail-call return.
+ const auto *AFI = MF.getInfo<AArch64FunctionInfo>();
+ return AFI->getArgumentStackToRestore() == 0 &&
+ AFI->getTailCallReservedStack() == 0;
+}
+
MachineInstr *
AArch64TargetLowering::EmitKCFIOutOfLineTrap(MachineInstr &Check,
MachineBasicBlock &TrapMBB,
const TargetInstrInfo *TII) const {
assert(Check.getOpcode() == AArch64::KCFI_CHECK && "Unexpected KCFI check");
- const MIMetadata MIMD(Check);
+ MIMetadata MIMD(Check);
Check.setDesc(TII->get(AArch64::KCFI_CMP));
- // The trap reports the registers that the check leaves the target and the
- // expected type in. AArch64AsmPrinter::LowerKCFI_CHECK picks them: the
- // expected type is in W17, or in W9 if the target is in X17, and a target of
- // XZR is in X16. They're live into the trap's block, so that a branch to it
- // that has to be relaxed doesn't clobber them.
- Register AddrReg = Check.getOperand(0).getReg();
- const Register TypeReg = AddrReg == AArch64::X17 ? AArch64::W9 : AArch64::W17;
- if (AddrReg == AArch64::XZR)
- AddrReg = AArch64::X16;
- TrapMBB.addLiveIn(AddrReg);
- TrapMBB.addLiveIn(getXRegFromWReg(TypeReg));
+ // Encode the registers where AArch64::getKCFICheckRegs leaves the target and
+ // expected type in KCFI_TRAP's BRK immediate. Keep TrapMBB out of cold
+ // sections via isMBBSafeToSplitToCold, and mark the operands undef rather
+ // than live-in so intra-section branch relaxation (>1 MiB) does not spill
+ // X16, with the trade-off that such a relaxed veneer could clobber X16 before
+ // BRK executes.
+ AArch64::KCFICheckRegs Regs =
+ AArch64::getKCFICheckRegs(Check.getOperand(0).getReg());
// Don't let the traps get merged, so that the address of one still says
// which check failed, as it does when they're inline.
BuildMI(&TrapMBB, MIMD, TII->get(AArch64::KCFI_TRAP))
- .addReg(AddrReg)
- .addReg(TypeReg)
+ .addReg(Regs.AddrReg, RegState::Undef)
+ .addReg(Regs.ExpectedTypeReg, RegState::Undef)
.setMIFlag(MachineInstr::NoMerge);
return BuildMI(*Check.getParent(), std::next(Check.getIterator()), MIMD,
TII->get(AArch64::Bcc))
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.h b/llvm/lib/Target/AArch64/AArch64ISelLowering.h
index 09a278445cdf88..fd65941d6d15c9 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.h
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.h
@@ -519,7 +519,7 @@ class AArch64TargetLowering : public TargetLowering {
MachineBasicBlock::instr_iterator &MBBI,
const TargetInstrInfo *TII) const override;
- bool supportKCFIOutOfLineTraps() const override { return true; }
+ bool supportKCFIOutOfLineTraps(const MachineFunction &MF) const override;
MachineInstr *
EmitKCFIOutOfLineTrap(MachineInstr &Check, MachineBasicBlock &TrapMBB,
diff --git a/llvm/lib/Target/AArch64/AArch64InstrInfo.cpp b/llvm/lib/Target/AArch64/AArch64InstrInfo.cpp
index 2371254f4f108f..014afcc64cb5b1 100644
--- a/llvm/lib/Target/AArch64/AArch64InstrInfo.cpp
+++ b/llvm/lib/Target/AArch64/AArch64InstrInfo.cpp
@@ -1649,6 +1649,9 @@ bool AArch64InstrInfo::isSchedulingBoundary(const MachineInstr &MI,
return true;
switch (MI.getOpcode()) {
+ case AArch64::KCFI_CMP:
+ // Keep KCFI_CMP immediately before its conditional branch terminator.
+ return true;
case AArch64::PAUTH_EPILOGUE:
// Keep authentication after the other epilogue instructions and close to
// the return, allowing them to combine into an authenticated return.
@@ -11321,9 +11324,15 @@ AArch64InstrInfo::getOutliningTypeImpl(const MachineModuleInfo &MMI,
unsigned Flags) const {
MachineInstr &MI = *MIT;
- // Don't outline anything used for return address signing. The outlined
- // function will get signed later if needed
+ // Don't outline anything used for return address signing or KCFI
+ // checks/traps. Signed functions will get signed later if needed, whereas
+ // outlining a KCFI check loses patchable-function-prefix from its caller (or
+ // risks linker veneers clobbering X16/X17/NZCV), and outlining a trap merges
+ // distinct check failure sites.
switch (MI.getOpcode()) {
+ case AArch64::KCFI_CHECK:
+ case AArch64::KCFI_CMP:
+ case AArch64::KCFI_TRAP:
case AArch64::PACM:
case AArch64::PACIASP:
case AArch64::PACIBSP:
@@ -11908,6 +11917,13 @@ bool AArch64InstrInfo::isFunctionSafeToSplit(const MachineFunction &MF) const {
return TargetInstrInfo::isFunctionSafeToSplit(MF);
}
+static bool endsWithKCFICmp(const MachineBasicBlock &MBB) {
+ MachineBasicBlock::const_iterator FirstTerm = MBB.getFirstTerminator();
+ if (FirstTerm == MBB.begin() || FirstTerm == MBB.end())
+ return false;
+ return prev_nodbg(FirstTerm, MBB.begin())->getOpcode() == AArch64::KCFI_CMP;
+}
+
bool AArch64InstrInfo::isMBBSafeToSplitToCold(
const MachineBasicBlock &MBB) const {
// Asm Goto blocks can contain conditional branches to goto labels, which can
@@ -11918,6 +11934,15 @@ bool AArch64InstrInfo::isMBBSafeToSplitToCold(
if (llvm::any_of(MBB, isAsmGoto) || MBB.isInlineAsmBrIndirectTarget())
return false;
+ // Keep KCFI checks, split call blocks, and trap blocks together out of cold
+ // sections so cold splitting never separates KCFI_CMP from its trap or
+ // fallthrough call (avoiding cross-section veneers/spills of X16/X17).
+ MachineBasicBlock::const_iterator LastI = MBB.getLastNonDebugInstr();
+ if ((LastI != MBB.end() && LastI->getOpcode() == AArch64::KCFI_TRAP) ||
+ endsWithKCFICmp(MBB) ||
+ (MBB.pred_size() == 1 && endsWithKCFICmp(**MBB.pred_begin())))
+ return false;
+
// Because jump tables are label-relative instead of table-relative, they all
// must be in the same section or relocation fixup handling will fail.
@@ -11937,11 +11962,6 @@ bool AArch64InstrInfo::isMBBSafeToSplitToCold(
case AArch64::JumpTableDest16:
case AArch64::JumpTableDest8:
return false;
- // A conditional branch can't go to another section, so splitting the trap
- // of a KCFI check from its check would have every check that passes take
- // a branch over an unconditional one, to save one cold instruction.
- case AArch64::KCFI_TRAP:
- return false;
default:
continue;
}
diff --git a/llvm/lib/Target/AArch64/AArch64InstrInfo.h b/llvm/lib/Target/AArch64/AArch64InstrInfo.h
index 8f34c3b6f48532..00f974db23f674 100644
--- a/llvm/lib/Target/AArch64/AArch64InstrInfo.h
+++ b/llvm/lib/Target/AArch64/AArch64InstrInfo.h
@@ -933,7 +933,23 @@ int32_t getSVERevInstr(uint32_t Opcode);
int32_t getSVENonRevInstr(uint32_t Opcode);
int32_t getSMEPseudoMap(uint32_t Opcode);
+
+struct KCFICheckRegs {
+ Register AddrReg;
+ Register TargetHashReg;
+ Register ExpectedTypeReg;
+};
+
+inline KCFICheckRegs getKCFICheckRegs(Register TargetReg) {
+ if (TargetReg == AArch64::XZR)
+ return {AArch64::X16, AArch64::W16, AArch64::W17};
+ return {
+ TargetReg,
+ TargetReg == AArch64::X16 ? AArch64::W9 : AArch64::W16,
+ TargetReg == AArch64::X17 ? AArch64::W9 : AArch64::W17,
+ };
}
+} // namespace AArch64
} // end namespace llvm
diff --git a/llvm/lib/Target/AArch64/AArch64InstrInfo.td b/llvm/lib/Target/AArch64/AArch64InstrInfo.td
index c53936e4be6854..d588b6a01c0930 100644
--- a/llvm/lib/Target/AArch64/AArch64InstrInfo.td
+++ b/llvm/lib/Target/AArch64/AArch64InstrInfo.td
@@ -2593,9 +2593,9 @@ def KCFI_CHECK : Pseudo<
(outs), (ins GPR64:$ptr, i32imm:$type), []>, Sched<[]>;
}
-// KCFI_CHECK without the trap: it only sets Z if the type of $ptr is $type.
+// KCFI_CHECK without the trap: it sets Z if the type of $ptr is $type.
// This gets lowered into a 16-byte instruction sequence.
-let Defs = [ X9, X16, X17, NZCV ], Size = 16 in {
+let Defs = [ X9, X16, X17, NZCV ], Size = 16, isNotDuplicable = 1 in {
def KCFI_CMP : Pseudo<
(outs), (ins GPR64:$ptr, i32imm:$type), []>, Sched<[]>;
}
@@ -2603,7 +2603,7 @@ def KCFI_CMP : Pseudo<
// The trap that the branch after a KCFI_CMP goes to if Z isn't set. It's a BRK
// that says that the KCFI_CMP left the target of the call in $ptr and the
// expected type in $type.
-let isTrap = 1, hasSideEffects = 1, Size = 4 in {
+let isTrap = 1, hasSideEffects = 1, Size = 4, isNotDuplicable = 1 in {
def KCFI_TRAP : Pseudo<
(outs), (ins GPR64:$ptr, GPR32:$type), []>, Sched<[]>;
}
diff --git a/llvm/lib/Target/AArch64/AArch64PointerAuth.cpp b/llvm/lib/Target/AArch64/AArch64PointerAuth.cpp
index f449b28c31ee3d..6abfb8c7fb4a14 100644
--- a/llvm/lib/Target/AArch64/AArch64PointerAuth.cpp
+++ b/llvm/lib/Target/AArch64/AArch64PointerAuth.cpp
@@ -295,7 +295,7 @@ void AArch64PointerAuthImpl::authenticateLR(
// Note that inserting new instructions "before MBBI" and "before TI" is
// not the same because if ShadowCallStack is enabled, its instructions
// are placed between MBBI and TI.
- MachineBasicBlock::iterator TI = MBB.getFirstInstrTerminator();
+ MachineBasicBlock::iterator TI = MBB.getFirstTerminator();
MCSymbol *PACSym = MFnI->getSigningInstrLabel();
auto &AFL = *static_cast<const AArch64FrameLowering *>(
diff --git a/llvm/lib/Target/X86/X86AsmPrinter.h b/llvm/lib/Target/X86/X86AsmPrinter.h
index a1dd9a63599cfd..8920f5931c84ca 100644
--- a/llvm/lib/Target/X86/X86AsmPrinter.h
+++ b/llvm/lib/Target/X86/X86AsmPrinter.h
@@ -129,6 +129,7 @@ class LLVM_LIBRARY_VISIBILITY X86AsmPrinter : public AsmPrinter {
void EmitKCFITypePadding(const MachineFunction &MF, bool HasType = true);
void LowerKCFI_CHECK(const MachineInstr &MI);
void LowerKCFI_TRAP(const MachineInstr &MI);
+ void emitKCFITrap(const MachineFunction &MF);
// Address sanitizer specific lowering for X86.
void LowerASAN_CHECK_MEMACCESS(const MachineInstr &MI);
diff --git a/llvm/lib/Target/X86/X86ISelLowering.cpp b/llvm/lib/Target/X86/X86ISelLowering.cpp
index 03734d362a9e65..9a03ed86282e76 100644
--- a/llvm/lib/Target/X86/X86ISelLowering.cpp
+++ b/llvm/lib/Target/X86/X86ISelLowering.cpp
@@ -65945,12 +65945,17 @@ X86TargetLowering::EmitKCFICheck(MachineBasicBlock &MBB,
.getInstr();
}
+bool X86TargetLowering::supportKCFIOutOfLineTraps(
+ const MachineFunction & /*MF*/) const {
+ return Subtarget.is64Bit();
+}
+
MachineInstr *
X86TargetLowering::EmitKCFIOutOfLineTrap(MachineInstr &Check,
MachineBasicBlock &TrapMBB,
const TargetInstrInfo *TII) const {
assert(Check.getOpcode() == X86::KCFI_CHECK && "Unexpected KCFI check");
- const MIMetadata MIMD(Check);
+ MIMetadata MIMD(Check);
Check.setDesc(TII->get(X86::KCFI_CMP));
// Don't let the traps get merged, so that the address of one still says
// which check failed, as it does when they're inline.
diff --git a/llvm/lib/Target/X86/X86ISelLowering.h b/llvm/lib/Target/X86/X86ISelLowering.h
index 7b313ce4473a17..e04075de278c49 100644
--- a/llvm/lib/Target/X86/X86ISelLowering.h
+++ b/llvm/lib/Target/X86/X86ISelLowering.h
@@ -694,7 +694,7 @@ namespace llvm {
MachineBasicBlock::instr_iterator &MBBI,
const TargetInstrInfo *TII) const override;
- bool supportKCFIOutOfLineTraps() const override { return true; }
+ bool supportKCFIOutOfLineTraps(const MachineFunction &MF) const override;
MachineInstr *
EmitKCFIOutOfLineTrap(MachineInstr &Check, MachineBasicBlock &TrapMBB,
diff --git a/llvm/lib/Target/X86/X86InstrCompiler.td b/llvm/lib/Target/X86/X86InstrCompiler.td
index 2b509f6325d8ad..9ef3da06473e5b 100644
--- a/llvm/lib/Target/X86/X86InstrCompiler.td
+++ b/llvm/lib/Target/X86/X86InstrCompiler.td
@@ -281,13 +281,14 @@ let
Defs = [R10, R11, EFLAGS] in {
def KCFI_CHECK : PseudoI<
(outs), (ins GR64:$ptr, i32imm:$type), []>, Sched<[]>;
-// KCFI_CHECK without the trap: it only sets ZF if the type of $ptr is $type.
+// KCFI_CHECK without the trap: it sets ZF if the type of $ptr is $type.
+let isNotDuplicable = 1 in
def KCFI_CMP : PseudoI<
(outs), (ins GR64:$ptr, i32imm:$type), []>, Sched<[]>;
}
// The trap that the branch after a KCFI_CMP goes to if ZF isn't set.
-let isTrap = 1, hasSideEffects = 1 in
+let isTrap = 1, hasSideEffects = 1, isNotDuplicable = 1 in
def KCFI_TRAP : PseudoI<(outs), (ins), []>, Sched<[]>;
//===----------------------------------------------------------------------===//
diff --git a/llvm/lib/Target/X86/X86InstrInfo.cpp b/llvm/lib/Target/X86/X86InstrInfo.cpp
index 6352fd87238229..dbd7e7e7578ceb 100644
--- a/llvm/lib/Target/X86/X86InstrInfo.cpp
+++ b/llvm/lib/Target/X86/X86InstrInfo.cpp
@@ -9289,7 +9289,7 @@ bool X86InstrInfo::isSchedulingBoundary(const MachineInstr &MI,
// ENDBR instructions should not be scheduled around.
unsigned Opcode = MI.getOpcode();
if (Opcode == X86::ENDBR64 || Opcode == X86::ENDBR32 ||
- Opcode == X86::PLDTILECFGV)
+ Opcode == X86::PLDTILECFGV || Opcode == X86::KCFI_CMP)
return true;
// Frame setup and destroy can't be scheduled around.
@@ -10794,6 +10794,13 @@ X86InstrInfo::getOutliningTypeImpl(const MachineModuleInfo &MMI,
unsigned Flags) const {
MachineInstr &MI = *MIT;
+ // Don't outline KCFI checks or traps: LowerKCFI_CHECK reads
+ // patchable-function-prefix from the parent function, and each trap must
+ // remain a distinct site in its caller's section.
+ if (MI.getOpcode() == X86::KCFI_CHECK || MI.getOpcode() == X86::KCFI_CMP ||
+ MI.getOpcode() == X86::KCFI_TRAP)
+ return outliner::InstrType::Illegal;
+
// Is this a terminator for a basic block?
if (MI.isTerminator())
// TargetInstrInfo::getOutliningType has already filtered out anything
diff --git a/llvm/lib/Target/X86/X86MCInstLower.cpp b/llvm/lib/Target/X86/X86MCInstLower.cpp
index 58bf56550bea30..640af92194f641 100644
--- a/llvm/lib/Target/X86/X86MCInstLower.cpp
+++ b/llvm/lib/Target/X86/X86MCInstLower.cpp
@@ -916,9 +916,16 @@ void X86AsmPrinter::LowerFENTRY_CALL(const MachineInstr &MI,
void X86AsmPrinter::LowerKCFI_CHECK(const MachineInstr &MI) {
// KCFI_CMP is KCFI_CHECK without the trap: it's followed by the branch to
// its KCFI_TRAP, which falls through to the call.
- const bool IsCmp = MI.getOpcode() == X86::KCFI_CMP;
- assert((IsCmp || std::next(MI.getIterator())->isCall()) &&
- "KCFI_CHECK not followed by a call instruction");
+ bool IsCmp = MI.getOpcode() == X86::KCFI_CMP;
+ if (IsCmp) {
+ MachineBasicBlock::const_instr_iterator Next =
+ next_nodbg(MI.getIterator(), MI.getParent()->instr_end());
+ assert(Next != MI.getParent()->instr_end() && Next->isConditionalBranch() &&
+ "KCFI_CMP not followed by a conditional branch");
+ } else {
+ assert(std::next(MI.getIterator())->isCall() &&
+ "KCFI_CHECK not followed by a call instruction");
+ }
// Adjust the offset for patchable-function-prefix. X86InstrInfo::getNop()
// returns a 1-byte X86::NOOP, which means the offset is the same in
@@ -933,11 +940,11 @@ void X86AsmPrinter::LowerKCFI_CHECK(const MachineInstr &MI) {
// and thus emitting potential call target gadgets at each indirect call
// site, load a negated constant to a register and compare that to the
// expected value at the call target.
- const Register AddrReg = MI.getOperand(0).getReg();
- const uint32_t Type = MI.getOperand(1).getImm();
+ Register AddrReg = MI.getOperand(0).getReg();
+ uint32_t Type = MI.getOperand(1).getImm();
// The check is immediately before the call. If the call target is in R10,
// we can clobber R11 for the check instead.
- unsigned TempReg = AddrReg == X86::R10 ? X86::R11D : X86::R10D;
+ Register TempReg = AddrReg == X86::R10 ? X86::R11D : X86::R10D;
EmitAndCountInstruction(
MCInstBuilder(X86::MOV32ri).addReg(TempReg).addImm(-MaskKCFIType(Type)));
EmitAndCountInstruction(MCInstBuilder(X86::ADD32rm)
@@ -958,18 +965,19 @@ void X86AsmPrinter::LowerKCFI_CHECK(const MachineInstr &MI) {
.addExpr(MCSymbolRefExpr::create(Pass, OutContext))
.addImm(X86::COND_E));
+ emitKCFITrap(MF);
+ OutStreamer->emitLabel(Pass);
+}
+
+void X86AsmPrinter::emitKCFITrap(const MachineFunction &MF) {
MCSymbol *Trap = OutContext.createTempSymbol();
OutStreamer->emitLabel(Trap);
EmitAndCountInstruction(MCInstBuilder(X86::TRAP));
emitKCFITrapEntry(MF, Trap);
- OutStreamer->emitLabel(Pass);
}
void X86AsmPrinter::LowerKCFI_TRAP(const MachineInstr &MI) {
- MCSymbol *Trap = OutContext.createTempSymbol();
- OutStreamer->emitLabel(Trap);
- EmitAndCountInstruction(MCInstBuilder(X86::TRAP));
- emitKCFITrapEntry(*MI.getMF(), Trap);
+ emitKCFITrap(*MI.getMF());
}
void X86AsmPrinter::LowerASAN_CHECK_MEMACCESS(const MachineInstr &MI) {
@@ -2574,8 +2582,6 @@ void X86AsmPrinter::emitInstruction(const MachineInstr *MI) {
return;
case X86::KCFI_CHECK:
- return LowerKCFI_CHECK(*MI);
-
case X86::KCFI_CMP:
return LowerKCFI_CHECK(*MI);
diff --git a/llvm/test/CodeGen/AArch64/kcfi-out-of-line-traps-xzr.mir b/llvm/test/CodeGen/AArch64/kcfi-out-of-line-traps-xzr.mir
index ee71526d3a4a82..c9891ca22190c4 100644
--- a/llvm/test/CodeGen/AArch64/kcfi-out-of-line-traps-xzr.mir
+++ b/llvm/test/CodeGen/AArch64/kcfi-out-of-line-traps-xzr.mir
@@ -1,9 +1,10 @@
# RUN: llc -mtriple=aarch64-- -start-before=kcfi -verify-machineinstrs %s -o - | FileCheck %s --check-prefix=ASM
-# RUN: llc -mtriple=aarch64-- -run-pass=kcfi -verify-machineinstrs %s -o - | FileCheck %s --check-prefix=MIR
+# RUN: llc -mtriple=aarch64-- -run-pass=kcfi -verify-machineinstrs %s -o - | llc -mtriple=aarch64-- -x mir -run-pass=none -verify-machineinstrs -o - | FileCheck %s --check-prefix=MIR
-## A check of a call to xzr compares the expected type with zero, which it has
-## in x16. The out-of-line trap says that the target is in x16, as an inline
-## one does.
+## A check of a call to xzr compares the expected type with zero, which it
+## moves into x16. The out-of-line trap encodes x16 in its BRK immediate without
+## marking x16 live-in to the trap block, and nomerge KCFI_TRAP round-trips
+## cleanly through MIR parsing.
# ASM-LABEL: f:
# ASM: mov x16, xzr
@@ -19,12 +20,12 @@
# MIR-LABEL: name: f
# MIR: KCFI_CMP $xzr, 12345678, implicit-def $x9, implicit-def $x16, implicit-def $x17, implicit-def $nzcv
-# MIR-NEXT: Bcc 1, %bb.1, implicit $nzcv
-# MIR: bb.2:
-# MIR-NEXT: BLR $xzr, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
-# MIR: bb.1:
-# MIR-NEXT: liveins: $x16, $x17
-# MIR: nomerge KCFI_TRAP $x16, $w17
+# MIR-NEXT: Bcc 1, %[[TRAP:bb\.[0-9]+]], implicit $nzcv
+# MIR: bb.{{[0-9]+}}:
+# MIR: BLR $xzr, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
+# MIR: [[TRAP]]:
+# MIR-NOT: liveins:
+# MIR: nomerge KCFI_TRAP undef $x16, undef $w17
--- |
define void @f() {
@@ -33,7 +34,7 @@
!llvm.module.flags = !{!0, !1}
!0 = !{i32 4, !"kcfi", i32 1}
- !1 = !{i32 4, !"kcfi-out-of-line-traps", i32 1}
+ !1 = !{i32 8, !"kcfi-out-of-line-traps", i32 1}
...
---
name: f
diff --git a/llvm/test/CodeGen/AArch64/kcfi-out-of-line-traps.ll b/llvm/test/CodeGen/AArch64/kcfi-out-of-line-traps.ll
index 097537a6663bd4..04f6fb1b9a7e41 100644
--- a/llvm/test/CodeGen/AArch64/kcfi-out-of-line-traps.ll
+++ b/llvm/test/CodeGen/AArch64/kcfi-out-of-line-traps.ll
@@ -1,4 +1,5 @@
-; RUN: llc -mtriple=aarch64-- -verify-machineinstrs < %s | FileCheck %s --check-prefix=ASM
+; RUN: llc -mtriple=aarch64-- -verify-machineinstrs < %s | FileCheck %s --check-prefixes=ASM,OUTLINED
+; RUN: llc -mtriple=aarch64-- -mcpu=cortex-a53 -enable-post-misched -verify-machineinstrs < %s | FileCheck %s --check-prefix=ASM
; RUN: llc -mtriple=aarch64-- -verify-machineinstrs -stop-after=kcfi < %s | FileCheck %s --check-prefix=MIR
; RUN: llc -mtriple=aarch64-- -verify-machineinstrs -function-splitting=all < %s | FileCheck %s --check-prefix=SPLIT
; RUN: llc -mtriple=aarch64-- -verify-machineinstrs -basic-block-sections=all < %s | FileCheck %s --check-prefix=BBS
@@ -30,8 +31,8 @@ define void @f1(ptr noundef %x) {
; MIR-NEXT: liveins: {{.*}}$x0
; MIR: BLR killed $x0, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
; MIR: bb.1 (%ir-block.0):
-; MIR-NEXT: liveins: $x0, $x17
-; MIR: nomerge KCFI_TRAP $x0, $w17
+; MIR-NOT: liveins:
+; MIR: nomerge KCFI_TRAP undef $x0, undef $w17
call void %x() [ "kcfi"(i32 12345678) ]
ret void
}
@@ -51,8 +52,8 @@ define void @f2(ptr noundef %x) {
; MIR: bb.2 (%ir-block.0):
; MIR: TCRETURNri killed $x0, 0, csr_aarch64_aapcs, implicit $sp
; MIR: bb.1 (%ir-block.0):
-; MIR-NEXT: liveins: $x0, $x17
-; MIR: nomerge KCFI_TRAP $x0, $w17
+; MIR-NOT: liveins:
+; MIR: nomerge KCFI_TRAP undef $x0, undef $w17
tail call void %x() [ "kcfi"(i32 12345678) ]
ret void
}
@@ -78,23 +79,21 @@ define void @f3(ptr noundef %x) #1 {
; MIR-NEXT: liveins: {{.*}}$x16
; MIR: TCRETURNrix16x17 killed $x16, 0, csr_aarch64_aapcs, implicit $sp
; MIR: bb.1 (%ir-block.0):
-; MIR-NEXT: liveins: $x16, $x17
-; MIR: nomerge KCFI_TRAP $x16, $w17
+; MIR-NOT: liveins:
+; MIR: nomerge KCFI_TRAP undef $x16, undef $w17
-;; The registers that the trap reports are live into its block. A branch to
-;; another section can get a veneer from the linker, which clobbers x16, so
-;; one to the trap saves x16 if the target is in it.
+;; With basic block sections enabled on AArch64, KCFI falls back to inline
+;; bundled checks so x16 never spills or routes through a veneer.
; BBS-LABEL: f3:
+; BBS-NOT: {{str|stp|ldr|ldp}} {{.*}}x16
+; BBS-NOT: f3.__part
+; BBS: ldur w9, [x16, #-4]
; BBS: cmp w9, w17
-; BBS-NEXT: b.eq [[PASS:.LBB[0-9_]+]]
-; BBS-NEXT: // %bb.{{[0-9]+}}:
-; BBS-NEXT: str x16, [sp, #-16]!
-; BBS-NEXT: b [[TRAP:f3.__part.[0-9]+]]
-; BBS: [[TRAP]]:
-; BBS-NEXT: .cfi_startproc
-; BBS-NEXT: ldr x16, [sp], #16
-; BBS-NEXT: // %bb.{{[0-9]+}}:
+; BBS-NEXT: b.eq [[PASS:\.Ltmp[0-9]+]]
; BBS-NEXT: brk #0x8230
+; BBS-NEXT: [[PASS]]:
+; BBS-NEXT: br x16
+; BBS: .size f3,
tail call void %x() [ "kcfi"(i32 12345678) ]
ret void
}
@@ -118,8 +117,8 @@ define void @f4(ptr noundef %x) #2 {
; MIR-NEXT: liveins: {{.*}}$x17
; MIR: TCRETURNrix17 killed $x17, 0, csr_aarch64_aapcs, implicit $sp
; MIR: bb.1 (%ir-block.0):
-; MIR-NEXT: liveins: $x17, $x9
-; MIR: nomerge KCFI_TRAP $x17, $w9
+; MIR-NOT: liveins:
+; MIR: nomerge KCFI_TRAP undef $x17, undef $w9
tail call void %x() [ "kcfi"(i32 12345678) ]
ret void
}
@@ -157,9 +156,11 @@ define void @f5(ptr noundef %x, ptr noundef %y) {
; MIR: bb.2 (%ir-block.0):
; MIR: BLR killed $x19, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
; MIR: bb.1 (%ir-block.0):
-; MIR: nomerge KCFI_TRAP $x19, $w17
+; MIR-NOT: liveins:
+; MIR: nomerge KCFI_TRAP undef $x19, undef $w17
; MIR: bb.3 (%ir-block.0):
-; MIR: nomerge KCFI_TRAP $x0, $w17
+; MIR-NOT: liveins:
+; MIR: nomerge KCFI_TRAP undef $x0, undef $w17
call void %x() [ "kcfi"(i32 12345678) ]
call void %y() [ "kcfi"(i32 12345678) ]
ret void
@@ -189,22 +190,25 @@ define void @f6(ptr noundef %x) #1 {
declare i32 @__gxx_personality_v0(...)
declare void @cleanup()
-;; A call in a block that has an EH pad successor can be an invoke, which has
-;; to stay next to the EH label that starts its try-range. It keeps an inline
-;; trap.
+;; A non-invoke call preceding another call/invoke in a block with an EH pad
+;; successor keeps an inline bundled check.
define void @f7(ptr noundef %x) personality ptr @__gxx_personality_v0 {
; ASM-LABEL: f7:
; ASM: cmp w16, w17
-; ASM-NEXT: b.eq [[PASS:.Ltmp[0-9]+]]
+; ASM-NEXT: b.eq [[PASS:\.Ltmp[0-9]+]]
; ASM-NEXT: brk #0x8220
; ASM-NEXT: [[PASS]]:
; ASM-NEXT: blr x0
+; ASM-NEXT: .Ltmp{{[0-9]+}}:
+; ASM-NEXT: bl cleanup
; MIR-LABEL: name: f7
; MIR: BUNDLE{{.*}} {
; MIR-NEXT: KCFI_CHECK $x0, 12345678, implicit-def $x9, implicit-def $x16, implicit-def $x17, implicit-def $nzcv
; MIR-NEXT: BLR killed $x0, csr_aarch64_aapcs, implicit-def dead $lr, implicit $sp, implicit-def $sp
; MIR-NEXT: }
+; MIR-NEXT: EH_LABEL
+; MIR-NEXT: BL @cleanup
; MIR-NEXT: EH_LABEL
call void %x() [ "kcfi"(i32 12345678) ]
invoke void @cleanup()
@@ -260,13 +264,160 @@ define void @f9(ptr noundef %x) uwtable(async) {
ret void
}
+;; When the block containing the check is cold, KCFI_CMP, its call block, and
+;; its trap stay out of the cold section so machine function splitting never
+;; splits across the conditional branch or fallthrough call.
+define void @f10(ptr noundef %x, i1 %c1, i1 %c2) !prof !17 {
+; SPLIT-LABEL: f10:
+; SPLIT-NOT: .section .text.split.f10
+; SPLIT: cmp w16, w17
+; SPLIT-NEXT: b.ne [[TRAP:\.LBB[0-9_]+]]
+; SPLIT: blr x0
+; SPLIT: [[TRAP]]:
+; SPLIT-NEXT: brk #0x8220
+; SPLIT: .section .text.split.f10,"ax", at progbits
+; SPLIT-NEXT: f10.cold:
+; SPLIT: bl cleanup
+; SPLIT-NOT: brk
+; SPLIT: .size f10.cold
+entry:
+ br i1 %c1, label %cold, label %check2, !prof !18
+check2:
+ br i1 %c2, label %cold2, label %hot, !prof !18
+cold:
+ call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+cold2:
+ call void @cleanup()
+ ret void
+hot:
+ ret void
+}
+
+;; KCFI_CMP and KCFI_TRAP are never outlined by the machine outliner, preserving
+;; patchable-function-prefix offsets, X16/X17/NZCV liveness, and distinct trap
+;; sites.
+define void @f11(ptr noundef %x, ptr %p) minsize "patchable-function-prefix"="2" {
+; OUTLINED-LABEL: f11:
+; OUTLINED: bl OUTLINED_FUNCTION_0
+; OUTLINED-NEXT: ldur w16, [x0, #-12]
+; OUTLINED: b.ne [[TRAP:.LBB[0-9_]+]]
+; OUTLINED: [[TRAP]]:
+; OUTLINED-NEXT: brk #0x8220
+ store i32 1, ptr %p, align 4
+ %p1 = getelementptr i32, ptr %p, i64 1
+ store i32 2, ptr %p1, align 4
+ %p2 = getelementptr i32, ptr %p, i64 2
+ store i32 3, ptr %p2, align 4
+ call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+define void @f12(ptr noundef %x, ptr %p) minsize "patchable-function-prefix"="2" {
+; OUTLINED-LABEL: f12:
+; OUTLINED: bl OUTLINED_FUNCTION_0
+; OUTLINED-NEXT: ldur w16, [x0, #-12]
+; OUTLINED: b.ne [[TRAP:.LBB[0-9_]+]]
+; OUTLINED: [[TRAP]]:
+; OUTLINED-NEXT: brk #0x8220
+ store i32 1, ptr %p, align 4
+ %p1 = getelementptr i32, ptr %p, i64 1
+ store i32 2, ptr %p1, align 4
+ %p2 = getelementptr i32, ptr %p, i64 2
+ store i32 3, ptr %p2, align 4
+ call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+;; Functions with callee-pop stack arguments fall back to inline KCFI checks.
+define tailcc void @f13(ptr noundef %x, i64, i64, i64, i64, i64, i64, i64, i64) {
+; ASM-LABEL: f13:
+; ASM: ldur w16, [x0, #-4]
+; ASM: cmp w16, w17
+; ASM-NEXT: b.eq [[PASS:\.Ltmp[0-9]+]]
+; ASM-NEXT: brk #0x8220
+; ASM-NEXT: [[PASS]]:
+; ASM-NEXT: br x0
+ call void @cleanup()
+ tail call tailcc void %x(ptr %x, i64 0, i64 0, i64 0, i64 0, i64 0, i64 0, i64 0, i64 0) [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+;; Functions with Speculative Load Hardening fall back to inline KCFI checks so
+;; KCFI_CMP does not interfere with X16 taint tracking or degrade hardening to
+;; full barriers.
+define void @f14(ptr noundef %x) speculative_load_hardening {
+; ASM-LABEL: f14:
+; ASM-NOT: dsb sy
+; ASM: cmp w16, w17
+; ASM-NEXT: b.eq [[PASS:.Ltmp[0-9]+]]
+; ASM-NEXT: brk #0x8220
+; ASM-NEXT: [[PASS]]:
+; ASM-NEXT: blr x0
+; ASM-NOT: dsb sy
+; ASM: ret
+ call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+;; Functions with XRay instrumentation attributes fall back to inline KCFI checks
+;; so PATCHABLE_TAIL_CALL does not clobber X16/W17 after a split check.
+define void @f15(ptr noundef %x, ptr noundef %y) "function-instrument"="xray-always" {
+; ASM-LABEL: f15:
+; ASM: cmp w16, w17
+; ASM-NEXT: b.eq [[PASS1:\.Ltmp[0-9]+]]
+; ASM-NEXT: brk #0x8220
+; ASM-NEXT: [[PASS1]]:
+; ASM-NEXT: blr x0
+; ASM: .Lxray_sled_{{[0-9]+}}:
+; ASM: ldur w16, [x0, #-4]
+; ASM: cmp w16, w17
+; ASM-NEXT: b.eq [[PASS2:\.Ltmp[0-9]+]]
+; ASM-NEXT: brk #0x8220
+; ASM-NEXT: [[PASS2]]:
+; ASM-NEXT: br x0
+ call void %x() [ "kcfi"(i32 12345678) ]
+ tail call void %y() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+;; Indirect invokes move their KCFI traps out of line on AArch64 while keeping
+;; their BeginLabel/EndLabel EH_LABEL pair around the call.
+define void @f16(ptr noundef %x) personality ptr @__gxx_personality_v0 {
+; ASM-LABEL: f16:
+; ASM: ldur w16, [x0, #-4]
+; ASM: cmp w16, w17
+; ASM-NEXT: b.ne [[TRAP:\.LBB[0-9_]+]]
+; ASM-NEXT: // %bb.{{[0-9]+}}:
+; ASM-NEXT: .Ltmp{{[0-9]+}}: // EH_LABEL
+; ASM-NEXT: blr x0
+; ASM-NEXT: .Ltmp{{[0-9]+}}: // EH_LABEL
+; ASM: [[TRAP]]:
+; ASM-NEXT: brk #0x8220
+ invoke void %x() [ "kcfi"(i32 12345678) ]
+ to label %cont
+ unwind label %err
+cont:
+ ret void
+err:
+ %exn = landingpad { ptr, i32 }
+ cleanup
+ call void @cleanup()
+ resume { ptr, i32 } %exn
+}
+
+; OUTLINED-LABEL: OUTLINED_FUNCTION_0:
+; OUTLINED-NOT: ldur
+; OUTLINED-NOT: brk
+; OUTLINED: ret
+
attributes #0 = { returns_twice }
attributes #1 = { "branch-target-enforcement" }
attributes #2 = { "branch-target-enforcement" "branch-protection-pauth-lr" "sign-return-address"="all" "sign-return-address-key"="a_key" }
!llvm.module.flags = !{!0, !1, !2, !3}
!0 = !{i32 4, !"kcfi", i32 1}
-!1 = !{i32 4, !"kcfi-out-of-line-traps", i32 1}
+!1 = !{i32 8, !"kcfi-out-of-line-traps", i32 1}
!2 = !{i32 8, !"branch-target-enforcement", i32 1}
!3 = !{i32 1, !"ProfileSummary", !4}
!4 = !{!5, !6, !7, !8, !9, !10, !11, !12}
diff --git a/llvm/test/CodeGen/X86/kcfi-out-of-line-traps-invoke.ll b/llvm/test/CodeGen/X86/kcfi-out-of-line-traps-invoke.ll
new file mode 100644
index 00000000000000..dba38e8cd76536
--- /dev/null
+++ b/llvm/test/CodeGen/X86/kcfi-out-of-line-traps-invoke.ll
@@ -0,0 +1,284 @@
+; RUN: split-file %s %t
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -verify-machineinstrs < %t/base.ll | FileCheck %s --check-prefix=ASM
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -verify-machineinstrs -stop-after=kcfi < %t/base.ll | FileCheck %s --check-prefix=MIR
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -verify-machineinstrs -verify-cfiinstrs -function-sections -basic-block-sections=all -basic-block-address-map -o %t.bbs.s < %t/base.ll
+; RUN: FileCheck %s --check-prefix=BBS < %t.bbs.s
+; RUN: llvm-mc -triple=x86_64-unknown-linux-gnu -filetype=obj -o /dev/null < %t.bbs.s
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -verify-machineinstrs -verify-cfiinstrs -function-sections -basic-block-sections=%t/prof -basic-block-address-map -o %t.prop.s 2> %t.prop.err < %t/base.ll
+; RUN: FileCheck %s --check-prefix=PROP < %t.prop.s
+; RUN: FileCheck %s --check-prefix=PROP-WARN < %t.prop.err
+; RUN: llvm-mc -triple=x86_64-unknown-linux-gnu -filetype=obj -o /dev/null < %t.prop.s
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -emulated-tls -verify-machineinstrs -verify-cfiinstrs -function-sections -basic-block-sections=all -o %t.emutls.s < %t/emutls.ll
+; RUN: FileCheck %s --check-prefix=EMUTLS < %t.emutls.s
+; RUN: llvm-mc -triple=x86_64-unknown-linux-gnu -filetype=obj -o /dev/null < %t.emutls.s
+
+;; Indirect invokes move their KCFI traps out of line, keeping the invoke's
+;; BeginLabel/EndLabel EH_LABEL pair and DW_CFA_GNU_args_size instruction with
+;; the call in the split block so LSDA call-site ranges and FDEs stay valid
+;; when basic block sections are enabled.
+
+;--- base.ll
+declare i32 @__gxx_personality_v0(...)
+declare void @cleanup()
+declare void @cleanup_8args(i64, i64, i64, i64, i64, i64, i64, i64)
+declare void @side(i32)
+
+define void @f1(ptr noundef %x) personality ptr @__gxx_personality_v0 {
+; ASM-LABEL: f1:
+; ASM: movl $4282621618, %r10d # imm = 0xFF439EB2
+; ASM-NEXT: addl -4(%rdi), %r10d
+; ASM-NEXT: jne [[TRAP:.LBB[0-9_]+]]
+; ASM-NEXT: # %bb.{{[0-9]+}}:
+; ASM-NEXT: .Ltmp{{[0-9]+}}: # EH_LABEL
+; ASM-NEXT: callq *%rdi
+; ASM-NEXT: .Ltmp{{[0-9]+}}: # EH_LABEL
+; ASM: [[TRAP]]:
+; ASM-NEXT: .cfi_def_cfa_offset 16
+; ASM-NEXT: .Ltmp{{[0-9]+}}:
+; ASM-NEXT: ud2
+
+; MIR-LABEL: name: f1
+; MIR: bb.0 (%ir-block.0):
+; MIR-NEXT: successors: %bb.4(0x80000000), %bb.3(0x00000000)
+; MIR: KCFI_CMP $rdi, 12345678, implicit-def $r10, implicit-def $r11, implicit-def $eflags
+; MIR-NEXT: JCC_1 %bb.3, 5, implicit $eflags
+; MIR: bb.4 (%ir-block.0):
+; MIR-NEXT: successors: %bb.1(0x7ffff800), %bb.2(0x00000800)
+; MIR-NEXT: liveins: {{.*}}$rdi
+; MIR: EH_LABEL
+; MIR-NEXT: CALL64r killed $rdi, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp
+; MIR-NEXT: EH_LABEL
+; MIR: bb.3 (%ir-block.0):
+; MIR-NEXT: nomerge KCFI_TRAP
+
+; BBS-LABEL: f1:
+; BBS: addl -4(%rdi), %r10d
+; BBS-NEXT: jne [[TRAP:f1\.__part\.[0-9]+]]
+; BBS-NEXT: jmp [[CALL:f1\.__part\.[0-9]+]]
+; BBS: [[CALL]]:
+; BBS: [[BEGIN:\.Ltmp[0-9]+]]: # EH_LABEL
+; BBS-NEXT: callq *%rdi
+; BBS-NEXT: .LBB{{[0-9]+}}_1_CS0:
+; BBS-NEXT: [[END:\.Ltmp[0-9]+]]: # EH_LABEL
+; BBS: [[TRAP]]:
+; BBS: ud2
+; BBS: [[ERR:f1\.__part\.[0-9]+]]: # %err
+; BBS: [[LPAD:\.Ltmp[0-9]+]]: # EH_LABEL
+; BBS: .section .gcc_except_table.f1
+; BBS: .uleb128 [[BEGIN]]-[[CALL]]
+; BBS-NEXT: .uleb128 [[END]]-[[BEGIN]]
+; BBS-NEXT: .uleb128 [[LPAD]]-[[ERR]]
+ invoke void %x() [ "kcfi"(i32 12345678) ]
+ to label %cont
+ unwind label %err
+cont:
+ ret void
+err:
+ %exn = landingpad { ptr, i32 }
+ cleanup
+ call void @cleanup()
+ resume { ptr, i32 } %exn
+}
+
+;; Stack-pushed call arguments carry DW_CFA_GNU_args_size (.cfi_escape 0x2e, 0x10)
+;; into the split invoke block's FDE when basic block sections are enabled.
+define void @f2(ptr noundef %x) personality ptr @__gxx_personality_v0 {
+; BBS-LABEL: f2:
+; BBS: pushq $8
+; BBS-NEXT: .cfi_adjust_cfa_offset 8
+; BBS-NEXT: pushq $7
+; BBS-NEXT: .cfi_adjust_cfa_offset 8
+; BBS-NEXT: movl $4282621618, %r10d # imm = 0xFF439EB2
+; BBS-NEXT: addl -4([[REG:%r[a-z0-9]+]]), %r10d
+; BBS-NEXT: jne [[TRAP:f2\.__part\.[0-9]+]]
+; BBS-NEXT: jmp [[CALL:f2\.__part\.[0-9]+]]
+; BBS: [[CALL]]:
+; BBS-NEXT: .cfi_startproc
+; BBS: .cfi_def_cfa %rsp, 32
+; BBS: [[BEGIN:\.Ltmp[0-9]+]]: # EH_LABEL
+; BBS-NEXT: .cfi_escape 0x2e, 0x10
+; BBS-NEXT: callq *[[REG]]
+; BBS-NEXT: .LBB{{[0-9]+}}_1_CS0:
+; BBS-NEXT: addq $16, %rsp
+; BBS-NEXT: .cfi_adjust_cfa_offset -16
+; BBS-NEXT: [[END:\.Ltmp[0-9]+]]: # EH_LABEL
+; BBS: [[TRAP]]:
+; BBS-NEXT: .cfi_startproc
+; BBS: .cfi_def_cfa %rsp, 32
+; BBS: ud2
+; BBS: .section .gcc_except_table.f2
+; BBS: .uleb128 [[BEGIN]]-[[CALL]]
+; BBS-NEXT: .uleb128 [[END]]-[[BEGIN]]
+ invoke void %x(i64 1, i64 2, i64 3, i64 4, i64 5, i64 6, i64 7, i64 8) [ "kcfi"(i32 12345678) ]
+ to label %cont
+ unwind label %err
+cont:
+ ret void
+err:
+ %exn = landingpad { ptr, i32 }
+ cleanup
+ call void @cleanup()
+ resume { ptr, i32 } %exn
+}
+
+;; Checked calls inside a landing pad keep inline traps so the landing pad
+;; EH_LABEL stays at the entry of its section without extra splitting.
+define void @f3(ptr noundef %x) personality ptr @__gxx_personality_v0 {
+; BBS-LABEL: f3:
+; BBS: [[BEGIN:\.Ltmp[0-9]+]]: # EH_LABEL
+; BBS: callq cleanup_8args at PLT
+; BBS: [[END:\.Ltmp[0-9]+]]: # EH_LABEL
+; BBS: [[ERR:f3\.__part\.[0-9]+]]: # %err
+; BBS: nop
+; BBS-NEXT: [[LPAD:\.Ltmp[0-9]+]]: # EH_LABEL
+; BBS: addl -4([[REG:%r[a-z0-9]+]]), %r10d
+; BBS-NEXT: je [[PASS:\.Ltmp[0-9]+]]
+; BBS: ud2
+; BBS: [[PASS]]:
+; BBS-NEXT: callq *[[REG]]
+; BBS: .section .gcc_except_table.f3
+; BBS: .uleb128 [[BEGIN]]-.Lfunc_begin
+; BBS-NEXT: .uleb128 [[END]]-[[BEGIN]]
+; BBS-NEXT: .uleb128 [[LPAD]]-[[ERR]]
+ invoke void @cleanup_8args(i64 1, i64 2, i64 3, i64 4, i64 5, i64 6, i64 7, i64 8)
+ to label %cont
+ unwind label %err
+cont:
+ ret void
+err:
+ %exn = landingpad { ptr, i32 }
+ cleanup
+ call void %x() [ "kcfi"(i32 12345678) ]
+ resume { ptr, i32 } %exn
+}
+
+;; Checked call followed by checked invoke in the same block, exercised with a
+;; Propeller profile that inserts code prefetch hints/targets on the split
+;; invoke block and attempts path cloning of the KCFI_CMP block (which
+;; BasicBlockPathCloning rejects because KCFI_CMP is marked isNotDuplicable).
+define void @f4(i1 %cond, ptr noundef %x, ptr noundef %y) personality ptr @__gxx_personality_v0 {
+; MIR-LABEL: name: f4
+; MIR: bb.3.join:
+; MIR-NEXT: successors: %bb.9(0x80000000), %bb.8(0x00000000)
+; MIR: CALL64pcrel32 target-flags(x86-plt) @side
+; MIR-NEXT: KCFI_CMP $r14, 12345678, implicit-def $r10, implicit-def $r11, implicit-def $eflags
+; MIR-NEXT: JCC_1 %bb.8, 5, implicit $eflags
+; MIR: bb.9.join:
+; MIR-NEXT: successors: %bb.7(0x80000000), %bb.6(0x00000000)
+; MIR: CALL64r killed $r14, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp
+; MIR-NEXT: KCFI_CMP $rbx, 12345678, implicit-def $r10, implicit-def $r11, implicit-def $eflags
+; MIR-NEXT: JCC_1 %bb.6, 5, implicit $eflags
+; MIR: bb.7.join:
+; MIR-NEXT: successors: %bb.4(0x7ffff800), %bb.5(0x00000800)
+; MIR: EH_LABEL
+; MIR-NEXT: CALL64r killed $rbx, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp
+; MIR-NEXT: EH_LABEL
+
+; PROP-LABEL: f4:
+; PROP: [[FNBEGIN:\.Lfunc_begin[0-9]+]]:
+; PROP: [[JOIN:\.LBB[0-9_]+]]: # %join
+; PROP: addl -4(%r14), %r10d
+; PROP-NEXT: jne f4.cold
+; PROP: callq *%r14
+; PROP: addl -4(%rbx), %r10d
+; PROP-NEXT: jne .LBB{{[0-9_]+}}
+; PROP: [[BEGIN:\.Ltmp[0-9]+]]: # EH_LABEL
+; PROP-NEXT: callq *%rbx
+; PROP: .globl __llvm_prefetch_target_f4_8_1
+; PROP-NEXT: __llvm_prefetch_target_f4_8_1:
+; PROP-NEXT: prefetchit1 __llvm_prefetch_target_side_0_0(%rip)
+; PROP: [[END:\.Ltmp[0-9]+]]: # EH_LABEL
+; PROP: retq
+; PROP: [[RIGHT:f4\.__part\.[0-9]+]]: # %right
+; PROP: movl $2, %edi
+; PROP-NEXT: jmp [[JOIN]]
+; PROP: f4.cold: # %join
+; PROP: .Ltmp{{[0-9]+}}:
+; PROP-NEXT: ud2
+; PROP: .LBB{{[0-9_]+}}: # %join
+; PROP-NEXT: .Ltmp{{[0-9]+}}:
+; PROP-NEXT: ud2
+; PROP: [[LPAD:\.Ltmp[0-9]+]]: # EH_LABEL
+; PROP: .section .gcc_except_table.f4
+; PROP: .uleb128 [[BEGIN]]-[[FNBEGIN]] # >> Call Site 2 <<
+; PROP-NEXT: .uleb128 [[END]]-[[BEGIN]]
+; PROP-NEXT: .uleb128 [[LPAD]]-f4.cold
+
+; PROP-WARN: warning: block #6 has non-duplicable instructions in function f4
+entry:
+ br i1 %cond, label %left, label %right
+left:
+ call void @side(i32 1)
+ br label %join
+right:
+ call void @side(i32 2)
+ br label %join
+join:
+ call void %x() [ "kcfi"(i32 12345678) ]
+ invoke void %y() [ "kcfi"(i32 12345678) ]
+ to label %cont
+ unwind label %err
+cont:
+ ret void
+err:
+ %exn = landingpad { ptr, i32 }
+ cleanup
+ call void @cleanup()
+ resume { ptr, i32 } %exn
+}
+
+!llvm.module.flags = !{!0, !1}
+!0 = !{i32 4, !"kcfi", i32 1}
+!1 = !{i32 8, !"kcfi-out-of-line-traps", i32 1}
+
+;--- prof
+v1
+f f4
+p 1 6 10
+c 0 1 6 10 8 4
+c 2
+t 8,1
+i 8,1 side,0,0
+
+;--- emutls.ll
+;; An invoke whose argument lowering inserts a call (such as emulated TLS)
+;; between BeginLabel and the indirect call keeps an inline trap so
+;; BeginLabel is not stranded across a KCFI block split.
+ at tls_var = thread_local global i32 0
+
+declare i32 @__gxx_personality_v0(...)
+declare void @cleanup()
+
+define void @f5(ptr noundef %x) personality ptr @__gxx_personality_v0 {
+; EMUTLS-LABEL: f5:
+; EMUTLS: [[FNBEGIN:\.Lfunc_begin[0-9]+]]:
+; EMUTLS: [[BEGIN:\.Ltmp[0-9]+]]: # EH_LABEL
+; EMUTLS: callq __emutls_get_address at PLT
+; EMUTLS: addl -4(%rbx), %r10d
+; EMUTLS-NEXT: je [[PASS:\.Ltmp[0-9]+]]
+; EMUTLS-NEXT: .Ltmp{{[0-9]+}}:
+; EMUTLS-NEXT: ud2
+; EMUTLS: [[PASS]]:
+; EMUTLS-NEXT: callq *%rbx
+; EMUTLS-NEXT: [[END:\.Ltmp[0-9]+]]: # EH_LABEL
+; EMUTLS: [[ERR:f5\.__part\.[0-9]+]]: # %err
+; EMUTLS: [[LPAD:\.Ltmp[0-9]+]]: # EH_LABEL
+; EMUTLS: .section .gcc_except_table.f5
+; EMUTLS: .uleb128 [[BEGIN]]-[[FNBEGIN]]
+; EMUTLS-NEXT: .uleb128 [[END]]-[[BEGIN]]
+; EMUTLS-NEXT: .uleb128 [[LPAD]]-[[ERR]]
+ invoke void %x(ptr @tls_var) [ "kcfi"(i32 12345678) ]
+ to label %cont
+ unwind label %err
+cont:
+ ret void
+err:
+ %exn = landingpad { ptr, i32 }
+ cleanup
+ call void @cleanup()
+ resume { ptr, i32 } %exn
+}
+
+!llvm.module.flags = !{!0, !1}
+!0 = !{i32 4, !"kcfi", i32 1}
+!1 = !{i32 8, !"kcfi-out-of-line-traps", i32 1}
diff --git a/llvm/test/CodeGen/X86/kcfi-out-of-line-traps-lto.ll b/llvm/test/CodeGen/X86/kcfi-out-of-line-traps-lto.ll
new file mode 100644
index 00000000000000..0766448ece65a2
--- /dev/null
+++ b/llvm/test/CodeGen/X86/kcfi-out-of-line-traps-lto.ll
@@ -0,0 +1,58 @@
+; RUN: split-file %s %t
+; RUN: llvm-link %t/ool.ll %t/inline.ll | llc -mtriple=x86_64-unknown-linux-gnu -verify-machineinstrs | FileCheck %s --check-prefix=INLINE
+; RUN: llvm-link %t/ool.ll %t/ool2.ll | llc -mtriple=x86_64-unknown-linux-gnu -verify-machineinstrs | FileCheck %s --check-prefix=OOL
+
+;; Verify kcfi-out-of-line-traps module flag semantics:
+;; Linking an opt-in module (Module::Min) with an absent module merges to 0
+;; so mixed LTO links keep inline traps, while linking two opt-in modules
+;; preserves out-of-line traps.
+
+;--- ool.ll
+define void @f_ool(ptr noundef %x) {
+ call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+!llvm.module.flags = !{!0, !1}
+!0 = !{i32 4, !"kcfi", i32 1}
+!1 = !{i32 8, !"kcfi-out-of-line-traps", i32 1}
+
+;--- ool2.ll
+define void @f_ool2(ptr noundef %x) {
+ call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+!llvm.module.flags = !{!0, !1}
+!0 = !{i32 4, !"kcfi", i32 1}
+!1 = !{i32 8, !"kcfi-out-of-line-traps", i32 1}
+
+; OOL-LABEL: f_ool:
+; OOL: jne [[TRAP1:.LBB[0-9_]+]]
+; OOL: callq *%rdi
+; OOL: [[TRAP1]]:
+; OOL: ud2
+; OOL-LABEL: f_ool2:
+; OOL: jne [[TRAP2:.LBB[0-9_]+]]
+; OOL: callq *%rdi
+; OOL: [[TRAP2]]:
+; OOL: ud2
+
+;--- inline.ll
+define void @f_inline(ptr noundef %x) {
+ call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+!llvm.module.flags = !{!0}
+!0 = !{i32 4, !"kcfi", i32 1}
+
+; INLINE-LABEL: f_ool:
+; INLINE: je [[PASS1:.Ltmp[0-9]+]]
+; INLINE-NEXT: .Ltmp{{[0-9]+}}:
+; INLINE-NEXT: ud2
+; INLINE: [[PASS1]]:
+; INLINE-NEXT: callq *%rdi
+; INLINE-LABEL: f_inline:
+; INLINE: je [[PASS2:.Ltmp[0-9]+]]
+; INLINE-NEXT: .Ltmp{{[0-9]+}}:
+; INLINE-NEXT: ud2
+; INLINE: [[PASS2]]:
+; INLINE-NEXT: callq *%rdi
diff --git a/llvm/test/CodeGen/X86/kcfi-out-of-line-traps-outliner.ll b/llvm/test/CodeGen/X86/kcfi-out-of-line-traps-outliner.ll
new file mode 100644
index 00000000000000..ba8adc69d3bbb9
--- /dev/null
+++ b/llvm/test/CodeGen/X86/kcfi-out-of-line-traps-outliner.ll
@@ -0,0 +1,52 @@
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -verify-machineinstrs -enable-machine-outliner < %s | FileCheck %s --check-prefix=OUTLINED
+
+;; KCFI_CMP and KCFI_TRAP are never outlined by the machine outliner, preserving
+;; patchable-function-prefix offsets and distinct trap sites. (Kept separate from
+;; kcfi-out-of-line-traps.ll because -enable-machine-outliner hits a
+;; pre-existing RetpolineThunkInserter assertion on retpoline calls.)
+define void @f1(ptr noundef %x, ptr %p) minsize "patchable-function-prefix"="4" {
+; OUTLINED-LABEL: f1:
+; OUTLINED: callq OUTLINED_FUNCTION_0
+; OUTLINED-NEXT: movl $4282621618, %r10d
+; OUTLINED-NEXT: addl -8(%rdi), %r10d
+; OUTLINED-NEXT: jne [[TRAP:.LBB[0-9_]+]]
+; OUTLINED: [[TRAP]]:
+; OUTLINED: ud2
+ store volatile i64 1, ptr %p, align 8
+ %p1 = getelementptr i64, ptr %p, i64 1
+ store volatile i64 2, ptr %p1, align 8
+ %p2 = getelementptr i64, ptr %p, i64 2
+ store volatile i64 3, ptr %p2, align 8
+ %p3 = getelementptr i64, ptr %p, i64 3
+ store volatile i64 4, ptr %p3, align 8
+ call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+define void @f2(ptr noundef %x, ptr %p) minsize "patchable-function-prefix"="4" {
+; OUTLINED-LABEL: f2:
+; OUTLINED: callq OUTLINED_FUNCTION_0
+; OUTLINED-NEXT: movl $4282621618, %r10d
+; OUTLINED-NEXT: addl -8(%rdi), %r10d
+; OUTLINED-NEXT: jne [[TRAP:.LBB[0-9_]+]]
+; OUTLINED: [[TRAP]]:
+; OUTLINED: ud2
+; OUTLINED-LABEL: OUTLINED_FUNCTION_0:
+; OUTLINED-NOT: addl
+; OUTLINED-NOT: ud2
+; OUTLINED: retq
+; OUTLINED-NOT: OUTLINED_FUNCTION_1:
+ store volatile i64 1, ptr %p, align 8
+ %p1 = getelementptr i64, ptr %p, i64 1
+ store volatile i64 2, ptr %p1, align 8
+ %p2 = getelementptr i64, ptr %p, i64 2
+ store volatile i64 3, ptr %p2, align 8
+ %p3 = getelementptr i64, ptr %p, i64 3
+ store volatile i64 4, ptr %p3, align 8
+ call void %x() [ "kcfi"(i32 12345678) ]
+ ret void
+}
+
+!llvm.module.flags = !{!0, !1}
+!0 = !{i32 4, !"kcfi", i32 1}
+!1 = !{i32 8, !"kcfi-out-of-line-traps", i32 1}
diff --git a/llvm/test/CodeGen/X86/kcfi-out-of-line-traps-wineh.ll b/llvm/test/CodeGen/X86/kcfi-out-of-line-traps-wineh.ll
new file mode 100644
index 00000000000000..d600f8be83fd7e
--- /dev/null
+++ b/llvm/test/CodeGen/X86/kcfi-out-of-line-traps-wineh.ll
@@ -0,0 +1,67 @@
+; RUN: llc -mtriple=x86_64-pc-windows-msvc -verify-machineinstrs < %s | FileCheck %s
+
+;; Windows EH (C++ EH and SEH) functions fall back to inline KCFI checks so
+;; faultable checks stay inside guarded ranges without extra EH state
+;; bookkeeping.
+
+declare i32 @__CxxFrameHandler3(...)
+declare i32 @__C_specific_handler(...)
+declare void @cleanup()
+
+define void @win_invoke(ptr noundef %x) personality ptr @__CxxFrameHandler3 {
+; CHECK-LABEL: win_invoke:
+; CHECK: [[BEGIN:\.Ltmp[0-9]+]]: # EH_LABEL
+; CHECK-NEXT: movl $4282621618, %r10d # imm = 0xFF439EB2
+; CHECK-NEXT: addl -4(%rcx), %r10d
+; CHECK-NEXT: je [[PASS:\.Ltmp[0-9]+]]
+; CHECK: ud2
+; CHECK-NEXT: [[PASS]]:
+; CHECK-NEXT: callq *%rcx
+; CHECK-NEXT: nop
+; CHECK-NEXT: [[END:\.Ltmp[0-9]+]]: # EH_LABEL
+; CHECK: $ip2state$win_invoke:
+; CHECK: .long [[BEGIN]]@IMGREL # IP
+; CHECK-NEXT: .long 0 # ToState
+; CHECK-NEXT: .long [[END]]@IMGREL # IP
+; CHECK-NEXT: .long -1 # ToState
+entry:
+ invoke void %x() [ "kcfi"(i32 12345678) ]
+ to label %cont
+ unwind label %ehcleanup
+cont:
+ ret void
+ehcleanup:
+ %cp = cleanuppad within none []
+ call void @cleanup() [ "funclet"(token %cp) ]
+ cleanupret from %cp unwind to caller
+}
+
+define i32 @win_seh(ptr noundef %x) personality ptr @__C_specific_handler {
+; CHECK-LABEL: win_seh:
+; CHECK: [[BEGIN:\.Ltmp[0-9]+]]: # EH_LABEL
+; CHECK-NEXT: movl $4282621618, %r10d # imm = 0xFF439EB2
+; CHECK-NEXT: addl -4(%rcx), %r10d
+; CHECK-NEXT: je [[PASS:\.Ltmp[0-9]+]]
+; CHECK: ud2
+; CHECK-NEXT: [[PASS]]:
+; CHECK-NEXT: callq *%rcx
+; CHECK-NEXT: nop
+; CHECK-NEXT: [[END:\.Ltmp[0-9]+]]: # EH_LABEL
+; CHECK: .long [[BEGIN]]@IMGREL # LabelStart
+; CHECK-NEXT: .long [[END]]@IMGREL # LabelEnd
+entry:
+ invoke void %x() [ "kcfi"(i32 12345678) ]
+ to label %cont
+ unwind label %catch.dispatch
+catch.dispatch:
+ %cs = catchswitch within none [label %catch] unwind to caller
+catch:
+ %cp = catchpad within %cs [ptr null]
+ catchret from %cp to label %cont
+cont:
+ ret i32 0
+}
+
+!llvm.module.flags = !{!0, !1}
+!0 = !{i32 4, !"kcfi", i32 1}
+!1 = !{i32 8, !"kcfi-out-of-line-traps", i32 1}
diff --git a/llvm/test/CodeGen/X86/kcfi-out-of-line-traps.ll b/llvm/test/CodeGen/X86/kcfi-out-of-line-traps.ll
index 2798f7689d5eb6..9ad4ce3d3019e8 100644
--- a/llvm/test/CodeGen/X86/kcfi-out-of-line-traps.ll
+++ b/llvm/test/CodeGen/X86/kcfi-out-of-line-traps.ll
@@ -1,6 +1,7 @@
; RUN: llc -mtriple=x86_64-unknown-linux-gnu -verify-machineinstrs < %s | FileCheck %s --check-prefix=ASM
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mcpu=x86-64-v3 -enable-post-misched -verify-machineinstrs < %s | FileCheck %s --check-prefix=ASM
; RUN: llc -mtriple=x86_64-unknown-linux-gnu -verify-machineinstrs -stop-after=kcfi < %s | FileCheck %s --check-prefix=MIR
-; RUN: llc -mtriple=x86_64-unknown-linux-gnu -stop-after=kcfi -enable-new-pm < %s | FileCheck %s --check-prefix=MIR
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -verify-machineinstrs -stop-after=kcfi -enable-new-pm < %s | FileCheck %s --check-prefix=MIR
; RUN: llc -mtriple=x86_64-unknown-linux-gnu -verify-machineinstrs -function-splitting=all < %s | FileCheck %s --check-prefix=SPLIT
;; With the kcfi-out-of-line-traps module flag, a KCFI check ends in a branch
@@ -87,6 +88,15 @@ define void @f3(ptr noundef %x) #0 {
;; So does one that had its memory operand unfolded.
define void @f4() {
+; ASM-LABEL: f4:
+; ASM: movq g at GOTPCREL(%rip), [[PTR:%r[a-z0-9]+]]
+; ASM-NEXT: movq ([[PTR]]), [[CALLEE:%r[a-z0-9]+]]
+; ASM: addl -4([[CALLEE]]), %r10d
+; ASM-NEXT: jne [[TRAP:\.LBB[0-9_]+]]
+; ASM: callq *[[CALLEE]]
+; ASM: [[TRAP]]:
+; ASM: ud2
+
; MIR-LABEL: name: f4
; MIR: $r11 = MOV64rm killed renamable $rax, 1, $noreg, 0, $noreg
; MIR-NEXT: KCFI_CMP $r11, 12345678, implicit-def $r10, implicit-def $r11, implicit-def $eflags
@@ -146,23 +156,26 @@ define void @f5(ptr noundef %x, ptr noundef %y) {
declare i32 @__gxx_personality_v0(...)
declare void @cleanup()
-;; A call in a block that has an EH pad successor can be an invoke, which has
-;; to stay next to the EH label that starts its try-range. It keeps an inline
-;; trap.
+;; A non-invoke call preceding another call/invoke in a block with an EH pad
+;; successor keeps an inline bundled check.
define void @f6(ptr noundef %x) personality ptr @__gxx_personality_v0 {
; ASM-LABEL: f6:
; ASM: addl -4(%rdi), %r10d
-; ASM-NEXT: je [[PASS:.Ltmp[0-9]+]]
+; ASM-NEXT: je [[PASS:\.Ltmp[0-9]+]]
; ASM-NEXT: .Ltmp{{[0-9]+}}:
; ASM-NEXT: ud2
; ASM: [[PASS]]:
; ASM-NEXT: callq *%rdi
+; ASM-NEXT: .Ltmp{{[0-9]+}}:
+; ASM-NEXT: callq cleanup at PLT
; MIR-LABEL: name: f6
; MIR: BUNDLE{{.*}} {
; MIR-NEXT: KCFI_CHECK $rdi, 12345678, implicit-def $r10, implicit-def $r11, implicit-def $eflags
; MIR-NEXT: CALL64r killed $rdi, csr_64, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp
; MIR-NEXT: }
+; MIR-NEXT: EH_LABEL
+; MIR-NEXT: CALL64pcrel32 target-flags(x86-plt) @cleanup
; MIR-NEXT: EH_LABEL
call void %x() [ "kcfi"(i32 12345678) ]
invoke void @cleanup()
@@ -199,7 +212,7 @@ attributes #0 = { "target-features"="+retpoline-indirect-branches,+retpoline-ind
!llvm.module.flags = !{!0, !1, !2}
!0 = !{i32 4, !"kcfi", i32 1}
-!1 = !{i32 4, !"kcfi-out-of-line-traps", i32 1}
+!1 = !{i32 8, !"kcfi-out-of-line-traps", i32 1}
!2 = !{i32 1, !"ProfileSummary", !3}
!3 = !{!4, !5, !6, !7, !8, !9, !10, !11}
!4 = !{!"ProfileFormat", !"InstrProf"}
More information about the cfe-commits
mailing list