[llvm] [BPF] Add exception handling support with .bpf_cleanup section (PR #192164)
via llvm-commits
llvm-commits at lists.llvm.org
Tue Apr 14 17:08:09 PDT 2026
https://github.com/4ast created https://github.com/llvm/llvm-project/pull/192164
Add support for invoke/landingpad/resume instructions in the BPF backend so that Rust programs compiled with panic=unwind can run cleanup code (Drop implementations) when bpf_throw fires.
Changes:
1. BPFISelLowering: Define exception pointer and selector registers (both R0) so SelectionDAG can lower landingpad instructions.
2. BPFAsmPrinter::emitFunctionBodyEnd: Emit a .bpf_cleanup section with a flat table of (begin, end, landing_pad) triples using R_BPF_64_NODYLD32 relocations.
The .bpf_cleanup section layout (12 bytes per entry):
u32 begin // start of the invoke region
u32 end // end of the invoke region
u32 landing_pad // address of the cleanup block
The invoke region [begin, end) includes argument setup instructions before the call. The runtime checks begin <= PC < end to find the matching landing pad.
Landing pad blocks survive optimization because invoke maintains CFG edges to them throughout codegen, same as every other backend. The standard .gcc_except_table and .eh_frame are also emitted by the existing DwarfCFIException handler; libbpf will ignore them.
In runtime:
- bpf_throw() is called (from panic handler)
- Kernel walks the BPF call stack with arch_bpf_stack_walk()
- For each frame, look up current PC in .bpf_cleanup table
- If match found: redirect execution to the cleanup function . cleanup function runs Drop impls (bpf_free, rcu_read_unlock, etc.) . calls _Unwind_Resume() which is patched to just 'ret' by the verifier . bpf_throw() pops frame goes to next
- If no match: go to next frame
>From 4bea207f7acc5457a616d807211bd88fd28ac46d Mon Sep 17 00:00:00 2001
From: Alexei Starovoitov <ast at kernel.org>
Date: Tue, 14 Apr 2026 16:01:48 -0700
Subject: [PATCH] [BPF] Add exception handling support with .bpf_cleanup
section
Add support for invoke/landingpad/resume instructions in the BPF
backend so that Rust programs compiled with panic=unwind can run
cleanup code (Drop implementations) when bpf_throw fires.
Changes:
1. BPFISelLowering: Define exception pointer and selector registers
(both R0) so SelectionDAG can lower landingpad instructions.
2. BPFAsmPrinter::emitFunctionBodyEnd: Emit a .bpf_cleanup section
with a flat table of (begin, end, landing_pad) triples using
R_BPF_64_NODYLD32 relocations.
The .bpf_cleanup section layout (12 bytes per entry):
u32 begin // start of the invoke region
u32 end // end of the invoke region
u32 landing_pad // address of the cleanup block
The invoke region [begin, end) includes argument setup instructions
before the call. The runtime checks begin <= PC < end to find the
matching landing pad.
Landing pad blocks survive optimization because invoke maintains
CFG edges to them throughout codegen, same as every other backend.
The standard .gcc_except_table and .eh_frame are also emitted by
the existing DwarfCFIException handler; libbpf will ignore them.
In runtime:
- bpf_throw() is called (from panic handler)
- Kernel walks the BPF call stack with arch_bpf_stack_walk()
- For each frame, look up current PC in .bpf_cleanup table
- If match found: redirect execution to the cleanup function
. cleanup function runs Drop impls (bpf_free, rcu_read_unlock, etc.)
. calls _Unwind_Resume() which is patched to just 'ret' by the verifier
. bpf_throw() pops frame goes to next
- If no match: go to next frame
Signed-off-by: Alexei Starovoitov <ast at kernel.org>
---
llvm/lib/Target/BPF/BPFAsmPrinter.cpp | 38 +++++++
llvm/lib/Target/BPF/BPFAsmPrinter.h | 1 +
llvm/lib/Target/BPF/BPFISelLowering.h | 8 ++
llvm/test/CodeGen/BPF/cleanup-section.ll | 121 +++++++++++++++++++++++
4 files changed, 168 insertions(+)
create mode 100644 llvm/test/CodeGen/BPF/cleanup-section.ll
diff --git a/llvm/lib/Target/BPF/BPFAsmPrinter.cpp b/llvm/lib/Target/BPF/BPFAsmPrinter.cpp
index abe081c0c76fd..6817d06df8736 100644
--- a/llvm/lib/Target/BPF/BPFAsmPrinter.cpp
+++ b/llvm/lib/Target/BPF/BPFAsmPrinter.cpp
@@ -31,6 +31,7 @@
#include "llvm/MC/MCInst.h"
#include "llvm/MC/MCStreamer.h"
#include "llvm/MC/MCSymbol.h"
+#include "llvm/MC/MCSectionELF.h"
#include "llvm/MC/MCSymbolELF.h"
#include "llvm/MC/TargetRegistry.h"
#include "llvm/Support/Compiler.h"
@@ -192,6 +193,43 @@ void BPFAsmPrinter::emitInstruction(const MachineInstr *MI) {
EmitToStreamer(*OutStreamer, TmpInst);
}
+void BPFAsmPrinter::emitFunctionBodyEnd() {
+ // Emit .bpf_cleanup section with a flat table of
+ // (call_site, landing_pad) pairs.
+ const std::vector<LandingPadInfo> &LandingPads = MF->getLandingPads();
+ if (LandingPads.empty())
+ return;
+
+ MCContext &Ctx = OutContext;
+ auto *CleanupSec = Ctx.getELFSection(".bpf_cleanup",
+ ELF::SHT_PROGBITS,
+ ELF::SHF_ALLOC);
+ OutStreamer->switchSection(CleanupSec);
+
+ // Each landing pad has BeginLabels/EndLabels marking the invoke
+ // call sites that unwind to it.
+ for (const LandingPadInfo &LP : LandingPads) {
+ MCSymbol *LPLabel = LP.LandingPadLabel;
+ if (!LPLabel)
+ continue;
+ for (unsigned i = 0, e = LP.BeginLabels.size(); i != e; ++i) {
+ MCSymbol *Begin = LP.BeginLabels[i];
+ MCSymbol *End = LP.EndLabels[i];
+
+ // Each entry is 3 x 4 bytes: begin, end, landing_pad.
+ // The invoke region [begin, end) may include argument setup
+ // before the call. The runtime checks begin <= PC < end.
+ OutStreamer->emitSymbolValue(Begin, 4);
+ OutStreamer->emitSymbolValue(End, 4);
+ OutStreamer->emitSymbolValue(LPLabel, 4);
+ }
+ }
+
+ // Switch back to the function's section.
+ OutStreamer->switchSection(MF->getSection());
+
+}
+
MCSymbol *BPFAsmPrinter::getJTPublicSymbol(unsigned JTI) {
SmallString<60> Name;
raw_svector_ostream(Name)
diff --git a/llvm/lib/Target/BPF/BPFAsmPrinter.h b/llvm/lib/Target/BPF/BPFAsmPrinter.h
index 75a1d7ed9f884..d1db65d878463 100644
--- a/llvm/lib/Target/BPF/BPFAsmPrinter.h
+++ b/llvm/lib/Target/BPF/BPFAsmPrinter.h
@@ -31,6 +31,7 @@ class BPFAsmPrinter : public AsmPrinter {
const char *ExtraCode, raw_ostream &O) override;
void emitInstruction(const MachineInstr *MI) override;
+ void emitFunctionBodyEnd() override;
MCSymbol *getJTPublicSymbol(unsigned JTI);
void emitJumpTableInfo() override;
diff --git a/llvm/lib/Target/BPF/BPFISelLowering.h b/llvm/lib/Target/BPF/BPFISelLowering.h
index f412e0b31c446..b16fce0dd418f 100644
--- a/llvm/lib/Target/BPF/BPFISelLowering.h
+++ b/llvm/lib/Target/BPF/BPFISelLowering.h
@@ -54,6 +54,14 @@ class BPFTargetLowering : public TargetLowering {
EVT getSetCCResultType(const DataLayout &DL, LLVMContext &Context,
EVT VT) const override;
+ // Exception handling support.
+ Register getExceptionPointerRegister(const Constant *) const override {
+ return BPF::R0;
+ }
+ Register getExceptionSelectorRegister(const Constant *) const override {
+ return BPF::R0;
+ }
+
MVT getScalarShiftAmountTy(const DataLayout &, EVT) const override;
unsigned getJumpTableEncoding() const override;
diff --git a/llvm/test/CodeGen/BPF/cleanup-section.ll b/llvm/test/CodeGen/BPF/cleanup-section.ll
new file mode 100644
index 0000000000000..c0e14864f720d
--- /dev/null
+++ b/llvm/test/CodeGen/BPF/cleanup-section.ll
@@ -0,0 +1,121 @@
+; RUN: llc -mtriple=bpfel -mcpu=v4 -filetype=asm -o - %s | FileCheck --check-prefix=ASM %s
+; RUN: llc -mtriple=bpfel -mcpu=v4 -filetype=obj -o %t %s
+; RUN: llvm-readelf -S %t | FileCheck --check-prefix=SECTIONS %s
+; RUN: llvm-readelf -r %t | FileCheck --check-prefix=RELOCS %s
+
+; Verify that invoke/landingpad with cleanup produces a .bpf_cleanup section
+; containing (call_site, landing_pad) pairs with R_BPF_64_NODYLD32 relocations.
+; Models a Rust BTreeMap insertion that may fail, with Drop cleanup
+; calling bpf_free to release allocated nodes.
+
+declare ptr @bpf_alloc(i64, i64)
+declare void @bpf_free(ptr)
+declare void @btree_insert(ptr, ptr, i64)
+declare i32 @rust_personality(i32, i64, ptr, ptr)
+
+; SECTIONS: .bpf_cleanup
+
+define void @stat_inc(ptr %map, ptr %key, i64 %val) personality ptr @rust_personality {
+entry:
+ %node = call ptr @bpf_alloc(i64 256, i64 0)
+ %null = icmp eq ptr %node, null
+ br i1 %null, label %oom, label %insert
+
+insert:
+ invoke void @btree_insert(ptr %map, ptr %node, i64 %val)
+ to label %done unwind label %cleanup
+
+done:
+ ret void
+
+cleanup:
+ %lp = landingpad { ptr, i32 } cleanup
+ call void @bpf_free(ptr %node)
+ resume { ptr, i32 } %lp
+
+oom:
+ ret void
+}
+
+; ASM-LABEL: stat_inc:
+; ASM: r1 = 256
+; ASM: call bpf_alloc
+; ASM: if r0 == 0 goto
+; Normal path: call btree_insert then exit
+; ASM: .Ltmp0:
+; ASM: call btree_insert
+; ASM: .Ltmp1:
+; ASM: exit
+; Cleanup landing pad: free the node, then resume unwinding
+; ASM: .Ltmp2:
+; ASM: r1 = r7
+; ASM: call bpf_free
+; ASM: call _Unwind_Resume
+; .bpf_cleanup entry: [.Ltmp0, .Ltmp1) -> .Ltmp2
+; ASM: .section .bpf_cleanup,"a", at progbits
+; ASM-NEXT: .long .Ltmp0
+; ASM-NEXT: .long .Ltmp1
+; ASM-NEXT: .long .Ltmp2
+
+define void @two_allocs(ptr %map) personality ptr @rust_personality {
+entry:
+ %n1 = call ptr @bpf_alloc(i64 256, i64 0)
+ invoke void @btree_insert(ptr %map, ptr %n1, i64 1)
+ to label %second unwind label %clean1
+
+second:
+ %n2 = call ptr @bpf_alloc(i64 512, i64 0)
+ invoke void @btree_insert(ptr %map, ptr %n2, i64 2)
+ to label %done unwind label %clean2
+
+done:
+ ret void
+
+clean2:
+ ; Must free both n2 and n1
+ %lp2 = landingpad { ptr, i32 } cleanup
+ call void @bpf_free(ptr %n2)
+ call void @bpf_free(ptr %n1)
+ resume { ptr, i32 } %lp2
+
+clean1:
+ ; Only n1 allocated at this point
+ %lp1 = landingpad { ptr, i32 } cleanup
+ call void @bpf_free(ptr %n1)
+ resume { ptr, i32 } %lp1
+}
+
+; ASM-LABEL: two_allocs:
+; First alloc + invoke
+; ASM: r1 = 256
+; ASM: call bpf_alloc
+; ASM: .Ltmp3:
+; ASM: call btree_insert
+; ASM: .Ltmp4:
+; Second alloc + invoke
+; ASM: r1 = 512
+; ASM: call bpf_alloc
+; ASM: .Ltmp6:
+; ASM: call btree_insert
+; ASM: .Ltmp7:
+; ASM: exit
+; clean2: free n2, then fall through to free n1
+; ASM: .Ltmp8:
+; ASM: call bpf_free
+; clean1 entry point, shared tail: free n1 then resume
+; ASM: .Ltmp5:
+; ASM: call bpf_free
+; ASM: call _Unwind_Resume
+; .bpf_cleanup entries: two triples
+; ASM: .section .bpf_cleanup,"a", at progbits
+; ASM-NEXT: .long .Ltmp3
+; ASM-NEXT: .long .Ltmp4
+; ASM-NEXT: .long .Ltmp5
+; ASM-NEXT: .long .Ltmp6
+; ASM-NEXT: .long .Ltmp7
+; ASM-NEXT: .long .Ltmp8
+
+; .bpf_cleanup should have R_BPF_64_NODYLD32 relocations
+; 3 entries x 3 fields = 9 relocs
+; RELOCS: .rel.bpf_cleanup
+; RELOCS-COUNT-9: R_BPF_64_NODYLD32 {{.*}} .text
More information about the llvm-commits
mailing list