[clang] [llvm] [Sanitizers] Support sanitizer instrumentation regions in KMSAN, KASAN, KCOV and KCSAN (PR #231374)
Alexander Potapenko via cfe-commits
cfe-commits at lists.llvm.org
Mon Oct 12 23:26:28 PDT 2026
https://github.com/ramosian-glider created https://github.com/llvm/llvm-project/pull/231374
Some code must not be instrumented as a whole but still has parts where instrumentation is allowed. The Linux kernel marks such functions `noinstr` and delimits the parts that may call instrumented code with `instrumentation_begin()` and `instrumentation_end()`. Sanitizers currently cannot see these parts: they either skip the whole function or instrument all of it. Under KMSAN this causes false positives on the kernel entry paths, because `noinstr` code never sets up parameter shadow for the instrumented functions it calls (e.g. https://lore.kernel.org/all/69e7ee1f.a00a0220.17a17.001d.GAE@google.com/T/#u). Other kernel sanitizers simply miss bugs in these regions.
This series adds region markers and implements them in four sanitizers. Each commit is self-contained and can be reviewed separately:
1. **[Sanitizers] Add sanitizer instrumentation region intrinsics and builtins**
`llvm.sanitizer.instrumentation.begin`/`end`, `__builtin_sanitizer_instrumentation_begin()`/`end()`, and `computeSanitizerInstrumentationRegions()`, a must-analysis over the region nesting depth shared by the passes (an instruction is in a region only if every path to it entered one, so it is always dominated by a `begin`). The intrinsics keep the default memory effects so that accesses to escaping memory do not move across region boundaries; codegen drops them. Clang only emits them for sanitizers that consume them.
2. **[MSan] Support sanitizer instrumentation regions in KMSAN**
In `disable_sanitizer_instrumentation` functions, fetch the context state at `begin` and propagate shadow / set up parameter shadow inside regions.
3. **[ASan] Support sanitizer instrumentation regions**
Check memory accesses and memory intrinsics inside regions of functions without `sanitize_address` or with `disable_sanitizer_instrumentation`.
4. **[SanitizerCoverage] Support sanitizer instrumentation regions**
Add coverage points for in-region code of `nosanitize_coverage` / `disable_sanitizer_instrumentation` functions (KCOV).
5. **[TSan] Support sanitizer instrumentation regions**
Instrument in-region accesses of `disable_sanitizer_instrumentation` functions and functions without `sanitize_thread` (KCSAN).
The semantics are documented in LangRef and the per-sanitizer docs. Tests: IR tests for each pass, a PhaseOrdering test that the markers survive the `-O2` pipeline, clang CodeGen tests for the builtins and for the full `-O2` pipelines (KMSAN, KASAN, KCOV, KCSAN, KMSAN+KCOV), CodeGen tests for SelectionDAG/FastISel/GlobalISel, and unit tests for the region analysis.
The kernel side (KMSAN first, then KASAN/KCOV/KCSAN) will use the builtins from `instrumentation_begin()`/`instrumentation_end()`; with KASAN, about 170 checks are added in `noinstr` code of a typical x86 config.
This supersedes #202603.
>From 43e8693c82bc039291885986b8bfb13d60de4170 Mon Sep 17 00:00:00 2001
From: Alexander Potapenko <glider at google.com>
Date: Mon, 5 Oct 2026 14:04:11 +0000
Subject: [PATCH 1/5] [Sanitizers] Add sanitizer instrumentation region
intrinsics and builtins
Some code must not be instrumented as a whole but still has parts where
instrumentation is allowed. The Linux kernel marks such functions
`noinstr` (lowered to `disable_sanitizer_instrumentation` and/or the
absence of `sanitize_<x>`), and delimits the parts that may call
instrumented code with `instrumentation_begin()` and
`instrumentation_end()`. Sanitizers currently cannot see these parts:
they either skip the whole function or instrument all of it.
This patch adds the common infrastructure that lets sanitizers
instrument such regions:
- `llvm.sanitizer.instrumentation.begin` and
`llvm.sanitizer.instrumentation.end` delimit a region of an otherwise
uninstrumented function in which sanitizer instrumentation is enabled
again. What counts as uninstrumented and what gets instrumented inside
regions is up to each sanitizer; sanitizers that do not support
regions ignore the intrinsics. The intrinsics keep the default memory
effects so that accesses to memory that may be visible outside the
function do not move across region boundaries, and are `IntrNoMerge`. SelectionDAG, FastISel and
GlobalISel drop them if they reach codegen.
- `computeSanitizerInstrumentationRegions()` in Utils/Instrumentation
computes region membership for the passes. It is a must-analysis over
the nesting depth (meet = min over predecessors): regions nest, an
unmatched end is ignored, and an instruction is in a region only if
every path to it entered one, so it is always dominated by a begin.
- `__builtin_sanitizer_instrumentation_begin()` and
`__builtin_sanitizer_instrumentation_end()` expose the intrinsics to
C code, which can detect them with `__has_builtin()`. They are only
lowered to the intrinsics for sanitizers that consume them, so that
they do not act as optimization barriers elsewhere; no sanitizer does
yet, so for now they lower to nothing.
The semantics are documented in LangRef. Follow-up patches implement
regions in KernelMemorySanitizer, AddressSanitizer, SanitizerCoverage
and ThreadSanitizer.
---
clang/include/clang/Basic/Builtins.td | 18 ++
clang/lib/CodeGen/CGBuiltin.cpp | 14 ++
.../sanitizer-instrumentation-builtins.c | 27 +++
llvm/docs/LangRef.md | 21 ++
llvm/include/llvm/IR/Intrinsics.td | 18 ++
.../llvm/Transforms/Utils/Instrumentation.h | 19 ++
llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp | 5 +
llvm/lib/CodeGen/SelectionDAG/FastISel.cpp | 2 +
.../SelectionDAG/SelectionDAGBuilder.cpp | 2 +
llvm/lib/Transforms/Utils/Instrumentation.cpp | 79 +++++++
.../sanitizer-instrumentation-intrinsics.ll | 19 ++
.../sanitizer-instrumentation-intrinsics.ll | 20 ++
.../sanitizer-instrumentation-markers.ll | 86 ++++++++
.../unittests/Transforms/Utils/CMakeLists.txt | 1 +
.../Transforms/Utils/SanitizerRegionsTest.cpp | 192 ++++++++++++++++++
15 files changed, 523 insertions(+)
create mode 100644 clang/test/CodeGen/sanitizer-instrumentation-builtins.c
create mode 100644 llvm/test/CodeGen/Generic/sanitizer-instrumentation-intrinsics.ll
create mode 100644 llvm/test/CodeGen/X86/sanitizer-instrumentation-intrinsics.ll
create mode 100644 llvm/test/Transforms/PhaseOrdering/sanitizer-instrumentation-markers.ll
create mode 100644 llvm/unittests/Transforms/Utils/SanitizerRegionsTest.cpp
diff --git a/clang/include/clang/Basic/Builtins.td b/clang/include/clang/Basic/Builtins.td
index ea8dbb96fab56..b1f84575d9dec 100644
--- a/clang/include/clang/Basic/Builtins.td
+++ b/clang/include/clang/Basic/Builtins.td
@@ -1469,6 +1469,24 @@ def BuiltinAllowSanitizeCheck : Builtin {
let Prototype = "bool(char const*)";
}
+// Sanitizer instrumentation region markers. These delimit a region of an
+// otherwise uninstrumented function (e.g. one marked with
+// __attribute__((disable_sanitizer_instrumentation))) in which sanitizer
+// instrumentation is enabled. They must not be Const or Pure: they are
+// optimization barriers, and moving code across them changes which
+// instructions belong to the region.
+def SanitizerInstrumentationBegin : Builtin {
+ let Spellings = ["__builtin_sanitizer_instrumentation_begin"];
+ let Attributes = [NoThrow];
+ let Prototype = "void()";
+}
+
+def SanitizerInstrumentationEnd : Builtin {
+ let Spellings = ["__builtin_sanitizer_instrumentation_end"];
+ let Attributes = [NoThrow];
+ let Prototype = "void()";
+}
+
def ShuffleVector : Builtin {
let Spellings = ["__builtin_shufflevector"];
let Attributes = [NoThrow, Const, CustomTypeChecking, Constexpr];
diff --git a/clang/lib/CodeGen/CGBuiltin.cpp b/clang/lib/CodeGen/CGBuiltin.cpp
index 4c1318f6543c1..71bdaefb57ac9 100644
--- a/clang/lib/CodeGen/CGBuiltin.cpp
+++ b/clang/lib/CodeGen/CGBuiltin.cpp
@@ -3908,6 +3908,20 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
llvm::MetadataAsValue::get(Ctx, llvm::MDString::get(Ctx, Kind)));
return RValue::get(Allow);
}
+ case Builtin::BI__builtin_sanitizer_instrumentation_begin:
+ case Builtin::BI__builtin_sanitizer_instrumentation_end: {
+ // Only emit the region markers for sanitizers that honour them. Elsewhere
+ // they would just be optimization barriers that no pass ever removes. This
+ // lets code guard its uses on __has_builtin alone.
+ SanitizerMask Consumers;
+ if (getLangOpts().Sanitize.hasOneOf(Consumers))
+ Builder.CreateCall(CGM.getIntrinsic(
+ BuiltinIDIfNoAsmLabel ==
+ Builtin::BI__builtin_sanitizer_instrumentation_begin
+ ? Intrinsic::sanitizer_instrumentation_begin
+ : Intrinsic::sanitizer_instrumentation_end));
+ return RValue::get(nullptr);
+ }
case Builtin::BI__builtin_allow_sanitize_check: {
Intrinsic::ID IntrID = Intrinsic::not_intrinsic;
StringRef Name =
diff --git a/clang/test/CodeGen/sanitizer-instrumentation-builtins.c b/clang/test/CodeGen/sanitizer-instrumentation-builtins.c
new file mode 100644
index 0000000000000..6fef449edd681
--- /dev/null
+++ b/clang/test/CodeGen/sanitizer-instrumentation-builtins.c
@@ -0,0 +1,27 @@
+// Check that the sanitizer region markers are available, and that they lower to
+// nothing without a sanitizer that consumes them.
+
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -disable-llvm-passes \
+// RUN: -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -fsanitize=thread \
+// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s
+
+#if !__has_builtin(__builtin_sanitizer_instrumentation_begin)
+#error "__builtin_sanitizer_instrumentation_begin not available"
+#endif
+#if !__has_builtin(__builtin_sanitizer_instrumentation_end)
+#error "__builtin_sanitizer_instrumentation_end not available"
+#endif
+
+extern void instrumented_callee(int);
+
+__attribute__((disable_sanitizer_instrumentation))
+void noinstr_function(int x) {
+ __builtin_sanitizer_instrumentation_begin();
+ instrumented_callee(x);
+ __builtin_sanitizer_instrumentation_end();
+}
+
+// CHECK-LABEL: define {{.*}}void @noinstr_function(
+// CHECK-NOT: llvm.sanitizer.instrumentation
+// CHECK: }
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index 6a3194271b838..524d88e461dbf 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -30489,3 +30489,24 @@ taken. A late optimization pass will convert this intrinsic to either
`llvm.cond.loop(true)` or `llvm.cond.loop(pred)`, where `pred`
is a predicate for a conditional branch leading to the intrinsic call,
if possible.
+
+#### '`llvm.sanitizer.instrumentation.begin`' and '`llvm.sanitizer.instrumentation.end`' Intrinsics
+
+##### Syntax:
+
+```llvm
+declare void @llvm.sanitizer.instrumentation.begin()
+declare void @llvm.sanitizer.instrumentation.end()
+```
+
+##### Overview:
+
+The '`llvm.sanitizer.instrumentation.begin`' and '`llvm.sanitizer.instrumentation.end`' intrinsics delimit a region of a function that a sanitizer does not otherwise instrument, in which that sanitizer's instrumentation is enabled again.
+
+##### Semantics:
+
+Inside a region, a sanitizer instruments the code as if the function were instrumented, to the extent it supports instrumenting part of a function; code outside regions stays uninstrumented. Sanitizers that do not support regions ignore the intrinsics, as do all sanitizers in functions that they instrument anyway. Which functions count as uninstrumented, and what is instrumented inside regions, depends on the sanitizer.
+
+Regions nest: `llvm.sanitizer.instrumentation.begin` increments a region nesting depth and `llvm.sanitizer.instrumentation.end` decrements it, so an inner `end` does not terminate an enclosing region. An unmatched `end` is ignored rather than making the depth negative.
+
+An instruction is considered to be inside a region only if *every* path from the function entry to it is inside a region, i.e. region membership is an intersection over predecessors. This makes an in-region instruction always dominated by a `llvm.sanitizer.instrumentation.begin`, so any per-function sanitizer state set up there is available. A region that is entered on some paths but not others is therefore instrumented only where it is unambiguous.
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 37c9c783465d6..f3b4cb0a9e2b8 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -2828,6 +2828,24 @@ def int_load_relative: DefaultAttrsIntrinsic<[llvm_ptr_ty],
def int_asan_check_memaccess :
Intrinsic<[],[llvm_ptr_ty, llvm_i32_ty], [ImmArg<ArgIndex<1>>]>;
+// Sanitizer instrumentation regions in functions that are otherwise not
+// instrumented.
+//
+// These are markers consumed by sanitizer passes, but they deliberately
+// keep the default (read/write any memory) effects and so act as memory
+// barriers. Relaxing them, e.g. to IntrInaccessibleMemOnly, lets the optimizer
+// move memory operations across a region boundary: a load can be CSEd with an
+// identical load from before begin() and lose its shadow check, and a store can
+// sink past end() and leave a stale shadow behind. Both defeat the purpose of
+// the region. What the explicit attribute list buys us is the rest of the
+// modelling - these calls do not throw, do not free, do not call back into the
+// module and always return - plus IntrNoMerge, which stops identical markers in
+// sibling blocks from being merged and thereby moving code out of a region.
+def int_sanitizer_instrumentation_begin
+ : DefaultAttrsIntrinsic<[], [], [IntrNoMerge]>;
+def int_sanitizer_instrumentation_end
+ : DefaultAttrsIntrinsic<[], [], [IntrNoMerge]>;
+
// Spin in an infinite loop (using instructions specified by the target) iff the
// argument is true. Used to implement efficient conditional traps.
def int_cond_loop : Intrinsic<[], [llvm_i1_ty],
diff --git a/llvm/include/llvm/Transforms/Utils/Instrumentation.h b/llvm/include/llvm/Transforms/Utils/Instrumentation.h
index 95a985ba3f0c4..dd143af599e25 100644
--- a/llvm/include/llvm/Transforms/Utils/Instrumentation.h
+++ b/llvm/include/llvm/Transforms/Utils/Instrumentation.h
@@ -13,6 +13,7 @@
#ifndef LLVM_TRANSFORMS_INSTRUMENTATION_H
#define LLVM_TRANSFORMS_INSTRUMENTATION_H
+#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/IR/BasicBlock.h"
#include "llvm/IR/DebugInfoMetadata.h"
@@ -60,6 +61,24 @@ LLVM_ABI Comdat *getOrCreateFunctionComdat(Function &F, Triple &T);
LLVM_ABI void setGlobalVariableLargeSection(const Triple &TargetTriple,
GlobalVariable &GV);
+/// Returns true if \p F calls llvm.sanitizer.instrumentation.begin or
+/// llvm.sanitizer.instrumentation.end.
+LLVM_ABI bool hasSanitizerInstrumentationRegions(const Function &F);
+
+/// Compute the instructions of \p F that lie inside an
+/// llvm.sanitizer.instrumentation.begin()/end() region, and add them to
+/// \p InRegion.
+///
+/// This is a *must* dataflow problem: an instruction is in a region only if
+/// *every* path from the function entry to it entered a region and did not
+/// leave it again, so an in-region instruction is always dominated by a
+/// begin(). Unbalanced regions are under-approximated, which agrees with
+/// objtool requiring them to be balanced on all paths. The lattice value is
+/// the nesting depth rather than a flag, so that in begin; begin; end; X; end
+/// the instruction X is still in a region.
+LLVM_ABI void computeSanitizerInstrumentationRegions(
+ const Function &F, SmallPtrSetImpl<const Instruction *> &InRegion);
+
// Insert GCOV profiling instrumentation
struct GCOVOptions {
LLVM_ABI static GCOVOptions getDefault();
diff --git a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
index 11e5ac36fc569..b609f61077c99 100644
--- a/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp
@@ -2533,6 +2533,11 @@ bool IRTranslator::translateKnownIntrinsic(const CallInst &CI, Intrinsic::ID ID,
case Intrinsic::experimental_noalias_scope_decl:
case Intrinsic::var_annotation:
case Intrinsic::sideeffect:
+ // The sanitizer markers are consumed by the sanitizer passes. They can still
+ // reach codegen when no such pass runs (or it does not look at a given
+ // function), in which case they have no effect.
+ case Intrinsic::sanitizer_instrumentation_begin:
+ case Intrinsic::sanitizer_instrumentation_end:
// Discard annotate attributes, assumptions, and artificial side-effects.
return true;
case Intrinsic::read_volatile_register:
diff --git a/llvm/lib/CodeGen/SelectionDAG/FastISel.cpp b/llvm/lib/CodeGen/SelectionDAG/FastISel.cpp
index 38c425e5b529b..b9c691e7ae4db 100644
--- a/llvm/lib/CodeGen/SelectionDAG/FastISel.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/FastISel.cpp
@@ -1385,6 +1385,8 @@ bool FastISel::selectIntrinsicCall(const IntrinsicInst *II) {
case Intrinsic::lifetime_end:
// The donothing intrinsic does, well, nothing.
case Intrinsic::donothing:
+ case Intrinsic::sanitizer_instrumentation_begin:
+ case Intrinsic::sanitizer_instrumentation_end:
// Neither does the sideeffect intrinsic.
case Intrinsic::sideeffect:
// Neither does the assume intrinsic; it's also OK not to codegen its operand.
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index dea008fd252a9..3923516537c67 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -7963,6 +7963,8 @@ void SelectionDAGBuilder::visitIntrinsicCall(const CallInst &I,
return;
}
case Intrinsic::donothing:
+ case Intrinsic::sanitizer_instrumentation_begin:
+ case Intrinsic::sanitizer_instrumentation_end:
case Intrinsic::seh_try_begin:
case Intrinsic::seh_scope_begin:
case Intrinsic::seh_try_end:
diff --git a/llvm/lib/Transforms/Utils/Instrumentation.cpp b/llvm/lib/Transforms/Utils/Instrumentation.cpp
index 3e43b85a11a3c..f44f47ec9aca9 100644
--- a/llvm/lib/Transforms/Utils/Instrumentation.cpp
+++ b/llvm/lib/Transforms/Utils/Instrumentation.cpp
@@ -12,6 +12,8 @@
//===----------------------------------------------------------------------===//
#include "llvm/Transforms/Utils/Instrumentation.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/IR/CFG.h"
#include "llvm/IR/DiagnosticInfo.h"
#include "llvm/IR/DiagnosticPrinter.h"
#include "llvm/IR/IntrinsicInst.h"
@@ -121,3 +123,80 @@ void llvm::setGlobalVariableLargeSection(const Triple &TargetTriple,
return;
GV.setCodeModel(CodeModel::Large);
}
+
+static bool isSanitizerInstrumentationMarker(const Instruction &I) {
+ const auto *II = dyn_cast<IntrinsicInst>(&I);
+ return II &&
+ (II->getIntrinsicID() == Intrinsic::sanitizer_instrumentation_begin ||
+ II->getIntrinsicID() == Intrinsic::sanitizer_instrumentation_end);
+}
+
+bool llvm::hasSanitizerInstrumentationRegions(const Function &F) {
+ for (const BasicBlock &BB : F)
+ for (const Instruction &I : BB)
+ if (isSanitizerInstrumentationMarker(I))
+ return true;
+ return false;
+}
+
+void llvm::computeSanitizerInstrumentationRegions(
+ const Function &F, SmallPtrSetImpl<const Instruction *> &InRegion) {
+ // Lattice top: this block has not been reached by the analysis yet, so it
+ // does not constrain the depth of its successors.
+ static constexpr unsigned UnreachedDepth = ~0U;
+
+ DenseMap<const BasicBlock *, unsigned> DepthIn;
+ DenseMap<const BasicBlock *, unsigned> DepthOut;
+ const BasicBlock *Entry = &F.getEntryBlock();
+ for (const BasicBlock &BB : F) {
+ DepthIn[&BB] = UnreachedDepth;
+ DepthOut[&BB] = UnreachedDepth;
+ }
+ DepthIn[Entry] = 0;
+
+ // Both maps only ever decrease and are bounded below by zero, so this
+ // fixed-point iteration terminates.
+ bool Changed = true;
+ while (Changed) {
+ Changed = false;
+ for (const BasicBlock &BB : F) {
+ if (&BB != Entry) {
+ unsigned In = UnreachedDepth;
+ for (const BasicBlock *Pred : predecessors(&BB))
+ In = std::min(In, DepthOut[Pred]);
+ if (In < DepthIn[&BB]) {
+ DepthIn[&BB] = In;
+ Changed = true;
+ }
+ }
+
+ unsigned Depth = DepthIn[&BB];
+ if (Depth == UnreachedDepth)
+ continue;
+
+ for (const Instruction &I : BB) {
+ if (const auto *II = dyn_cast<IntrinsicInst>(&I)) {
+ if (II->getIntrinsicID() ==
+ Intrinsic::sanitizer_instrumentation_begin)
+ ++Depth;
+ else if (II->getIntrinsicID() ==
+ Intrinsic::sanitizer_instrumentation_end &&
+ Depth)
+ // Saturate at zero: an unbalanced end() is malformed input, and
+ // wrapping around would spuriously instrument the rest of the
+ // function.
+ --Depth;
+ }
+ if (Depth > 0)
+ InRegion.insert(&I);
+ else
+ InRegion.erase(&I);
+ }
+
+ if (Depth < DepthOut[&BB]) {
+ DepthOut[&BB] = Depth;
+ Changed = true;
+ }
+ }
+ }
+}
diff --git a/llvm/test/CodeGen/Generic/sanitizer-instrumentation-intrinsics.ll b/llvm/test/CodeGen/Generic/sanitizer-instrumentation-intrinsics.ll
new file mode 100644
index 0000000000000..e3de9022ad54f
--- /dev/null
+++ b/llvm/test/CodeGen/Generic/sanitizer-instrumentation-intrinsics.ll
@@ -0,0 +1,19 @@
+; RUN: llc -O0 < %s | FileCheck %s
+; RUN: llc -O2 < %s | FileCheck %s
+; RUN: llc -O0 -fast-isel < %s | FileCheck %s
+
+; Verify that the backend lowers the sanitizer region intrinsics to nothing and
+; does not crash when compiling a module where no sanitizer pass ran.
+
+declare void @llvm.sanitizer.instrumentation.begin()
+declare void @llvm.sanitizer.instrumentation.end()
+
+define void @test_sanitizer_intrinsics() {
+; CHECK-LABEL: test_sanitizer_intrinsics:
+; CHECK: # %bb.0:
+; CHECK-NEXT: ret{{[ql]?}}
+entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.sanitizer.instrumentation.end()
+ ret void
+}
diff --git a/llvm/test/CodeGen/X86/sanitizer-instrumentation-intrinsics.ll b/llvm/test/CodeGen/X86/sanitizer-instrumentation-intrinsics.ll
new file mode 100644
index 0000000000000..b52aea91a03b1
--- /dev/null
+++ b/llvm/test/CodeGen/X86/sanitizer-instrumentation-intrinsics.ll
@@ -0,0 +1,20 @@
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -O0 < %s | FileCheck %s
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -O0 -fast-isel < %s | FileCheck %s
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -O0 -global-isel -global-isel-abort=1 < %s | FileCheck %s
+
+; The sanitizer region intrinsics are consumed by sanitizer passes, but they can
+; still reach codegen when no such pass runs. All three instruction selectors
+; have to drop them instead of failing to select.
+
+declare void @llvm.sanitizer.instrumentation.begin()
+declare void @llvm.sanitizer.instrumentation.end()
+
+define void @test_sanitizer_intrinsics() {
+; CHECK-LABEL: test_sanitizer_intrinsics:
+; CHECK-NOT: call
+; CHECK: retq
+entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.sanitizer.instrumentation.end()
+ ret void
+}
diff --git a/llvm/test/Transforms/PhaseOrdering/sanitizer-instrumentation-markers.ll b/llvm/test/Transforms/PhaseOrdering/sanitizer-instrumentation-markers.ll
new file mode 100644
index 0000000000000..453d336a4494b
--- /dev/null
+++ b/llvm/test/Transforms/PhaseOrdering/sanitizer-instrumentation-markers.ll
@@ -0,0 +1,86 @@
+; RUN: opt -passes='default<O2>' -S < %s | FileCheck %s
+
+; The sanitizer instrumentation region markers may access any memory that is
+; visible outside the function, so the optimizer must not move accesses to such
+; memory into or out of a region, and must not merge, hoist, sink or delete the
+; markers themselves.
+
+declare void @llvm.sanitizer.instrumentation.begin()
+declare void @llvm.sanitizer.instrumentation.end()
+declare void @callee(i32)
+
+define i32 @no_cse_across_markers(ptr %p) disable_sanitizer_instrumentation {
+; CHECK-LABEL: define i32 @no_cse_across_markers(
+; CHECK: %a = load i32, ptr %p
+; CHECK-NEXT: tail call void @llvm.sanitizer.instrumentation.begin()
+; CHECK-NEXT: %b = load i32, ptr %p
+; CHECK-NEXT: store i32 1, ptr %p
+; CHECK-NEXT: tail call void @llvm.sanitizer.instrumentation.end()
+; CHECK-NEXT: store i32 2, ptr %p
+ %a = load i32, ptr %p
+ call void @llvm.sanitizer.instrumentation.begin()
+ %b = load i32, ptr %p
+ store i32 1, ptr %p
+ call void @llvm.sanitizer.instrumentation.end()
+ store i32 2, ptr %p
+ %r = add i32 %a, %b
+ ret i32 %r
+}
+
+define void @no_licm_out_of_region(ptr %p, i32 %n) disable_sanitizer_instrumentation {
+; CHECK-LABEL: define void @no_licm_out_of_region(
+; CHECK: loop:
+; CHECK: tail call void @llvm.sanitizer.instrumentation.begin()
+; CHECK-NEXT: %v = load i32, ptr %p
+; CHECK-NEXT: tail call void @callee(i32 %v)
+; CHECK-NEXT: tail call void @llvm.sanitizer.instrumentation.end()
+entry:
+ br label %loop
+loop:
+ %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+ call void @llvm.sanitizer.instrumentation.begin()
+ %v = load i32, ptr %p
+ call void @callee(i32 %v)
+ call void @llvm.sanitizer.instrumentation.end()
+ %i.next = add i32 %i, 1
+ %c = icmp slt i32 %i.next, %n
+ br i1 %c, label %loop, label %exit
+exit:
+ ret void
+}
+
+define void @no_merge(i1 %c, i32 %x) disable_sanitizer_instrumentation {
+; CHECK-LABEL: define void @no_merge(
+; CHECK: a:
+; CHECK-NEXT: tail call void @llvm.sanitizer.instrumentation.begin()
+; CHECK-NEXT: tail call void @callee(i32 %x)
+; CHECK-NEXT: tail call void @llvm.sanitizer.instrumentation.end()
+; CHECK: b:
+; CHECK-NEXT: tail call void @llvm.sanitizer.instrumentation.begin()
+; CHECK-NEXT: tail call void @callee(i32 %x)
+; CHECK-NEXT: tail call void @llvm.sanitizer.instrumentation.end()
+entry:
+ br i1 %c, label %a, label %b
+a:
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @callee(i32 %x)
+ call void @llvm.sanitizer.instrumentation.end()
+ br label %exit
+b:
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @callee(i32 %x)
+ call void @llvm.sanitizer.instrumentation.end()
+ br label %exit
+exit:
+ ret void
+}
+
+define void @empty_region() disable_sanitizer_instrumentation {
+; CHECK-LABEL: define void @empty_region(
+; CHECK-NEXT: tail call void @llvm.sanitizer.instrumentation.begin()
+; CHECK-NEXT: tail call void @llvm.sanitizer.instrumentation.end()
+; CHECK-NEXT: ret void
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.sanitizer.instrumentation.end()
+ ret void
+}
diff --git a/llvm/unittests/Transforms/Utils/CMakeLists.txt b/llvm/unittests/Transforms/Utils/CMakeLists.txt
index 7d649f25c830a..b0b0473b9d256 100644
--- a/llvm/unittests/Transforms/Utils/CMakeLists.txt
+++ b/llvm/unittests/Transforms/Utils/CMakeLists.txt
@@ -28,6 +28,7 @@ add_llvm_unittest(UtilsTests
MemTransferLowering.cpp
ModuleUtilsTest.cpp
ScalarEvolutionExpanderTest.cpp
+ SanitizerRegionsTest.cpp
SizeOptsTest.cpp
SSAUpdaterBulkTest.cpp
UnrollLoopTest.cpp
diff --git a/llvm/unittests/Transforms/Utils/SanitizerRegionsTest.cpp b/llvm/unittests/Transforms/Utils/SanitizerRegionsTest.cpp
new file mode 100644
index 0000000000000..e2fbe466f0ee5
--- /dev/null
+++ b/llvm/unittests/Transforms/Utils/SanitizerRegionsTest.cpp
@@ -0,0 +1,192 @@
+//===- SanitizerRegionsTest.cpp - Unit tests for sanitizer regions --------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ADT/SmallPtrSet.h"
+#include "llvm/AsmParser/Parser.h"
+#include "llvm/IR/InstIterator.h"
+#include "llvm/IR/LLVMContext.h"
+#include "llvm/IR/Module.h"
+#include "llvm/Support/SourceMgr.h"
+#include "llvm/Transforms/Utils/Instrumentation.h"
+#include "gtest/gtest.h"
+
+using namespace llvm;
+
+namespace {
+
+// Every test function loads from %p into values named "in*" for instructions
+// expected to be in a region and "out*" for the rest.
+class SanitizerRegionsTest : public testing::Test {
+protected:
+ LLVMContext C;
+ std::unique_ptr<Module> M;
+
+ Function *parse(StringRef Body) {
+ std::string IR = (Twine(R"(
+ declare void @llvm.sanitizer.instrumentation.begin()
+ declare void @llvm.sanitizer.instrumentation.end()
+ declare void @callee()
+ define void @f(ptr %p, i1 %c) {
+ )") + Body + "}\n")
+ .str();
+ SMDiagnostic Err;
+ M = parseAssemblyString(IR, Err, C);
+ if (!M) {
+ Err.print("SanitizerRegionsTest", errs());
+ return nullptr;
+ }
+ return M->getFunction("f");
+ }
+
+ void expectRegions(StringRef Body, bool ExpectHasRegions = true) {
+ Function *F = parse(Body);
+ ASSERT_TRUE(F);
+ EXPECT_EQ(hasSanitizerInstrumentationRegions(*F), ExpectHasRegions);
+ SmallPtrSet<const Instruction *, 16> InRegion;
+ computeSanitizerInstrumentationRegions(*F, InRegion);
+ unsigned Named = 0;
+ for (const Instruction &I : instructions(*F)) {
+ if (!I.hasName())
+ continue;
+ ++Named;
+ EXPECT_EQ(InRegion.contains(&I), I.getName().starts_with("in"))
+ << I.getName().str();
+ }
+ EXPECT_GT(Named, 0u);
+ }
+};
+
+TEST_F(SanitizerRegionsTest, NoRegions) {
+ expectRegions(R"(
+ %out = load i32, ptr %p
+ ret void
+ )",
+ /*ExpectHasRegions=*/false);
+}
+
+TEST_F(SanitizerRegionsTest, Straight) {
+ expectRegions(R"(
+ %out1 = load i32, ptr %p
+ call void @llvm.sanitizer.instrumentation.begin()
+ %in = load i32, ptr %p
+ call void @callee()
+ call void @llvm.sanitizer.instrumentation.end()
+ %out2 = load i32, ptr %p
+ ret void
+ )");
+}
+
+TEST_F(SanitizerRegionsTest, Nested) {
+ expectRegions(R"(
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.sanitizer.instrumentation.begin()
+ %in1 = load i32, ptr %p
+ call void @llvm.sanitizer.instrumentation.end()
+ %in2 = load i32, ptr %p
+ call void @llvm.sanitizer.instrumentation.end()
+ %out = load i32, ptr %p
+ ret void
+ )");
+}
+
+TEST_F(SanitizerRegionsTest, UnmatchedEnd) {
+ expectRegions(R"(
+ call void @llvm.sanitizer.instrumentation.end()
+ %out1 = load i32, ptr %p
+ call void @llvm.sanitizer.instrumentation.begin()
+ %in = load i32, ptr %p
+ call void @llvm.sanitizer.instrumentation.end()
+ %out2 = load i32, ptr %p
+ ret void
+ )");
+}
+
+// A block that can be reached without passing a begin() is not in a region.
+TEST_F(SanitizerRegionsTest, BeginOnOnePath) {
+ expectRegions(R"(
+ entry:
+ br i1 %c, label %then, label %join
+ then:
+ call void @llvm.sanitizer.instrumentation.begin()
+ %in = load i32, ptr %p
+ br label %join
+ join:
+ %out = load i32, ptr %p
+ ret void
+ )");
+}
+
+// Likewise for a block that can be reached after an end().
+TEST_F(SanitizerRegionsTest, EndOnOnePath) {
+ expectRegions(R"(
+ entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ %in1 = load i32, ptr %p
+ br i1 %c, label %then, label %join
+ then:
+ call void @llvm.sanitizer.instrumentation.end()
+ %out1 = load i32, ptr %p
+ br label %join
+ join:
+ %out2 = load i32, ptr %p
+ ret void
+ )");
+}
+
+TEST_F(SanitizerRegionsTest, BalancedOnBothPaths) {
+ expectRegions(R"(
+ entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ br i1 %c, label %a, label %b
+ a:
+ %in1 = load i32, ptr %p
+ br label %join
+ b:
+ %in2 = load i32, ptr %p
+ br label %join
+ join:
+ %in3 = load i32, ptr %p
+ call void @llvm.sanitizer.instrumentation.end()
+ %out = load i32, ptr %p
+ ret void
+ )");
+}
+
+// A loop whose body leaves the region on the back edge.
+TEST_F(SanitizerRegionsTest, LoopLeavingRegion) {
+ expectRegions(R"(
+ entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ br label %loop
+ loop:
+ %out1 = load i32, ptr %p
+ call void @llvm.sanitizer.instrumentation.end()
+ br i1 %c, label %loop, label %exit
+ exit:
+ %out2 = load i32, ptr %p
+ ret void
+ )");
+}
+
+// A loop entirely inside a region.
+TEST_F(SanitizerRegionsTest, LoopInsideRegion) {
+ expectRegions(R"(
+ entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ br label %loop
+ loop:
+ %in1 = load i32, ptr %p
+ br i1 %c, label %loop, label %exit
+ exit:
+ %in2 = load i32, ptr %p
+ call void @llvm.sanitizer.instrumentation.end()
+ ret void
+ )");
+}
+
+} // namespace
>From d3801b65b70bcd489328b297091e0f86a2e3a481 Mon Sep 17 00:00:00 2001
From: Alexander Potapenko <glider at google.com>
Date: Mon, 5 Oct 2026 14:04:37 +0000
Subject: [PATCH 2/5] [MSan] Support sanitizer instrumentation regions in KMSAN
The Linux kernel marks functions that must not be instrumented as
`noinstr`, which under KMSAN means `disable_sanitizer_instrumentation`.
Such functions still call instrumented code between the kernel's
`instrumentation_begin()` and `instrumentation_end()` markers. Because
the caller is not instrumented, it never writes parameter shadow, and
the callee reads whatever the last instrumented caller left in the
per-context `param_tls`. This produces false positive KMSAN reports on
the kernel entry paths, e.g.
https://lore.kernel.org/all/69e7ee1f.a00a0220.17a17.001d.GAE@google.com/T/#u
Implement `llvm.sanitizer.instrumentation.begin`/`end` regions in
MemorySanitizer in kernel mode: in `disable_sanitizer_instrumentation`
functions, fetch the context state at `begin`, and propagate shadow and
set up parameter shadow for calls inside regions, as in a fully
instrumented function. Code outside regions stays uninstrumented; values
computed there are treated as initialized when used inside a region,
and memory written there keeps its previous shadow. Memory intrinsics
outside regions are not lowered to `__msan_mem*`, and vararg
bookkeeping is skipped in region-only functions. The context state slot
is null-initialized in the entry block; the must-analysis in
`computeSanitizerInstrumentationRegions()` guarantees that every
in-region use is dominated by a `begin` that stores to it.
The region markers are left in place for the sanitizer passes that
run after MemorySanitizer; codegen drops them.
Clang now lowers the region builtins to the intrinsics under
`-fsanitize=kernel-memory`. Userspace MSan ignores the intrinsics.
Tests: `llvm/test/Instrumentation/MemorySanitizer/kmsan-*.ll`,
`clang/test/CodeGen/sanitizer-instrumentation-builtins.c`,
`clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c`.
---
clang/docs/MemorySanitizer.md | 8 +
clang/lib/CodeGen/CGBuiltin.cpp | 2 +-
.../sanitizer-instrumentation-builtins.c | 27 +-
...nitizer-instrumentation-regions-pipeline.c | 39 +++
llvm/docs/LangRef.md | 2 +
.../Instrumentation/MemorySanitizer.cpp | 292 +++++++++++++++---
.../MemorySanitizer/kmsan-intrinsics.ll | 138 +++++++++
.../MemorySanitizer/kmsan-region-dataflow.ll | 124 ++++++++
.../kmsan-region-mem-intrinsics.ll | 41 +++
.../MemorySanitizer/kmsan-region-userspace.ll | 26 ++
.../MemorySanitizer/kmsan-region-varargs.ll | 36 +++
11 files changed, 684 insertions(+), 51 deletions(-)
create mode 100644 clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c
create mode 100644 llvm/test/Instrumentation/MemorySanitizer/kmsan-intrinsics.ll
create mode 100644 llvm/test/Instrumentation/MemorySanitizer/kmsan-region-dataflow.ll
create mode 100644 llvm/test/Instrumentation/MemorySanitizer/kmsan-region-mem-intrinsics.ll
create mode 100644 llvm/test/Instrumentation/MemorySanitizer/kmsan-region-userspace.ll
create mode 100644 llvm/test/Instrumentation/MemorySanitizer/kmsan-region-varargs.ll
diff --git a/clang/docs/MemorySanitizer.md b/clang/docs/MemorySanitizer.md
index c6d5d337952de..88f8811ab6117 100644
--- a/clang/docs/MemorySanitizer.md
+++ b/clang/docs/MemorySanitizer.md
@@ -94,6 +94,14 @@ positives and therefore should be used with care, and only if absolutely
required; for example for certain code that cannot tolerate any instrumentation
and resulting side-effects. This attribute overrides `no_sanitize("memory")`.
+In the context of KernelMemorySanitizer (KMSAN), this attribute skips the standard
+function-level ABI prologue and MSan instrumentation. However,
+`__builtin_sanitizer_instrumentation_begin()` and
+`__builtin_sanitizer_instrumentation_end()` can be used within a function marked
+with this attribute to selectively enable instrumentation for a specific region
+of code. Values computed outside such a region are treated as initialized when
+used inside it, and memory written outside a region keeps its previous shadow.
+
### Interaction of Inlining with Disabling Sanitizer Instrumentation
- A `no_sanitize` function will not be inlined heuristically by the compiler into a sanitized function.
diff --git a/clang/lib/CodeGen/CGBuiltin.cpp b/clang/lib/CodeGen/CGBuiltin.cpp
index 71bdaefb57ac9..aa8fbca7caecc 100644
--- a/clang/lib/CodeGen/CGBuiltin.cpp
+++ b/clang/lib/CodeGen/CGBuiltin.cpp
@@ -3913,7 +3913,7 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
// Only emit the region markers for sanitizers that honour them. Elsewhere
// they would just be optimization barriers that no pass ever removes. This
// lets code guard its uses on __has_builtin alone.
- SanitizerMask Consumers;
+ SanitizerMask Consumers = SanitizerKind::KernelMemory;
if (getLangOpts().Sanitize.hasOneOf(Consumers))
Builder.CreateCall(CGM.getIntrinsic(
BuiltinIDIfNoAsmLabel ==
diff --git a/clang/test/CodeGen/sanitizer-instrumentation-builtins.c b/clang/test/CodeGen/sanitizer-instrumentation-builtins.c
index 6fef449edd681..65f36048637cf 100644
--- a/clang/test/CodeGen/sanitizer-instrumentation-builtins.c
+++ b/clang/test/CodeGen/sanitizer-instrumentation-builtins.c
@@ -1,10 +1,15 @@
-// Check that the sanitizer region markers are available, and that they lower to
-// nothing without a sanitizer that consumes them.
+// Check that the sanitizer region markers lower to their intrinsics under
+// -fsanitize=kernel-memory, and to nothing without a sanitizer that consumes
+// them.
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -fsanitize=kernel-memory \
+// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s
// RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -disable-llvm-passes \
-// RUN: -o - %s | FileCheck %s
+// RUN: -o - %s | FileCheck %s --check-prefix=NONE
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -fsanitize=address \
+// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s --check-prefix=NONE
// RUN: %clang_cc1 -triple x86_64-linux-gnu -fsanitize=thread \
-// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s
+// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s --check-prefix=NONE
#if !__has_builtin(__builtin_sanitizer_instrumentation_begin)
#error "__builtin_sanitizer_instrumentation_begin not available"
@@ -23,5 +28,15 @@ void noinstr_function(int x) {
}
// CHECK-LABEL: define {{.*}}void @noinstr_function(
-// CHECK-NOT: llvm.sanitizer.instrumentation
-// CHECK: }
+// CHECK: call void @llvm.sanitizer.instrumentation.begin()
+// CHECK: call void @instrumented_callee(
+// CHECK: call void @llvm.sanitizer.instrumentation.end()
+
+// The markers are optimization barriers, so they must not be marked readnone
+// or speculatable; only the default attributes plus nomerge are expected.
+// CHECK: declare void @llvm.sanitizer.instrumentation.begin() #[[ATTR:[0-9]+]]
+// CHECK: attributes #[[ATTR]] = { nocallback nofree nomerge nosync nounwind willreturn }
+
+// NONE-LABEL: define {{.*}}void @noinstr_function(
+// NONE-NOT: llvm.sanitizer.instrumentation
+// NONE: }
diff --git a/clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c b/clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c
new file mode 100644
index 0000000000000..8ec56d40cb580
--- /dev/null
+++ b/clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c
@@ -0,0 +1,39 @@
+// Check that, after the full -O2 pipeline, each sanitizer instruments the code
+// inside a region of a function it otherwise does not instrument, and nothing
+// else. The region markers come from an always_inline helper, as in the Linux
+// kernel. Debug info is enabled so that the verifier checks that the
+// instrumentation calls get debug locations.
+
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -O2 -debug-info-kind=line-tables-only \
+// RUN: -fsanitize=kernel-memory -emit-llvm -o - %s \
+// RUN: | FileCheck %s --check-prefixes=CHECK,KMSAN
+
+static inline __attribute__((always_inline)) void instrumentation_begin(void) {
+ __builtin_sanitizer_instrumentation_begin();
+}
+
+static inline __attribute__((always_inline)) void instrumentation_end(void) {
+ __builtin_sanitizer_instrumentation_end();
+}
+
+void instrumented_callee(int);
+
+__attribute__((noinline, disable_sanitizer_instrumentation))
+void noinstr_function(int *p, int x) {
+ int v = *p + (x == 42);
+ instrumentation_begin();
+ if (x == 7)
+ instrumented_callee(*p + v);
+ instrumentation_end();
+ *p = v;
+}
+
+// CHECK-LABEL: define {{.*}}void @noinstr_function(
+// CHECK-NOT: call {{.*}}@__{{msan|asan|tsan|sanitizer_cov}}
+// CHECK: call void @llvm.sanitizer.instrumentation.begin()
+// KMSAN: call ptr @__msan_get_context_state()
+// KMSAN: call { ptr, ptr } @__msan_metadata_ptr_for_load_4(
+// CHECK: call void @instrumented_callee(
+// CHECK: call void @llvm.sanitizer.instrumentation.end()
+// CHECK-NOT: call {{.*}}@__{{msan|asan|tsan|sanitizer_cov}}
+// CHECK: ret void
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index 524d88e461dbf..0b08303ad3052 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -30510,3 +30510,5 @@ Inside a region, a sanitizer instruments the code as if the function were instru
Regions nest: `llvm.sanitizer.instrumentation.begin` increments a region nesting depth and `llvm.sanitizer.instrumentation.end` decrements it, so an inner `end` does not terminate an enclosing region. An unmatched `end` is ignored rather than making the depth negative.
An instruction is considered to be inside a region only if *every* path from the function entry to it is inside a region, i.e. region membership is an intersection over predecessors. This makes an in-region instruction always dominated by a `llvm.sanitizer.instrumentation.begin`, so any per-function sanitizer state set up there is available. A region that is entered on some paths but not others is therefore instrumented only where it is unambiguous.
+
+MemorySanitizer supports regions in kernel mode (KMSAN), in functions marked with `disable_sanitizer_instrumentation`. The function-level ABI prologue is skipped, the MSan context state is fetched at `llvm.sanitizer.instrumentation.begin`, and shadow propagation and parameter shadow setup are only performed inside regions. Memory written outside a region keeps whatever shadow it had, and values computed outside a region are treated as fully initialized when they are used inside one.
diff --git a/llvm/lib/Transforms/Instrumentation/MemorySanitizer.cpp b/llvm/lib/Transforms/Instrumentation/MemorySanitizer.cpp
index 8691849eead22..452e98f897aba 100644
--- a/llvm/lib/Transforms/Instrumentation/MemorySanitizer.cpp
+++ b/llvm/lib/Transforms/Instrumentation/MemorySanitizer.cpp
@@ -170,11 +170,13 @@
#include "llvm/IR/Constants.h"
#include "llvm/IR/DataLayout.h"
#include "llvm/IR/DerivedTypes.h"
+#include "llvm/IR/DiagnosticInfo.h"
#include "llvm/IR/Function.h"
#include "llvm/IR/GlobalValue.h"
#include "llvm/IR/GlobalVariable.h"
#include "llvm/IR/IRBuilder.h"
#include "llvm/IR/InlineAsm.h"
+#include "llvm/IR/InstIterator.h"
#include "llvm/IR/InstVisitor.h"
#include "llvm/IR/InstrTypes.h"
#include "llvm/IR/Instruction.h"
@@ -1247,10 +1249,55 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
SmallVector<std::pair<IntrinsicInst *, AllocaInst *>, 16> LifetimeStartList;
SmallVector<StoreInst *, 16> StoreList;
int64_t SplittableBlocksCount = 0;
+ bool IsRegionInstrumentedOnly = false;
+
+ Value *ContextStateAlloca = nullptr;
+
+ Value *getKmsanTLS(IRBuilderBase &IRB, int Index, const Twine &Name) {
+ Value *Ctx = IRB.CreateLoad(MS.PtrTy, ContextStateAlloca);
+ Constant *Zero = IRB.getInt32(0);
+ return IRB.CreateGEP(MS.MsanContextStateTy, Ctx,
+ {Zero, IRB.getInt32(Index)}, Name);
+ }
+
+ Value *getParamTLS(IRBuilderBase &IRB) {
+ return ContextStateAlloca ? getKmsanTLS(IRB, 0, "param_shadow")
+ : MS.ParamTLS;
+ }
+ Value *getRetvalTLS(IRBuilderBase &IRB) {
+ return ContextStateAlloca ? getKmsanTLS(IRB, 1, "retval_shadow")
+ : MS.RetvalTLS;
+ }
+ Value *getVAArgTLS(IRBuilderBase &IRB) {
+ return ContextStateAlloca ? getKmsanTLS(IRB, 2, "va_arg_shadow")
+ : MS.VAArgTLS;
+ }
+ Value *getVAArgOriginTLS(IRBuilderBase &IRB) {
+ return ContextStateAlloca ? getKmsanTLS(IRB, 3, "va_arg_origin")
+ : MS.VAArgOriginTLS;
+ }
+ Value *getVAArgOverflowSizeTLS(IRBuilderBase &IRB) {
+ return ContextStateAlloca ? getKmsanTLS(IRB, 4, "va_arg_overflow_size")
+ : MS.VAArgOverflowSizeTLS;
+ }
+ Value *getParamOriginTLS(IRBuilderBase &IRB) {
+ return ContextStateAlloca ? getKmsanTLS(IRB, 5, "param_origin")
+ : MS.ParamOriginTLS;
+ }
+ Value *getRetvalOriginTLS(IRBuilderBase &IRB) {
+ return ContextStateAlloca ? getKmsanTLS(IRB, 6, "retval_origin")
+ : MS.RetvalOriginTLS;
+ }
+
+ SmallPtrSet<const Instruction *, 16> InRegion;
MemorySanitizerVisitor(Function &F, MemorySanitizer &MS,
const TargetLibraryInfo &TLI)
: F(F), MS(MS), VAHelper(CreateVarArgHelper(F, MS, *this)), TLI(&TLI) {
+ IsRegionInstrumentedOnly =
+ MS.CompileKernel &&
+ F.hasFnAttribute(Attribute::DisableSanitizerInstrumentation);
+
bool SanitizeFunction =
F.hasFnAttribute(Attribute::SanitizeMemory) && !ClDisableChecks;
InsertChecks = SanitizeFunction;
@@ -1272,7 +1319,24 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
if (MS.CompileKernel) {
IRBuilder<> IRB(FnPrologueEnd);
- insertKmsanPrologue(IRB);
+ if (MS.TargetTriple.getArch() == Triple::systemz)
+ MS.MsanMetadataAlloca = IRB.CreateAlloca(MS.MsanMetadata, 0u);
+ // A region-instrumented function must not call into the runtime outside
+ // a region, so it cannot fetch the context in the prologue. Each begin()
+ // fetches it instead and keeps it in a stack slot for the region.
+ if (!IsRegionInstrumentedOnly) {
+ insertKmsanPrologue(IRB);
+ } else {
+ ContextStateAlloca =
+ IRB.CreateAlloca(MS.PtrTy, nullptr, "msan_context_state");
+ // No context is available until the first begin().
+ // computeRegionMembership() guarantees that every instruction that
+ // loads this slot is dominated by a begin() that stored into it, so
+ // this store is dead and will be removed. Keep it anyway: should that
+ // invariant ever be broken, faulting on a null pointer is a far better
+ // failure mode than corrupting memory through an uninitialized one.
+ IRB.CreateStore(Constant::getNullValue(MS.PtrTy), ContextStateAlloca);
+ }
}
LLVM_DEBUG(if (!InsertChecks) dbgs()
@@ -1612,7 +1676,8 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
LLVM_DEBUG(dbgs() << "DONE:\n" << F);
}
- // Returns the last instruction in the new prologue
+ /// Fetch the current KMSAN context and cache it for the rest of the
+ /// function in SSA values.
void insertKmsanPrologue(IRBuilder<> &IRB) {
Value *ContextState = IRB.CreateCall(MS.MsanGetContextStateFn, {});
Constant *Zero = IRB.getInt32(0);
@@ -1632,12 +1697,38 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
MS.RetvalOriginTLS =
IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
{Zero, IRB.getInt32(6)}, "retval_origin");
- if (MS.TargetTriple.getArch() == Triple::systemz)
- MS.MsanMetadataAlloca = IRB.CreateAlloca(MS.MsanMetadata, 0u);
+ }
+
+ /// Fetch the current KMSAN context and cache it in ContextStateAlloca. Used
+ /// at every llvm.sanitizer.instrumentation.begin() of a region-instrumented
+ /// function.
+ void loadKmsanContextState(IRBuilderBase &IRB) {
+ IRB.CreateStore(IRB.CreateCall(MS.MsanGetContextStateFn, {}),
+ ContextStateAlloca);
+ }
+
+ /// Compute, for every instruction of a region-instrumented function, whether
+ /// it belongs to an llvm.sanitizer.instrumentation.begin()/end() region.
+ ///
+ /// computeSanitizerInstrumentationRegions() solves this as a *must* dataflow
+ /// problem, which MSan relies on for the soundness of the cached context
+ /// state: every begin() stores the result of __msan_get_context_state() into
+ /// %msan_context_state, and in-region instrumentation loads the pointer back
+ /// from that slot. With a "may" (union) join, a block reachable both from
+ /// inside and from outside a region would be instrumented, and on the path
+ /// that never executed begin() the instrumentation would load an
+ /// uninitialized alloca and then read and *write* through the resulting wild
+ /// pointer. With the intersection join an in-region instruction is always
+ /// dominated by a begin(), so the slot is always initialized.
+ void computeRegionMembership() {
+ computeSanitizerInstrumentationRegions(F, InRegion);
}
/// Add MemorySanitizer instrumentation to a function.
bool runOnFunction() {
+ if (IsRegionInstrumentedOnly)
+ computeRegionMembership();
+
// Iterate all BBs in depth-first order and create shadow instructions
// for all instructions (where applicable).
// For PHI nodes we create dummy shadow PHIs which will be finalized later.
@@ -1646,11 +1737,22 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
// `visit` above only collects instructions. Process them after iterating
// CFG to avoid requirement on CFG transformations.
- for (Instruction *I : Instructions)
+ for (Instruction *I : Instructions) {
+ if (IsRegionInstrumentedOnly) {
+ bool State = InRegion.contains(I);
+ PropagateShadow = State;
+ InsertChecks = State;
+ }
InstVisitor<MemorySanitizerVisitor>::visit(*I);
+ }
// Finalize PHI nodes.
for (PHINode *PN : ShadowPHINodes) {
+ if (IsRegionInstrumentedOnly) {
+ bool State = InRegion.contains(PN);
+ PropagateShadow = State;
+ InsertChecks = State;
+ }
PHINode *PNS = cast<PHINode>(getShadow(PN));
PHINode *PNO = MS.TrackOrigins ? cast<PHINode>(getOrigin(PN)) : nullptr;
size_t NumValues = PN->getNumIncomingValues();
@@ -1667,14 +1769,17 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
// instrumenting only allocas.
if (ClHandleLifetimeIntrinsics) {
for (auto Item : LifetimeStartList) {
- instrumentAlloca(*Item.second, Item.first);
+ if (!IsRegionInstrumentedOnly || InRegion.contains(Item.first))
+ instrumentAlloca(*Item.second, Item.first);
AllocaSet.remove(Item.second);
}
}
// Poison the allocas for which we didn't instrument the corresponding
// lifetime intrinsics.
- for (AllocaInst *AI : AllocaSet)
- instrumentAlloca(*AI);
+ for (AllocaInst *AI : AllocaSet) {
+ if (!IsRegionInstrumentedOnly || InRegion.contains(AI))
+ instrumentAlloca(*AI);
+ }
// Insert shadow value checks.
materializeChecks();
@@ -1976,7 +2081,7 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
///
/// Shadow = ParamTLS+ArgOffset.
Value *getShadowPtrForArgument(IRBuilder<> &IRB, int ArgOffset) {
- return IRB.CreatePtrAdd(MS.ParamTLS,
+ return IRB.CreatePtrAdd(getParamTLS(IRB),
ConstantInt::get(MS.IntptrTy, ArgOffset), "_msarg");
}
@@ -1984,20 +2089,20 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
Value *getOriginPtrForArgument(IRBuilder<> &IRB, int ArgOffset) {
if (!MS.TrackOrigins)
return nullptr;
- return IRB.CreatePtrAdd(MS.ParamOriginTLS,
+ return IRB.CreatePtrAdd(getParamOriginTLS(IRB),
ConstantInt::get(MS.IntptrTy, ArgOffset),
"_msarg_o");
}
/// Compute the shadow address for a retval.
Value *getShadowPtrForRetval(IRBuilder<> &IRB) {
- return IRB.CreatePointerCast(MS.RetvalTLS, IRB.getPtrTy(0), "_msret");
+ return IRB.CreatePointerCast(getRetvalTLS(IRB), IRB.getPtrTy(0), "_msret");
}
/// Compute the origin address for a retval.
- Value *getOriginPtrForRetval() {
+ Value *getOriginPtrForRetval(IRBuilderBase &IRB) {
// We keep a single origin for the entire retval. Might be too optimistic.
- return MS.RetvalOriginTLS;
+ return getRetvalOriginTLS(IRB);
}
/// Set SV to be the shadow value for V.
@@ -2083,6 +2188,11 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
return AllOnes;
}
if (Argument *A = dyn_cast<Argument>(V)) {
+ if (IsRegionInstrumentedOnly) {
+ if (MS.TrackOrigins && !OriginMap.count(A))
+ setOrigin(A, getCleanOrigin());
+ return getCleanShadow(V);
+ }
// For arguments we compute the shadow on demand and store it in the map.
Value *&ShadowPtr = ShadowMap[V];
if (ShadowPtr)
@@ -2353,6 +2463,7 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
// Don't want to visit if we're in the prologue
if (isInPrologue(I))
return;
+
if (!DebugCounter::shouldExecute(DebugInstrumentInstruction)) {
LLVM_DEBUG(dbgs() << "Skipping instruction: " << I << "\n");
// We still need to set the shadow and origin to clean values.
@@ -2407,12 +2518,27 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
/// Stores the corresponding shadow and (optionally) origin.
/// Optionally, checks that the store address is fully defined.
void visitStoreInst(StoreInst &I) {
+ if (IsRegionInstrumentedOnly && !PropagateShadow) {
+ setShadow(&I, getCleanShadow(&I));
+ setOrigin(&I, getCleanOrigin());
+ // The shadow of the written memory is deliberately left as it was: the
+ // only way to update it is a KMSAN runtime call, which must not be
+ // emitted outside a region. A later in-region load of this location can
+ // therefore see a stale shadow; this is documented in LangRef.
+ return;
+ }
StoreList.push_back(&I);
if (ClCheckAccessAddress)
insertCheckShadowOf(I.getPointerOperand(), &I);
}
void handleCASOrRMW(Instruction &I) {
+ if (IsRegionInstrumentedOnly && !PropagateShadow) {
+ setShadow(&I, getCleanShadow(&I));
+ setOrigin(&I, getCleanOrigin());
+ return;
+ }
+
assert(isa<AtomicRMWInst>(I) || isa<AtomicCmpXchgInst>(I));
IRBuilder<> IRB(&I);
@@ -3374,7 +3500,18 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
/// of overlapping regions. So, we simply lower the intrinsic to a call.
///
/// Similar situation exists for memcpy and memset.
+ /// Memory intrinsics are replaced with calls into the KMSAN runtime. In a
+ /// function that is only instrumented inside regions such a call must not be
+ /// emitted outside one: instrumenting code outside a region is not what the
+ /// region markers ask for, and in noinstr kernel code the call leaves
+ /// .noinstr.text, which objtool rejects. Leave the intrinsic untouched.
+ bool skipMemIntrinsicOutsideRegion() const {
+ return IsRegionInstrumentedOnly && !PropagateShadow;
+ }
+
void visitMemMoveInst(MemMoveInst &I) {
+ if (skipMemIntrinsicOutsideRegion())
+ return;
getShadow(I.getArgOperand(1)); // Ensure shadow initialized
IRBuilder<> IRB(&I);
IRB.CreateCall(MS.MemmoveFn,
@@ -3398,6 +3535,8 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
/// __msan_memcpy(). Should this be wrong, such as when implementing memcpy()
/// itself, instrumentation should be disabled with the no_sanitize attribute.
void visitMemCpyInst(MemCpyInst &I) {
+ if (skipMemIntrinsicOutsideRegion())
+ return;
getShadow(I.getArgOperand(1)); // Ensure shadow initialized
IRBuilder<> IRB(&I);
IRB.CreateCall(MS.MemcpyFn,
@@ -3408,6 +3547,8 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
// Same as memcpy.
void visitMemSetInst(MemSetInst &I) {
+ if (skipMemIntrinsicOutsideRegion())
+ return;
IRBuilder<> IRB(&I);
IRB.CreateCall(
MS.MemsetFn,
@@ -7456,6 +7597,24 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
}
void visitIntrinsicInst(IntrinsicInst &I) {
+ switch (I.getIntrinsicID()) {
+ case Intrinsic::sanitizer_instrumentation_begin: {
+ // Only meaningful in a region-instrumented function: elsewhere the
+ // context is already live from the function prologue.
+ if (IsRegionInstrumentedOnly) {
+ NextNodeIRBuilder IRB(&I);
+ loadKmsanContextState(IRB);
+ }
+ // The region markers are shared by all sanitizers, so leave them for the
+ // passes that run after this one. They are dropped during codegen.
+ return;
+ }
+ case Intrinsic::sanitizer_instrumentation_end:
+ return;
+ default:
+ break;
+ }
+
if (maybeHandleCrossPlatformIntrinsic(I))
return;
@@ -7527,6 +7686,12 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
}
void visitCallBase(CallBase &CB) {
+ if (IsRegionInstrumentedOnly && !PropagateShadow) {
+ setShadow(&CB, getCleanShadow(&CB));
+ setOrigin(&CB, getCleanOrigin());
+ return;
+ }
+
assert(!CB.getMetadata(LLVMContext::MD_nosanitize));
if (CB.isInlineAsm()) {
// For inline asm (either a call to asm function, or callbr instruction),
@@ -7727,7 +7892,8 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
"_msret");
setShadow(&CB, RetvalShadow);
if (MS.TrackOrigins)
- setOrigin(&CB, IRBAfter.CreateLoad(MS.OriginTy, getOriginPtrForRetval()));
+ setOrigin(&CB,
+ IRBAfter.CreateLoad(MS.OriginTy, getOriginPtrForRetval(IRB)));
}
bool isAMustTailRetVal(Value *RetVal) {
@@ -7741,6 +7907,9 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
}
void visitReturnInst(ReturnInst &I) {
+ if (IsRegionInstrumentedOnly && !PropagateShadow)
+ return;
+
IRBuilder<> IRB(&I);
Value *RetVal = I.getReturnValue();
if (!RetVal)
@@ -7768,7 +7937,7 @@ struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
if (StoreShadow) {
IRB.CreateAlignedStore(Shadow, ShadowPtr, kShadowTLSAlignment);
if (MS.TrackOrigins && StoreOrigin)
- IRB.CreateStore(getOrigin(RetVal), getOriginPtrForRetval());
+ IRB.CreateStore(getOrigin(RetVal), getOriginPtrForRetval(IRB));
}
}
@@ -8166,14 +8335,15 @@ struct VarArgHelperBase : public VarArgHelper {
: F(F), MS(MS), MSV(MSV), VAListTagSize(VAListTagSize) {}
Value *getShadowAddrForVAArgument(IRBuilder<> &IRB, unsigned ArgOffset) {
- Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
+ Value *Base = IRB.CreatePointerCast(MSV.getVAArgTLS(IRB), MS.IntptrTy);
return IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
}
/// Compute the shadow address for a given va_arg.
Value *getShadowPtrForVAArgument(IRBuilder<> &IRB, unsigned ArgOffset) {
- return IRB.CreatePtrAdd(
- MS.VAArgTLS, ConstantInt::get(MS.IntptrTy, ArgOffset), "_msarg_va_s");
+ return IRB.CreatePtrAdd(MSV.getVAArgTLS(IRB),
+ ConstantInt::get(MS.IntptrTy, ArgOffset),
+ "_msarg_va_s");
}
/// Compute the shadow address for a given va_arg.
@@ -8190,7 +8360,7 @@ struct VarArgHelperBase : public VarArgHelper {
// getOriginPtrForVAArgument() is always called after
// getShadowPtrForVAArgument(), so __msan_va_arg_origin_tls can never
// overflow.
- return IRB.CreatePtrAdd(MS.VAArgOriginTLS,
+ return IRB.CreatePtrAdd(MSV.getVAArgOriginTLS(IRB),
ConstantInt::get(MS.IntptrTy, ArgOffset),
"_msarg_va_o");
}
@@ -8381,7 +8551,7 @@ struct VarArgAMD64Helper : public VarArgHelperBase {
}
Constant *OverflowSize =
ConstantInt::get(IRB.getInt64Ty(), OverflowOffset - AMD64FpEndOffset);
- IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS);
+ IRB.CreateStore(OverflowSize, MSV.getVAArgOverflowSizeTLS(IRB));
}
void finalizeInstrumentation() override {
@@ -8392,7 +8562,7 @@ struct VarArgAMD64Helper : public VarArgHelperBase {
// va_arg_tls somewhere in the function entry block.
IRBuilder<> IRB(MSV.FnPrologueEnd);
VAArgOverflowSize =
- IRB.CreateLoad(IRB.getInt64Ty(), MS.VAArgOverflowSizeTLS);
+ IRB.CreateLoad(IRB.getInt64Ty(), MSV.getVAArgOverflowSizeTLS(IRB));
Value *CopySize = IRB.CreateAdd(
ConstantInt::get(MS.IntptrTy, AMD64FpEndOffset), VAArgOverflowSize);
VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
@@ -8403,13 +8573,14 @@ struct VarArgAMD64Helper : public VarArgHelperBase {
Value *SrcSize = IRB.CreateBinaryIntrinsic(
Intrinsic::umin, CopySize,
ConstantInt::get(MS.IntptrTy, kParamTLSSize));
- IRB.CreateMemCpy(VAArgTLSCopy, kShadowTLSAlignment, MS.VAArgTLS,
+ IRB.CreateMemCpy(VAArgTLSCopy, kShadowTLSAlignment, MSV.getVAArgTLS(IRB),
kShadowTLSAlignment, SrcSize);
if (MS.TrackOrigins) {
VAArgTLSOriginCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
VAArgTLSOriginCopy->setAlignment(kShadowTLSAlignment);
IRB.CreateMemCpy(VAArgTLSOriginCopy, kShadowTLSAlignment,
- MS.VAArgOriginTLS, kShadowTLSAlignment, SrcSize);
+ MSV.getVAArgOriginTLS(IRB), kShadowTLSAlignment,
+ SrcSize);
}
}
@@ -8558,7 +8729,7 @@ struct VarArgAArch64Helper : public VarArgHelperBase {
}
Constant *OverflowSize =
ConstantInt::get(IRB.getInt64Ty(), OverflowOffset - AArch64VAEndOffset);
- IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS);
+ IRB.CreateStore(OverflowSize, MSV.getVAArgOverflowSizeTLS(IRB));
}
// Retrieve a va_list field of 'void*' size.
@@ -8584,7 +8755,7 @@ struct VarArgAArch64Helper : public VarArgHelperBase {
// va_arg_tls somewhere in the function entry block.
IRBuilder<> IRB(MSV.FnPrologueEnd);
VAArgOverflowSize =
- IRB.CreateLoad(IRB.getInt64Ty(), MS.VAArgOverflowSizeTLS);
+ IRB.CreateLoad(IRB.getInt64Ty(), MSV.getVAArgOverflowSizeTLS(IRB));
Value *CopySize = IRB.CreateAdd(
ConstantInt::get(MS.IntptrTy, AArch64VAEndOffset), VAArgOverflowSize);
VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
@@ -8595,7 +8766,7 @@ struct VarArgAArch64Helper : public VarArgHelperBase {
Value *SrcSize = IRB.CreateBinaryIntrinsic(
Intrinsic::umin, CopySize,
ConstantInt::get(MS.IntptrTy, kParamTLSSize));
- IRB.CreateMemCpy(VAArgTLSCopy, kShadowTLSAlignment, MS.VAArgTLS,
+ IRB.CreateMemCpy(VAArgTLSCopy, kShadowTLSAlignment, MSV.getVAArgTLS(IRB),
kShadowTLSAlignment, SrcSize);
}
@@ -8785,14 +8956,15 @@ struct VarArgPowerPC64Helper : public VarArgHelperBase {
ConstantInt::get(MS.IntptrTy, VAArgOffset - VAArgBase);
// Here using VAArgOverflowSizeTLS as VAArgSizeTLS to avoid creation of
// a new class member i.e. it is the total size of all VarArgs.
- IRB.CreateStore(TotalVAArgSize, MS.VAArgOverflowSizeTLS);
+ IRB.CreateStore(TotalVAArgSize, MSV.getVAArgOverflowSizeTLS(IRB));
}
void finalizeInstrumentation() override {
assert(!VAArgSize && !VAArgTLSCopy &&
"finalizeInstrumentation called twice");
IRBuilder<> IRB(MSV.FnPrologueEnd);
- VAArgSize = IRB.CreateLoad(IRB.getInt64Ty(), MS.VAArgOverflowSizeTLS);
+ VAArgSize =
+ IRB.CreateLoad(IRB.getInt64Ty(), MSV.getVAArgOverflowSizeTLS(IRB));
Value *CopySize = VAArgSize;
if (!VAStartInstrumentationList.empty()) {
@@ -8807,7 +8979,7 @@ struct VarArgPowerPC64Helper : public VarArgHelperBase {
Value *SrcSize = IRB.CreateBinaryIntrinsic(
Intrinsic::umin, CopySize,
ConstantInt::get(IRB.getInt64Ty(), kParamTLSSize));
- IRB.CreateMemCpy(VAArgTLSCopy, kShadowTLSAlignment, MS.VAArgTLS,
+ IRB.CreateMemCpy(VAArgTLSCopy, kShadowTLSAlignment, MSV.getVAArgTLS(IRB),
kShadowTLSAlignment, SrcSize);
}
@@ -8921,14 +9093,14 @@ struct VarArgPowerPC32Helper : public VarArgHelperBase {
ConstantInt::get(MS.IntptrTy, VAArgOffset - VAArgBase);
// Here using VAArgOverflowSizeTLS as VAArgSizeTLS to avoid creation of
// a new class member i.e. it is the total size of all VarArgs.
- IRB.CreateStore(TotalVAArgSize, MS.VAArgOverflowSizeTLS);
+ IRB.CreateStore(TotalVAArgSize, MSV.getVAArgOverflowSizeTLS(IRB));
}
void finalizeInstrumentation() override {
assert(!VAArgSize && !VAArgTLSCopy &&
"finalizeInstrumentation called twice");
IRBuilder<> IRB(MSV.FnPrologueEnd);
- VAArgSize = IRB.CreateLoad(MS.IntptrTy, MS.VAArgOverflowSizeTLS);
+ VAArgSize = IRB.CreateLoad(MS.IntptrTy, MSV.getVAArgOverflowSizeTLS(IRB));
Value *CopySize = VAArgSize;
if (!VAStartInstrumentationList.empty()) {
@@ -8943,7 +9115,7 @@ struct VarArgPowerPC32Helper : public VarArgHelperBase {
Value *SrcSize = IRB.CreateBinaryIntrinsic(
Intrinsic::umin, CopySize,
ConstantInt::get(MS.IntptrTy, kParamTLSSize));
- IRB.CreateMemCpy(VAArgTLSCopy, kShadowTLSAlignment, MS.VAArgTLS,
+ IRB.CreateMemCpy(VAArgTLSCopy, kShadowTLSAlignment, MSV.getVAArgTLS(IRB),
kShadowTLSAlignment, SrcSize);
}
@@ -9206,7 +9378,7 @@ struct VarArgSystemZHelper : public VarArgHelperBase {
}
Constant *OverflowSize = ConstantInt::get(
IRB.getInt64Ty(), OverflowOffset - SystemZOverflowOffset);
- IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS);
+ IRB.CreateStore(OverflowSize, MSV.getVAArgOverflowSizeTLS(IRB));
}
void copyRegSaveArea(IRBuilder<> &IRB, Value *VAListTag) {
@@ -9267,7 +9439,7 @@ struct VarArgSystemZHelper : public VarArgHelperBase {
// va_arg_tls somewhere in the function entry block.
IRBuilder<> IRB(MSV.FnPrologueEnd);
VAArgOverflowSize =
- IRB.CreateLoad(IRB.getInt64Ty(), MS.VAArgOverflowSizeTLS);
+ IRB.CreateLoad(IRB.getInt64Ty(), MSV.getVAArgOverflowSizeTLS(IRB));
Value *CopySize =
IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, SystemZOverflowOffset),
VAArgOverflowSize);
@@ -9279,13 +9451,14 @@ struct VarArgSystemZHelper : public VarArgHelperBase {
Value *SrcSize = IRB.CreateBinaryIntrinsic(
Intrinsic::umin, CopySize,
ConstantInt::get(MS.IntptrTy, kParamTLSSize));
- IRB.CreateMemCpy(VAArgTLSCopy, kShadowTLSAlignment, MS.VAArgTLS,
+ IRB.CreateMemCpy(VAArgTLSCopy, kShadowTLSAlignment, MSV.getVAArgTLS(IRB),
kShadowTLSAlignment, SrcSize);
if (MS.TrackOrigins) {
VAArgTLSOriginCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
VAArgTLSOriginCopy->setAlignment(kShadowTLSAlignment);
IRB.CreateMemCpy(VAArgTLSOriginCopy, kShadowTLSAlignment,
- MS.VAArgOriginTLS, kShadowTLSAlignment, SrcSize);
+ MSV.getVAArgOriginTLS(IRB), kShadowTLSAlignment,
+ SrcSize);
}
}
@@ -9361,14 +9534,14 @@ struct VarArgI386Helper : public VarArgHelperBase {
Constant *TotalVAArgSize = ConstantInt::get(MS.IntptrTy, VAArgOffset);
// Here using VAArgOverflowSizeTLS as VAArgSizeTLS to avoid creation of
// a new class member i.e. it is the total size of all VarArgs.
- IRB.CreateStore(TotalVAArgSize, MS.VAArgOverflowSizeTLS);
+ IRB.CreateStore(TotalVAArgSize, MSV.getVAArgOverflowSizeTLS(IRB));
}
void finalizeInstrumentation() override {
assert(!VAArgSize && !VAArgTLSCopy &&
"finalizeInstrumentation called twice");
IRBuilder<> IRB(MSV.FnPrologueEnd);
- VAArgSize = IRB.CreateLoad(MS.IntptrTy, MS.VAArgOverflowSizeTLS);
+ VAArgSize = IRB.CreateLoad(MS.IntptrTy, MSV.getVAArgOverflowSizeTLS(IRB));
Value *CopySize = VAArgSize;
if (!VAStartInstrumentationList.empty()) {
@@ -9382,7 +9555,7 @@ struct VarArgI386Helper : public VarArgHelperBase {
Value *SrcSize = IRB.CreateBinaryIntrinsic(
Intrinsic::umin, CopySize,
ConstantInt::get(MS.IntptrTy, kParamTLSSize));
- IRB.CreateMemCpy(VAArgTLSCopy, kShadowTLSAlignment, MS.VAArgTLS,
+ IRB.CreateMemCpy(VAArgTLSCopy, kShadowTLSAlignment, MSV.getVAArgTLS(IRB),
kShadowTLSAlignment, SrcSize);
}
@@ -9446,14 +9619,14 @@ struct VarArgGenericHelper : public VarArgHelperBase {
Constant *TotalVAArgSize = ConstantInt::get(MS.IntptrTy, VAArgOffset);
// Here using VAArgOverflowSizeTLS as VAArgSizeTLS to avoid creation of
// a new class member i.e. it is the total size of all VarArgs.
- IRB.CreateStore(TotalVAArgSize, MS.VAArgOverflowSizeTLS);
+ IRB.CreateStore(TotalVAArgSize, MSV.getVAArgOverflowSizeTLS(IRB));
}
void finalizeInstrumentation() override {
assert(!VAArgSize && !VAArgTLSCopy &&
"finalizeInstrumentation called twice");
IRBuilder<> IRB(MSV.FnPrologueEnd);
- VAArgSize = IRB.CreateLoad(MS.IntptrTy, MS.VAArgOverflowSizeTLS);
+ VAArgSize = IRB.CreateLoad(MS.IntptrTy, MSV.getVAArgOverflowSizeTLS(IRB));
Value *CopySize = VAArgSize;
if (!VAStartInstrumentationList.empty()) {
@@ -9467,7 +9640,7 @@ struct VarArgGenericHelper : public VarArgHelperBase {
Value *SrcSize = IRB.CreateBinaryIntrinsic(
Intrinsic::umin, CopySize,
ConstantInt::get(MS.IntptrTy, kParamTLSSize));
- IRB.CreateMemCpy(VAArgTLSCopy, kShadowTLSAlignment, MS.VAArgTLS,
+ IRB.CreateMemCpy(VAArgTLSCopy, kShadowTLSAlignment, MSV.getVAArgTLS(IRB),
kShadowTLSAlignment, SrcSize);
}
@@ -9521,6 +9694,15 @@ struct VarArgNoOpHelper : public VarArgHelper {
static VarArgHelper *CreateVarArgHelper(Function &Func, MemorySanitizer &Msan,
MemorySanitizerVisitor &Visitor) {
+ // A region-instrumented function is only instrumented between
+ // llvm.sanitizer.instrumentation.begin() and end(). VarArgHelpers emit their
+ // setup into the function prologue, which is outside every region: the MSan
+ // context state has not been fetched there yet, so that code would read and
+ // write through an uninitialized pointer. Leave varargs alone instead.
+ // sanitizeFunction() only gets here for such a function in kernel mode.
+ if (Func.hasFnAttribute(Attribute::DisableSanitizerInstrumentation))
+ return new VarArgNoOpHelper(Func, Msan, Visitor);
+
// VarArg handling is only implemented on AMD64. False positives are possible
// on other platforms.
Triple TargetTriple(Func.getParent()->getTargetTriple());
@@ -9574,8 +9756,30 @@ bool MemorySanitizer::sanitizeFunction(Function &F, TargetLibraryInfo &TLI) {
if (!CompileKernel && F.getName() == kMsanModuleCtorName)
return false;
- if (F.hasFnAttribute(Attribute::DisableSanitizerInstrumentation))
- return false;
+ // Functions marked disable_sanitizer_instrumentation are normally skipped
+ // entirely. They are still instrumented inside
+ // llvm.sanitizer.instrumentation.begin()/end() regions, so look for those
+ // first.
+ if (F.hasFnAttribute(Attribute::DisableSanitizerInstrumentation)) {
+ // Region instrumentation is a KernelMemorySanitizer feature: it refers to
+ // the context state returned by __msan_get_context_state(), which only
+ // exists in the kernel runtime.
+ if (!CompileKernel)
+ return false;
+ bool HasKmsanRegionIntrinsics = false;
+ for (Instruction &I : instructions(F)) {
+ if (auto *II = dyn_cast<IntrinsicInst>(&I)) {
+ if (II->getIntrinsicID() ==
+ Intrinsic::sanitizer_instrumentation_begin ||
+ II->getIntrinsicID() == Intrinsic::sanitizer_instrumentation_end) {
+ HasKmsanRegionIntrinsics = true;
+ break;
+ }
+ }
+ }
+ if (!HasKmsanRegionIntrinsics)
+ return false;
+ }
MemorySanitizerVisitor Visitor(F, *this, TLI);
diff --git a/llvm/test/Instrumentation/MemorySanitizer/kmsan-intrinsics.ll b/llvm/test/Instrumentation/MemorySanitizer/kmsan-intrinsics.ll
new file mode 100644
index 0000000000000..51837797a95c1
--- /dev/null
+++ b/llvm/test/Instrumentation/MemorySanitizer/kmsan-intrinsics.ll
@@ -0,0 +1,138 @@
+; RUN: opt < %s -msan-check-access-address=0 -msan-kernel=1 -S -passes=msan 2>&1 | FileCheck %s
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @instrumented_function() sanitize_memory {
+entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.sanitizer.instrumentation.end()
+ ret void
+}
+
+define void @no_sanitize_function() {
+entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.sanitizer.instrumentation.end()
+ ret void
+}
+
+define void @noinstr_function(ptr %p, i64 %val) disable_sanitizer_instrumentation {
+entry:
+ %x = load i64, ptr %p
+ %add = add i64 %x, %val
+ call void @llvm.sanitizer.instrumentation.begin()
+ %y = load i64, ptr %p
+ %add2 = add i64 %y, %add
+ store i64 %add2, ptr %p
+ %z = load i64, ptr %p
+ store i64 %z, ptr %p
+ call void @llvm.sanitizer.instrumentation.end()
+ %w = load i64, ptr %p
+ ret void
+}
+
+; CHECK-LABEL: @noinstr_function(
+; CHECK-NOT: __msan_get_context_state
+; CHECK: %x = load i64, ptr %p
+; CHECK: %add = add i64 %x, %val
+; CHECK: [[CTX1:%.*]] = call ptr @__msan_get_context_state()
+; CHECK: store ptr [[CTX1]], ptr %msan_context_state
+; CHECK: %y = load i64, ptr %p
+; CHECK: call { ptr, ptr } @__msan_metadata_ptr_for_load_8(ptr %p)
+; CHECK: %add2 = add i64 %y, %add
+; CHECK: call { ptr, ptr } @__msan_metadata_ptr_for_store_8(ptr %p)
+; CHECK: store i64 %add2, ptr %p
+; CHECK-NOT: call ptr @__msan_get_context_state()
+; CHECK: %z = load i64, ptr %p
+; CHECK: call { ptr, ptr } @__msan_metadata_ptr_for_load_8(ptr %p)
+; CHECK: call { ptr, ptr } @__msan_metadata_ptr_for_store_8(ptr %p)
+; CHECK: store i64 %z, ptr %p
+; CHECK-NOT: call ptr @__msan_get_context_state()
+; CHECK-NOT: call { ptr, ptr } @__msan_metadata_ptr_for_load_8
+; CHECK: %w = load i64, ptr %p
+; CHECK: ret void
+
+declare void @llvm.sanitizer.instrumentation.begin()
+declare void @llvm.sanitizer.instrumentation.end()
+
+define void @noinstr_with_branch(ptr %p, i1 %cond) disable_sanitizer_instrumentation {
+entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ br i1 %cond, label %if.then, label %if.else
+
+if.then:
+ %a = load i64, ptr %p
+ store i64 %a, ptr %p
+ br label %if.end
+
+if.else:
+ %b = load i64, ptr %p
+ store i64 %b, ptr %p
+ br label %if.end
+
+if.end:
+ call void @llvm.sanitizer.instrumentation.end()
+ ret void
+}
+
+; CHECK-LABEL: @noinstr_with_branch(
+; CHECK: [[CTX3:%.*]] = call ptr @__msan_get_context_state()
+; CHECK: store ptr [[CTX3]], ptr %msan_context_state
+; CHECK: if.then:
+; CHECK: call { ptr, ptr } @__msan_metadata_ptr_for_load_8(ptr %p)
+; CHECK: if.else:
+; CHECK: call { ptr, ptr } @__msan_metadata_ptr_for_load_8(ptr %p)
+
+define void @noinstr_with_loop(ptr %p, i64 %n) disable_sanitizer_instrumentation {
+entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ br label %loop
+
+loop:
+ %i = phi i64 [ 0, %entry ], [ %inc, %loop ]
+ %c = load i64, ptr %p
+ store i64 %c, ptr %p
+ %inc = add i64 %i, 1
+ %cmp = icmp ult i64 %inc, %n
+ br i1 %cmp, label %loop, label %exit
+
+exit:
+ call void @llvm.sanitizer.instrumentation.end()
+ ret void
+}
+
+; CHECK-LABEL: @noinstr_with_loop(
+; CHECK: [[CTX4:%.*]] = call ptr @__msan_get_context_state()
+; CHECK: store ptr [[CTX4]], ptr %msan_context_state
+; CHECK: loop:
+; CHECK: call { ptr, ptr } @__msan_metadata_ptr_for_load_8(ptr %p)
+; CHECK: call { ptr, ptr } @__msan_metadata_ptr_for_store_8(ptr %p)
+; CHECK: exit:
+
+declare i64 @dummy_call(i64)
+
+define i64 @noinstr_with_various_insts(ptr %p, i64 %val) disable_sanitizer_instrumentation {
+entry:
+ %a = alloca i64
+ %rmw1 = atomicrmw add ptr %p, i64 %val seq_cst
+ %call1 = call i64 @dummy_call(i64 %rmw1)
+ store i64 %call1, ptr %a
+
+ call void @llvm.sanitizer.instrumentation.begin()
+ %rmw2 = atomicrmw add ptr %p, i64 %val seq_cst
+ %call2 = call i64 @dummy_call(i64 %rmw2)
+ store i64 %call2, ptr %a
+ call void @llvm.sanitizer.instrumentation.end()
+
+ %rmw3 = atomicrmw add ptr %p, i64 %val seq_cst
+ %call3 = call i64 @dummy_call(i64 %rmw3)
+ store i64 %call3, ptr %a
+ ret i64 %call3
+}
+
+; CHECK-LABEL: @noinstr_with_various_insts(
+; CHECK-NOT: __msan_metadata
+; CHECK: call ptr @__msan_get_context_state()
+; CHECK: call { ptr, ptr } @__msan_metadata_ptr_for_store_8(ptr %p)
+; CHECK: ret i64 %call3
\ No newline at end of file
diff --git a/llvm/test/Instrumentation/MemorySanitizer/kmsan-region-dataflow.ll b/llvm/test/Instrumentation/MemorySanitizer/kmsan-region-dataflow.ll
new file mode 100644
index 0000000000000..aee6d9fdb20ea
--- /dev/null
+++ b/llvm/test/Instrumentation/MemorySanitizer/kmsan-region-dataflow.ll
@@ -0,0 +1,124 @@
+; RUN: opt < %s -msan-check-access-address=0 -msan-kernel=1 -S -passes=msan 2>&1 | FileCheck %s
+
+; Region membership is a "must" dataflow problem over the region nesting depth.
+; These tests cover the cases where an "any path" or flag-based analysis gets it
+; wrong.
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+declare void @llvm.sanitizer.instrumentation.begin()
+declare void @llvm.sanitizer.instrumentation.end()
+
+; A block reachable both from inside and from outside a region must not be
+; instrumented: on the path that never executed begin(), %msan_context_state
+; holds no context state, and the instrumentation would load a wild pointer out
+; of it and then store through it.
+
+define i32 @unbalanced_join(ptr %p, i1 %cond) disable_sanitizer_instrumentation {
+entry:
+ br i1 %cond, label %left, label %right
+
+left:
+ call void @llvm.sanitizer.instrumentation.begin()
+ br label %merge
+
+right:
+ br label %merge
+
+merge:
+ %v = load i32, ptr %p
+ ret i32 %v
+}
+
+; CHECK-LABEL: @unbalanced_join(
+; CHECK: merge:
+; CHECK-NOT: load ptr, ptr %msan_context_state
+; CHECK-NOT: __msan_metadata_ptr_for_load_4
+; CHECK: ret i32 %v
+
+; The same join is instrumented once every incoming path has entered a region.
+
+define i32 @balanced_join(ptr %p, i1 %cond) disable_sanitizer_instrumentation {
+entry:
+ br i1 %cond, label %left, label %right
+
+left:
+ call void @llvm.sanitizer.instrumentation.begin()
+ br label %merge
+
+right:
+ call void @llvm.sanitizer.instrumentation.begin()
+ br label %merge
+
+merge:
+ %v = load i32, ptr %p
+ call void @llvm.sanitizer.instrumentation.end()
+ ret i32 %v
+}
+
+; CHECK-LABEL: @balanced_join(
+; CHECK: merge:
+; CHECK: %v = load i32, ptr %p
+; CHECK: call { ptr, ptr } @__msan_metadata_ptr_for_load_4(ptr %p)
+
+; Regions nest: the inner end() must not terminate the outer region.
+
+define void @nested_regions(ptr %p) disable_sanitizer_instrumentation {
+entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.sanitizer.instrumentation.begin()
+ %a = load i32, ptr %p
+ call void @llvm.sanitizer.instrumentation.end()
+ %b = load i32, ptr %p
+ call void @llvm.sanitizer.instrumentation.end()
+ %c = load i32, ptr %p
+ ret void
+}
+
+; CHECK-LABEL: @nested_regions(
+; CHECK: %a = load i32, ptr %p
+; CHECK: call { ptr, ptr } @__msan_metadata_ptr_for_load_4(ptr %p)
+; CHECK: %b = load i32, ptr %p
+; CHECK: call { ptr, ptr } @__msan_metadata_ptr_for_load_4(ptr %p)
+; CHECK: %c = load i32, ptr %p
+; CHECK-NOT: __msan_metadata_ptr_for_load_4
+; CHECK: ret void
+
+; Nesting also has to survive a join: both predecessors of %merge are at depth
+; one, so %merge is still inside the outer region.
+
+define void @nested_regions_join(ptr %p, i1 %cond) disable_sanitizer_instrumentation {
+entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ br i1 %cond, label %inner, label %merge
+
+inner:
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.sanitizer.instrumentation.end()
+ br label %merge
+
+merge:
+ %v = load i32, ptr %p
+ call void @llvm.sanitizer.instrumentation.end()
+ ret void
+}
+
+; CHECK-LABEL: @nested_regions_join(
+; CHECK: merge:
+; CHECK: %v = load i32, ptr %p
+; CHECK: call { ptr, ptr } @__msan_metadata_ptr_for_load_4(ptr %p)
+
+; An end() without a matching begin() is malformed input. Saturate the depth at
+; zero rather than wrapping around and instrumenting the rest of the function.
+
+define void @unmatched_end(ptr %p) disable_sanitizer_instrumentation {
+entry:
+ call void @llvm.sanitizer.instrumentation.end()
+ %v = load i32, ptr %p
+ ret void
+}
+
+; CHECK-LABEL: @unmatched_end(
+; CHECK-NOT: __msan_metadata_ptr_for_load_4
+; CHECK: ret void
diff --git a/llvm/test/Instrumentation/MemorySanitizer/kmsan-region-mem-intrinsics.ll b/llvm/test/Instrumentation/MemorySanitizer/kmsan-region-mem-intrinsics.ll
new file mode 100644
index 0000000000000..0643ce3331c95
--- /dev/null
+++ b/llvm/test/Instrumentation/MemorySanitizer/kmsan-region-mem-intrinsics.ll
@@ -0,0 +1,41 @@
+; RUN: opt < %s -msan-check-access-address=0 -msan-kernel=1 -S -passes=msan 2>&1 | FileCheck %s
+
+; Memory intrinsics are rewritten into KMSAN runtime calls. Outside an
+; instrumentation region they have to be left alone: emitting the call there
+; instruments code the region markers explicitly excluded, and in noinstr kernel
+; code it makes objtool report
+; "call to __msan_memcpy() leaves .noinstr.text section".
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+declare void @llvm.sanitizer.instrumentation.begin()
+declare void @llvm.sanitizer.instrumentation.end()
+declare void @llvm.memcpy.p0.p0.i64(ptr, ptr, i64, i1)
+declare void @llvm.memmove.p0.p0.i64(ptr, ptr, i64, i1)
+declare void @llvm.memset.p0.i64(ptr, i8, i64, i1)
+
+define void @mem_intrinsics(ptr %dst, ptr %src) disable_sanitizer_instrumentation {
+entry:
+ call void @llvm.memcpy.p0.p0.i64(ptr %dst, ptr %src, i64 64, i1 false)
+ call void @llvm.memmove.p0.p0.i64(ptr %dst, ptr %src, i64 64, i1 false)
+ call void @llvm.memset.p0.i64(ptr %dst, i8 0, i64 64, i1 false)
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.memcpy.p0.p0.i64(ptr %dst, ptr %src, i64 64, i1 false)
+ call void @llvm.memmove.p0.p0.i64(ptr %dst, ptr %src, i64 64, i1 false)
+ call void @llvm.memset.p0.i64(ptr %dst, i8 0, i64 64, i1 false)
+ call void @llvm.sanitizer.instrumentation.end()
+ call void @llvm.memcpy.p0.p0.i64(ptr %dst, ptr %src, i64 64, i1 false)
+ ret void
+}
+
+; CHECK-LABEL: @mem_intrinsics(
+; CHECK: call void @llvm.memcpy.p0.p0.i64(ptr %dst, ptr %src, i64 64, i1 false)
+; CHECK: call void @llvm.memmove.p0.p0.i64(ptr %dst, ptr %src, i64 64, i1 false)
+; CHECK: call void @llvm.memset.p0.i64(ptr %dst, i8 0, i64 64, i1 false)
+; CHECK: call ptr @__msan_get_context_state()
+; CHECK: call ptr @__msan_memcpy(ptr %dst, ptr %src, i64 64)
+; CHECK: call ptr @__msan_memmove(ptr %dst, ptr %src, i64 64)
+; CHECK: call ptr @__msan_memset(ptr %dst, i32 0, i64 64)
+; CHECK: call void @llvm.memcpy.p0.p0.i64(ptr %dst, ptr %src, i64 64, i1 false)
+; CHECK: ret void
diff --git a/llvm/test/Instrumentation/MemorySanitizer/kmsan-region-userspace.ll b/llvm/test/Instrumentation/MemorySanitizer/kmsan-region-userspace.ll
new file mode 100644
index 0000000000000..8c1d79d8243bc
--- /dev/null
+++ b/llvm/test/Instrumentation/MemorySanitizer/kmsan-region-userspace.ll
@@ -0,0 +1,26 @@
+; RUN: opt < %s -S -passes=msan 2>&1 | FileCheck %s
+
+; Region instrumentation is a KernelMemorySanitizer feature: it refers to the
+; context state returned by __msan_get_context_state(), which only exists in the
+; kernel runtime. Under userspace MemorySanitizer the markers must simply be
+; ignored - this used to crash the compiler on a null MsanGetContextStateFn.
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+declare void @llvm.sanitizer.instrumentation.begin()
+declare void @llvm.sanitizer.instrumentation.end()
+
+define i32 @region_in_userspace_msan(ptr %p) disable_sanitizer_instrumentation {
+entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ %v = load i32, ptr %p
+ call void @llvm.sanitizer.instrumentation.end()
+ ret i32 %v
+}
+
+; CHECK-LABEL: @region_in_userspace_msan(
+; CHECK-NOT: __msan_get_context_state
+; CHECK-NOT: msan_context_state
+; CHECK: %v = load i32, ptr %p
+; CHECK: ret i32 %v
diff --git a/llvm/test/Instrumentation/MemorySanitizer/kmsan-region-varargs.ll b/llvm/test/Instrumentation/MemorySanitizer/kmsan-region-varargs.ll
new file mode 100644
index 0000000000000..fa08752df90ad
--- /dev/null
+++ b/llvm/test/Instrumentation/MemorySanitizer/kmsan-region-varargs.ll
@@ -0,0 +1,36 @@
+; RUN: opt < %s -msan-check-access-address=0 -msan-kernel=1 -S -passes=msan 2>&1 | FileCheck %s
+
+; VarArgHelpers emit their setup into the function prologue. In a
+; region-instrumented function the prologue is outside every region, so
+; %msan_context_state has not been stored to yet: the emitted code used to load
+; that slot and then read and write through the resulting pointer. Varargs must
+; be left alone in region-instrumented functions.
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+declare void @llvm.sanitizer.instrumentation.begin()
+declare void @llvm.sanitizer.instrumentation.end()
+declare void @llvm.va_start.p0(ptr)
+declare void @llvm.va_end.p0(ptr)
+
+define i32 @va_noinstr(i32 %n, ...) disable_sanitizer_instrumentation {
+entry:
+ %ap = alloca [1 x { i32, i32, ptr, ptr }]
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.va_start.p0(ptr %ap)
+ %v = va_arg ptr %ap, i32
+ call void @llvm.va_end.p0(ptr %ap)
+ call void @llvm.sanitizer.instrumentation.end()
+ ret i32 %v
+}
+
+; The context state must not be read before the first begin() stores it. The
+; first thing touching the slot is the store from __msan_get_context_state().
+; CHECK-LABEL: @va_noinstr(
+; CHECK: %msan_context_state = alloca ptr
+; CHECK-NOT: load ptr, ptr %msan_context_state
+; CHECK-NOT: va_arg_overflow_size
+; CHECK-NOT: va_arg_shadow
+; CHECK: call ptr @__msan_get_context_state()
+; CHECK: store ptr {{.*}}, ptr %msan_context_state
>From 376b51b15bb1a1d4aad2cd9746211c110b659660 Mon Sep 17 00:00:00 2001
From: Alexander Potapenko <glider at google.com>
Date: Mon, 5 Oct 2026 14:04:47 +0000
Subject: [PATCH 3/5] [ASan] Support sanitizer instrumentation regions
Implement `llvm.sanitizer.instrumentation.begin`/`end` regions in
AddressSanitizer, in both user and kernel mode. Functions that lack
`sanitize_address` or have `disable_sanitizer_instrumentation` are
normally not instrumented at all; if they contain regions, memory
accesses and memory intrinsics inside the regions are now checked.
The Linux kernel uses this for `noinstr` functions, which may call
instrumented code between `instrumentation_begin()` and
`instrumentation_end()`, but whose own memory accesses in those
regions were never checked by KASAN.
Only memory accesses are instrumented in such functions. Their stack
frame is not laid out by ASan, so stack poisoning, use-after-return and
the `__asan_handle_no_return()` calls are skipped, and accesses to the
function's own allocas, which have no redzones, are not checked. A call
outside a region still resets the per-block deduplication of checks.
Clang now lowers the region builtins to the intrinsics under
`-fsanitize=address` and `-fsanitize=kernel-address`.
Tests: `llvm/test/Instrumentation/AddressSanitizer/sanitizer-instrumentation-regions.ll`,
`clang/test/CodeGen/sanitizer-instrumentation-builtins.c`,
`clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c`.
---
clang/docs/AddressSanitizer.md | 21 ++
clang/lib/CodeGen/CGBuiltin.cpp | 4 +-
.../sanitizer-instrumentation-builtins.c | 12 +-
...nitizer-instrumentation-regions-pipeline.c | 4 +
llvm/docs/LangRef.md | 2 +
.../Instrumentation/AddressSanitizer.cpp | 49 ++++-
.../sanitizer-instrumentation-regions.ll | 183 ++++++++++++++++++
7 files changed, 262 insertions(+), 13 deletions(-)
create mode 100644 llvm/test/Instrumentation/AddressSanitizer/sanitizer-instrumentation-regions.ll
diff --git a/clang/docs/AddressSanitizer.md b/clang/docs/AddressSanitizer.md
index 160c63dce2d97..1f502584b1de9 100644
--- a/clang/docs/AddressSanitizer.md
+++ b/clang/docs/AddressSanitizer.md
@@ -243,6 +243,27 @@ AddressSanitizer also supports
works similarly to `__attribute__((no_sanitize("address")))`, but it also
prevents instrumentation performed by other sanitizers.
+### Re-enabling Instrumentation in Regions
+
+Inside a function that AddressSanitizer does not instrument, a region
+delimited by `__builtin_sanitizer_instrumentation_begin()` and
+`__builtin_sanitizer_instrumentation_end()` is instrumented again: memory
+accesses and memory intrinsics in it are checked. The function's stack frame
+is still not instrumented, and accesses to its own local variables are not
+checked. Regions nest, and code reachable without passing through a
+`__builtin_sanitizer_instrumentation_begin()` is not considered to be in a
+region.
+
+```c
+__attribute__((no_sanitize("address")))
+void mostly_uninstrumented(int *p) {
+ ... // not checked
+ __builtin_sanitizer_instrumentation_begin();
+ *p = 0; // checked
+ __builtin_sanitizer_instrumentation_end();
+}
+```
+
### Interaction of Inlining with Disabling Sanitizer Instrumentation
- A `no_sanitize` function will not be inlined heuristically by the compiler into a sanitized function.
diff --git a/clang/lib/CodeGen/CGBuiltin.cpp b/clang/lib/CodeGen/CGBuiltin.cpp
index aa8fbca7caecc..5509fd384eaa6 100644
--- a/clang/lib/CodeGen/CGBuiltin.cpp
+++ b/clang/lib/CodeGen/CGBuiltin.cpp
@@ -3913,7 +3913,9 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
// Only emit the region markers for sanitizers that honour them. Elsewhere
// they would just be optimization barriers that no pass ever removes. This
// lets code guard its uses on __has_builtin alone.
- SanitizerMask Consumers = SanitizerKind::KernelMemory;
+ SanitizerMask Consumers = SanitizerKind::KernelMemory |
+ SanitizerKind::Address |
+ SanitizerKind::KernelAddress;
if (getLangOpts().Sanitize.hasOneOf(Consumers))
Builder.CreateCall(CGM.getIntrinsic(
BuiltinIDIfNoAsmLabel ==
diff --git a/clang/test/CodeGen/sanitizer-instrumentation-builtins.c b/clang/test/CodeGen/sanitizer-instrumentation-builtins.c
index 65f36048637cf..fdc1e3187bb06 100644
--- a/clang/test/CodeGen/sanitizer-instrumentation-builtins.c
+++ b/clang/test/CodeGen/sanitizer-instrumentation-builtins.c
@@ -1,13 +1,15 @@
// Check that the sanitizer region markers lower to their intrinsics under
-// -fsanitize=kernel-memory, and to nothing without a sanitizer that consumes
-// them.
+// -fsanitize=kernel-memory, kernel-address and address, and to nothing without
+// a sanitizer that consumes them.
// RUN: %clang_cc1 -triple x86_64-linux-gnu -fsanitize=kernel-memory \
// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s
// RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -disable-llvm-passes \
// RUN: -o - %s | FileCheck %s --check-prefix=NONE
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -fsanitize=kernel-address \
+// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s --check-prefix=ASAN
// RUN: %clang_cc1 -triple x86_64-linux-gnu -fsanitize=address \
-// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s --check-prefix=NONE
+// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s --check-prefix=ASAN
// RUN: %clang_cc1 -triple x86_64-linux-gnu -fsanitize=thread \
// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s --check-prefix=NONE
@@ -37,6 +39,10 @@ void noinstr_function(int x) {
// CHECK: declare void @llvm.sanitizer.instrumentation.begin() #[[ATTR:[0-9]+]]
// CHECK: attributes #[[ATTR]] = { nocallback nofree nomerge nosync nounwind willreturn }
+// ASAN-LABEL: define {{.*}}void @noinstr_function(
+// ASAN: call void @llvm.sanitizer.instrumentation.begin()
+// ASAN: call void @llvm.sanitizer.instrumentation.end()
+
// NONE-LABEL: define {{.*}}void @noinstr_function(
// NONE-NOT: llvm.sanitizer.instrumentation
// NONE: }
diff --git a/clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c b/clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c
index 8ec56d40cb580..1a9229790441d 100644
--- a/clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c
+++ b/clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c
@@ -7,6 +7,9 @@
// RUN: %clang_cc1 -triple x86_64-linux-gnu -O2 -debug-info-kind=line-tables-only \
// RUN: -fsanitize=kernel-memory -emit-llvm -o - %s \
// RUN: | FileCheck %s --check-prefixes=CHECK,KMSAN
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -O2 -debug-info-kind=line-tables-only \
+// RUN: -fsanitize=kernel-address -emit-llvm -o - %s \
+// RUN: | FileCheck %s --check-prefixes=CHECK,KASAN
static inline __attribute__((always_inline)) void instrumentation_begin(void) {
__builtin_sanitizer_instrumentation_begin();
@@ -33,6 +36,7 @@ void noinstr_function(int *p, int x) {
// CHECK: call void @llvm.sanitizer.instrumentation.begin()
// KMSAN: call ptr @__msan_get_context_state()
// KMSAN: call { ptr, ptr } @__msan_metadata_ptr_for_load_4(
+// KASAN: call void @__asan_report_load4(
// CHECK: call void @instrumented_callee(
// CHECK: call void @llvm.sanitizer.instrumentation.end()
// CHECK-NOT: call {{.*}}@__{{msan|asan|tsan|sanitizer_cov}}
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index 0b08303ad3052..9abdfb3679bea 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -30512,3 +30512,5 @@ Regions nest: `llvm.sanitizer.instrumentation.begin` increments a region nesting
An instruction is considered to be inside a region only if *every* path from the function entry to it is inside a region, i.e. region membership is an intersection over predecessors. This makes an in-region instruction always dominated by a `llvm.sanitizer.instrumentation.begin`, so any per-function sanitizer state set up there is available. A region that is entered on some paths but not others is therefore instrumented only where it is unambiguous.
MemorySanitizer supports regions in kernel mode (KMSAN), in functions marked with `disable_sanitizer_instrumentation`. The function-level ABI prologue is skipped, the MSan context state is fetched at `llvm.sanitizer.instrumentation.begin`, and shadow propagation and parameter shadow setup are only performed inside regions. Memory written outside a region keeps whatever shadow it had, and values computed outside a region are treated as fully initialized when they are used inside one.
+
+AddressSanitizer supports regions in functions that lack the `sanitize_address` attribute or are marked with `disable_sanitizer_instrumentation`. Inside regions, memory accesses and memory intrinsics are instrumented. The stack frame of such a function is not instrumented, accesses to its own `alloca`s are not checked, and no `__asan_handle_no_return` calls are inserted.
diff --git a/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp b/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp
index d358318195265..976e0789d891a 100644
--- a/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp
+++ b/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp
@@ -3108,13 +3108,23 @@ bool AddressSanitizer::instrumentFunction(Function &F,
if (maybeInsertAsanInitAtFunctionEntry(F))
FunctionModified = true;
- // Leave if the function doesn't need instrumentation.
- if (!F.hasFnAttribute(Attribute::SanitizeAddress)) return FunctionModified;
-
- if (F.hasFnAttribute(Attribute::DisableSanitizerInstrumentation))
- return FunctionModified;
-
- LLVM_DEBUG(dbgs() << "ASAN instrumenting:\n" << F << "\n");
+ // Leave if the function doesn't need instrumentation. Functions that are
+ // not instrumented as a whole are still instrumented inside
+ // llvm.sanitizer.instrumentation.begin()/end() regions. Only memory
+ // accesses are instrumented there: the stack frame of such a function is
+ // not laid out by ASan, so stack poisoning, use-after-return and the
+ // handling of noreturn calls (which unpoisons the stack) are left out.
+ bool RegionOnly = false;
+ if (!F.hasFnAttribute(Attribute::SanitizeAddress) ||
+ F.hasFnAttribute(Attribute::DisableSanitizerInstrumentation)) {
+ if (!hasSanitizerInstrumentationRegions(F))
+ return FunctionModified;
+ RegionOnly = true;
+ }
+
+ LLVM_DEBUG(dbgs() << "ASAN instrumenting" << (RegionOnly ? " regions" : "")
+ << ":\n"
+ << F << "\n");
initializeCallbacks(TLI);
@@ -3131,6 +3141,10 @@ bool AddressSanitizer::instrumentFunction(Function &F,
if (TargetTriple.isOSWindows())
markCatchParametersAsUninteresting(F);
+ SmallPtrSet<const Instruction *, 16> InRegion;
+ if (RegionOnly)
+ computeSanitizerInstrumentationRegions(F, InRegion);
+
// We want to instrument every address only once per basic block (unless there
// are calls between uses).
SmallPtrSet<Value *, 16> TempsToInstrument;
@@ -3150,11 +3164,23 @@ bool AddressSanitizer::instrumentFunction(Function &F,
// Skip instructions inserted by another instrumentation.
if (Inst.hasMetadata(LLVMContext::MD_nosanitize))
continue;
+ if (RegionOnly && !InRegion.contains(&Inst)) {
+ // A call between two regions can free memory like any other call.
+ if (isa<CallBase>(Inst))
+ TempsToInstrument.clear();
+ continue;
+ }
SmallVector<InterestingMemoryOperand, 1> InterestingOperands;
getInterestingMemoryOperands(&Inst, InterestingOperands, TTI);
if (!InterestingOperands.empty()) {
for (auto &Operand : InterestingOperands) {
+ // The locals of a function that is only instrumented in regions
+ // have no redzones, so checking them can only trip over stale
+ // stack poisoning.
+ if (RegionOnly &&
+ isa<AllocaInst>(getUnderlyingObject(Operand.getPtr())))
+ continue;
if (ClOpt && ClOptSameTemp) {
Value *Ptr = Operand.getPtr();
// If we have a mask, skip instrumentation if we've already
@@ -3214,8 +3240,13 @@ bool AddressSanitizer::instrumentFunction(Function &F,
FunctionModified = true;
}
- FunctionStackPoisoner FSP(F, *this, RTCI);
- bool ChangedStack = FSP.runOnFunction();
+ bool ChangedStack = false;
+ if (!RegionOnly) {
+ FunctionStackPoisoner FSP(F, *this, RTCI);
+ ChangedStack = FSP.runOnFunction();
+ } else {
+ NoReturnCalls.clear();
+ }
// We must unpoison the stack before NoReturn calls (throw, _exit, etc).
// See e.g. https://github.com/google/sanitizers/issues/37
diff --git a/llvm/test/Instrumentation/AddressSanitizer/sanitizer-instrumentation-regions.ll b/llvm/test/Instrumentation/AddressSanitizer/sanitizer-instrumentation-regions.ll
new file mode 100644
index 0000000000000..d82e185e962c6
--- /dev/null
+++ b/llvm/test/Instrumentation/AddressSanitizer/sanitizer-instrumentation-regions.ll
@@ -0,0 +1,183 @@
+; Test that functions not instrumented by AddressSanitizer are still
+; instrumented inside llvm.sanitizer.instrumentation.begin()/end() regions, and
+; only there.
+;
+; RUN: opt < %s -passes=asan -asan-kernel=1 \
+; RUN: -asan-instrumentation-with-call-threshold=0 -S | FileCheck %s
+; RUN: opt < %s -passes=asan -asan-instrumentation-with-call-threshold=0 -S \
+; RUN: | FileCheck %s
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+declare void @llvm.sanitizer.instrumentation.begin()
+declare void @llvm.sanitizer.instrumentation.end()
+declare void @llvm.memcpy.p0.p0.i64(ptr, ptr, i64, i1)
+declare void @callee()
+
+; Only the access between begin() and end() is checked.
+define i32 @region(ptr %a, ptr %b, ptr %c) {
+entry:
+ %x = load i32, ptr %a
+ call void @llvm.sanitizer.instrumentation.begin()
+ %y = load i32, ptr %b
+ call void @llvm.sanitizer.instrumentation.end()
+ store i32 %y, ptr %c
+ %s = add i32 %x, %y
+ ret i32 %s
+}
+
+; CHECK-LABEL: define i32 @region(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: %x = load i32, ptr %a
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.begin()
+; CHECK: call void @__asan_load4{{(_noabort)?}}(i64 %{{.*}})
+; CHECK-NEXT: %y = load i32, ptr %b
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.end()
+; CHECK-NEXT: store i32 %y, ptr %c
+; CHECK-NOT: __asan_
+; CHECK: ret i32
+
+; The same for functions with disable_sanitizer_instrumentation.
+define i32 @region_disable(ptr %a, ptr %b) disable_sanitizer_instrumentation {
+entry:
+ %x = load i32, ptr %a
+ call void @llvm.sanitizer.instrumentation.begin()
+ store i32 %x, ptr %b
+ call void @llvm.sanitizer.instrumentation.end()
+ ret i32 %x
+}
+
+; CHECK-LABEL: define i32 @region_disable(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: %x = load i32, ptr %a
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.begin()
+; CHECK: call void @__asan_store4{{(_noabort)?}}(i64 %{{.*}})
+; CHECK-NEXT: store i32 %x, ptr %b
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.end()
+; CHECK-NEXT: ret i32 %x
+
+; A block reachable without passing begin() is not in a region.
+define i32 @diamond(i1 %cond, ptr %a) {
+entry:
+ br i1 %cond, label %then, label %join
+then:
+ call void @llvm.sanitizer.instrumentation.begin()
+ br label %join
+join:
+ %x = load i32, ptr %a
+ ret i32 %x
+}
+
+; CHECK-LABEL: define i32 @diamond(
+; CHECK-NOT: __asan_
+; CHECK: ret i32
+
+; The inner end() of nested regions does not end the outer region.
+define void @nested(ptr %a) {
+entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.sanitizer.instrumentation.end()
+ store i32 0, ptr %a
+ call void @llvm.sanitizer.instrumentation.end()
+ store i32 1, ptr %a
+ ret void
+}
+
+; CHECK-LABEL: define void @nested(
+; CHECK: call void @llvm.sanitizer.instrumentation.end()
+; CHECK: call void @__asan_store4{{(_noabort)?}}(i64 %{{.*}})
+; CHECK-NEXT: store i32 0, ptr %a
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.end()
+; CHECK-NEXT: store i32 1, ptr %a
+; CHECK-NEXT: ret void
+
+; The function's own locals have no redzones and are not checked; neither is
+; its stack frame laid out by ASan.
+define i32 @locals(ptr %p) {
+entry:
+ %local = alloca i32
+ call void @llvm.sanitizer.instrumentation.begin()
+ store i32 1, ptr %local
+ %v = load i32, ptr %p
+ call void @llvm.sanitizer.instrumentation.end()
+ ret i32 %v
+}
+
+; CHECK-LABEL: define i32 @locals(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: %local = alloca i32
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.begin()
+; CHECK-NEXT: store i32 1, ptr %local
+; CHECK: call void @__asan_load4{{(_noabort)?}}(i64 %{{.*}})
+; CHECK-NEXT: %v = load i32, ptr %p
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.end()
+; CHECK-NEXT: ret i32 %v
+
+; A call between two regions may free memory, so the second access to the
+; same address is checked again.
+define void @call_between_regions(ptr %p) {
+entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ store i32 0, ptr %p
+ call void @llvm.sanitizer.instrumentation.end()
+ call void @callee()
+ call void @llvm.sanitizer.instrumentation.begin()
+ store i32 1, ptr %p
+ call void @llvm.sanitizer.instrumentation.end()
+ ret void
+}
+
+; CHECK-LABEL: define void @call_between_regions(
+; CHECK: call void @__asan_store4{{(_noabort)?}}(i64 %{{.*}})
+; CHECK-NEXT: store i32 0, ptr %p
+; CHECK: call void @callee()
+; CHECK: call void @__asan_store4{{(_noabort)?}}(i64 %{{.*}})
+; CHECK-NEXT: store i32 1, ptr %p
+
+; Memory intrinsics are only replaced inside regions.
+define void @memcpy(ptr %d, ptr %s) {
+entry:
+ call void @llvm.memcpy.p0.p0.i64(ptr %d, ptr %s, i64 16, i1 false)
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.memcpy.p0.p0.i64(ptr %d, ptr %s, i64 16, i1 false)
+ call void @llvm.sanitizer.instrumentation.end()
+ ret void
+}
+
+; CHECK-LABEL: define void @memcpy(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr %d, ptr %s, i64 16, i1 false)
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.begin()
+; CHECK-NEXT: call ptr @{{(__asan_)?}}memcpy(ptr %d, ptr %s, i64 16)
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.end()
+
+; Without regions, functions lacking sanitize_address are left alone.
+define i32 @no_region(ptr %a) {
+entry:
+ %x = load i32, ptr %a
+ ret i32 %x
+}
+
+; CHECK-LABEL: define i32 @no_region(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: %x = load i32, ptr %a
+; CHECK-NEXT: ret i32 %x
+
+; Fully instrumented functions are unaffected by regions.
+define i32 @sanitized(ptr %a, ptr %b) sanitize_address {
+entry:
+ %x = load i32, ptr %a
+ call void @llvm.sanitizer.instrumentation.begin()
+ %y = load i32, ptr %b
+ call void @llvm.sanitizer.instrumentation.end()
+ %s = add i32 %x, %y
+ ret i32 %s
+}
+
+; CHECK-LABEL: define i32 @sanitized(
+; CHECK: call void @__asan_load4{{(_noabort)?}}(i64 %{{.*}})
+; CHECK-NEXT: %x = load i32, ptr %a
+; CHECK: call void @__asan_load4{{(_noabort)?}}(i64 %{{.*}})
+; CHECK-NEXT: %y = load i32, ptr %b
>From 51135a189d966ab18bc4c6da4ab8725153e8f1cb Mon Sep 17 00:00:00 2001
From: Alexander Potapenko <glider at google.com>
Date: Tue, 6 Oct 2026 10:23:12 +0000
Subject: [PATCH 4/5] [SanitizerCoverage] Support sanitizer instrumentation
regions
Implement `llvm.sanitizer.instrumentation.begin`/`end` regions in
SanitizerCoverage. Functions with `nosanitize_coverage` or
`disable_sanitizer_instrumentation` are normally not instrumented at
all; if they contain regions, the code inside the regions is now
instrumented.
The Linux kernel uses this for `noinstr` functions, which may call
instrumented code between `instrumentation_begin()` and
`instrumentation_end()`. Today KCOV does not see that code, so the
fuzzer gets no coverage or comparison feedback for it, e.g. for the
syscall and exception entry paths.
Each block that enters a region gets a coverage point right after the
point where it does, and comparisons, switches, divisions, GEPs, loads,
stores and indirect calls inside regions are traced as requested. A
block whose coverage point is not at its start, or that is the entry
block, is split there, so that the callback runs inside the region and
the PC table can refer to the block by address. Pruning is disabled in
such functions, as it assumes that all blocks are instrumented, and
function-level instrumentation (entry/exit callbacks, stack depth
tracing, control flow tables) is not emitted, as it would run outside
the regions.
Clang now lowers the region builtins to the intrinsics whenever
`-fsanitize-coverage` instrumentation is enabled.
Tests: `llvm/test/Instrumentation/SanitizerCoverage/sanitizer-instrumentation-regions.ll`,
`clang/test/CodeGen/sanitizer-instrumentation-builtins.c`,
`clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c`.
---
clang/docs/SanitizerCoverage.md | 25 +++
clang/lib/CodeGen/CGBuiltin.cpp | 3 +-
.../sanitizer-instrumentation-builtins.c | 17 +-
...nitizer-instrumentation-regions-pipeline.c | 11 +
llvm/docs/LangRef.md | 2 +
.../Instrumentation/SanitizerCoverage.cpp | 66 +++++-
.../sanitizer-instrumentation-regions.ll | 205 ++++++++++++++++++
7 files changed, 314 insertions(+), 15 deletions(-)
create mode 100644 llvm/test/Instrumentation/SanitizerCoverage/sanitizer-instrumentation-regions.ll
diff --git a/clang/docs/SanitizerCoverage.md b/clang/docs/SanitizerCoverage.md
index f859838616b49..0e5015466a1a0 100644
--- a/clang/docs/SanitizerCoverage.md
+++ b/clang/docs/SanitizerCoverage.md
@@ -449,6 +449,31 @@ function attribute `__attribute__((no_sanitize("coverage")))`. Because this
attribute may not be supported by other compilers, it is recommended to use it
together with `__has_feature(coverage_sanitizer)`.
+### Re-enabling instrumentation in regions
+
+Inside a function that has `__attribute__((no_sanitize("coverage")))` or
+`__attribute__((disable_sanitizer_instrumentation))`, a region delimited by
+`__builtin_sanitizer_instrumentation_begin()` and
+`__builtin_sanitizer_instrumentation_end()` is instrumented again: every basic
+block that enters a region gets a coverage point right after it does, and
+comparisons, divisions, GEPs, loads, stores and indirect calls in regions are
+traced as requested. Coverage points are not pruned in such functions, and the
+function entry and exit callbacks, stack depth tracing and control flow tables
+are not emitted. Regions nest, and code reachable without passing through a
+`__builtin_sanitizer_instrumentation_begin()` is not considered to be in a
+region.
+
+```c
+__attribute__((no_sanitize("coverage")))
+void mostly_uninstrumented(int x) {
+ ... // not covered
+ __builtin_sanitizer_instrumentation_begin();
+ if (x) // covered, and traced with -fsanitize-coverage=trace-cmp
+ foo();
+ __builtin_sanitizer_instrumentation_end();
+}
+```
+
## Disabling instrumentation without source modification
It is sometimes useful to tell SanitizerCoverage to instrument only a subset of the
diff --git a/clang/lib/CodeGen/CGBuiltin.cpp b/clang/lib/CodeGen/CGBuiltin.cpp
index 5509fd384eaa6..fafe7ddba5454 100644
--- a/clang/lib/CodeGen/CGBuiltin.cpp
+++ b/clang/lib/CodeGen/CGBuiltin.cpp
@@ -3916,7 +3916,8 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
SanitizerMask Consumers = SanitizerKind::KernelMemory |
SanitizerKind::Address |
SanitizerKind::KernelAddress;
- if (getLangOpts().Sanitize.hasOneOf(Consumers))
+ if (getLangOpts().Sanitize.hasOneOf(Consumers) ||
+ CGM.getCodeGenOpts().hasSanitizeCoverage())
Builder.CreateCall(CGM.getIntrinsic(
BuiltinIDIfNoAsmLabel ==
Builtin::BI__builtin_sanitizer_instrumentation_begin
diff --git a/clang/test/CodeGen/sanitizer-instrumentation-builtins.c b/clang/test/CodeGen/sanitizer-instrumentation-builtins.c
index fdc1e3187bb06..36123ff4a4453 100644
--- a/clang/test/CodeGen/sanitizer-instrumentation-builtins.c
+++ b/clang/test/CodeGen/sanitizer-instrumentation-builtins.c
@@ -1,15 +1,18 @@
// Check that the sanitizer region markers lower to their intrinsics under
-// -fsanitize=kernel-memory, kernel-address and address, and to nothing without
-// a sanitizer that consumes them.
+// -fsanitize=kernel-memory, kernel-address and address, and with
+// -fsanitize-coverage, and to nothing without a sanitizer that consumes them.
// RUN: %clang_cc1 -triple x86_64-linux-gnu -fsanitize=kernel-memory \
// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s
// RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -disable-llvm-passes \
// RUN: -o - %s | FileCheck %s --check-prefix=NONE
// RUN: %clang_cc1 -triple x86_64-linux-gnu -fsanitize=kernel-address \
-// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s --check-prefix=ASAN
+// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s --check-prefix=MARKERS
// RUN: %clang_cc1 -triple x86_64-linux-gnu -fsanitize=address \
-// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s --check-prefix=ASAN
+// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s --check-prefix=MARKERS
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -fsanitize-coverage-type=3 \
+// RUN: -fsanitize-coverage-trace-pc -emit-llvm -disable-llvm-passes -o - %s \
+// RUN: | FileCheck %s --check-prefix=MARKERS
// RUN: %clang_cc1 -triple x86_64-linux-gnu -fsanitize=thread \
// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s --check-prefix=NONE
@@ -39,9 +42,9 @@ void noinstr_function(int x) {
// CHECK: declare void @llvm.sanitizer.instrumentation.begin() #[[ATTR:[0-9]+]]
// CHECK: attributes #[[ATTR]] = { nocallback nofree nomerge nosync nounwind willreturn }
-// ASAN-LABEL: define {{.*}}void @noinstr_function(
-// ASAN: call void @llvm.sanitizer.instrumentation.begin()
-// ASAN: call void @llvm.sanitizer.instrumentation.end()
+// MARKERS-LABEL: define {{.*}}void @noinstr_function(
+// MARKERS: call void @llvm.sanitizer.instrumentation.begin()
+// MARKERS: call void @llvm.sanitizer.instrumentation.end()
// NONE-LABEL: define {{.*}}void @noinstr_function(
// NONE-NOT: llvm.sanitizer.instrumentation
diff --git a/clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c b/clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c
index 1a9229790441d..d16868f2c8e16 100644
--- a/clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c
+++ b/clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c
@@ -10,6 +10,15 @@
// RUN: %clang_cc1 -triple x86_64-linux-gnu -O2 -debug-info-kind=line-tables-only \
// RUN: -fsanitize=kernel-address -emit-llvm -o - %s \
// RUN: | FileCheck %s --check-prefixes=CHECK,KASAN
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -O2 -debug-info-kind=line-tables-only \
+// RUN: -fsanitize-coverage-type=3 -fsanitize-coverage-trace-pc \
+// RUN: -fsanitize-coverage-trace-cmp -emit-llvm -o - %s \
+// RUN: | FileCheck %s --check-prefixes=CHECK,KCOV
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -O2 -debug-info-kind=line-tables-only \
+// RUN: -fsanitize=kernel-memory -fsanitize-coverage-type=3 \
+// RUN: -fsanitize-coverage-trace-pc -fsanitize-coverage-trace-cmp \
+// RUN: -emit-llvm -o - %s \
+// RUN: | FileCheck %s --check-prefixes=CHECK,KMSAN,KCOV
static inline __attribute__((always_inline)) void instrumentation_begin(void) {
__builtin_sanitizer_instrumentation_begin();
@@ -35,6 +44,8 @@ void noinstr_function(int *p, int x) {
// CHECK-NOT: call {{.*}}@__{{msan|asan|tsan|sanitizer_cov}}
// CHECK: call void @llvm.sanitizer.instrumentation.begin()
// KMSAN: call ptr @__msan_get_context_state()
+// KCOV: call void @__sanitizer_cov_trace_pc()
+// KCOV: call void @__sanitizer_cov_trace_const_cmp4(i32 7,
// KMSAN: call { ptr, ptr } @__msan_metadata_ptr_for_load_4(
// KASAN: call void @__asan_report_load4(
// CHECK: call void @instrumented_callee(
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index 9abdfb3679bea..a8629a1360006 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -30514,3 +30514,5 @@ An instruction is considered to be inside a region only if *every* path from the
MemorySanitizer supports regions in kernel mode (KMSAN), in functions marked with `disable_sanitizer_instrumentation`. The function-level ABI prologue is skipped, the MSan context state is fetched at `llvm.sanitizer.instrumentation.begin`, and shadow propagation and parameter shadow setup are only performed inside regions. Memory written outside a region keeps whatever shadow it had, and values computed outside a region are treated as fully initialized when they are used inside one.
AddressSanitizer supports regions in functions that lack the `sanitize_address` attribute or are marked with `disable_sanitizer_instrumentation`. Inside regions, memory accesses and memory intrinsics are instrumented. The stack frame of such a function is not instrumented, accesses to its own `alloca`s are not checked, and no `__asan_handle_no_return` calls are inserted.
+
+SanitizerCoverage supports regions in functions marked with `nosanitize_coverage` or `disable_sanitizer_instrumentation`. Each basic block that enters a region gets a coverage point right after the point where it does (splitting the block if necessary), without pruning, and the requested tracing of comparisons, divisions, GEPs, loads, stores and indirect calls is performed inside regions. Function entry/exit callbacks, stack depth tracing and control flow tables are not emitted for such functions.
diff --git a/llvm/lib/Transforms/Instrumentation/SanitizerCoverage.cpp b/llvm/lib/Transforms/Instrumentation/SanitizerCoverage.cpp
index df9675a02824e..b7912b49f4292 100644
--- a/llvm/lib/Transforms/Instrumentation/SanitizerCoverage.cpp
+++ b/llvm/lib/Transforms/Instrumentation/SanitizerCoverage.cpp
@@ -36,6 +36,7 @@
#include "llvm/TargetParser/Triple.h"
#include "llvm/Transforms/Utils/BasicBlockUtils.h"
#include "llvm/Transforms/Utils/EscapeEnumerator.h"
+#include "llvm/Transforms/Utils/Instrumentation.h"
#include "llvm/Transforms/Utils/ModuleUtils.h"
using namespace llvm;
@@ -635,6 +636,26 @@ static bool shouldInstrumentBlock(const Function &F, const BasicBlock *BB,
!(isFullPostDominator(BB, PDT) && !BB->getSinglePredecessor());
}
+// Returns the first instruction of \p BB inside a sanitizer instrumentation
+// region before which coverage can be inserted, or nullptr if there is none.
+// The region's begin() marker is skipped: it is the first instruction of the
+// region, but code inserted before it would run outside the region.
+static Instruction *
+getRegionCoveragePoint(BasicBlock &BB,
+ const SmallPtrSetImpl<const Instruction *> &InRegion) {
+ if (isa<UnreachableInst>(BB.getFirstNonPHIOrDbgOrLifetime()))
+ return nullptr;
+ for (Instruction &I : make_range(BB.getFirstInsertionPt(), BB.end())) {
+ if (!InRegion.contains(&I))
+ continue;
+ if (auto *II = dyn_cast<IntrinsicInst>(&I))
+ if (II->getIntrinsicID() == Intrinsic::sanitizer_instrumentation_begin)
+ continue;
+ return &I;
+ }
+ return nullptr;
+}
+
// Returns true iff From->To is a backedge.
// A twist here is that we treat From->To as a backedge if
// * To dominates From or
@@ -695,14 +716,27 @@ void ModuleSanitizerCoverage::instrumentFunction(Function &F) {
// Do not apply any instrumentation for naked functions.
if (F.hasFnAttribute(Attribute::Naked))
return;
- if (F.hasFnAttribute(Attribute::NoSanitizeCoverage))
- return;
- if (F.hasFnAttribute(Attribute::DisableSanitizerInstrumentation))
- return;
+ // Functions that are not instrumented as a whole are still instrumented
+ // inside llvm.sanitizer.instrumentation.begin()/end() regions. There, every
+ // block that reaches a region gets a coverage point right after the point
+ // where it enters the region; function-level instrumentation (entry/exit
+ // callbacks, stack depth, control flow tables) is left out, as it would run
+ // outside of the regions.
+ bool RegionOnly = false;
+ if (F.hasFnAttribute(Attribute::NoSanitizeCoverage) ||
+ F.hasFnAttribute(Attribute::DisableSanitizerInstrumentation)) {
+ if (!hasSanitizerInstrumentationRegions(F))
+ return;
+ RegionOnly = true;
+ }
if (Options.CoverageType >= SanitizerCoverageOptions::SCK_Edge) {
SplitAllCriticalEdges(
F, CriticalEdgeSplittingOptions().setIgnoreUnreachableDests());
}
+ SmallPtrSet<const Instruction *, 16> InRegion;
+ if (RegionOnly)
+ computeSanitizerInstrumentationRegions(F, InRegion);
+ SmallVector<Instruction *, 16> RegionCoveragePoints;
SmallVector<Instruction *, 8> IndirCalls;
SmallVector<BasicBlock *, 16> BlocksToInstrument;
SmallVector<Instruction *, 8> CmpTraceTargets;
@@ -717,9 +751,17 @@ void ModuleSanitizerCoverage::instrumentFunction(Function &F) {
bool IsLeafFunc = true;
for (auto &BB : F) {
- if (shouldInstrumentBlock(F, &BB, DT, PDT, Options))
+ if (RegionOnly) {
+ // Block pruning relies on every block being instrumented, so it is not
+ // applied to region-only functions.
+ if (Instruction *IP = getRegionCoveragePoint(BB, InRegion))
+ RegionCoveragePoints.push_back(IP);
+ } else if (shouldInstrumentBlock(F, &BB, DT, PDT, Options)) {
BlocksToInstrument.push_back(&BB);
+ }
for (auto &Inst : BB) {
+ if (RegionOnly && !InRegion.contains(&Inst))
+ continue;
if (Options.IndirectCalls) {
CallBase *CB = dyn_cast<CallBase>(&Inst);
if (CB && CB->isIndirectCall())
@@ -753,7 +795,17 @@ void ModuleSanitizerCoverage::instrumentFunction(Function &F) {
}
}
- if (Options.CollectControlFlow)
+ // Coverage is inserted at the start of a block, and the PC table refers to
+ // blocks by address. Start a new block at each region coverage point that is
+ // not already at the start of a block other than the entry block.
+ for (Instruction *IP : RegionCoveragePoints) {
+ BasicBlock *BB = IP->getParent();
+ if (BB->isEntryBlock() || IP != &*BB->getFirstInsertionPt())
+ BB = BB->splitBasicBlock(IP, "sancov.region");
+ BlocksToInstrument.push_back(BB);
+ }
+
+ if (Options.CollectControlFlow && !RegionOnly)
createFunctionControlFlow(F);
Value *FunctionGateCmp = nullptr;
@@ -765,7 +817,7 @@ void ModuleSanitizerCoverage::instrumentFunction(Function &F) {
InjectTraceForGep(F, GepTraceTargets);
InjectTraceForLoadsAndStores(F, Loads, Stores);
- if (Options.TracePCEntryExit)
+ if (Options.TracePCEntryExit && !RegionOnly)
InjectTraceForExits(F);
}
diff --git a/llvm/test/Instrumentation/SanitizerCoverage/sanitizer-instrumentation-regions.ll b/llvm/test/Instrumentation/SanitizerCoverage/sanitizer-instrumentation-regions.ll
new file mode 100644
index 0000000000000..d9fffa2c1f6d5
--- /dev/null
+++ b/llvm/test/Instrumentation/SanitizerCoverage/sanitizer-instrumentation-regions.ll
@@ -0,0 +1,205 @@
+; Test that functions that are not instrumented as a whole are instrumented
+; inside llvm.sanitizer.instrumentation.begin()/end() regions.
+; RUN: opt < %s -passes='module(sancov-module)' -sanitizer-coverage-level=4 \
+; RUN: -sanitizer-coverage-trace-pc -sanitizer-coverage-trace-compares \
+; RUN: -sanitizer-coverage-trace-loads -sanitizer-coverage-trace-stores \
+; RUN: -sanitizer-coverage-trace-divs \
+; RUN: -sanitizer-coverage-trace-pc-entry-exit -sanitizer-coverage-stack-depth \
+; RUN: -sanitizer-coverage-control-flow -S | FileCheck %s
+; RUN: opt < %s -passes='module(sancov-module)' -sanitizer-coverage-level=3 \
+; RUN: -sanitizer-coverage-inline-8bit-counters -sanitizer-coverage-pc-table \
+; RUN: -S | FileCheck %s --check-prefix=PCTABLE
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+declare void @llvm.sanitizer.instrumentation.begin()
+declare void @llvm.sanitizer.instrumentation.end()
+declare void @foo()
+
+; PCTABLE: @__sancov_gen_{{.*}} = private constant {{.*}} [ptr blockaddress(@region_disable, %sancov.region), ptr null], section "__sancov_pcs"
+
+; Only the code between begin() and end() is instrumented. The block that
+; contains begin() is split right after it, so that the coverage callback runs
+; inside the region, and the function gets no entry/exit callbacks, no stack
+; depth tracking and no control flow table.
+define i32 @region(ptr %p, i32 %x, ptr %fp) nosanitize_coverage {
+entry:
+ %out.cmp = icmp eq i32 %x, 42
+ %out.load = load i32, ptr %p
+ call void @llvm.sanitizer.instrumentation.begin()
+ %in.cmp = icmp eq i32 %x, 7
+ %in.load = load i32, ptr %p
+ store i32 %x, ptr %p
+ %in.div = udiv i32 %in.load, %x
+ call void %fp()
+ call void @llvm.sanitizer.instrumentation.end()
+ store i32 %out.load, ptr %p
+ call void %fp()
+ %r = select i1 %out.cmp, i32 %in.div, i32 0
+ ret i32 %r
+}
+
+; CHECK-LABEL: define i32 @region(
+; CHECK-NEXT: entry:
+; CHECK-NOT: call void @__sanitizer_cov
+; CHECK: call void @llvm.sanitizer.instrumentation.begin()
+; CHECK-NEXT: br label %sancov.region
+; CHECK: sancov.region:
+; CHECK-NEXT: call void @__sanitizer_cov_trace_pc()
+; CHECK-NEXT: call void @__sanitizer_cov_trace_const_cmp4(i32 7, i32 %x)
+; CHECK-NEXT: %in.cmp = icmp eq i32 %x, 7
+; CHECK-NEXT: call void @__sanitizer_cov_load4(ptr %p)
+; CHECK-NEXT: %in.load = load i32, ptr %p
+; CHECK-NEXT: call void @__sanitizer_cov_store4(ptr %p)
+; CHECK-NEXT: store i32 %x, ptr %p
+; CHECK-NEXT: call void @__sanitizer_cov_trace_div4(i32 %x)
+; CHECK-NEXT: %in.div = udiv i32 %in.load, %x
+; CHECK-NEXT: %0 = ptrtoint ptr %fp to i64
+; CHECK-NEXT: call void @__sanitizer_cov_trace_pc_indir(i64 %0)
+; CHECK-NEXT: call void %fp()
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.end()
+; CHECK-NOT: call void @__sanitizer_cov
+; CHECK: ret i32 %r
+
+; Every block of a region is covered, including the blocks inserted to split
+; critical edges: pruning does not apply, as it assumes that all blocks are
+; instrumented. A block whose only in-region instruction is end() is not.
+define void @region_blocks(i1 %c) nosanitize_coverage {
+entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ br i1 %c, label %then, label %join
+
+then:
+ call void @foo()
+ br label %join
+
+join:
+ call void @llvm.sanitizer.instrumentation.end()
+ br label %exit
+
+exit:
+ ret void
+}
+
+; CHECK-LABEL: define void @region_blocks(
+; CHECK: entry:
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.begin()
+; CHECK-NEXT: br label %sancov.region
+; CHECK: sancov.region:
+; CHECK-NEXT: call void @__sanitizer_cov_trace_pc()
+; CHECK-NEXT: br i1 %c, label %then, label %entry.join_crit_edge
+; CHECK: entry.join_crit_edge:
+; CHECK-NEXT: call void @__sanitizer_cov_trace_pc()
+; CHECK: then:
+; CHECK-NEXT: call void @__sanitizer_cov_trace_pc()
+; CHECK-NEXT: call void @foo()
+; CHECK: join:
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.end()
+; CHECK: exit:
+; CHECK-NEXT: ret void
+
+; disable_sanitizer_instrumentation functions are handled the same way.
+define void @region_disable(i32 %x) disable_sanitizer_instrumentation {
+entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ %c = icmp eq i32 %x, 7
+ call void @llvm.sanitizer.instrumentation.end()
+ ret void
+}
+
+; CHECK-LABEL: define void @region_disable(
+; CHECK: call void @llvm.sanitizer.instrumentation.begin()
+; CHECK: sancov.region:
+; CHECK-NEXT: call void @__sanitizer_cov_trace_pc()
+; CHECK-NEXT: call void @__sanitizer_cov_trace_const_cmp4(i32 7, i32 %x)
+; CHECK: call void @llvm.sanitizer.instrumentation.end()
+; CHECK-NEXT: ret void
+
+; A block reachable without passing through begin() is not in a region.
+define void @diamond(i1 %c, i32 %x) nosanitize_coverage {
+entry:
+ br i1 %c, label %enter, label %join
+
+enter:
+ call void @llvm.sanitizer.instrumentation.begin()
+ br label %join
+
+join:
+ %cmp = icmp eq i32 %x, 7
+ call void @foo()
+ call void @llvm.sanitizer.instrumentation.end()
+ ret void
+}
+
+; CHECK-LABEL: define void @diamond(
+; CHECK: enter:
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.begin()
+; CHECK-NEXT: br label %sancov.region
+; CHECK: sancov.region:
+; CHECK-NEXT: call void @__sanitizer_cov_trace_pc()
+; CHECK: join:
+; CHECK-NOT: call void @__sanitizer_cov
+; CHECK: ret void
+
+; Regions nest: the code after an inner end() is still covered.
+define void @nested(i32 %x) nosanitize_coverage {
+entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.sanitizer.instrumentation.end()
+ %cmp = icmp eq i32 %x, 7
+ call void @llvm.sanitizer.instrumentation.end()
+ %cmp2 = icmp eq i32 %x, 8
+ ret void
+}
+
+; CHECK-LABEL: define void @nested(
+; CHECK: call void @llvm.sanitizer.instrumentation.begin()
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.begin()
+; CHECK-NEXT: br label %sancov.region
+; CHECK: sancov.region:
+; CHECK-NEXT: call void @__sanitizer_cov_trace_pc()
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.end()
+; CHECK-NEXT: call void @__sanitizer_cov_trace_const_cmp4(i32 7, i32 %x)
+; CHECK-NEXT: %cmp = icmp eq i32 %x, 7
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.end()
+; CHECK-NEXT: %cmp2 = icmp eq i32 %x, 8
+; CHECK-NEXT: ret void
+
+; Without regions, nosanitize_coverage functions are left alone.
+define void @no_region(i32 %x) nosanitize_coverage {
+entry:
+ %cmp = icmp eq i32 %x, 7
+ call void @foo()
+ ret void
+}
+
+; CHECK-LABEL: define void @no_region(
+; CHECK-NOT: call void @__sanitizer_cov
+; CHECK: ret void
+
+; Instrumented functions are instrumented as usual, regions or not.
+define void @instrumented(i32 %x) nounwind {
+entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @foo()
+ call void @llvm.sanitizer.instrumentation.end()
+ ret void
+}
+
+; CHECK-LABEL: define void @instrumented(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: call void @__sanitizer_cov_trace_pc_entry()
+; CHECK: call void @llvm.sanitizer.instrumentation.begin()
+; CHECK-NEXT: call void @foo()
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.end()
+; CHECK-NEXT: call void @__sanitizer_cov_trace_pc_exit()
+
+; The PC table refers to the split-off region block, never to the entry block of
+; a region-only function.
+; PCTABLE-LABEL: define void @region_disable(
+; PCTABLE: call void @llvm.sanitizer.instrumentation.begin()
+; PCTABLE: sancov.region:
+; PCTABLE-NEXT: load i8, ptr @__sancov_gen_
+; PCTABLE: call void @llvm.sanitizer.instrumentation.end()
>From 699d86dd11d3456bb95b3637c0e2327c02a7d6d2 Mon Sep 17 00:00:00 2001
From: Alexander Potapenko <glider at google.com>
Date: Tue, 6 Oct 2026 10:24:59 +0000
Subject: [PATCH 5/5] [TSan] Support sanitizer instrumentation regions
Implement `llvm.sanitizer.instrumentation.begin`/`end` regions in
ThreadSanitizer.
Functions with `disable_sanitizer_instrumentation` are normally not
instrumented at all. If they contain regions, plain and atomic memory
accesses and memory intrinsics inside the regions are now instrumented.
They still get no function entry/exit hooks or runtime ignores, as
those would run outside the regions. A call outside a region still
separates the accesses on either side of it when deciding which reads
can be folded into a following write.
Functions without `sanitize_thread` (and without
`disable_sanitizer_instrumentation`) already get their atomics and
entry/exit hooks instrumented; regions now also enable plain memory
access and memory intrinsic instrumentation in them.
The Linux kernel uses this for `noinstr` functions, which KCSAN marks
`disable_sanitizer_instrumentation` (via `__no_kcsan`), and which may
call instrumented code between `instrumentation_begin()` and
`instrumentation_end()`. Data races on accesses in those regions, e.g.
in the syscall and exception entry code, are currently invisible to
KCSAN.
Clang now lowers the region builtins to the intrinsics under
`-fsanitize=thread`.
Tests: `llvm/test/Instrumentation/ThreadSanitizer/sanitizer-instrumentation-regions.ll`,
`clang/test/CodeGen/sanitizer-instrumentation-builtins.c`,
`clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c`.
---
clang/docs/ThreadSanitizer.md | 23 +++
clang/lib/CodeGen/CGBuiltin.cpp | 6 +-
.../sanitizer-instrumentation-builtins.c | 4 +-
...nitizer-instrumentation-regions-pipeline.c | 4 +
llvm/docs/LangRef.md | 2 +
.../Instrumentation/ThreadSanitizer.cpp | 46 ++++-
.../sanitizer-instrumentation-regions.ll | 178 ++++++++++++++++++
7 files changed, 249 insertions(+), 14 deletions(-)
create mode 100644 llvm/test/Instrumentation/ThreadSanitizer/sanitizer-instrumentation-regions.ll
diff --git a/clang/docs/ThreadSanitizer.md b/clang/docs/ThreadSanitizer.md
index 244f2ffee0a19..e251f306cb9dc 100644
--- a/clang/docs/ThreadSanitizer.md
+++ b/clang/docs/ThreadSanitizer.md
@@ -102,6 +102,29 @@ and only if absolutely required; for example for certain code that cannot
tolerate any instrumentation and resulting side-effects. This attribute
overrides `no_sanitize("thread")`.
+## Re-enabling Instrumentation in Regions
+
+Inside a function with `__attribute__((disable_sanitizer_instrumentation))` or
+`__attribute__((no_sanitize("thread")))`, a region delimited by
+`__builtin_sanitizer_instrumentation_begin()` and
+`__builtin_sanitizer_instrumentation_end()` is instrumented again: plain
+memory accesses and memory intrinsics in it are instrumented. A
+`disable_sanitizer_instrumentation` function also gets its atomic accesses
+instrumented only inside regions, and no function entry/exit hooks. Regions
+nest, and code reachable without passing through a
+`__builtin_sanitizer_instrumentation_begin()` is not considered to be in a
+region.
+
+```c
+__attribute__((disable_sanitizer_instrumentation))
+void mostly_uninstrumented(int *p) {
+ ... // not instrumented
+ __builtin_sanitizer_instrumentation_begin();
+ *p = 0; // instrumented
+ __builtin_sanitizer_instrumentation_end();
+}
+```
+
## Interaction of Inlining with Disabling Sanitizer Instrumentation
- A `no_sanitize` function will not be inlined heuristically by the compiler into a sanitized function.
diff --git a/clang/lib/CodeGen/CGBuiltin.cpp b/clang/lib/CodeGen/CGBuiltin.cpp
index fafe7ddba5454..6aabc993a3217 100644
--- a/clang/lib/CodeGen/CGBuiltin.cpp
+++ b/clang/lib/CodeGen/CGBuiltin.cpp
@@ -3913,9 +3913,9 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
// Only emit the region markers for sanitizers that honour them. Elsewhere
// they would just be optimization barriers that no pass ever removes. This
// lets code guard its uses on __has_builtin alone.
- SanitizerMask Consumers = SanitizerKind::KernelMemory |
- SanitizerKind::Address |
- SanitizerKind::KernelAddress;
+ SanitizerMask Consumers =
+ SanitizerKind::KernelMemory | SanitizerKind::Address |
+ SanitizerKind::KernelAddress | SanitizerKind::Thread;
if (getLangOpts().Sanitize.hasOneOf(Consumers) ||
CGM.getCodeGenOpts().hasSanitizeCoverage())
Builder.CreateCall(CGM.getIntrinsic(
diff --git a/clang/test/CodeGen/sanitizer-instrumentation-builtins.c b/clang/test/CodeGen/sanitizer-instrumentation-builtins.c
index 36123ff4a4453..5fe7083ac668f 100644
--- a/clang/test/CodeGen/sanitizer-instrumentation-builtins.c
+++ b/clang/test/CodeGen/sanitizer-instrumentation-builtins.c
@@ -1,5 +1,5 @@
// Check that the sanitizer region markers lower to their intrinsics under
-// -fsanitize=kernel-memory, kernel-address and address, and with
+// -fsanitize=kernel-memory, kernel-address, address and thread, and with
// -fsanitize-coverage, and to nothing without a sanitizer that consumes them.
// RUN: %clang_cc1 -triple x86_64-linux-gnu -fsanitize=kernel-memory \
@@ -14,6 +14,8 @@
// RUN: -fsanitize-coverage-trace-pc -emit-llvm -disable-llvm-passes -o - %s \
// RUN: | FileCheck %s --check-prefix=MARKERS
// RUN: %clang_cc1 -triple x86_64-linux-gnu -fsanitize=thread \
+// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s --check-prefix=MARKERS
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -fsanitize=null \
// RUN: -emit-llvm -disable-llvm-passes -o - %s | FileCheck %s --check-prefix=NONE
#if !__has_builtin(__builtin_sanitizer_instrumentation_begin)
diff --git a/clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c b/clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c
index d16868f2c8e16..69ecd6166d6e0 100644
--- a/clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c
+++ b/clang/test/CodeGen/sanitizer-instrumentation-regions-pipeline.c
@@ -19,6 +19,9 @@
// RUN: -fsanitize-coverage-trace-pc -fsanitize-coverage-trace-cmp \
// RUN: -emit-llvm -o - %s \
// RUN: | FileCheck %s --check-prefixes=CHECK,KMSAN,KCOV
+// RUN: %clang_cc1 -triple x86_64-linux-gnu -O2 -debug-info-kind=line-tables-only \
+// RUN: -fsanitize=thread -mllvm -tsan-instrument-func-entry-exit=0 \
+// RUN: -emit-llvm -o - %s | FileCheck %s --check-prefixes=CHECK,KCSAN
static inline __attribute__((always_inline)) void instrumentation_begin(void) {
__builtin_sanitizer_instrumentation_begin();
@@ -48,6 +51,7 @@ void noinstr_function(int *p, int x) {
// KCOV: call void @__sanitizer_cov_trace_const_cmp4(i32 7,
// KMSAN: call { ptr, ptr } @__msan_metadata_ptr_for_load_4(
// KASAN: call void @__asan_report_load4(
+// KCSAN: call void @__tsan_read4(
// CHECK: call void @instrumented_callee(
// CHECK: call void @llvm.sanitizer.instrumentation.end()
// CHECK-NOT: call {{.*}}@__{{msan|asan|tsan|sanitizer_cov}}
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index a8629a1360006..8173b697fefc2 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -30516,3 +30516,5 @@ MemorySanitizer supports regions in kernel mode (KMSAN), in functions marked wit
AddressSanitizer supports regions in functions that lack the `sanitize_address` attribute or are marked with `disable_sanitizer_instrumentation`. Inside regions, memory accesses and memory intrinsics are instrumented. The stack frame of such a function is not instrumented, accesses to its own `alloca`s are not checked, and no `__asan_handle_no_return` calls are inserted.
SanitizerCoverage supports regions in functions marked with `nosanitize_coverage` or `disable_sanitizer_instrumentation`. Each basic block that enters a region gets a coverage point right after the point where it does (splitting the block if necessary), without pruning, and the requested tracing of comparisons, divisions, GEPs, loads, stores and indirect calls is performed inside regions. Function entry/exit callbacks, stack depth tracing and control flow tables are not emitted for such functions.
+
+ThreadSanitizer supports regions in functions marked with `disable_sanitizer_instrumentation`, and in functions that lack the `sanitize_thread` attribute. In the former, plain and atomic memory accesses and memory intrinsics are instrumented only inside regions, and no function entry/exit hooks are inserted. In the latter, which already get their atomic accesses and function entry/exit instrumented, plain memory accesses and memory intrinsics inside regions are instrumented as well.
diff --git a/llvm/lib/Transforms/Instrumentation/ThreadSanitizer.cpp b/llvm/lib/Transforms/Instrumentation/ThreadSanitizer.cpp
index f05efd863fb74..4de3cf138e528 100644
--- a/llvm/lib/Transforms/Instrumentation/ThreadSanitizer.cpp
+++ b/llvm/lib/Transforms/Instrumentation/ThreadSanitizer.cpp
@@ -507,10 +507,23 @@ bool ThreadSanitizer::sanitizeFunction(Function &F,
if (F.hasFnAttribute(Attribute::Naked))
return false;
+ bool SanitizeFunction = F.hasFnAttribute(Attribute::SanitizeThread);
// __attribute__(disable_sanitizer_instrumentation) prevents all kinds of
- // instrumentation.
- if (F.hasFnAttribute(Attribute::DisableSanitizerInstrumentation))
+ // instrumentation, except inside llvm.sanitizer.instrumentation.begin()/end()
+ // regions. Such a function is instrumented only there, and gets no
+ // function-level instrumentation (entry/exit hooks, runtime ignores), as that
+ // would run outside of the regions.
+ bool RegionOnly =
+ F.hasFnAttribute(Attribute::DisableSanitizerInstrumentation);
+ if (RegionOnly && !hasSanitizerInstrumentationRegions(F))
return false;
+ // Regions also enable memory access instrumentation in functions that lack
+ // the sanitize_thread attribute, which otherwise only get their atomics and
+ // entry/exit instrumented.
+ SmallPtrSet<const Instruction *, 16> InRegion;
+ if (RegionOnly ||
+ (!SanitizeFunction && hasSanitizerInstrumentationRegions(F)))
+ computeSanitizerInstrumentationRegions(F, InRegion);
initialize(*F.getParent(), TLI);
SmallVector<InstructionInfo, 8> AllLoadsAndStores;
@@ -519,7 +532,6 @@ bool ThreadSanitizer::sanitizeFunction(Function &F,
SmallVector<Instruction*, 8> MemIntrinCalls;
bool Res = false;
bool HasCalls = false;
- bool SanitizeFunction = F.hasFnAttribute(Attribute::SanitizeThread);
const DataLayout &DL = F.getDataLayout();
// Traverse all instructions, collect loads/stores/returns, check for calls.
@@ -528,14 +540,24 @@ bool ThreadSanitizer::sanitizeFunction(Function &F,
// Skip instructions inserted by another instrumentation.
if (Inst.hasMetadata(LLVMContext::MD_nosanitize))
continue;
+ bool InstrumentAccesses =
+ (SanitizeFunction && !RegionOnly) || InRegion.contains(&Inst);
+ if (RegionOnly && !InstrumentAccesses) {
+ // A call outside a region still separates the accesses around it.
+ if (isa<CallBase>(Inst))
+ chooseInstructionsToInstrument(LocalLoadsAndStores, AllLoadsAndStores,
+ DL);
+ continue;
+ }
if (isTsanAtomic(&Inst))
AtomicAccesses.push_back(&Inst);
- else if (isa<LoadInst>(Inst) || isa<StoreInst>(Inst))
- LocalLoadsAndStores.push_back(&Inst);
- else if (isa<CallInst>(Inst) || isa<InvokeInst>(Inst)) {
+ else if (isa<LoadInst>(Inst) || isa<StoreInst>(Inst)) {
+ if (InstrumentAccesses)
+ LocalLoadsAndStores.push_back(&Inst);
+ } else if (isa<CallInst>(Inst) || isa<InvokeInst>(Inst)) {
if (CallInst *CI = dyn_cast<CallInst>(&Inst))
maybeMarkSanitizerLibraryCallNoBuiltin(CI, &TLI);
- if (isa<MemIntrinsic>(Inst))
+ if (isa<MemIntrinsic>(Inst) && InstrumentAccesses)
MemIntrinCalls.push_back(&Inst);
HasCalls = true;
chooseInstructionsToInstrument(LocalLoadsAndStores, AllLoadsAndStores,
@@ -549,8 +571,9 @@ bool ThreadSanitizer::sanitizeFunction(Function &F,
// FIXME: many of these accesses do not need to be checked for races
// (e.g. variables that do not escape, etc).
- // Instrument memory accesses only if we want to report bugs in the function.
- if (ClInstrumentMemoryAccesses && SanitizeFunction)
+ // Instrument memory accesses only if we want to report bugs in the function
+ // (or in the region they belong to).
+ if (ClInstrumentMemoryAccesses)
for (const auto &II : AllLoadsAndStores) {
Res |= instrumentLoadOrStore(II, DL);
}
@@ -562,11 +585,14 @@ bool ThreadSanitizer::sanitizeFunction(Function &F,
Res |= instrumentAtomic(Inst, DL);
}
- if (ClInstrumentMemIntrinsics && SanitizeFunction)
+ if (ClInstrumentMemIntrinsics)
for (auto *Inst : MemIntrinCalls) {
Res |= instrumentMemIntrinsic(Inst);
}
+ if (RegionOnly)
+ return Res;
+
if (F.hasFnAttribute("sanitize_thread_no_checking_at_run_time")) {
assert(!F.hasFnAttribute(Attribute::SanitizeThread));
if (HasCalls)
diff --git a/llvm/test/Instrumentation/ThreadSanitizer/sanitizer-instrumentation-regions.ll b/llvm/test/Instrumentation/ThreadSanitizer/sanitizer-instrumentation-regions.ll
new file mode 100644
index 0000000000000..5606e19768cc3
--- /dev/null
+++ b/llvm/test/Instrumentation/ThreadSanitizer/sanitizer-instrumentation-regions.ll
@@ -0,0 +1,178 @@
+; Test that functions that are not instrumented as a whole are instrumented
+; inside llvm.sanitizer.instrumentation.begin()/end() regions.
+; RUN: opt < %s -passes='function(tsan),module(tsan-module)' -S | FileCheck %s
+; KCSAN disables the entry/exit hooks.
+; RUN: opt < %s -passes='function(tsan),module(tsan-module)' \
+; RUN: -tsan-instrument-func-entry-exit=0 -tsan-distinguish-volatile=1 -S \
+; RUN: | FileCheck %s --check-prefix=KCSAN
+
+target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
+target triple = "x86_64-unknown-linux-gnu"
+
+declare void @llvm.sanitizer.instrumentation.begin()
+declare void @llvm.sanitizer.instrumentation.end()
+declare void @llvm.memcpy.p0.p0.i64(ptr, ptr, i64, i1)
+declare void @foo()
+
+; In a disable_sanitizer_instrumentation function, plain and atomic accesses
+; and memory intrinsics are instrumented only inside regions, and there are no
+; entry/exit hooks.
+define i32 @region(ptr %a, ptr %b, ptr %d) disable_sanitizer_instrumentation {
+entry:
+ %out.load = load i32, ptr %a
+ %out.atomic = load atomic i32, ptr %b seq_cst, align 4
+ call void @llvm.memcpy.p0.p0.i64(ptr %d, ptr %a, i64 16, i1 false)
+ call void @llvm.sanitizer.instrumentation.begin()
+ %in.load = load i32, ptr %a
+ store i32 %in.load, ptr %b
+ %in.atomic = load atomic i32, ptr %b seq_cst, align 4
+ store volatile i32 %in.atomic, ptr %d
+ call void @llvm.memcpy.p0.p0.i64(ptr %d, ptr %a, i64 16, i1 false)
+ call void @llvm.sanitizer.instrumentation.end()
+ store i32 %out.load, ptr %b
+ store atomic i32 %out.atomic, ptr %b seq_cst, align 4
+ ret i32 %in.load
+}
+
+; CHECK-LABEL: define i32 @region(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: %out.load = load i32, ptr %a
+; CHECK-NEXT: %out.atomic = load atomic i32, ptr %b seq_cst
+; CHECK-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr %d, ptr %a, i64 16, i1 false)
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.begin()
+; CHECK-NEXT: call void @__tsan_read4(ptr %a)
+; CHECK-NEXT: %in.load = load i32, ptr %a
+; CHECK-NEXT: call void @__tsan_write4(ptr %b)
+; CHECK-NEXT: store i32 %in.load, ptr %b
+; CHECK-NEXT: %0 = call i32 @__tsan_atomic32_load(ptr %b, i32 5)
+; CHECK: call void @__tsan_write4(ptr %d)
+; CHECK-NEXT: store volatile i32
+; CHECK-NEXT: call ptr @__tsan_memcpy(ptr %d, ptr %a, i64 16)
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.end()
+; CHECK-NEXT: store i32 %out.load, ptr %b
+; CHECK-NEXT: store atomic i32 %out.atomic, ptr %b seq_cst
+; CHECK-NEXT: ret i32 %in.load
+
+; KCSAN-LABEL: define i32 @region(
+; KCSAN: call void @llvm.sanitizer.instrumentation.begin()
+; KCSAN-NEXT: call void @__tsan_read4(ptr %a)
+; KCSAN: call void @__tsan_volatile_write4(ptr %d)
+; KCSAN: call void @llvm.sanitizer.instrumentation.end()
+; KCSAN-NOT: call {{.*}}@__tsan_
+; KCSAN: ret i32 %in.load
+
+; A block reachable without passing through begin() is not in a region.
+define void @diamond(i1 %c, ptr %a) disable_sanitizer_instrumentation {
+entry:
+ br i1 %c, label %enter, label %join
+
+enter:
+ call void @llvm.sanitizer.instrumentation.begin()
+ store i32 1, ptr %a
+ br label %join
+
+join:
+ store i32 2, ptr %a
+ call void @llvm.sanitizer.instrumentation.end()
+ ret void
+}
+
+; CHECK-LABEL: define void @diamond(
+; CHECK: enter:
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.begin()
+; CHECK-NEXT: call void @__tsan_write4(ptr %a)
+; CHECK-NEXT: store i32 1, ptr %a
+; CHECK: join:
+; CHECK-NEXT: store i32 2, ptr %a
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.end()
+; CHECK-NEXT: ret void
+
+; Regions nest.
+define void @nested(ptr %a) disable_sanitizer_instrumentation {
+entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.sanitizer.instrumentation.end()
+ store i32 1, ptr %a
+ call void @llvm.sanitizer.instrumentation.end()
+ store i32 2, ptr %a
+ ret void
+}
+
+; CHECK-LABEL: define void @nested(
+; CHECK: call void @llvm.sanitizer.instrumentation.end()
+; CHECK-NEXT: call void @__tsan_write4(ptr %a)
+; CHECK-NEXT: store i32 1, ptr %a
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.end()
+; CHECK-NEXT: store i32 2, ptr %a
+; CHECK-NEXT: ret void
+
+; A call outside the regions separates the accesses on either side of it, so
+; the read before the write is not folded into the write.
+define void @call_between_regions(ptr %a) disable_sanitizer_instrumentation {
+entry:
+ call void @llvm.sanitizer.instrumentation.begin()
+ %v = load i32, ptr %a
+ call void @llvm.sanitizer.instrumentation.end()
+ call void @foo()
+ call void @llvm.sanitizer.instrumentation.begin()
+ store i32 %v, ptr %a
+ call void @llvm.sanitizer.instrumentation.end()
+ ret void
+}
+
+; CHECK-LABEL: define void @call_between_regions(
+; CHECK: call void @__tsan_read4(ptr %a)
+; CHECK: call void @foo()
+; CHECK: call void @__tsan_write4(ptr %a)
+
+; Functions without sanitize_thread still get their atomics and entry/exit
+; hooks instrumented everywhere; regions add plain accesses.
+define i32 @no_sanitize_thread(ptr %a) {
+entry:
+ %out.load = load i32, ptr %a
+ call void @llvm.sanitizer.instrumentation.begin()
+ %in.load = load i32, ptr %a
+ call void @llvm.sanitizer.instrumentation.end()
+ %atomic = load atomic i32, ptr %a seq_cst, align 4
+ %r = add i32 %out.load, %in.load
+ ret i32 %r
+}
+
+; CHECK-LABEL: define i32 @no_sanitize_thread(
+; CHECK: call void @__tsan_func_entry(
+; CHECK-NEXT: %out.load = load i32, ptr %a
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.begin()
+; CHECK-NEXT: call void @__tsan_read4(ptr %a)
+; CHECK-NEXT: %in.load = load i32, ptr %a
+; CHECK-NEXT: call void @llvm.sanitizer.instrumentation.end()
+; CHECK-NEXT: call i32 @__tsan_atomic32_load(ptr %a, i32 5)
+; CHECK: call void @__tsan_func_exit()
+
+; Without regions, disable_sanitizer_instrumentation functions are left alone.
+define i32 @no_region(ptr %a) disable_sanitizer_instrumentation {
+entry:
+ %v = load i32, ptr %a
+ %atomic = load atomic i32, ptr %a seq_cst, align 4
+ ret i32 %v
+}
+
+; CHECK-LABEL: define i32 @no_region(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: %v = load i32, ptr %a
+; CHECK-NEXT: %atomic = load atomic i32, ptr %a seq_cst
+; CHECK-NEXT: ret i32 %v
+
+; Instrumented functions are instrumented as usual, regions or not.
+define i32 @sanitized(ptr %a) sanitize_thread {
+entry:
+ %out.load = load i32, ptr %a
+ call void @llvm.sanitizer.instrumentation.begin()
+ call void @llvm.sanitizer.instrumentation.end()
+ ret i32 %out.load
+}
+
+; CHECK-LABEL: define i32 @sanitized(
+; CHECK: call void @__tsan_func_entry(
+; CHECK-NEXT: call void @__tsan_read4(ptr %a)
+; CHECK-NEXT: %out.load = load i32, ptr %a
More information about the cfe-commits
mailing list