[llvm-branch-commits] [clang] [compiler-rt] [compiler-rt] Add 'csan' library for the concurrency sanitizer (PR #225782)
via llvm-branch-commits
llvm-branch-commits at lists.llvm.org
Wed Sep 23 07:02:50 PDT 2026
llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-backend-amdgpu
Author: Joseph Huber (jhuber6)
<details>
<summary>Changes</summary>
Summary:
Adds the runtime for the concurrency sanitizer, both CPU and GPU.
Fundamentally, this works using the following pseudocode:
```c
static u64 watchpoints[N]; // Hash-indexed, zero is empty.
// Emitted before the access, so we never trip on our own write.
void check_access(volatile void *addr, u32 size, u32 type) {
// Every access probes. A read conflicts only with a watched write, a
// write conflicts with either.
if (u64 *wp = find_watchpoint(addr, size, type))
consume(wp, this_pc()); // Hand our location to the owner.
if (!should_sample()) // Wave-uniform, 1-in-N chance.
return;
u64 *wp = arm_watchpoint(addr, size, type);
if (!wp) // Slot is already taken.
return;
auto old = read(addr, size);
delay(ctx.rand()); // Randomized, bounded.
if (void *peer = disarm(wp))
report_race(addr, this_pc(), peer);
else if (read(addr, size) != old)
report_race(addr, this_pc(), UNKNOWN);
}
```
This implementation is kept intentionally minimal to simplify the review
process. Many options are planned for later. The intended use is for
users to pass `-fsanitize=concurrency` for supported compilations.
---
<sub>Stack created with <a href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
---
Patch is 117.79 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/225782.diff
58 Files Affected:
- (added) clang/docs/ConcurrencySanitizer.md (+212)
- (modified) clang/docs/index.md (+1)
- (modified) compiler-rt/cmake/caches/AMDGPU.cmake (+1-1)
- (modified) compiler-rt/cmake/config-ix.cmake (+10-1)
- (added) compiler-rt/lib/csan/CMakeLists.txt (+60)
- (added) compiler-rt/lib/csan/csan.cpp (+334)
- (added) compiler-rt/lib/csan/csan.h (+64)
- (added) compiler-rt/lib/csan/csan_defs.h (+35)
- (added) compiler-rt/lib/csan/csan_flags.inc (+23)
- (added) compiler-rt/lib/csan/csan_gpu.cpp (+456)
- (added) compiler-rt/lib/csan/csan_offload_packet.h (+39)
- (added) compiler-rt/lib/csan/csan_report.cpp (+279)
- (added) compiler-rt/lib/csan/csan_watch.h (+163)
- (added) compiler-rt/lib/csan/offload/CMakeLists.txt (+25)
- (added) compiler-rt/lib/csan/offload/csan_offload.h (+31)
- (added) compiler-rt/lib/csan/offload/csan_offload_hsa_interceptors.cpp (+370)
- (added) compiler-rt/lib/csan/offload/csan_offload_report.cpp (+184)
- (modified) compiler-rt/lib/sanitizer_common/sanitizer_internal_defs.h (+3)
- (modified) compiler-rt/lib/sanitizer_common/sanitizer_offload.cpp (+46-3)
- (modified) compiler-rt/lib/sanitizer_common/sanitizer_offload.h (+3)
- (modified) compiler-rt/lib/sanitizer_common/sanitizer_offload_hsa.h (+2)
- (modified) compiler-rt/lib/sanitizer_common/sanitizer_offload_image.cpp (+21)
- (modified) compiler-rt/lib/sanitizer_common/sanitizer_offload_opcodes.h (+1)
- (modified) compiler-rt/lib/sanitizer_common/sanitizer_symbolizer.h (+4)
- (modified) compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cpp (+15)
- (added) compiler-rt/test/csan/AMDGPU/aba-race.hip (+33)
- (added) compiler-rt/test/csan/AMDGPU/array-race.hip (+22)
- (added) compiler-rt/test/csan/AMDGPU/atomic-nonatomic.hip (+26)
- (added) compiler-rt/test/csan/AMDGPU/atomic.hip (+17)
- (added) compiler-rt/test/csan/AMDGPU/disjoint.hip (+20)
- (added) compiler-rt/test/csan/AMDGPU/global-race.hip (+23)
- (added) compiler-rt/test/csan/AMDGPU/helper-race.hip (+26)
- (added) compiler-rt/test/csan/AMDGPU/lds-disjoint.hip (+21)
- (added) compiler-rt/test/csan/AMDGPU/lds-race.hip (+21)
- (added) compiler-rt/test/csan/AMDGPU/lit.local.cfg.py (+11)
- (added) compiler-rt/test/csan/AMDGPU/memcpy-large-race.hip (+28)
- (added) compiler-rt/test/csan/AMDGPU/memcpy-race.hip (+22)
- (added) compiler-rt/test/csan/AMDGPU/memmove-race.hip (+28)
- (added) compiler-rt/test/csan/AMDGPU/openmp-race.cpp (+19)
- (added) compiler-rt/test/csan/AMDGPU/race.h (+32)
- (added) compiler-rt/test/csan/AMDGPU/shared-watchpoints.hip (+59)
- (added) compiler-rt/test/csan/AMDGPU/single-thread.hip (+17)
- (added) compiler-rt/test/csan/AMDGPU/write-read-race.hip (+26)
- (added) compiler-rt/test/csan/CMakeLists.txt (+25)
- (added) compiler-rt/test/csan/access-sizes.cpp (+62)
- (added) compiler-rt/test/csan/atomic-nonatomic.cpp (+27)
- (added) compiler-rt/test/csan/atomic.cpp (+22)
- (added) compiler-rt/test/csan/halt-on-error.cpp (+24)
- (added) compiler-rt/test/csan/ignore-thread.cpp (+45)
- (added) compiler-rt/test/csan/large-access.cpp (+19)
- (added) compiler-rt/test/csan/large-range-race.cpp (+25)
- (added) compiler-rt/test/csan/lit.common.cfg.py (+62)
- (added) compiler-rt/test/csan/lit.site.cfg.py.in (+8)
- (added) compiler-rt/test/csan/race.cpp (+26)
- (added) compiler-rt/test/csan/read-read.cpp (+23)
- (added) compiler-rt/test/csan/single-thread.cpp (+11)
- (added) compiler-rt/test/csan/unknown-origin.cpp (+31)
- (modified) compiler-rt/test/lit.common.cfg.py (+1)
``````````diff
diff --git a/clang/docs/ConcurrencySanitizer.md b/clang/docs/ConcurrencySanitizer.md
new file mode 100644
index 00000000000000..fdf62e9f87d447
--- /dev/null
+++ b/clang/docs/ConcurrencySanitizer.md
@@ -0,0 +1,212 @@
+# ConcurrencySanitizer
+
+## Introduction
+
+ConcurrencySanitizer (CSan) is a data race detector for CPU and GPU programs.
+It consists of an LLVM instrumentation pass and run-time libraries in
+compiler-rt.
+
+Unlike ThreadSanitizer, CSan does not maintain a happens-before model or shadow
+state for every memory location. Memory accesses are sampled at random and
+stalled to detect unordered accesses. Importantly, this method has **no false
+positives** and **fixed memory overhead.** However, the probabalistic nature
+means that many runs are required to state confidently that the application is
+not racy.
+
+## How to build
+
+Build LLVM/Clang with [CMake](https://llvm.org/docs/CMake.html). To enable
+offloading support, the easiest configuration is provided in a CMake cache file.
+
+```sh
+cmake ../llvm -G Ninja \
+ -C ../offload/cmake/caches/Offload.cmake \
+ -DCMAKE_BUILD_TYPE=Release \
+ -DCMAKE_INSTALL_PREFIX=<PATH>
+```
+
+## Supported platforms
+
+CSan currently supports:
+
+- Linux on x86-64.
+- AMDGPU devices on Linux through the HSA runtime.
+
+Support for other hosts and GPU targets is not currently implemented.
+
+## Usage
+
+Compile and link the complete program with `-fsanitize=concurrency`. Use `-g`
+to include source locations in reports and `-O1` or higher for representative
+optimized code.
+
+For a CPU program:
+
+```console
+$ clang++ -fsanitize=concurrency -g -O1 race.cpp -pthread
+$ ./a.out
+```
+
+For a HIP program:
+
+```console
+$ clang++ -x hip --offload-arch=gfx1030 -fsanitize=concurrency \
+ -g -O1 race.hip
+$ ./a.out
+```
+
+For an OpenMP offload program:
+
+```console
+$ clang++ -fopenmp --offload-arch=gfx1030 -fsanitize=concurrency \
+ -g -O1 race.cpp
+$ ./a.out
+```
+
+When CSan detects a race, it writes a report to standard error. A watchpoint
+report normally identifies both conflicting accesses:
+
+```text
+WARNING: ConcurrencySanitizer: data race
+ Write of size 4 at 0x...:
+ #0 update race.cpp:12
+ Previous write of size 4 at 0x...:
+ #0 update race.cpp:13
+```
+
+A value change without a matching instrumented access is reported as a
+`data race of unknown origin` and contains only the sampled access. GPU reports
+also identify the block, thread, and lane and may name the global variable
+containing the raced address.
+
+## How it works
+
+The instrumentation marks memory accesses with a runtime function. The runtime
+then samples these accesses. Sampling is fundamentally stalling a given value
+while checking if any other threads have touched it during that window. This is
+done using a **watchpoint** table and by **value comparison**. Each probed
+access sets up a tripwire and checks later if another thread triggered it like
+in the following pseudocode:
+
+```c
+static u64 watchpoints[N]; // Hash-indexed, zero is empty.
+
+// Emitted before the access to simulate a stalled operation.
+void check_access(volatile void *addr, u32 size, u32 type) {
+ // Every access probes. A read conflicts only with a watched write, a
+ // write conflicts with either.
+ if (u64 *wp = find_watchpoint(addr, size, type))
+ consume(wp, this_pc()); // Hand our location to the owner.
+
+ if (!should_sample()) // 1-in-N chance for a non-atomic access.
+ return;
+
+ u64 *wp = arm_watchpoint(addr, size, type);
+ if (!wp) // Slot is already taken.
+ return;
+
+ auto old = read(addr, size);
+ delay(ctx.rand()); // Randomized, bounded.
+ auto new = read(addr, size);
+
+ if (void *peer = disarm(wp))
+ report_race(addr, this_pc(), peer);
+ else if (new != old)
+ report_race(addr, this_pc(), UNKNOWN);
+}
+```
+
+The probability of detecting a race and the overhead of the sanitizer is
+directly related to the probability of sampling and the length of the delay. The
+watchpoint table can be omitted to make a minimal detector with less coverage.
+
+## `__has_feature(concurrency_sanitizer)`
+
+Code can test whether CSan instrumentation is enabled:
+
+```c
+#if defined(__has_feature)
+# if __has_feature(concurrency_sanitizer)
+// Code built with ConcurrencySanitizer.
+# endif
+#endif
+```
+
+Clang also defines `__SANITIZE_CONCURRENCY__` while compiling with
+`-fsanitize=concurrency`.
+
+## Disabling instrumentation
+
+Use `no_sanitize("concurrency")` to disable CSan memory-access instrumentation
+for a function:
+
+```c
+__attribute__((no_sanitize("concurrency")))
+void uninstrumented_function() {
+ // ...
+}
+```
+
+The `disable_sanitizer_instrumentation` attribute disables all sanitizer
+instrumentation and takes precedence over `no_sanitize` attributes. Use either
+form carefully: uninstrumented accesses cannot consume watchpoints and may
+reduce detection or produce reports of unknown origin.
+
+## Run-time flags
+
+CSan reads options from the `CSAN_OPTIONS` environment variable:
+
+```console
+$ CSAN_OPTIONS="skip_watch=1000:udelay=200" ./a.out
+```
+
+The CSan-specific host options are:
+
+- `skip_watch` (default: `4000`): Approximate number of accesses skipped by a
+ thread before arming another watchpoint.
+- `udelay` (default: `80`): Microseconds to delay after arming a watchpoint.
+- `halt_on_error` (default: `false`): Terminate after the first host report.
+
+These options currently control the host detector. The AMDGPU sampling rate and
+delay interval are fixed. Common sanitizer options still control facilities
+such as symbolization and report formatting on the host side of GPU reporting.
+
+Use `CSAN_OPTIONS=help=1` to print all CSan and sanitizer-common options.
+Programs may provide host defaults by defining:
+
+```c++
+extern "C" const char *__csan_default_options() {
+ return "skip_watch=1000:udelay=200";
+}
+```
+
+Environment options take precedence over `__csan_default_options`.
+
+## Limitations
+
+- CSan is probabilistic. It detects races that overlap a sampled observation
+ window rather than proving that a program is race-free.
+- CSan does not implement ThreadSanitizer's happens-before model, lock-order
+ analysis, thread history, or synchronization diagnostics.
+- CSan cannot be combined with several other full sanitizers, including
+ ThreadSanitizer and AddressSanitizer.
+
+## Security considerations
+
+ConcurrencySanitizer is a testing tool. Its runtime is not intended for
+production executables and was not developed under security-sensitive runtime
+constraints. Instrumented programs deliberately change scheduling and reserve
+additional memory.
+
+## Current status
+
+ConcurrencySanitizer is experimental. Its interface, reports, supported targets,
+sampling policy, and run-time options may change. The compiler-rt test suite can
+be run with:
+
+```console
+$ ninja check-csan
+```
+
+Minimized test cases and reports from real CPU and GPU applications are
+welcome.
diff --git a/clang/docs/index.md b/clang/docs/index.md
index bb2ba828637af4..e7eaced355d3c9 100644
--- a/clang/docs/index.md
+++ b/clang/docs/index.md
@@ -36,6 +36,7 @@ FunctionEffectAnalysis
AddressSanitizer
HardwareAssistedAddressSanitizer
ThreadSanitizer
+ConcurrencySanitizer
MemorySanitizer
UndefinedBehaviorSanitizer
DataFlowSanitizer
diff --git a/compiler-rt/cmake/caches/AMDGPU.cmake b/compiler-rt/cmake/caches/AMDGPU.cmake
index bb5ab22edaf69e..34a5559fe52990 100644
--- a/compiler-rt/cmake/caches/AMDGPU.cmake
+++ b/compiler-rt/cmake/caches/AMDGPU.cmake
@@ -7,7 +7,7 @@ set(COMPILER_RT_BUILD_BUILTINS ON CACHE BOOL "")
set(COMPILER_RT_BAREMETAL_BUILD ON CACHE BOOL "")
set(COMPILER_RT_BUILD_CRT OFF CACHE BOOL "")
set(COMPILER_RT_BUILD_SANITIZERS ON CACHE BOOL "")
-set(COMPILER_RT_SANITIZERS_TO_BUILD "ubsan;ubsan_minimal" CACHE STRING "")
+set(COMPILER_RT_SANITIZERS_TO_BUILD "ubsan;ubsan_minimal;csan" CACHE STRING "")
set(COMPILER_RT_BUILD_XRAY OFF CACHE BOOL "")
set(COMPILER_RT_BUILD_LIBFUZZER OFF CACHE BOOL "")
set(COMPILER_RT_BUILD_PROFILE ON CACHE BOOL "")
diff --git a/compiler-rt/cmake/config-ix.cmake b/compiler-rt/cmake/config-ix.cmake
index e1abc7eb5c44e9..02ba1edb2de155 100644
--- a/compiler-rt/cmake/config-ix.cmake
+++ b/compiler-rt/cmake/config-ix.cmake
@@ -768,7 +768,7 @@ if(COMPILER_RT_SUPPORTED_ARCH)
endif()
message(STATUS "Compiler-RT supported architectures: ${COMPILER_RT_SUPPORTED_ARCH}")
-set(ALL_SANITIZERS asan;rtsan;dfsan;msan;hwasan;tsan;tysan;safestack;cfi;scudo_standalone;ubsan_minimal;gwp_asan;nsan;asan_abi)
+set(ALL_SANITIZERS asan;rtsan;dfsan;msan;hwasan;tsan;csan;tysan;safestack;cfi;scudo_standalone;ubsan_minimal;gwp_asan;nsan;asan_abi)
set(COMPILER_RT_SANITIZERS_TO_BUILD all CACHE STRING
"sanitizers to build if supported on the target (all;${ALL_SANITIZERS})")
list_replace(COMPILER_RT_SANITIZERS_TO_BUILD all "${ALL_SANITIZERS}")
@@ -897,6 +897,15 @@ else()
set(COMPILER_RT_HAS_UBSAN FALSE)
endif()
+if ((COMPILER_RT_HAS_SANITIZER_COMMON AND UBSAN_SUPPORTED_ARCH AND
+ OS_NAME MATCHES "Linux" AND NOT ANDROID)
+ OR (COMPILER_RT_GPU_BUILD AND COMPILER_RT_TARGET_AMDGPU AND
+ UBSAN_SUPPORTED_ARCH))
+ set(COMPILER_RT_HAS_CSAN TRUE)
+else()
+ set(COMPILER_RT_HAS_CSAN FALSE)
+endif()
+
if (UBSAN_SUPPORTED_ARCH AND
(OS_NAME MATCHES "Linux|FreeBSD|NetBSD|Android|Darwin|SunOS" OR
COMPILER_RT_GPU_BUILD))
diff --git a/compiler-rt/lib/csan/CMakeLists.txt b/compiler-rt/lib/csan/CMakeLists.txt
new file mode 100644
index 00000000000000..b0791c39e2a4cf
--- /dev/null
+++ b/compiler-rt/lib/csan/CMakeLists.txt
@@ -0,0 +1,60 @@
+# Build for the ConcurrencySanitizer runtime support library.
+
+include_directories(.)
+include_directories(..)
+include_directories(../../include)
+
+# GPU targets only need the watchpoint runtime. The host interceptor archive is
+# built in the offload/ project.
+if(COMPILER_RT_GPU_BUILD)
+ include(FindLibcCommonUtils)
+ if(NOT TARGET llvm-libc-common-utilities)
+ add_compiler_rt_component(csan)
+ return()
+ endif()
+
+ set(CSAN_SOURCES csan_gpu.cpp)
+ set(CSAN_HEADERS csan_defs.h csan_offload_packet.h csan_watch.h)
+
+ set(CSAN_CFLAGS
+ ${SANITIZER_COMMON_CFLAGS}
+ -DSANITIZER_COMMON_NO_REDEFINE_BUILTINS)
+ append_rtti_flag(OFF CSAN_CFLAGS)
+
+ add_compiler_rt_component(csan)
+
+ add_compiler_rt_runtime(clang_rt.csan
+ STATIC
+ ARCHS ${UBSAN_SUPPORTED_ARCH}
+ SOURCES ${CSAN_SOURCES}
+ ADDITIONAL_HEADERS ${CSAN_HEADERS}
+ CFLAGS ${CSAN_CFLAGS}
+ LINK_LIBS llvm-libc-common-utilities
+ PARENT_TARGET csan)
+
+ return()
+endif()
+
+set(CSAN_CFLAGS ${SANITIZER_COMMON_CFLAGS})
+append_rtti_flag(OFF CSAN_CFLAGS)
+
+add_compiler_rt_component(csan)
+
+add_compiler_rt_runtime(clang_rt.csan
+ STATIC
+ ARCHS ${UBSAN_SUPPORTED_ARCH}
+ SOURCES csan.cpp csan_report.cpp
+ ADDITIONAL_HEADERS csan.h csan_defs.h csan_flags.inc csan_watch.h
+ OBJECT_LIBS RTSanitizerCommon
+ RTSanitizerCommonLibc
+ RTSanitizerCommonCoverage
+ RTSanitizerCommonSymbolizer
+ RTSanitizerCommonSymbolizerInternal
+ RTInterception
+ CFLAGS ${CSAN_CFLAGS}
+ PARENT_TARGET csan)
+
+# Thin host interceptors for offloading. Linked with -u to keep it separate.
+if(OS_NAME MATCHES "Linux" AND NOT ANDROID)
+ add_subdirectory(offload)
+endif()
diff --git a/compiler-rt/lib/csan/csan.cpp b/compiler-rt/lib/csan/csan.cpp
new file mode 100644
index 00000000000000..148e30643333ae
--- /dev/null
+++ b/compiler-rt/lib/csan/csan.cpp
@@ -0,0 +1,334 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Watchpoint-based host data race detector inspired by KCSAN. The host and GPU
+/// runtimes share a configurable packed-watchpoint implementation.
+///
+//===----------------------------------------------------------------------===//
+
+#include "csan.h"
+#include "csan_watch.h"
+
+#include "sanitizer_common/sanitizer_atomic.h"
+#include "sanitizer_common/sanitizer_common.h"
+#include "sanitizer_common/sanitizer_flag_parser.h"
+#include "sanitizer_common/sanitizer_flags.h"
+#include "sanitizer_common/sanitizer_internal_defs.h"
+#include "sanitizer_common/sanitizer_libc.h"
+#include "sanitizer_common/sanitizer_mutex.h"
+#include "sanitizer_common/sanitizer_stacktrace.h"
+#include "sanitizer_common/sanitizer_symbolizer.h"
+
+using namespace __sanitizer;
+
+#define INTERFACE extern "C" SANITIZER_INTERFACE_ATTRIBUTE
+
+static constexpr u32 kHostWatchpointEntries = 64;
+static constexpr u32 kHostCheckAdjacent = 1;
+static constexpr uptr kHostSlotRange = 4096;
+static constexpr uptr kHostMaxAccessSize =
+ kHostSlotRange * (1 + kHostCheckAdjacent);
+static_assert(__atomic_always_lock_free(sizeof(u64), nullptr),
+ "host watchpoints must be lock-free");
+
+using HostWatchpointTable =
+ __csan::WatchpointTable<kHostMaxAccessSize, kHostCheckAdjacent>;
+static u64 HostWatchpoints[kHostWatchpointEntries +
+ HostWatchpointTable::OverflowEntries];
+static_assert(HostWatchpointTable::AddressBits == 48,
+ "host watchpoints require 48-bit pointers");
+
+static HostWatchpointTable GetHostWatchpoints() {
+ return HostWatchpointTable(HostWatchpoints);
+}
+
+static u32 HostWatchpointSlot(uptr Address) {
+ return Address / kHostSlotRange % kHostWatchpointEntries;
+}
+
+namespace __csan {
+
+Flags flags_data;
+
+static void Initialize();
+
+static atomic_uint64_t NumDataRaces;
+static THREADLOCAL u32 DisableCount;
+static THREADLOCAL s32 Skip;
+static THREADLOCAL u32 RandState;
+
+namespace {
+struct ScopedDisable {
+ ScopedDisable() { ++DisableCount; }
+ ~ScopedDisable() { --DisableCount; }
+};
+} // namespace
+
+void RecordDataRace() {
+ atomic_fetch_add(&NumDataRaces, 1, memory_order_relaxed);
+}
+
+void Flags::SetDefaults() {
+#define CSAN_FLAG(Type, Name, DefaultValue, Description) Name = DefaultValue;
+#include "csan_flags.inc"
+#undef CSAN_FLAG
+}
+
+static void RegisterCsanFlags(FlagParser *Parser, Flags *F) {
+#define CSAN_FLAG(Type, Name, DefaultValue, Description) \
+ RegisterFlag(Parser, #Name, Description, &F->Name);
+#include "csan_flags.inc"
+#undef CSAN_FLAG
+}
+
+void InitializeFlags() {
+ SetCommonFlagsDefaults();
+ {
+ CommonFlags CF;
+ CF.CopyFrom(*common_flags());
+ CF.external_symbolizer_path = GetEnv("CSAN_SYMBOLIZER_PATH");
+ OverrideCommonFlags(CF);
+ }
+
+ flags()->SetDefaults();
+
+ FlagParser Parser;
+ RegisterCommonFlags(&Parser);
+ RegisterCsanFlags(&Parser, flags());
+ Parser.ParseString(__csan_default_options());
+ Parser.ParseStringFromEnv("CSAN_OPTIONS");
+ InitializeCommonFlags();
+ if (Verbosity())
+ ReportUnrecognizedFlags();
+ if (common_flags()->help)
+ Parser.PrintFlagDescriptions();
+}
+
+static u32 Random(u32 EpRo) {
+ if (EpRo <= 1)
+ return 0;
+ u32 State = RandState;
+ if (!State)
+ State = (u32)__builtin_readcyclecounter() | 1u;
+ State = 1664525u * State + 1013904223u;
+ RandState = State;
+ return State % EpRo;
+}
+
+static void ResetSkip() {
+ s32 Count = flags()->skip_watch;
+ if (Count < 0)
+ Count = 0;
+ if (Count)
+ Count -= (s32)Random((u32)Count);
+ Skip = Count;
+}
+
+static bool ShouldWatch(int Type) {
+ if (Type & CSAN_ACCESS_ATOMIC)
+ return false;
+ if (--Skip >= 0)
+ return false;
+ return true;
+}
+
+static void DelayAccess(int Type) {
+ s32 Delay = flags()->udelay;
+ if (Delay < 0)
+ Delay = 0;
+ if (Delay) {
+ u32 Skew = (Type & CSAN_ACCESS_COMPOUND) ? 1u : 0u;
+ u32 Span = (u32)Delay >> Skew;
+ if (!Span)
+ Span = (u32)Delay;
+ Delay -= (s32)Random(Span);
+ }
+ if (Delay)
+ internal_usleep((u64)Delay);
+}
+
+static u64 ReadRange(const volatile u8 *Bytes, uptr Size) {
+ u64 Sum = 0xcbf29ce484222325ull;
+ uptr I = 0;
+ for (; I < Size && ((uptr)(Bytes + I) & 7u); ++I)
+ Sum = (Sum ^ Bytes[I]) * 0x100000001b3ull;
+ for (; I + 8 <= Size; I += 8)
+ Sum = (Sum ^ *(const volatile u64 *)(Bytes + I)) * 0x100000001b3ull;
+ for (; I < Size; ++I)
+ Sum = (Sum ^ Bytes[I]) * 0x100000001b3ull;
+ return Sum;
+}
+
+static u64 ReadInstrumented(const volatile void *Ptr, uptr Size) {
+ switch (Size) {
+ case 1:
+ return *(const volatile u8 *)Ptr;
+ case 2:
+ return *(const volatile u16 *)Ptr;
+ case 4:
+ return *(const volatile u32 *)Ptr;
+ case 8:
+ return *(const volatile u64 *)Ptr;
+ default:
+ return ReadRange((const volatile u8 *)Ptr, Size);
+ }
+}
+
+static AccessInfo MakeAccessInfo(const volatile void *Ptr, uptr Size, int Type,
+ uptr PC, uptr BP) {
+ AccessInfo AI;
+ AI.ptr = Ptr;
+ AI.size = Size;
+ AI.access_type = Type;
+ AI.tid = (u32)GetTid();
+ AI.pc = PC;
+ AI.bp = BP;
+ return AI;
+}
+
+NOINLINE static void FoundWatchpoint(const volatile void *Ptr, uptr Size,
+ int Type, uptr PC, uptr, u64 *WP,
+ u64 Encoded) {
+ ScopedDisable Disable;
+ GetHostWatchpoints().TryConsume(WP, Encoded, PC,
+ (Type & CSAN_ACCESS_WRITE) != 0, Size);
+}
+
+NOINLINE static void SetupWatchpoint(const volatile void *Ptr, uptr Size,
+ int Type, uptr PC, uptr BP) {
+ ScopedDisable Disable;
+ ResetSkip();
+
+ if ((uptr)Ptr < GetPageSizeCached())
+ return;
+
+ u64 *WP = GetHostWatchpoints().Insert((uptr)Ptr, Size,
+ (Type & CSAN_ACCESS_WRITE) != 0,
+ HostWatchpointSlot((uptr)Ptr));
+ if (!WP)
+ return;
+
+ const u64 Old = ReadInstrumented(Ptr, Size);
+ DelayAccess(Type);
+ const u64 New = ReadInstrumented(Ptr, Size);
+
+ ValueChange VC = Old != New ? kValueChangeTrue : kValueChangeMaybe;
+
+ const AccessInfo AI = MakeAccessInfo(Ptr, Size, Type, PC, BP);
+ void *Peer;
+ int PeerAccess;
+ u32 PeerSize;
+ if (!GetHostWatchpoints().Consume(WP, Peer, PeerAccess, PeerSize)) {
+ ReportKnownOrigin(AI, VC, (uptr)Peer, PeerAccess, PeerSize, Old, New);
+ } else if (VC == kValueChangeTrue) {
+ ReportUnknownOrigin(AI, Old, New);
+ }
+
+ GetHostWatchpoints().Remove(WP);
+}
+
+ALWAYS_INLINE static void CheckAccess(const volatile void *Ptr, uptr Size,
+ int Type, uptr PC, uptr BP) {
+ if (UNLIKELY(!Ptr || !Size))
+ return;
+ if (Size > kHostMaxAccessSize)
+ Size = kHostMaxAccessSize;
+ if (UNLIKELY(uptr(Ptr) & ~HostWatchpointTable::AddressMask))
+ return;
+ Initialize();
+ if (UNLIKELY(DisableCount))
+ return;
+
+ u64 Encoded;
+ u64 *WP =
+ GetHostWatchpoints().Find((uptr)Ptr, Size, !(Type & CSAN_ACCESS_WRITE),
+ HostWatchpointSlot((uptr)Ptr), Encoded);
+ if (UNLIKELY(WP != nullptr))
+ FoundWatchpoint(Ptr, Size, Type, PC, BP, WP, Encoded);
+ else if (UNLIKELY(ShouldWatch(Type)))
+ SetupWatchpoint(Ptr, Size, Type, PC, BP);
+}
+
+static StaticSpinMutex InitMutex;
+static atomic_uint8_t Initialized;
+
+void Initialize() {
+ if (LIKELY(atomic_load(&Initialized, memory_order_acquire)))
+ return;
+ SpinMutexLock L(&InitMutex);
+ if (atomic_load(&Initialized, memory_order_relaxed))
+ return;
+ SanitizerToolName = "ConcurrencySanitizer";
+ CacheBinaryName();
+ InitializeFlags();
+ atomic_store(&Initialized, 1, memory_order_release);
+ Symbolizer::LateInitialize();
+}
+
+} // namespace __csan
+
+SANITIZER_INTERFACE_WEAK_DEF(const char *, __csan_default_options, void) {
+ return "";
+}
+
+INTERFACE u64 __csan_get_num_data_races() {
+ return atomic_load(&__csan::NumDataRaces, memory_order_relaxed);
+}
+
+INTERFACE void __csan_init() { __csan::Initialize(); }
+
+INTERFACE void __csan_func_entry(void *) {}
+INTERFACE void __csan_func_exit() {}
+INTERFACE void __csan_ignore_thread_begin() { ++__csan::DisableCount; }
+INTERFACE void __csan_ignore_thread_end() {
+ if (__csan::DisableCount)
+ --__csan::DisableCount;
+}
+
+static int AccessFlags(int Flags, bool IsWrite) {
+ return Flags | (I...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/225782
More information about the llvm-branch-commits
mailing list