[compiler-rt] [ASAN][sanitizers][win] Allow windows-asan to be built with /MDd and intercept functions from the debug runtimes. (PR #77840)

[Charlie Barto via llvm-commits]

https://github.com/barcharcraz created https://github.com/llvm/llvm-project/pull/77840

It turns out this works _mostly_ fine, even when mixing debug versions of asan with programs built with the release runtime. Using /MT (or /MTd) with a dynamically linked asan has never really worked that well, and I am planning on opening a PR that will completely remove the static-asan configuration for windows and make programs linked with the static CRT/runtime work with the DLL version of asan. This is better than the current situation because the static linked version of asan doesn't work well on windows if there are multiple DLLs in the process using it.

The check for building asan with only /MD or /MT has been removed. It was in AsanDoesNotSupportStaticLinkage, but was checking for debug CRTs, not static linkage. The kind of static linkage this function is supposed to check for (on linux for example) doesn't really exist on windows.

Note: There is one outstanding issue with this approach, if you mix a /MDd DLLs and /MD dlls in the same process then the "real" function called by asan interceptors will be the same for calls from both contexts, potentially screwing up things like errno. This only happens if you mix /MD and /MDd in the same process, because otherwise asan won't find functions from both runtimes to intercept. We are working on a fix for this, and it mainly hits with the CRT functions exported from both ucrtbase and ntdll.


This change is being upstreamed from our fork.

>From 71a300510b9cbfff9904015697fb4a872cdbbdee Mon Sep 17 00:00:00 2001
From: Charlie Barto <chbarto at microsoft.com>
Date: Wed, 10 Jan 2024 16:59:26 -0800
Subject: [PATCH 1/2] Remove the restriction on building asan with the debug
 runtime (build changes not included)

---
 compiler-rt/lib/asan/asan_win.cpp | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/compiler-rt/lib/asan/asan_win.cpp b/compiler-rt/lib/asan/asan_win.cpp
index d5a30f471e2b0d..2a62f2546a9ead 100644
--- a/compiler-rt/lib/asan/asan_win.cpp
+++ b/compiler-rt/lib/asan/asan_win.cpp
@@ -265,9 +265,6 @@ void PlatformTSDDtor(void *tsd) { AsanThread::TSDDtor(tsd); }
 
 // ---------------------- Various stuff ---------------- {{{
 void *AsanDoesNotSupportStaticLinkage() {
-#if defined(_DEBUG)
-#error Please build the runtime with a non-debug CRT: /MD or /MT
-#endif
   return 0;
 }
 

>From 74ec76c287be5911856497c4e5c15306d40ea3d3 Mon Sep 17 00:00:00 2001
From: Charlie Barto <chbarto at microsoft.com>
Date: Thu, 11 Jan 2024 10:15:43 -0800
Subject: [PATCH 2/2] Hook into both debug and release DLLs

---
 compiler-rt/lib/interception/interception_win.cpp | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/compiler-rt/lib/interception/interception_win.cpp b/compiler-rt/lib/interception/interception_win.cpp
index f1a549b938e202..9ae21170580ba4 100644
--- a/compiler-rt/lib/interception/interception_win.cpp
+++ b/compiler-rt/lib/interception/interception_win.cpp
@@ -946,6 +946,11 @@ bool OverrideFunction(
 static void **InterestingDLLsAvailable() {
   static const char *InterestingDLLs[] = {
       "kernel32.dll",
+      "msvcr100d.dll",                // VS2010
+      "msvcr110d.dll",                // VS2012
+      "msvcr120d.dll",                // VS2013
+      "vcruntime140d.dll",            // VS2015
+      "ucrtbased.dll",  // Universal CRT
       "msvcr100.dll",      // VS2010
       "msvcr110.dll",      // VS2012
       "msvcr120.dll",      // VS2013