[llvm] [asan] Disable instrumentation for available_externally global with COFF (PR #81109)

[Wu Yingcong via llvm-commits]

https://github.com/yingcong-wu created https://github.com/llvm/llvm-project/pull/81109

For COFF, available_externally global will be instrumented because of the lack of filtering, and will trigger the Verifier pass assertion and crash the compilation. This patch will filter out the available_externally global for COFF.

For non-COFF, `!G->hasExactDefinition()` in line 1954 will filter out the available_externally globals.

There is a related bug reported in https://www.mail-archive.com/llvm-bugs@lists.llvm.org/msg46110.html. I tried the reproducer posted on the page and this will fix the problem.

>From 658d5f1e05627759fcd977ba7a561bb9e5f74577 Mon Sep 17 00:00:00 2001
From: "Wu, Yingcong" <yingcong.wu at intel.com>
Date: Thu, 8 Feb 2024 00:05:50 -0800
Subject: [PATCH 1/2] donot instrument available_externally globals

---
 .../Transforms/Instrumentation/AddressSanitizer.cpp    |  4 ++++
 .../do-not-instrument-globals-windows.ll               | 10 ++++++++++
 2 files changed, 14 insertions(+)
 create mode 100644 llvm/test/Instrumentation/AddressSanitizer/do-not-instrument-globals-windows.ll

diff --git a/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp b/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp
index d4f5bf8c393568..80643e5b048608 100644
--- a/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp
+++ b/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp
@@ -1957,6 +1957,10 @@ bool ModuleAddressSanitizer::shouldInstrumentGlobal(GlobalVariable *G) const {
     // On COFF, don't instrument non-ODR linkages.
     if (G->isInterposable())
       return false;
+    // If the global has AvailableExternally linkage, then it is not in this 
+    // module, which means it does not need to be instrumented.
+    if (G->hasAvailableExternallyLinkage())
+      return false;
   }
 
   // If a comdat is present, it must have a selection kind that implies ODR
diff --git a/llvm/test/Instrumentation/AddressSanitizer/do-not-instrument-globals-windows.ll b/llvm/test/Instrumentation/AddressSanitizer/do-not-instrument-globals-windows.ll
new file mode 100644
index 00000000000000..2ffa652f5fcdcb
--- /dev/null
+++ b/llvm/test/Instrumentation/AddressSanitizer/do-not-instrument-globals-windows.ll
@@ -0,0 +1,10 @@
+; This test checks that we are not instrumenting unnecessary globals
+; RUN: opt < %s -passes=asan -S | FileCheck %s
+
+target datalayout = "e-m:w-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-pc-windows-msvc"
+
+ at v_available_externally = available_externally global i32 zeroinitializer
+; CHECK-NOT: {{asan_gen.*v_available_externally}}
+
+; CHECK: @asan.module_ctor

>From d403f55d10561a430bd72b1ca63bac24df166dce Mon Sep 17 00:00:00 2001
From: "Wu, Yingcong" <yingcong.wu at intel.com>
Date: Thu, 8 Feb 2024 00:46:10 -0800
Subject: [PATCH 2/2] update datalayout

---
 .../AddressSanitizer/do-not-instrument-globals-windows.ll       | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/llvm/test/Instrumentation/AddressSanitizer/do-not-instrument-globals-windows.ll b/llvm/test/Instrumentation/AddressSanitizer/do-not-instrument-globals-windows.ll
index 2ffa652f5fcdcb..c143f69f126a85 100644
--- a/llvm/test/Instrumentation/AddressSanitizer/do-not-instrument-globals-windows.ll
+++ b/llvm/test/Instrumentation/AddressSanitizer/do-not-instrument-globals-windows.ll
@@ -1,7 +1,7 @@
 ; This test checks that we are not instrumenting unnecessary globals
 ; RUN: opt < %s -passes=asan -S | FileCheck %s
 
-target datalayout = "e-m:w-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-pc-windows-msvc"
 
 @v_available_externally = available_externally global i32 zeroinitializer