[compiler-rt] [win/asan] Don't intercept memset etc. in ntdll (PR #120397)

[Reid Kleckner via llvm-commits]

rnk wrote:

First, this is just venting, I think everyone who works on Windows already knows this and accepts this, but: The Windows C runtime ecosystem is such a nightmare. ⚡🧠⚡ 

At a really high level, I think the best direction here is to de-escalate the hotpatching wars.

Regarding `msvcrt.dll` ASan compatibility, I cannot remember any attempt to intentionally support WinASan usage in this configuration, so I'm somewhat inclined to say that it was an accident that it ever worked at all. Contributions to support it would be welcome, but we should keep this change in to prioritize the users (Chromium & Firefox) who benefit from this change.

If we want to intentionally support msvcrt.dll, there is prior art for relaxing the CHECK failure in `malloc_usable_size`. See these checks for OWNED_BY_RTL:
https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/asan/asan_malloc_win.cpp#L204

Adding checks for heap ownership on the hot malloc/free codepaths are prohibitively expensive, so we want to minimize those, but these malloc_usable_size APIs are usually much colder and its probably acceptable to add a wide variety of defensive checks there in the name of compatibility and ease of adoption.

My assumption is that calls to malloc from /instrumented DSOs/ are going to be very hot, while malloc calls from precompiled DLLs, system or third party, are going to be much colder. If we could find a way to separate the malloc entry points between instrumented and non-instrumented code, that could allow us to make the non-instrumented memory allocator entry points more flexible, meaning we have fewer CHECK failures and more recovery paths.

This would all be worth doing if it makes future Windows updates smoother.

https://github.com/llvm/llvm-project/pull/120397