[compiler-rt] [ASan] Honor `allocator_may_return_null` when set through user-function and fix large alloc edge case (PR #117929)

[via llvm-commits]

================
@@ -164,7 +164,24 @@ void UnmapOrDie(void *addr, uptr size, bool raw_report) {
 static void *ReturnNullptrOnOOMOrDie(uptr size, const char *mem_type,
                                      const char *mmap_type) {
   error_t last_error = GetLastError();
-  if (last_error == ERROR_NOT_ENOUGH_MEMORY)
+
+  // Assumption: VirtualAlloc is the last system call that was invoked before
+  //   this method.
+  // VirtualAlloc emits one of 2 error codes when running out of memory
+  // 1. ERROR_NOT_ENOUGH_MEMORY:
+  //  There's not enough memory to execute the command
+  // 2. ERROR_INVALID_PARAMETER:
+  //  VirtualAlloc will return this if the request would allocate memory at an
+  //  address exceeding or being very close to the maximum application address
+  //  (the `lpMaximumApplicationAddress` field within the `SystemInfo` struct).
+  //  This does not seem to be officially documented, but is corroborated here:
+  //  https://stackoverflow.com/questions/45833674/why-does-virtualalloc-fail-for-lpaddress-greater-than-0x6ffffffffff
+
+  // Note - It's possible that 'ERROR_COMMITMENT_LIMIT' needs to be handled here
+  // as well. It is currently not handled due to the lack of a reproducer that
+  // induces the error code.
----------------
zmodem wrote:

That looks reasonable to me. Want to send it again as a PR? I think we could merge that without a test case.

https://github.com/llvm/llvm-project/pull/117929