[compiler-rt] [asan] Use InternalMmapVector for error_message_buffer, reallocate wh… (PR #77488)

[via llvm-commits]

Enna1 wrote:

Hi @vitalybuka , thanks for looking into this.
I have reverted the previous change using InternalMmapVector, 
because it caused a deadlock when running the [asan/TestCases/Linux/rlimit_mmap_test.cpp](https://github.com/llvm/llvm-project/blob/main/compiler-rt/test/asan/TestCases/Linux/rlimit_mmap_test.cpp) test.
See https://buildkite.com/llvm-project/github-pull-requests/builds/27725.

This is because when asan reports an OOM, the PrintfAndReportCallback `AppendToErrorMessageBuffer()` is called.
As I use InternalMmapVector for error_message_buffer, `MmapOrDie()` is called for allocating buffer, but will fail because of OOM.
When `MmapOrDie()` fails, `ReportMmapFailureAndDie()` is called, then `Report()` is called, finally the PrintfAndReportCallback `AppendToErrorMessageBuffer()` is called again, causes a deadlock.

The following is the deadlock stack trace:
```
#0  0x0000560049fb9ae9 in __sanitizer::FutexWait (p=p at entry=0x56004a023180 <__asan::error_message_buf_mutex+16>, cmp=cmp at entry=0)
    at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp:724
#1  0x0000560049fba9f5 in __sanitizer::Semaphore::Wait (this=this at entry=0x56004a023180 <__asan::error_message_buf_mutex+16>)
    at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_mutex.cpp:35
#2  0x0000560049fa418c in __sanitizer::Mutex::Lock (this=0x56004a023170 <__asan::error_message_buf_mutex>)
    at llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_mutex.h:196
#3  __sanitizer::GenericScopedLock<__sanitizer::Mutex>::GenericScopedLock (mu=0x56004a023170 <__asan::error_message_buf_mutex>, this=<synthetic pointer>)
    at llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_mutex.h:383
#4  __asan::AppendToErrorMessageBuffer (
    buffer=0x7fff8eb03640 "==225129==ERROR: AddressSanitizer: out of memory: failed to allocate 0x10000 (65536) bytes of InternalMmapVector (error code: 12)\n")
    at llvm-project/compiler-rt/lib/asan/asan_report.cpp:41
#5  0x0000560049fbd294 in __sanitizer::CallPrintfAndReportCallback (
    str=0x7fff8eb03640 "==225129==ERROR: AddressSanitizer: out of memory: failed to allocate 0x10000 (65536) bytes of InternalMmapVector (error code: 12)\n")
    at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_printf.cpp:244
#6  __sanitizer::SharedPrintfCodeNoBuffer(bool, char *, int, const char *, typedef __va_list_tag __va_list_tag *) (append_pid=append_pid at entry=true,
    local_buffer=local_buffer at entry=0x7fff8eb03640 "==225129==ERROR: AddressSanitizer: out of memory: failed to allocate 0x10000 (65536) bytes of InternalMmapVector (error code: 12)\n",
    buffer_size=buffer_size at entry=400, format=format at entry=0x560049ff4200 "ERROR: %s: out of memory: failed to %s 0x%zx (%zd) bytes of %s (error code: %d)\n", args=args at entry=0x7fff8eb037e8)
    at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_printf.cpp:296
#7  0x0000560049fbd2d8 in __sanitizer::SharedPrintfCode(bool, const char *, typedef __va_list_tag __va_list_tag *) (append_pid=append_pid at entry=true,
    format=format at entry=0x560049ff4200 "ERROR: %s: out of memory: failed to %s 0x%zx (%zd) bytes of %s (error code: %d)\n", args=args at entry=0x7fff8eb037e8)
    at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_printf.cpp:309
#8  0x0000560049fbd410 in __sanitizer::Report (format=format at entry=0x560049ff4200 "ERROR: %s: out of memory: failed to %s 0x%zx (%zd) bytes of %s (error code: %d)\n")
    at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_printf.cpp:324
#9  0x0000560049fb3566 in __sanitizer::ReportMmapFailureAndDie (size=size at entry=65536, mem_type=mem_type at entry=0x560049fea341 "InternalMmapVector",
    mmap_type=mmap_type at entry=0x560049ff7db0 "allocate", err=12, raw_report=raw_report at entry=false)
    at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_common.cpp:50
#10 0x0000560049fbbb58 in __sanitizer::MmapOrDie (size=size at entry=65536, mem_type=mem_type at entry=0x560049fea341 "InternalMmapVector", raw_report=raw_report at entry=false)
    at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_posix.cpp:52
#11 0x0000560049f0905b in __sanitizer::InternalMmapVectorNoCtor<char>::Realloc (this=0x7f27f87f2e18, new_capacity=new_capacity at entry=65536)
    at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_common.h:599
#12 0x0000560049fa422a in __sanitizer::InternalMmapVectorNoCtor<char>::reserve (new_size=65536, this=<optimized out>)
    at llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common.h:558
#13 __asan::AppendToErrorMessageBuffer (buffer=0x7fff8eb03a40 '=' <repeats 65 times>, "\n") at llvm-project/compiler-rt/lib/asan/asan_report.cpp:45
#14 0x0000560049fbd294 in __sanitizer::CallPrintfAndReportCallback (str=0x7fff8eb03a40 '=' <repeats 65 times>, "\n")
    at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_printf.cpp:244
#15 __sanitizer::SharedPrintfCodeNoBuffer(bool, char *, int, const char *, typedef __va_list_tag __va_list_tag *) (append_pid=append_pid at entry=false,
    local_buffer=local_buffer at entry=0x7fff8eb03a40 '=' <repeats 65 times>, "\n", buffer_size=buffer_size at entry=400, format=format at entry=0x560049ff2628 '=' <repeats 65 times>, "\n",
    args=args at entry=0x7fff8eb03be8) at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_printf.cpp:296
#16 0x0000560049fbd2d8 in __sanitizer::SharedPrintfCode(bool, const char *, typedef __va_list_tag __va_list_tag *) (append_pid=append_pid at entry=false,
    format=format at entry=0x560049ff2628 '=' <repeats 65 times>, "\n", args=args at entry=0x7fff8eb03be8)
    at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_printf.cpp:309
#17 0x0000560049fbd374 in __sanitizer::Printf (format=format at entry=0x560049ff2628 '=' <repeats 65 times>, "\n")
    at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_printf.cpp:316
#18 0x0000560049fa6818 in __asan::ScopedInErrorReport::ScopedInErrorReport (fatal=true, this=0x7fff8eb0476e)
    at llvm-project/compiler-rt/lib/asan/asan_report.cpp:130
#19 __asan::ReportOutOfMemory (requested_size=requested_size at entry=10000000, stack=stack at entry=0x7fff8eb04870)
    at llvm-project/compiler-rt/lib/asan/asan_report.cpp:321
--Type <RET> for more, q to quit, c to continue without paging--
#20 0x0000560049f0196d in __asan::Allocator::Allocate (this=this at entry=0x56004a025d60 <__asan::instance>, size=<optimized out>, size at entry=10000000, alignment=alignment at entry=8,
    stack=stack at entry=0x7fff8eb04870, alloc_type=alloc_type at entry=__asan::FROM_MALLOC, can_fill=can_fill at entry=true)
    at llvm-project/compiler-rt/lib/asan/asan_allocator.cpp:592
#21 0x0000560049eff57f in __asan::asan_malloc (size=size at entry=10000000, stack=stack at entry=0x7fff8eb04870)
    at llvm-project/compiler-rt/lib/asan/asan_allocator.cpp:1000
#22 0x0000560049f9e009 in ___interceptor_malloc (size=10000000) at llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69
#23 0x0000560049fe8426 in main () at llvm-project/compiler-rt/test/asan/TestCases/Linux/rlimit_mmap_test.cpp:13
```
---

So in the lastest patch, I manually call `MmapOrDieQuietly()` and `UnmapOrDieQuietly()` to reallocate error_message_buffer.

If we can provide a variant InternalMmapVector, which calls `MmapOrDie()` with `raw_report=true` instead of  calling `MmapOrDie()` with default `raw_report=false`, the deadlock will be fixed. 
I think we can add `raw_report`  as a template parameter for InternalMmapVector ?


https://github.com/llvm/llvm-project/pull/77488