[cfe-dev] Question about Clang/LLVM addresssanitizer

Kostya Serebryany via cfe-dev cfe-dev at lists.llvm.org
Tue Sep 20 22:05:14 PDT 2016 

On Tue, Sep 20, 2016 at 10:00 PM, ZhaoKang <zhaokang at mail.tsinghua.edu.cn>
wrote:

> Hello Kostya,
>
> I run the same binary on different VMs, one is redhat5.9 and the other one
> is redhat 6.5.
>
> And dump the outputs to two log files (attachments) .
>
> I'm sorry for our VM cannot be accessed by other groups.
>

Please check if you can reproduce this on any of the OS types available on
GCE.
https://cloud.google.com/compute/docs/images
If/when you verify that it reproduces on one of them, I'll try to debug it
there.


> Please see the attached files, one is failed and another is right.
>

I need the results of a run under strace.


> Many thanks!
>
>
> -----原始邮件-----
> *发件人:* "Kostya Serebryany" <kcc at google.com>
> *发送时间:* 2016-09-21 11:41:07 (星期三)
> *收件人:* ZhaoKang <zhaokang at mail.tsinghua.edu.cn>
> *抄送:* "Clang Dev" <cfe-dev at lists.llvm.org>
> *主题:* Re: Re: Re: Re: [cfe-dev] Question about Clang/LLVM addresssanitizer
>
>
>
>
> On Tue, Sep 20, 2016 at 6:47 PM, ZhaoKang <zhaokang at mail.tsinghua.edu.cn>
> wrote:
>
>> You mentioned that 'Some global settings in the system may for some
>> reason reject such huge mappings.'
>> And I want to know which kinds of settings may have this effect?
>>
>
> Something related to memory overcommit.
>
>
>> And did any other users encounter this kind of issue before?
>>
>
> Not on Linux.
>
>
>> And the output error message said that '==41651==ERROR: AddressSanitizer
>> failed to allocate 0xdfff0001000 (15392894357504) bytes at address
>> 2008fff7000 (errno: 12)'
>> Does this feature need allocate so large memory?
>>
>
> Yes.
> You may read https://github.com/google/sanitizers/wiki/
> AddressSanitizerAlgorithm for details.
>
> My suggestion: build a single binary with asan and run it under strace on
> both systems that you have,
> then compare the output.
>
> Can you give me access to a machine where this fails?
> Is this OS variant available on GCE (https://cloud.google.com/compute/)?
>
>
>> or this feature dump the error message?
>>
>> Thanks a lot!
>>
>>
>> -----原始邮件-----
>> *发件人:* "Kostya Serebryany" <kcc at google.com>
>> *发送时间:* 2016-09-21 00:35:55 (星期三)
>> *收件人:* ZhaoKang <zhaokang at mail.tsinghua.edu.cn>
>> *抄送:* "Clang Dev" <cfe-dev at lists.llvm.org>
>> *主题:* Re: Re: Re: [cfe-dev] Question about Clang/LLVM addresssanitizer
>>
>>
>>
>>
>> On Mon, Sep 19, 2016 at 9:41 PM, ZhaoKang <zhaokang at mail.tsinghua.edu.cn>
>> wrote:
>>
>>> Yes, if just a small case like following also failed:
>>>
>>> int main() {
>>>
>>>     int *mem = new int[100];
>>>
>>>     for (unsigned i = 0; i<=100; i++ ) {
>>>
>>>         mem[i] = i;
>>>
>>>     }
>>>
>>> return 0;
>>>
>>> }
>>>
>>>
>>>
>>> And I build like this:
>>>
>>> fangqing@[xcoapps57 small_case]$ clang++  -fsanitize=address
>>> -fsanitize=undefined main.cpp
>>>
>>> fangqing@[xcoapps57 small_case]$ ll -h a.out
>>>
>>> -rwxr-x--- 1 fangqing hd 9.6M Sep 19 21:24 a.out
>>>
>>
>> 9.6M is a bit unusual, on my box I get a 2.4M binary,
>> but not entirely bad. asan links lots of stuff into a binary.
>>
>>
>>>
>>>
>>> fangqing@[xcoapps57 small_case]$ ASAN_OPTIONS=verbosity=1 ./a.out
>>>
>>> ==41651==AddressSanitizer: failed to intercept '__isoc99_printf'
>>>
>>> ==41651==AddressSanitizer: failed to intercept '__isoc99_sprintf'
>>>
>>> ==41651==AddressSanitizer: failed to intercept '__isoc99_snprintf'
>>>
>>> ==41651==AddressSanitizer: failed to intercept '__isoc99_fprintf'
>>>
>>> ==41651==AddressSanitizer: failed to intercept '__isoc99_vprintf'
>>>
>>> ==41651==AddressSanitizer: failed to intercept '__isoc99_vsprintf'
>>>
>>> ==41651==AddressSanitizer: failed to intercept '__isoc99_vsnprintf'
>>>
>>> ==41651==AddressSanitizer: failed to intercept '__isoc99_vfprintf'
>>>
>>> ==41651==AddressSanitizer: failed to intercept 'process_vm_readv'
>>>
>>> ==41651==AddressSanitizer: failed to intercept 'process_vm_writev'
>>>
>>> ==41651==AddressSanitizer: libc interceptors initialized
>>>
>>> || `[0x10007fff8000, 0x7fffffffffff]` || HighMem    ||
>>>
>>> || `[0x02008fff7000, 0x10007fff7fff]` || HighShadow ||
>>>
>>> || `[0x005000000000, 0x02008fff6fff]` || ShadowGap3 ||
>>>
>>> || `[0x003000000000, 0x004fffffffff]` || MidMem     ||
>>>
>>> || `[0x000a7fff8000, 0x002fffffffff]` || ShadowGap2 ||
>>>
>>> || `[0x00067fff8000, 0x000a7fff7fff]` || MidShadow  ||
>>>
>>> || `[0x00008fff7000, 0x00067fff7fff]` || ShadowGap  ||
>>>
>>> || `[0x00007fff8000, 0x00008fff6fff]` || LowShadow  ||
>>>
>>> || `[0x000000000000, 0x00007fff7fff]` || LowMem     ||
>>>
>>> MemToShadow(shadow): 0x00008fff7000 0x000091ff6dff 0x004091ff6e00
>>> 0x02008fff6fff 0x00014fff7000 0x0001cfff6fff
>>>
>>> redzone=16
>>>
>>> max_redzone=2048
>>>
>>> quarantine_size_mb=256M
>>>
>>> malloc_context_size=30
>>>
>>> SHADOW_SCALE: 3
>>>
>>> SHADOW_GRANULARITY: 8
>>>
>>> SHADOW_OFFSET: 0x7fff8000
>>>
>>> ==41651==ERROR: AddressSanitizer failed to allocate 0xdfff0001000
>>> (15392894357504) bytes at address 2008fff7000 (errno: 12)
>>>
>>> ==41651==ReserveShadowMemoryRange failed while trying to map
>>> 0xdfff0001000 bytes. Perhaps you're using ulimit -v
>>>
>>> Aborted (core dumped)
>>>
>>>
>>>
>>> If compile this small case without address-sanitizer, the generated
>>> executable file is just 6.8K, and if with this feature, the executable
>>> file is 9.6M.
>>>
>>> fangqing@[xcoapps57 small_case]$ clang++  main.cpp
>>>
>>> fangqing@[xcoapps57 small_case]$ ll -h a.out
>>>
>>> -rwxr-x--- 1 fangqing hd 6.8K Sep 19 21:28 a.out
>>>
>>>
>>>
>>> *And the most important is that if we move this case to another Linux
>>> server, the address-sanitizer feature works fine.*
>>>
>>> Now the failed case is built on following RedHat server:
>>>
>>> fangqing@[xcoapps57 small_case]$ uname -a
>>>
>>> Linux xcoapps57 2.6.32-504.el6.x86_64 #1 SMP Tue Sep 16 01:56:35 EDT
>>> 2014 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> fangqing@[xcoapps57 small_case]$ cat /etc/redhat-release
>>>
>>> Red Hat Enterprise Linux Workstation release 6.6 (Santiago)
>>>
>>>
>>>
>>> So is it related to different platform? And what does this feature
>>> depend on?
>>>
>>
>> asan depends on the ability to mmap a huge range of address space
>> with MAP_NORESERVE.
>> Some global settings in the system may for some reason reject such huge
>> mappings.
>> But I have not seen this before.
>>
>> --kcc
>>
>>
>>>
>>>
>>> Thanks a lot!
>>>
>>>
>>> ----------
>>> *From:* "Kostya Serebryany" <kcc at google.com>
>>> *Time:* 2016-09-20 10:36:16 (星期二)
>>> *To:* ZhaoKang <zhaokang at mail.tsinghua.edu.cn>
>>> *cc:* "Clang Dev" <cfe-dev at lists.llvm.org>
>>> *Title:* Re: Re: [cfe-dev] Question about Clang/LLVM addresssanitizer
>>>
>>>
>>> Does this happen with any small application on your system, or only with
>>> this (presumably big) one?
>>> You will need to figure out why the asan's mmap fails here.
>>> Best is to run the process under strace and see
>>>   * are there any mmaps that intersect with this one
>>>   * are there any syscalls that limit the address space (setrlimit)
>>>
>>> --kcc
>>>
>>>
>>> On Mon, Sep 19, 2016 at 7:18 PM, ZhaoKang <zhaokang at mail.tsinghua.edu.cn
>>> > wrote:
>>>
>>>> Dear Kostya,
>>>>
>>>> Thanks a lot for your reply!
>>>> For the first question, we have dump out the info, please see the
>>>> following:
>>>> =============================================
>>>>
>>>> ASAN_OPTIONS=verbosity=1 ./csim.exe
>>>>
>>>> ==25048==AddressSanitizer: failed to intercept '__isoc99_printf'
>>>>
>>>> ==25048==AddressSanitizer: failed to intercept '__isoc99_sprintf'
>>>>
>>>> ==25048==AddressSanitizer: failed to intercept '__isoc99_snprintf'
>>>>
>>>> ==25048==AddressSanitizer: failed to intercept '__isoc99_fprintf'
>>>>
>>>> ==25048==AddressSanitizer: failed to intercept '__isoc99_vprintf'
>>>>
>>>> ==25048==AddressSanitizer: failed to intercept '__isoc99_vsprintf'
>>>>
>>>> ==25048==AddressSanitizer: failed to intercept '__isoc99_vsnprintf'
>>>>
>>>> ==25048==AddressSanitizer: failed to intercept '__isoc99_vfprintf'
>>>>
>>>> ==25048==AddressSanitizer: failed to intercept 'process_vm_readv'
>>>>
>>>> ==25048==AddressSanitizer: failed to intercept 'process_vm_writev'
>>>>
>>>> ==25048==AddressSanitizer: libc interceptors initialized
>>>>
>>>> || `[0x10007fff8000, 0x7fffffffffff]` || HighMem    ||
>>>>
>>>> || `[0x02008fff7000, 0x10007fff7fff]` || HighShadow ||
>>>>
>>>> || `[0x005000000000, 0x02008fff6fff]` || ShadowGap3 ||
>>>>
>>>> || `[0x003000000000, 0x004fffffffff]` || MidMem     ||
>>>>
>>>> || `[0x000a7fff8000, 0x002fffffffff]` || ShadowGap2 ||
>>>>
>>>> || `[0x00067fff8000, 0x000a7fff7fff]` || MidShadow  ||
>>>>
>>>> || `[0x00008fff7000, 0x00067fff7fff]` || ShadowGap  ||
>>>>
>>>> || `[0x00007fff8000, 0x00008fff6fff]` || LowShadow  ||
>>>>
>>>> || `[0x000000000000, 0x00007fff7fff]` || LowMem     ||
>>>>
>>>> MemToShadow(shadow): 0x00008fff7000 0x000091ff6dff 0x004091ff6e00
>>>> 0x02008fff6fff 0x00014fff7000 0x0001cfff6fff
>>>>
>>>> redzone=16
>>>>
>>>> max_redzone=2048
>>>>
>>>> quarantine_size_mb=256M
>>>>
>>>> malloc_context_size=30
>>>>
>>>> SHADOW_SCALE: 3
>>>>
>>>> SHADOW_GRANULARITY: 8
>>>>
>>>> SHADOW_OFFSET: 0x7fff8000
>>>>
>>>> ==25048==ERROR: AddressSanitizer failed to allocate 0xdfff0001000
>>>> (15392894357504) bytes at address 2008fff7000 (errno: 12)
>>>>
>>>> ==25048==ReserveShadowMemoryRange failed while trying to map
>>>> 0xdfff0001000 bytes. Perhaps you're using ulimit -v
>>>>
>>>> Aborted (core dumped)
>>>>
>>>>  =============================================
>>>>
>>>> Thanks!
>>>>
>>>> Kang
>>>>
>>>> ----------
>>>> *From:* "Kostya Serebryany" <kcc at google.com>
>>>> *Time:* 2016-09-09 05:50:03
>>>> *Time:* ZhaoKang <zhaokang at mail.tsinghua.edu.cn>
>>>> *CC:* "Clang Dev" <cfe-dev at lists.llvm.org>
>>>> *Subject:* Re: [cfe-dev] Question about Clang/LLVM addresssanitizer
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Sep 8, 2016 at 1:09 AM, ZhaoKang via cfe-dev <
>>>> cfe-dev at lists.llvm.org> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>>
>>>>>
>>>>> We have one question about the clang compiler option:
>>>>> -fsanitize=address. (We want to use the feature to detect potential bug in
>>>>> out c++ design.)
>>>>>
>>>>> However, when using clang to compile our two cases with this option,
>>>>> one case error out with following message:
>>>>>
>>>>>
>>>>>
>>>>> ==31183==ERROR: AddressSanitizer *failed to allocate 0x400000000*
>>>>> (17179869184) bytes at address 67fff8000 (errno: 12)
>>>>>
>>>>> ==31183==ReserveShadowMemoryRange failed while trying to map
>>>>> 0x400000000 bytes. Perhaps you're using ulimit –v
>>>>>
>>>>
>>>> This happens at startup right?
>>>> something causes asan to fail to allocate the shadow.
>>>>
>>>> please send the output of
>>>>    ASAN_OPTIONS=verbosity=1 ./your-binary
>>>>
>>>> or, better, send the reproducer.
>>>>
>>>>
>>>>>
>>>>>
>>>>> The other case error out with following message:
>>>>>
>>>>> ==30711==ERROR: AddressSanitizer: *stack-buffer-overflow* on address
>>>>> 0x7fff8a931dcd at pc 0x000000861eec bp 0x7fff8a9303f0 sp 0x7fff8a9303e8
>>>>>
>>>>
>>>> that smells like a real bug in your code.
>>>>
>>>>
>>>>> READ of size 1 at 0x7fff8a931dcd thread T0
>>>>>
>>>>>     #0 0x861eeb in ap_private<8, false, true>::RType<32, true>::mult
>>>>> ap_private<8, false, true>::operator*<32, true>(ap_private<32, true,
>>>>> (32)<=(64)> const&) const (/wrk/xbj_vdi/fangqing/work/sp
>>>>> rite/hls/BugSpray/crs/810730/hscale/solution1/csim/build/csi
>>>>> m.exe+0x861eeb)
>>>>>
>>>>>     #1 0x7d717f in ap_private<8, false, (8)<=(64)>::RType<32,
>>>>> true>::mult operator*<8, false>(ap_private<8, false, (8)<=(64)> const&,
>>>>> int) (/wrk/xbj_vdi/fangqing/work/sprite/hls/BugSpray/crs/810730/h
>>>>> scale/solution1/csim/build/csim.exe+0x7d717f)
>>>>>
>>>>>     …
>>>>>
>>>>>     #6 0x3099a1d9c3 in __libc_start_main (/lib64/libc.so.6+0x3099a1d9c3
>>>>> )
>>>>>
>>>>>     #7 0x4c3648 in _start (/wrk/xbj_vdi/fangqing/work/sp
>>>>> rite/hls/BugSpray/crs/810730/hscale/solution1/csim/build/csi
>>>>> m.exe+0x4c3648)
>>>>>
>>>>>
>>>>>
>>>>> Address 0x7fff8a931dcd is located in stack of thread T0 at offset 1549
>>>>> in frame
>>>>>
>>>>>     #0 0x78628f in hscale_core(hls::stream<HSC_MPIX_STRUCT>&,
>>>>> ap_uint<4>, ap_uint<16>, ap_uint<16>, ap_uint<16>, ap_uint<32>, ap_uint<2>,
>>>>> ap_int<16> (*) [8], hls::stream<HSC_MPIX_STRUCT>&)
>>>>> (/wrk/xbj_vdi/fangqing/work/sprite/hls/BugSpray/crs/810730/h
>>>>> scale/solution1/csim/build/csim.exe+0x78628f)
>>>>>
>>>>>
>>>>>
>>>>>   This frame has 215 object(s):
>>>>>
>>>>> [32, 33) 'RegSmplsPerClk'
>>>>>
>>>>> [48, 49) 'RegBitsPerCol'
>>>>>
>>>>> [64, 66) 'TotalLines'
>>>>>
>>>>> …
>>>>>
>>>>> [1552, 1553) 'ref.tmp65' <== Memory access at offset 1549 underflows
>>>>> this variable
>>>>>
>>>>> [1568, 1569) 'ref.tmp68'
>>>>>
>>>>> …
>>>>>
>>>>>
>>>>>
>>>>> Both of them can be compiled successfully and run correctly when
>>>>> compiled with clang without this addrsanitizer option. However both of them
>>>>> failed when add this option.
>>>>>
>>>>> From the error message we can see AddressSanitizer need to allocate a
>>>>> very large memory (about 16G byte) from heap memory pool (1st case),
>>>>> or occupy large stack memory and cause stack-buffer-overflow (2nd
>>>>> case). (ulimit –v shows unlimited)
>>>>>
>>>>> So our question is it is this feature’s shortcoming or there is
>>>>> something wrong with our development environment?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> cfe-dev mailing list
>>>>> cfe-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20160920/bffb9694/attachment.html>

More information about the cfe-dev mailing list