[LLVMdev] Porting ASAN to new platform

[Alexander Potapenko]

On Wed, Feb 19, 2014 at 4:41 AM, Ryan Govostes <rzg at apple.com> wrote:
> + llvmdev
>
> On Feb 18, 2014, at 2:55 AM, Alexander Potapenko <glider at google.com> wrote:
>
>> Hi Ryan,
>>
>> (CCing Kostya and Evgeniy)
>>
>> On Feb 15, 2014 10:49 AM, "Ryan Govostes" <rzg at apple.com> wrote:
>>>
>>> Hi Alexander,
>>>
>>> I spent some time trying to get ASAN running on ARM64, but I’m not sure what the necessary changes are. (Note I’m on r199351 which is a little old. Just trying to get the basic steps down.)
>>>
>>>
>>> - I defined a Mapping.Offset in llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp’s getShadowMapping() —— but I don’t know what the value should be
>> Mapping.Offset must be the same as SHADOW_OFFSET in asan_mapping.h.
>> This is the address at which the shadow memory range starts (the
>> shadow occupies 1/8 of address space, 16Tb by default on 64-bit
>> systems). I'd try to attach vmmap (or gdb, or whatever) to an existing
>> non-ASan process (if iOS allows that) and look at the memory layout on
>> ARM64.
>
> I still have ASAN_FLEXIBLE_MAPPING_AND_OFFSET defined so I did not need to specify SHADOW_OFFSET. This was removed somewhat recently on trunk, I think. I’ll try to update.
Yes, please. There've been some changes regarding AArch64 that you may
find useful.
>>
>>> - In compiler-rt/lib/asan/asan_mapping.h, I defined kLowMemBeg to 0x100000000ULL which I understand is the correct lower bound on 64-bit iOS.
>>>
>>> - In compiler-rt/lib/sanitizer_common/sanitizer_posix.cc, I made GetMaxVirtualAddress() return (0x1a0000000ULL - 1) which I understand is the correct upper bound on 64-bit iOS.
>> So you've only 2 gigabytes (0x100000000 to 0x1a0000000) of memory
>> available to the user? That's not much, and ASan is gonna take 256Mb
>> of that space for shadow, plus some for the allocator.
>
> I believe that is accurate.
Sadly, this makes the usage of ASan on 64 bit iOS quite limited.
>>
>>> - In compiler-rt/lib/asan/asan_mac.cc, I redefined GetPcSpBp to return the ARM pc, sp, and lr registers.
>>>
>>> - I made numerous hacks to the build scripts to basically build the OS X libclang_rt.asan_osx_dynamic.dylib for iOS instead. Maybe there are ARM-specific build settings used for the existing Android support that I should also use?
>> You should take a look at how libclang_rt.asan_iossim_dynamic.dylib is
>> built, I think that's closer to your needs. There might be some issues
>> related to Xcode sysroots which I've resolved when porting to iossim.
>> Actually, the support for iOS is now limited to iossim, and I haven't
>> even tried to build ASan for iOS, because we didn't need to yet.
>> That's why I haven't looked into the possible issues on iOS yet. My
>> biggest concern is that the function interceptors (based on dyld
>> interposing) may not work correctly, because they require re-execing
>> the process with DYLD_INSERT_LIBRARIES. Do you know if that's at all
>> possible?
>
> I believe iOS does support DYLD_INSERT_LIBRARIES in some cases, such as for debugging with libgmalloc.
Good.
>>>
>>>
>>> Beyond setting [ memory low, shadow offset, memory high ], what other platform-specific changes might I need to make? I expect maybe to have to write some new interceptors, but so far I’m stuck on just initializing ASAN.
>> Nothing comes to mind, let us try to figure out what's wrong with startup first.
>>>
>>> Right now, with the shadow offset as 0x110000000ULL, dyld dies by jumping into shadow memory while trying to resolve pthread_create. This doesn’t make too much sense to me, as the dyld implementations are effectively the same on OS X and iOS.
>> The implementations can be the same, but IIUC the address space
>> available to the user is much smaller on iOS, so we may be hitting
>> some addresses used by dyld.
>> There's a check in __asan_init() for existing Mach-O sections mapping
>> into the shadow range, but if dyld allocates some memory in the shadow
>> space using mmap or malloc we won't notice that.
>> Or perhaps someone is loading code after __asan_init() has mapped the shadow?
>> Is it possible to step over the program using a debugger?
>
> Looking at vmmap of a non-ASAN process as you suggest, it does look like /usr/lib/dyld is loaded at ~0x120000000 which is inside the shadow map I chose. I’ll look for a bigger unused gap.
>
> It looks like I might also need to set kAllocatorSpace and kAllocatorSize in asan_allocator2.cc, as the offset and size are well outside the bounds of what I have available. Any guidance on choosing appropriate values?
You need to use the 32-bit allocator instead of the 64-one, that's the
default since r201303.
> Thanks for your help.
>
> Ryan



-- 
Alexander Potapenko
Software Engineer
Google Moscow