[PATCH] D77330: Consider increasing the default ARM32 page size to 64k.

[Ryan Prichard via Phabricator via llvm-commits]

rprichard added a subscriber: victoryang.
rprichard added a comment.

In D77330#1961215 <https://reviews.llvm.org/D77330#1961215>, @srhines wrote:

> So it turns out that the ld shipped in the NDK actually did have the 64KB max-page-size patch, so not taking this patch would change current user behavior. Thanks to @MaskRay for sharing the exact binutils patch we were looking for to verify that it has it. For that reason, this patch should be fine for our 32-bit ARM NDK users. For the platform, we'll see if we should just set the max-page-size back to 4KB explicitly in the build rules, but that shouldn't affect this CL.


It's a bit hairier than this. The NDK ships ld.bfd, ld.gold, and ld.lld. The plain "ld" for arm32 is ld.gold, and "ld" for arm64 is ld.bfd. (AFAIK, that inconsistency has existed at least since NDK r10e. We're hoping to switch "ld" to ld.lld soon-ish for all architectures.)

Before the platform switched to lld, I think it was using ld.gold on all architectures.

- For arm32, the NDK ld.gold(default), defaults to a max-page-size of 0x1000, but ld.bfd defaults to 0x10000 instead. (Even upstream arm32 gold looks like it uses 0x1000 still.)
- For arm64, it looks like the NDK ld.bfd(default) has used a max-page-size of 0x10000 since at least r10e, but ld.gold switched from 0x1000 to 0x10000 between r15 and r16.

For arm32, I think a 64KiB max-page-size (on an OS where the pages are actually 4KiB) would create vaddr gaps between each object's segments. It looks like the Bionic loader maps a gap as inaccessible memory, but the kernel loader leaves the gap unmapped, so something else could be mapped into an executable's gap. If another DSO were mapped into that area, perhaps it could break something? (e.g. arm32 C++ exception unwinding can break <https://gist.github.com/rprichard/04c522bc811956d05b4cf458243fb3d9>. That was fixed <https://android-review.googlesource.com/c/platform/bionic/+/916470/> for Q+, but old platform versions are still unfixed.) Newer devices seem to keep non-fixed mmap VMAs away from the executable's VMAs, though, which avoids the issue.

I think each DSO gap adds another VMA, and I think this change could add up to 3 of them per DSO. (IIRC, VMA usage is a problem because the bookkeeping uses expensive non-pageable kernel memory.) This issue is easy to fix, though, by building the arm32 platform with `-z,max-page-size=0x1000`. (I checked a 32-bit device's system_server. I saw 318 DSOs, and it looked like a 64KiB max-page-size could add up to 192KiB of vaddr padding per DSO, for a total increase of up to 60MiB of vaddr usage.)


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D77330/new/

https://reviews.llvm.org/D77330