[all-commits] [llvm/llvm-project] 91dc54: Set Huge Page mode on shadow regions based on no_h...

[Z via All-commits]

  Branch: refs/heads/master
  Home:   https://github.com/llvm/llvm-project
  Commit: 91dc545bf24daa60c21c93039408061194dd0ab3
      https://github.com/llvm/llvm-project/commit/91dc545bf24daa60c21c93039408061194dd0ab3
  Author: Jianzhou Zhao <jianzhouzh at google.com>
  Date:   2020-10-20 (Tue, 20 Oct 2020)

  Changed paths:
    M compiler-rt/lib/dfsan/dfsan.cpp
    M compiler-rt/test/dfsan/release_shadow_space.c

  Log Message:
  -----------
  Set Huge Page mode on shadow regions based on no_huge_pages_for_shadow

It turned out that at dynamic shared library mode, the memory access
pattern can increase memory footprint significantly on OS when transparent
hugepages (THP) are enabled. This could cause >70x memory overhead than
running a static linked binary. For example, a static binary with RSS
overhead 300M can use > 23G RSS if it is built dynamically.
/proc/../smaps shows in 6204552 kB RSS 6141952 kB relates to
AnonHugePages.

Also such a high RSS happens in some rate: around 25% runs may use > 23G RSS, the
rest uses in between 6-23G. I guess this may relate to how user memory
is allocated and distributted across huge pages.

THP is a trade-off between time and space. We have a flag
no_huge_pages_for_shadow for sanitizer. It is true by default but DFSan
did not follow this. Depending on if a target is built statically or
dynamically, maybe Clang can set no_huge_pages_for_shadow accordingly
after this change. But it still seems fine to follow the default setting of
no_huge_pages_for_shadow. If time is an issue, and users are fine with
high RSS, this flag can be set to false selectively.