[all-commits] [llvm/llvm-project] d19470: [sanitizer] Add a ForEach callback interface for A...

Thu Nov 11 11:31:44 PST 2021

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: d19470540a0769313d219295f245975af5589edb
      https://github.com/llvm/llvm-project/commit/d19470540a0769313d219295f245975af5589edb
  Author: Snehasish Kumar <snehasishk at google.com>
  Date:   2021-11-11 (Thu, 11 Nov 2021)

  Changed paths:
    M compiler-rt/lib/sanitizer_common/sanitizer_addrhashmap.h
    M compiler-rt/lib/sanitizer_common/tests/CMakeLists.txt
    A compiler-rt/lib/sanitizer_common/tests/sanitizer_addrhashmap_test.cpp

  Log Message:
  -----------
  [sanitizer] Add a ForEach callback interface for AddrHashMap.

This change adds a ForEach method to the AddrHashMap class which can
then be used to iterate over all the key value pairs in the hash map.
I intend to use this in an upcoming change to the memprof runtime.

Added a unit test to cover basic insertion and the ForEach callback.

Differential Revision: https://reviews.llvm.org/D111368

  Commit: fc7162414ede116a453ddb402a55f67d8e50c31b
      https://github.com/llvm/llvm-project/commit/fc7162414ede116a453ddb402a55f67d8e50c31b
  Author: Snehasish Kumar <snehasishk at google.com>
  Date:   2021-11-11 (Thu, 11 Nov 2021)

  Changed paths:
    M compiler-rt/lib/memprof/CMakeLists.txt
    M compiler-rt/lib/memprof/memprof_allocator.cpp
    A compiler-rt/lib/memprof/memprof_meminfoblock.h

  Log Message:
  -----------
  [memprof] Move the MemInfoBlock definition to a separate header.

Move the memprof MemInfoBlock struct to it's own header as requested
during the review of D111676.

Differential Revision: https://reviews.llvm.org/D113315

  Commit: 1243cef245f6131c093d65235e87319e8e124e7f
      https://github.com/llvm/llvm-project/commit/1243cef245f6131c093d65235e87319e8e124e7f
  Author: Snehasish Kumar <snehasishk at google.com>
  Date:   2021-11-11 (Thu, 11 Nov 2021)

  Changed paths:
    M compiler-rt/lib/memprof/CMakeLists.txt
    M compiler-rt/lib/memprof/memprof_allocator.cpp
    M compiler-rt/lib/memprof/memprof_flags.inc
    A compiler-rt/lib/memprof/memprof_mibmap.cpp
    A compiler-rt/lib/memprof/memprof_mibmap.h
    R compiler-rt/test/memprof/TestCases/mem_info_cache_entries.cpp
    A compiler-rt/test/memprof/TestCases/memprof_merge_mib.cpp
    R compiler-rt/test/memprof/TestCases/print_miss_rate.cpp

  Log Message:
  -----------
  [memprof] Replace the block cache with a hashmap.

The existing implementation uses a cache + eviction based scheme to
record heap profile information. This design was adopted to ensure a
constant memory overhead (due to fixed number of cache entries) along
with incremental write-to-disk for evictions. We find that since the
number to entries to track is O(unique-allocation-contexts) the overhead
of keeping all contexts in memory is not very high. On a clang workload,
the max number of unique allocation contexts was ~35K, median ~11K.
For each context, we (currently) store 64 bytes of data - this amounts
to 5.5MB (max). Given the low overheads for a complex workload, we can
simplify the implementation by using a hashmap without eviction.

Other changes:
* Memory map is dumped at the end rather than startup. The relative
order in the profile dump is unchanged since we no longer have evicted
entries at runtime.
* Added a test to check meminfoblocks are merged.

Differential Revision: https://reviews.llvm.org/D111676

  Commit: 545866cb05b90329fc61f361c6afffedb3aaf938
      https://github.com/llvm/llvm-project/commit/545866cb05b90329fc61f361c6afffedb3aaf938
  Author: Snehasish Kumar <snehasishk at google.com>
  Date:   2021-11-11 (Thu, 11 Nov 2021)

  Changed paths:
    M compiler-rt/lib/memprof/CMakeLists.txt
    M compiler-rt/lib/memprof/memprof_allocator.cpp
    M compiler-rt/lib/memprof/memprof_flags.inc
    A compiler-rt/lib/memprof/memprof_rawprofile.cpp
    A compiler-rt/lib/memprof/memprof_rawprofile.h
    A compiler-rt/lib/memprof/tests/CMakeLists.txt
    A compiler-rt/lib/memprof/tests/driver.cpp
    A compiler-rt/lib/memprof/tests/rawprofile.cpp
    M compiler-rt/lib/sanitizer_common/sanitizer_procmaps.h
    M compiler-rt/test/memprof/TestCases/memprof_profile_dump.cpp

  Log Message:
  -----------
  [memprof] Add a raw binary format to serialize memprof profiles.

This change implements the raw binary format discussed in
https://lists.llvm.org/pipermail/llvm-dev/2021-September/153007.html

Summary of changes
* Add a new memprof option to choose binary or text (default) format.
* Add a rawprofile library which serializes the MIB map to profile.
* Add a unit test for rawprofile.
* Mark sanitizer procmaps methods as virtual to be able to mock them.
* Extend memprof_profile_dump regression test.

Differential Revision: https://reviews.llvm.org/D113317

  Commit: aacaebc6c22f81b13598aac69dee4b076f669e3e
      https://github.com/llvm/llvm-project/commit/aacaebc6c22f81b13598aac69dee4b076f669e3e
  Author: Snehasish Kumar <snehasishk at google.com>
  Date:   2021-11-11 (Thu, 11 Nov 2021)

  Changed paths:
    M compiler-rt/lib/memprof/memprof_flags.inc
    M compiler-rt/test/memprof/TestCases/atexit_stats.cpp
    M compiler-rt/test/memprof/TestCases/dump_process_map.cpp
    M compiler-rt/test/memprof/TestCases/log_path_test.cpp
    M compiler-rt/test/memprof/TestCases/malloc-size-too-big.cpp
    M compiler-rt/test/memprof/TestCases/memprof_merge_mib.cpp
    M compiler-rt/test/memprof/TestCases/memprof_profile_dump.cpp
    M compiler-rt/test/memprof/TestCases/stress_dtls.c
    M compiler-rt/test/memprof/TestCases/test_malloc_load_store.c
    M compiler-rt/test/memprof/TestCases/test_memintrin.cpp
    M compiler-rt/test/memprof/TestCases/test_new_load_store.cpp
    M compiler-rt/test/memprof/TestCases/test_terse.cpp
    M compiler-rt/test/memprof/TestCases/unaligned_loads_and_stores.cpp
    M compiler-rt/test/memprof/lit.cfg.py

  Log Message:
  -----------
  [memprof] Make the raw binary format the default.

Set the default memprof serialization format as binary. 9 tests are
updated to use print_text=true. Also fixed an issue with concatenation
of default and test specified options (missing separator).

Differential Revision: https://reviews.llvm.org/D113617

Compare: https://github.com/llvm/llvm-project/compare/986416251b35...aacaebc6c22f