[PATCH] D41152: Use a custom container to reduce AFDO's memory usage by ~60%

[George Burgess IV via Phabricator via llvm-commits]

george.burgess.iv created this revision.
george.burgess.iv added reviewers: chandlerc, danielcdh.
Herald added a subscriber: mgorny.

This patch introduces a new associative container that's optimized for either being empty or having one element, and makes our sampling profile data structures use it.

As it turns out, AFDO profiles build lots of empty (or almost-empty) maps. :)

The results of three tests I ran with/without the patch:

- Merging 3.1GB of profile data (split across 226 profiles) consumed 10.2GB of RAM and took 42 seconds to terminate with this patch. Without, it took 26.3GB of RAM and took 70 seconds.
- Given a 32MB profile in `$foo`, running `echo | clang -O0 -x c - -fprofile-sample-use=$foo` took 250MB of RAM with this patch, and 650MB without. Clang itself ate 40MB without `-fprofile-sample-use` being passed.
- Given a 50MB profile collected by someone else, I observed a drop from 1.6GB of RAM consumption to 569MB with this patch.

Aside: I'm unsure of what the requirements are to land a new data structure in ADT. If this seems too tailored to AFDO's uses or too incomplete (e.g. it lacks `insert`, `remove`, and `emplace`; it doesn't support a custom less-than op, ...) on its own, I'm happy to move it elsewhere or add whatever's missing.


https://reviews.llvm.org/D41152

Files:
  include/llvm/ADT/TinyAssociativeVector.h
  include/llvm/ProfileData/SampleProf.h
  lib/ProfileData/SampleProf.cpp
  lib/ProfileData/SampleProfWriter.cpp
  lib/Transforms/IPO/SampleProfile.cpp
  test/tools/llvm-profdata/sample-profile-basic.test
  unittests/ADT/CMakeLists.txt
  unittests/ADT/TinyAssociativeVectorTest.cpp

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D41152.126668.patch
Type: text/x-patch
Size: 21548 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20171213/7aac7274/attachment.bin>