[PATCH] D78950: Adds LRU caching of compile units in DWARFContext.
Hyoun Kyu Cho via Phabricator via llvm-commits
llvm-commits at lists.llvm.org
Mon Apr 27 11:49:27 PDT 2020
netforce created this revision.
netforce added a reviewer: dblaikie.
netforce added a project: debug-info.
Herald added subscribers: hiraditya, aprantl.
Herald added a project: LLVM.
This adds LRU caching of compile units in DWARFContext to reduce memory consumption of llvm-symbolizer. When llvm-symbolize symbolizes addresses from various compile units, it keeps the internal data structure (e.g. DIE and line table) for all relevant compile units until it finishes running. This leads to memory bloat when symbolizing many addresses. The memory usage can be limited by only keeping a fixed number of compile units in the memory.
This might make it longer to run because sometimes it has to parse the same compile unit as it's kicked out of memory. For better performance, LRU caching is used to keep the recently used compile units in the memory while kicking out the ones not used recently.
Confirmed this reduces the memory usage significantly (1.3GB -> 441MB when symbolizing a clang binary).
$ nm clang | sort -n | gawk '{printf "0x%s\n", $1}' | /usr/bin/time -f "RSS: %M KB\nexecution_time: %E\n" ./llvm-symbolizer-before -a -demangle -obj clang > /dev/null
RSS: 1314996 KB
execution_time: 0:04.11
$ nm clang | sort -n | gawk '{printf "0x%s\n", $1}' | /usr/bin/time -f "RSS: %M KB\nexecution_time: %E\n" ./llvm-symbolizer-after -a -demangle -obj clang > /dev/null
RSS: 441300 KB
execution_time: 0:08.15
Using Valgrind/Massif we could also confirm it limits the memory usage growth while it keeps growing without the change.
$ nm clang | sort -n | gawk '{printf "0x%s\n", $1}' | /usr/bin/valgrind --tool=massif --massif-out-file=massif.out.before ./llvm-symbolizer-before -a -demangle -obj clang > /dev/null
$ ms_print massif.out.before
`--------------------------------------------------------------------------------
Command: ./llvm-symbolizer-before -a -demangle -obj clang
Massif arguments: --massif-out-file=massif.out.before
ms_print arguments: massif.out.before
-------------------------------------
GB
1.208^ :
| @@@#::
| @@@@@@#::
| @@@@@@@@@#::
| @@@@@@@@@@@@@#::
| ::@@ @@@@@@@@@@@#::
| ::::::@@ @@@@@@@@@@@#::
| @@@::: ::@@ @@@@@@@@@@@#::
| @@@@@@::: ::@@ @@@@@@@@@@@#::
| @@@@@@ @@@::: ::@@ @@@@@@@@@@@#::
| @@@@@ @@@ @@@::: ::@@ @@@@@@@@@@@#::
| ::@:@@@@@ @@@ @@@::: ::@@ @@@@@@@@@@@#::
| ::: @:@@@@@ @@@ @@@::: ::@@ @@@@@@@@@@@#::
| :@:::::: @:@@@@@ @@@ @@@::: ::@@ @@@@@@@@@@@#::
| @@@::@:: ::: @:@@@@@ @@@ @@@::: ::@@ @@@@@@@@@@@#::
| @@@ @::@:: ::: @:@@@@@ @@@ @@@::: ::@@ @@@@@@@@@@@#::
| @@:@@@ @::@:: ::: @:@@@@@ @@@ @@@::: ::@@ @@@@@@@@@@@#::
| @@@@@@@ @@@ @::@:: ::: @:@@@@@ @@@ @@@::: ::@@ @@@@@@@@@@@#::
| ::@@@ @@ @@ @@@ @::@:: ::: @:@@@@@ @@@ @@@::: ::@@ @@@@@@@@@@@#::
| @@@: @@@ @@ @@ @@@ @::@:: ::: @:@@@@@ @@@ @@@::: ::@@ @@@@@@@@@@@#::
0 +----------------------------------------------------------------------->Gi
0 19.40
...`
$ nm clang | sort -n | gawk '{printf "0x%s\n", $1}' | /usr/bin/valgrind --tool=massif --massif-out-file=massif.out.after ./llvm-symbolizer-after -a -demangle -obj clang > /dev/null
$ ms_print massif.out.after
`--------------------------------------------------------------------------------
Command: ./llvm-symbolizer-after -a -demangle -obj clang
Massif arguments: --massif-out-file=massif.out.after
ms_print arguments: massif.out.after
------------------------------------
MB
97.32^ #
| #
| #
| # : ::
| # : :
| # : :
| # : :
| # : @@: : :
| # : @ : : : :
| :# : @ : : : :
| : :# : : @ : : : : @ :
| : :# @@ ::: :: @ : ::: : @:::
| ::::# @ :: :: @ : ::::: : ::@: : :
| : : :: :#@@ :@ :::: : : :::::@ :: :::: ::: : : ::@: ::: : :
| ::::::::::: :#@ ::::@ : :: ::: : :::@ ::@::: : ::: :::::::::@: :::@::::
| :: ::::: :: :#@ :: :@ : :: :::@: :::@ ::@::: : ::: :::::: ::@: :::@::::
| :: ::::: :: :#@ :: :@ : :: :::@: :::@ ::@::: : ::: :::::: ::@: :::@::::
| :: ::::: :: :#@ :: :@ : :: :::@: :::@ ::@::: : ::: :::::: ::@: :::@::::
| :: ::::: :: :#@ :: :@ : :: :::@: :::@ ::@::: : ::: :::::: ::@: :::@::::
| :: ::::: :: :#@ :: :@ : :: :::@: :::@ ::@::: : ::: :::::: ::@: :::@::::
0 +----------------------------------------------------------------------->Gi
0 49.33
...`
Repository:
rG LLVM Github Monorepo
https://reviews.llvm.org/D78950
Files:
llvm/include/llvm/DebugInfo/DWARF/DWARFContext.h
llvm/include/llvm/DebugInfo/DWARF/DWARFDebugLine.h
llvm/lib/DebugInfo/DWARF/DWARFContext.cpp
llvm/lib/DebugInfo/DWARF/DWARFDebugLine.cpp
llvm/lib/DebugInfo/DWARF/DWARFUnit.cpp
-------------- next part --------------
A non-text attachment was scrubbed...
Name: D78950.260379.patch
Type: text/x-patch
Size: 9554 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20200427/2012e9a5/attachment.bin>
More information about the llvm-commits
mailing list