[cfe-dev] libclang: Memory management

[Jusufadis Bakamovic via cfe-dev]

Hi,

I am a little bit puzzled about the memory management in libclang. What I
observed is, when run on mid-sized code base, a very high (understandable)
memory consumption whereas it seems it is not possible to reclaim all the
consumed memory back once we are finished with whatever we have been doing
with libclang. For example, this is a code excerpt which should explain my
words:

// 1. Create an index
vector<CXTranslationUnit> tunits;
CXIndex idx = clang_createIndex(1, 1);

// 2. Parse each file found in directory and store corresponding TU into
the vector
for (auto file& : directory) {
    CXTranslationUnit tu;
    if (!clang_parseTranslationUnit2(idx, file.path().c_str(), cmd_args,
cmd_args_len, 0, 0, CXTranslationUnit_DetailedPreprocessingRecord, &tu)
        tunits.push_back(tu);
}

// 3. Cleanup
for (auto tu& : tunits) {
    clang_disposeTranslationUnit(tu);
}
clang_disposeIndex(idx);


If I run this code on `cppcheck` code base (
https://github.com/danmar/cppcheck) which is somewhat a mid-sized project
(it counts cca 300 C/C++ files), I get the following figures in terms of
memory consumption for that particular process (based on `htop` output):

* app memory consumption after 2nd step: virt(5763M) / res(5533M) /
shr(380M)
* app memory consumption after 3rd step:  virt(4423M) / res(4288M) /
shr(65188)


So, as it can be seen from the figures, even after the cleanup stage, both
virtual and resident memory figures are still very high. Seems like the
only part which has been reclaimed is the memory that has been associated
with the TU's. All the other, I can guess, parsing artifacts are still
being hold somewhere in the memory to which we don't have access to neither
we can flush them using the libclang API. I even ran the code with valgrind
and there were no memory leaks detected (only `still reachable` blocks).

Either I am missing something here or this might impose a memory issues for
long-running non-single-shot tools (i.e. think indexer).

Can anyone comment on this issue?

Thanks,
Adi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20170309/d07b5a34/attachment.html>