[cfe-dev] libclang: Memory management

[Jusufadis Bakamovic via cfe-dev]

Reproducing the issue is fairly easy. If you like I could provide a
smallish demo which monitors the (system) memory consumption at various
steps of execution?

Better question is if there is anything we can do to improve this? I
believe this is a highly important aspect of the library.

On 9 March 2017 at 17:53, Manuel Klimek <klimek at google.com> wrote:

> Benjamin had tried to come up with a repro at some point, too, iirc
>
> On Thu, Mar 9, 2017 at 4:48 PM Jusufadis Bakamovic via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
>> Hi,
>>
>> I am a little bit puzzled about the memory management in libclang. What I
>> observed is, when run on mid-sized code base, a very high (understandable)
>> memory consumption whereas it seems it is not possible to reclaim all the
>> consumed memory back once we are finished with whatever we have been doing
>> with libclang. For example, this is a code excerpt which should explain my
>> words:
>>
>> // 1. Create an index
>> vector<CXTranslationUnit> tunits;
>> CXIndex idx = clang_createIndex(1, 1);
>>
>> // 2. Parse each file found in directory and store corresponding TU into
>> the vector
>> for (auto file& : directory) {
>>     CXTranslationUnit tu;
>>     if (!clang_parseTranslationUnit2(idx, file.path().c_str(), cmd_args,
>> cmd_args_len, 0, 0, CXTranslationUnit_DetailedPreprocessingRecord, &tu)
>>         tunits.push_back(tu);
>> }
>>
>> // 3. Cleanup
>> for (auto tu& : tunits) {
>>     clang_disposeTranslationUnit(tu);
>> }
>> clang_disposeIndex(idx);
>>
>>
>> If I run this code on `cppcheck` code base (https://github.com/danmar/
>> cppcheck) which is somewhat a mid-sized project (it counts cca 300
>> C/C++ files), I get the following figures in terms of memory consumption
>> for that particular process (based on `htop` output):
>>
>> * app memory consumption after 2nd step: virt(5763M) / res(5533M) /
>> shr(380M)
>> * app memory consumption after 3rd step:  virt(4423M) / res(4288M) /
>> shr(65188)
>>
>>
>> So, as it can be seen from the figures, even after the cleanup stage,
>> both virtual and resident memory figures are still very high. Seems like
>> the only part which has been reclaimed is the memory that has been
>> associated with the TU's. All the other, I can guess, parsing artifacts are
>> still being hold somewhere in the memory to which we don't have access to
>> neither we can flush them using the libclang API. I even ran the code with
>> valgrind and there were no memory leaks detected (only `still reachable`
>> blocks).
>>
>> Either I am missing something here or this might impose a memory issues
>> for long-running non-single-shot tools (i.e. think indexer).
>>
>> Can anyone comment on this issue?
>>
>> Thanks,
>> Adi
>>
>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20170310/bdc9af1b/attachment.html>