[cfe-dev] libclang: Memory management

[mats petersson via cfe-dev]

Is this not a general case of "heaps (sometimes/often) don't shrink once
they have grown"? In other words, once the OS or runtime has used a large
amount of memory, unless there are other components in the system to "push
out" those memory allocations to swap, the memory now freed is still owned
by the process that originally allocated it. It's a bit more complex as
well, since both Linux and Windows have multiple ways of allocating memory,
with choices made both by the programmer (picking different APIs) and the
runtime/OS itself. The memory that has been released is (assuming there's
no real leak) free to be used for other purposes.

The reason for this behaviour is that applications quite often build up
large sets of heap allocations, release said allocations, and then allocate
large amounts again.

You can try this out by doing something like:

int main()
{
     int *p = new int[10000000];
     ... fill p with data to actually use the memory ...
     delete [] p;
     cout << "Memory freed, now sleeping..." << std::endl;
     sleep(20);
     return 0;
}

If when it's freed memory and sleeping, the memory usag is still some
400MB, you know that the heap is not immediately released to the OS.

--
Mats

On 10 March 2017 at 10:21, Jusufadis Bakamovic via cfe-dev <
cfe-dev at lists.llvm.org> wrote:

> Reproducing the issue is fairly easy. If you like I could provide a
> smallish demo which monitors the (system) memory consumption at various
> steps of execution?
>
> Better question is if there is anything we can do to improve this? I
> believe this is a highly important aspect of the library.
>
> On 9 March 2017 at 17:53, Manuel Klimek <klimek at google.com> wrote:
>
>> Benjamin had tried to come up with a repro at some point, too, iirc
>>
>> On Thu, Mar 9, 2017 at 4:48 PM Jusufadis Bakamovic via cfe-dev <
>> cfe-dev at lists.llvm.org> wrote:
>>
>>> Hi,
>>>
>>> I am a little bit puzzled about the memory management in libclang. What
>>> I observed is, when run on mid-sized code base, a very high
>>> (understandable) memory consumption whereas it seems it is not possible to
>>> reclaim all the consumed memory back once we are finished with whatever we
>>> have been doing with libclang. For example, this is a code excerpt which
>>> should explain my words:
>>>
>>> // 1. Create an index
>>> vector<CXTranslationUnit> tunits;
>>> CXIndex idx = clang_createIndex(1, 1);
>>>
>>> // 2. Parse each file found in directory and store corresponding TU into
>>> the vector
>>> for (auto file& : directory) {
>>>     CXTranslationUnit tu;
>>>     if (!clang_parseTranslationUnit2(idx, file.path().c_str(),
>>> cmd_args, cmd_args_len, 0, 0, CXTranslationUnit_DetailedPreprocessingRecord,
>>> &tu)
>>>         tunits.push_back(tu);
>>> }
>>>
>>> // 3. Cleanup
>>> for (auto tu& : tunits) {
>>>     clang_disposeTranslationUnit(tu);
>>> }
>>> clang_disposeIndex(idx);
>>>
>>>
>>> If I run this code on `cppcheck` code base (
>>> https://github.com/danmar/cppcheck) which is somewhat a mid-sized
>>> project (it counts cca 300 C/C++ files), I get the following figures in
>>> terms of memory consumption for that particular process (based on `htop`
>>> output):
>>>
>>> * app memory consumption after 2nd step: virt(5763M) / res(5533M) /
>>> shr(380M)
>>> * app memory consumption after 3rd step:  virt(4423M) / res(4288M) /
>>> shr(65188)
>>>
>>>
>>> So, as it can be seen from the figures, even after the cleanup stage,
>>> both virtual and resident memory figures are still very high. Seems like
>>> the only part which has been reclaimed is the memory that has been
>>> associated with the TU's. All the other, I can guess, parsing artifacts are
>>> still being hold somewhere in the memory to which we don't have access to
>>> neither we can flush them using the libclang API. I even ran the code with
>>> valgrind and there were no memory leaks detected (only `still reachable`
>>> blocks).
>>>
>>> Either I am missing something here or this might impose a memory issues
>>> for long-running non-single-shot tools (i.e. think indexer).
>>>
>>> Can anyone comment on this issue?
>>>
>>> Thanks,
>>> Adi
>>>
>>>
>>> _______________________________________________
>>> cfe-dev mailing list
>>> cfe-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>
>>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20170310/0afef08e/attachment.html>