[cfe-dev] libclang: Memory management

[Jusufadis Bakamovic via cfe-dev]

Mats, I think it actually makes sense what you're saying and it might be
very well that we are experiencing exactly this case. It shouldn't be too
hard to check it out though. If it proves right, then allocating additional
memory (like in your example) _after_ we finish with the libclang
acitivites (disposal), memory consumption should _not_ increase. Expected
is that the memory owned by the process will be reused. I will make an
experiment and come back with the results.

Cheers,
Adi

On 10 March 2017 at 11:46, mats petersson <mats at planetcatfish.com> wrote:

> Is this not a general case of "heaps (sometimes/often) don't shrink once
> they have grown"? In other words, once the OS or runtime has used a large
> amount of memory, unless there are other components in the system to "push
> out" those memory allocations to swap, the memory now freed is still owned
> by the process that originally allocated it. It's a bit more complex as
> well, since both Linux and Windows have multiple ways of allocating memory,
> with choices made both by the programmer (picking different APIs) and the
> runtime/OS itself. The memory that has been released is (assuming there's
> no real leak) free to be used for other purposes.
>
> The reason for this behaviour is that applications quite often build up
> large sets of heap allocations, release said allocations, and then allocate
> large amounts again.
>
> You can try this out by doing something like:
>
> int main()
> {
>      int *p = new int[10000000];
>      ... fill p with data to actually use the memory ...
>      delete [] p;
>      cout << "Memory freed, now sleeping..." << std::endl;
>      sleep(20);
>      return 0;
> }
>
> If when it's freed memory and sleeping, the memory usag is still some
> 400MB, you know that the heap is not immediately released to the OS.
>
> --
> Mats
>
> On 10 March 2017 at 10:21, Jusufadis Bakamovic via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
>> Reproducing the issue is fairly easy. If you like I could provide a
>> smallish demo which monitors the (system) memory consumption at various
>> steps of execution?
>>
>> Better question is if there is anything we can do to improve this? I
>> believe this is a highly important aspect of the library.
>>
>> On 9 March 2017 at 17:53, Manuel Klimek <klimek at google.com> wrote:
>>
>>> Benjamin had tried to come up with a repro at some point, too, iirc
>>>
>>> On Thu, Mar 9, 2017 at 4:48 PM Jusufadis Bakamovic via cfe-dev <
>>> cfe-dev at lists.llvm.org> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am a little bit puzzled about the memory management in libclang. What
>>>> I observed is, when run on mid-sized code base, a very high
>>>> (understandable) memory consumption whereas it seems it is not possible to
>>>> reclaim all the consumed memory back once we are finished with whatever we
>>>> have been doing with libclang. For example, this is a code excerpt which
>>>> should explain my words:
>>>>
>>>> // 1. Create an index
>>>> vector<CXTranslationUnit> tunits;
>>>> CXIndex idx = clang_createIndex(1, 1);
>>>>
>>>> // 2. Parse each file found in directory and store corresponding TU
>>>> into the vector
>>>> for (auto file& : directory) {
>>>>     CXTranslationUnit tu;
>>>>     if (!clang_parseTranslationUnit2(idx, file.path().c_str(),
>>>> cmd_args, cmd_args_len, 0, 0, CXTranslationUnit_DetailedPreprocessingRecord,
>>>> &tu)
>>>>         tunits.push_back(tu);
>>>> }
>>>>
>>>> // 3. Cleanup
>>>> for (auto tu& : tunits) {
>>>>     clang_disposeTranslationUnit(tu);
>>>> }
>>>> clang_disposeIndex(idx);
>>>>
>>>>
>>>> If I run this code on `cppcheck` code base (
>>>> https://github.com/danmar/cppcheck) which is somewhat a mid-sized
>>>> project (it counts cca 300 C/C++ files), I get the following figures in
>>>> terms of memory consumption for that particular process (based on `htop`
>>>> output):
>>>>
>>>> * app memory consumption after 2nd step: virt(5763M) / res(5533M) /
>>>> shr(380M)
>>>> * app memory consumption after 3rd step:  virt(4423M) / res(4288M) /
>>>> shr(65188)
>>>>
>>>>
>>>> So, as it can be seen from the figures, even after the cleanup stage,
>>>> both virtual and resident memory figures are still very high. Seems like
>>>> the only part which has been reclaimed is the memory that has been
>>>> associated with the TU's. All the other, I can guess, parsing artifacts are
>>>> still being hold somewhere in the memory to which we don't have access to
>>>> neither we can flush them using the libclang API. I even ran the code with
>>>> valgrind and there were no memory leaks detected (only `still reachable`
>>>> blocks).
>>>>
>>>> Either I am missing something here or this might impose a memory issues
>>>> for long-running non-single-shot tools (i.e. think indexer).
>>>>
>>>> Can anyone comment on this issue?
>>>>
>>>> Thanks,
>>>> Adi
>>>>
>>>>
>>>> _______________________________________________
>>>> cfe-dev mailing list
>>>> cfe-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>
>>>
>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20170310/8b9438e8/attachment.html>