[cfe-dev] libclang: Memory management

[Jusufadis Bakamovic via cfe-dev]

Yes, I think there's not much that can be done from libclang POV. This is
all memory allocator stuff and should be handled from client code if
required but nevertheless interesting observation we should be aware of
when building tools.

Thanks everybody for the fruitful discussion and details provided.

Cheers,
Adi

On 10 March 2017 at 17:56, mats petersson <mats at planetcatfish.com> wrote:

> Like I said (and David points out), the rules for when something is
> reused, and when it isn't, depends A LOT on how that memory was allocated
> (and the details of the allocator implementation itself). If we assume that
> you are using a normal `malloc` (or the `new` operator), such as it is
> normally done in Linux (e.g. "glibc"), then I happen to know that an
> allocation that is considered "large", (from memory, large means anything
> more than 128KB, but could be some other number), the fresh memory required
> for the allocation is not done through "sbrk" (growing the heap), but from
> `mmap` (using a null file). This "large" allocation is also released
> "immediately".
>
> I bet that if you redo that code to for example use a few million small
> allocations, it will not grow the original size (but again, it all depends
> on what things are in the free-list, and how that matches what you are
> currently allocating.
>
> The way that compilers and parsers work, you quite often create small-ish
> lists of things, such as argument lists, statements in blocks, all the
> functions, etc, so there will be lots of small and intermediate size
> allocations - and some big ones in some places.
>
> So I think we can conclude that there isn't any BIG leaks at least... :)
>
> --
> Mats
>
> On 10 March 2017 at 14:16, Jusufadis Bakamovic <jbakam at gmail.com> wrote:
>
>> On 10 March 2017 at 13:28, David Chisnall <David.Chisnall at cl.cam.ac.uk>
>> wrote:
>>
>>> On 10 Mar 2017, at 12:19, Jusufadis Bakamovic via cfe-dev <
>>> cfe-dev at lists.llvm.org> wrote:
>>> >
>>> > If it proves right, then allocating additional memory (like in your
>>> example) _after_ we finish with the libclang acitivites (disposal), memory
>>> consumption should _not_ increase. Expected is that the memory owned by the
>>> process will be reused
>>>
>>> That doesn’t necessarily hold.  A typical modern memory allocator will
>>> have different pools for different-sized allocations and will provide a
>>> hint to the OS about whether the physical memory backing virtual regions
>>> can be reclaimed.  If you allocate with a different size to the ones that
>>> libclang has used, then you may end up with new zones being allocated from
>>> the OS for the memory allocator to reuse and if memory pressure is low then
>>> the OS may not recycle the physical pages that are backing free’d
>>> allocations (and if there is a lot of fragmentation, then you may have a
>>> very small number of allocations holding a lot of pages together).
>>>
>>> David
>>>
>>
>> Alright, I am really not familiar with all of the implementation details
>> of memory allocators but I was aware that it is not that straightforward as
>> my words probably were telling.
>>
>> Anyhow, with this in mind what both David and Mats said, I have made two
>> different experiments:
>>
>> 1. Do the libclang stuff (create index, parse multiple files, cleanup)
>> and then allocate 100MB on heap
>> 2. Do the libclang stuff (create index, parse multiple files, cleanup)
>> two times in a row
>>
>>
>> Results of first experiment were:
>>
>> Before libclang cleanup:    VIRT: 365M RES: 227M SHR: 75M
>> After libclang cleanup:       VIRT: 277M RES: 166M SHR: 63M
>> Allocated another 100MB: VIRT: 377M RES: 265M SHR: 63M
>> Deallocated 100MB:          VIRT: 277M RES: 166M SHR: 63M
>>
>>
>> Results of second experiment were:
>>
>> Before 1st libclang cleanup:    VIRT: 365M RES: 227M SHR: 75M
>> After 1st libclang cleanup:       VIRT: 277M RES: 166M SHR: 63M
>> Before 2nd libclang cleanup:   VIRT: 354M RES: 228M SHR: 76M
>>
>> After 2nd libclang cleanup:      VIRT: 277M RES: 173M SHR: 63M
>>
>>
>> Additional allocations were done as:
>>
>> char *p = new char[100*1024*1024]; // 100MB
>> for (int idx = 0; idx < 100*1024*1024; idx+=4096) p[idx] = idx; // touch
>> pages to force them go into the resident memory
>>
>>
>> So it looks like my assumptions were proved wrong (probably due to what
>> David said). What I can see as a conclusion is that physical (resident)
>> memory which is still owned by the process will be able to get (partially)
>> recycled if we utilize it in a similar way we did in previous iterations
>> (i.e. second experiment). Otherwise, physical (resident) memory consumption
>> will increase (i.e. first experiment).
>>
>> Cheers,
>> Adi
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20170310/a53376a1/attachment.html>