[cfe-dev] libclang: Memory management

[Jusufadis Bakamovic via cfe-dev]

On 10 March 2017 at 13:28, David Chisnall <David.Chisnall at cl.cam.ac.uk>
wrote:

> On 10 Mar 2017, at 12:19, Jusufadis Bakamovic via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
> >
> > If it proves right, then allocating additional memory (like in your
> example) _after_ we finish with the libclang acitivites (disposal), memory
> consumption should _not_ increase. Expected is that the memory owned by the
> process will be reused
>
> That doesn’t necessarily hold.  A typical modern memory allocator will
> have different pools for different-sized allocations and will provide a
> hint to the OS about whether the physical memory backing virtual regions
> can be reclaimed.  If you allocate with a different size to the ones that
> libclang has used, then you may end up with new zones being allocated from
> the OS for the memory allocator to reuse and if memory pressure is low then
> the OS may not recycle the physical pages that are backing free’d
> allocations (and if there is a lot of fragmentation, then you may have a
> very small number of allocations holding a lot of pages together).
>
> David
>

Alright, I am really not familiar with all of the implementation details of
memory allocators but I was aware that it is not that straightforward as my
words probably were telling.

Anyhow, with this in mind what both David and Mats said, I have made two
different experiments:

1. Do the libclang stuff (create index, parse multiple files, cleanup) and
then allocate 100MB on heap
2. Do the libclang stuff (create index, parse multiple files, cleanup) two
times in a row


Results of first experiment were:

Before libclang cleanup:    VIRT: 365M RES: 227M SHR: 75M
After libclang cleanup:       VIRT: 277M RES: 166M SHR: 63M
Allocated another 100MB: VIRT: 377M RES: 265M SHR: 63M
Deallocated 100MB:          VIRT: 277M RES: 166M SHR: 63M


Results of second experiment were:

Before 1st libclang cleanup:    VIRT: 365M RES: 227M SHR: 75M
After 1st libclang cleanup:       VIRT: 277M RES: 166M SHR: 63M
Before 2nd libclang cleanup:   VIRT: 354M RES: 228M SHR: 76M

After 2nd libclang cleanup:      VIRT: 277M RES: 173M SHR: 63M


Additional allocations were done as:

char *p = new char[100*1024*1024]; // 100MB
for (int idx = 0; idx < 100*1024*1024; idx+=4096) p[idx] = idx; // touch
pages to force them go into the resident memory


So it looks like my assumptions were proved wrong (probably due to what
David said). What I can see as a conclusion is that physical (resident)
memory which is still owned by the process will be able to get (partially)
recycled if we utilize it in a similar way we did in previous iterations
(i.e. second experiment). Otherwise, physical (resident) memory consumption
will increase (i.e. first experiment).

Cheers,
Adi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20170310/c58d5822/attachment.html>