<div dir="ltr">Yes, I think there's not much that can be done from libclang POV. This is all memory allocator stuff and should be handled from client code if required but nevertheless interesting observation we should be aware of when building tools.<div><br></div><div>Thanks everybody for the fruitful discussion and details provided.</div><div><br></div><div>Cheers,</div><div>Adi<br><div class="gmail_extra"><br><div class="gmail_quote">On 10 March 2017 at 17:56, mats petersson <span dir="ltr"><<a href="mailto:mats@planetcatfish.com" target="_blank">mats@planetcatfish.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>Like I said (and David points out), the rules for when something is reused, and when it isn't, depends A LOT on how that memory was allocated (and the details of the allocator implementation itself). If we assume that you are using a normal `malloc` (or the `new` operator), such as it is normally done in Linux (e.g. "glibc"), then I happen to know that an allocation that is considered "large", (from memory, large means anything more than 128KB, but could be some other number), the fresh memory required for the allocation is not done through "sbrk" (growing the heap), but from `mmap` (using a null file). This "large" allocation is also released "immediately". <br><br></div>I bet that if you redo that code to for example use a few million small allocations, it will not grow the original size (but again, it all depends on what things are in the free-list, and how that matches what you are currently allocating.<br><br></div>The way that compilers and parsers work, you quite often create small-ish lists of things, such as argument lists, statements in blocks, all the functions, etc, so there will be lots of small and intermediate size allocations - and some big ones in some places. <br><br></div>So I think we can conclude that there isn't any BIG leaks at least... :)<br><br>--<br></div>Mats<br> </div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On 10 March 2017 at 14:16, Jusufadis Bakamovic <span dir="ltr"><<a href="mailto:jbakam@gmail.com" target="_blank">jbakam@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div class="m_6482996037418888304h5"><div class="gmail_extra"><div class="gmail_quote">On 10 March 2017 at 13:28, David Chisnall <span dir="ltr"><<a href="mailto:David.Chisnall@cl.cam.ac.uk" target="_blank">David.Chisnall@cl.cam.ac.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="m_6482996037418888304m_6047996845311170006gmail-">On 10 Mar 2017, at 12:19, Jusufadis Bakamovic via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<br>
><br>
> If it proves right, then allocating additional memory (like in your example) _after_ we finish with the libclang acitivites (disposal), memory consumption should _not_ increase. Expected is that the memory owned by the process will be reused<br>
<br>
</span>That doesn’t necessarily hold. A typical modern memory allocator will have different pools for different-sized allocations and will provide a hint to the OS about whether the physical memory backing virtual regions can be reclaimed. If you allocate with a different size to the ones that libclang has used, then you may end up with new zones being allocated from the OS for the memory allocator to reuse and if memory pressure is low then the OS may not recycle the physical pages that are backing free’d allocations (and if there is a lot of fragmentation, then you may have a very small number of allocations holding a lot of pages together).<br>
<span class="m_6482996037418888304m_6047996845311170006gmail-HOEnZb"><font color="#888888"><br>
David<br>
</font></span></blockquote></div></div><div><br></div></div></div><div>Alright, I am really not familiar with all of the implementation details of memory allocators but I was aware that it is not that straightforward as my words probably were telling.</div><div><br></div><div>Anyhow, with this in mind what both David and Mats said, I have made two different experiments:</div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>1. Do the libclang stuff (create index, parse multiple files, cleanup) and then allocate 100MB on heap</div><div>2. Do the libclang stuff (create index, parse multiple files, cleanup) two times in a row</div></blockquote><div><br></div><div>Results of first experiment were:</div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>Before libclang cleanup: VIRT: 365M RES: 227M SHR: 75M</div><div>After libclang cleanup: VIRT: 277M RES: 166M SHR: 63M</div><div>Allocated another 100MB: VIRT: 377M RES: 265M SHR: 63M</div><div>Deallocated 100MB: VIRT: 277M RES: 166M SHR: 63M</div></blockquote></div><div><br></div><div>Results of second experiment were:</div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>Before 1st libclang cleanup: VIRT: 365M RES: 227M SHR: 75M</div>After 1st libclang cleanup: VIRT: 277M RES: 166M SHR: 63M<br>Before 2nd libclang cleanup: VIRT: 354M RES: 228M SHR: 76M</blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">After 2nd libclang cleanup: VIRT: 277M RES: 173M SHR: 63M<br></blockquote><br></div><div>Additional allocations were done as:</div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>char *p = new char[100*1024*1024]; // 100MB</div></div><div><div>for (int idx = 0; idx < 100*1024*1024; idx+=4096) p[idx] = idx; // touch pages to force them go into the resident memory</div></div></blockquote><div><br></div><div>So it looks like my assumptions were proved wrong (probably due to what David said). What I can see as a conclusion is that physical (resident) memory which is still owned by the process will be able to get (partially) recycled if we utilize it in a similar way we did in previous iterations (i.e. second experiment). Otherwise, physical (resident) memory consumption will increase (i.e. first experiment).</div><div class="gmail_extra"><div><br></div><div>Cheers,</div><div>Adi</div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div>