<div dir="ltr"><div>Mats, I think it actually makes sense what you're saying and it might be very well that we are experiencing exactly this case. It shouldn't be too hard to check it out though. If it proves right, then allocating additional memory (like in your example) _after_ we finish with the libclang acitivites (disposal), memory consumption should _not_ increase. Expected is that the memory owned by the process will be reused. I will make an experiment and come back with the results.</div><div><br></div><div>Cheers,</div><div>Adi</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 10 March 2017 at 11:46, mats petersson <span dir="ltr"><<a href="mailto:mats@planetcatfish.com" target="_blank">mats@planetcatfish.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div><div><div>Is this not a general case of "heaps (sometimes/often) don't shrink once they have grown"? In other words, once the OS or runtime has used a large amount of memory, unless there are other components in the system to "push out" those memory allocations to swap, the memory now freed is still owned by the process that originally allocated it. It's a bit more complex as well, since both Linux and Windows have multiple ways of allocating memory, with choices made both by the programmer (picking different APIs) and the runtime/OS itself. The memory that has been released is (assuming there's no real leak) free to be used for other purposes.<br><br></div>The reason for this behaviour is that applications quite often build up large sets of heap allocations, release said allocations, and then allocate large amounts again.<br><br></div>You can try this out by doing something like:<br><br></div>int main()<br>{<br></div> int *p = new int[10000000];<br></div> ... fill p with data to actually use the memory ... <br></div> delete [] p;<br></div><div> cout << "Memory freed, now sleeping..." << std::endl;<br></div> sleep(20);<br></div> return 0;<br>}<br><br></div>If when it's freed memory and sleeping, the memory usag is still some 400MB, you know that the heap is not immediately released to the OS.<br><br>--<br></div>Mats<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On 10 March 2017 at 10:21, Jusufadis Bakamovic via cfe-dev <span dir="ltr"><<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Reproducing the issue is fairly easy. If you like I could provide a smallish demo which monitors the (system) memory consumption at various steps of execution?<div><br></div><div>Better question is if there is anything we can do to improve this? I believe this is a highly important aspect of the library.</div></div><div class="m_-4322472898859046701HOEnZb"><div class="m_-4322472898859046701h5"><div class="gmail_extra"><br><div class="gmail_quote">On 9 March 2017 at 17:53, Manuel Klimek <span dir="ltr"><<a href="mailto:klimek@google.com" target="_blank">klimek@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Benjamin had tried to come up with a repro at some point, too, iirc<br><br><div class="gmail_quote"><div><div class="m_-4322472898859046701m_1885391402009476353h5"><div dir="ltr">On Thu, Mar 9, 2017 at 4:48 PM Jusufadis Bakamovic via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_-4322472898859046701m_1885391402009476353h5"><div dir="ltr" class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">Hi,<div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"><br class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"></div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">I am a little bit puzzled about the memory management in libclang. What I observed is, when run on mid-sized code base, a very high (understandable) memory consumption whereas it seems it is not possible to reclaim all the consumed memory back once we are finished with whatever we have been doing with libclang. For example, this is a code excerpt which should explain my words:</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"><br class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">// 1. Create an index</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">vector<CXTranslationUnit> tunits;</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">CXIndex idx = clang_createIndex(1, 1);</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"><br class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"></div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">// 2. Parse each file found in directory and store corresponding TU into the vector</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">for (auto file& : directory) {</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"> CXTranslationUnit tu;</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"> if (!clang_parseTranslationUnit2(<wbr>idx, file.path().c_str(), cmd_args, cmd_args_len, 0, 0, CXTranslationUnit_DetailedPrep<wbr>rocessingRecord, &tu)</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"> tunits.push_back(tu);</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">}</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"><br class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"></div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">// 3. Cleanup</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">for (auto tu& : tunits) {</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"> clang_disposeTranslationUnit(t<wbr>u);</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">}</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">clang_disposeIndex(idx);</div></blockquote><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"><br class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"></div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">If I run this code on `cppcheck` code base (<a href="https://github.com/danmar/cppcheck" class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg" target="_blank">https://github.com/danmar/cpp<wbr>check</a>) which is somewhat a mid-sized project (it counts cca 300 C/C++ files), I get the following figures in terms of memory consumption for that particular process (based on `htop` output):</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"><br class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px" class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">* app memory consumption after 2nd step: virt(5763M) / res(5533M) / shr(380M)</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">* app memory consumption after 3rd step: virt(4423M) / res(4288M) / shr(65188)</div></blockquote><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"><br class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"></div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">So, as it can be seen from the figures, even after the cleanup stage, both virtual and resident memory figures are still very high. Seems like the only part which has been reclaimed is the memory that has been associated with the TU's. All the other, I can guess, parsing artifacts are still being hold somewhere in the memory to which we don't have access to neither we can flush them using the libclang API. I even ran the code with valgrind and there were no memory leaks detected (only `still reachable` blocks).</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"><br class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"></div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">Either I am missing something here or this might impose a memory issues for long-running non-single-shot tools (i.e. think indexer).</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"><br class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"></div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">Can anyone comment on this issue?</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"><br class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"></div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">Thanks,</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">Adi</div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"><br class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"></div><div class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"><br class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg"></div></div></div></div></div>
______________________________<wbr>_________________<br class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">
cfe-dev mailing list<br class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">
<a href="mailto:cfe-dev@lists.llvm.org" class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg" target="_blank">cfe-dev@lists.llvm.org</a><br class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/cfe-dev</a><br class="m_-4322472898859046701m_1885391402009476353m_-964102771787330168gmail_msg">
</blockquote></div></div>
</blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/cfe-dev</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>