<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 10 March 2017 at 10:21, Jusufadis Bakamovic via cfe-dev <span dir="ltr"><<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Reproducing the issue is fairly easy. If you like I could provide a smallish demo which monitors the (system) memory consumption at various steps of execution?</div></blockquote><div><br></div><div>Hi,<div><br></div><div>I am interested in running the demo and investigating the issue.</div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Better question is if there is anything we can do to improve this? I believe this is a highly important aspect of the library.</div></div></blockquote><div><br></div><div>If it's caused by data that we don't need but that's still hanging around then yes.</div><div><br></div><div>Alex<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div class="gmail-HOEnZb"><div class="gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On 9 March 2017 at 17:53, Manuel Klimek <span dir="ltr"><<a href="mailto:klimek@google.com" target="_blank">klimek@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Benjamin had tried to come up with a repro at some point, too, iirc<br><br><div class="gmail_quote"><div><div class="gmail-m_2778857413344757691h5"><div dir="ltr">On Thu, Mar 9, 2017 at 4:48 PM Jusufadis Bakamovic via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div><div class="gmail-m_2778857413344757691h5"><div dir="ltr" class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">Hi,<div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"><br class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"></div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">I am a little bit puzzled about the memory management in libclang. What I observed is, when run on mid-sized code base, a very high (understandable) memory consumption whereas it seems it is not possible to reclaim all the consumed memory back once we are finished with whatever we have been doing with libclang. For example, this is a code excerpt which should explain my words:</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"><br class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">// 1. Create an index</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">vector<CXTranslationUnit> tunits;</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">CXIndex idx = clang_createIndex(1, 1);</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"><br class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"></div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">// 2. Parse each file found in directory and store corresponding TU into the vector</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">for (auto file& : directory) {</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"> CXTranslationUnit tu;</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"> if (!clang_parseTranslationUnit2(<wbr>idx, file.path().c_str(), cmd_args, cmd_args_len, 0, 0, CXTranslationUnit_DetailedPrep<wbr>rocessingRecord, &tu)</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"> tunits.push_back(tu);</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">}</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"><br class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"></div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">// 3. Cleanup</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">for (auto tu& : tunits) {</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"> clang_disposeTranslationUnit(t<wbr>u);</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">}</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">clang_disposeIndex(idx);</div></blockquote><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"><br class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"></div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">If I run this code on `cppcheck` code base (<a href="https://github.com/danmar/cppcheck" class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg" target="_blank">https://github.com/danmar/cpp<wbr>check</a>) which is somewhat a mid-sized project (it counts cca 300 C/C++ files), I get the following figures in terms of memory consumption for that particular process (based on `htop` output):</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"><br class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">* app memory consumption after 2nd step: virt(5763M) / res(5533M) / shr(380M)</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">* app memory consumption after 3rd step: virt(4423M) / res(4288M) / shr(65188)</div></blockquote><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"><br class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"></div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">So, as it can be seen from the figures, even after the cleanup stage, both virtual and resident memory figures are still very high. Seems like the only part which has been reclaimed is the memory that has been associated with the TU's. All the other, I can guess, parsing artifacts are still being hold somewhere in the memory to which we don't have access to neither we can flush them using the libclang API. I even ran the code with valgrind and there were no memory leaks detected (only `still reachable` blocks).</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"><br class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"></div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">Either I am missing something here or this might impose a memory issues for long-running non-single-shot tools (i.e. think indexer).</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"><br class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"></div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">Can anyone comment on this issue?</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"><br class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"></div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">Thanks,</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">Adi</div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"><br class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"></div><div class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"><br class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg"></div></div></div></div></div>
______________________________<wbr>_________________<br class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">
cfe-dev mailing list<br class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">
<a href="mailto:cfe-dev@lists.llvm.org" class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg" target="_blank">cfe-dev@lists.llvm.org</a><br class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/cfe-dev</a><br class="gmail-m_2778857413344757691m_-964102771787330168gmail_msg">
</blockquote></div></div>
</blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/cfe-dev</a><br>
<br></blockquote></div><br></div></div>