[cfe-dev] libclang: Why do I have to dispose CXTUs in LIFO order for memory to be freed?

[Harald Hvaal]

I have been trying to find out why the clang completer plugin in vim's
YouCompleteMe plugin makes vim take up more and more memory as I open
new buffers. On the github issue we eventually narrowed it down to a
simple program parsing 10 cpp files, then disposing their TUs.

When disposing them in the order that they were created, linux would
reuse the memory if the same files were parsed, but the RSS in ps
would never decrease, only increase when new files were parsed. I
eventually discovered that by disposing them in the opposite order, ie
LIFO order, the RSS stat would go up and down as expected.

For a completer plugin that needs to throw out the oldest TUs and
definitely not the newest ones, this is bad news. I would like to hear
your input on what it is in libclang that is causing this
fragmentation, and your advice for making it work.

The original discussion on github is here:

https://github.com/Valloric/YouCompleteMe/issues/184

And below is the (slightly modified) example program that demonstrates
the issue.

Harald Hvaal
harald.hvaal at gmail.com

---

#include <iostream>
#include <vector>
#include <list>

#include <clang-c/Index.h>

/*
 * Compile with:
 * clang++ -O3 -std=c++11 -stdlib=libc++ -lcxxrt -ldl -lclang
parse-bimap-samples.cpp -o parse-bimap-samples
 * or
 * g++ -std=c++11 -lclang
 */

int main()
{
    const bool dispose_in_FIFO_order = false;

    std::vector<std::string> tu_names = {
        "/usr/share/doc/libboost1.40-doc/examples/libs/bimap/example/at_function_examples.cpp",
        "/usr/share/doc/libboost1.40-doc/examples/libs/bimap/example/mighty_bimap.cpp",
        "/usr/share/doc/libboost1.40-doc/examples/libs/bimap/example/population_bimap.cpp",
        "/usr/share/doc/libboost1.40-doc/examples/libs/bimap/example/projection.cpp",
        "/usr/share/doc/libboost1.40-doc/examples/libs/bimap/example/repetitions_counter.cpp",
        "/usr/share/doc/libboost1.40-doc/examples/libs/bimap/example/simple_bimap.cpp",
        "/usr/share/doc/libboost1.40-doc/examples/libs/bimap/example/standard_map_comparison.cpp",
        "/usr/share/doc/libboost1.40-doc/examples/libs/bimap/example/step_by_step.cpp",
        "/usr/share/doc/libboost1.40-doc/examples/libs/bimap/example/tagged_simple_bimap.cpp",
        "/usr/share/doc/libboost1.40-doc/examples/libs/bimap/example/unconstrained_collection.cpp",
        "/usr/share/doc/libboost1.40-doc/examples/libs/bimap/example/user_defined_names.cpp"
    };

    for(int i = 0; i < 2; ++i)
    {
        std::cout << "Checkpoint" << std::endl;

        std::list<CXTranslationUnit> tus;

        CXIndex idx = clang_createIndex(1, 0);
        if (!idx)
        {
            std::cerr << "createIndex failed" << std::endl;
            return 2;
        }

        for(const auto &name : tu_names)
        {
            CXTranslationUnit tu = clang_parseTranslationUnit(
                    idx,
                    name.c_str(),
                    nullptr,
                    0,
                    nullptr,
                    0,
                    CXTranslationUnit_PrecompiledPreamble);

            if (!tu)
            {
                std::cerr << "parseTranslationUnit failed with " <<
name << std::endl;
                return 2;
            }

            clang_reparseTranslationUnit(tu, 0, 0, 0);
            std::cout << "Parsed " << name << std::endl;

            if (dispose_in_FIFO_order)
                tus.push_back(tu);
            else
                tus.push_front(tu);
        }

        std::cout << "All files parsed, go check the process memory
usage. Press ENTER to continue." << std::endl;
        std::cin.get();

        for(auto tu : tus)
            clang_disposeTranslationUnit(tu);

        std::cout << "TUs now disposed, check the memory usage again.
Press ENTER to continue." << std::endl;
        std::cin.get();

        clang_disposeIndex(idx);

        std::cout << "Index is disposed. Press ENTER to continue." << std::endl;
        std::cin.get();
    }

    return 0;
}