[cfe-dev] clang memory usage with C++ template metaprogramming

[John Bytheway]

On 08/06/10 23:03, Douglas Gregor wrote:
> 
> On Jun 8, 2010, at 2:55 PM, John Bytheway wrote:
> 
>> Several years ago I embarked on a project involving some heavy-duty
>> C++ template metaprogramming.  In the end I abandoned it because
>> the compile times and memory usage with g++ were too big.
>> 
>> On seeing clang's promised reduction of such requirements, I
>> thought I'd go back to my project and see how clang fared when
>> compiling it. Although it does indeed run much faster than g++, it
>> actually uses *more* memory.  I'm just posting here to ask if this
>> is to be expected. If it might be indicative of some issue or if
>> you'd like to know where all this memory is being used then I'd be
>> happy to try some profiling.
<snip>

> I've also seen this with template metaprogramming-heavy code, but
> aside from some idle speculation (we think it has to do with
> type-location information in Clang), we haven't looked into it
> closely.

Fair enough.  I was curious, so I ran valgrind/massif to get an idea.
In short:

16.53% (259,009,024B) in 722 places, all below massif's threshold
14.49% (227,086,336B) clang::DeclContext::CreateStoredDeclsMap
12.85% (201,326,592B) clang::SourceManager::createInstantiationLoc
12.83% (201,068,544B) clang::ASTContext::CreateTypeSourceInfo
08.86% (138,792,960B) clang::ASTContext::getTemplateSpecializationType
06.82% (106,841,236B) clang::TokenLexer::ExpandFunctionArguments
04.94% (77,463,552B) clang::CXXConstructorDecl::Create
02.86% (44,883,968B) clang::ClassTemplateSpecializationDecl::Create
02.71% (42,532,864B) clang::ParmVarDecl::Create
02.25% (35,332,096B) clang::TagDecl::startDefinition
02.15% (33,763,328B) clang::TemplateArgumentList::TemplateArgumentList
02.14% (33,554,432B) std::vector<clang::Type*,
std::allocator<clang::Type*> >::_M_insert_aux
02.06% (32,329,728B) clang::CXXRecordDecl::Create
02.05% (32,157,696B) clang::CXXMethodDecl::Create
01.82% (28,585,984B) clang::CXXDestructorDecl::Create
01.59% (24,907,776B) clang::ASTContext::getFunctionType
01.27% (19,861,504B) clang::ASTContext::getLValueReferenceType
01.08% (16,908,288B) clang::TypedefDecl::Create

So indeed type location information is a significant part, but nothing
is overwhelming, which I guess is a good sign and nothing is worth changing.

I wonder idly: How plausible would it be to allow execution in a mode
where no source information was maintained, and thus reduce memory usage
(at the expense of useful errors/warnings)?  Such a mode might be useful
at times.  I'm guessing it would be prohibitively difficult.

John Bytheway