<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Feb 10, 2011, at 4:26 PM, Douglas Gregor wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Author: dgregor<br>Date: Thu Feb 10 18:26:14 2011<br>New Revision: 125324<br><br>URL: <a href="http://llvm.org/viewvc/llvm-project?rev=125324&view=rev">http://llvm.org/viewvc/llvm-project?rev=125324&view=rev</a><br>Log:<br>Implement AST/PCH chaining support for macro definitions. Previously,<br>we would deserialize all of the macro definitions we knew about while<br>serializing the macro definitions at the end of the AST/PCH file. Even<br>though we skipped most of them (since they were unchanged), it's still<br>a performance problem.<br><br>Now, we do the standard AST/PCH chaining trick: watch what identifiers<br>are deserialized as macro names, and consider only those identifiers<br>(along with macro definitions that have been deserialized/written in<br>the source) when serializing the preprocessor state.<br></div></blockquote><div><br></div><div>For those of us keeping score, this takes us from:</div><div><br></div><div><div>*** AST File Statistics:</div><div><div>*** AST File Statistics:</div><div> 2 stat cache hits</div><div> 6 stat cache misses</div><div> 1/64463 source location entries read (1.469060%)</div><div> 15606/16956 types read (92.038216%)</div><div> 59266/89334 declarations read (66.342041%)</div><div> 47651/61393 identifiers read (77.616341%)</div><div> 0/7778 selectors read (0.000000%)</div><div> 24192/34644 statements read (69.830276%)</div><div> 8809/8809 macros read (100.000000%)</div><div> 2095/5189 lexical declcontexts read (40.373867%)</div><div> 0/4587 visible declcontexts read (0.000000%)</div><div> 0/7716 method pool entries read (0.000000%)</div><div> 0 method pool misses</div><div><br></div></div><div>to:</div><div><br></div><div>*** AST File Statistics:</div></div><div><div> 2 stat cache hits</div><div> 6 stat cache misses</div><div> 1/64463 source location entries read (0.001551%)</div><div> 15606/16956 types read (92.038216%)</div><div> 59266/89334 declarations read (66.342041%)</div><div> 38952/61393 identifiers read (63.446976%)</div><div> 0/7778 selectors read (0.000000%)</div><div> 24192/34644 statements read (69.830276%)</div><div> 388/8809 macros read (4.404586%)</div><div> 2095/5189 lexical declcontexts read (40.373867%)</div><div> 0/4587 visible declcontexts read (0.000000%)</div><div> 0/7716 method pool entries read (0.000000%)</div><div> 0 method pool misses</div></div><div><br></div><div>when creating a chained PCH file for a file t.h that includes Cocoa.h and adds one simple function declaration. We're still getting 388 macros read because of the huge number of declarations and identifiers being read; those are the subject of my next commit.</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- Doug</div><div><br></div><blockquote type="cite"><div><br>Modified:<br> cfe/trunk/include/clang/Serialization/ASTWriter.h<br> cfe/trunk/lib/Serialization/ASTWriter.cpp<br> cfe/trunk/test/PCH/reinclude1.h<br><br>Modified: cfe/trunk/include/clang/Serialization/ASTWriter.h<br>URL: <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Serialization/ASTWriter.h?rev=125324&r1=125323&r2=125324&view=diff">http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Serialization/ASTWriter.h?rev=125324&r1=125323&r2=125324&view=diff</a><br>==============================================================================<br>--- cfe/trunk/include/clang/Serialization/ASTWriter.h (original)<br>+++ cfe/trunk/include/clang/Serialization/ASTWriter.h Thu Feb 10 18:26:14 2011<br>@@ -184,6 +184,9 @@<br> /// defined.<br> llvm::DenseMap<const IdentifierInfo *, uint64_t> MacroOffsets;<br><br>+ /// \brief The set of identifiers that had macro definitions at some point.<br>+ std::vector<const IdentifierInfo *> DeserializedMacroNames;<br>+ <br> /// \brief The first ID number we can use for our own macro definitions.<br> serialization::MacroID FirstMacroID;<br><br><br>Modified: cfe/trunk/lib/Serialization/ASTWriter.cpp<br>URL: <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Serialization/ASTWriter.cpp?rev=125324&r1=125323&r2=125324&view=diff">http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Serialization/ASTWriter.cpp?rev=125324&r1=125323&r2=125324&view=diff</a><br>==============================================================================<br>--- cfe/trunk/lib/Serialization/ASTWriter.cpp (original)<br>+++ cfe/trunk/lib/Serialization/ASTWriter.cpp Thu Feb 10 18:26:14 2011<br>@@ -1536,7 +1536,8 @@<br> llvm::SmallVector<std::pair<const IdentifierInfo *, MacroInfo *>, 2> <br> MacrosToEmit;<br> llvm::SmallPtrSet<const IdentifierInfo*, 4> MacroDefinitionsSeen;<br>- for (Preprocessor::macro_iterator I = PP.macro_begin(), E = PP.macro_end();<br>+ for (Preprocessor::macro_iterator I = PP.macro_begin(Chain == 0), <br>+ E = PP.macro_end(Chain == 0);<br> I != E; ++I) {<br> MacroDefinitionsSeen.insert(I->first);<br> MacrosToEmit.push_back(std::make_pair(I->first, I->second));<br>@@ -1547,10 +1548,21 @@<br> llvm::array_pod_sort(MacrosToEmit.begin(), MacrosToEmit.end(), <br> &compareMacroDefinitions);<br><br>+ // Resolve any identifiers that defined macros at the time they were<br>+ // deserialized, adding them to the list of macros to emit (if appropriate).<br>+ for (unsigned I = 0, N = DeserializedMacroNames.size(); I != N; ++I) {<br>+ IdentifierInfo *Name<br>+ = const_cast<IdentifierInfo *>(DeserializedMacroNames[I]);<br>+ if (Name->hasMacroDefinition() && MacroDefinitionsSeen.insert(Name))<br>+ MacrosToEmit.push_back(std::make_pair(Name, PP.getMacroInfo(Name)));<br>+ }<br>+ <br> for (unsigned I = 0, N = MacrosToEmit.size(); I != N; ++I) {<br> const IdentifierInfo *Name = MacrosToEmit[I].first;<br> MacroInfo *MI = MacrosToEmit[I].second;<br>-<br>+ if (!MI)<br>+ continue;<br>+ <br> // Don't emit builtin macros like __LINE__ to the AST file unless they have<br> // been redefined by the header (in which case they are not isBuiltinMacro).<br> // Also skip macros from a AST file if we're chaining.<br>@@ -1603,7 +1615,6 @@<br> // FIXME: When reading literal tokens, reconstruct the literal pointer if<br> // it is needed.<br> AddIdentifierRef(Tok.getIdentifierInfo(), Record);<br>-<br> // FIXME: Should translate token kind to a stable encoding.<br> Record.push_back(Tok.getKind());<br> // FIXME: Should translate token flags to a stable encoding.<br>@@ -3681,6 +3692,8 @@<br><br> void ASTWriter::IdentifierRead(IdentID ID, IdentifierInfo *II) {<br> IdentifierIDs[II] = ID;<br>+ if (II->hasMacroDefinition())<br>+ DeserializedMacroNames.push_back(II);<br> }<br><br> void ASTWriter::TypeRead(TypeIdx Idx, QualType T) {<br><br>Modified: cfe/trunk/test/PCH/reinclude1.h<br>URL: <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/test/PCH/reinclude1.h?rev=125324&r1=125323&r2=125324&view=diff">http://llvm.org/viewvc/llvm-project/cfe/trunk/test/PCH/reinclude1.h?rev=125324&r1=125323&r2=125324&view=diff</a><br>==============================================================================<br>--- cfe/trunk/test/PCH/reinclude1.h (original)<br>+++ cfe/trunk/test/PCH/reinclude1.h Thu Feb 10 18:26:14 2011<br>@@ -2,3 +2,6 @@<br> int x;<br> int y;<br> }<br>+<br>+int foo;<br>+#define foo foo<br><br><br>_______________________________________________<br>cfe-commits mailing list<br><a href="mailto:cfe-commits@cs.uiuc.edu">cfe-commits@cs.uiuc.edu</a><br>http://lists.cs.uiuc.edu/mailman/listinfo/cfe-commits<br></div></blockquote></div><br></body></html>