<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jun 17, 2013 at 1:15 PM, Thompson, John <span dir="ltr"><<a href="mailto:John_Thompson@playstation.sony.com" target="_blank">John_Thompson@playstation.sony.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Sean,<u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks again for your help.<u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Sorry, I overlooked revising the function that was doing some old C-style string handling. I’ve addressed that now (in the new ModularizePPDirective::print
function).<u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">The only other item I didn’t address was the macro substitutions function, which was intentional as I described. I’ve looked into this further, but I’d still
like to punt on it for now. Because there isn’t such a function already, it appears the only clean way to do it would be to instantiate a new Lexer and Preprocessor object, linking the new preprocessor to the old so it can find the macros (there seems to
be a pointer for this). It would also require adding at least a new Lexer constructor, so it can set up the buffer pointers and correct state. The constructor for lexing pragmas is close, but it sets the lexer in raw mode, which I don’t want. There’s also
the problem that it will call the MacroExpanded handler again, and I also worry there might be other side-effects due to the preprocessor and lexer states being changed inadvertently. Since this is an external tool, I’m kind of thinking that it would be simpler
to just use the function I put in for now, and I will look into it some more when I try to figure out how to do the function-style macro expansion, possibly using the existing code in the Preprocessor class.</span></p></div>
</div></blockquote><div><br></div><div style>Please ask on cfe-dev about what you are trying to achieve and how best to achieve it. While this specific functionality (a function that does this) may not be available, clang may have facilities for producing the end result in a simpler way. Since clang already gives detailed information about macro expansion for many diagnostics, it seems likely that your goals here can be achieved with the current code.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Regarding breaking the patch up, it might be easier to consider the changes in three parts:<u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>1.<span style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">The new classes.<u></u><u></u></span></p>
<p><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>2.<span style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">The changes to Modularize.cpp.<u></u><u></u></span></p>
<p><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>3.<span style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">The changes to the tests.</span> </p></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple"><div><p><span style="color:rgb(31,73,125);font-family:Calibri,sans-serif;font-size:11pt">I’ll provide these in separate patch files, though the associated changes will need to go in at the same time, however.</span></p>
</div></div></blockquote><div style>In the LLVM community "breaking the patch up" is understood to mean breaking into a series of patches that can be applied in series and that compile and run (and pass all tests) after each patch is applied. Try to keep changes that need to be applied together together in a single patch file (this includes tests); this is basically always achievable (the exception is typically changes that affect multiple projects (e.g. changes to LLVM that require changes to clang), but that isn't an issue in this situation).</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><p class="">
<span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">The new classes (mod_2013_06_17_new_patch.txt). I should give some description of the new classes added and how they work together:<u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>1.<span style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ModularizePPCallbacks – Derives from the Clang PPCallbacks class to track preprocessor actions, such as changing files and handling preprocessor directives
and macro expansions. It has to figure out when a new header file is entered and left, as the provided handler is not particularly clear about it. It also stores a map of macro expansions obtained from the MacroExpands callback, for use by a function that
effectively preprocesses a conditional. It handles the top-level aspects of collecting header file instance information, and tracking the preprocessor conditional directives.<u></u><u></u></span></p>
<p><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>2.<span style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ModularizePPDirective – Stores information about one preprocessor directive instance, presently limited to #if, #elif, #ifdef, and #ifndef, since
that is all modularize needs for now. It stores the source file line number, a directive kind code, and both the unpreprocessed and preprocessed conditional source code snippet.<u></u><u></u></span></p>
<p><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>3.<span style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ModularizeHeaderFile – Store a header file name and a vector of ModularizePPDirective instances collected for that header file.<u></u><u></u></span></p>
<p><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>4.<span style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ModularizeHeaderInstance – Stores a pointer to a ModularizeHeaderFile for a header, and a vector of header file names for the headers from the modularize
header list that reference the particular header, either directly or indirectly via some nested include. If separate instances of the header are encountered when modularize processes its header list, if the preprocessed directive conditionals stored in the
ModularizePPDirective vector are the same for and existing ModularizeHeaderFile object, the top-level header name is added to the instance, effectively reusing the ModularizeHeaderFile object. If a header is seen for the first time, or if the preprocessed
conditionals for the stored directives don’t match those of an instance of the header seen before, a new ModularizeHeaderInstance object is created and saved.<u></u><u></u></span></p>
<p><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>5.<span style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ModularizeHeaderTracker – Tracks the instances of one particular header. It stores the header name and a vector of ModularizeHeaderInstance’s. If
all instances of a header seen have the same conditionals after preprocessing, there will only be one ModularizeHeaderInstance. If one or more conditionals were difference, there will be two or more instance objects saved.<u></u><u></u></span></p>
<p><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>6.<span style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ModularizeMasterHeaderTracker – Stores a map of all the ModularizeHeaderTracker objects, and provides an “addHeaderFile” function for handling a header
file, and a “report” function for outputting the warnings about the preprocessor conditional directive mismatches.</span></p></div></blockquote><div><br></div><div style>Please put these descriptions in doxygen comments for their respective classes.</div>
<div style> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple">
<div><p><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">The changes to Modularize.cpp (mod_2013_06_17_modularize_patch.txt):<u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>1.<span style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Add an option for disabling the preprocessing consistency checking. This is a fallback, in case of problems with the mechanism, or to reduce warnings
volume.<u></u><u></u></span></p>
<p><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>2.<span style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Set up a ModularizePPCallbacks object for tracking the preprocessor. This is done in the CollectEntitiesConsumer object.<u></u><u></u></span></p>
<p><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>3.<span style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Set up a ModularizeMasterHeaderTracker for storing the header instance data. This is done in the CollectEntitiesConsumer object.<u></u><u></u></span></p>
<p><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>4.<span style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Call the ModularizeMasterHeaderTracker::report function to report any warnings about the preprocessor conditional directive mismatches.<u></u><u></u></span></p>
<p><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>5.<span style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Fixed some naming convention issues.</span></p></div></div></blockquote><div><br></div><div style>At the very least number 5 can be a separate patch (feel free to commit it directly).</div>
<div style> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple">
<div><p><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">The changes to the tests (mod_2013_06_17_test_patch.txt):<u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>1.<span style="font-style:normal;font-variant:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Some new lines in a couple of files for the new feature.<u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">I’m also including a zip with the changed files.</span></p></div></div></blockquote><div style>For future reference, including a zip like this is not necessary (nor particularly desirable for most people's workflow I assume).</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">I’m hoping I can check this in soon, as it makes me nervous to sit on so much, and makes it harder to continue experimenting. Since this is still an experimental
tool, I’m hoping we can improve it in incremental steps.</span></p></div></div></blockquote><div style>At the very least, the stylistic issues (variable naming, line endings, tabs, formatting, etc.) will need to be cleared up before committing.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">You mentioned you have other suggestions too. Please do feel free to send them.<u></u><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> </span></p></div></div></blockquote><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple"><p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u></span></p>
<p class=""><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">One thing I’m aware of is that the collections are probably leaking the objects’ memory. I can fix that if necessary.</span></p>
</div></blockquote><div><br></div><div style>Yes, please fix that; memory leaks are not acceptable.</div><div style><br></div><div style>Some other things:</div><div style><br></div><div style>* Please rename things called `ModularizeFoo` to be just `Foo` if they are already in the `Modularize` namespace.</div>
<div style>* Please name your variables in accordance with LLVM naming conventions and put `*` and `&` on the right.</div><div style>* Remove hard tabs.</div><div style>* Use Unix (LF) line endings throughout (I see many CRLF) and make sure to have <=80 columns per line.</div>
<div style><br></div><div style><div>+bool ModularizeHeaderTracker::report() {</div><div>+ int headerInstanceIndex;</div><div>+ int headerInstanceCount = getHeaderInstanceCount();</div><div>+ ModularizeHeaderInstance *headerInstance;</div>
<div>+ ModularizeHeaderInstance *headerInstance0;</div><div>+ ModularizeHeaderFile *headerFile;</div><div>+ ModularizeHeaderFile *headerFile0;</div><div>+ int directiveIndex;</div><div>+ ModularizePPDirective *directive;</div>
<div>+ ModularizePPDirective *directive0;</div><div>+ bool mismatch;</div><div><br></div><div>Please move these declarations to their point of first use. Also clarify what the `0` suffix on some of these names means (possibly by renaming to a more useful name).<br>
</div><div><br></div><div>+ if (getHeaderInstanceCount() > 1) {</div><div>+ int directiveCount = 0x7fffffff; // Big number.</div><div>+ ModularizeHeaderInstanceVectorIterator hiIter = getHeaderInstancesBegin();</div>
<div><br></div><div style>Reverse the condition of this `if` and use an early exit to simplify and reduce indentation <<a href="http://llvm.org/docs/CodingStandards.html#use-early-exits-and-continue-to-simplify-code">http://llvm.org/docs/CodingStandards.html#use-early-exits-and-continue-to-simplify-code</a>>.</div>
<div style><br></div><div style><div>+ if (mismatch) {</div><div>+ errs() << "warning: The instances of header " << Name << " have different contents after preprocessing:\n";</div>
<div><br></div><div style>Same for this `if`.</div><div style><br></div><div style><div>+// Represents a preprocessor directive kind.</div><div>+enum ModularizePPDirectiveKind {</div><div>+ MPPD_If,</div><div>+ MPPD_ElIf,</div>
<div>+ MPPD_IfDef,</div><div>+ MPPD_IfNDef</div><div>+};</div><div><br></div><div style>Surely clang already has an enum that can serve this purpose?</div><div style><br></div></div><div style><div>+// Get directive spelling.</div>
<div>+const char *ModularizePPDirective::getDirectiveSpelling(</div><div>+ ModularizePPDirectiveKind kind) {</div><div>+ const char *directiveString;</div><div>+ switch (kind) {</div><div>+ case MPPD_If:</div><div>
+ directiveString = "if";</div><div><br></div><div style>Surely clang already has something that can do this?</div><div style><br></div><div style><div>+ ModularizeHeaderFile *mhf = new ModularizeHeaderFile(topFile.str(), PP);</div>
<div>+ addHeaderFile(mhf);</div><div>+ CurrentHeaderFile = mhf;</div><div>+ RootHeaderFile = mhf;</div><div><br></div><div style>Who owns this? When is it freed?</div><div style><br></div><div style><div>+ mhf = getHeaderFile(*fileName);</div>
<div>+ if (mhf == NULL) {</div><div>+ mhf = new ModularizeHeaderFile(*fileName, PP);</div><div><br></div><div style>Same here.</div><div style><br></div><div style><div>+// Retrieve source snippet from file image.</div>
<div>+std::string ModularizePPCallbacks::getSourceSnippet(SourceRange sourceRange) {</div><div><br></div><div style>Surely clang has this functionality somewhere already?</div><div style><br></div><div style><div>+ std::string returnValue(bPtr, length);</div>
<div>+</div><div>+ return returnValue;</div><div><br></div><div style>You can directly `return std::string(...)`.</div></div></div></div></div></div></div><div><br></div><div><div>+ // Trim snippet.</div><div>+ while ((*bPtr <= ' ') && (length != 0)) {</div>
<div>+ bPtr++;</div><div>+ length--;</div><div>+ }</div><div>+</div><div>+ while ((length != 0) && (bPtr[length - 1] <= ' '))</div><div>+ length--;</div></div><div><br></div><div style>StringRef::trim?</div>
<div style><br></div><div style><div>+ using namespace clang;</div><div>+ using namespace llvm;</div><div><br></div><div style>No `using namespace` in headers. (Second sentence of <<a href="http://llvm.org/docs/CodingStandards.html#do-not-use-using-namespace-std">http://llvm.org/docs/CodingStandards.html#do-not-use-using-namespace-std</a>>)</div>
<div style><br></div><div style>+ return (*iter).second;<br></div><div style><br></div><div style>Use operator-> (e.g. iter->second), here and in other places.</div><div style><br></div><div style><div>+#ifndef MODULARIZEHEADERTRACKER_H</div>
<div>+#define MODULARIZEHEADERTRACKER_H</div><div><br></div><div style>In LLVM we typically put underscores between "words" in header guard macros. So e.g. this should be MODULARIZE_HEADER_TRACKER_H.</div></div>
</div><div style><br></div><div style>-- Sean Silva</div><div style><br></div></div></div></div></div>