<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Apr 25, 2008, at 11:35 AM, Emerson Murphy-Hill wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">I'm having a bit of difficulty with the lexer. It's been working dandy for me for the most part, but has some problems when lexing over macro expansions that are empty. </div></blockquote><div><br class="webkit-block-placeholder"></div>Please paste the code you're using to lex over this. The "clang -dump-tokens" option presumably works on this, right?</div><div><br class="webkit-block-placeholder"></div><div><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); "><font class="Apple-style-span" color="#000000" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">is no problem. But defining the macro thusly:</span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); "><font class="Apple-style-span" color="#000000" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); "><font class="Apple-style-span" color="#000000" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); ">#ifndef MyMacro</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); ">#define MyMacro</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); ">#endif</div></span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); "><font class="Apple-style-span" color="#000000" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); "><font class="Apple-style-span" color="#000000" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">or:</span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); "><font class="Apple-style-span" color="#000000" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); "><font class="Apple-style-span" color="#000000" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); ">#ifndef MyMacro</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); ">#define MyMacro <span style="color: #007400">/*some comment*/</span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); ">#endif</div></span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); "><font class="Apple-style-span" color="#000000" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); "><font class="Apple-style-span" color="#000000" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">causes a EXC_BAD_ACCESS here:</span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); "><font class="Apple-style-span" color="#000000" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); "><font class="Apple-style-span" color="#000000" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">#0</span></font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"> </span></font></span><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">clang::TokenLexer::isAtEnd at TokenLexer.h:122</span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">#1</span></font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"> </span></font></span><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">clang::TokenLexer::Lex at TokenLexer.cpp:279</span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">#2</span></font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"> </span></font></span><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">clang::Preprocessor::Lex at Preprocessor.h:262</span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">#3</span></font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"> </span></font></span><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">clang::Preprocessor::HandleMacroExpandedIdentifier at PPMacroExpansion.cpp:205</span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">#4</span></font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"> </span></font></span><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">clang::Preprocessor::HandleIdentifier at Preprocessor.cpp:532</span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">#5</span></font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"> </span></font></span><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">clang::Lexer::LexIdentifier at Lexer.cpp:534</span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">#6</span></font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"> </span></font></span><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">clang::Lexer::LexTokenInternal at Lexer.cpp:1314</span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">#7</span></font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"> </span></font></span><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">clang::Lexer::Lex at Lexer.h:141</span></font></div><div><font class="Apple-style-span" color="#643820" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><br></span></font></div><div>(Actually, in the comment case, I'm not entirely sure if the comment gets expanded or simply removed by the preprocessor before expansion.)</div></span></font></div></div></div></blockquote><div><br class="webkit-block-placeholder"></div>Are you sure you're not lexing past EOF?</div><div><br><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); "><font class="Apple-style-span" color="#000000" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"><div><span class="Apple-style-span" style="color: rgb(20, 79, 174); -webkit-text-stroke-width: -1; ">This problem seems odd because the source file is getting parsed by clang correctly the first time around (for instance, I can run the clang driver and emit HTML, no problem). But when I make a new lexer (re-using the preprocessor from the original parsing), I get the above errors. Is there some lexer or preprocessor setting that I'm missing?</span></div><div><br></div></span></font></div></div></div></blockquote>when reusing a preprocessor like this, be aware that you're starting out with the full t-u full of macro definitions already installed. This means that the macro is already defined, so the #ifndef block isn't entered.</div><div><br class="webkit-block-placeholder"></div><div>-Chris</div></body></html>