<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br class="Apple-interchange-newline">On Apr 25, 2008, at 2:45 PM, Chris Lattner wrote:<br class="Apple-interchange-newline"><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Apr 25, 2008, at 11:35 AM, Emerson Murphy-Hill wrote:</div><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">I'm having a bit of difficulty with the lexer. It's been working dandy for me for the most part, but has some problems when lexing over macro expansions that are empty. </div></blockquote><div><br class="webkit-block-placeholder"></div>Please paste the code you're using to lex over this. The "clang -dump-tokens" option presumably works on this, right?</div></div></blockquote><div><br></div><div>dump-tokens works. Here's some crashing code:</div><div><br></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "> <span style="color: rgb(92, 38, 153); ">Lexer</span>* lexer = <span style="color: rgb(170, 13, 145); ">new</span> <span style="color: rgb(92, 38, 153); ">Lexer</span>(LocStart,*(<span style="color: rgb(63, 110, 116); ">provider</span>-><span style="color: rgb(38, 71, 75); ">preprocessor</span>()),startBuf,endBuf);</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; min-height: 14px; "> <br class="webkit-block-placeholder"></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "> <span style="color: rgb(63, 110, 116); ">Token</span> current;</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "> <span style="color: rgb(170, 13, 145); ">do</span> {</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "> lexer-><span style="color: rgb(63, 110, 116); ">Lex</span>(current);</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "> }<span style="color: rgb(170, 13, 145); ">while</span>(!current.<span style="color: rgb(46, 13, 110); ">is</span>(<span style="color: rgb(92, 38, 153); ">tok</span>::<span style="color: rgb(46, 13, 110); ">eof</span>) && !current.<span style="color: rgb(46, 13, 110); ">is</span>(<span style="color: rgb(92, 38, 153); ">tok</span>::<span style="color: rgb(46, 13, 110); ">r_brace</span>));</div></div><div><div><br></div><div>Works fine when the macro has something in it, crashes when it's empty.</div><div><br></div></div><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); "><font class="Apple-style-span" color="#000000" face="Helvetica" size="3"><div><font class="Apple-style-span" color="#643820" face="Monaco" size="2"><br></font></div><div>(Actually, in the comment case, I'm not entirely sure if the comment gets expanded or simply removed by the preprocessor before expansion.)</div></font></div></div></div></blockquote><div><br class="webkit-block-placeholder"></div>Are you sure you're not lexing past EOF?</div></div></blockquote><div><br></div><div>Pretty sure.</div><br><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(100, 56, 32); "><font class="Apple-style-span" color="#000000" face="Helvetica" size="3"><div><span class="Apple-style-span" style="color: rgb(20, 79, 174); -webkit-text-stroke-width: -1; ">This problem seems odd because the source file is getting parsed by clang correctly the first time around (for instance, I can run the clang driver and emit HTML, no problem). But when I make a new lexer (re-using the preprocessor from the original parsing), I get the above errors. Is there some lexer or preprocessor setting that I'm missing?</span></div><div><br></div></font></div></div></div></blockquote>when reusing a preprocessor like this, be aware that you're starting out with the full t-u full of macro definitions already installed. This means that the macro is already defined, so the #ifndef block isn't entered.</div></div></blockquote><div><br></div>That makes sense.</div><div><br></div><div>Thanks,</div><div><br></div><div>e</div><div><br></div></div></div></body></html>