<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Sep 9, 2008, at 9:48 PM, <a href="mailto:b.j.burgers@student.utwente.nl">b.j.burgers@student.utwente.nl</a> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div lang="NL" link="blue" vlink="purple" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div class="Section1"><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Thanks for your quick response Snaroff.<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I choose to change the function Lexer::LexNumericConstant and generate my own kind of token if I find an s at the end of an numerical_constant.<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Now I want to separate the number from the s.<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Is there an easy way of getting the text of a token in the C file? </span></div></div></div></span></blockquote><div><br></div><div>Not directly. You need to go through the SourceManager as follows...</div><div><br></div><div>const char *sourceText = SM->getCharacterData(Tok.getLocation());</div><div><br></div><div>If you want the token type name, this will suffice...</div><div><br></div>const char *tokenName = Tok.getName();<br><blockquote type="cite"><span class="Apple-style-span" style="color: rgb(0, 0, 0); "><div lang="NL" link="blue" vlink="purple" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div class="Section1"></div></div></span></blockquote></div><div><br></div><div><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div lang="NL" link="blue" vlink="purple" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div class="Section1"><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I searched for this a while ago too but couldn’t find it. I found the SourceManager.GetCharacterData(Sourcelocation ) function, but this returns me all characters starting from the Sourceloaction.<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Is there a way to get a char * of text between 2 SourceLocations ?<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div></div></div></span></blockquote><div><br></div><div>These should do the trick...</div><div><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "> // converts SourceLocation's into "char *'s"<span style="color: #aa0d91"></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "><span style="color: #aa0d91"> const</span> <span style="color: #aa0d91">char</span> *startBuf = <span style="color: #3f6e74">SM</span>-><span style="color: #26474b">getCharacterData</span>(LocStart);</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "> <span style="color: #aa0d91">const</span> <span style="color: #aa0d91">char</span> *endBuf = <span style="color: #3f6e74">SM</span>-><span style="color: #26474b">getCharacterData</span>(LocEnd);</div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><br></span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"> // converts a "char *" offset into a SourceLocation</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "><span style=""> </span><span style="color: #3f6e74">SourceLocation</span> OptionalLoc = LocStart.<span style="color: #26474b">getFileLocWithOffset</span>(<span style="color: #3f6e74">p</span>-startBuf);</div><div><br></div><div>snaroff</div><div><br></div></span></font></div><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div lang="NL" link="blue" vlink="purple" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div class="Section1"><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Thanks for all the help,<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Bas<o:p></o:p></span></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div><div style="border-right-style: none; border-bottom-style: none; border-left-style: none; border-width: initial; border-color: initial; border-top-style: solid; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding-top: 3pt; padding-right: 0cm; padding-bottom: 0cm; padding-left: 0cm; "><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; ">Van:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; "><span class="Apple-converted-space"> </span>steve naroff [<a href="mailto:snaroff@apple.com" style="color: blue; text-decoration: underline; ">mailto:snaroff@apple.com</a>]<span class="Apple-converted-space"> </span><br><b>Verzonden:</b><span class="Apple-converted-space"> </span>dinsdag 9 september 2008 15:02<br><b>Aan:</b><span class="Apple-converted-space"> </span>Burgers, B.J. (Bas, Student EMSYS)<br><b>CC:</b><span class="Apple-converted-space"> </span><a href="mailto:cfe-dev@cs.uiuc.edu" style="color: blue; text-decoration: underline; ">cfe-dev@cs.uiuc.edu</a><br><b>Onderwerp:</b><span class="Apple-converted-space"> </span>Re: [cfe-dev] changing the lexer or parser<o:p></o:p></span></div></div></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Hi Bas,<o:p></o:p></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">clang currently implements C integer constants by including the trailing suffix (see C99 6.4.4.1 for more details).<o:p></o:p></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Sema::ActOnNumbericConstant() is then responsible for determining the type of constant (integer, floating) and size.<o:p></o:p></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">I haven't thought about adapting clang's lexer to generate tokens that don't conform to C.<o:p></o:p></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">That said, you could simply examine the "suffix" by hand (without fiddling with the lexer directly).<o:p></o:p></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">snaroff<o:p></o:p></div></div><div><div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">On Sep 9, 2008, at 5:27 PM,<span class="Apple-converted-space"> </span><a href="mailto:b.j.burgers@student.utwente.nl" style="color: blue; text-decoration: underline; ">b.j.burgers@student.utwente.nl</a><span class="Apple-converted-space"> </span>wrote:<o:p></o:p></div></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><br><br><o:p></o:p></div><div><div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; ">Hello,</span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; "><o:p></o:p></span></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; "> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; "><o:p></o:p></span></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; ">I’m working on a tool that allows time construct in C. I implemented this tool by adapting Clang.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; "><o:p></o:p></span></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; ">In these time constructs I like to allow arguments like “1000s”, “1000 s”, “1000 s”, “100ms”, “100 ms”, etc.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; "><o:p></o:p></span></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; ">The lexer creates 1 token called numerical_token if the argument is “1000s” even if ‘s’ is added as keyword or token in TokenKinds.def. I hoped the lexer would have generated two tokens, 1 numerical_constant and an identifier (or self defined token).</span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; "><o:p></o:p></span></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; ">What is the best way to allow these kind of arguments ? Do I have to create a new token that allows some digits followed by an ‘s’?</span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; "><o:p></o:p></span></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; "> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; "><o:p></o:p></span></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; ">Thanks for any help,</span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; "><o:p></o:p></span></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; "> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; "><o:p></o:p></span></div></div><div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; ">Bas Burgers</span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: black; "><o:p></o:p></span></div></div></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: black; ">_______________________________________________<br>cfe-dev mailing list<br><a href="mailto:cfe-dev@cs.uiuc.edu" style="color: blue; text-decoration: underline; ">cfe-dev@cs.uiuc.edu</a><br><a href="http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev" style="color: blue; text-decoration: underline; ">http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev</a><o:p></o:p></span></div></div></div><div style="margin-top: 0cm; margin-right: 0cm; margin-left: 0cm; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div></div>_______________________________________________<br>cfe-dev mailing list<br><a href="mailto:cfe-dev@cs.uiuc.edu" style="color: blue; text-decoration: underline; ">cfe-dev@cs.uiuc.edu</a><br><a href="http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev" style="color: blue; text-decoration: underline; ">http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev</a><br></div></span></blockquote></div><br></body></html>