<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Nov 2, 2013 at 11:45 AM, Daniel Jasper <span dir="ltr"><<a href="mailto:djasper@google.com" target="_blank">djasper@google.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">More specifically:<div>- "var" is lexed just fined. It is the identifier lookup/parsing that you are thinking of and we can easily do that.</div>
<div>- let ===/!== just be lexed as ==/!= and =.. We can set the spacing/line break appropriately. As I mentioned. We don't need to understand the language, we just need to format it correctly. Or just do the post-processing I was describing and merge the two tokens received from the lexer.</div>
</div></blockquote><div><br></div><div>While those don't seem like a big deal in terms of hacking around Clang's lexer not being a javascript lexer, JavaScript's regex literals do seem like they would be quite a bit of work. For example, you might have:</div>
<div><br></div><div>if ((m = /^(\s*)([a-zA-Z0-9._-]+)\s*=\s*/.exec(chunk))) {</div><div><br></div><div>As long as clang's lexer doesn't choke on stuff like this, you probably can reconstruct it just fine with a bit of work. It just seems like a source of a lot of complexity. As a side note, there are some edge cases where the regex literal can contains something that clang will lex as a comment, e.g. /[/*]/ or /[//]/, but I doubt those are very common and so not problematic. Languages with "raw" string literals that aren't lexically identical to C++'s will also present a similar problem (e.g. a URL in a raw string literal will result in http:// interpreted as starting a comment).</div>
<div><br></div><div>It just seems like using one language's lexer for another language is a big hack, and very prone to running into a "showstopper" problem. There are perfectly good javascript lexers around (e.g. <<a href="http://esprima.org/demo/parse.html">http://esprima.org/demo/parse.html</a>>, click on "Tokens"; check "Line and column based" to get source location info and "include comments" for comments). Could clang-format just have a component that accepts a stream of tokens on stdin in some specified format and produces a list of replacements?</div>
<div><br></div><div>-- Sean Silva</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">
</div><div class=""><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Nov 2, 2013 at 4:34 PM, Manuel Klimek <span dir="ltr"><<a href="mailto:klimek@google.com" target="_blank">klimek@google.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>On Sat, Nov 2, 2013 at 4:18 PM, Sean Silva <span dir="ltr"><<a href="mailto:silvas@purdue.edu" target="_blank">silvas@purdue.edu</a>></span> wrote:<br>
</div><div class="gmail_extra"><div class="gmail_quote"><div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">
<div>On Fri, Nov 1, 2013 at 5:50 PM, Daniel Jasper <span dir="ltr"><<a href="mailto:djasper@google.com" target="_blank">djasper@google.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">My gut feeling is that we won't need to change the lexer. Bear in mind, that (same as with everything else in clang-format) we only need to understand the language good enough to format it. There might always be corner cases where we aren't correct, but these are rare in practice.</div>
</blockquote><div><br></div></div><div>How do you intend to lex JavaScript's === and !== operators? Or the `var` keyword? These aren't "corner cases" at all.</div></div></div></div></blockquote><div><br>
</div></div><div>What do you expect to be formatted differently because of those?</div><div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote"><span><font color="#888888"><div><br></div><div>-- Sean Silva</div></font></span><div><div><br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>
<br></div><div>In fact, I would like to go ahead and see whether we really hit the limit somewhere and if so, what the problems are. Once we have sufficient information, we can make a good decision on how to continue. Options would be allowing different lexers or post-processing the output of Clang's lexer. I fully agree that we should not modify the lexer to accommodate other languages.</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On Fri, Nov 1, 2013 at 9:17 PM, Chandler Carruth <span dir="ltr"><<a href="mailto:chandlerc@google.com" target="_blank">chandlerc@google.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div><div dir="ltr"><div><div class="gmail_extra">
<br><div class="gmail_quote">On Fri, Nov 1, 2013 at 1:13 PM, Ryan Gonzalez <span dir="ltr"><<a href="mailto:rymg19@gmail.com" target="_blank">rymg19@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">What if the lexer was an overly large, non-abstract base class? Then, the derived classes can just override the tokens as needed.</blockquote>
</div><br></div></div><div class="gmail_extra">(FWIW, I wouldn't try to design changes to the lexer in this thread, in the abstract... If this is an interesting path to pursue, I suspect Daniel or others should produce concrete proposed patches that enable the features needed and minimize the pollution of the lexer with other languages...)</div>
</div>
<br></div></div><div>_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@cs.uiuc.edu" target="_blank">cfe-dev@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev</a><br>
<br></div></blockquote></div><br></div>
<br>_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@cs.uiuc.edu" target="_blank">cfe-dev@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev</a><br>
<br></blockquote></div></div><br></div></div>
<br>_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@cs.uiuc.edu" target="_blank">cfe-dev@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev</a><br>
<br></blockquote></div></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>