<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Mon, Feb 6, 2017 at 6:18 PM Gaetano Checinski <<a href="mailto:gaetano.checinski@gmail.com">gaetano.checinski@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg">Your comment is very surprising to me. <br class="gmail_msg"></div><div class="gmail_msg"><div class="gmail_msg">The points you make sound to me like advantages not disadvantages:<br class="gmail_msg"><br class="gmail_msg"></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg">> <span style="font-size:12.8px" class="gmail_msg">C++ AST can be created that is both useful and not overly specific to a single compiler.</span></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg">Maybe, but what more important is an AST you can build, analyse and transform easily.</span></div></div></blockquote><div><br></div><div>Well, "useful" seems like the most important part to me :)</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg">> <span style="font-size:12.8px" class="gmail_msg">The IPR does not handle macros</span><span style="font-size:12.8px" class="gmail_msg"> [...]</span></div><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg">Why should it ? - what would it be good for, besides having source-locations ? - how is this currently handled in clang ?</span></div><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg"> As far as i know clang's AST has no notion of macros.</span></div></div></blockquote><div><br></div><div>Having full information about what source locations each part of the AST was generated from would be enough; that's what clang provides. Transformations for C++ code need to take that into account, though.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_msg">I imagine it could be very handy to have multiple ASTs:</div><div class="gmail_msg">One having only nodes for Preprocessor (ppAST) constructs and one for C++ (cppAST).</div><div class="gmail_msg">The Preprocessor could process ppAST and return cppAST alongside with a sourcemap.</div><div class="gmail_msg">We could even go a step further and have separate ASTs for C++ with and without templates.</div></div><div class="gmail_msg">This would make the codebase more functional and composable.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">To get a proper source-location we then just need to follow the sourcemaps.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Dealing with real (immutable) ASTs would make (de-)serialization, cloning and comparing an easy task.</div><div class="gmail_msg">As far as i can see, the preprocessor is a big liability:</div><div class="gmail_msg">The preprocessor is stateful and macros can transform the sourcefile in almost any imaginable way.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">As far as i can see the c++-parser could be paralyzed. </div><div class="gmail_msg">However this is only possible if the parser is decoupled from the preprocessor.</div></div></blockquote><div><br></div><div>Preprocessing is fast; parallelization is made hard by C++ (the language), not by the preprocessor.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg"></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div style="font-size:12.8px" class="gmail_msg">> - does not mimic C++ language irregularities; general rules are used, rather than long lists of special cases</div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div style="font-size:12.8px" class="gmail_msg">We still can write an validator to be sure the AST conforms to a specific c++-standard.</div><div style="font-size:12.8px" class="gmail_msg">I think that having a nice and regular AST would simplify working with the AST. </div><div style="font-size:12.8px" class="gmail_msg">Writing visitors and pattern matchers for semantic analysis might become easier. </div></div></div></blockquote><div><br></div><div>Yes, the question is what you do when you want to match some arcane detail of the language, and the information is not there?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div style="font-size:12.8px" class="gmail_msg">> <span style="font-size:12.8px" class="gmail_msg">Generally, I would not trust a representation that I can't generate code from to be correct enough for tools.</span></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div style="font-size:12.8px" class="gmail_msg">Well, my goal would definitely be to generate code from the AST. If information are missing in the representation then we need to improve it.</div><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg">IMHO the main message of the paper is that there is that you can build an AST representation that is more regular and still complete.</span></div></div></div></blockquote><div><br></div><div>So, if somebody builds it and maintains it that's great (I'd find that very cool), but it sound like a rather large project :)</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg">> <span style="font-size:12.8px" class="gmail_msg">- Unfortunately, a program cannot fully automate the generation of “skeletons.” If our aim is portability, we still need to (by hand) eliminate non-standard additions to the contents of header file.</span></div><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg"><br class="gmail_msg"></span></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg">can you elaborate? - what do you mean by "skeletons" and to which header files are you referring to?</span></div></div></blockquote><div><br></div><div>It means that the authors admit that their representation cannot be generated for system headers, so manual work to model system headers is necessary. This is problematic at scale, unless I'm misunderstanding something.</div><div><br></div><div>Don't get me wrong, I'd love it if it was possible to create a simplified AST for C++ that's still useful (-> efficient to generate, memory efficient, has all necessary information).</div><div>Somebody has to prove that it's possible by writing it, I guess. Time will tell. Given the information I have today, I'm not holding my breath, and it seems like there are higher priority things to spend time on.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg">2017-02-06 15:45 GMT+00:00 Manuel Klimek <span dir="ltr" class="gmail_msg"><<a href="mailto:klimek@google.com" class="gmail_msg" target="_blank">klimek@google.com</a>></span>:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail_msg">I haven't looked too deeply into it, but from talking to various clang developers, the common theme is disbelieve that a high-level C++ AST can be created that is both useful and not overly specific to a single compiler.<div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">From the paper referenced in the project, I see multiple things that make it seem not interesting to me from a point of refactoring and semantic analysis:</div><div class="gmail_msg">- The IPR does not handle macros before their expansion in the preprocessor.<br class="gmail_msg"></div><div class="gmail_msg">- does not mimic C++ language irregularities; general rules are used, rather than long lists of special cases</div><div class="gmail_msg">- Unfortunately, a program cannot fully automate the generation of “skeletons.” If our aim is portability, we still need to (by hand) eliminate non-standard additions to the contents of header file.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Generally, I would not trust a representation that I can't generate code from to be correct enough for tools.</div></div><div class="m_-7709448869939002193gmail-m_-3548957017129714506HOEnZb gmail_msg"><div class="m_-7709448869939002193gmail-m_-3548957017129714506h5 gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Wed, Feb 1, 2017 at 8:32 PM Reid Kleckner via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" class="gmail_msg" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg">Yes, the clang "abstract syntax tree" is often jokingly referred to as the "concrete syntax graph". We try to provide a generally useful representation, but being a fast production C++ compiler comes first. Clang's AST is very concrete. You have to know a lot about it to navigate it. There is no "Node" base class that you can use as a cursor to navigate around Decls, Exprs, Types, and TemplateArguments. Template instantiation, the closest thing I can think of to cloning, is done relatively manually with TreeTransform.<div class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg"><br class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg"></div><div class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg">I think it would be nice to revisit the design of clang's AST to simplify it, normalize it, and abstract it, but it is not a task to be taken lightly, and I don't expect it to happen in the near future.</div></div><div class="gmail_extra m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg"><br class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg"><div class="gmail_quote m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg">On Wed, Feb 1, 2017 at 8:44 AM, Gaetano Checinski via cfe-dev <span dir="ltr" class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg"><<a href="mailto:cfe-dev@lists.llvm.org" class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg" target="_blank">cfe-dev@lists.llvm.org</a>></span> wrote:<br class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg"><blockquote class="gmail_quote m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg">As the AST is not really a Tree as it seems to have circular references, working with the AST is sometimes a bit messy (eg. cloning).<div class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg"><br class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg"></div><div class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg">A while ago Stroustroup pointed me to Gabriel Dos Reis' work on a different approach to represent C++-AST: <a href="https://mailtrack.io/trace/link/a5c184fad2cf94fbf0449fa233263072ebc34ea8?url=https%3A%2F%2Fgithub.com%2FGabrielDosReis%2Fipr&signature=9a19f2a0a41d5ceb" class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg" target="_blank">https://github.com/GabrielDosReis/ipr</a></div><div class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg"><br class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg"></div><div class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg">Did anyone try to integrate his work into clang or has an opinion to share ?<br class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg"></div><img width="0" height="0" class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520m_-1034651269622219800m_-8147391716180187190mailtrack-img m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg"></div>
<br class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg">_______________________________________________<br class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg">
cfe-dev mailing list<br class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg">
<a href="mailto:cfe-dev@lists.llvm.org" class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg" target="_blank">cfe-dev@lists.llvm.org</a><br class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg">
<br class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg"></blockquote></div><br class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg"></div>
_______________________________________________<br class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg">
cfe-dev mailing list<br class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg">
<a href="mailto:cfe-dev@lists.llvm.org" class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg" target="_blank">cfe-dev@lists.llvm.org</a><br class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br class="m_-7709448869939002193gmail-m_-3548957017129714506m_-5452629544365183520gmail_msg gmail_msg">
</blockquote></div>
</div></div></blockquote></div><br class="gmail_msg"></div><span class="m_-7709448869939002193gmail-mt-tool-email-notification-true gmail_msg"></span><span class="m_-7709448869939002193gmail-mt-tool-email-tracking-true gmail_msg"></span><br class="gmail_msg"><br class="gmail_msg"><br class="gmail_msg"></div><div dir="ltr" class="gmail_msg"><div class="m_-7709448869939002193mt-signature gmail_msg"><a href="https://mailtrack.io/trace/link/29fe4deb755f2a860d95181bcf80131f16c45e93?url=https%3A%2F%2Fmailtrack.io%2F&signature=d1df2a3c7469f8d7" class="m_-7709448869939002193gmail-mt-signature-logo gmail_msg" style="text-decoration:none" target="_blank"><img src="https://s3-eu-west-1.amazonaws.com/mailtrack-crx/icon-signature.png" width="16" height="14" class="gmail_msg"> </a><font color="#999999" class="m_-7709448869939002193mt-signature-text gmail_msg">Sent with <a href="https://mailtrack.io/install?source=signature&lang=en&referral=gaetano.checinski@gmail.com&idSignature=22" class="m_-7709448869939002193mt-install gmail_msg" target="_blank">Mailtrack</a></font></div><img width="0" height="0" class="m_-7709448869939002193mailtrack-img gmail_msg" src="https://mailtrack.io/trace/mail/8ba1b8824c012f875bf1a67de8774b902460c2d5.png?u=931501"></div>
</blockquote></div></div>