[PATCH] D148997: [clang] Add a new annotation token: annot_repl_input_end

[Richard Smith - zygoloid via Phabricator via cfe-commits]

rsmith added a comment.

This looks like a nice improvement to me.



================
Comment at: clang/lib/Interpreter/IncrementalParser.cpp:162
+  if (P->getCurToken().is(tok::annot_repl_input_end)) {
+    P->ConsumeAnyToken();
     // FIXME: Clang does not call ExitScope on finalizing the regular TU, we
----------------



================
Comment at: clang/lib/Interpreter/IncrementalParser.cpp:254-268
   if (PP.getLangOpts().DelayedTemplateParsing) {
     // Microsoft-specific:
     // Late parsed templates can leave unswallowed "macro"-like tokens.
     // They will seriously confuse the Parser when entering the next
     // source file. So lex until we are EOF.
     Token Tok;
     do {
----------------
Not really related to this patch, but in `DelayedTemplateParsing` mode this code appears to expect two `annot_repl_input_end` tokens in a row.


================
Comment at: clang/lib/Parse/ParseStmt.cpp:546-556
+  Token *CurTok = nullptr;
+  // If we're parsing an ExprStmt and the last semicolon is missing and the
+  // incremental externsion is enabled and we're reaching the end, consider we
+  // want to do value printing. Note this is only enable in C++ mode
+  // since part of the implementation requires C++ language features.
+  //
+  // Note we shouldn't eat the token since the callback need it.
----------------
I don't think we need to check whether incremental processing is enabled here -- if not, we shouldn't see the "end of REPL input" token at all.


================
Comment at: clang/lib/Parse/Parser.cpp:620-621
   // processing
-  if (PP.isIncrementalProcessingEnabled() && Tok.is(tok::eof))
-    ConsumeToken();
+  if (PP.isIncrementalProcessingEnabled() && Tok.is(tok::annot_repl_input_end))
+    ConsumeAnnotationToken();
 
----------------
Do we need to do this here? `IncrementalParser` already seems to take care of this, and the logic here would be easier to reason about if `Parser` never steps past an `annot_repl_input_end` token, and such tokens instead are only ever consumed by the REPL.

Are there other users of incremental processing mode, other than the REPL / `IncrementalParser`?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D148997/new/

https://reviews.llvm.org/D148997