[PATCH] D148997: [clang] Add a new annotation token: annot_repl_input_end

[Aaron Ballman via Phabricator via cfe-commits]

aaron.ballman added a comment.

In D148997#4561620 <https://reviews.llvm.org/D148997#4561620>, @v.g.vassilev wrote:

> In D148997#4561614 <https://reviews.llvm.org/D148997#4561614>, @bnbarham wrote:
>
>> In D148997#4561577 <https://reviews.llvm.org/D148997#4561577>, @v.g.vassilev wrote:
>>
>>>> My other concern here is that it seems our use cases are somewhat different, eg. we wouldn't want any parsing differences and while I don't know why this is yet, the removal of:
>>>
>>> I think I understand now. We basically want a mode which keeps some of the clang objects alive and ready to process more input. And on top, for clang-repl we want to enable some special parsing mode for the top-level statement declarations.
>>
>> Yeah, I'd say that sums it up pretty well.
>>
>>>>   // Skip over the EOF token, flagging end of previous input for incremental
>>>>   // processing
>>>>   if (PP.isIncrementalProcessingEnabled() && Tok.is(tok::eof))
>>>>     ConsumeToken();
>>>>
>>>> is causing issues for us as well.
>>>
>>> Yes, that now seems to be moved on the user side as we thought it would be dead code. We can bring that one back if that's the only issue you see with that approach.
>>
>> I think it would make sense to bring it back under the mode you spoke about above, yeah 👍
>
> So, in that case we should bring back the boolean flag for incremental processing and keep the `IncrementalExtensions` LanguageOption separate. In that case `IncrementalExtensions` would mean that we must turn on incremental processing for managing lifetime and only use the language option when extending the parsing logic. However, I think the problem would be what to do with the `tok::eof` and `tok::annot_repl_input_end`? I'd probably need @aaron.ballman or @rsmith here...

I missed this ping earlier in the month, sorry for that! My intuition is that this is handled by `isEofOrEom()` as done in this patch -- if incremental extensions is not enabled, then we wouldn't see an `annot_repl_input_end` token in the first place, right?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D148997/new/

https://reviews.llvm.org/D148997