[PATCH] D158413: [Lex] Introduce Preprocessor::LexTokensUntilEOF()

[Aaron Ballman via Phabricator via cfe-commits]

aaron.ballman added inline comments.


================
Comment at: clang/lib/Lex/Preprocessor.cpp:1000
+  std::vector<Token> toks;
+  while (1) {
+    Token tok;
----------------
v.g.vassilev wrote:
> aaron.ballman wrote:
> > Hahnfeld wrote:
> > > aaron.ballman wrote:
> > > > I'd prefer not to assume the token stream has an EOF token (perhaps the stream is one only being used to parse until the `eod` token instead), so if we can turn this into a non-infinite loop, that would make me more comfortable.
> > > I'm not sure I understand entirely. Do you want something like
> > > ```
> > > tok.isOneOf(tok::unknown, tok::eof, tok::eod)
> > > ```
> > > instead of `tok.is(tok::eof)`? Can this happen at the level of the `Preprocessor`?
> > I was thinking something more along the lines of:
> > ```
> > if (Tokens) {
> >   for (Token Tok; !Tok.isOneOf(tok::eof, tok::eod); Lex(Tok))
> >     Tokens->push_back(Tok);
> > }
> > ```
> > but I hadn't thought about `tok::unknown`; that might be a good one to also include given that clangd operates on partial sources.
> > 
> I was wondering if we could somehow merge this routine with `Parser::SkipUntil` since they seem to be doing a very similar tasks.
That could perhaps end up looking reasonable (they do similar tasks aside from collecting the tokens that are being skipped). Do you need the interface to be on `Preprocessor` or `Parser` though (or does it not really matter for you)?


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D158413/new/

https://reviews.llvm.org/D158413