[PATCH] D158413: [Lex] Introduce Preprocessor::LexTokensUntilEOF()

[Aaron Ballman via Phabricator via cfe-commits]

aaron.ballman added inline comments.


================
Comment at: clang/lib/Lex/Preprocessor.cpp:1000
+  std::vector<Token> toks;
+  while (1) {
+    Token tok;
----------------
Hahnfeld wrote:
> aaron.ballman wrote:
> > I'd prefer not to assume the token stream has an EOF token (perhaps the stream is one only being used to parse until the `eod` token instead), so if we can turn this into a non-infinite loop, that would make me more comfortable.
> I'm not sure I understand entirely. Do you want something like
> ```
> tok.isOneOf(tok::unknown, tok::eof, tok::eod)
> ```
> instead of `tok.is(tok::eof)`? Can this happen at the level of the `Preprocessor`?
I was thinking something more along the lines of:
```
if (Tokens) {
  for (Token Tok; !Tok.isOneOf(tok::eof, tok::eod); Lex(Tok))
    Tokens->push_back(Tok);
}
```
but I hadn't thought about `tok::unknown`; that might be a good one to also include given that clangd operates on partial sources.



CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D158413/new/

https://reviews.llvm.org/D158413