[PATCH] D158413: [Lex] Introduce Preprocessor::LexAll()

[Aaron Ballman via Phabricator via cfe-commits]

aaron.ballman added inline comments.


================
Comment at: clang/lib/Lex/Preprocessor.cpp:998
 
+std::vector<Token> Preprocessor::LexAll() {
+  std::vector<Token> toks;
----------------
v.g.vassilev wrote:
> Shouldn't we take the results as a `LexAll(std::vector<Token> &result)`?
> 
> Perhaps we should rename this interface to `LexTokensUntilEOF`?
I think both suggestions make sense, especially given this doesn't return the EOF token (so it doesn't really lex *all*). Though, given how many callers do not want to store the tokens, perhaps it should take a `std::optional` or a pointer to the vector so we can skip the storage work if it's unnecessary?


================
Comment at: clang/lib/Lex/Preprocessor.cpp:1000
+  std::vector<Token> toks;
+  while (1) {
+    Token tok;
----------------
I'd prefer not to assume the token stream has an EOF token (perhaps the stream is one only being used to parse until the `eod` token instead), so if we can turn this into a non-infinite loop, that would make me more comfortable.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D158413/new/

https://reviews.llvm.org/D158413