[PATCH] D74962: [clang][Tooling] Add a way to tokenize a FileRange

[Kadir Cetinkaya via Phabricator via cfe-commits]

kadircet marked 2 inline comments as done.
kadircet added inline comments.


================
Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:335
   clang::Token T;
-  while (!L.LexFromRawLexer(T))
+  while (!L.LexFromRawLexer(T) && L.getCurrentBufferOffset() < FR.endOffset())
     AddToken(T);
----------------
sammccall wrote:
> Discussed offline, this loop includes an extra token if the truncation is at whitespace between tokens. (Please test this case)
> 
> Also the eof comment is confusing.
> 
> I think the loop should be rewritten.
instead of re-writing the loop I just changed the after-the-loop check to verify latest lexed token to be in range.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D74962/new/

https://reviews.llvm.org/D74962