[PATCH] D74962: [clang][Tooling] Add a way to tokenize a FileRange

[Sam McCall via Phabricator via cfe-commits]

sammccall added inline comments.


================
Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:335
   clang::Token T;
-  while (!L.LexFromRawLexer(T))
+  while (!L.LexFromRawLexer(T) && L.getCurrentBufferOffset() < FR.endOffset())
     AddToken(T);
----------------
Discussed offline, this loop includes an extra token if the truncation is at whitespace between tokens. (Please test this case)

Also the eof comment is confusing.

I think the loop should be rewritten.


================
Comment at: clang/lib/Tooling/Syntax/Tokens.cpp:346
+                                            const LangOptions &LO) {
+  return tokenize(syntax::FileRange(SM, SM.getLocForStartOfFile(FID),
+                                    SM.getLocForEndOfFile(FID)),
----------------
nit: FileRange(FID, 0, SM.getFileIDSize(FID)) is a lot more direct :-)


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D74962/new/

https://reviews.llvm.org/D74962