[clang-tools-extra] [clang-tidy] Add ranges-style view for tokenizing source code (PR #172508)

[Baranov Victor via cfe-commits]

================
@@ -127,6 +129,78 @@ SourceLocation getUnifiedEndLoc(const Stmt &S, const SourceManager &SM,
 SourceLocation getLocationForNoexceptSpecifier(const FunctionDecl *FuncDecl,
                                                const SourceManager &SM);
 
+class TokenView {
----------------
vbvictor wrote:

Could we please add tests for this (if it is possibe). I want to beleave that there are some unit tests for `Lexer`

https://github.com/llvm/llvm-project/pull/172508