[clang] [clang] Make -dump-tokens option align tokens (PR #164894)

[Aaron Ballman via cfe-commits]

================
@@ -240,14 +241,59 @@ void Preprocessor::FinalizeForModelFile() {
 }
 
 void Preprocessor::DumpToken(const Token &Tok, bool DumpFlags) const {
-  llvm::errs() << tok::getTokenName(Tok.getKind());
+  std::string TokenStr;
+  llvm::raw_string_ostream OS(TokenStr);
+
+  // The alignment of 16 is chosen to comfortably fit most identifiers.
+  OS << llvm::formatv("{0,-16} ", tok::getTokenName(Tok.getKind()));
+
+  // Annotation tokens are just markers that don't have a spelling -- they
----------------
AaronBallman wrote:

I think that's reasonable, but it's also existing behavior: https://godbolt.org/z/YoM9zr843 so I think it's fine in a follow-up too if that's preferred.

https://github.com/llvm/llvm-project/pull/164894