[clang] [clang] Make -dump-tokens option align tokens (PR #164894)

[via cfe-commits]

alexpaniman wrote:

Hi! Apologies for the delay in response.

I took into account all comments and did a few adjustments:
1. The test now uses `--strict-whitespace` to verify the exact formatting.
2. The test now includes examples of annotations, and the implementation now includes a comment explaining them.
3. I now escape token spelling to prevent multiline tokens from breaking formatting (for example, the test includes an `eod` token; its spelled as a newline that is now escaped as `'\n'`)
4. I added dynamic padding logic to align the trailing markers (like `[StartOfLine]`) for tokens originating in the same file. In the same files, because I pad every line as if it was at line 9999 and column 99 - it keeps the implementation simple and stateless, while still making most of the trailing markers in majority of files aligned. See the comment in implementation for more details.

This is how `-Xclang -dump-tokens` of the updated test looks like:
<img width="1926" height="865" alt="image" src="https://github.com/user-attachments/assets/4c91e0ee-9d5e-4bbc-9e79-846648000b9f" />

I'd appreciate a review, @AaronBallman! Do you think this can be merged? Thanks.



https://github.com/llvm/llvm-project/pull/164894