[PATCH] D128411: [syntax] Introduce a BaseToken class.

Ilya Biryukov via Phabricator via cfe-commits cfe-commits at lists.llvm.org
Fri Jun 24 06:15:55 PDT 2022

ilya-biryukov added inline comments.

Comment at: clang/include/clang/Tooling/Syntax/TokenManager.h:33
+  /// The syntax-tree Leaf node holds a Key.
+  using Key = const void *;
+  /// Gets the text of token identified by the key.
I have just realized that we were discussing having opaque index as a key, but there may also be tokens in the syntax tree that are not backed by the `TokenBuffer`.
Those can be synthesized, e.g. imagine someone wants to change `!(A && B)` to `!A || !B`. They will need to synthesize at least the `||` token as it's not in the source code. There is a way to do this now and it prohibits the use of an index to the `TokenBuffer`.

So having the opaque pointer is probably ok for now, it should enable the pseudo-parser to build syntax trees.
We might want to add an operation to synthesize tokens into the `TokenManager` at some point, but that's a discussion for another day.

  rG LLVM Github Monorepo



More information about the cfe-commits mailing list