[PATCH] D61637: [Syntax] Introduce syntax trees

[Ilya Biryukov via Phabricator via cfe-commits]

ilya-biryukov added inline comments.


================
Comment at: clang/include/clang/Tooling/Syntax/Corpus.h:38
+  std::pair<FileID, llvm::ArrayRef<syntax::Token>>
+  tokenizeBuffer(std::unique_ptr<llvm::MemoryBuffer> Buffer);
+
----------------
sammccall wrote:
> ilya-biryukov wrote:
> > sammccall wrote:
> > > Are you planning to have a way to add tokens directly? Having to turn them into text and re-lex them seems like it might be inconvenient.
> > The tokens have source locations and refer to a text in some buffer. `tokenizeBuffer` makes is easier, not harder, to mock tokens.
> Fair enough.
> 
> `lexBuffer` might be a slightly clearer name?
> 
> Who are the intended non-test users of this function? Are they better served by being able (and responsible) for constructing a MemoryBuffer with e.g. a sensible name and ownership, or would it be better to pass a StringRef and have the Arena come up with a sensible "anonymous" name?
The only use-case in my prototype so far is creating token nodes for punctuation nodes, e.g. say you want to create an expr of the form `<a>+<b>`, where both `a` and `b` are existing expressions and you need to synthesize a leaf node for `+`.
We use this function to synthesize a buffer with the corresponding token.

All the use-cases I can imagine are around synthesizing syntax trees (as opposed to constructing them from the AST).


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D61637/new/

https://reviews.llvm.org/D61637