[PATCH] D59887: [Syntax] Introduce TokenBuffer, start clangToolingSyntax library

[Ilya Biryukov via Phabricator via cfe-commits]

ilya-biryukov marked 2 inline comments as done.
ilya-biryukov added inline comments.


================
Comment at: include/clang/Tooling/Syntax/Tokens.h:206
+  ///     DECL(a);
+  /// spelledTokens() returns {"#", "define", "DECL", "(", "name", ")", "eof"}.
+  /// FIXME: we do not yet store tokens of directives, like #include, #define,
----------------
ilya-biryukov wrote:
> nridge wrote:
> > Is this comment correct?
> > 
> > If so:
> >   * Why are the tokens "int", "name", "=", "10" not included?
> >   * Why are the tokens `"DECL", "(", "a", ")" not included?
> > 
> It isn't. Thanks for catching that.
> This patch went through so many revisions, it was close to impossible to track this down.
Also:
- we do not store 'eof' in the spelled tokens anymore
- the FIXME is stale, we do store tokens of macro directives now

759c90456d418ffe69e1a2b4bcea2792491a6b5a updates the comment.


Repository:
  rC Clang

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D59887/new/

https://reviews.llvm.org/D59887