[PATCH] D152416: add -dump-tokens-bg option for clang

[Anarion via Phabricator via cfe-commits]

Anarion-zuo added a comment.

We intend to build a deep learning model with code as input, then noticed clang does not currently have a convenient way of dumping tokens in huge projects.

The dumped tokens are supposed to be converted to vectors, then given to a deep learning model.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D152416/new/

https://reviews.llvm.org/D152416