[PATCH] D119162: [Pseudo] Token/TokenStream, PP directive parser.

[Sam McCall via Phabricator via cfe-commits]

sammccall created this revision.
sammccall added a reviewer: hokein.
Herald added a subscriber: mgorny.
sammccall requested review of this revision.
Herald added a project: clang.
Herald added a subscriber: cfe-commits.

The TokenStream class is the representation of the source code that will
be fed into the GLR parser.

This patch allows a "raw" TokenStream to be built by reading source code.
It also supports scanning a TokenStream to find the directive structure.

Next steps (with placeholders in the code): heuristically choosing a
path through #ifs, preprocessing the code by stripping directives and
comments, cooking raw_identifiers.
These will produce a suitable stream to feed into the parser proper.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D119162

Files:
  clang/include/clang/Tooling/Syntax/Pseudo/Preprocess.h
  clang/include/clang/Tooling/Syntax/Pseudo/Token.h
  clang/lib/Tooling/Syntax/Pseudo/CMakeLists.txt
  clang/lib/Tooling/Syntax/Pseudo/Lex.cpp
  clang/lib/Tooling/Syntax/Pseudo/Preprocess.cpp
  clang/lib/Tooling/Syntax/Pseudo/Token.cpp
  clang/test/Syntax/Inputs/example.c
  clang/test/Syntax/lex.test
  clang/tools/clang-pseudo/ClangPseudo.cpp

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D119162.406520.patch
Type: text/x-patch
Size: 30239 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20220207/596314d4/attachment-0001.bin>