[all-commits] [llvm/llvm-project] 7c1ee5: [Pseudo] Token/TokenStream, PP directive parser.

[Sam McCall via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 7c1ee5e95f3159e13edef644db0509a7d49921c3
      https://github.com/llvm/llvm-project/commit/7c1ee5e95f3159e13edef644db0509a7d49921c3
  Author: Sam McCall <sam.mccall at gmail.com>
  Date:   2022-02-23 (Wed, 23 Feb 2022)

  Changed paths:
    M clang/include/clang/Basic/TokenKinds.h
    A clang/include/clang/Tooling/Syntax/Pseudo/Preprocess.h
    A clang/include/clang/Tooling/Syntax/Pseudo/Token.h
    M clang/lib/Basic/TokenKinds.cpp
    M clang/lib/Tooling/Syntax/Pseudo/CMakeLists.txt
    A clang/lib/Tooling/Syntax/Pseudo/Lex.cpp
    A clang/lib/Tooling/Syntax/Pseudo/Preprocess.cpp
    A clang/lib/Tooling/Syntax/Pseudo/Token.cpp
    A clang/test/Syntax/lex.c
    M clang/tools/clang-pseudo/ClangPseudo.cpp
    M clang/unittests/Tooling/Syntax/Pseudo/CMakeLists.txt
    A clang/unittests/Tooling/Syntax/Pseudo/PreprocessTest.cpp
    A clang/unittests/Tooling/Syntax/Pseudo/TokenTest.cpp

  Log Message:
  -----------
  [Pseudo] Token/TokenStream, PP directive parser.

The TokenStream class is the representation of the source code that will
be fed into the GLR parser.

This patch allows a "raw" TokenStream to be built by reading source code.
It also supports scanning a TokenStream to find the directive structure.

Next steps (with placeholders in the code): heuristically choosing a
path through #ifs, preprocessing the code by stripping directives and comments.
These will produce a suitable stream to feed into the parser proper.

Differential Revision: https://reviews.llvm.org/D119162