[PATCH] D143144: [clang][lex] Add TryGrowLexerBuffer/SourceFileGrower

[Sunho Kim via Phabricator via cfe-commits]

sunho created this revision.
Herald added a project: All.
sunho retitled this revision from "asdfasdf" to "[clang][lex] Add TryExpandBuffer callback".
sunho edited the summary of this revision.
sunho updated this revision to Diff 494978.
sunho added a comment.
sunho updated this revision to Diff 495037.
sunho updated this revision to Diff 495039.
sunho retitled this revision from "[clang][lex] Add TryExpandBuffer callback" to "[clang][lex] Add TryGrowLexerBuffer/SourceFileGrower".
sunho edited the summary of this revision.
sunho edited the summary of this revision.
sunho edited the summary of this revision.
sunho edited the summary of this revision.
sunho published this revision for review.
Herald added a project: clang.
Herald added a subscriber: cfe-commits.

Update


sunho added a comment.

Update


sunho added a comment.

Update


Add TryGrowLexerBuffer to Lexer and SourceFileGrower to Preprocessor that can be used to grow the source code buffer when Lexer reaches the eof of the buffer.

Since clang-repl receive source code incrementally line by line, we need to grow source code buffer. This change adds the interfaces to Lexer/Preprocessor in order to accommodate this need. If SourceFileGrower interface is registered to Preprocessor, it will call TryGrowFile method when Lexer reached eof. Inside this method, the user can grow the file and return true to request Lexer to continue the lexing from the last point.

When Lexer reaches eof it will call TryGrowLexerBuffer callback specified in the constructor to try growing buffer and if it got new buffer, it will continue the lexing from the last point. Preprocessor registers TryGrowLexerBuffer callback in order to implement handling of SourceFileGrower.

Note that practically all the code changes will not affect the AOT clang world since they are all disabled when Preprocessor doesn't have SourceFileGrower instance.

NOTE: This is part 2 and 3 of https://discourse.llvm.org/t/rfc-flexible-lexer-buffering-for-handling-incomplete-input-in-interactive-c-c/64180


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D143144

Files:
  clang/include/clang/Lex/Lexer.h
  clang/include/clang/Lex/Preprocessor.h
  clang/lib/Lex/Lexer.cpp
  clang/lib/Lex/PPLexerChange.cpp
  clang/lib/Lex/Preprocessor.cpp
  clang/unittests/Lex/LexerTest.cpp

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D143144.495039.patch
Type: text/x-patch
Size: 22598 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20230206/812e6d19/attachment-0001.bin>