[cfe-commits] A patch for PPCallbacks::If() & Elif() functions to fix what the FIXME says.

[Wei]

Hi, Douglas,

How could I re-lex the tokens? Does clang store some state information after
lexing tokens? if so, simply set the Lexer::BufferPtr to the position of #
is not enough.

My current though is to still use the clang Lex mechanism
1) In Preprocessor::HandleIfDirective(), first call
Preprocessor::EnableBacktrackAtThisPos().
2) In PPCallbacks::If(), call Preprocessor::Backtrack(), so that we can use
the lexed tokens, and don't need to worry about the correct state of
re-lexing.

How about your idea? I want to sync with u for the design before I start the
coding again :)

Wei Hu
http://code.google.com/u/wei.hu.tw/
http://www.csie.ntu.edu.tw/~r88052/



2011/4/21 Douglas Gregor <dgregor at apple.com>

> Hello Wei,
>
> On Apr 20, 2011, at 7:09 AM, Wei wrote:
>
> > Hi, all,
> >
> > I have made a patch to make clang::PPCallbacks::If() &
> clang::PPCallbacks::Elif() can pass-back a token tree.
> > That's what the original FIXME of these 2 functions said: "better to pass
> in a list (or tree!) of Tokens."
> >
> > This patch has passed all the expected tests in the clang-test. (I use
> Visual C++ 2010 Express as my dev. environment)
> > 1>  Testing Time: 738.14s
> > 1>    Expected Passes    : 2877
> > 1>    Expected Failures  : 26
> > 1>    Unsupported Tests  : 5
> >
> > I hope this patch can be merged into clang main trunk, and if you have
> any problems about this patch, please let me know, I'll modify it so that it
> can be merged.
>
> The preprocessor is an extremely performance-sensitive part of Clang, so we
> need to be very careful about what computation we perform. Building a
> tree-like data structure such as
>
> @@ -264,6 +266,57 @@
>
>  };
>
> +class TokenTreeNode {
> + private:
> +  llvm::BumpPtrAllocator *Allocator;
> +  Token Self;
> +
> +  typedef llvm::SmallVector<TokenTreeNode*,3> ChildrenTy;
> +  typedef ChildrenTy::iterator iterator;
> +
> +  ChildrenTy Children;
> +
> + public:
> +  TokenTreeNode(Token &token, llvm::BumpPtrAllocator *allocator) :
> +      Allocator(allocator),
> +      Self(token) {
> +    assert(Allocator != 0);
> +  }
> +
> +  ~TokenTreeNode() {
> +    for (iterator iter = Children.begin(); iter != Children.end(); ++iter)
> {
> +      (*iter)->~TokenTreeNode();
> +      Allocator->Deallocate(*iter);
> +    }
> +  }
> +
> +  inline ChildrenTy &getChildren() {
> +    return Children;
> +  }
> +
> +  void addChildren(iterator I, Token &T) {
> +    TokenTreeNode *Node = Allocator->Allocate<TokenTreeNode>();
> +    new (Node) TokenTreeNode(T, Allocator);
> +    Children.insert(I, Node);
> +  }
> +
> +  void addChildren(iterator I, TokenTreeNode *N) {
> +    assert(N != 0);
> +    Children.insert(I, N);
> +  }
> +
> +  inline void addChildren(Token &T) {
> +    addChildren(Children.end(), T);
> +  }
> +
> +  inline void addChildren(TokenTreeNode *N) {
> +    assert(N != 0);
> +    addChildren(Children.end(), N);
> +  }
> +
> +  void dump(unsigned IndentLevel);
> +};
>
> is likely to incur an unacceptable performance penalty. It's probably
> reasonable to pass the SourceRange of the expression down to If/ElIf, from
> which one could re-lex the tokens if desired.
>
>        - Doug
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20110526/f2dfb20a/attachment.html>