Allow peeking forward several tokens with MCAsmLexer

[Benjamin Kramer via llvm-commits]

> On 16.08.2015, at 16:53, Dylan McKay <dylanmckay34 at gmail.com> wrote:
> 
> I fixed everything you mentioned Benjamin - thanks for the review!
> 
> Attached is the new patch.

You don't need the size_t Count argument anymore, the MutableArrayRef already carries the size.

- Ben

> 
> On Sun, Aug 16, 2015 at 10:57 PM, Benjamin Kramer <benny.kra at gmail.com> wrote:
> 
> > On 15.08.2015, at 16:23, Dylan McKay via llvm-commits <llvm-commits at lists.llvm.org> wrote:
> >
> > From b7094fa637ffd59b10776e0a4bbf7f8c04235306 Mon Sep 17 00:00:00 2001
> > From: Dylan McKay <dylanmckay34 at gmail.com>
> > Date: Sun, 16 Aug 2015 02:09:05 +1200
> > Subject: [PATCH] Extend MCAsmLexer so that it can peek forward several tokens
> >
> > This commit adds a virtual `peekTokens()` function to `MCAsmLexer`
> > which can peek forward an arbitrary number of tokens.
> >
> > It also makes the `peekTok()` method call `peekTokens()` method, but
> > only requesting one token.
> >
> > The idea is to better support targets which more more ambiguous
> > assembly syntaxes.
> > ---
> >  include/llvm/MC/MCParser/AsmLexer.h   |  3 ++-
> >  include/llvm/MC/MCParser/MCAsmLexer.h | 13 ++++++++++++-
> >  lib/MC/MCParser/AsmLexer.cpp          | 16 +++++++++++++---
> >  3 files changed, 27 insertions(+), 5 deletions(-)
> >
> > diff --git a/include/llvm/MC/MCParser/AsmLexer.h b/include/llvm/MC/MCParser/AsmLexer.h
> > index 62d39b2..77023e4 100644
> > --- a/include/llvm/MC/MCParser/AsmLexer.h
> > +++ b/include/llvm/MC/MCParser/AsmLexer.h
> > @@ -47,7 +47,8 @@ public:
> >    StringRef LexUntilEndOfStatement() override;
> >    StringRef LexUntilEndOfLine();
> >
> > -  const AsmToken peekTok(bool ShouldSkipSpace = true) override;
> > +  size_t peekTokens(AsmToken *Buf, size_t Size,
> > +                    bool ShouldSkipSpace = true) override;
> 
> This could be a MutableArrayRef<AsmToken> instead of a pointer+size pair.
> 
> >
> >    bool isAtStartOfComment(const char *Ptr);
> >    bool isAtStatementSeparator(const char *Ptr);
> > diff --git a/include/llvm/MC/MCParser/MCAsmLexer.h b/include/llvm/MC/MCParser/MCAsmLexer.h
> > index 71f15b3..fb9d32d 100644
> > --- a/include/llvm/MC/MCParser/MCAsmLexer.h
> > +++ b/include/llvm/MC/MCParser/MCAsmLexer.h
> > @@ -162,7 +162,18 @@ public:
> >    }
> >
> >    /// Look ahead at the next token to be lexed.
> > -  virtual const AsmToken peekTok(bool ShouldSkipSpace = true) = 0;
> > +  const AsmToken peekTok(bool ShouldSkipSpace = true) {
> > +    AsmToken Buf;
> > +
> > +    size_t ReadCount = peekTokens(&Buf, 1, ShouldSkipSpace);
> > +    assert(ReadCount == 1);
> 
> Add (void)ReadCount; here to avoid unused variable warnings when assertions are disabled.
> 
> > +
> > +    return Buf;
> > +  }
> > +
> > +  /// Look ahead an arbitrary number of tokens.
> > +  virtual size_t peekTokens(AsmToken *Buf, size_t Size,
> > +                            bool ShouldSkipSpace = true) = 0;
> >
> >    /// Get the current error location
> >    const SMLoc &getErrLoc() {
> > diff --git a/lib/MC/MCParser/AsmLexer.cpp b/lib/MC/MCParser/AsmLexer.cpp
> > index b983d99..fcedb42 100644
> > --- a/lib/MC/MCParser/AsmLexer.cpp
> > +++ b/lib/MC/MCParser/AsmLexer.cpp
> > @@ -436,7 +436,8 @@ StringRef AsmLexer::LexUntilEndOfLine() {
> >    return StringRef(TokStart, CurPtr-TokStart);
> >  }
> >
> > -const AsmToken AsmLexer::peekTok(bool ShouldSkipSpace) {
> > +size_t AsmLexer::peekTokens(AsmToken *Buf, size_t Size,
> > +                         bool ShouldSkipSpace) {
> >    const char *SavedTokStart = TokStart;
> >    const char *SavedCurPtr = CurPtr;
> >    bool SavedAtStartOfLine = isAtStartOfLine;
> > @@ -446,8 +447,17 @@ const AsmToken AsmLexer::peekTok(bool ShouldSkipSpace) {
> >    SMLoc SavedErrLoc = getErrLoc();
> >
> >    SkipSpace = ShouldSkipSpace;
> > -  AsmToken Token = LexToken();
> >
> > +  size_t ReadCount;
> > +  for(ReadCount = 0; ReadCount<Size; ++ReadCount) {
> > +    AsmToken Token = LexToken();
> > +
> > +    Buf[ReadCount] = Token;
> > +
> > +    if(Token.is(AsmToken::Eof))
> > +        break;
> > +
> > +  }
> 
> This hunk has funky identation. clang-format?
> 
> Patch looks good when my comments are fixed.
> 
> - Ben
> 
> >    SetError(SavedErrLoc, SavedErr);
> >
> >    SkipSpace = SavedSkipSpace;
> > @@ -455,7 +465,7 @@ const AsmToken AsmLexer::peekTok(bool ShouldSkipSpace) {
> >    CurPtr = SavedCurPtr;
> >    TokStart = SavedTokStart;
> >
> > -  return Token;
> > +  return ReadCount;
> >  }
> >
> >  bool AsmLexer::isAtStartOfComment(const char *Ptr) {
> > --
> > 2.4.4
> >
> 
> 
> <0001-Extend-MCAsmLexer-so-that-it-can-peek-forward-severa.patch>