r225619 - Parse: Get rid of tok::cxx_defaultarg_end, use EOF instead

[Richard Smith]

Thanks!

Is it possible to do the same to tok::cxx_exceptspec_end? (I also note we
don't currently use an end-of-delayed-tokens marker when delay-parsing
function bodies. It probably takes a bit more work to construct a testcase
that goes awry, but I suspect that case would benefit from using a
synthetic tok::eof too.)

On Sun, Jan 11, 2015 at 9:17 PM, David Majnemer <david.majnemer at gmail.com>
wrote:

> Author: majnemer
> Date: Sun Jan 11 23:17:40 2015
> New Revision: 225619
>
> URL: http://llvm.org/viewvc/llvm-project?rev=225619&view=rev
> Log:
> Parse: Get rid of tok::cxx_defaultarg_end, use EOF instead
>
> I added setEofData/getEofData to solve this sort of problem back in
> r224505.  Use the Param's decl to tell us if this is *our* EOF token.
>
> Modified:
>     cfe/trunk/include/clang/Basic/TokenKinds.def
>     cfe/trunk/lib/Parse/ParseCXXInlineMethods.cpp
>     cfe/trunk/lib/Parse/ParseDecl.cpp
>     cfe/trunk/lib/Parse/Parser.cpp
>
> Modified: cfe/trunk/include/clang/Basic/TokenKinds.def
> URL:
> http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/TokenKinds.def?rev=225619&r1=225618&r2=225619&view=diff
>
> ==============================================================================
> --- cfe/trunk/include/clang/Basic/TokenKinds.def (original)
> +++ cfe/trunk/include/clang/Basic/TokenKinds.def Sun Jan 11 23:17:40 2015
> @@ -116,7 +116,6 @@ TOK(eof)                 // End of file.
>  TOK(eod)                 // End of preprocessing directive (end of line
> inside a
>                           // directive).
>  TOK(code_completion)     // Code completion marker
> -TOK(cxx_defaultarg_end)  // C++ default argument end marker
>  TOK(cxx_exceptspec_end)  // C++ exception-specification end marker
>
>  // C99 6.4.9: Comments.
>
> Modified: cfe/trunk/lib/Parse/ParseCXXInlineMethods.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Parse/ParseCXXInlineMethods.cpp?rev=225619&r1=225618&r2=225619&view=diff
>
> ==============================================================================
> --- cfe/trunk/lib/Parse/ParseCXXInlineMethods.cpp (original)
> +++ cfe/trunk/lib/Parse/ParseCXXInlineMethods.cpp Sun Jan 11 23:17:40 2015
> @@ -342,7 +342,9 @@ void Parser::ParseLexedMethodDeclaration
>          Actions.ActOnParamDefaultArgumentError(LM.DefaultArgs[I].Param,
>                                                 EqualLoc);
>        else {
> -        if (!TryConsumeToken(tok::cxx_defaultarg_end)) {
> +        if (Tok.is(tok::eof) && Tok.getEofData() ==
> LM.DefaultArgs[I].Param) {
> +          ConsumeAnyToken();
> +        } else {
>            // The last two tokens are the terminator and the saved value of
>            // Tok; the last token in the default argument is the one before
>            // those.
> @@ -360,8 +362,11 @@ void Parser::ParseLexedMethodDeclaration
>               "ParseAssignmentExpression went over the default arg
> tokens!");
>        // There could be leftover tokens (e.g. because of an error).
>        // Skip through until we reach the original token position.
> -      while (Tok.getLocation() != origLoc && Tok.isNot(tok::eof))
> +      while (Tok.getLocation() != origLoc) {
> +        if (Tok.is(tok::eof) && Tok.getEofData() !=
> LM.DefaultArgs[I].Param)
> +          break;
>          ConsumeAnyToken();
> +      }
>
>        delete Toks;
>        LM.DefaultArgs[I].Toks = nullptr;
> @@ -652,7 +657,6 @@ bool Parser::ConsumeAndStoreUntil(tok::T
>
>      switch (Tok.getKind()) {
>      case tok::eof:
> -    case tok::cxx_defaultarg_end:
>      case tok::annot_module_begin:
>      case tok::annot_module_end:
>      case tok::annot_module_include:
>
> Modified: cfe/trunk/lib/Parse/ParseDecl.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Parse/ParseDecl.cpp?rev=225619&r1=225618&r2=225619&view=diff
>
> ==============================================================================
> --- cfe/trunk/lib/Parse/ParseDecl.cpp (original)
> +++ cfe/trunk/lib/Parse/ParseDecl.cpp Sun Jan 11 23:17:40 2015
> @@ -5631,8 +5631,9 @@ void Parser::ParseParameterDeclarationCl
>              // stop when we parse it later on.
>              Token DefArgEnd;
>              DefArgEnd.startToken();
> -            DefArgEnd.setKind(tok::cxx_defaultarg_end);
> +            DefArgEnd.setKind(tok::eof);
>              DefArgEnd.setLocation(Tok.getLocation());
> +            DefArgEnd.setEofData(Param);
>              DefArgToks->push_back(DefArgEnd);
>              Actions.ActOnParamUnparsedDefaultArgument(Param, EqualLoc,
>
>  (*DefArgToks)[1].getLocation());
>
> Modified: cfe/trunk/lib/Parse/Parser.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Parse/Parser.cpp?rev=225619&r1=225618&r2=225619&view=diff
>
> ==============================================================================
> --- cfe/trunk/lib/Parse/Parser.cpp (original)
> +++ cfe/trunk/lib/Parse/Parser.cpp Sun Jan 11 23:17:40 2015
> @@ -262,10 +262,6 @@ bool Parser::SkipUntil(ArrayRef<tok::Tok
>        // Ran out of tokens.
>        return false;
>
> -    case tok::cxx_defaultarg_end:
> -      // It's never desirable to consume the 'end-of-default-argument'
> token.
> -      return false;
> -
>      case tok::annot_pragma_openmp_end:
>        // Stop before an OpenMP pragma boundary.
>      case tok::annot_module_begin:
>
>
> _______________________________________________
> cfe-commits mailing list
> cfe-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20150111/0e551543/attachment.html>