r195543 - Generate a marker token when entering or leaving a submodule when building a
Rafael Espíndola
rafael.espindola at gmail.com
Mon Dec 2 17:58:52 PST 2013
looks like this found a real issue with OS X 10.9's headers. Should we
just xfail the test? Including a system header (<stdint.h>) in a
clang test seems like a bad idea, no? I am getting:
Command Output (stderr):
--
While building module 'Darwin' imported from
/Users/respindola/llvm/clang/test/Headers/cxx11.cpp:18:
In file included from <module-includes>:10:
/usr/include/inttypes.h:252:1: error: expected '}'
#include <sys/_types/_wchar_t.h>
^
/usr/include/inttypes.h:235:1: note: to match this '{'
__BEGIN_DECLS
^
/usr/include/sys/cdefs.h:71:34: note: expanded from macro '__BEGIN_DECLS'
#define __BEGIN_DECLS extern "C" {
^
While building module 'Darwin' imported from
/Users/respindola/llvm/clang/test/Headers/cxx11.cpp:18:
In file included from <module-includes>:10:
/usr/include/inttypes.h:263:1: error: extraneous closing brace ('}')
__END_DECLS
^
/usr/include/sys/cdefs.h:72:21: note: expanded from macro '__END_DECLS'
#define __END_DECLS }
^
While building module 'Darwin' imported from
/Users/respindola/llvm/clang/test/Headers/cxx11.cpp:18:
In file included from <module-includes>:25:
/usr/include/stdlib.h:238:1: error: expected '}'
#include <machine/types.h>
^
/usr/include/stdlib.h:127:1: note: to match this '{'
__BEGIN_DECLS
^
/usr/include/sys/cdefs.h:71:34: note: expanded from macro '__BEGIN_DECLS'
#define __BEGIN_DECLS extern "C" {
^
While building module 'Darwin' imported from
/Users/respindola/llvm/clang/test/Headers/cxx11.cpp:18:
In file included from <module-includes>:25:
/usr/include/stdlib.h:323:1: error: extraneous closing brace ('}')
__END_DECLS
^
/usr/include/sys/cdefs.h:72:21: note: expanded from macro '__END_DECLS'
#define __END_DECLS }
^
While building module 'Darwin' imported from
/Users/respindola/llvm/clang/test/Headers/cxx11.cpp:18:
In file included from <module-includes>:49:
/usr/include/iconv.h:80:1: error: expected '}'
#include <sys/_types/_wchar_t.h>
^
/usr/include/iconv.h:56:12: note: to match this '{'
extern "C" {
^
/usr/include/iconv.h:185:1: error: extraneous closing brace ('}')
}
^
While building module 'Darwin' imported from
/Users/respindola/llvm/clang/test/Headers/cxx11.cpp:18:
In file included from <module-includes>:56:
In file included from /usr/include/net/if.h:106:
In file included from /usr/include/net/if_var.h:70:
/usr/include/sys/time.h:193:1: error: expected '}'
#include <sys/_select.h> /* select() prototype */
^
/usr/include/sys/time.h:181:1: note: to match this '{'
__BEGIN_DECLS
^
/usr/include/sys/cdefs.h:71:34: note: expanded from macro '__BEGIN_DECLS'
#define __BEGIN_DECLS extern "C" {
^
While building module 'Darwin' imported from
/Users/respindola/llvm/clang/test/Headers/cxx11.cpp:18:
In file included from <module-includes>:56:
In file included from /usr/include/net/if.h:106:
In file included from /usr/include/net/if_var.h:70:
/usr/include/sys/time.h:199:1: error: extraneous closing brace ('}')
__END_DECLS
^
/usr/include/sys/cdefs.h:72:21: note: expanded from macro '__END_DECLS'
#define __END_DECLS }
^
While building module 'Darwin' imported from
/Users/respindola/llvm/clang/test/Headers/cxx11.cpp:18:
In file included from <module-includes>:185:
/usr/include/unistd.h:707:7: error: declaration of 'valloc' has a
different language linkage
void *valloc(size_t);
^
/usr/include/stdlib.h:316:7: note: previous declaration is here
void *valloc(size_t);
^
While building module 'Darwin' imported from
/Users/respindola/llvm/clang/test/Headers/cxx11.cpp:18:
In file included from <module-includes>:185:
/usr/include/unistd.h:709:14: error: declaration of 'suboptarg' has a
different language linkage
extern char *suboptarg; /* getsubopt(3) external variable */
^
/usr/include/stdlib.h:315:14: note: previous definition is here
extern char *suboptarg; /* getsubopt(3) external variable */
^
While building module 'Darwin' imported from
/Users/respindola/llvm/clang/test/Headers/cxx11.cpp:18:
In file included from <module-includes>:325:
In file included from /usr/include/ncurses.h:141:
/usr/include/unctrl.h:54:1: error: expected '}'
#include <curses.h>
^
/usr/include/unctrl.h:51:12: note: to match this '{'
extern "C" {
^
/usr/include/unctrl.h:60:1: error: extraneous closing brace ('}')
}
^
/Users/respindola/llvm/clang/test/Headers/cxx11.cpp:18:10: fatal
error: could not build module 'Darwin'
#include <stdint.h>
~~~~~~~~^
13 errors generated.
On 22 November 2013 23:06, Richard Smith <richard-llvm at metafoo.co.uk> wrote:
> Author: rsmith
> Date: Fri Nov 22 22:06:09 2013
> New Revision: 195543
>
> URL: http://llvm.org/viewvc/llvm-project?rev=195543&view=rev
> Log:
> Generate a marker token when entering or leaving a submodule when building a
> module. Use the marker to diagnose cases where we try to transition between
> submodules when not at the top level (most likely because a closing brace was
> missing at the end of a header file, but is also possible if submodule headers
> attempt to do something fundamentally non-modular, like our .def files).
>
> Added:
> cfe/trunk/test/Modules/Inputs/malformed/
> cfe/trunk/test/Modules/Inputs/malformed/a1.h
> cfe/trunk/test/Modules/Inputs/malformed/a2.h
> cfe/trunk/test/Modules/Inputs/malformed/b1.h
> cfe/trunk/test/Modules/Inputs/malformed/b2.h
> cfe/trunk/test/Modules/Inputs/malformed/module.map
> cfe/trunk/test/Modules/malformed.cpp
> Modified:
> cfe/trunk/include/clang/Basic/TokenKinds.def
> cfe/trunk/include/clang/Lex/Preprocessor.h
> cfe/trunk/include/clang/Parse/Parser.h
> cfe/trunk/lib/Frontend/PrintPreprocessedOutput.cpp
> cfe/trunk/lib/Lex/PPDirectives.cpp
> cfe/trunk/lib/Lex/PPLexerChange.cpp
> cfe/trunk/lib/Lex/Preprocessor.cpp
> cfe/trunk/lib/Parse/ParseCXXInlineMethods.cpp
> cfe/trunk/lib/Parse/ParseDecl.cpp
> cfe/trunk/lib/Parse/ParseDeclCXX.cpp
> cfe/trunk/lib/Parse/ParseInit.cpp
> cfe/trunk/lib/Parse/ParseObjc.cpp
> cfe/trunk/lib/Parse/ParseStmt.cpp
> cfe/trunk/lib/Parse/Parser.cpp
> cfe/trunk/test/Modules/auto-module-import.m
>
> Modified: cfe/trunk/include/clang/Basic/TokenKinds.def
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/TokenKinds.def?rev=195543&r1=195542&r2=195543&view=diff
> ==============================================================================
> --- cfe/trunk/include/clang/Basic/TokenKinds.def (original)
> +++ cfe/trunk/include/clang/Basic/TokenKinds.def Fri Nov 22 22:06:09 2013
> @@ -667,8 +667,10 @@ ANNOTATION(pragma_opencl_extension)
> ANNOTATION(pragma_openmp)
> ANNOTATION(pragma_openmp_end)
>
> -// Annotation for module import translated from #include etc.
> +// Annotations for module import translated from #include etc.
> ANNOTATION(module_include)
> +ANNOTATION(module_begin)
> +ANNOTATION(module_end)
>
> #undef ANNOTATION
> #undef TESTING_KEYWORD
>
> Modified: cfe/trunk/include/clang/Lex/Preprocessor.h
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Lex/Preprocessor.h?rev=195543&r1=195542&r2=195543&view=diff
> ==============================================================================
> --- cfe/trunk/include/clang/Lex/Preprocessor.h (original)
> +++ cfe/trunk/include/clang/Lex/Preprocessor.h Fri Nov 22 22:06:09 2013
> @@ -276,22 +276,26 @@ class Preprocessor : public RefCountedBa
> CLK_LexAfterModuleImport
> } CurLexerKind;
>
> + /// \brief True if the current lexer is for a submodule.
> + bool CurIsSubmodule;
> +
> /// IncludeMacroStack - This keeps track of the stack of files currently
> /// \#included, and macros currently being expanded from, not counting
> /// CurLexer/CurTokenLexer.
> struct IncludeStackInfo {
> enum CurLexerKind CurLexerKind;
> + bool IsSubmodule;
> Lexer *TheLexer;
> PTHLexer *ThePTHLexer;
> PreprocessorLexer *ThePPLexer;
> TokenLexer *TheTokenLexer;
> const DirectoryLookup *TheDirLookup;
>
> - IncludeStackInfo(enum CurLexerKind K, Lexer *L, PTHLexer* P,
> - PreprocessorLexer* PPL,
> - TokenLexer* TL, const DirectoryLookup *D)
> - : CurLexerKind(K), TheLexer(L), ThePTHLexer(P), ThePPLexer(PPL),
> - TheTokenLexer(TL), TheDirLookup(D) {}
> + IncludeStackInfo(enum CurLexerKind K, bool SM, Lexer *L, PTHLexer *P,
> + PreprocessorLexer *PPL, TokenLexer *TL,
> + const DirectoryLookup *D)
> + : CurLexerKind(K), IsSubmodule(SM), TheLexer(L), ThePTHLexer(P),
> + ThePPLexer(PPL), TheTokenLexer(TL), TheDirLookup(D) {}
> };
> std::vector<IncludeStackInfo> IncludeMacroStack;
>
> @@ -662,7 +666,7 @@ public:
> /// start lexing tokens from it instead of the current buffer. Emit an error
> /// and don't enter the file on error.
> void EnterSourceFile(FileID CurFileID, const DirectoryLookup *Dir,
> - SourceLocation Loc);
> + SourceLocation Loc, bool IsSubmodule = false);
>
> /// EnterMacro - Add a Macro to the top of the include stack and start lexing
> /// tokens from it instead of the current buffer. Args specifies the
> @@ -1155,6 +1159,9 @@ private:
> IdentifierInfo *Ident__abnormal_termination,
> *Ident___abnormal_termination,
> *Ident_AbnormalTermination;
> +
> + const char *getCurLexerEndPos();
> +
> public:
> void PoisonSEHIdentifiers(bool Poison = true); // Borland
>
> @@ -1265,6 +1272,7 @@ private:
>
> void PushIncludeMacroStack() {
> IncludeMacroStack.push_back(IncludeStackInfo(CurLexerKind,
> + CurIsSubmodule,
> CurLexer.take(),
> CurPTHLexer.take(),
> CurPPLexer,
> @@ -1280,6 +1288,7 @@ private:
> CurTokenLexer.reset(IncludeMacroStack.back().TheTokenLexer);
> CurDirLookup = IncludeMacroStack.back().TheDirLookup;
> CurLexerKind = IncludeMacroStack.back().CurLexerKind;
> + CurIsSubmodule = IncludeMacroStack.back().IsSubmodule;
> IncludeMacroStack.pop_back();
> }
>
> @@ -1380,11 +1389,13 @@ private:
>
> /// EnterSourceFileWithLexer - Add a lexer to the top of the include stack and
> /// start lexing tokens from it instead of the current buffer.
> - void EnterSourceFileWithLexer(Lexer *TheLexer, const DirectoryLookup *Dir);
> + void EnterSourceFileWithLexer(Lexer *TheLexer, const DirectoryLookup *Dir,
> + bool IsSubmodule = false);
>
> /// EnterSourceFileWithPTH - Add a lexer to the top of the include stack and
> /// start getting tokens from it using the PTH cache.
> - void EnterSourceFileWithPTH(PTHLexer *PL, const DirectoryLookup *Dir);
> + void EnterSourceFileWithPTH(PTHLexer *PL, const DirectoryLookup *Dir,
> + bool IsSubmodule = false);
>
> /// \brief Set the file ID for the preprocessor predefines.
> void setPredefinesFileID(FileID FID) {
>
> Modified: cfe/trunk/include/clang/Parse/Parser.h
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Parse/Parser.h?rev=195543&r1=195542&r2=195543&view=diff
> ==============================================================================
> --- cfe/trunk/include/clang/Parse/Parser.h (original)
> +++ cfe/trunk/include/clang/Parse/Parser.h Fri Nov 22 22:06:09 2013
> @@ -73,7 +73,7 @@ class Parser : public CodeCompletionHand
> SourceLocation PrevTokLocation;
>
> unsigned short ParenCount, BracketCount, BraceCount;
> -
> +
> /// Actions - These are the callbacks we invoke as we parse various constructs
> /// in the file.
> Sema &Actions;
> @@ -408,6 +408,14 @@ private:
> Tok.setKind(tok::eof);
> }
>
> + /// \brief Determine if we're at the end of the file or at a transition
> + /// between modules.
> + bool isEofOrEom() {
> + tok::TokenKind Kind = Tok.getKind();
> + return Kind == tok::eof || Kind == tok::annot_module_begin ||
> + Kind == tok::annot_module_end || Kind == tok::annot_module_include;
> + }
> +
> /// \brief Handle the annotation token produced for #pragma unused(...)
> void HandlePragmaUnused();
>
>
> Modified: cfe/trunk/lib/Frontend/PrintPreprocessedOutput.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Frontend/PrintPreprocessedOutput.cpp?rev=195543&r1=195542&r2=195543&view=diff
> ==============================================================================
> --- cfe/trunk/lib/Frontend/PrintPreprocessedOutput.cpp (original)
> +++ cfe/trunk/lib/Frontend/PrintPreprocessedOutput.cpp Fri Nov 22 22:06:09 2013
> @@ -657,7 +657,9 @@ static void PrintPreprocessedTokens(Prep
> // -traditional-cpp the lexer keeps /all/ whitespace, including comments.
> SourceLocation StartLoc = Tok.getLocation();
> Callbacks->MoveToLine(StartLoc.getLocWithOffset(Tok.getLength()));
> - } else if (Tok.is(tok::annot_module_include)) {
> + } else if (Tok.is(tok::annot_module_include) ||
> + Tok.is(tok::annot_module_begin) ||
> + Tok.is(tok::annot_module_end)) {
> // PrintPPOutputPPCallbacks::InclusionDirective handles producing
> // appropriate output here. Ignore this token entirely.
> PP.Lex(Tok);
>
> Modified: cfe/trunk/lib/Lex/PPDirectives.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Lex/PPDirectives.cpp?rev=195543&r1=195542&r2=195543&view=diff
> ==============================================================================
> --- cfe/trunk/lib/Lex/PPDirectives.cpp (original)
> +++ cfe/trunk/lib/Lex/PPDirectives.cpp Fri Nov 22 22:06:09 2013
> @@ -1389,6 +1389,19 @@ bool Preprocessor::ConcatenateIncludeNam
> return true;
> }
>
> +/// \brief Push a token onto the token stream containing an annotation.
> +static void EnterAnnotationToken(Preprocessor &PP,
> + SourceLocation Begin, SourceLocation End,
> + tok::TokenKind Kind, void *AnnotationVal) {
> + Token *Tok = new Token[1];
> + Tok[0].startToken();
> + Tok[0].setKind(Kind);
> + Tok[0].setLocation(Begin);
> + Tok[0].setAnnotationEndLoc(End);
> + Tok[0].setAnnotationValue(AnnotationVal);
> + PP.EnterTokenStream(Tok, 1, true, true);
> +}
> +
> /// HandleIncludeDirective - The "\#include" tokens have just been read, read
> /// the file to be included from the lexer, then include it! This is a common
> /// routine with functionality shared between \#include, \#include_next and
> @@ -1590,7 +1603,7 @@ void Preprocessor::HandleIncludeDirectiv
> // include directive maps to.
> bool BuildingImportedModule
> = Path[0].first->getName() == getLangOpts().CurrentModule;
> -
> +
> if (!BuildingImportedModule && getLangOpts().ObjC2) {
> // If we're not building the imported module, warn that we're going
> // to automatically turn this inclusion directive into a module import.
> @@ -1639,13 +1652,8 @@ void Preprocessor::HandleIncludeDirectiv
> // make the module visible.
> // FIXME: Produce this as the current token directly, rather than
> // allocating a new token for it.
> - Token *Tok = new Token[1];
> - Tok[0].startToken();
> - Tok[0].setKind(tok::annot_module_include);
> - Tok[0].setLocation(HashLoc);
> - Tok[0].setAnnotationEndLoc(End);
> - Tok[0].setAnnotationValue(Imported);
> - EnterTokenStream(Tok, 1, true, true);
> + EnterAnnotationToken(*this, HashLoc, End, tok::annot_module_include,
> + Imported);
> }
> return;
> }
> @@ -1692,8 +1700,23 @@ void Preprocessor::HandleIncludeDirectiv
> FileID FID = SourceMgr.createFileID(File, IncludePos, FileCharacter);
> assert(!FID.isInvalid() && "Expected valid file ID");
>
> - // Finally, if all is good, enter the new file!
> - EnterSourceFile(FID, CurDir, FilenameTok.getLocation());
> + // Determine if we're switching to building a new submodule, and which one.
> + ModuleMap::KnownHeader BuildingModule;
> + if (getLangOpts().Modules && !getLangOpts().CurrentModule.empty()) {
> + Module *RequestingModule = getModuleForLocation(FilenameLoc);
> + BuildingModule =
> + HeaderInfo.getModuleMap().findModuleForHeader(File, RequestingModule);
> + }
> +
> + // If all is good, enter the new file!
> + EnterSourceFile(FID, CurDir, FilenameTok.getLocation(),
> + static_cast<bool>(BuildingModule));
> +
> + // If we're walking into another part of the same module, let the parser
> + // know that any future declarations are within that other submodule.
> + if (BuildingModule)
> + EnterAnnotationToken(*this, HashLoc, End, tok::annot_module_begin,
> + BuildingModule.getModule());
> }
>
> /// HandleIncludeNextDirective - Implements \#include_next.
>
> Modified: cfe/trunk/lib/Lex/PPLexerChange.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Lex/PPLexerChange.cpp?rev=195543&r1=195542&r2=195543&view=diff
> ==============================================================================
> --- cfe/trunk/lib/Lex/PPLexerChange.cpp (original)
> +++ cfe/trunk/lib/Lex/PPLexerChange.cpp Fri Nov 22 22:06:09 2013
> @@ -69,7 +69,7 @@ PreprocessorLexer *Preprocessor::getCurr
> /// EnterSourceFile - Add a source file to the top of the include stack and
> /// start lexing tokens from it instead of the current buffer.
> void Preprocessor::EnterSourceFile(FileID FID, const DirectoryLookup *CurDir,
> - SourceLocation Loc) {
> + SourceLocation Loc, bool IsSubmodule) {
> assert(!CurTokenLexer && "Cannot #include a file inside a macro!");
> ++NumEnteredSourceFiles;
>
> @@ -78,7 +78,7 @@ void Preprocessor::EnterSourceFile(FileI
>
> if (PTH) {
> if (PTHLexer *PL = PTH->CreateLexer(FID)) {
> - EnterSourceFileWithPTH(PL, CurDir);
> + EnterSourceFileWithPTH(PL, CurDir, IsSubmodule);
> return;
> }
> }
> @@ -101,14 +101,16 @@ void Preprocessor::EnterSourceFile(FileI
> CodeCompletionFileLoc.getLocWithOffset(CodeCompletionOffset);
> }
>
> - EnterSourceFileWithLexer(new Lexer(FID, InputFile, *this), CurDir);
> + EnterSourceFileWithLexer(new Lexer(FID, InputFile, *this), CurDir,
> + IsSubmodule);
> return;
> }
>
> /// EnterSourceFileWithLexer - Add a source file to the top of the include stack
> /// and start lexing tokens from it instead of the current buffer.
> void Preprocessor::EnterSourceFileWithLexer(Lexer *TheLexer,
> - const DirectoryLookup *CurDir) {
> + const DirectoryLookup *CurDir,
> + bool IsSubmodule) {
>
> // Add the current lexer to the include stack.
> if (CurPPLexer || CurTokenLexer)
> @@ -117,6 +119,7 @@ void Preprocessor::EnterSourceFileWithLe
> CurLexer.reset(TheLexer);
> CurPPLexer = TheLexer;
> CurDirLookup = CurDir;
> + CurIsSubmodule = IsSubmodule;
> if (CurLexerKind != CLK_LexAfterModuleImport)
> CurLexerKind = CLK_Lexer;
>
> @@ -133,7 +136,8 @@ void Preprocessor::EnterSourceFileWithLe
> /// EnterSourceFileWithPTH - Add a source file to the top of the include stack
> /// and start getting tokens from it using the PTH cache.
> void Preprocessor::EnterSourceFileWithPTH(PTHLexer *PL,
> - const DirectoryLookup *CurDir) {
> + const DirectoryLookup *CurDir,
> + bool IsSubmodule) {
>
> if (CurPPLexer || CurTokenLexer)
> PushIncludeMacroStack();
> @@ -141,6 +145,7 @@ void Preprocessor::EnterSourceFileWithPT
> CurDirLookup = CurDir;
> CurPTHLexer.reset(PL);
> CurPPLexer = CurPTHLexer.get();
> + CurIsSubmodule = IsSubmodule;
> if (CurLexerKind != CLK_LexAfterModuleImport)
> CurLexerKind = CLK_PTHLexer;
>
> @@ -244,6 +249,29 @@ void Preprocessor::PropagateLineStartLea
> // but it might if they're empty?
> }
>
> +/// \brief Determine the location to use as the end of the buffer for a lexer.
> +///
> +/// If the file ends with a newline, form the EOF token on the newline itself,
> +/// rather than "on the line following it", which doesn't exist. This makes
> +/// diagnostics relating to the end of file include the last file that the user
> +/// actually typed, which is goodness.
> +const char *Preprocessor::getCurLexerEndPos() {
> + const char *EndPos = CurLexer->BufferEnd;
> + if (EndPos != CurLexer->BufferStart &&
> + (EndPos[-1] == '\n' || EndPos[-1] == '\r')) {
> + --EndPos;
> +
> + // Handle \n\r and \r\n:
> + if (EndPos != CurLexer->BufferStart &&
> + (EndPos[-1] == '\n' || EndPos[-1] == '\r') &&
> + EndPos[-1] != EndPos[0])
> + --EndPos;
> + }
> +
> + return EndPos;
> +}
> +
> +
> /// HandleEndOfFile - This callback is invoked when the lexer hits the end of
> /// the current file. This either returns the EOF token or pops a level off
> /// the include stack and keeps going.
> @@ -342,7 +370,19 @@ bool Preprocessor::HandleEndOfFile(Token
> FileID ExitedFID;
> if (Callbacks && !isEndOfMacro && CurPPLexer)
> ExitedFID = CurPPLexer->getFileID();
> -
> +
> + // If this file corresponded to a submodule, notify the parser that we've
> + // left that submodule.
> + bool LeavingSubmodule = CurIsSubmodule && CurLexer;
> + if (LeavingSubmodule) {
> + const char *EndPos = getCurLexerEndPos();
> + Result.startToken();
> + CurLexer->BufferPtr = EndPos;
> + CurLexer->FormTokenWithChars(Result, EndPos, tok::annot_module_end);
> + Result.setAnnotationEndLoc(Result.getLocation());
> + Result.setAnnotationValue(0);
> + }
> +
> // We're done with the #included file.
> RemoveTopOfLexerStack();
>
> @@ -357,27 +397,13 @@ bool Preprocessor::HandleEndOfFile(Token
> PPCallbacks::ExitFile, FileType, ExitedFID);
> }
>
> - // Client should lex another token.
> - return false;
> + // Client should lex another token unless we generated an EOM.
> + return LeavingSubmodule;
> }
>
> - // If the file ends with a newline, form the EOF token on the newline itself,
> - // rather than "on the line following it", which doesn't exist. This makes
> - // diagnostics relating to the end of file include the last file that the user
> - // actually typed, which is goodness.
> + // If this is the end of the main file, form an EOF token.
> if (CurLexer) {
> - const char *EndPos = CurLexer->BufferEnd;
> - if (EndPos != CurLexer->BufferStart &&
> - (EndPos[-1] == '\n' || EndPos[-1] == '\r')) {
> - --EndPos;
> -
> - // Handle \n\r and \r\n:
> - if (EndPos != CurLexer->BufferStart &&
> - (EndPos[-1] == '\n' || EndPos[-1] == '\r') &&
> - EndPos[-1] != EndPos[0])
> - --EndPos;
> - }
> -
> + const char *EndPos = getCurLexerEndPos();
> Result.startToken();
> CurLexer->BufferPtr = EndPos;
> CurLexer->FormTokenWithChars(Result, EndPos, tok::eof);
>
> Modified: cfe/trunk/lib/Lex/Preprocessor.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Lex/Preprocessor.cpp?rev=195543&r1=195542&r2=195543&view=diff
> ==============================================================================
> --- cfe/trunk/lib/Lex/Preprocessor.cpp (original)
> +++ cfe/trunk/lib/Lex/Preprocessor.cpp Fri Nov 22 22:06:09 2013
> @@ -67,8 +67,8 @@ Preprocessor::Preprocessor(IntrusiveRefC
> CodeComplete(0), CodeCompletionFile(0), CodeCompletionOffset(0),
> LastTokenWasAt(false), ModuleImportExpectsIdentifier(false),
> CodeCompletionReached(0), SkipMainFilePreamble(0, true), CurPPLexer(0),
> - CurDirLookup(0), CurLexerKind(CLK_Lexer), Callbacks(0),
> - MacroArgCache(0), Record(0), MIChainHead(0), MICache(0),
> + CurDirLookup(0), CurLexerKind(CLK_Lexer), CurIsSubmodule(false),
> + Callbacks(0), MacroArgCache(0), Record(0), MIChainHead(0), MICache(0),
> DeserialMIChainHead(0) {
> OwnsHeaderSearch = OwnsHeaders;
>
>
> Modified: cfe/trunk/lib/Parse/ParseCXXInlineMethods.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Parse/ParseCXXInlineMethods.cpp?rev=195543&r1=195542&r2=195543&view=diff
> ==============================================================================
> --- cfe/trunk/lib/Parse/ParseCXXInlineMethods.cpp (original)
> +++ cfe/trunk/lib/Parse/ParseCXXInlineMethods.cpp Fri Nov 22 22:06:09 2013
> @@ -580,6 +580,9 @@ bool Parser::ConsumeAndStoreUntil(tok::T
>
> switch (Tok.getKind()) {
> case tok::eof:
> + case tok::annot_module_begin:
> + case tok::annot_module_end:
> + case tok::annot_module_include:
> // Ran out of tokens.
> return false;
>
> @@ -965,6 +968,9 @@ bool Parser::ConsumeAndStoreInitializer(
> goto consume_token;
>
> case tok::eof:
> + case tok::annot_module_begin:
> + case tok::annot_module_end:
> + case tok::annot_module_include:
> // Ran out of tokens.
> return false;
>
>
> Modified: cfe/trunk/lib/Parse/ParseDecl.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Parse/ParseDecl.cpp?rev=195543&r1=195542&r2=195543&view=diff
> ==============================================================================
> --- cfe/trunk/lib/Parse/ParseDecl.cpp (original)
> +++ cfe/trunk/lib/Parse/ParseDecl.cpp Fri Nov 22 22:06:09 2013
> @@ -1561,6 +1561,9 @@ void Parser::SkipMalformedDecl() {
> break;
>
> case tok::eof:
> + case tok::annot_module_begin:
> + case tok::annot_module_end:
> + case tok::annot_module_include:
> return;
>
> default:
> @@ -3371,7 +3374,7 @@ void Parser::ParseStructUnionBody(Source
> SmallVector<Decl *, 32> FieldDecls;
>
> // While we still have something to read, read the declarations in the struct.
> - while (Tok.isNot(tok::r_brace) && Tok.isNot(tok::eof)) {
> + while (Tok.isNot(tok::r_brace) && !isEofOrEom()) {
> // Each iteration of this loop reads one struct-declaration.
>
> // Check for extraneous top-level semicolon.
>
> Modified: cfe/trunk/lib/Parse/ParseDeclCXX.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Parse/ParseDeclCXX.cpp?rev=195543&r1=195542&r2=195543&view=diff
> ==============================================================================
> --- cfe/trunk/lib/Parse/ParseDeclCXX.cpp (original)
> +++ cfe/trunk/lib/Parse/ParseDeclCXX.cpp Fri Nov 22 22:06:09 2013
> @@ -195,7 +195,7 @@ void Parser::ParseInnerNamespace(std::ve
> ParsedAttributes& attrs,
> BalancedDelimiterTracker &Tracker) {
> if (index == Ident.size()) {
> - while (Tok.isNot(tok::r_brace) && Tok.isNot(tok::eof)) {
> + while (Tok.isNot(tok::r_brace) && !isEofOrEom()) {
> ParsedAttributesWithRange attrs(AttrFactory);
> MaybeParseCXX11Attributes(attrs);
> MaybeParseMicrosoftAttributes(attrs);
> @@ -318,7 +318,7 @@ Decl *Parser::ParseLinkage(ParsingDeclSp
>
> BalancedDelimiterTracker T(*this, tok::l_brace);
> T.consumeOpen();
> - while (Tok.isNot(tok::r_brace) && Tok.isNot(tok::eof)) {
> + while (Tok.isNot(tok::r_brace) && !isEofOrEom()) {
> ParsedAttributesWithRange attrs(AttrFactory);
> MaybeParseCXX11Attributes(attrs);
> MaybeParseMicrosoftAttributes(attrs);
> @@ -2452,7 +2452,7 @@ ExprResult Parser::ParseCXXMemberInitial
> // a top-level comma always ends the initializer expression.
> const Token &Next = NextToken();
> if (IsFunction || Next.is(tok::semi) || Next.is(tok::comma) ||
> - Next.is(tok::eof)) {
> + Next.is(tok::eof)) {
> if (IsFunction)
> Diag(ConsumeToken(), diag::err_default_delete_in_multiple_declaration)
> << 1 /* delete */;
> @@ -2597,7 +2597,7 @@ void Parser::ParseCXXMemberSpecification
>
> if (TagDecl) {
> // While we still have something to read, read the member-declarations.
> - while (Tok.isNot(tok::r_brace) && Tok.isNot(tok::eof)) {
> + while (Tok.isNot(tok::r_brace) && !isEofOrEom()) {
> // Each iteration of this loop reads one member-declaration.
>
> if (getLangOpts().MicrosoftExt && (Tok.is(tok::kw___if_exists) ||
> @@ -3420,7 +3420,7 @@ void Parser::ParseMicrosoftIfExistsClass
> return;
> }
>
> - while (Tok.isNot(tok::r_brace) && Tok.isNot(tok::eof)) {
> + while (Tok.isNot(tok::r_brace) && !isEofOrEom()) {
> // __if_exists, __if_not_exists can nest.
> if ((Tok.is(tok::kw___if_exists) || Tok.is(tok::kw___if_not_exists))) {
> ParseMicrosoftIfExistsClassDeclaration((DeclSpec::TST)TagType, CurAS);
>
> Modified: cfe/trunk/lib/Parse/ParseInit.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Parse/ParseInit.cpp?rev=195543&r1=195542&r2=195543&view=diff
> ==============================================================================
> --- cfe/trunk/lib/Parse/ParseInit.cpp (original)
> +++ cfe/trunk/lib/Parse/ParseInit.cpp Fri Nov 22 22:06:09 2013
> @@ -512,7 +512,7 @@ bool Parser::ParseMicrosoftIfExistsBrace
> return false;
> }
>
> - while (Tok.isNot(tok::eof)) {
> + while (!isEofOrEom()) {
> trailingComma = false;
> // If we know that this cannot be a designation, just parse the nested
> // initializer directly.
>
> Modified: cfe/trunk/lib/Parse/ParseObjc.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Parse/ParseObjc.cpp?rev=195543&r1=195542&r2=195543&view=diff
> ==============================================================================
> --- cfe/trunk/lib/Parse/ParseObjc.cpp (original)
> +++ cfe/trunk/lib/Parse/ParseObjc.cpp Fri Nov 22 22:06:09 2013
> @@ -423,7 +423,7 @@ void Parser::ParseObjCInterfaceDeclList(
> }
>
> // If we got to the end of the file, exit the loop.
> - if (Tok.is(tok::eof))
> + if (isEofOrEom())
> break;
>
> // Code completion within an Objective-C interface.
> @@ -1289,7 +1289,7 @@ void Parser::ParseObjCClassInstanceVaria
> BalancedDelimiterTracker T(*this, tok::l_brace);
> T.consumeOpen();
> // While we still have something to read, read the instance variables.
> - while (Tok.isNot(tok::r_brace) && Tok.isNot(tok::eof)) {
> + while (Tok.isNot(tok::r_brace) && !isEofOrEom()) {
> // Each iteration of this loop reads one objc-instance-variable-decl.
>
> // Check for extraneous top-level semicolon.
> @@ -1582,7 +1582,7 @@ Parser::ParseObjCAtImplementationDeclara
>
> {
> ObjCImplParsingDataRAII ObjCImplParsing(*this, ObjCImpDecl);
> - while (!ObjCImplParsing.isFinished() && Tok.isNot(tok::eof)) {
> + while (!ObjCImplParsing.isFinished() && !isEofOrEom()) {
> ParsedAttributesWithRange attrs(AttrFactory);
> MaybeParseCXX11Attributes(attrs);
> MaybeParseMicrosoftAttributes(attrs);
> @@ -1612,7 +1612,7 @@ Parser::ParseObjCAtEndDeclaration(Source
> Parser::ObjCImplParsingDataRAII::~ObjCImplParsingDataRAII() {
> if (!Finished) {
> finish(P.Tok.getLocation());
> - if (P.Tok.is(tok::eof)) {
> + if (P.isEofOrEom()) {
> P.Diag(P.Tok, diag::err_objc_missing_end)
> << FixItHint::CreateInsertion(P.Tok.getLocation(), "\n at end\n");
> P.Diag(Dcl->getLocStart(), diag::note_objc_container_start)
>
> Modified: cfe/trunk/lib/Parse/ParseStmt.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Parse/ParseStmt.cpp?rev=195543&r1=195542&r2=195543&view=diff
> ==============================================================================
> --- cfe/trunk/lib/Parse/ParseStmt.cpp (original)
> +++ cfe/trunk/lib/Parse/ParseStmt.cpp Fri Nov 22 22:06:09 2013
> @@ -890,7 +890,7 @@ StmtResult Parser::ParseCompoundStatemen
> Stmts.push_back(R.release());
> }
>
> - while (Tok.isNot(tok::r_brace) && Tok.isNot(tok::eof)) {
> + while (Tok.isNot(tok::r_brace) && !isEofOrEom()) {
> if (Tok.is(tok::annot_pragma_unused)) {
> HandlePragmaUnused();
> continue;
> @@ -2058,7 +2058,7 @@ StmtResult Parser::ParseMicrosoftAsmStat
> SourceLocation TokLoc = Tok.getLocation();
> do {
> // If we hit EOF, we're done, period.
> - if (Tok.is(tok::eof))
> + if (isEofOrEom())
> break;
>
> if (!InAsmComment && Tok.is(tok::semi)) {
>
> Modified: cfe/trunk/lib/Parse/Parser.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Parse/Parser.cpp?rev=195543&r1=195542&r2=195543&view=diff
> ==============================================================================
> --- cfe/trunk/lib/Parse/Parser.cpp (original)
> +++ cfe/trunk/lib/Parse/Parser.cpp Fri Nov 22 22:06:09 2013
> @@ -288,7 +288,7 @@ bool Parser::SkipUntil(ArrayRef<tok::Tok
> if (Toks.size() == 1 && Toks[0] == tok::eof &&
> !HasFlagsSet(Flags, StopAtSemi) &&
> !HasFlagsSet(Flags, StopAtCodeCompletion)) {
> - while (Tok.getKind() != tok::eof)
> + while (Tok.isNot(tok::eof))
> ConsumeAnyToken();
> return true;
> }
> @@ -297,7 +297,15 @@ bool Parser::SkipUntil(ArrayRef<tok::Tok
> case tok::eof:
> // Ran out of tokens.
> return false;
> -
> +
> + case tok::annot_module_begin:
> + case tok::annot_module_end:
> + case tok::annot_module_include:
> + // Stop before we change submodules. They generally indicate a "good"
> + // place to pick up parsing again (except in the special case where
> + // we're trying to skip to EOF).
> + return false;
> +
> case tok::code_completion:
> if (!HasFlagsSet(Flags, StopAtCodeCompletion))
> ConsumeToken();
> @@ -574,10 +582,12 @@ namespace {
> bool Parser::ParseTopLevelDecl(DeclGroupPtrTy &Result) {
> DestroyTemplateIdAnnotationsRAIIObj CleanupRAII(TemplateIds);
>
> - // Skip over the EOF token, flagging end of previous input for incremental
> + // Skip over the EOF token, flagging end of previous input for incremental
> // processing
> - if (PP.isIncrementalProcessingEnabled() && Tok.is(tok::eof))
> + if (PP.isIncrementalProcessingEnabled() && Tok.is(tok::eof)) {
> ConsumeToken();
> + return false;
> + }
>
> Result = DeclGroupPtrTy();
> switch (Tok.getKind()) {
> @@ -592,6 +602,12 @@ bool Parser::ParseTopLevelDecl(DeclGroup
> ConsumeToken();
> return false;
>
> + case tok::annot_module_begin:
> + case tok::annot_module_end:
> + // FIXME: Update visibility based on the submodule we're in.
> + ConsumeToken();
> + return false;
> +
> case tok::eof:
> // Late template parsing can begin.
> if (getLangOpts().DelayedTemplateParsing)
> @@ -1917,14 +1933,15 @@ void Parser::ParseMicrosoftIfExistsExter
> }
>
> // Parse the declarations.
> - while (Tok.isNot(tok::r_brace) && Tok.isNot(tok::eof)) {
> + // FIXME: Support module import within __if_exists?
> + while (Tok.isNot(tok::r_brace) && !isEofOrEom()) {
> ParsedAttributesWithRange attrs(AttrFactory);
> MaybeParseCXX11Attributes(attrs);
> MaybeParseMicrosoftAttributes(attrs);
> DeclGroupPtrTy Result = ParseExternalDeclaration(attrs);
> if (Result && !getCurScope()->getParent())
> Actions.getASTConsumer().HandleTopLevelDecl(Result.get());
> - }
> + }
> Braces.consumeClose();
> }
>
> @@ -1980,8 +1997,8 @@ bool BalancedDelimiterTracker::diagnoseO
> P.Diag(P.Tok, diag::err_bracket_depth_exceeded)
> << P.getLangOpts().BracketDepth;
> P.Diag(P.Tok, diag::note_bracket_depth);
> - P.SkipUntil(tok::eof);
> - return true;
> + P.cutOffParsing();
> + return true;
> }
>
> bool BalancedDelimiterTracker::expectAndConsume(unsigned DiagID,
>
> Added: cfe/trunk/test/Modules/Inputs/malformed/a1.h
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/Inputs/malformed/a1.h?rev=195543&view=auto
> ==============================================================================
> --- cfe/trunk/test/Modules/Inputs/malformed/a1.h (added)
> +++ cfe/trunk/test/Modules/Inputs/malformed/a1.h Fri Nov 22 22:06:09 2013
> @@ -0,0 +1 @@
> +void f() {
>
> Added: cfe/trunk/test/Modules/Inputs/malformed/a2.h
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/Inputs/malformed/a2.h?rev=195543&view=auto
> ==============================================================================
> --- cfe/trunk/test/Modules/Inputs/malformed/a2.h (added)
> +++ cfe/trunk/test/Modules/Inputs/malformed/a2.h Fri Nov 22 22:06:09 2013
> @@ -0,0 +1 @@
> +}
>
> Added: cfe/trunk/test/Modules/Inputs/malformed/b1.h
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/Inputs/malformed/b1.h?rev=195543&view=auto
> ==============================================================================
> --- cfe/trunk/test/Modules/Inputs/malformed/b1.h (added)
> +++ cfe/trunk/test/Modules/Inputs/malformed/b1.h Fri Nov 22 22:06:09 2013
> @@ -0,0 +1,3 @@
> +struct S {
> + #include "b2.h"
> +};
>
> Added: cfe/trunk/test/Modules/Inputs/malformed/b2.h
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/Inputs/malformed/b2.h?rev=195543&view=auto
> ==============================================================================
> --- cfe/trunk/test/Modules/Inputs/malformed/b2.h (added)
> +++ cfe/trunk/test/Modules/Inputs/malformed/b2.h Fri Nov 22 22:06:09 2013
> @@ -0,0 +1 @@
> +void g() {}
>
> Added: cfe/trunk/test/Modules/Inputs/malformed/module.map
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/Inputs/malformed/module.map?rev=195543&view=auto
> ==============================================================================
> --- cfe/trunk/test/Modules/Inputs/malformed/module.map (added)
> +++ cfe/trunk/test/Modules/Inputs/malformed/module.map Fri Nov 22 22:06:09 2013
> @@ -0,0 +1,8 @@
> +module a {
> + module a1 { header "a1.h" }
> + module a2 { header "a2.h" }
> +}
> +module b {
> + module b1 { header "b1.h" }
> + module b2 { header "b2.h" }
> +}
>
> Modified: cfe/trunk/test/Modules/auto-module-import.m
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/auto-module-import.m?rev=195543&r1=195542&r2=195543&view=diff
> ==============================================================================
> --- cfe/trunk/test/Modules/auto-module-import.m (original)
> +++ cfe/trunk/test/Modules/auto-module-import.m Fri Nov 22 22:06:09 2013
> @@ -83,6 +83,6 @@ int getNotInModule() {
> return not_in_module;
> }
>
> -void includeNotAtTopLevel() {
> - #include <NoUmbrella/A.h> // expected-warning {{treating #include as an import}} expected-error {{expected expression}}
> -}
> +void includeNotAtTopLevel() { // expected-note {{to match this '{'}}
> + #include <NoUmbrella/A.h> // expected-warning {{treating #include as an import}} expected-error {{expected '}'}}
> +} // expected-error {{extraneous closing brace}}
>
> Added: cfe/trunk/test/Modules/malformed.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/malformed.cpp?rev=195543&view=auto
> ==============================================================================
> --- cfe/trunk/test/Modules/malformed.cpp (added)
> +++ cfe/trunk/test/Modules/malformed.cpp Fri Nov 22 22:06:09 2013
> @@ -0,0 +1,23 @@
> +// RUN: rm -rf %t
> +// RUN: not %clang_cc1 -fmodules -fmodules-cache-path=%t -I %S/Inputs/malformed -DHEADER="a1.h" %s 2>&1 | FileCheck %s --check-prefix=CHECK-A
> +// RUN: not %clang_cc1 -fmodules -fmodules-cache-path=%t -I %S/Inputs/malformed -DHEADER="b1.h" %s 2>&1 | FileCheck %s --check-prefix=CHECK-B
> +
> +#define STR2(x) #x
> +#define STR(x) STR2(x)
> +#include STR(HEADER)
> +
> +// CHECK-A: While building module 'a'
> +// CHECK-A: a1.h:1:{{.*}} error: expected '}'
> +// CHECK-A: a1.h:1:{{.*}} note: to match this '{'
> +//
> +// CHECK-A: While building module 'a'
> +// CHECK-A: a2.h:1:{{.*}} error: extraneous closing brace
> +
> +// CHECK-B: While building module 'b'
> +// CHECK-B: b1.h:2:{{.*}} error: expected '}'
> +// CHECK-B: b1.h:1:{{.*}} note: to match this '{'
> +// CHECK-B: b1.h:3:{{.*}} error: extraneous closing brace ('}')
> +//
> +// CHECK-B: While building module 'b'
> +// CHECK-B: b2.h:1:{{.*}} error: redefinition of 'g'
> +// CHECK-B: b2.h:1:{{.*}} note: previous definition is here
>
>
> _______________________________________________
> cfe-commits mailing list
> cfe-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-commits
More information about the cfe-commits
mailing list