r304515 - Support lazy stat'ing of files referenced by module maps.
Alexander Kornienko via cfe-commits
cfe-commits at lists.llvm.org
Fri Jun 2 09:59:04 PDT 2017
I've not yet figured out exactly, but I have a suspicion that this commit
causes crashes when run under asan. Specifically, when running
test/Modules/preprocess-module.cpp
The stack trace is:
#0 0x1d644c6 in (anonymous
namespace)::HeaderFileInfoTrait::EmitData(llvm::raw_ostream&, (anonymous
namespace)::HeaderFileInfoTrait::key_type const&, (anonymous
namespace)::HeaderFileInfoTrait::data_type const&, unsigned int)
llvm/tools/clang/lib/Serialization/ASTWriter.cpp:1926:39
#1 0x1d322f3 in llvm::OnDiskChainedHashTableGenerator<(anonymous
namespace)::HeaderFileInfoTrait>::Emit(llvm::raw_ostream&, (anonymous
namespace)::HeaderFileInfoTrait&)
llvm/include/llvm/Support/OnDiskHashTable.h:198:17
#2 0x1d3168d in clang::ASTWriter::WriteHeaderSearch(clang::HeaderSearch
const&) llvm/tools/clang/lib/Serialization/ASTWriter.cpp:2092:30
#3 0x1d52f4c in clang::ASTWriter::WriteASTCore(clang::Sema&,
llvm::StringRef, std::string const&, clang::Module*)
llvm/tools/clang/lib/Serialization/ASTWriter.cpp:4857:3
#4 0x1d4f7db in clang::ASTWriter::WriteAST(clang::Sema&, std::string
const&, clang::Module*, llvm::StringRef, bool)
llvm/tools/clang/lib/Serialization/ASTWriter.cpp:4475:7
#5 0x1cd0333 in
clang::PCHGenerator::HandleTranslationUnit(clang::ASTContext&)
llvm/tools/clang/lib/Serialization/GeneratePCH.cpp:62:14
#6 0x1f56cc7 in
clang::MultiplexConsumer::HandleTranslationUnit(clang::ASTContext&)
llvm/tools/clang/lib/Frontend/MultiplexConsumer.cpp:305:15
#7 0x24ebc9c in clang::ParseAST(clang::Sema&, bool, bool)
llvm/tools/clang/lib/Parse/ParseAST.cpp:159:13
#8 0x1f4b283 in clang::FrontendAction::Execute()
llvm/tools/clang/lib/Frontend/FrontendAction.cpp:856:8
#9 0x1caf916 in
clang::CompilerInstance::ExecuteAction(clang::FrontendAction&)
llvm/tools/clang/lib/Frontend/CompilerInstance.cpp:970:11
#10 0x5ed12c in
clang::ExecuteCompilerInvocation(clang::CompilerInstance*)
llvm/tools/clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:249:25
Could you take a look?
On Fri, Jun 2, 2017 at 3:55 AM, Richard Smith via cfe-commits <
cfe-commits at lists.llvm.org> wrote:
> Author: rsmith
> Date: Thu Jun 1 20:55:39 2017
> New Revision: 304515
>
> URL: http://llvm.org/viewvc/llvm-project?rev=304515&view=rev
> Log:
> Support lazy stat'ing of files referenced by module maps.
>
> This patch adds support for a `header` declaration in a module map to
> specify
> certain `stat` information (currently, size and mtime) about that header
> file.
> This has two purposes:
>
> - It removes the need to eagerly `stat` every file referenced by a module
> map.
> Instead, we track a list of unresolved header files with each size /
> mtime
> (actually, for simplicity, we track submodules with such headers), and
> when
> attempting to look up a header file based on a `FileEntry`, we check if
> there
> are any unresolved header directives with that `FileEntry`'s size /
> mtime and
> perform deferred `stat`s if so.
>
> - It permits a preprocessed module to be compiled without the original
> files
> being present on disk. The only reason we used to need those files was
> to get
> the `stat` information in order to do header -> module lookups when
> using the
> module. If we're provided with the `stat` information in the preprocessed
> module, we can avoid requiring the files to exist.
>
> Unlike most `header` directives, if a `header` directive with `stat`
> information has no corresponding on-disk file the enclosing module is *not*
> marked unavailable (so that behavior is consistent regardless of whether
> we've
> resolved a header directive, and so that preprocessed modules don't get
> marked
> unavailable). We could actually do this for all `header` directives: the
> only
> reason we mark the module unavailable if headers are missing is to give a
> diagnostic slightly earlier (rather than waiting until we actually try to
> build
> the module / load and validate its .pcm file).
>
> Differential Revision: https://reviews.llvm.org/D33703
>
> Added:
> cfe/trunk/test/Modules/Inputs/header-attribs/
> cfe/trunk/test/Modules/Inputs/header-attribs/bar.h
> cfe/trunk/test/Modules/Inputs/header-attribs/baz.h
> cfe/trunk/test/Modules/Inputs/header-attribs/foo.h
> cfe/trunk/test/Modules/Inputs/header-attribs/modular.modulemap
> cfe/trunk/test/Modules/Inputs/header-attribs/textual.modulemap
> cfe/trunk/test/Modules/header-attribs.cpp
> cfe/trunk/test/Modules/preprocess-missing.modulemap
> Modified:
> cfe/trunk/docs/Modules.rst
> cfe/trunk/include/clang/Basic/DiagnosticLexKinds.td
> cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td
> cfe/trunk/include/clang/Basic/Module.h
> cfe/trunk/include/clang/Lex/ModuleMap.h
> cfe/trunk/lib/Basic/Module.cpp
> cfe/trunk/lib/Frontend/FrontendAction.cpp
> cfe/trunk/lib/Lex/HeaderSearch.cpp
> cfe/trunk/lib/Lex/ModuleMap.cpp
> cfe/trunk/lib/Lex/PPDirectives.cpp
> cfe/trunk/lib/Serialization/ASTWriter.cpp
> cfe/trunk/test/Modules/diagnostics.modulemap
> cfe/trunk/test/Modules/preprocess-module.cpp
>
> Modified: cfe/trunk/docs/Modules.rst
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/docs/
> Modules.rst?rev=304515&r1=304514&r2=304515&view=diff
> ============================================================
> ==================
> --- cfe/trunk/docs/Modules.rst (original)
> +++ cfe/trunk/docs/Modules.rst Thu Jun 1 20:55:39 2017
> @@ -469,9 +469,16 @@ A header declaration specifies that a pa
> .. parsed-literal::
>
> *header-declaration*:
> - ``private``:sub:`opt` ``textual``:sub:`opt` ``header``
> *string-literal*
> - ``umbrella`` ``header`` *string-literal*
> - ``exclude`` ``header`` *string-literal*
> + ``private``:sub:`opt` ``textual``:sub:`opt` ``header``
> *string-literal* *header-attrs*:sub:`opt`
> + ``umbrella`` ``header`` *string-literal* *header-attrs*:sub:`opt`
> + ``exclude`` ``header`` *string-literal* *header-attrs*:sub:`opt`
> +
> + *header-attrs*:
> + '{' *header-attr** '}'
> +
> + *header-attr*:
> + ``size`` *integer-literal*
> + ``mtime`` *integer-literal*
>
> A header declaration that does not contain ``exclude`` nor ``textual``
> specifies a header that contributes to the enclosing module. Specifically,
> when the module is built, the named header will be parsed and its
> declarations will be (logically) placed into the enclosing submodule.
>
> @@ -504,6 +511,18 @@ A header with the ``exclude`` specifier
>
> A given header shall not be referenced by more than one
> *header-declaration*.
>
> +Two *header-declaration*\s, or a *header-declaration* and a ``#include``,
> are
> +considered to refer to the same file if the paths resolve to the same file
> +and the specified *header-attr*\s (if any) match the attributes of that
> file,
> +even if the file is named differently (for instance, by a relative path or
> +via symlinks).
> +
> +.. note::
> + The use of *header-attr*\s avoids the need for Clang to speculatively
> + ``stat`` every header referenced by a module map. It is recommended
> that
> + *header-attr*\s only be used in machine-generated module maps, to
> avoid
> + mismatches between attribute values and the corresponding files.
> +
> Umbrella directory declaration
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> An umbrella directory declaration specifies that all of the headers in
> the specified directory should be included within the module.
>
> Modified: cfe/trunk/include/clang/Basic/DiagnosticLexKinds.td
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/
> DiagnosticLexKinds.td?rev=304515&r1=304514&r2=304515&view=diff
> ============================================================
> ==================
> --- cfe/trunk/include/clang/Basic/DiagnosticLexKinds.td (original)
> +++ cfe/trunk/include/clang/Basic/DiagnosticLexKinds.td Thu Jun 1
> 20:55:39 2017
> @@ -664,6 +664,12 @@ def warn_mmap_mismatched_top_level_priva
> InGroup<PrivateModule>;
> def note_mmap_rename_top_level_private_as_submodule : Note<
> "make '%0' a submodule of '%1' to ensure it can be found by name">;
> +def err_mmap_duplicate_header_attribute : Error<
> + "header attribute '%0' specified multiple times">;
> +def err_mmap_invalid_header_attribute_value : Error<
> + "expected integer literal as value for header attribute '%0'">;
> +def err_mmap_expected_header_attribute : Error<
> + "expected a header attribute name ('size' or 'mtime')">;
>
> def warn_auto_module_import : Warning<
> "treating #%select{include|import|include_next|__include_macros}0 as
> an "
>
> Modified: cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/
> DiagnosticSerializationKinds.td?rev=304515&r1=304514&r2=304515&view=diff
> ============================================================
> ==================
> --- cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td
> (original)
> +++ cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td Thu
> Jun 1 20:55:39 2017
> @@ -174,10 +174,6 @@ def note_module_odr_violation_mismatch_d
> "method %2 with %ordinal3 parameter of type %4%select{| decayed from
> %6}5|"
> "method %2 with %ordinal3 parameter named %4}1">;
>
> -def warn_module_uses_date_time : Warning<
> - "%select{precompiled header|module}0 uses __DATE__ or __TIME__">,
> - InGroup<DiagGroup<"pch-date-time">>;
> -
> def warn_duplicate_module_file_extension : Warning<
> "duplicate module file extension block name '%0'">,
> InGroup<ModuleFileExtension>;
> @@ -186,7 +182,15 @@ def warn_module_system_bit_conflict : Wa
> "module file '%0' was validated as a system module and is now being
> imported "
> "as a non-system module; any difference in diagnostic options will be
> ignored">,
> InGroup<ModuleConflict>;
> +} // let CategoryName
>
> +let CategoryName = "AST Serialization Issue" in {
> +def warn_module_uses_date_time : Warning<
> + "%select{precompiled header|module}0 uses __DATE__ or __TIME__">,
> + InGroup<DiagGroup<"pch-date-time">>;
> +def err_module_no_size_mtime_for_header : Error<
> + "cannot emit module %0: %select{size|mtime}1 must be explicitly
> specified "
> + "for missing header file \"%2\"">;
> } // let CategoryName
> } // let Component
>
>
> Modified: cfe/trunk/include/clang/Basic/Module.h
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/
> clang/Basic/Module.h?rev=304515&r1=304514&r2=304515&view=diff
> ============================================================
> ==================
> --- cfe/trunk/include/clang/Basic/Module.h (original)
> +++ cfe/trunk/include/clang/Basic/Module.h Thu Jun 1 20:55:39 2017
> @@ -154,11 +154,19 @@ public:
> /// \brief Stored information about a header directive that was found
> in the
> /// module map file but has not been resolved to a file.
> struct UnresolvedHeaderDirective {
> + HeaderKind Kind = HK_Normal;
> SourceLocation FileNameLoc;
> std::string FileName;
> - bool IsUmbrella;
> + bool IsUmbrella = false;
> + bool HasBuiltinHeader = false;
> + Optional<off_t> Size;
> + Optional<time_t> ModTime;
> };
>
> + /// Headers that are mentioned in the module map file but that we have
> not
> + /// yet attempted to resolve to a file on the file system.
> + SmallVector<UnresolvedHeaderDirective, 1> UnresolvedHeaders;
> +
> /// \brief Headers that are mentioned in the module map file but could
> not be
> /// found on the file system.
> SmallVector<UnresolvedHeaderDirective, 1> MissingHeaders;
>
> Modified: cfe/trunk/include/clang/Lex/ModuleMap.h
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/
> clang/Lex/ModuleMap.h?rev=304515&r1=304514&r2=304515&view=diff
> ============================================================
> ==================
> --- cfe/trunk/include/clang/Lex/ModuleMap.h (original)
> +++ cfe/trunk/include/clang/Lex/ModuleMap.h Thu Jun 1 20:55:39 2017
> @@ -26,6 +26,7 @@
> #include "llvm/ADT/SmallVector.h"
> #include "llvm/ADT/StringMap.h"
> #include "llvm/ADT/StringRef.h"
> +#include "llvm/ADT/TinyPtrVector.h"
> #include "llvm/ADT/Twine.h"
> #include <algorithm>
> #include <memory>
> @@ -116,6 +117,11 @@ public:
> // Adjust ModuleMap::addHeader.
> };
>
> + /// Convert a header kind to a role. Requires Kind to not be
> HK_Excluded.
> + static ModuleHeaderRole headerKindToRole(Module::HeaderKind Kind);
> + /// Convert a header role to a kind.
> + static Module::HeaderKind headerRoleToKind(ModuleHeaderRole Role);
> +
> /// \brief A header that is known to reside within a given module,
> /// whether it was included or excluded.
> class KnownHeader {
> @@ -165,7 +171,13 @@ private:
> /// \brief Mapping from each header to the module that owns the
> contents of
> /// that header.
> HeadersMap Headers;
> -
> +
> + /// Map from file sizes to modules with lazy header directives of that
> size.
> + mutable llvm::DenseMap<off_t, llvm::TinyPtrVector<Module*>>
> LazyHeadersBySize;
> + /// Map from mtimes to modules with lazy header directives with those
> mtimes.
> + mutable llvm::DenseMap<time_t, llvm::TinyPtrVector<Module*>>
> + LazyHeadersByModTime;
> +
> /// \brief Mapping from directories with umbrella headers to the module
> /// that is generated from the umbrella header.
> ///
> @@ -257,22 +269,30 @@ private:
> /// resolved.
> Module *resolveModuleId(const ModuleId &Id, Module *Mod, bool Complain)
> const;
>
> - /// Resolve the given header directive to an actual header file.
> + /// Add an unresolved header to a module.
> + void addUnresolvedHeader(Module *Mod,
> + Module::UnresolvedHeaderDirective Header);
> +
> + /// Look up the given header directive to find an actual header file.
> ///
> /// \param M The module in which we're resolving the header directive.
> /// \param Header The header directive to resolve.
> /// \param RelativePathName Filled in with the relative path name from
> the
> /// module to the resolved header.
> /// \return The resolved file, if any.
> - const FileEntry *resolveHeader(Module *M,
> - Module::UnresolvedHeaderDirective
> Header,
> - SmallVectorImpl<char> &RelativePathName);
> + const FileEntry *findHeader(Module *M,
> + const Module::UnresolvedHeaderDirective
> &Header,
> + SmallVectorImpl<char> &RelativePathName);
> +
> + /// Resolve the given header directive.
> + void resolveHeader(Module *M,
> + const Module::UnresolvedHeaderDirective &Header);
>
> /// Attempt to resolve the specified header directive as naming a
> builtin
> /// header.
> - const FileEntry *
> - resolveAsBuiltinHeader(Module *M, Module::UnresolvedHeaderDirective
> Header,
> - SmallVectorImpl<char> &BuiltinPathName);
> + /// \return \c true if a corresponding builtin header was found.
> + bool resolveAsBuiltinHeader(Module *M,
> + const Module::UnresolvedHeaderDirective
> &Header);
>
> /// \brief Looks up the modules that \p File corresponds to.
> ///
> @@ -368,6 +388,15 @@ public:
> /// the preferred module for the header.
> ArrayRef<KnownHeader> findAllModulesForHeader(const FileEntry *File)
> const;
>
> + /// Resolve all lazy header directives for the specified file.
> + ///
> + /// This ensures that the HeaderFileInfo on HeaderSearch is up to date.
> This
> + /// is effectively internal, but is exposed so HeaderSearch can call it.
> + void resolveHeaderDirectives(const FileEntry *File) const;
> +
> + /// Resolve all lazy header directives for the specified module.
> + void resolveHeaderDirectives(Module *Mod) const;
> +
> /// \brief Reports errors if a module must not include a specific file.
> ///
> /// \param RequestingModule The module including a file.
>
> Modified: cfe/trunk/lib/Basic/Module.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Basic/
> Module.cpp?rev=304515&r1=304514&r2=304515&view=diff
> ============================================================
> ==================
> --- cfe/trunk/lib/Basic/Module.cpp (original)
> +++ cfe/trunk/lib/Basic/Module.cpp Thu Jun 1 20:55:39 2017
> @@ -394,11 +394,30 @@ void Module::print(raw_ostream &OS, unsi
> {"exclude ", HK_Excluded}};
>
> for (auto &K : Kinds) {
> + assert(&K == &Kinds[K.Kind] && "kinds in wrong order");
> for (auto &H : Headers[K.Kind]) {
> OS.indent(Indent + 2);
> OS << K.Prefix << "header \"";
> OS.write_escaped(H.NameAsWritten);
> - OS << "\"\n";
> + OS << "\" { size " << H.Entry->getSize()
> + << " mtime " << H.Entry->getModificationTime() << " }\n";
> + }
> + }
> + for (auto *Unresolved : {&UnresolvedHeaders, &MissingHeaders}) {
> + for (auto &U : *Unresolved) {
> + OS.indent(Indent + 2);
> + OS << Kinds[U.Kind].Prefix << "header \"";
> + OS.write_escaped(U.FileName);
> + OS << "\"";
> + if (U.Size || U.ModTime) {
> + OS << " {";
> + if (U.Size)
> + OS << " size " << *U.Size;
> + if (U.ModTime)
> + OS << " mtime " << *U.ModTime;
> + OS << " }";
> + }
> + OS << "\n";
> }
> }
>
>
> Modified: cfe/trunk/lib/Frontend/FrontendAction.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/
> Frontend/FrontendAction.cpp?rev=304515&r1=304514&r2=304515&view=diff
> ============================================================
> ==================
> --- cfe/trunk/lib/Frontend/FrontendAction.cpp (original)
> +++ cfe/trunk/lib/Frontend/FrontendAction.cpp Thu Jun 1 20:55:39 2017
> @@ -289,14 +289,28 @@ static void addHeaderInclude(StringRef H
> ///
> /// \param Includes Will be augmented with the set of \#includes or
> \#imports
> /// needed to load all of the named headers.
> -static std::error_code
> -collectModuleHeaderIncludes(const LangOptions &LangOpts, FileManager
> &FileMgr,
> - ModuleMap &ModMap, clang::Module *Module,
> - SmallVectorImpl<char> &Includes) {
> +static std::error_code collectModuleHeaderIncludes(
> + const LangOptions &LangOpts, FileManager &FileMgr, DiagnosticsEngine
> &Diag,
> + ModuleMap &ModMap, clang::Module *Module, SmallVectorImpl<char>
> &Includes) {
> // Don't collect any headers for unavailable modules.
> if (!Module->isAvailable())
> return std::error_code();
>
> + // Resolve all lazy header directives to header files.
> + ModMap.resolveHeaderDirectives(Module);
> +
> + // If any headers are missing, we can't build this module. In most
> cases,
> + // diagnostics for this should have already been produced; we only get
> here
> + // if explicit stat information was provided.
> + // FIXME: If the name resolves to a file with different stat
> information,
> + // produce a better diagnostic.
> + if (!Module->MissingHeaders.empty()) {
> + auto &MissingHeader = Module->MissingHeaders.front();
> + Diag.Report(MissingHeader.FileNameLoc, diag::err_module_header_
> missing)
> + << MissingHeader.IsUmbrella << MissingHeader.FileName;
> + return std::error_code();
> + }
> +
> // Add includes for each of these headers.
> for (auto HK : {Module::HK_Normal, Module::HK_Private}) {
> for (Module::Header &H : Module->Headers[HK]) {
> @@ -367,7 +381,7 @@ collectModuleHeaderIncludes(const LangOp
> SubEnd = Module->submodule_end();
> Sub != SubEnd; ++Sub)
> if (std::error_code Err = collectModuleHeaderIncludes(
> - LangOpts, FileMgr, ModMap, *Sub, Includes))
> + LangOpts, FileMgr, Diag, ModMap, *Sub, Includes))
> return Err;
>
> return std::error_code();
> @@ -494,7 +508,7 @@ getInputBufferForModule(CompilerInstance
> addHeaderInclude(UmbrellaHeader.NameAsWritten, HeaderContents,
> CI.getLangOpts(), M->IsExternC);
> Err = collectModuleHeaderIncludes(
> - CI.getLangOpts(), FileMgr,
> + CI.getLangOpts(), FileMgr, CI.getDiagnostics(),
> CI.getPreprocessor().getHeaderSearchInfo().getModuleMap(), M,
> HeaderContents);
>
>
> Modified: cfe/trunk/lib/Lex/HeaderSearch.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Lex/
> HeaderSearch.cpp?rev=304515&r1=304514&r2=304515&view=diff
> ============================================================
> ==================
> --- cfe/trunk/lib/Lex/HeaderSearch.cpp (original)
> +++ cfe/trunk/lib/Lex/HeaderSearch.cpp Thu Jun 1 20:55:39 2017
> @@ -1114,6 +1114,8 @@ bool HeaderSearch::ShouldEnterIncludeFil
> auto TryEnterImported = [&](void) -> bool {
> if (!ModulesEnabled)
> return false;
> + // Ensure FileInfo bits are up to date.
> + ModMap.resolveHeaderDirectives(File);
> // Modules with builtins are special; multiple modules use builtins as
> // modular headers, example:
> //
>
> Modified: cfe/trunk/lib/Lex/ModuleMap.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Lex/
> ModuleMap.cpp?rev=304515&r1=304514&r2=304515&view=diff
> ============================================================
> ==================
> --- cfe/trunk/lib/Lex/ModuleMap.cpp (original)
> +++ cfe/trunk/lib/Lex/ModuleMap.cpp Thu Jun 1 20:55:39 2017
> @@ -36,6 +36,37 @@
> #endif
> using namespace clang;
>
> +Module::HeaderKind ModuleMap::headerRoleToKind(ModuleHeaderRole Role) {
> + switch ((int)Role) {
> + default: llvm_unreachable("unknown header role");
> + case NormalHeader:
> + return Module::HK_Normal;
> + case PrivateHeader:
> + return Module::HK_Private;
> + case TextualHeader:
> + return Module::HK_Textual;
> + case PrivateHeader | TextualHeader:
> + return Module::HK_PrivateTextual;
> + }
> +}
> +
> +ModuleMap::ModuleHeaderRole
> +ModuleMap::headerKindToRole(Module::HeaderKind Kind) {
> + switch ((int)Kind) {
> + case Module::HK_Normal:
> + return NormalHeader;
> + case Module::HK_Private:
> + return PrivateHeader;
> + case Module::HK_Textual:
> + return TextualHeader;
> + case Module::HK_PrivateTextual:
> + return ModuleHeaderRole(PrivateHeader | TextualHeader);
> + case Module::HK_Excluded:
> + llvm_unreachable("unexpected header kind");
> + }
> + llvm_unreachable("unknown header kind");
> +}
> +
> Module::ExportDecl
> ModuleMap::resolveExport(Module *Mod,
> const Module::UnresolvedExportDecl &Unresolved,
> @@ -104,12 +135,22 @@ static void appendSubframeworkPaths(Modu
> }
>
> const FileEntry *
> -ModuleMap::resolveHeader(Module *M, Module::UnresolvedHeaderDirective
> Header,
> - SmallVectorImpl<char> &RelativePathName) {
> +ModuleMap::findHeader(Module *M,
> + const Module::UnresolvedHeaderDirective &Header,
> + SmallVectorImpl<char> &RelativePathName) {
> + auto GetFile = [&](StringRef Filename) -> const FileEntry * {
> + auto *File = SourceMgr.getFileManager().getFile(Filename);
> + if (!File ||
> + (Header.Size && File->getSize() != *Header.Size) ||
> + (Header.ModTime && File->getModificationTime() !=
> *Header.ModTime))
> + return nullptr;
> + return File;
> + };
> +
> if (llvm::sys::path::is_absolute(Header.FileName)) {
> RelativePathName.clear();
> RelativePathName.append(Header.FileName.begin(),
> Header.FileName.end());
> - return SourceMgr.getFileManager().getFile(Header.FileName);
> + return GetFile(Header.FileName);
> }
>
> // Search for the header file within the module's home directory.
> @@ -124,7 +165,7 @@ ModuleMap::resolveHeader(Module *M, Modu
> // Check whether this file is in the public headers.
> llvm::sys::path::append(RelativePathName, "Headers",
> Header.FileName);
> llvm::sys::path::append(FullPathName, RelativePathName);
> - if (auto *File = SourceMgr.getFileManager().getFile(FullPathName))
> + if (auto *File = GetFile(FullPathName))
> return File;
>
> // Check whether this file is in the private headers.
> @@ -141,31 +182,74 @@ ModuleMap::resolveHeader(Module *M, Modu
> llvm::sys::path::append(RelativePathName, "PrivateHeaders",
> Header.FileName);
> llvm::sys::path::append(FullPathName, RelativePathName);
> - return SourceMgr.getFileManager().getFile(FullPathName);
> + return GetFile(FullPathName);
> }
>
> // Lookup for normal headers.
> llvm::sys::path::append(RelativePathName, Header.FileName);
> llvm::sys::path::append(FullPathName, RelativePathName);
> - return SourceMgr.getFileManager().getFile(FullPathName);
> + return GetFile(FullPathName);
> }
>
> -const FileEntry *
> -ModuleMap::resolveAsBuiltinHeader(Module *M,
> - Module::UnresolvedHeaderDirective
> Header,
> - SmallVectorImpl<char> &BuiltinPathName)
> {
> - if (llvm::sys::path::is_absolute(Header.FileName) ||
> M->isPartOfFramework() ||
> - !M->IsSystem || Header.IsUmbrella || !BuiltinIncludeDir ||
> - BuiltinIncludeDir == M->Directory || !isBuiltinHeader(Header.
> FileName))
> - return nullptr;
> +void ModuleMap::resolveHeader(Module *Mod,
> + const Module::UnresolvedHeaderDirective
> &Header) {
> + SmallString<128> RelativePathName;
> + if (const FileEntry *File = findHeader(Mod, Header, RelativePathName)) {
> + if (Header.IsUmbrella) {
> + const DirectoryEntry *UmbrellaDir = File->getDir();
> + if (Module *UmbrellaMod = UmbrellaDirs[UmbrellaDir])
> + Diags.Report(Header.FileNameLoc, diag::err_mmap_umbrella_clash)
> + << UmbrellaMod->getFullModuleName();
> + else
> + // Record this umbrella header.
> + setUmbrellaHeader(Mod, File, RelativePathName.str());
> + } else {
> + Module::Header H = {RelativePathName.str(), File};
> + if (Header.Kind == Module::HK_Excluded)
> + excludeHeader(Mod, H);
> + else
> + addHeader(Mod, H, headerKindToRole(Header.Kind));
> + }
> + } else if (Header.HasBuiltinHeader && !Header.Size && !Header.ModTime) {
> + // There's a builtin header but no corresponding on-disk header.
> Assume
> + // this was supposed to modularize the builtin header alone.
> + } else if (Header.Kind == Module::HK_Excluded) {
> + // Ignore missing excluded header files. They're optional anyway.
> + } else {
> + // If we find a module that has a missing header, we mark this module
> as
> + // unavailable and store the header directive for displaying
> diagnostics.
> + Mod->MissingHeaders.push_back(Header);
> + // A missing header with stat information doesn't make the module
> + // unavailable; this keeps our behavior consistent as headers are
> lazily
> + // resolved. (Such a module still can't be built though, except from
> + // preprocessed source.)
> + if (!Header.Size && !Header.ModTime)
> + Mod->markUnavailable();
> + }
> +}
> +
> +bool ModuleMap::resolveAsBuiltinHeader(
> + Module *Mod, const Module::UnresolvedHeaderDirective &Header) {
> + if (Header.Kind == Module::HK_Excluded ||
> + llvm::sys::path::is_absolute(Header.FileName) ||
> + Mod->isPartOfFramework() || !Mod->IsSystem || Header.IsUmbrella ||
> + !BuiltinIncludeDir || BuiltinIncludeDir == Mod->Directory ||
> + !isBuiltinHeader(Header.FileName))
> + return false;
>
> // This is a system module with a top-level header. This header
> // may have a counterpart (or replacement) in the set of headers
> // supplied by Clang. Find that builtin header.
> - llvm::sys::path::append(BuiltinPathName, BuiltinIncludeDir->getName(),
> - Header.FileName);
> - return SourceMgr.getFileManager().getFile(
> - StringRef(BuiltinPathName.data(), BuiltinPathName.size()));
> + SmallString<128> Path;
> + llvm::sys::path::append(Path, BuiltinIncludeDir->getName(),
> Header.FileName);
> + auto *File = SourceMgr.getFileManager().getFile(Path);
> + if (!File)
> + return false;
> +
> + auto Role = headerKindToRole(Header.Kind);
> + Module::Header H = {Path.str(), File};
> + addHeader(Mod, H, Role);
> + return true;
> }
>
> ModuleMap::ModuleMap(SourceManager &SourceMgr, DiagnosticsEngine &Diags,
> @@ -246,6 +330,7 @@ bool ModuleMap::isBuiltinHeader(StringRe
>
> ModuleMap::HeadersMap::iterator
> ModuleMap::findKnownHeader(const FileEntry *File) {
> + resolveHeaderDirectives(File);
> HeadersMap::iterator Known = Headers.find(File);
> if (HeaderInfo.getHeaderSearchOpts().ImplicitModuleMaps &&
> Known == Headers.end() && File->getDir() == BuiltinIncludeDir &&
> @@ -328,8 +413,10 @@ void ModuleMap::diagnoseHeaderInclusion(
> if (getTopLevelOrNull(RequestingModule) != getTopLevelOrNull(
> SourceModule))
> return;
>
> - if (RequestingModule)
> + if (RequestingModule) {
> resolveUses(RequestingModule, /*Complain=*/false);
> + resolveHeaderDirectives(RequestingModule);
> + }
>
> bool Excluded = false;
> Module *Private = nullptr;
> @@ -511,6 +598,7 @@ ModuleMap::findOrCreateModuleForHeaderIn
>
> ArrayRef<ModuleMap::KnownHeader>
> ModuleMap::findAllModulesForHeader(const FileEntry *File) const {
> + resolveHeaderDirectives(File);
> auto It = Headers.find(File);
> if (It == Headers.end())
> return None;
> @@ -524,6 +612,7 @@ bool ModuleMap::isHeaderInUnavailableMod
> bool
> ModuleMap::isHeaderUnavailableInModule(const FileEntry *Header,
> const Module *RequestingModule)
> const {
> + resolveHeaderDirectives(Header);
> HeadersMap::const_iterator Known = Headers.find(Header);
> if (Known != Headers.end()) {
> for (SmallVectorImpl<KnownHeader>::const_iterator
> @@ -896,20 +985,65 @@ void ModuleMap::setUmbrellaDir(Module *M
> UmbrellaDirs[UmbrellaDir] = Mod;
> }
>
> -static Module::HeaderKind headerRoleToKind(ModuleMap::ModuleHeaderRole
> Role) {
> - switch ((int)Role) {
> - default: llvm_unreachable("unknown header role");
> - case ModuleMap::NormalHeader:
> - return Module::HK_Normal;
> - case ModuleMap::PrivateHeader:
> - return Module::HK_Private;
> - case ModuleMap::TextualHeader:
> - return Module::HK_Textual;
> - case ModuleMap::PrivateHeader | ModuleMap::TextualHeader:
> - return Module::HK_PrivateTextual;
> +void ModuleMap::addUnresolvedHeader(Module *Mod,
> + Module::UnresolvedHeaderDirective
> Header) {
> + // If there is a builtin counterpart to this file, add it now so it can
> + // wrap the system header.
> + if (resolveAsBuiltinHeader(Mod, Header)) {
> + // If we have both a builtin and system version of the file, the
> + // builtin version may want to inject macros into the system header,
> so
> + // force the system header to be treated as a textual header in this
> + // case.
> + Header.Kind = headerRoleToKind(ModuleMap::ModuleHeaderRole(
> + headerKindToRole(Header.Kind) | ModuleMap::TextualHeader));
> + Header.HasBuiltinHeader = true;
> + }
> +
> + // If possible, don't stat the header until we need to. This requires
> the
> + // user to have provided us with some stat information about the file.
> + // FIXME: Add support for lazily stat'ing umbrella headers and excluded
> + // headers.
> + if ((Header.Size || Header.ModTime) && !Header.IsUmbrella &&
> + Header.Kind != Module::HK_Excluded) {
> + // We expect more variation in mtime than size, so if we're given
> both,
> + // use the mtime as the key.
> + if (Header.ModTime)
> + LazyHeadersByModTime[*Header.ModTime].push_back(Mod);
> + else
> + LazyHeadersBySize[*Header.Size].push_back(Mod);
> + Mod->UnresolvedHeaders.push_back(Header);
> + return;
> + }
> +
> + // We don't have stat information or can't defer looking this file up.
> + // Perform the lookup now.
> + resolveHeader(Mod, Header);
> +}
> +
> +void ModuleMap::resolveHeaderDirectives(const FileEntry *File) const {
> + auto BySize = LazyHeadersBySize.find(File->getSize());
> + if (BySize != LazyHeadersBySize.end()) {
> + for (auto *M : BySize->second)
> + resolveHeaderDirectives(M);
> + LazyHeadersBySize.erase(BySize);
> + }
> +
> + auto ByModTime = LazyHeadersByModTime.find(
> File->getModificationTime());
> + if (ByModTime != LazyHeadersByModTime.end()) {
> + for (auto *M : ByModTime->second)
> + resolveHeaderDirectives(M);
> + LazyHeadersByModTime.erase(ByModTime);
> }
> }
>
> +void ModuleMap::resolveHeaderDirectives(Module *Mod) const {
> + for (auto &Header : Mod->UnresolvedHeaders)
> + // This operation is logically const; we're just changing how we
> represent
> + // the header information for this file.
> + const_cast<ModuleMap*>(this)->resolveHeader(Mod, Header);
> + Mod->UnresolvedHeaders.clear();
> +}
> +
> void ModuleMap::addHeader(Module *Mod, Module::Header Header,
> ModuleHeaderRole Role, bool Imported) {
> KnownHeader KH(Mod, Role);
> @@ -1063,6 +1197,7 @@ namespace clang {
> RequiresKeyword,
> Star,
> StringLiteral,
> + IntegerLiteral,
> TextualKeyword,
> LBrace,
> RBrace,
> @@ -1072,7 +1207,12 @@ namespace clang {
>
> unsigned Location;
> unsigned StringLength;
> - const char *StringData;
> + union {
> + // If Kind != IntegerLiteral.
> + const char *StringData;
> + // If Kind == IntegerLiteral.
> + uint64_t IntegerValue;
> + };
>
> void clear() {
> Kind = EndOfFile;
> @@ -1086,9 +1226,14 @@ namespace clang {
> SourceLocation getLocation() const {
> return SourceLocation::getFromRawEncoding(Location);
> }
> +
> + uint64_t getInteger() const {
> + return Kind == IntegerLiteral ? IntegerValue : 0;
> + }
>
> StringRef getString() const {
> - return StringRef(StringData, StringLength);
> + return Kind == IntegerLiteral ? StringRef()
> + : StringRef(StringData, StringLength);
> }
> };
>
> @@ -1278,6 +1423,25 @@ retry:
> Tok.StringLength = Length;
> break;
> }
> +
> + case tok::numeric_constant: {
> + // We don't support any suffixes or other complications.
> + SmallString<32> SpellingBuffer;
> + SpellingBuffer.resize(LToken.getLength() + 1);
> + const char *Start = SpellingBuffer.data();
> + unsigned Length =
> + Lexer::getSpelling(LToken, Start, SourceMgr, L.getLangOpts());
> + uint64_t Value;
> + if (StringRef(Start, Length).getAsInteger(0, Value)) {
> + Diags.Report(Tok.getLocation(), diag::err_mmap_unknown_token);
> + HadError = true;
> + goto retry;
> + }
> +
> + Tok.Kind = MMToken::IntegerLiteral;
> + Tok.IntegerValue = Value;
> + break;
> + }
>
> case tok::comment:
> goto retry;
> @@ -1904,6 +2068,9 @@ void ModuleMapParser::parseHeaderDecl(MM
> Header.FileName = Tok.getString();
> Header.FileNameLoc = consumeToken();
> Header.IsUmbrella = LeadingToken == MMToken::UmbrellaKeyword;
> + Header.Kind =
> + (LeadingToken == MMToken::ExcludeKeyword ? Module::HK_Excluded
> + :
> Map.headerRoleToKind(Role));
>
> // Check whether we already have an umbrella.
> if (Header.IsUmbrella && ActiveModule->Umbrella) {
> @@ -1913,64 +2080,62 @@ void ModuleMapParser::parseHeaderDecl(MM
> return;
> }
>
> - // Look for this file by name if we don't have any stat information.
> - SmallString<128> RelativePathName, BuiltinPathName;
> - const FileEntry *File =
> - Map.resolveHeader(ActiveModule, Header, RelativePathName);
> - const FileEntry *BuiltinFile =
> - Map.resolveAsBuiltinHeader(ActiveModule, Header, BuiltinPathName);
> -
> - // If Clang supplies this header but the underlying system does not,
> - // just silently swap in our builtin version. Otherwise, we'll end
> - // up adding both (later).
> - if (BuiltinFile && !File) {
> - RelativePathName = BuiltinPathName;
> - File = BuiltinFile;
> - BuiltinFile = nullptr;
> - }
> -
> - // FIXME: We shouldn't be eagerly stat'ing every file named in a module
> map.
> - // Come up with a lazy way to do this.
> - if (File) {
> - if (Header.IsUmbrella) {
> - const DirectoryEntry *UmbrellaDir = File->getDir();
> - if (Module *UmbrellaModule = Map.UmbrellaDirs[UmbrellaDir]) {
> - Diags.Report(LeadingLoc, diag::err_mmap_umbrella_clash)
> - << UmbrellaModule->getFullModuleName();
> - HadError = true;
> - } else {
> - // Record this umbrella header.
> - Map.setUmbrellaHeader(ActiveModule, File,
> RelativePathName.str());
> - }
> - } else if (LeadingToken == MMToken::ExcludeKeyword) {
> - Module::Header H = {RelativePathName.str(), File};
> - Map.excludeHeader(ActiveModule, H);
> - } else {
> - // If there is a builtin counterpart to this file, add it now so it
> can
> - // wrap the system header.
> - if (BuiltinFile) {
> - Module::Header H = { BuiltinPathName.str(), BuiltinFile };
> - Map.addHeader(ActiveModule, H, Role);
> -
> - // If we have both a builtin and system version of the file, the
> - // builtin version may want to inject macros into the system
> header, so
> - // force the system header to be treated as a textual header in
> this
> - // case.
> - Role = ModuleMap::ModuleHeaderRole(Role |
> ModuleMap::TextualHeader);
> - }
> + // If we were given stat information, parse it so we can skip looking
> for
> + // the file.
> + if (Tok.is(MMToken::LBrace)) {
> + SourceLocation LBraceLoc = consumeToken();
> +
> + while (!Tok.is(MMToken::RBrace) && !Tok.is(MMToken::EndOfFile)) {
> + enum Attribute { Size, ModTime, Unknown };
> + StringRef Str = Tok.getString();
> + SourceLocation Loc = consumeToken();
> + switch (llvm::StringSwitch<Attribute>(Str)
> + .Case("size", Size)
> + .Case("mtime", ModTime)
> + .Default(Unknown)) {
> + case Size:
> + if (Header.Size)
> + Diags.Report(Loc, diag::err_mmap_duplicate_header_attribute)
> << Str;
> + if (!Tok.is(MMToken::IntegerLiteral)) {
> + Diags.Report(Tok.getLocation(),
> + diag::err_mmap_invalid_header_attribute_value) <<
> Str;
> + skipUntil(MMToken::RBrace);
> + break;
> + }
> + Header.Size = Tok.getInteger();
> + consumeToken();
> + break;
>
> - // Record this header.
> - Module::Header H = { RelativePathName.str(), File };
> - Map.addHeader(ActiveModule, H, Role);
> + case ModTime:
> + if (Header.ModTime)
> + Diags.Report(Loc, diag::err_mmap_duplicate_header_attribute)
> << Str;
> + if (!Tok.is(MMToken::IntegerLiteral)) {
> + Diags.Report(Tok.getLocation(),
> + diag::err_mmap_invalid_header_attribute_value) <<
> Str;
> + skipUntil(MMToken::RBrace);
> + break;
> + }
> + Header.ModTime = Tok.getInteger();
> + consumeToken();
> + break;
> +
> + case Unknown:
> + Diags.Report(Loc, diag::err_mmap_expected_header_attribute);
> + skipUntil(MMToken::RBrace);
> + break;
> + }
> }
> - } else if (LeadingToken != MMToken::ExcludeKeyword) {
> - // Ignore excluded header files. They're optional anyway.
>
> - // If we find a module that has a missing header, we mark this module
> as
> - // unavailable and store the header directive for displaying
> diagnostics.
> - ActiveModule->markUnavailable();
> - ActiveModule->MissingHeaders.push_back(Header);
> + if (Tok.is(MMToken::RBrace))
> + consumeToken();
> + else {
> + Diags.Report(Tok.getLocation(), diag::err_mmap_expected_rbrace);
> + Diags.Report(LBraceLoc, diag::note_mmap_lbrace_match);
> + HadError = true;
> + }
> }
> +
> + Map.addUnresolvedHeader(ActiveModule, std::move(Header));
> }
>
> static int compareModuleHeaders(const Module::Header *A,
> @@ -2521,6 +2686,7 @@ bool ModuleMapParser::parseModuleMapFile
> case MMToken::RequiresKeyword:
> case MMToken::Star:
> case MMToken::StringLiteral:
> + case MMToken::IntegerLiteral:
> case MMToken::TextualKeyword:
> case MMToken::UmbrellaKeyword:
> case MMToken::UseKeyword:
>
> Modified: cfe/trunk/lib/Lex/PPDirectives.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Lex/
> PPDirectives.cpp?rev=304515&r1=304514&r2=304515&view=diff
> ============================================================
> ==================
> --- cfe/trunk/lib/Lex/PPDirectives.cpp (original)
> +++ cfe/trunk/lib/Lex/PPDirectives.cpp Thu Jun 1 20:55:39 2017
> @@ -689,6 +689,8 @@ Preprocessor::getModuleHeaderToIncludeFo
> while (!Loc.isInvalid() && !SM.isInMainFile(Loc)) {
> auto ID = SM.getFileID(SM.getExpansionLoc(Loc));
> auto *FE = SM.getFileEntryForID(ID);
> + if (!FE)
> + break;
>
> bool InTextualHeader = false;
> for (auto Header : HeaderInfo.getModuleMap().findAllModulesForHeader(FE))
> {
>
> Modified: cfe/trunk/lib/Serialization/ASTWriter.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/
> Serialization/ASTWriter.cpp?rev=304515&r1=304514&r2=304515&view=diff
> ============================================================
> ==================
> --- cfe/trunk/lib/Serialization/ASTWriter.cpp (original)
> +++ cfe/trunk/lib/Serialization/ASTWriter.cpp Thu Jun 1 20:55:39 2017
> @@ -1856,24 +1856,31 @@ namespace {
> // Trait used for the on-disk hash table of header search information.
> class HeaderFileInfoTrait {
> ASTWriter &Writer;
> - const HeaderSearch &HS;
>
> // Keep track of the framework names we've used during serialization.
> SmallVector<char, 128> FrameworkStringData;
> llvm::StringMap<unsigned> FrameworkNameOffset;
>
> public:
> - HeaderFileInfoTrait(ASTWriter &Writer, const HeaderSearch &HS)
> - : Writer(Writer), HS(HS) { }
> -
> + HeaderFileInfoTrait(ASTWriter &Writer) : Writer(Writer) {}
> +
> struct key_type {
> - const FileEntry *FE;
> StringRef Filename;
> + off_t Size;
> + time_t ModTime;
> };
> typedef const key_type &key_type_ref;
> +
> + using UnresolvedModule =
> + llvm::PointerIntPair<Module *, 2, ModuleMap::ModuleHeaderRole>;
>
> - typedef HeaderFileInfo data_type;
> + struct data_type {
> + const HeaderFileInfo &HFI;
> + ArrayRef<ModuleMap::KnownHeader> KnownHeaders;
> + UnresolvedModule Unresolved;
> + };
> typedef const data_type &data_type_ref;
> +
> typedef unsigned hash_value_type;
> typedef unsigned offset_type;
>
> @@ -1881,8 +1888,7 @@ namespace {
> // The hash is based only on size/time of the file, so that the
> reader can
> // match even when symlinking or excess path elements ("foo/../",
> "../")
> // change the form of the name. However, complete path is still the
> key.
> - return llvm::hash_combine(key.FE->getSize(),
> - Writer.getTimestampForOutput(key.FE));
> + return llvm::hash_combine(key.Size, key.ModTime);
> }
>
> std::pair<unsigned,unsigned>
> @@ -1892,68 +1898,74 @@ namespace {
> unsigned KeyLen = key.Filename.size() + 1 + 8 + 8;
> LE.write<uint16_t>(KeyLen);
> unsigned DataLen = 1 + 2 + 4 + 4;
> - for (auto ModInfo : HS.getModuleMap().findAllModulesForHeader(key.
> FE))
> + for (auto ModInfo : Data.KnownHeaders)
> if (Writer.getLocalOrImportedSubmoduleID(ModInfo.getModule()))
> DataLen += 4;
> + if (Data.Unresolved.getPointer())
> + DataLen += 4;
> LE.write<uint8_t>(DataLen);
> return std::make_pair(KeyLen, DataLen);
> }
> -
> +
> void EmitKey(raw_ostream& Out, key_type_ref key, unsigned KeyLen) {
> using namespace llvm::support;
> endian::Writer<little> LE(Out);
> - LE.write<uint64_t>(key.FE->getSize());
> + LE.write<uint64_t>(key.Size);
> KeyLen -= 8;
> - LE.write<uint64_t>(Writer.getTimestampForOutput(key.FE));
> + LE.write<uint64_t>(key.ModTime);
> KeyLen -= 8;
> Out.write(key.Filename.data(), KeyLen);
> }
> -
> +
> void EmitData(raw_ostream &Out, key_type_ref key,
> data_type_ref Data, unsigned DataLen) {
> using namespace llvm::support;
> endian::Writer<little> LE(Out);
> uint64_t Start = Out.tell(); (void)Start;
>
> - unsigned char Flags = (Data.isImport << 4)
> - | (Data.isPragmaOnce << 3)
> - | (Data.DirInfo << 1)
> - | Data.IndexHeaderMapHeader;
> + unsigned char Flags = (Data.HFI.isImport << 4)
> + | (Data.HFI.isPragmaOnce << 3)
> + | (Data.HFI.DirInfo << 1)
> + | Data.HFI.IndexHeaderMapHeader;
> LE.write<uint8_t>(Flags);
> - LE.write<uint16_t>(Data.NumIncludes);
> + LE.write<uint16_t>(Data.HFI.NumIncludes);
>
> - if (!Data.ControllingMacro)
> - LE.write<uint32_t>(Data.ControllingMacroID);
> + if (!Data.HFI.ControllingMacro)
> + LE.write<uint32_t>(Data.HFI.ControllingMacroID);
> else
> - LE.write<uint32_t>(Writer.getIdentifierRef(Data.
> ControllingMacro));
> -
> + LE.write<uint32_t>(Writer.getIdentifierRef(Data.HFI.
> ControllingMacro));
> +
> unsigned Offset = 0;
> - if (!Data.Framework.empty()) {
> + if (!Data.HFI.Framework.empty()) {
> // If this header refers into a framework, save the framework
> name.
> llvm::StringMap<unsigned>::iterator Pos
> - = FrameworkNameOffset.find(Data.Framework);
> + = FrameworkNameOffset.find(Data.HFI.Framework);
> if (Pos == FrameworkNameOffset.end()) {
> Offset = FrameworkStringData.size() + 1;
> - FrameworkStringData.append(Data.Framework.begin(),
> - Data.Framework.end());
> + FrameworkStringData.append(Data.HFI.Framework.begin(),
> + Data.HFI.Framework.end());
> FrameworkStringData.push_back(0);
>
> - FrameworkNameOffset[Data.Framework] = Offset;
> + FrameworkNameOffset[Data.HFI.Framework] = Offset;
> } else
> Offset = Pos->second;
> }
> LE.write<uint32_t>(Offset);
>
> - // FIXME: If the header is excluded, we should write out some
> - // record of that fact.
> - for (auto ModInfo : HS.getModuleMap().findAllModulesForHeader(key.FE))
> {
> - if (uint32_t ModID =
> - Writer.getLocalOrImportedSubmoduleID(ModInfo.getModule()))
> {
> - uint32_t Value = (ModID << 2) | (unsigned)ModInfo.getRole();
> + auto EmitModule = [&](Module *M, ModuleMap::ModuleHeaderRole Role) {
> + if (uint32_t ModID = Writer.getLocalOrImportedSubmoduleID(M)) {
> + uint32_t Value = (ModID << 2) | (unsigned)Role;
> assert((Value >> 2) == ModID && "overflow in header module
> info");
> LE.write<uint32_t>(Value);
> }
> - }
> + };
> +
> + // FIXME: If the header is excluded, we should write out some
> + // record of that fact.
> + for (auto ModInfo : Data.KnownHeaders)
> + EmitModule(ModInfo.getModule(), ModInfo.getRole());
> + if (Data.Unresolved.getPointer())
> + EmitModule(Data.Unresolved.getPointer(),
> Data.Unresolved.getInt());
>
> assert(Out.tell() - Start == DataLen && "Wrong data length");
> }
> @@ -1968,16 +1980,71 @@ namespace {
> ///
> /// \param HS The header search structure to save.
> void ASTWriter::WriteHeaderSearch(const HeaderSearch &HS) {
> + HeaderFileInfoTrait GeneratorTrait(*this);
> + llvm::OnDiskChainedHashTableGenerator<HeaderFileInfoTrait> Generator;
> + SmallVector<const char *, 4> SavedStrings;
> + unsigned NumHeaderSearchEntries = 0;
> +
> + // Find all unresolved headers for the current module. We generally will
> + // have resolved them before we get here, but not necessarily: we might
> be
> + // compiling a preprocessed module, where there is no requirement for
> the
> + // original files to exist any more.
> + if (WritingModule) {
> + llvm::SmallVector<Module *, 16> Worklist(1, WritingModule);
> + while (!Worklist.empty()) {
> + Module *M = Worklist.pop_back_val();
> + if (!M->isAvailable())
> + continue;
> +
> + // Map to disk files where possible, to pick up any missing stat
> + // information. This also means we don't need to check the
> unresolved
> + // headers list when emitting resolved headers in the first loop
> below.
> + // FIXME: It'd be preferable to avoid doing this if we were given
> + // sufficient stat information in the module map.
> + HS.getModuleMap().resolveHeaderDirectives(M);
> +
> + // If the file didn't exist, we can still create a module if we
> were given
> + // enough information in the module map.
> + for (auto U : M->MissingHeaders) {
> + // Check that we were given enough information to build a module
> + // without this file existing on disk.
> + if (!U.Size || (!U.ModTime && IncludeTimestamps)) {
> + PP->Diag(U.FileNameLoc, diag::err_module_no_size_
> mtime_for_header)
> + << WritingModule->getFullModuleName() << U.Size.hasValue()
> + << U.FileName;
> + continue;
> + }
> +
> + // Form the effective relative pathname for the file.
> + SmallString<128> Filename(M->Directory->getName());
> + llvm::sys::path::append(Filename, U.FileName);
> + PreparePathForOutput(Filename);
> +
> + StringRef FilenameDup = strdup(Filename.c_str());
> + SavedStrings.push_back(FilenameDup.data());
> +
> + HeaderFileInfoTrait::key_type Key = {
> + FilenameDup, *U.Size, IncludeTimestamps ? *U.ModTime : 0
> + };
> + HeaderFileInfoTrait::data_type Data = {
> + {}, {}, {M, ModuleMap::headerKindToRole(U.Kind)}
> + };
> + // FIXME: Deal with cases where there are multiple unresolved
> header
> + // directives in different submodules for the same header.
> + Generator.insert(Key, Data, GeneratorTrait);
> + ++NumHeaderSearchEntries;
> + }
> +
> + Worklist.append(M->submodule_begin(), M->submodule_end());
> + }
> + }
> +
> SmallVector<const FileEntry *, 16> FilesByUID;
> HS.getFileMgr().GetUniqueIDMapping(FilesByUID);
>
> if (FilesByUID.size() > HS.header_file_size())
> FilesByUID.resize(HS.header_file_size());
> -
> - HeaderFileInfoTrait GeneratorTrait(*this, HS);
> - llvm::OnDiskChainedHashTableGenerator<HeaderFileInfoTrait> Generator;
> - SmallVector<const char *, 4> SavedStrings;
> - unsigned NumHeaderSearchEntries = 0;
> +
> for (unsigned UID = 0, LastUID = FilesByUID.size(); UID != LastUID;
> ++UID) {
> const FileEntry *File = FilesByUID[UID];
> if (!File)
> @@ -2004,11 +2071,16 @@ void ASTWriter::WriteHeaderSearch(const
> SavedStrings.push_back(Filename.data());
> }
>
> - HeaderFileInfoTrait::key_type key = { File, Filename };
> - Generator.insert(key, *HFI, GeneratorTrait);
> + HeaderFileInfoTrait::key_type Key = {
> + Filename, File->getSize(), getTimestampForOutput(File)
> + };
> + HeaderFileInfoTrait::data_type Data = {
> + *HFI, HS.getModuleMap().findAllModulesForHeader(File), {}
> + };
> + Generator.insert(Key, Data, GeneratorTrait);
> ++NumHeaderSearchEntries;
> }
> -
> +
> // Create the on-disk hash table in a buffer.
> SmallString<4096> TableData;
> uint32_t BucketOffset;
>
> Added: cfe/trunk/test/Modules/Inputs/header-attribs/bar.h
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/
> Modules/Inputs/header-attribs/bar.h?rev=304515&view=auto
> ============================================================
> ==================
> --- cfe/trunk/test/Modules/Inputs/header-attribs/bar.h (added)
> +++ cfe/trunk/test/Modules/Inputs/header-attribs/bar.h Thu Jun 1
> 20:55:39 2017
> @@ -0,0 +1 @@
> +extern int b;
>
> Added: cfe/trunk/test/Modules/Inputs/header-attribs/baz.h
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/
> Modules/Inputs/header-attribs/baz.h?rev=304515&view=auto
> ============================================================
> ==================
> --- cfe/trunk/test/Modules/Inputs/header-attribs/baz.h (added)
> +++ cfe/trunk/test/Modules/Inputs/header-attribs/baz.h Thu Jun 1
> 20:55:39 2017
> @@ -0,0 +1 @@
> +extern int c;
>
> Added: cfe/trunk/test/Modules/Inputs/header-attribs/foo.h
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/
> Modules/Inputs/header-attribs/foo.h?rev=304515&view=auto
> ============================================================
> ==================
> --- cfe/trunk/test/Modules/Inputs/header-attribs/foo.h (added)
> +++ cfe/trunk/test/Modules/Inputs/header-attribs/foo.h Thu Jun 1
> 20:55:39 2017
> @@ -0,0 +1 @@
> +extern int a;
> \ No newline at end of file
>
> Added: cfe/trunk/test/Modules/Inputs/header-attribs/modular.modulemap
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/
> Modules/Inputs/header-attribs/modular.modulemap?rev=304515&view=auto
> ============================================================
> ==================
> --- cfe/trunk/test/Modules/Inputs/header-attribs/modular.modulemap (added)
> +++ cfe/trunk/test/Modules/Inputs/header-attribs/modular.modulemap Thu
> Jun 1 20:55:39 2017
> @@ -0,0 +1,5 @@
> +module A {
> + header "foo.h" { size 13 }
> + header "bar.h" { size 1000 }
> + header "baz.h" { mtime 1 }
> +}
>
> Added: cfe/trunk/test/Modules/Inputs/header-attribs/textual.modulemap
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/
> Modules/Inputs/header-attribs/textual.modulemap?rev=304515&view=auto
> ============================================================
> ==================
> --- cfe/trunk/test/Modules/Inputs/header-attribs/textual.modulemap (added)
> +++ cfe/trunk/test/Modules/Inputs/header-attribs/textual.modulemap Thu
> Jun 1 20:55:39 2017
> @@ -0,0 +1,5 @@
> +module A {
> + textual header "foo.h" { size 13 }
> + textual header "bar.h" { size 1000 }
> + textual header "baz.h" { mtime 1 }
> +}
>
> Modified: cfe/trunk/test/Modules/diagnostics.modulemap
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/
> Modules/diagnostics.modulemap?rev=304515&r1=304514&r2=304515&view=diff
> ============================================================
> ==================
> --- cfe/trunk/test/Modules/diagnostics.modulemap (original)
> +++ cfe/trunk/test/Modules/diagnostics.modulemap Thu Jun 1 20:55:39 2017
> @@ -1,4 +1,4 @@
> -// RUN: not %clang_cc1 -fmodules -fmodules-cache-path=%t
> -fmodule-map-file=%S/Inputs/diagnostics-aux.modulemap
> -fmodule-map-file=%s -fsyntax-only -x c++ /dev/null 2>&1 | FileCheck %s
> +// RUN: not %clang_cc1 -fmodules -fmodules-cache-path=%t
> -fmodule-map-file=%S/Inputs/diagnostics-aux.modulemap
> -fmodule-map-file=%s -fsyntax-only -x c++ /dev/null 2>&1 | FileCheck %s
> --implicit-check-not error:
>
> // CHECK: In file included from {{.*}}diagnostics-aux.modulemap:3:
> // CHECK: diagnostics-aux-2.modulemap:2:3: error: expected
> @@ -15,3 +15,15 @@ module bad_use {
> // CHECK: diagnostics.modulemap:[[@LINE+1]]:22: error: use
> declarations are only allowed in top-level modules
> module submodule { use foo }
> }
> +
> +module header_attr {
> + // CHECK: diagnostics.modulemap:[[@LINE+1]]:20: error: expected a
> header attribute name
> + header "foo.h" { x }
> + // CHECK: diagnostics.modulemap:[[@LINE+1]]:27: error: header
> attribute 'size' specified multiple times
> + header "bar.h" { size 1 size 2 }
> + // CHECK: diagnostics.modulemap:[[@LINE+1]]:25: error: expected
> integer literal as value for header attribute 'size'
> + header "baz.h" { size "30 kilobytes" }
> +
> + header "quux.h" { size 1 mtime 2 }
> + header "no_attrs.h" {}
> +}
>
> Added: cfe/trunk/test/Modules/header-attribs.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/
> Modules/header-attribs.cpp?rev=304515&view=auto
> ============================================================
> ==================
> --- cfe/trunk/test/Modules/header-attribs.cpp (added)
> +++ cfe/trunk/test/Modules/header-attribs.cpp Thu Jun 1 20:55:39 2017
> @@ -0,0 +1,10 @@
> +// RUN: rm -rf %t
> +// RUN: %clang_cc1 -fmodules -I%S/Inputs/header-attribs
> -fmodule-map-file=%S/Inputs/header-attribs/textual.modulemap
> -fmodules-cache-path=%t -verify %s -fmodule-name=A -fmodules-strict-decluse
> +// RUN: not %clang_cc1 -fmodules -I%S/Inputs/header-attribs -emit-module
> -x c++-module-map %S/Inputs/header-attribs/modular.modulemap
> -fmodules-cache-path=%t -fmodule-name=A 2>&1 | FileCheck %s --check-prefix
> BUILD-MODULAR
> +
> +#include "foo.h" // ok, stats match
> +#include "bar.h" // expected-error {{does not depend on a module
> exporting 'bar.h'}}
> +#include "baz.h" // expected-error {{does not depend on a module
> exporting 'baz.h'}}
> +
> +// FIXME: Explain why the 'bar.h' found on disk doesn't match the module
> map.
> +// BUILD-MODULAR: error: header 'bar.h' not found
>
> Added: cfe/trunk/test/Modules/preprocess-missing.modulemap
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/
> Modules/preprocess-missing.modulemap?rev=304515&view=auto
> ============================================================
> ==================
> --- cfe/trunk/test/Modules/preprocess-missing.modulemap (added)
> +++ cfe/trunk/test/Modules/preprocess-missing.modulemap Thu Jun 1
> 20:55:39 2017
> @@ -0,0 +1,7 @@
> +// RUN: %clang_cc1 -fmodules -fmodule-name=A -x c++-module-map %s
> -emit-module -o /dev/null -verify
> +module A {
> + header "does not exist" { size 12345 } // ok, do not need mtime for
> explicit module build
> + header "also does not exist" { mtime 12345 }
> +}
> +#pragma clang module contents
> +// expected-error at 4 {{cannot emit module A: size must be explicitly
> specified for missing header file "also does not exist"}}
>
> Modified: cfe/trunk/test/Modules/preprocess-module.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/
> Modules/preprocess-module.cpp?rev=304515&r1=304514&r2=304515&view=diff
> ============================================================
> ==================
> --- cfe/trunk/test/Modules/preprocess-module.cpp (original)
> +++ cfe/trunk/test/Modules/preprocess-module.cpp Thu Jun 1 20:55:39 2017
> @@ -28,12 +28,21 @@
> // RUN: %clang_cc1 -fmodules -fmodule-file=%t/no-rewrite.pcm %s -I%t
> -verify -fno-modules-error-recovery -DINCLUDE -I%S/Inputs/preprocess
> // RUN: %clang_cc1 -fmodules -fmodule-file=%t/rewrite.pcm %s -I%t -verify
> -fno-modules-error-recovery -DREWRITE -DINCLUDE -I%S/Inputs/preprocess
>
> +// Now try building the module when the header files are missing.
> +// RUN: cp %S/Inputs/preprocess/fwd.h %S/Inputs/preprocess/file.h
> %S/Inputs/preprocess/file2.h %S/Inputs/preprocess/module.modulemap %t
> +// RUN: %clang_cc1 -fmodules -fmodule-name=file -fmodule-file=%t/fwd.pcm
> -I%t -x c++-module-map %t/module.modulemap -E -frewrite-includes -o
> %t/copy.ii
> +// RUN: rm %t/fwd.h %t/file.h %t/file2.h %t/module.modulemap
> +// RUN: %clang_cc1 -fmodules -fmodule-name=file -fmodule-file=%t/fwd.pcm
> -x c++-module-map-cpp-output %t/copy.ii -emit-module -o %t/copy.pcm
> +
> +// Finally, check that our module contains correct mapping information
> for the headers.
> +// RUN: cp %S/Inputs/preprocess/fwd.h %S/Inputs/preprocess/file.h
> %S/Inputs/preprocess/file2.h %S/Inputs/preprocess/module.modulemap %t
> +// RUN: %clang_cc1 -fmodules -fmodule-file=%t/copy.pcm %s -I%t -verify
> -fno-modules-error-recovery -DCOPY -DINCLUDE
>
> // == module map
> // CHECK: # 1 "{{.*}}module.modulemap"
> // CHECK: module file {
> -// CHECK: header "file.h"
> -// CHECK: header "file2.h"
> +// CHECK: header "file.h" { size
> +// CHECK: header "file2.h" { size
> // CHECK: }
>
> // == file.h
> @@ -98,6 +107,8 @@
> __FILE *a; // expected-error {{declaration of '__FILE' must be imported}}
> #ifdef REWRITE
> // expected-note at rewrite.ii:1 {{here}}
> +#elif COPY
> +// expected-note at copy.ii:1 {{here}}
> #else
> // expected-note at no-rewrite.ii:1 {{here}}
> #endif
>
>
> _______________________________________________
> cfe-commits mailing list
> cfe-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20170602/b893319b/attachment-0001.html>
More information about the cfe-commits
mailing list