r304515 - Support lazy stat'ing of files referenced by module maps.
Benjamin Kramer via cfe-commits
cfe-commits at lists.llvm.org
Fri Jun 2 10:31:01 PDT 2017
I committed a workaround in r304568.
On Fri, Jun 2, 2017 at 6:59 PM, Alexander Kornienko via cfe-commits
<cfe-commits at lists.llvm.org> wrote:
> I've not yet figured out exactly, but I have a suspicion that this commit
> causes crashes when run under asan. Specifically, when running
> test/Modules/preprocess-module.cpp
>
> The stack trace is:
> #0 0x1d644c6 in (anonymous
> namespace)::HeaderFileInfoTrait::EmitData(llvm::raw_ostream&, (anonymous
> namespace)::HeaderFileInfoTrait::key_type const&, (anonymous
> namespace)::HeaderFileInfoTrait::data_type const&, unsigned int)
> llvm/tools/clang/lib/Serialization/ASTWriter.cpp:1926:39
> #1 0x1d322f3 in llvm::OnDiskChainedHashTableGenerator<(anonymous
> namespace)::HeaderFileInfoTrait>::Emit(llvm::raw_ostream&, (anonymous
> namespace)::HeaderFileInfoTrait&)
> llvm/include/llvm/Support/OnDiskHashTable.h:198:17
> #2 0x1d3168d in clang::ASTWriter::WriteHeaderSearch(clang::HeaderSearch
> const&) llvm/tools/clang/lib/Serialization/ASTWriter.cpp:2092:30
> #3 0x1d52f4c in clang::ASTWriter::WriteASTCore(clang::Sema&,
> llvm::StringRef, std::string const&, clang::Module*)
> llvm/tools/clang/lib/Serialization/ASTWriter.cpp:4857:3
> #4 0x1d4f7db in clang::ASTWriter::WriteAST(clang::Sema&, std::string
> const&, clang::Module*, llvm::StringRef, bool)
> llvm/tools/clang/lib/Serialization/ASTWriter.cpp:4475:7
> #5 0x1cd0333 in
> clang::PCHGenerator::HandleTranslationUnit(clang::ASTContext&)
> llvm/tools/clang/lib/Serialization/GeneratePCH.cpp:62:14
> #6 0x1f56cc7 in
> clang::MultiplexConsumer::HandleTranslationUnit(clang::ASTContext&)
> llvm/tools/clang/lib/Frontend/MultiplexConsumer.cpp:305:15
> #7 0x24ebc9c in clang::ParseAST(clang::Sema&, bool, bool)
> llvm/tools/clang/lib/Parse/ParseAST.cpp:159:13
> #8 0x1f4b283 in clang::FrontendAction::Execute()
> llvm/tools/clang/lib/Frontend/FrontendAction.cpp:856:8
> #9 0x1caf916 in
> clang::CompilerInstance::ExecuteAction(clang::FrontendAction&)
> llvm/tools/clang/lib/Frontend/CompilerInstance.cpp:970:11
> #10 0x5ed12c in
> clang::ExecuteCompilerInvocation(clang::CompilerInstance*)
> llvm/tools/clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:249:25
>
> Could you take a look?
>
>
> On Fri, Jun 2, 2017 at 3:55 AM, Richard Smith via cfe-commits
> <cfe-commits at lists.llvm.org> wrote:
>>
>> Author: rsmith
>> Date: Thu Jun 1 20:55:39 2017
>> New Revision: 304515
>>
>> URL: http://llvm.org/viewvc/llvm-project?rev=304515&view=rev
>> Log:
>> Support lazy stat'ing of files referenced by module maps.
>>
>> This patch adds support for a `header` declaration in a module map to
>> specify
>> certain `stat` information (currently, size and mtime) about that header
>> file.
>> This has two purposes:
>>
>> - It removes the need to eagerly `stat` every file referenced by a module
>> map.
>> Instead, we track a list of unresolved header files with each size /
>> mtime
>> (actually, for simplicity, we track submodules with such headers), and
>> when
>> attempting to look up a header file based on a `FileEntry`, we check if
>> there
>> are any unresolved header directives with that `FileEntry`'s size /
>> mtime and
>> perform deferred `stat`s if so.
>>
>> - It permits a preprocessed module to be compiled without the original
>> files
>> being present on disk. The only reason we used to need those files was
>> to get
>> the `stat` information in order to do header -> module lookups when
>> using the
>> module. If we're provided with the `stat` information in the
>> preprocessed
>> module, we can avoid requiring the files to exist.
>>
>> Unlike most `header` directives, if a `header` directive with `stat`
>> information has no corresponding on-disk file the enclosing module is
>> *not*
>> marked unavailable (so that behavior is consistent regardless of whether
>> we've
>> resolved a header directive, and so that preprocessed modules don't get
>> marked
>> unavailable). We could actually do this for all `header` directives: the
>> only
>> reason we mark the module unavailable if headers are missing is to give a
>> diagnostic slightly earlier (rather than waiting until we actually try to
>> build
>> the module / load and validate its .pcm file).
>>
>> Differential Revision: https://reviews.llvm.org/D33703
>>
>> Added:
>> cfe/trunk/test/Modules/Inputs/header-attribs/
>> cfe/trunk/test/Modules/Inputs/header-attribs/bar.h
>> cfe/trunk/test/Modules/Inputs/header-attribs/baz.h
>> cfe/trunk/test/Modules/Inputs/header-attribs/foo.h
>> cfe/trunk/test/Modules/Inputs/header-attribs/modular.modulemap
>> cfe/trunk/test/Modules/Inputs/header-attribs/textual.modulemap
>> cfe/trunk/test/Modules/header-attribs.cpp
>> cfe/trunk/test/Modules/preprocess-missing.modulemap
>> Modified:
>> cfe/trunk/docs/Modules.rst
>> cfe/trunk/include/clang/Basic/DiagnosticLexKinds.td
>> cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td
>> cfe/trunk/include/clang/Basic/Module.h
>> cfe/trunk/include/clang/Lex/ModuleMap.h
>> cfe/trunk/lib/Basic/Module.cpp
>> cfe/trunk/lib/Frontend/FrontendAction.cpp
>> cfe/trunk/lib/Lex/HeaderSearch.cpp
>> cfe/trunk/lib/Lex/ModuleMap.cpp
>> cfe/trunk/lib/Lex/PPDirectives.cpp
>> cfe/trunk/lib/Serialization/ASTWriter.cpp
>> cfe/trunk/test/Modules/diagnostics.modulemap
>> cfe/trunk/test/Modules/preprocess-module.cpp
>>
>> Modified: cfe/trunk/docs/Modules.rst
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/docs/Modules.rst?rev=304515&r1=304514&r2=304515&view=diff
>>
>> ==============================================================================
>> --- cfe/trunk/docs/Modules.rst (original)
>> +++ cfe/trunk/docs/Modules.rst Thu Jun 1 20:55:39 2017
>> @@ -469,9 +469,16 @@ A header declaration specifies that a pa
>> .. parsed-literal::
>>
>> *header-declaration*:
>> - ``private``:sub:`opt` ``textual``:sub:`opt` ``header``
>> *string-literal*
>> - ``umbrella`` ``header`` *string-literal*
>> - ``exclude`` ``header`` *string-literal*
>> + ``private``:sub:`opt` ``textual``:sub:`opt` ``header``
>> *string-literal* *header-attrs*:sub:`opt`
>> + ``umbrella`` ``header`` *string-literal* *header-attrs*:sub:`opt`
>> + ``exclude`` ``header`` *string-literal* *header-attrs*:sub:`opt`
>> +
>> + *header-attrs*:
>> + '{' *header-attr** '}'
>> +
>> + *header-attr*:
>> + ``size`` *integer-literal*
>> + ``mtime`` *integer-literal*
>>
>> A header declaration that does not contain ``exclude`` nor ``textual``
>> specifies a header that contributes to the enclosing module. Specifically,
>> when the module is built, the named header will be parsed and its
>> declarations will be (logically) placed into the enclosing submodule.
>>
>> @@ -504,6 +511,18 @@ A header with the ``exclude`` specifier
>>
>> A given header shall not be referenced by more than one
>> *header-declaration*.
>>
>> +Two *header-declaration*\s, or a *header-declaration* and a ``#include``,
>> are
>> +considered to refer to the same file if the paths resolve to the same
>> file
>> +and the specified *header-attr*\s (if any) match the attributes of that
>> file,
>> +even if the file is named differently (for instance, by a relative path
>> or
>> +via symlinks).
>> +
>> +.. note::
>> + The use of *header-attr*\s avoids the need for Clang to speculatively
>> + ``stat`` every header referenced by a module map. It is recommended
>> that
>> + *header-attr*\s only be used in machine-generated module maps, to
>> avoid
>> + mismatches between attribute values and the corresponding files.
>> +
>> Umbrella directory declaration
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> An umbrella directory declaration specifies that all of the headers in
>> the specified directory should be included within the module.
>>
>> Modified: cfe/trunk/include/clang/Basic/DiagnosticLexKinds.td
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/DiagnosticLexKinds.td?rev=304515&r1=304514&r2=304515&view=diff
>>
>> ==============================================================================
>> --- cfe/trunk/include/clang/Basic/DiagnosticLexKinds.td (original)
>> +++ cfe/trunk/include/clang/Basic/DiagnosticLexKinds.td Thu Jun 1
>> 20:55:39 2017
>> @@ -664,6 +664,12 @@ def warn_mmap_mismatched_top_level_priva
>> InGroup<PrivateModule>;
>> def note_mmap_rename_top_level_private_as_submodule : Note<
>> "make '%0' a submodule of '%1' to ensure it can be found by name">;
>> +def err_mmap_duplicate_header_attribute : Error<
>> + "header attribute '%0' specified multiple times">;
>> +def err_mmap_invalid_header_attribute_value : Error<
>> + "expected integer literal as value for header attribute '%0'">;
>> +def err_mmap_expected_header_attribute : Error<
>> + "expected a header attribute name ('size' or 'mtime')">;
>>
>> def warn_auto_module_import : Warning<
>> "treating #%select{include|import|include_next|__include_macros}0 as an
>> "
>>
>> Modified: cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td?rev=304515&r1=304514&r2=304515&view=diff
>>
>> ==============================================================================
>> --- cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td
>> (original)
>> +++ cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td Thu Jun
>> 1 20:55:39 2017
>> @@ -174,10 +174,6 @@ def note_module_odr_violation_mismatch_d
>> "method %2 with %ordinal3 parameter of type %4%select{| decayed from
>> %6}5|"
>> "method %2 with %ordinal3 parameter named %4}1">;
>>
>> -def warn_module_uses_date_time : Warning<
>> - "%select{precompiled header|module}0 uses __DATE__ or __TIME__">,
>> - InGroup<DiagGroup<"pch-date-time">>;
>> -
>> def warn_duplicate_module_file_extension : Warning<
>> "duplicate module file extension block name '%0'">,
>> InGroup<ModuleFileExtension>;
>> @@ -186,7 +182,15 @@ def warn_module_system_bit_conflict : Wa
>> "module file '%0' was validated as a system module and is now being
>> imported "
>> "as a non-system module; any difference in diagnostic options will be
>> ignored">,
>> InGroup<ModuleConflict>;
>> +} // let CategoryName
>>
>> +let CategoryName = "AST Serialization Issue" in {
>> +def warn_module_uses_date_time : Warning<
>> + "%select{precompiled header|module}0 uses __DATE__ or __TIME__">,
>> + InGroup<DiagGroup<"pch-date-time">>;
>> +def err_module_no_size_mtime_for_header : Error<
>> + "cannot emit module %0: %select{size|mtime}1 must be explicitly
>> specified "
>> + "for missing header file \"%2\"">;
>> } // let CategoryName
>> } // let Component
>>
>>
>> Modified: cfe/trunk/include/clang/Basic/Module.h
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/Module.h?rev=304515&r1=304514&r2=304515&view=diff
>>
>> ==============================================================================
>> --- cfe/trunk/include/clang/Basic/Module.h (original)
>> +++ cfe/trunk/include/clang/Basic/Module.h Thu Jun 1 20:55:39 2017
>> @@ -154,11 +154,19 @@ public:
>> /// \brief Stored information about a header directive that was found
>> in the
>> /// module map file but has not been resolved to a file.
>> struct UnresolvedHeaderDirective {
>> + HeaderKind Kind = HK_Normal;
>> SourceLocation FileNameLoc;
>> std::string FileName;
>> - bool IsUmbrella;
>> + bool IsUmbrella = false;
>> + bool HasBuiltinHeader = false;
>> + Optional<off_t> Size;
>> + Optional<time_t> ModTime;
>> };
>>
>> + /// Headers that are mentioned in the module map file but that we have
>> not
>> + /// yet attempted to resolve to a file on the file system.
>> + SmallVector<UnresolvedHeaderDirective, 1> UnresolvedHeaders;
>> +
>> /// \brief Headers that are mentioned in the module map file but could
>> not be
>> /// found on the file system.
>> SmallVector<UnresolvedHeaderDirective, 1> MissingHeaders;
>>
>> Modified: cfe/trunk/include/clang/Lex/ModuleMap.h
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Lex/ModuleMap.h?rev=304515&r1=304514&r2=304515&view=diff
>>
>> ==============================================================================
>> --- cfe/trunk/include/clang/Lex/ModuleMap.h (original)
>> +++ cfe/trunk/include/clang/Lex/ModuleMap.h Thu Jun 1 20:55:39 2017
>> @@ -26,6 +26,7 @@
>> #include "llvm/ADT/SmallVector.h"
>> #include "llvm/ADT/StringMap.h"
>> #include "llvm/ADT/StringRef.h"
>> +#include "llvm/ADT/TinyPtrVector.h"
>> #include "llvm/ADT/Twine.h"
>> #include <algorithm>
>> #include <memory>
>> @@ -116,6 +117,11 @@ public:
>> // Adjust ModuleMap::addHeader.
>> };
>>
>> + /// Convert a header kind to a role. Requires Kind to not be
>> HK_Excluded.
>> + static ModuleHeaderRole headerKindToRole(Module::HeaderKind Kind);
>> + /// Convert a header role to a kind.
>> + static Module::HeaderKind headerRoleToKind(ModuleHeaderRole Role);
>> +
>> /// \brief A header that is known to reside within a given module,
>> /// whether it was included or excluded.
>> class KnownHeader {
>> @@ -165,7 +171,13 @@ private:
>> /// \brief Mapping from each header to the module that owns the
>> contents of
>> /// that header.
>> HeadersMap Headers;
>> -
>> +
>> + /// Map from file sizes to modules with lazy header directives of that
>> size.
>> + mutable llvm::DenseMap<off_t, llvm::TinyPtrVector<Module*>>
>> LazyHeadersBySize;
>> + /// Map from mtimes to modules with lazy header directives with those
>> mtimes.
>> + mutable llvm::DenseMap<time_t, llvm::TinyPtrVector<Module*>>
>> + LazyHeadersByModTime;
>> +
>> /// \brief Mapping from directories with umbrella headers to the module
>> /// that is generated from the umbrella header.
>> ///
>> @@ -257,22 +269,30 @@ private:
>> /// resolved.
>> Module *resolveModuleId(const ModuleId &Id, Module *Mod, bool Complain)
>> const;
>>
>> - /// Resolve the given header directive to an actual header file.
>> + /// Add an unresolved header to a module.
>> + void addUnresolvedHeader(Module *Mod,
>> + Module::UnresolvedHeaderDirective Header);
>> +
>> + /// Look up the given header directive to find an actual header file.
>> ///
>> /// \param M The module in which we're resolving the header directive.
>> /// \param Header The header directive to resolve.
>> /// \param RelativePathName Filled in with the relative path name from
>> the
>> /// module to the resolved header.
>> /// \return The resolved file, if any.
>> - const FileEntry *resolveHeader(Module *M,
>> - Module::UnresolvedHeaderDirective
>> Header,
>> - SmallVectorImpl<char>
>> &RelativePathName);
>> + const FileEntry *findHeader(Module *M,
>> + const Module::UnresolvedHeaderDirective
>> &Header,
>> + SmallVectorImpl<char> &RelativePathName);
>> +
>> + /// Resolve the given header directive.
>> + void resolveHeader(Module *M,
>> + const Module::UnresolvedHeaderDirective &Header);
>>
>> /// Attempt to resolve the specified header directive as naming a
>> builtin
>> /// header.
>> - const FileEntry *
>> - resolveAsBuiltinHeader(Module *M, Module::UnresolvedHeaderDirective
>> Header,
>> - SmallVectorImpl<char> &BuiltinPathName);
>> + /// \return \c true if a corresponding builtin header was found.
>> + bool resolveAsBuiltinHeader(Module *M,
>> + const Module::UnresolvedHeaderDirective
>> &Header);
>>
>> /// \brief Looks up the modules that \p File corresponds to.
>> ///
>> @@ -368,6 +388,15 @@ public:
>> /// the preferred module for the header.
>> ArrayRef<KnownHeader> findAllModulesForHeader(const FileEntry *File)
>> const;
>>
>> + /// Resolve all lazy header directives for the specified file.
>> + ///
>> + /// This ensures that the HeaderFileInfo on HeaderSearch is up to date.
>> This
>> + /// is effectively internal, but is exposed so HeaderSearch can call
>> it.
>> + void resolveHeaderDirectives(const FileEntry *File) const;
>> +
>> + /// Resolve all lazy header directives for the specified module.
>> + void resolveHeaderDirectives(Module *Mod) const;
>> +
>> /// \brief Reports errors if a module must not include a specific file.
>> ///
>> /// \param RequestingModule The module including a file.
>>
>> Modified: cfe/trunk/lib/Basic/Module.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Basic/Module.cpp?rev=304515&r1=304514&r2=304515&view=diff
>>
>> ==============================================================================
>> --- cfe/trunk/lib/Basic/Module.cpp (original)
>> +++ cfe/trunk/lib/Basic/Module.cpp Thu Jun 1 20:55:39 2017
>> @@ -394,11 +394,30 @@ void Module::print(raw_ostream &OS, unsi
>> {"exclude ", HK_Excluded}};
>>
>> for (auto &K : Kinds) {
>> + assert(&K == &Kinds[K.Kind] && "kinds in wrong order");
>> for (auto &H : Headers[K.Kind]) {
>> OS.indent(Indent + 2);
>> OS << K.Prefix << "header \"";
>> OS.write_escaped(H.NameAsWritten);
>> - OS << "\"\n";
>> + OS << "\" { size " << H.Entry->getSize()
>> + << " mtime " << H.Entry->getModificationTime() << " }\n";
>> + }
>> + }
>> + for (auto *Unresolved : {&UnresolvedHeaders, &MissingHeaders}) {
>> + for (auto &U : *Unresolved) {
>> + OS.indent(Indent + 2);
>> + OS << Kinds[U.Kind].Prefix << "header \"";
>> + OS.write_escaped(U.FileName);
>> + OS << "\"";
>> + if (U.Size || U.ModTime) {
>> + OS << " {";
>> + if (U.Size)
>> + OS << " size " << *U.Size;
>> + if (U.ModTime)
>> + OS << " mtime " << *U.ModTime;
>> + OS << " }";
>> + }
>> + OS << "\n";
>> }
>> }
>>
>>
>> Modified: cfe/trunk/lib/Frontend/FrontendAction.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Frontend/FrontendAction.cpp?rev=304515&r1=304514&r2=304515&view=diff
>>
>> ==============================================================================
>> --- cfe/trunk/lib/Frontend/FrontendAction.cpp (original)
>> +++ cfe/trunk/lib/Frontend/FrontendAction.cpp Thu Jun 1 20:55:39 2017
>> @@ -289,14 +289,28 @@ static void addHeaderInclude(StringRef H
>> ///
>> /// \param Includes Will be augmented with the set of \#includes or
>> \#imports
>> /// needed to load all of the named headers.
>> -static std::error_code
>> -collectModuleHeaderIncludes(const LangOptions &LangOpts, FileManager
>> &FileMgr,
>> - ModuleMap &ModMap, clang::Module *Module,
>> - SmallVectorImpl<char> &Includes) {
>> +static std::error_code collectModuleHeaderIncludes(
>> + const LangOptions &LangOpts, FileManager &FileMgr, DiagnosticsEngine
>> &Diag,
>> + ModuleMap &ModMap, clang::Module *Module, SmallVectorImpl<char>
>> &Includes) {
>> // Don't collect any headers for unavailable modules.
>> if (!Module->isAvailable())
>> return std::error_code();
>>
>> + // Resolve all lazy header directives to header files.
>> + ModMap.resolveHeaderDirectives(Module);
>> +
>> + // If any headers are missing, we can't build this module. In most
>> cases,
>> + // diagnostics for this should have already been produced; we only get
>> here
>> + // if explicit stat information was provided.
>> + // FIXME: If the name resolves to a file with different stat
>> information,
>> + // produce a better diagnostic.
>> + if (!Module->MissingHeaders.empty()) {
>> + auto &MissingHeader = Module->MissingHeaders.front();
>> + Diag.Report(MissingHeader.FileNameLoc,
>> diag::err_module_header_missing)
>> + << MissingHeader.IsUmbrella << MissingHeader.FileName;
>> + return std::error_code();
>> + }
>> +
>> // Add includes for each of these headers.
>> for (auto HK : {Module::HK_Normal, Module::HK_Private}) {
>> for (Module::Header &H : Module->Headers[HK]) {
>> @@ -367,7 +381,7 @@ collectModuleHeaderIncludes(const LangOp
>> SubEnd = Module->submodule_end();
>> Sub != SubEnd; ++Sub)
>> if (std::error_code Err = collectModuleHeaderIncludes(
>> - LangOpts, FileMgr, ModMap, *Sub, Includes))
>> + LangOpts, FileMgr, Diag, ModMap, *Sub, Includes))
>> return Err;
>>
>> return std::error_code();
>> @@ -494,7 +508,7 @@ getInputBufferForModule(CompilerInstance
>> addHeaderInclude(UmbrellaHeader.NameAsWritten, HeaderContents,
>> CI.getLangOpts(), M->IsExternC);
>> Err = collectModuleHeaderIncludes(
>> - CI.getLangOpts(), FileMgr,
>> + CI.getLangOpts(), FileMgr, CI.getDiagnostics(),
>> CI.getPreprocessor().getHeaderSearchInfo().getModuleMap(), M,
>> HeaderContents);
>>
>>
>> Modified: cfe/trunk/lib/Lex/HeaderSearch.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Lex/HeaderSearch.cpp?rev=304515&r1=304514&r2=304515&view=diff
>>
>> ==============================================================================
>> --- cfe/trunk/lib/Lex/HeaderSearch.cpp (original)
>> +++ cfe/trunk/lib/Lex/HeaderSearch.cpp Thu Jun 1 20:55:39 2017
>> @@ -1114,6 +1114,8 @@ bool HeaderSearch::ShouldEnterIncludeFil
>> auto TryEnterImported = [&](void) -> bool {
>> if (!ModulesEnabled)
>> return false;
>> + // Ensure FileInfo bits are up to date.
>> + ModMap.resolveHeaderDirectives(File);
>> // Modules with builtins are special; multiple modules use builtins
>> as
>> // modular headers, example:
>> //
>>
>> Modified: cfe/trunk/lib/Lex/ModuleMap.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Lex/ModuleMap.cpp?rev=304515&r1=304514&r2=304515&view=diff
>>
>> ==============================================================================
>> --- cfe/trunk/lib/Lex/ModuleMap.cpp (original)
>> +++ cfe/trunk/lib/Lex/ModuleMap.cpp Thu Jun 1 20:55:39 2017
>> @@ -36,6 +36,37 @@
>> #endif
>> using namespace clang;
>>
>> +Module::HeaderKind ModuleMap::headerRoleToKind(ModuleHeaderRole Role) {
>> + switch ((int)Role) {
>> + default: llvm_unreachable("unknown header role");
>> + case NormalHeader:
>> + return Module::HK_Normal;
>> + case PrivateHeader:
>> + return Module::HK_Private;
>> + case TextualHeader:
>> + return Module::HK_Textual;
>> + case PrivateHeader | TextualHeader:
>> + return Module::HK_PrivateTextual;
>> + }
>> +}
>> +
>> +ModuleMap::ModuleHeaderRole
>> +ModuleMap::headerKindToRole(Module::HeaderKind Kind) {
>> + switch ((int)Kind) {
>> + case Module::HK_Normal:
>> + return NormalHeader;
>> + case Module::HK_Private:
>> + return PrivateHeader;
>> + case Module::HK_Textual:
>> + return TextualHeader;
>> + case Module::HK_PrivateTextual:
>> + return ModuleHeaderRole(PrivateHeader | TextualHeader);
>> + case Module::HK_Excluded:
>> + llvm_unreachable("unexpected header kind");
>> + }
>> + llvm_unreachable("unknown header kind");
>> +}
>> +
>> Module::ExportDecl
>> ModuleMap::resolveExport(Module *Mod,
>> const Module::UnresolvedExportDecl &Unresolved,
>> @@ -104,12 +135,22 @@ static void appendSubframeworkPaths(Modu
>> }
>>
>> const FileEntry *
>> -ModuleMap::resolveHeader(Module *M, Module::UnresolvedHeaderDirective
>> Header,
>> - SmallVectorImpl<char> &RelativePathName) {
>> +ModuleMap::findHeader(Module *M,
>> + const Module::UnresolvedHeaderDirective &Header,
>> + SmallVectorImpl<char> &RelativePathName) {
>> + auto GetFile = [&](StringRef Filename) -> const FileEntry * {
>> + auto *File = SourceMgr.getFileManager().getFile(Filename);
>> + if (!File ||
>> + (Header.Size && File->getSize() != *Header.Size) ||
>> + (Header.ModTime && File->getModificationTime() !=
>> *Header.ModTime))
>> + return nullptr;
>> + return File;
>> + };
>> +
>> if (llvm::sys::path::is_absolute(Header.FileName)) {
>> RelativePathName.clear();
>> RelativePathName.append(Header.FileName.begin(),
>> Header.FileName.end());
>> - return SourceMgr.getFileManager().getFile(Header.FileName);
>> + return GetFile(Header.FileName);
>> }
>>
>> // Search for the header file within the module's home directory.
>> @@ -124,7 +165,7 @@ ModuleMap::resolveHeader(Module *M, Modu
>> // Check whether this file is in the public headers.
>> llvm::sys::path::append(RelativePathName, "Headers",
>> Header.FileName);
>> llvm::sys::path::append(FullPathName, RelativePathName);
>> - if (auto *File = SourceMgr.getFileManager().getFile(FullPathName))
>> + if (auto *File = GetFile(FullPathName))
>> return File;
>>
>> // Check whether this file is in the private headers.
>> @@ -141,31 +182,74 @@ ModuleMap::resolveHeader(Module *M, Modu
>> llvm::sys::path::append(RelativePathName, "PrivateHeaders",
>> Header.FileName);
>> llvm::sys::path::append(FullPathName, RelativePathName);
>> - return SourceMgr.getFileManager().getFile(FullPathName);
>> + return GetFile(FullPathName);
>> }
>>
>> // Lookup for normal headers.
>> llvm::sys::path::append(RelativePathName, Header.FileName);
>> llvm::sys::path::append(FullPathName, RelativePathName);
>> - return SourceMgr.getFileManager().getFile(FullPathName);
>> + return GetFile(FullPathName);
>> }
>>
>> -const FileEntry *
>> -ModuleMap::resolveAsBuiltinHeader(Module *M,
>> - Module::UnresolvedHeaderDirective
>> Header,
>> - SmallVectorImpl<char> &BuiltinPathName)
>> {
>> - if (llvm::sys::path::is_absolute(Header.FileName) ||
>> M->isPartOfFramework() ||
>> - !M->IsSystem || Header.IsUmbrella || !BuiltinIncludeDir ||
>> - BuiltinIncludeDir == M->Directory ||
>> !isBuiltinHeader(Header.FileName))
>> - return nullptr;
>> +void ModuleMap::resolveHeader(Module *Mod,
>> + const Module::UnresolvedHeaderDirective
>> &Header) {
>> + SmallString<128> RelativePathName;
>> + if (const FileEntry *File = findHeader(Mod, Header, RelativePathName))
>> {
>> + if (Header.IsUmbrella) {
>> + const DirectoryEntry *UmbrellaDir = File->getDir();
>> + if (Module *UmbrellaMod = UmbrellaDirs[UmbrellaDir])
>> + Diags.Report(Header.FileNameLoc, diag::err_mmap_umbrella_clash)
>> + << UmbrellaMod->getFullModuleName();
>> + else
>> + // Record this umbrella header.
>> + setUmbrellaHeader(Mod, File, RelativePathName.str());
>> + } else {
>> + Module::Header H = {RelativePathName.str(), File};
>> + if (Header.Kind == Module::HK_Excluded)
>> + excludeHeader(Mod, H);
>> + else
>> + addHeader(Mod, H, headerKindToRole(Header.Kind));
>> + }
>> + } else if (Header.HasBuiltinHeader && !Header.Size && !Header.ModTime)
>> {
>> + // There's a builtin header but no corresponding on-disk header.
>> Assume
>> + // this was supposed to modularize the builtin header alone.
>> + } else if (Header.Kind == Module::HK_Excluded) {
>> + // Ignore missing excluded header files. They're optional anyway.
>> + } else {
>> + // If we find a module that has a missing header, we mark this module
>> as
>> + // unavailable and store the header directive for displaying
>> diagnostics.
>> + Mod->MissingHeaders.push_back(Header);
>> + // A missing header with stat information doesn't make the module
>> + // unavailable; this keeps our behavior consistent as headers are
>> lazily
>> + // resolved. (Such a module still can't be built though, except from
>> + // preprocessed source.)
>> + if (!Header.Size && !Header.ModTime)
>> + Mod->markUnavailable();
>> + }
>> +}
>> +
>> +bool ModuleMap::resolveAsBuiltinHeader(
>> + Module *Mod, const Module::UnresolvedHeaderDirective &Header) {
>> + if (Header.Kind == Module::HK_Excluded ||
>> + llvm::sys::path::is_absolute(Header.FileName) ||
>> + Mod->isPartOfFramework() || !Mod->IsSystem || Header.IsUmbrella ||
>> + !BuiltinIncludeDir || BuiltinIncludeDir == Mod->Directory ||
>> + !isBuiltinHeader(Header.FileName))
>> + return false;
>>
>> // This is a system module with a top-level header. This header
>> // may have a counterpart (or replacement) in the set of headers
>> // supplied by Clang. Find that builtin header.
>> - llvm::sys::path::append(BuiltinPathName, BuiltinIncludeDir->getName(),
>> - Header.FileName);
>> - return SourceMgr.getFileManager().getFile(
>> - StringRef(BuiltinPathName.data(), BuiltinPathName.size()));
>> + SmallString<128> Path;
>> + llvm::sys::path::append(Path, BuiltinIncludeDir->getName(),
>> Header.FileName);
>> + auto *File = SourceMgr.getFileManager().getFile(Path);
>> + if (!File)
>> + return false;
>> +
>> + auto Role = headerKindToRole(Header.Kind);
>> + Module::Header H = {Path.str(), File};
>> + addHeader(Mod, H, Role);
>> + return true;
>> }
>>
>> ModuleMap::ModuleMap(SourceManager &SourceMgr, DiagnosticsEngine &Diags,
>> @@ -246,6 +330,7 @@ bool ModuleMap::isBuiltinHeader(StringRe
>>
>> ModuleMap::HeadersMap::iterator
>> ModuleMap::findKnownHeader(const FileEntry *File) {
>> + resolveHeaderDirectives(File);
>> HeadersMap::iterator Known = Headers.find(File);
>> if (HeaderInfo.getHeaderSearchOpts().ImplicitModuleMaps &&
>> Known == Headers.end() && File->getDir() == BuiltinIncludeDir &&
>> @@ -328,8 +413,10 @@ void ModuleMap::diagnoseHeaderInclusion(
>> if (getTopLevelOrNull(RequestingModule) !=
>> getTopLevelOrNull(SourceModule))
>> return;
>>
>> - if (RequestingModule)
>> + if (RequestingModule) {
>> resolveUses(RequestingModule, /*Complain=*/false);
>> + resolveHeaderDirectives(RequestingModule);
>> + }
>>
>> bool Excluded = false;
>> Module *Private = nullptr;
>> @@ -511,6 +598,7 @@ ModuleMap::findOrCreateModuleForHeaderIn
>>
>> ArrayRef<ModuleMap::KnownHeader>
>> ModuleMap::findAllModulesForHeader(const FileEntry *File) const {
>> + resolveHeaderDirectives(File);
>> auto It = Headers.find(File);
>> if (It == Headers.end())
>> return None;
>> @@ -524,6 +612,7 @@ bool ModuleMap::isHeaderInUnavailableMod
>> bool
>> ModuleMap::isHeaderUnavailableInModule(const FileEntry *Header,
>> const Module *RequestingModule)
>> const {
>> + resolveHeaderDirectives(Header);
>> HeadersMap::const_iterator Known = Headers.find(Header);
>> if (Known != Headers.end()) {
>> for (SmallVectorImpl<KnownHeader>::const_iterator
>> @@ -896,20 +985,65 @@ void ModuleMap::setUmbrellaDir(Module *M
>> UmbrellaDirs[UmbrellaDir] = Mod;
>> }
>>
>> -static Module::HeaderKind headerRoleToKind(ModuleMap::ModuleHeaderRole
>> Role) {
>> - switch ((int)Role) {
>> - default: llvm_unreachable("unknown header role");
>> - case ModuleMap::NormalHeader:
>> - return Module::HK_Normal;
>> - case ModuleMap::PrivateHeader:
>> - return Module::HK_Private;
>> - case ModuleMap::TextualHeader:
>> - return Module::HK_Textual;
>> - case ModuleMap::PrivateHeader | ModuleMap::TextualHeader:
>> - return Module::HK_PrivateTextual;
>> +void ModuleMap::addUnresolvedHeader(Module *Mod,
>> + Module::UnresolvedHeaderDirective
>> Header) {
>> + // If there is a builtin counterpart to this file, add it now so it can
>> + // wrap the system header.
>> + if (resolveAsBuiltinHeader(Mod, Header)) {
>> + // If we have both a builtin and system version of the file, the
>> + // builtin version may want to inject macros into the system header,
>> so
>> + // force the system header to be treated as a textual header in this
>> + // case.
>> + Header.Kind = headerRoleToKind(ModuleMap::ModuleHeaderRole(
>> + headerKindToRole(Header.Kind) | ModuleMap::TextualHeader));
>> + Header.HasBuiltinHeader = true;
>> + }
>> +
>> + // If possible, don't stat the header until we need to. This requires
>> the
>> + // user to have provided us with some stat information about the file.
>> + // FIXME: Add support for lazily stat'ing umbrella headers and excluded
>> + // headers.
>> + if ((Header.Size || Header.ModTime) && !Header.IsUmbrella &&
>> + Header.Kind != Module::HK_Excluded) {
>> + // We expect more variation in mtime than size, so if we're given
>> both,
>> + // use the mtime as the key.
>> + if (Header.ModTime)
>> + LazyHeadersByModTime[*Header.ModTime].push_back(Mod);
>> + else
>> + LazyHeadersBySize[*Header.Size].push_back(Mod);
>> + Mod->UnresolvedHeaders.push_back(Header);
>> + return;
>> + }
>> +
>> + // We don't have stat information or can't defer looking this file up.
>> + // Perform the lookup now.
>> + resolveHeader(Mod, Header);
>> +}
>> +
>> +void ModuleMap::resolveHeaderDirectives(const FileEntry *File) const {
>> + auto BySize = LazyHeadersBySize.find(File->getSize());
>> + if (BySize != LazyHeadersBySize.end()) {
>> + for (auto *M : BySize->second)
>> + resolveHeaderDirectives(M);
>> + LazyHeadersBySize.erase(BySize);
>> + }
>> +
>> + auto ByModTime =
>> LazyHeadersByModTime.find(File->getModificationTime());
>> + if (ByModTime != LazyHeadersByModTime.end()) {
>> + for (auto *M : ByModTime->second)
>> + resolveHeaderDirectives(M);
>> + LazyHeadersByModTime.erase(ByModTime);
>> }
>> }
>>
>> +void ModuleMap::resolveHeaderDirectives(Module *Mod) const {
>> + for (auto &Header : Mod->UnresolvedHeaders)
>> + // This operation is logically const; we're just changing how we
>> represent
>> + // the header information for this file.
>> + const_cast<ModuleMap*>(this)->resolveHeader(Mod, Header);
>> + Mod->UnresolvedHeaders.clear();
>> +}
>> +
>> void ModuleMap::addHeader(Module *Mod, Module::Header Header,
>> ModuleHeaderRole Role, bool Imported) {
>> KnownHeader KH(Mod, Role);
>> @@ -1063,6 +1197,7 @@ namespace clang {
>> RequiresKeyword,
>> Star,
>> StringLiteral,
>> + IntegerLiteral,
>> TextualKeyword,
>> LBrace,
>> RBrace,
>> @@ -1072,7 +1207,12 @@ namespace clang {
>>
>> unsigned Location;
>> unsigned StringLength;
>> - const char *StringData;
>> + union {
>> + // If Kind != IntegerLiteral.
>> + const char *StringData;
>> + // If Kind == IntegerLiteral.
>> + uint64_t IntegerValue;
>> + };
>>
>> void clear() {
>> Kind = EndOfFile;
>> @@ -1086,9 +1226,14 @@ namespace clang {
>> SourceLocation getLocation() const {
>> return SourceLocation::getFromRawEncoding(Location);
>> }
>> +
>> + uint64_t getInteger() const {
>> + return Kind == IntegerLiteral ? IntegerValue : 0;
>> + }
>>
>> StringRef getString() const {
>> - return StringRef(StringData, StringLength);
>> + return Kind == IntegerLiteral ? StringRef()
>> + : StringRef(StringData,
>> StringLength);
>> }
>> };
>>
>> @@ -1278,6 +1423,25 @@ retry:
>> Tok.StringLength = Length;
>> break;
>> }
>> +
>> + case tok::numeric_constant: {
>> + // We don't support any suffixes or other complications.
>> + SmallString<32> SpellingBuffer;
>> + SpellingBuffer.resize(LToken.getLength() + 1);
>> + const char *Start = SpellingBuffer.data();
>> + unsigned Length =
>> + Lexer::getSpelling(LToken, Start, SourceMgr, L.getLangOpts());
>> + uint64_t Value;
>> + if (StringRef(Start, Length).getAsInteger(0, Value)) {
>> + Diags.Report(Tok.getLocation(), diag::err_mmap_unknown_token);
>> + HadError = true;
>> + goto retry;
>> + }
>> +
>> + Tok.Kind = MMToken::IntegerLiteral;
>> + Tok.IntegerValue = Value;
>> + break;
>> + }
>>
>> case tok::comment:
>> goto retry;
>> @@ -1904,6 +2068,9 @@ void ModuleMapParser::parseHeaderDecl(MM
>> Header.FileName = Tok.getString();
>> Header.FileNameLoc = consumeToken();
>> Header.IsUmbrella = LeadingToken == MMToken::UmbrellaKeyword;
>> + Header.Kind =
>> + (LeadingToken == MMToken::ExcludeKeyword ? Module::HK_Excluded
>> + :
>> Map.headerRoleToKind(Role));
>>
>> // Check whether we already have an umbrella.
>> if (Header.IsUmbrella && ActiveModule->Umbrella) {
>> @@ -1913,64 +2080,62 @@ void ModuleMapParser::parseHeaderDecl(MM
>> return;
>> }
>>
>> - // Look for this file by name if we don't have any stat information.
>> - SmallString<128> RelativePathName, BuiltinPathName;
>> - const FileEntry *File =
>> - Map.resolveHeader(ActiveModule, Header, RelativePathName);
>> - const FileEntry *BuiltinFile =
>> - Map.resolveAsBuiltinHeader(ActiveModule, Header, BuiltinPathName);
>> -
>> - // If Clang supplies this header but the underlying system does not,
>> - // just silently swap in our builtin version. Otherwise, we'll end
>> - // up adding both (later).
>> - if (BuiltinFile && !File) {
>> - RelativePathName = BuiltinPathName;
>> - File = BuiltinFile;
>> - BuiltinFile = nullptr;
>> - }
>> -
>> - // FIXME: We shouldn't be eagerly stat'ing every file named in a module
>> map.
>> - // Come up with a lazy way to do this.
>> - if (File) {
>> - if (Header.IsUmbrella) {
>> - const DirectoryEntry *UmbrellaDir = File->getDir();
>> - if (Module *UmbrellaModule = Map.UmbrellaDirs[UmbrellaDir]) {
>> - Diags.Report(LeadingLoc, diag::err_mmap_umbrella_clash)
>> - << UmbrellaModule->getFullModuleName();
>> - HadError = true;
>> - } else {
>> - // Record this umbrella header.
>> - Map.setUmbrellaHeader(ActiveModule, File,
>> RelativePathName.str());
>> - }
>> - } else if (LeadingToken == MMToken::ExcludeKeyword) {
>> - Module::Header H = {RelativePathName.str(), File};
>> - Map.excludeHeader(ActiveModule, H);
>> - } else {
>> - // If there is a builtin counterpart to this file, add it now so it
>> can
>> - // wrap the system header.
>> - if (BuiltinFile) {
>> - Module::Header H = { BuiltinPathName.str(), BuiltinFile };
>> - Map.addHeader(ActiveModule, H, Role);
>> -
>> - // If we have both a builtin and system version of the file, the
>> - // builtin version may want to inject macros into the system
>> header, so
>> - // force the system header to be treated as a textual header in
>> this
>> - // case.
>> - Role = ModuleMap::ModuleHeaderRole(Role |
>> ModuleMap::TextualHeader);
>> - }
>> + // If we were given stat information, parse it so we can skip looking
>> for
>> + // the file.
>> + if (Tok.is(MMToken::LBrace)) {
>> + SourceLocation LBraceLoc = consumeToken();
>> +
>> + while (!Tok.is(MMToken::RBrace) && !Tok.is(MMToken::EndOfFile)) {
>> + enum Attribute { Size, ModTime, Unknown };
>> + StringRef Str = Tok.getString();
>> + SourceLocation Loc = consumeToken();
>> + switch (llvm::StringSwitch<Attribute>(Str)
>> + .Case("size", Size)
>> + .Case("mtime", ModTime)
>> + .Default(Unknown)) {
>> + case Size:
>> + if (Header.Size)
>> + Diags.Report(Loc, diag::err_mmap_duplicate_header_attribute) <<
>> Str;
>> + if (!Tok.is(MMToken::IntegerLiteral)) {
>> + Diags.Report(Tok.getLocation(),
>> + diag::err_mmap_invalid_header_attribute_value) <<
>> Str;
>> + skipUntil(MMToken::RBrace);
>> + break;
>> + }
>> + Header.Size = Tok.getInteger();
>> + consumeToken();
>> + break;
>>
>> - // Record this header.
>> - Module::Header H = { RelativePathName.str(), File };
>> - Map.addHeader(ActiveModule, H, Role);
>> + case ModTime:
>> + if (Header.ModTime)
>> + Diags.Report(Loc, diag::err_mmap_duplicate_header_attribute) <<
>> Str;
>> + if (!Tok.is(MMToken::IntegerLiteral)) {
>> + Diags.Report(Tok.getLocation(),
>> + diag::err_mmap_invalid_header_attribute_value) <<
>> Str;
>> + skipUntil(MMToken::RBrace);
>> + break;
>> + }
>> + Header.ModTime = Tok.getInteger();
>> + consumeToken();
>> + break;
>> +
>> + case Unknown:
>> + Diags.Report(Loc, diag::err_mmap_expected_header_attribute);
>> + skipUntil(MMToken::RBrace);
>> + break;
>> + }
>> }
>> - } else if (LeadingToken != MMToken::ExcludeKeyword) {
>> - // Ignore excluded header files. They're optional anyway.
>>
>> - // If we find a module that has a missing header, we mark this module
>> as
>> - // unavailable and store the header directive for displaying
>> diagnostics.
>> - ActiveModule->markUnavailable();
>> - ActiveModule->MissingHeaders.push_back(Header);
>> + if (Tok.is(MMToken::RBrace))
>> + consumeToken();
>> + else {
>> + Diags.Report(Tok.getLocation(), diag::err_mmap_expected_rbrace);
>> + Diags.Report(LBraceLoc, diag::note_mmap_lbrace_match);
>> + HadError = true;
>> + }
>> }
>> +
>> + Map.addUnresolvedHeader(ActiveModule, std::move(Header));
>> }
>>
>> static int compareModuleHeaders(const Module::Header *A,
>> @@ -2521,6 +2686,7 @@ bool ModuleMapParser::parseModuleMapFile
>> case MMToken::RequiresKeyword:
>> case MMToken::Star:
>> case MMToken::StringLiteral:
>> + case MMToken::IntegerLiteral:
>> case MMToken::TextualKeyword:
>> case MMToken::UmbrellaKeyword:
>> case MMToken::UseKeyword:
>>
>> Modified: cfe/trunk/lib/Lex/PPDirectives.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Lex/PPDirectives.cpp?rev=304515&r1=304514&r2=304515&view=diff
>>
>> ==============================================================================
>> --- cfe/trunk/lib/Lex/PPDirectives.cpp (original)
>> +++ cfe/trunk/lib/Lex/PPDirectives.cpp Thu Jun 1 20:55:39 2017
>> @@ -689,6 +689,8 @@ Preprocessor::getModuleHeaderToIncludeFo
>> while (!Loc.isInvalid() && !SM.isInMainFile(Loc)) {
>> auto ID = SM.getFileID(SM.getExpansionLoc(Loc));
>> auto *FE = SM.getFileEntryForID(ID);
>> + if (!FE)
>> + break;
>>
>> bool InTextualHeader = false;
>> for (auto Header :
>> HeaderInfo.getModuleMap().findAllModulesForHeader(FE)) {
>>
>> Modified: cfe/trunk/lib/Serialization/ASTWriter.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Serialization/ASTWriter.cpp?rev=304515&r1=304514&r2=304515&view=diff
>>
>> ==============================================================================
>> --- cfe/trunk/lib/Serialization/ASTWriter.cpp (original)
>> +++ cfe/trunk/lib/Serialization/ASTWriter.cpp Thu Jun 1 20:55:39 2017
>> @@ -1856,24 +1856,31 @@ namespace {
>> // Trait used for the on-disk hash table of header search information.
>> class HeaderFileInfoTrait {
>> ASTWriter &Writer;
>> - const HeaderSearch &HS;
>>
>> // Keep track of the framework names we've used during serialization.
>> SmallVector<char, 128> FrameworkStringData;
>> llvm::StringMap<unsigned> FrameworkNameOffset;
>>
>> public:
>> - HeaderFileInfoTrait(ASTWriter &Writer, const HeaderSearch &HS)
>> - : Writer(Writer), HS(HS) { }
>> -
>> + HeaderFileInfoTrait(ASTWriter &Writer) : Writer(Writer) {}
>> +
>> struct key_type {
>> - const FileEntry *FE;
>> StringRef Filename;
>> + off_t Size;
>> + time_t ModTime;
>> };
>> typedef const key_type &key_type_ref;
>> +
>> + using UnresolvedModule =
>> + llvm::PointerIntPair<Module *, 2, ModuleMap::ModuleHeaderRole>;
>>
>> - typedef HeaderFileInfo data_type;
>> + struct data_type {
>> + const HeaderFileInfo &HFI;
>> + ArrayRef<ModuleMap::KnownHeader> KnownHeaders;
>> + UnresolvedModule Unresolved;
>> + };
>> typedef const data_type &data_type_ref;
>> +
>> typedef unsigned hash_value_type;
>> typedef unsigned offset_type;
>>
>> @@ -1881,8 +1888,7 @@ namespace {
>> // The hash is based only on size/time of the file, so that the
>> reader can
>> // match even when symlinking or excess path elements ("foo/../",
>> "../")
>> // change the form of the name. However, complete path is still the
>> key.
>> - return llvm::hash_combine(key.FE->getSize(),
>> - Writer.getTimestampForOutput(key.FE));
>> + return llvm::hash_combine(key.Size, key.ModTime);
>> }
>>
>> std::pair<unsigned,unsigned>
>> @@ -1892,68 +1898,74 @@ namespace {
>> unsigned KeyLen = key.Filename.size() + 1 + 8 + 8;
>> LE.write<uint16_t>(KeyLen);
>> unsigned DataLen = 1 + 2 + 4 + 4;
>> - for (auto ModInfo :
>> HS.getModuleMap().findAllModulesForHeader(key.FE))
>> + for (auto ModInfo : Data.KnownHeaders)
>> if (Writer.getLocalOrImportedSubmoduleID(ModInfo.getModule()))
>> DataLen += 4;
>> + if (Data.Unresolved.getPointer())
>> + DataLen += 4;
>> LE.write<uint8_t>(DataLen);
>> return std::make_pair(KeyLen, DataLen);
>> }
>> -
>> +
>> void EmitKey(raw_ostream& Out, key_type_ref key, unsigned KeyLen) {
>> using namespace llvm::support;
>> endian::Writer<little> LE(Out);
>> - LE.write<uint64_t>(key.FE->getSize());
>> + LE.write<uint64_t>(key.Size);
>> KeyLen -= 8;
>> - LE.write<uint64_t>(Writer.getTimestampForOutput(key.FE));
>> + LE.write<uint64_t>(key.ModTime);
>> KeyLen -= 8;
>> Out.write(key.Filename.data(), KeyLen);
>> }
>> -
>> +
>> void EmitData(raw_ostream &Out, key_type_ref key,
>> data_type_ref Data, unsigned DataLen) {
>> using namespace llvm::support;
>> endian::Writer<little> LE(Out);
>> uint64_t Start = Out.tell(); (void)Start;
>>
>> - unsigned char Flags = (Data.isImport << 4)
>> - | (Data.isPragmaOnce << 3)
>> - | (Data.DirInfo << 1)
>> - | Data.IndexHeaderMapHeader;
>> + unsigned char Flags = (Data.HFI.isImport << 4)
>> + | (Data.HFI.isPragmaOnce << 3)
>> + | (Data.HFI.DirInfo << 1)
>> + | Data.HFI.IndexHeaderMapHeader;
>> LE.write<uint8_t>(Flags);
>> - LE.write<uint16_t>(Data.NumIncludes);
>> + LE.write<uint16_t>(Data.HFI.NumIncludes);
>>
>> - if (!Data.ControllingMacro)
>> - LE.write<uint32_t>(Data.ControllingMacroID);
>> + if (!Data.HFI.ControllingMacro)
>> + LE.write<uint32_t>(Data.HFI.ControllingMacroID);
>> else
>> -
>> LE.write<uint32_t>(Writer.getIdentifierRef(Data.ControllingMacro));
>> -
>> +
>> LE.write<uint32_t>(Writer.getIdentifierRef(Data.HFI.ControllingMacro));
>> +
>> unsigned Offset = 0;
>> - if (!Data.Framework.empty()) {
>> + if (!Data.HFI.Framework.empty()) {
>> // If this header refers into a framework, save the framework
>> name.
>> llvm::StringMap<unsigned>::iterator Pos
>> - = FrameworkNameOffset.find(Data.Framework);
>> + = FrameworkNameOffset.find(Data.HFI.Framework);
>> if (Pos == FrameworkNameOffset.end()) {
>> Offset = FrameworkStringData.size() + 1;
>> - FrameworkStringData.append(Data.Framework.begin(),
>> - Data.Framework.end());
>> + FrameworkStringData.append(Data.HFI.Framework.begin(),
>> + Data.HFI.Framework.end());
>> FrameworkStringData.push_back(0);
>>
>> - FrameworkNameOffset[Data.Framework] = Offset;
>> + FrameworkNameOffset[Data.HFI.Framework] = Offset;
>> } else
>> Offset = Pos->second;
>> }
>> LE.write<uint32_t>(Offset);
>>
>> - // FIXME: If the header is excluded, we should write out some
>> - // record of that fact.
>> - for (auto ModInfo :
>> HS.getModuleMap().findAllModulesForHeader(key.FE)) {
>> - if (uint32_t ModID =
>> -
>> Writer.getLocalOrImportedSubmoduleID(ModInfo.getModule())) {
>> - uint32_t Value = (ModID << 2) | (unsigned)ModInfo.getRole();
>> + auto EmitModule = [&](Module *M, ModuleMap::ModuleHeaderRole Role)
>> {
>> + if (uint32_t ModID = Writer.getLocalOrImportedSubmoduleID(M)) {
>> + uint32_t Value = (ModID << 2) | (unsigned)Role;
>> assert((Value >> 2) == ModID && "overflow in header module
>> info");
>> LE.write<uint32_t>(Value);
>> }
>> - }
>> + };
>> +
>> + // FIXME: If the header is excluded, we should write out some
>> + // record of that fact.
>> + for (auto ModInfo : Data.KnownHeaders)
>> + EmitModule(ModInfo.getModule(), ModInfo.getRole());
>> + if (Data.Unresolved.getPointer())
>> + EmitModule(Data.Unresolved.getPointer(),
>> Data.Unresolved.getInt());
>>
>> assert(Out.tell() - Start == DataLen && "Wrong data length");
>> }
>> @@ -1968,16 +1980,71 @@ namespace {
>> ///
>> /// \param HS The header search structure to save.
>> void ASTWriter::WriteHeaderSearch(const HeaderSearch &HS) {
>> + HeaderFileInfoTrait GeneratorTrait(*this);
>> + llvm::OnDiskChainedHashTableGenerator<HeaderFileInfoTrait> Generator;
>> + SmallVector<const char *, 4> SavedStrings;
>> + unsigned NumHeaderSearchEntries = 0;
>> +
>> + // Find all unresolved headers for the current module. We generally
>> will
>> + // have resolved them before we get here, but not necessarily: we might
>> be
>> + // compiling a preprocessed module, where there is no requirement for
>> the
>> + // original files to exist any more.
>> + if (WritingModule) {
>> + llvm::SmallVector<Module *, 16> Worklist(1, WritingModule);
>> + while (!Worklist.empty()) {
>> + Module *M = Worklist.pop_back_val();
>> + if (!M->isAvailable())
>> + continue;
>> +
>> + // Map to disk files where possible, to pick up any missing stat
>> + // information. This also means we don't need to check the
>> unresolved
>> + // headers list when emitting resolved headers in the first loop
>> below.
>> + // FIXME: It'd be preferable to avoid doing this if we were given
>> + // sufficient stat information in the module map.
>> + HS.getModuleMap().resolveHeaderDirectives(M);
>> +
>> + // If the file didn't exist, we can still create a module if we
>> were given
>> + // enough information in the module map.
>> + for (auto U : M->MissingHeaders) {
>> + // Check that we were given enough information to build a module
>> + // without this file existing on disk.
>> + if (!U.Size || (!U.ModTime && IncludeTimestamps)) {
>> + PP->Diag(U.FileNameLoc,
>> diag::err_module_no_size_mtime_for_header)
>> + << WritingModule->getFullModuleName() << U.Size.hasValue()
>> + << U.FileName;
>> + continue;
>> + }
>> +
>> + // Form the effective relative pathname for the file.
>> + SmallString<128> Filename(M->Directory->getName());
>> + llvm::sys::path::append(Filename, U.FileName);
>> + PreparePathForOutput(Filename);
>> +
>> + StringRef FilenameDup = strdup(Filename.c_str());
>> + SavedStrings.push_back(FilenameDup.data());
>> +
>> + HeaderFileInfoTrait::key_type Key = {
>> + FilenameDup, *U.Size, IncludeTimestamps ? *U.ModTime : 0
>> + };
>> + HeaderFileInfoTrait::data_type Data = {
>> + {}, {}, {M, ModuleMap::headerKindToRole(U.Kind)}
>> + };
>> + // FIXME: Deal with cases where there are multiple unresolved
>> header
>> + // directives in different submodules for the same header.
>> + Generator.insert(Key, Data, GeneratorTrait);
>> + ++NumHeaderSearchEntries;
>> + }
>> +
>> + Worklist.append(M->submodule_begin(), M->submodule_end());
>> + }
>> + }
>> +
>> SmallVector<const FileEntry *, 16> FilesByUID;
>> HS.getFileMgr().GetUniqueIDMapping(FilesByUID);
>>
>> if (FilesByUID.size() > HS.header_file_size())
>> FilesByUID.resize(HS.header_file_size());
>> -
>> - HeaderFileInfoTrait GeneratorTrait(*this, HS);
>> - llvm::OnDiskChainedHashTableGenerator<HeaderFileInfoTrait> Generator;
>> - SmallVector<const char *, 4> SavedStrings;
>> - unsigned NumHeaderSearchEntries = 0;
>> +
>> for (unsigned UID = 0, LastUID = FilesByUID.size(); UID != LastUID;
>> ++UID) {
>> const FileEntry *File = FilesByUID[UID];
>> if (!File)
>> @@ -2004,11 +2071,16 @@ void ASTWriter::WriteHeaderSearch(const
>> SavedStrings.push_back(Filename.data());
>> }
>>
>> - HeaderFileInfoTrait::key_type key = { File, Filename };
>> - Generator.insert(key, *HFI, GeneratorTrait);
>> + HeaderFileInfoTrait::key_type Key = {
>> + Filename, File->getSize(), getTimestampForOutput(File)
>> + };
>> + HeaderFileInfoTrait::data_type Data = {
>> + *HFI, HS.getModuleMap().findAllModulesForHeader(File), {}
>> + };
>> + Generator.insert(Key, Data, GeneratorTrait);
>> ++NumHeaderSearchEntries;
>> }
>> -
>> +
>> // Create the on-disk hash table in a buffer.
>> SmallString<4096> TableData;
>> uint32_t BucketOffset;
>>
>> Added: cfe/trunk/test/Modules/Inputs/header-attribs/bar.h
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/Inputs/header-attribs/bar.h?rev=304515&view=auto
>>
>> ==============================================================================
>> --- cfe/trunk/test/Modules/Inputs/header-attribs/bar.h (added)
>> +++ cfe/trunk/test/Modules/Inputs/header-attribs/bar.h Thu Jun 1 20:55:39
>> 2017
>> @@ -0,0 +1 @@
>> +extern int b;
>>
>> Added: cfe/trunk/test/Modules/Inputs/header-attribs/baz.h
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/Inputs/header-attribs/baz.h?rev=304515&view=auto
>>
>> ==============================================================================
>> --- cfe/trunk/test/Modules/Inputs/header-attribs/baz.h (added)
>> +++ cfe/trunk/test/Modules/Inputs/header-attribs/baz.h Thu Jun 1 20:55:39
>> 2017
>> @@ -0,0 +1 @@
>> +extern int c;
>>
>> Added: cfe/trunk/test/Modules/Inputs/header-attribs/foo.h
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/Inputs/header-attribs/foo.h?rev=304515&view=auto
>>
>> ==============================================================================
>> --- cfe/trunk/test/Modules/Inputs/header-attribs/foo.h (added)
>> +++ cfe/trunk/test/Modules/Inputs/header-attribs/foo.h Thu Jun 1 20:55:39
>> 2017
>> @@ -0,0 +1 @@
>> +extern int a;
>> \ No newline at end of file
>>
>> Added: cfe/trunk/test/Modules/Inputs/header-attribs/modular.modulemap
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/Inputs/header-attribs/modular.modulemap?rev=304515&view=auto
>>
>> ==============================================================================
>> --- cfe/trunk/test/Modules/Inputs/header-attribs/modular.modulemap (added)
>> +++ cfe/trunk/test/Modules/Inputs/header-attribs/modular.modulemap Thu Jun
>> 1 20:55:39 2017
>> @@ -0,0 +1,5 @@
>> +module A {
>> + header "foo.h" { size 13 }
>> + header "bar.h" { size 1000 }
>> + header "baz.h" { mtime 1 }
>> +}
>>
>> Added: cfe/trunk/test/Modules/Inputs/header-attribs/textual.modulemap
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/Inputs/header-attribs/textual.modulemap?rev=304515&view=auto
>>
>> ==============================================================================
>> --- cfe/trunk/test/Modules/Inputs/header-attribs/textual.modulemap (added)
>> +++ cfe/trunk/test/Modules/Inputs/header-attribs/textual.modulemap Thu Jun
>> 1 20:55:39 2017
>> @@ -0,0 +1,5 @@
>> +module A {
>> + textual header "foo.h" { size 13 }
>> + textual header "bar.h" { size 1000 }
>> + textual header "baz.h" { mtime 1 }
>> +}
>>
>> Modified: cfe/trunk/test/Modules/diagnostics.modulemap
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/diagnostics.modulemap?rev=304515&r1=304514&r2=304515&view=diff
>>
>> ==============================================================================
>> --- cfe/trunk/test/Modules/diagnostics.modulemap (original)
>> +++ cfe/trunk/test/Modules/diagnostics.modulemap Thu Jun 1 20:55:39 2017
>> @@ -1,4 +1,4 @@
>> -// RUN: not %clang_cc1 -fmodules -fmodules-cache-path=%t
>> -fmodule-map-file=%S/Inputs/diagnostics-aux.modulemap -fmodule-map-file=%s
>> -fsyntax-only -x c++ /dev/null 2>&1 | FileCheck %s
>> +// RUN: not %clang_cc1 -fmodules -fmodules-cache-path=%t
>> -fmodule-map-file=%S/Inputs/diagnostics-aux.modulemap -fmodule-map-file=%s
>> -fsyntax-only -x c++ /dev/null 2>&1 | FileCheck %s --implicit-check-not
>> error:
>>
>> // CHECK: In file included from {{.*}}diagnostics-aux.modulemap:3:
>> // CHECK: diagnostics-aux-2.modulemap:2:3: error: expected
>> @@ -15,3 +15,15 @@ module bad_use {
>> // CHECK: diagnostics.modulemap:[[@LINE+1]]:22: error: use declarations
>> are only allowed in top-level modules
>> module submodule { use foo }
>> }
>> +
>> +module header_attr {
>> + // CHECK: diagnostics.modulemap:[[@LINE+1]]:20: error: expected a
>> header attribute name
>> + header "foo.h" { x }
>> + // CHECK: diagnostics.modulemap:[[@LINE+1]]:27: error: header attribute
>> 'size' specified multiple times
>> + header "bar.h" { size 1 size 2 }
>> + // CHECK: diagnostics.modulemap:[[@LINE+1]]:25: error: expected integer
>> literal as value for header attribute 'size'
>> + header "baz.h" { size "30 kilobytes" }
>> +
>> + header "quux.h" { size 1 mtime 2 }
>> + header "no_attrs.h" {}
>> +}
>>
>> Added: cfe/trunk/test/Modules/header-attribs.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/header-attribs.cpp?rev=304515&view=auto
>>
>> ==============================================================================
>> --- cfe/trunk/test/Modules/header-attribs.cpp (added)
>> +++ cfe/trunk/test/Modules/header-attribs.cpp Thu Jun 1 20:55:39 2017
>> @@ -0,0 +1,10 @@
>> +// RUN: rm -rf %t
>> +// RUN: %clang_cc1 -fmodules -I%S/Inputs/header-attribs
>> -fmodule-map-file=%S/Inputs/header-attribs/textual.modulemap
>> -fmodules-cache-path=%t -verify %s -fmodule-name=A -fmodules-strict-decluse
>> +// RUN: not %clang_cc1 -fmodules -I%S/Inputs/header-attribs -emit-module
>> -x c++-module-map %S/Inputs/header-attribs/modular.modulemap
>> -fmodules-cache-path=%t -fmodule-name=A 2>&1 | FileCheck %s --check-prefix
>> BUILD-MODULAR
>> +
>> +#include "foo.h" // ok, stats match
>> +#include "bar.h" // expected-error {{does not depend on a module
>> exporting 'bar.h'}}
>> +#include "baz.h" // expected-error {{does not depend on a module
>> exporting 'baz.h'}}
>> +
>> +// FIXME: Explain why the 'bar.h' found on disk doesn't match the module
>> map.
>> +// BUILD-MODULAR: error: header 'bar.h' not found
>>
>> Added: cfe/trunk/test/Modules/preprocess-missing.modulemap
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/preprocess-missing.modulemap?rev=304515&view=auto
>>
>> ==============================================================================
>> --- cfe/trunk/test/Modules/preprocess-missing.modulemap (added)
>> +++ cfe/trunk/test/Modules/preprocess-missing.modulemap Thu Jun 1
>> 20:55:39 2017
>> @@ -0,0 +1,7 @@
>> +// RUN: %clang_cc1 -fmodules -fmodule-name=A -x c++-module-map %s
>> -emit-module -o /dev/null -verify
>> +module A {
>> + header "does not exist" { size 12345 } // ok, do not need mtime for
>> explicit module build
>> + header "also does not exist" { mtime 12345 }
>> +}
>> +#pragma clang module contents
>> +// expected-error at 4 {{cannot emit module A: size must be explicitly
>> specified for missing header file "also does not exist"}}
>>
>> Modified: cfe/trunk/test/Modules/preprocess-module.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/preprocess-module.cpp?rev=304515&r1=304514&r2=304515&view=diff
>>
>> ==============================================================================
>> --- cfe/trunk/test/Modules/preprocess-module.cpp (original)
>> +++ cfe/trunk/test/Modules/preprocess-module.cpp Thu Jun 1 20:55:39 2017
>> @@ -28,12 +28,21 @@
>> // RUN: %clang_cc1 -fmodules -fmodule-file=%t/no-rewrite.pcm %s -I%t
>> -verify -fno-modules-error-recovery -DINCLUDE -I%S/Inputs/preprocess
>> // RUN: %clang_cc1 -fmodules -fmodule-file=%t/rewrite.pcm %s -I%t -verify
>> -fno-modules-error-recovery -DREWRITE -DINCLUDE -I%S/Inputs/preprocess
>>
>> +// Now try building the module when the header files are missing.
>> +// RUN: cp %S/Inputs/preprocess/fwd.h %S/Inputs/preprocess/file.h
>> %S/Inputs/preprocess/file2.h %S/Inputs/preprocess/module.modulemap %t
>> +// RUN: %clang_cc1 -fmodules -fmodule-name=file -fmodule-file=%t/fwd.pcm
>> -I%t -x c++-module-map %t/module.modulemap -E -frewrite-includes -o
>> %t/copy.ii
>> +// RUN: rm %t/fwd.h %t/file.h %t/file2.h %t/module.modulemap
>> +// RUN: %clang_cc1 -fmodules -fmodule-name=file -fmodule-file=%t/fwd.pcm
>> -x c++-module-map-cpp-output %t/copy.ii -emit-module -o %t/copy.pcm
>> +
>> +// Finally, check that our module contains correct mapping information
>> for the headers.
>> +// RUN: cp %S/Inputs/preprocess/fwd.h %S/Inputs/preprocess/file.h
>> %S/Inputs/preprocess/file2.h %S/Inputs/preprocess/module.modulemap %t
>> +// RUN: %clang_cc1 -fmodules -fmodule-file=%t/copy.pcm %s -I%t -verify
>> -fno-modules-error-recovery -DCOPY -DINCLUDE
>>
>> // == module map
>> // CHECK: # 1 "{{.*}}module.modulemap"
>> // CHECK: module file {
>> -// CHECK: header "file.h"
>> -// CHECK: header "file2.h"
>> +// CHECK: header "file.h" { size
>> +// CHECK: header "file2.h" { size
>> // CHECK: }
>>
>> // == file.h
>> @@ -98,6 +107,8 @@
>> __FILE *a; // expected-error {{declaration of '__FILE' must be imported}}
>> #ifdef REWRITE
>> // expected-note at rewrite.ii:1 {{here}}
>> +#elif COPY
>> +// expected-note at copy.ii:1 {{here}}
>> #else
>> // expected-note at no-rewrite.ii:1 {{here}}
>> #endif
>>
>>
>> _______________________________________________
>> cfe-commits mailing list
>> cfe-commits at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
>
>
>
> _______________________________________________
> cfe-commits mailing list
> cfe-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
>
More information about the cfe-commits
mailing list