r374841 - [Modules][PCH] Hash input files content
Eric Christopher via cfe-commits
cfe-commits at lists.llvm.org
Mon Oct 14 16:12:28 PDT 2019
This was breaking a few bots and I couldn't find you on irc so I've
reverted it thusly:
echristo at jhereg ~/s/llvm-project> git llvm push
Pushing 1 commit:
175b1b856ea Temporarily Revert [Modules][PCH] Hash input files
content as it's breaking a few bots.
Sending cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td
Sending cfe/trunk/include/clang/Driver/Options.td
Sending cfe/trunk/include/clang/Lex/HeaderSearchOptions.h
Sending cfe/trunk/include/clang/Serialization/ASTBitCodes.h
Sending cfe/trunk/include/clang/Serialization/ASTReader.h
Sending cfe/trunk/lib/Driver/ToolChains/Clang.cpp
Sending cfe/trunk/lib/Frontend/CompilerInstance.cpp
Sending cfe/trunk/lib/Frontend/CompilerInvocation.cpp
Sending cfe/trunk/lib/Serialization/ASTReader.cpp
Sending cfe/trunk/lib/Serialization/ASTWriter.cpp
Deleting cfe/trunk/test/Modules/validate-file-content.m
Deleting cfe/trunk/test/PCH/validate-file-content.m
Transmitting file data ..........done
Committing transaction...
Committed revision 374842.
Committed 175b1b856ea to svn.
Sorry for the inconvenience!
-eric
On Mon, Oct 14, 2019 at 3:59 PM Bruno Cardoso Lopes via cfe-commits
<cfe-commits at lists.llvm.org> wrote:
>
> Author: bruno
> Date: Mon Oct 14 16:02:03 2019
> New Revision: 374841
>
> URL: http://llvm.org/viewvc/llvm-project?rev=374841&view=rev
> Log:
> [Modules][PCH] Hash input files content
>
> Summary:
> When files often get touched during builds, the mtime based validation
> leads to different problems in implicit modules builds, even when the
> content doesn't actually change:
>
> - Modules only: module invalidation due to out of date files. Usually causing rebuild traffic.
> - Modules + PCH: build failures because clang cannot rebuild a module if it comes from building a PCH.
> - PCH: build failures because clang cannot rebuild a PCH in case one of the input headers has different mtime.
>
> This patch proposes hashing the content of input files (headers and
> module maps), which is performed during serialization time. When looking
> at input files for validation, clang only computes the hash in case
> there's a mtime mismatch.
>
> I've tested a couple of different hash algorithms availble in LLVM in
> face of building modules+pch for `#import <Cocoa/Cocoa.h>`:
> - `hash_code`: performace diff within the noise, total module cache increased by 0.07%.
> - `SHA1`: 5% slowdown. Haven't done real size measurements, but it'd be BLOCK_ID+20 bytes per input file, instead of BLOCK_ID+8 bytes from `hash_code`.
> - `MD5`: 3% slowdown. Like above, but BLOCK_ID+16 bytes per input file.
>
> Given the numbers above, the patch uses `hash_code`. The patch also
> improves invalidation error msgs to point out which type of problem the
> user is facing: "mtime", "size" or "content".
>
> rdar://problem/29320105
>
> Reviewers: dexonsmith, arphaman, rsmith, aprantl
>
> Subscribers: jkorous, cfe-commits, ributzka
>
> Tags: #clang
>
> Differential Revision: https://reviews.llvm.org/D67249
>
> Added:
> cfe/trunk/test/Modules/validate-file-content.m
> cfe/trunk/test/PCH/validate-file-content.m
> Modified:
> cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td
> cfe/trunk/include/clang/Driver/Options.td
> cfe/trunk/include/clang/Lex/HeaderSearchOptions.h
> cfe/trunk/include/clang/Serialization/ASTBitCodes.h
> cfe/trunk/include/clang/Serialization/ASTReader.h
> cfe/trunk/lib/Driver/ToolChains/Clang.cpp
> cfe/trunk/lib/Frontend/CompilerInstance.cpp
> cfe/trunk/lib/Frontend/CompilerInvocation.cpp
> cfe/trunk/lib/Serialization/ASTReader.cpp
> cfe/trunk/lib/Serialization/ASTWriter.cpp
>
> Modified: cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td?rev=374841&r1=374840&r2=374841&view=diff
> ==============================================================================
> --- cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td (original)
> +++ cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td Mon Oct 14 16:02:03 2019
> @@ -18,13 +18,16 @@ def err_fe_pch_malformed : Error<
> def err_fe_pch_malformed_block : Error<
> "malformed block record in PCH file: '%0'">, DefaultFatal;
> def err_fe_pch_file_modified : Error<
> - "file '%0' has been modified since the precompiled header '%1' was built">,
> + "file '%0' has been modified since the precompiled header '%1' was built"
> + ": %select{size|mtime|content}2 changed">,
> DefaultFatal;
> def err_fe_module_file_modified : Error<
> - "file '%0' has been modified since the module file '%1' was built">,
> + "file '%0' has been modified since the module file '%1' was built"
> + ": %select{size|mtime|content}2 changed">,
> DefaultFatal;
> def err_fe_ast_file_modified : Error<
> - "file '%0' has been modified since the AST file '%1' was built">,
> + "file '%0' has been modified since the AST file '%1' was built"
> + ": %select{size|mtime|content}2 changed">,
> DefaultFatal;
> def err_fe_pch_file_overridden : Error<
> "file '%0' from the precompiled header has been overridden">;
> @@ -399,6 +402,8 @@ def warn_module_uses_date_time : Warning
> def err_module_no_size_mtime_for_header : Error<
> "cannot emit module %0: %select{size|mtime}1 must be explicitly specified "
> "for missing header file \"%2\"">;
> +def err_module_unable_to_hash_content : Error<
> + "failed to hash content for '%0' because memory buffer cannot be retrieved">;
> } // let CategoryName
> } // let Component
>
>
> Modified: cfe/trunk/include/clang/Driver/Options.td
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Driver/Options.td?rev=374841&r1=374840&r2=374841&view=diff
> ==============================================================================
> --- cfe/trunk/include/clang/Driver/Options.td (original)
> +++ cfe/trunk/include/clang/Driver/Options.td Mon Oct 14 16:02:03 2019
> @@ -1368,6 +1368,28 @@ def fmodules_validate_system_headers : F
> HelpText<"Validate the system headers that a module depends on when loading the module">;
> def fno_modules_validate_system_headers : Flag<["-"], "fno-modules-validate-system-headers">,
> Group<i_Group>, Flags<[DriverOption]>;
> +
> +def fvalidate_ast_input_files_content:
> + Flag <["-"], "fvalidate-ast-input-files-content">,
> + Group<f_Group>, Flags<[CC1Option]>,
> + HelpText<"Compute and store the hash of input files used to build an AST."
> + " Files with mismatching mtime's are considered valid"
> + " if both contents is identical">;
> +def fmodules_validate_input_files_content:
> + Flag <["-"], "fmodules-validate-input-files-content">,
> + Group<f_Group>, Flags<[DriverOption]>,
> + HelpText<"Validate PCM input files based on content if mtime differs">;
> +def fno_modules_validate_input_files_content:
> + Flag <["-"], "fno_modules-validate-input-files-content">,
> + Group<f_Group>, Flags<[DriverOption]>;
> +def fpch_validate_input_files_content:
> + Flag <["-"], "fpch-validate-input-files-content">,
> + Group<f_Group>, Flags<[DriverOption]>,
> + HelpText<"Validate PCH input files based on content if mtime differs">;
> +def fno_pch_validate_input_files_content:
> + Flag <["-"], "fno_pch-validate-input-files-content">,
> + Group<f_Group>, Flags<[DriverOption]>;
> +
> def fmodules : Flag <["-"], "fmodules">, Group<f_Group>,
> Flags<[DriverOption, CC1Option]>,
> HelpText<"Enable the 'modules' language feature">;
>
> Modified: cfe/trunk/include/clang/Lex/HeaderSearchOptions.h
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Lex/HeaderSearchOptions.h?rev=374841&r1=374840&r2=374841&view=diff
> ==============================================================================
> --- cfe/trunk/include/clang/Lex/HeaderSearchOptions.h (original)
> +++ cfe/trunk/include/clang/Lex/HeaderSearchOptions.h Mon Oct 14 16:02:03 2019
> @@ -195,6 +195,10 @@ public:
> /// Whether to validate system input files when a module is loaded.
> unsigned ModulesValidateSystemHeaders : 1;
>
> + // Whether the content of input files should be hashed and used to
> + // validate consistency.
> + unsigned ValidateASTInputFilesContent : 1;
> +
> /// Whether the module includes debug information (-gmodules).
> unsigned UseDebugInfo : 1;
>
> @@ -208,7 +212,8 @@ public:
> UseBuiltinIncludes(true), UseStandardSystemIncludes(true),
> UseStandardCXXIncludes(true), UseLibcxx(false), Verbose(false),
> ModulesValidateOncePerBuildSession(false),
> - ModulesValidateSystemHeaders(false), UseDebugInfo(false),
> + ModulesValidateSystemHeaders(false),
> + ValidateASTInputFilesContent(false), UseDebugInfo(false),
> ModulesValidateDiagnosticOptions(true), ModulesHashContent(false) {}
>
> /// AddPath - Add the \p Path path to the specified \p Group list.
>
> Modified: cfe/trunk/include/clang/Serialization/ASTBitCodes.h
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Serialization/ASTBitCodes.h?rev=374841&r1=374840&r2=374841&view=diff
> ==============================================================================
> --- cfe/trunk/include/clang/Serialization/ASTBitCodes.h (original)
> +++ cfe/trunk/include/clang/Serialization/ASTBitCodes.h Mon Oct 14 16:02:03 2019
> @@ -382,7 +382,10 @@ namespace serialization {
> /// inside the control block.
> enum InputFileRecordTypes {
> /// An input file.
> - INPUT_FILE = 1
> + INPUT_FILE = 1,
> +
> + /// The input file content hash
> + INPUT_FILE_HASH
> };
>
> /// Record types that occur within the AST block itself.
>
> Modified: cfe/trunk/include/clang/Serialization/ASTReader.h
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Serialization/ASTReader.h?rev=374841&r1=374840&r2=374841&view=diff
> ==============================================================================
> --- cfe/trunk/include/clang/Serialization/ASTReader.h (original)
> +++ cfe/trunk/include/clang/Serialization/ASTReader.h Mon Oct 14 16:02:03 2019
> @@ -930,6 +930,9 @@ private:
> /// Whether validate system input files.
> bool ValidateSystemInputs;
>
> + /// Whether validate headers and module maps using hash based on contents.
> + bool ValidateASTInputFilesContent;
> +
> /// Whether we are allowed to use the global module index.
> bool UseGlobalIndex;
>
> @@ -1203,6 +1206,7 @@ private:
>
> struct InputFileInfo {
> std::string Filename;
> + uint64_t ContentHash;
> off_t StoredSize;
> time_t StoredTime;
> bool Overridden;
> @@ -1437,6 +1441,8 @@ private:
> void Error(StringRef Msg) const;
> void Error(unsigned DiagID, StringRef Arg1 = StringRef(),
> StringRef Arg2 = StringRef()) const;
> + void Error(unsigned DiagID, StringRef Arg1, StringRef Arg2,
> + unsigned Select) const;
> void Error(llvm::Error &&Err) const;
>
> public:
> @@ -1485,7 +1491,9 @@ public:
> StringRef isysroot = "", bool DisableValidation = false,
> bool AllowASTWithCompilerErrors = false,
> bool AllowConfigurationMismatch = false,
> - bool ValidateSystemInputs = false, bool UseGlobalIndex = true,
> + bool ValidateSystemInputs = false,
> + bool ValidateASTInputFilesContent = false,
> + bool UseGlobalIndex = true,
> std::unique_ptr<llvm::Timer> ReadTimer = {});
> ASTReader(const ASTReader &) = delete;
> ASTReader &operator=(const ASTReader &) = delete;
>
> Modified: cfe/trunk/lib/Driver/ToolChains/Clang.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Driver/ToolChains/Clang.cpp?rev=374841&r1=374840&r2=374841&view=diff
> ==============================================================================
> --- cfe/trunk/lib/Driver/ToolChains/Clang.cpp (original)
> +++ cfe/trunk/lib/Driver/ToolChains/Clang.cpp Mon Oct 14 16:02:03 2019
> @@ -2777,6 +2777,10 @@ static void RenderModulesOptions(Compila
> std::string("-fprebuilt-module-path=") + A->getValue()));
> A->claim();
> }
> + if (Args.hasFlag(options::OPT_fmodules_validate_input_files_content,
> + options::OPT_fno_modules_validate_input_files_content,
> + false))
> + CmdArgs.push_back("-fvalidate-ast-input-files-content");
> }
>
> // -fmodule-name specifies the module that is currently being built (or
> @@ -4899,6 +4903,10 @@ void Clang::ConstructJob(Compilation &C,
> Std && (Std->containsValue("c++2a") || Std->containsValue("c++latest"));
> RenderModulesOptions(C, D, Args, Input, Output, CmdArgs, HaveModules);
>
> + if (Args.hasFlag(options::OPT_fpch_validate_input_files_content,
> + options::OPT_fno_pch_validate_input_files_content, false))
> + CmdArgs.push_back("-fvalidate-ast-input-files-content");
> +
> Args.AddLastArg(CmdArgs, options::OPT_fexperimental_new_pass_manager,
> options::OPT_fno_experimental_new_pass_manager);
>
>
> Modified: cfe/trunk/lib/Frontend/CompilerInstance.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Frontend/CompilerInstance.cpp?rev=374841&r1=374840&r2=374841&view=diff
> ==============================================================================
> --- cfe/trunk/lib/Frontend/CompilerInstance.cpp (original)
> +++ cfe/trunk/lib/Frontend/CompilerInstance.cpp Mon Oct 14 16:02:03 2019
> @@ -510,7 +510,8 @@ IntrusiveRefCntPtr<ASTReader> CompilerIn
> PP, ModuleCache, &Context, PCHContainerRdr, Extensions,
> Sysroot.empty() ? "" : Sysroot.data(), DisablePCHValidation,
> AllowPCHWithCompilerErrors, /*AllowConfigurationMismatch*/ false,
> - HSOpts.ModulesValidateSystemHeaders, UseGlobalModuleIndex));
> + HSOpts.ModulesValidateSystemHeaders, HSOpts.ValidateASTInputFilesContent,
> + UseGlobalModuleIndex));
>
> // We need the external source to be set up before we read the AST, because
> // eagerly-deserialized declarations may use it.
> @@ -1492,6 +1493,7 @@ void CompilerInstance::createModuleManag
> /*AllowASTWithCompilerErrors=*/false,
> /*AllowConfigurationMismatch=*/false,
> HSOpts.ModulesValidateSystemHeaders,
> + HSOpts.ValidateASTInputFilesContent,
> getFrontendOpts().UseGlobalModuleIndex, std::move(ReadTimer));
> if (hasASTConsumer()) {
> ModuleManager->setDeserializationListener(
>
> Modified: cfe/trunk/lib/Frontend/CompilerInvocation.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Frontend/CompilerInvocation.cpp?rev=374841&r1=374840&r2=374841&view=diff
> ==============================================================================
> --- cfe/trunk/lib/Frontend/CompilerInvocation.cpp (original)
> +++ cfe/trunk/lib/Frontend/CompilerInvocation.cpp Mon Oct 14 16:02:03 2019
> @@ -2081,6 +2081,8 @@ static void ParseHeaderSearchArgs(Header
> getLastArgUInt64Value(Args, OPT_fbuild_session_timestamp, 0);
> Opts.ModulesValidateSystemHeaders =
> Args.hasArg(OPT_fmodules_validate_system_headers);
> + Opts.ValidateASTInputFilesContent =
> + Args.hasArg(OPT_fvalidate_ast_input_files_content);
> if (const Arg *A = Args.getLastArg(OPT_fmodule_format_EQ))
> Opts.ModuleFormat = A->getValue();
>
>
> Modified: cfe/trunk/lib/Serialization/ASTReader.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Serialization/ASTReader.cpp?rev=374841&r1=374840&r2=374841&view=diff
> ==============================================================================
> --- cfe/trunk/lib/Serialization/ASTReader.cpp (original)
> +++ cfe/trunk/lib/Serialization/ASTReader.cpp Mon Oct 14 16:02:03 2019
> @@ -1247,6 +1247,12 @@ void ASTReader::Error(unsigned DiagID,
> Diag(DiagID) << Arg1 << Arg2;
> }
>
> +void ASTReader::Error(unsigned DiagID, StringRef Arg1, StringRef Arg2,
> + unsigned Select) const {
> + if (!Diags.isDiagnosticInFlight())
> + Diag(DiagID) << Arg1 << Arg2 << Select;
> +}
> +
> void ASTReader::Error(llvm::Error &&Err) const {
> Error(toString(std::move(Err)));
> }
> @@ -2241,6 +2247,24 @@ ASTReader::readInputFileInfo(ModuleFile
> R.TopLevelModuleMap = static_cast<bool>(Record[5]);
> R.Filename = Blob;
> ResolveImportedPath(F, R.Filename);
> +
> + Expected<llvm::BitstreamEntry> MaybeEntry = Cursor.advance();
> + if (!MaybeEntry) // FIXME this drops errors on the floor.
> + consumeError(MaybeEntry.takeError());
> + llvm::BitstreamEntry Entry = MaybeEntry.get();
> + assert(Entry.Kind == llvm::BitstreamEntry::Record &&
> + "expected record type for input file hash");
> +
> + Record.clear();
> + if (Expected<unsigned> Maybe = Cursor.readRecord(Entry.ID, Record))
> + assert(static_cast<InputFileRecordTypes>(Maybe.get()) == INPUT_FILE_HASH &&
> + "invalid record type for input file hash");
> + else {
> + // FIXME this drops errors on the floor.
> + consumeError(Maybe.takeError());
> + }
> + R.ContentHash = (static_cast<uint64_t>(Record[1]) << 32) |
> + static_cast<uint64_t>(Record[0]);
> return R;
> }
>
> @@ -2271,6 +2295,7 @@ InputFile ASTReader::getInputFile(Module
> bool Overridden = FI.Overridden;
> bool Transient = FI.Transient;
> StringRef Filename = FI.Filename;
> + uint64_t StoredContentHash = FI.ContentHash;
>
> const FileEntry *File = nullptr;
> if (auto FE = FileMgr.getFile(Filename, /*OpenFile=*/false))
> @@ -2325,14 +2350,46 @@ InputFile ASTReader::getInputFile(Module
> }
> }
>
> - bool IsOutOfDate = false;
> + enum ModificationType {
> + Size,
> + ModTime,
> + Content,
> + None,
> + };
> + auto HasInputFileChanged = [&]() {
> + if (StoredSize != File->getSize())
> + return ModificationType::Size;
> + if (!DisableValidation && StoredTime &&
> + StoredTime != File->getModificationTime()) {
> + // In case the modification time changes but not the content,
> + // accept the cached file as legit.
> + if (ValidateASTInputFilesContent &&
> + StoredContentHash != static_cast<uint64_t>(llvm::hash_code(-1))) {
> + auto MemBuffOrError = FileMgr.getBufferForFile(File);
> + if (!MemBuffOrError) {
> + if (!Complain)
> + return ModificationType::ModTime;
> + std::string ErrorStr = "could not get buffer for file '";
> + ErrorStr += File->getName();
> + ErrorStr += "'";
> + Error(ErrorStr);
> + return ModificationType::ModTime;
> + }
> +
> + auto ContentHash = hash_value(MemBuffOrError.get()->getBuffer());
> + if (StoredContentHash == static_cast<uint64_t>(ContentHash))
> + return ModificationType::None;
> + return ModificationType::Content;
> + }
> + return ModificationType::ModTime;
> + }
> + return ModificationType::None;
> + };
>
> + bool IsOutOfDate = false;
> + auto FileChange = HasInputFileChanged();
> // For an overridden file, there is nothing to validate.
> - if (!Overridden && //
> - (StoredSize != File->getSize() ||
> - (StoredTime && StoredTime != File->getModificationTime() &&
> - !DisableValidation)
> - )) {
> + if (!Overridden && FileChange != ModificationType::None) {
> if (Complain) {
> // Build a list of the PCH imports that got us here (in reverse).
> SmallVector<ModuleFile *, 4> ImportStack(1, &F);
> @@ -2341,13 +2398,17 @@ InputFile ASTReader::getInputFile(Module
>
> // The top-level PCH is stale.
> StringRef TopLevelPCHName(ImportStack.back()->FileName);
> - unsigned DiagnosticKind = moduleKindForDiagnostic(ImportStack.back()->Kind);
> + unsigned DiagnosticKind =
> + moduleKindForDiagnostic(ImportStack.back()->Kind);
> if (DiagnosticKind == 0)
> - Error(diag::err_fe_pch_file_modified, Filename, TopLevelPCHName);
> + Error(diag::err_fe_pch_file_modified, Filename, TopLevelPCHName,
> + (unsigned)FileChange);
> else if (DiagnosticKind == 1)
> - Error(diag::err_fe_module_file_modified, Filename, TopLevelPCHName);
> + Error(diag::err_fe_module_file_modified, Filename, TopLevelPCHName,
> + (unsigned)FileChange);
> else
> - Error(diag::err_fe_ast_file_modified, Filename, TopLevelPCHName);
> + Error(diag::err_fe_ast_file_modified, Filename, TopLevelPCHName,
> + (unsigned)FileChange);
>
> // Print the import stack.
> if (ImportStack.size() > 1 && !Diags.isDiagnosticInFlight()) {
> @@ -5192,6 +5253,8 @@ bool ASTReader::readASTFileControlBlock(
> consumeError(MaybeRecordType.takeError());
> }
> switch ((InputFileRecordTypes)MaybeRecordType.get()) {
> + case INPUT_FILE_HASH:
> + break;
> case INPUT_FILE:
> bool Overridden = static_cast<bool>(Record[3]);
> std::string Filename = Blob;
> @@ -12153,7 +12216,7 @@ ASTReader::ASTReader(Preprocessor &PP, I
> StringRef isysroot, bool DisableValidation,
> bool AllowASTWithCompilerErrors,
> bool AllowConfigurationMismatch, bool ValidateSystemInputs,
> - bool UseGlobalIndex,
> + bool ValidateASTInputFilesContent, bool UseGlobalIndex,
> std::unique_ptr<llvm::Timer> ReadTimer)
> : Listener(DisableValidation
> ? cast<ASTReaderListener>(new SimpleASTReaderListener(PP))
> @@ -12167,6 +12230,7 @@ ASTReader::ASTReader(Preprocessor &PP, I
> AllowASTWithCompilerErrors(AllowASTWithCompilerErrors),
> AllowConfigurationMismatch(AllowConfigurationMismatch),
> ValidateSystemInputs(ValidateSystemInputs),
> + ValidateASTInputFilesContent(ValidateASTInputFilesContent),
> UseGlobalIndex(UseGlobalIndex), CurrSwitchCaseStmts(&SwitchCaseStmts) {
> SourceMgr.setExternalSLocEntrySource(this);
>
>
> Modified: cfe/trunk/lib/Serialization/ASTWriter.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Serialization/ASTWriter.cpp?rev=374841&r1=374840&r2=374841&view=diff
> ==============================================================================
> --- cfe/trunk/lib/Serialization/ASTWriter.cpp (original)
> +++ cfe/trunk/lib/Serialization/ASTWriter.cpp Mon Oct 14 16:02:03 2019
> @@ -1099,6 +1099,7 @@ void ASTWriter::WriteBlockInfoBlock() {
>
> BLOCK(INPUT_FILES_BLOCK);
> RECORD(INPUT_FILE);
> + RECORD(INPUT_FILE_HASH);
>
> // AST Top-Level Block.
> BLOCK(AST_BLOCK);
> @@ -1764,6 +1765,7 @@ struct InputFileEntry {
> bool IsTransient;
> bool BufferOverridden;
> bool IsTopLevelModuleMap;
> + uint32_t ContentHash[2];
> };
>
> } // namespace
> @@ -1787,6 +1789,13 @@ void ASTWriter::WriteInputFiles(SourceMa
> IFAbbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // File name
> unsigned IFAbbrevCode = Stream.EmitAbbrev(std::move(IFAbbrev));
>
> + // Create input file hash abbreviation.
> + auto IFHAbbrev = std::make_shared<BitCodeAbbrev>();
> + IFHAbbrev->Add(BitCodeAbbrevOp(INPUT_FILE_HASH));
> + IFHAbbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32));
> + IFHAbbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32));
> + unsigned IFHAbbrevCode = Stream.EmitAbbrev(std::move(IFHAbbrev));
> +
> // Get all ContentCache objects for files, sorted by whether the file is a
> // system one or not. System files go at the back, users files at the front.
> std::deque<InputFileEntry> SortedFiles;
> @@ -1810,6 +1819,25 @@ void ASTWriter::WriteInputFiles(SourceMa
> Entry.BufferOverridden = Cache->BufferOverridden;
> Entry.IsTopLevelModuleMap = isModuleMap(File.getFileCharacteristic()) &&
> File.getIncludeLoc().isInvalid();
> +
> + auto ContentHash = hash_code(-1);
> + if (PP->getHeaderSearchInfo()
> + .getHeaderSearchOpts()
> + .ValidateASTInputFilesContent) {
> + auto *MemBuff = Cache->getRawBuffer();
> + if (MemBuff)
> + ContentHash = hash_value(MemBuff->getBuffer());
> + else
> + // FIXME: The path should be taken from the FileEntryRef.
> + PP->Diag(SourceLocation(), diag::err_module_unable_to_hash_content)
> + << Entry.File->getName();
> + }
> + auto CH = llvm::APInt(64, ContentHash);
> + Entry.ContentHash[0] =
> + static_cast<uint32_t>(CH.getLoBits(32).getZExtValue());
> + Entry.ContentHash[1] =
> + static_cast<uint32_t>(CH.getHiBits(32).getZExtValue());
> +
> if (Entry.IsSystemFile)
> SortedFiles.push_back(Entry);
> else
> @@ -1834,17 +1862,26 @@ void ASTWriter::WriteInputFiles(SourceMa
>
> // Emit size/modification time for this file.
> // And whether this file was overridden.
> - RecordData::value_type Record[] = {
> - INPUT_FILE,
> - InputFileOffsets.size(),
> - (uint64_t)Entry.File->getSize(),
> - (uint64_t)getTimestampForOutput(Entry.File),
> - Entry.BufferOverridden,
> - Entry.IsTransient,
> - Entry.IsTopLevelModuleMap};
> + {
> + RecordData::value_type Record[] = {
> + INPUT_FILE,
> + InputFileOffsets.size(),
> + (uint64_t)Entry.File->getSize(),
> + (uint64_t)getTimestampForOutput(Entry.File),
> + Entry.BufferOverridden,
> + Entry.IsTransient,
> + Entry.IsTopLevelModuleMap};
> +
> + // FIXME: The path should be taken from the FileEntryRef.
> + EmitRecordWithPath(IFAbbrevCode, Record, Entry.File->getName());
> + }
>
> - // FIXME: The path should be taken from the FileEntryRef.
> - EmitRecordWithPath(IFAbbrevCode, Record, Entry.File->getName());
> + // Emit content hash for this file.
> + {
> + RecordData::value_type Record[] = {INPUT_FILE_HASH, Entry.ContentHash[0],
> + Entry.ContentHash[1]};
> + Stream.EmitRecordWithAbbrev(IFHAbbrevCode, Record);
> + }
> }
>
> Stream.ExitBlock();
>
> Added: cfe/trunk/test/Modules/validate-file-content.m
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/validate-file-content.m?rev=374841&view=auto
> ==============================================================================
> --- cfe/trunk/test/Modules/validate-file-content.m (added)
> +++ cfe/trunk/test/Modules/validate-file-content.m Mon Oct 14 16:02:03 2019
> @@ -0,0 +1,33 @@
> +// REQUIRES: shell
> +//
> +// Check driver works
> +// RUN: %clang -fmodules -fsyntax-only -fmodules-validate-input-files-content %s -### 2>&1 | FileCheck --check-prefix=CHECK-CC1 %s
> +// CHECK-CC1: -fvalidate-ast-input-files-content
> +//
> +// PCH+Modules: Test that a mtime mismatch without content change is fine
> +// RUN: rm -rf %t
> +// RUN: mkdir %t
> +// RUN: echo '// m.h' > %t/m.h
> +// RUN: echo '#include "m.h"' > %t/a.h
> +// RUN: echo 'module m { header "m.h" }' > %t/module.modulemap
> +// RUN: %clang_cc1 -emit-pch -fmodules-cache-path=%t/cache -fmodules -fimplicit-module-maps -o %t/a.pch -I %t -x objective-c-header %t/a.h -fvalidate-ast-input-files-content
> +// RUN: touch -m -a -t 202901010000 %t/m.h
> +// RUN: %clang_cc1 -fsyntax-only -fmodules-cache-path=%t/cache -fmodules -fimplicit-module-maps -I %t -include-pch %t/a.pch %s -verify -fvalidate-ast-input-files-content
> +//
> +// PCH+Modules: Test that a mtime mismatch with content change
> +// RUN: rm -rf %t
> +// RUN: mkdir %t
> +// RUN: echo '// m.h' > %t/m.h
> +// RUN: echo '#include "m.h"' > %t/a.h
> +// RUN: echo 'module m { header "m.h" }' > %t/module.modulemap
> +// RUN: %clang_cc1 -emit-pch -fmodules-cache-path=%t/cache -fmodules -fimplicit-module-maps -o %t/a.pch -I %t -x objective-c-header %t/a.h -fvalidate-ast-input-files-content
> +// RUN: echo '// m.x' > %t/m.h
> +// RUN: touch -m -a -t 202901010000 %t/m.h
> +// RUN: not %clang_cc1 -fsyntax-only -fmodules-cache-path=%t/cache -fmodules -fimplicit-module-maps -I %t -include-pch %t/a.pch %s -fvalidate-ast-input-files-content 2> %t/stderr
> +// RUN: FileCheck %s < %t/stderr
> +//
> +// CHECK: file '[[M_H:.*[/\\]m\.h]]' has been modified since the precompiled header '[[A_PCH:.*/a\.pch]]' was built: content changed
> +// CHECK: '[[M_H]]' required by '[[M_PCM:.*[/\\]m.*\.pcm]]'
> +// CHECK: '[[M_PCM]]' required by '[[A_PCH]]'
> +// CHECK: please rebuild precompiled header '[[A_PCH]]'
> +// expected-no-diagnostics
>
> Added: cfe/trunk/test/PCH/validate-file-content.m
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/PCH/validate-file-content.m?rev=374841&view=auto
> ==============================================================================
> --- cfe/trunk/test/PCH/validate-file-content.m (added)
> +++ cfe/trunk/test/PCH/validate-file-content.m Mon Oct 14 16:02:03 2019
> @@ -0,0 +1,29 @@
> +// REQUIRES: shell
> +//
> +// Check driver works
> +// RUN: %clang -x objective-c-header -fsyntax-only -fpch-validate-input-files-content %t/a.h -### 2>&1 | FileCheck --check-prefix=CHECK-CC1 %s
> +// CHECK-CC1: -fvalidate-ast-input-files-content
> +//
> +// PCH only: Test that a mtime mismatch without content change is fine
> +// RUN: rm -rf %t
> +// RUN: mkdir %t
> +// RUN: echo '// m.h' > %t/m.h
> +// RUN: echo '#include "m.h"' > %t/a.h
> +// RUN: %clang_cc1 -emit-pch -o %t/a.pch -I %t -x objective-c-header %t/a.h -fvalidate-ast-input-files-content
> +// RUN: touch -m -a -t 202901010000 %t/m.h
> +// RUN: %clang_cc1 -fsyntax-only -I %t -include-pch %t/a.pch %s -verify -fvalidate-ast-input-files-content
> +//
> +// PCH only: Test that a mtime mismatch with content change
> +// RUN: rm -rf %t
> +// RUN: mkdir %t
> +// RUN: echo '// m.h' > %t/m.h
> +// RUN: echo '#include "m.h"' > %t/a.h
> +// RUN: %clang_cc1 -emit-pch -o %t/a.pch -I %t -x objective-c-header %t/a.h -fvalidate-ast-input-files-content
> +// RUN: echo '// m.x' > %t/m.h
> +// RUN: touch -m -a -t 202901010000 %t/m.h
> +// RUN: not %clang_cc1 -fsyntax-only -I %t -include-pch %t/a.pch %s -fvalidate-ast-input-files-content 2> %t/stderr
> +// RUN: FileCheck %s < %t/stderr
> +//
> +// CHECK: file '[[M_H:.*[/\\]m\.h]]' has been modified since the precompiled header '[[A_PCH:.*/a\.pch]]' was built: content changed
> +// CHECK: please rebuild precompiled header '[[A_PCH]]'
> +// expected-no-diagnostics
>
>
> _______________________________________________
> cfe-commits mailing list
> cfe-commits at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
More information about the cfe-commits
mailing list