[PATCH] D67249: [Modules][PCH] Hash input files content

[Bruno Cardoso Lopes via Phabricator via cfe-commits]

bruno created this revision.
bruno added reviewers: dexonsmith, arphaman, rsmith, aprantl.
Herald added subscribers: cfe-commits, jkorous.
Herald added a project: clang.

When files often get touched during builds, the mtime based validation
leads to different problems in implicit modules builds, even when the
content doesn't actually change:

- Modules only: module invalidation due to out of date files. Usually

causing rebuild traffic.

- Modules + PCH: build failures because clang cannot rebuild a module if

it comes from building a PCH.

- PCH: build failures because clang cannot rebuild a PCH in case one of

the input headers has different mtime.

This patch proposes hashing the content of input files (headers and
module maps), which is performed during serialization time. When looking
at input files for validation, clang only computes the hash in case
there's a mtime mismatch.

I've tested a couple of different hash algorithms availble in LLVM in
face of building modules+pch for `#import <Cocoa/Cocoa.h>`:

- `hash_code`: performace diff within the noise, total module cache

increased by 0.07%.

- `SHA1`: 5% slowdown. Haven't done real size measurements, but it'd be

BLOCK_ID+20 bytes per input file, instead of BLOCK_ID+8 bytes from
`hash_code`.

- `MD5`: 3% slowdown. Like above, but BLOCK_ID+16 bytes per input file.

Given the numbers above, the patch uses `hash_code`. The patch also
improves invalidation error msgs to point out which type of problem the
user is facing: "mtime", "size" or "content".

rdar://problem/29320105


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D67249

Files:
  clang/include/clang/Basic/DiagnosticSerializationKinds.td
  clang/include/clang/Driver/Options.td
  clang/include/clang/Lex/HeaderSearchOptions.h
  clang/include/clang/Serialization/ASTBitCodes.h
  clang/include/clang/Serialization/ASTReader.h
  clang/lib/Driver/ToolChains/Clang.cpp
  clang/lib/Frontend/CompilerInstance.cpp
  clang/lib/Frontend/CompilerInvocation.cpp
  clang/lib/Serialization/ASTReader.cpp
  clang/lib/Serialization/ASTWriter.cpp
  clang/test/Modules/validate-file-content.m
  clang/test/PCH/validate-file-content.m

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D67249.219013.patch
Type: text/x-patch
Size: 21785 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20190906/5bd23b10/attachment-0001.bin>