r233115 - [Modules] Start making explicit modules produce deterministic output.

[Chandler Carruth]

Author: chandlerc
Date: Tue Mar 24 16:18:10 2015
New Revision: 233115

URL: http://llvm.org/viewvc/llvm-project?rev=233115&view=rev
Log:
[Modules] Start making explicit modules produce deterministic output.

There are two aspects of non-determinism fixed here, which was the
minimum required to cause at least an empty module to be deterministic.

First, the random number signature is only inserted into the module when
we are building modules implicitly. The use case for these random
signatures is to work around the very fact that modules are not
deterministic in their output when working with the implicitly built and
populated module cache. Eventually this should go away entirely when
we're confident that Clang is producing deterministic output.

Second, the on-disk hash table is populated based on the order of
iteration over a DenseMap. Instead, use a MapVector so that we can walk
it in insertion order.

I've added a test that an empty module, when built twice, produces the
same binary PCM file.

Added:
    cfe/trunk/test/Modules/Inputs/empty/
    cfe/trunk/test/Modules/Inputs/empty/empty.h
    cfe/trunk/test/Modules/empty.modulemap
Modified:
    cfe/trunk/include/clang/Serialization/ASTWriter.h
    cfe/trunk/lib/Serialization/ASTWriter.cpp

Modified: cfe/trunk/include/clang/Serialization/ASTWriter.h
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Serialization/ASTWriter.h?rev=233115&r1=233114&r2=233115&view=diff
==============================================================================
--- cfe/trunk/include/clang/Serialization/ASTWriter.h (original)
+++ cfe/trunk/include/clang/Serialization/ASTWriter.h Tue Mar 24 16:18:10 2015
@@ -225,7 +225,7 @@ private:
   /// The ID numbers for identifiers are consecutive (in order of
   /// discovery), starting at 1. An ID of zero refers to a NULL
   /// IdentifierInfo.
-  llvm::DenseMap<const IdentifierInfo *, serialization::IdentID> IdentifierIDs;
+  llvm::MapVector<const IdentifierInfo *, serialization::IdentID> IdentifierIDs;
 
   /// \brief The first ID number we can use for our own macros.
   serialization::MacroID FirstMacroID;

Modified: cfe/trunk/lib/Serialization/ASTWriter.cpp
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Serialization/ASTWriter.cpp?rev=233115&r1=233114&r2=233115&view=diff
==============================================================================
--- cfe/trunk/lib/Serialization/ASTWriter.cpp (original)
+++ cfe/trunk/lib/Serialization/ASTWriter.cpp Tue Mar 24 16:18:10 2015
@@ -1160,12 +1160,17 @@ void ASTWriter::WriteControlBlock(Prepro
   Stream.EmitRecordWithBlob(MetadataAbbrevCode, Record,
                             getClangFullRepositoryVersion());
 
-  // Signature
-  Record.clear();
-  Record.push_back(getSignature());
-  Stream.EmitRecord(SIGNATURE, Record);
-
   if (WritingModule) {
+    // For implicit modules we output a signature that we can use to ensure
+    // duplicate module builds don't collide in the cache as their output order
+    // is non-deterministic.
+    // FIXME: Remove this when output is deterministic.
+    if (Context.getLangOpts().ImplicitModules) {
+      Record.clear();
+      Record.push_back(getSignature());
+      Stream.EmitRecord(SIGNATURE, Record);
+    }
+
     // Module name
     BitCodeAbbrev *Abbrev = new BitCodeAbbrev();
     Abbrev->Add(BitCodeAbbrevOp(MODULE_NAME));
@@ -3507,14 +3512,12 @@ void ASTWriter::WriteIdentifierTable(Pre
     // Create the on-disk hash table representation. We only store offsets
     // for identifiers that appear here for the first time.
     IdentifierOffsets.resize(NextIdentID - FirstIdentID);
-    for (llvm::DenseMap<const IdentifierInfo *, IdentID>::iterator
-           ID = IdentifierIDs.begin(), IDEnd = IdentifierIDs.end();
-         ID != IDEnd; ++ID) {
-      assert(ID->first && "NULL identifier in identifier table");
-      if (!Chain || !ID->first->isFromAST() || 
-          ID->first->hasChangedSinceDeserialization())
-        Generator.insert(const_cast<IdentifierInfo *>(ID->first), ID->second,
-                         Trait);
+    for (auto IdentIDPair : IdentifierIDs) {
+      IdentifierInfo *II = const_cast<IdentifierInfo *>(IdentIDPair.first);
+      IdentID ID = IdentIDPair.second;
+      assert(II && "NULL identifier in identifier table");
+      if (!Chain || !II->isFromAST() || II->hasChangedSinceDeserialization())
+        Generator.insert(II, ID, Trait);
     }
 
     // Create the on-disk hash table in a buffer.

Added: cfe/trunk/test/Modules/Inputs/empty/empty.h
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/Inputs/empty/empty.h?rev=233115&view=auto
==============================================================================
--- cfe/trunk/test/Modules/Inputs/empty/empty.h (added)
+++ cfe/trunk/test/Modules/Inputs/empty/empty.h Tue Mar 24 16:18:10 2015
@@ -0,0 +1 @@
+// This file intentionally left empty.

Added: cfe/trunk/test/Modules/empty.modulemap
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/empty.modulemap?rev=233115&view=auto
==============================================================================
--- cfe/trunk/test/Modules/empty.modulemap (added)
+++ cfe/trunk/test/Modules/empty.modulemap Tue Mar 24 16:18:10 2015
@@ -0,0 +1,15 @@
+// RUN: rm -rf %t
+//
+// RUN: %clang_cc1 -fmodules -x c++ -fmodules-cache-path=%t \
+// RUN:   -fno-implicit-modules -fno-modules-implicit-maps \
+// RUN:   -emit-module -fmodule-name=empty -o %t/base.pcm \
+// RUN:   %s
+//
+// RUN: %clang_cc1 -fmodules -x c++ -fmodules-cache-path=%t \
+// RUN:   -fno-implicit-modules -fno-modules-implicit-maps \
+// RUN:   -emit-module -fmodule-name=empty -o %t/check.pcm \
+// RUN:   %s
+//
+// RUN: diff %t/base.pcm %t/check.pcm
+
+module empty { header "Inputs/empty.h" export * }