[all-commits] [llvm/llvm-project] f7b056: [clang][Serialization] Record SLoc dedup metadata ...

[Yuxuan Chen via All-commits]

  Branch: refs/heads/users/yuxuanchen1997/modules-sloc-dedup
  Home:   https://github.com/llvm/llvm-project
  Commit: f7b056ae862c0cc5d5a1ea1fad6da8792f4b6537
      https://github.com/llvm/llvm-project/commit/f7b056ae862c0cc5d5a1ea1fad6da8792f4b6537
  Author: Yuxuan Chen <i at yuxuan.ch>
  Date:   2026-07-08 (Wed, 08 Jul 2026)

  Changed paths:
    M clang/include/clang/Serialization/ASTBitCodes.h
    M clang/include/clang/Serialization/ModuleFile.h
    M clang/lib/Serialization/ASTReader.cpp
    M clang/lib/Serialization/ASTWriter.cpp
    A clang/test/Modules/cxx20-hu-sloc-dedup-info.cpp

  Log Message:
  -----------
  [clang][Serialization] Record SLoc dedup metadata in PCMs

Loaded module files currently reserve SourceManager address space per PCM. Header units that textually include the same large header therefore serialize enough information to recognize identical file contents, but the reader has no eager per-SLoc metadata it can consult before allocating the module's loaded source-location range.

Add an eagerly readable SLOC_ENTRY_DEDUP_INFO record next to SOURCE_LOCATION_OFFSETS. The record is parallel to the serialized SLoc entry table and is emitted unconditionally whenever source-location offsets are written, so the on-disk layout has a single shape regardless of whether a later reader chooses to use the metadata for deduplication.

Each entry records the serialized local source offset, the SLoc-space length, xxh3_128 content hash data for file entries, and flags describing whether the row is a file, dedupable, has line directives, and which file characteristic it uses. Non-dedupable entries keep zero hashes but still occupy their parallel metadata row.

This commit only records and reads the metadata pointer; it does not change module loading allocation. Keeping the format commit independent from the reader policy lets later commits gate or expand deduplication without adding more PCM layout variants.


  Commit: 54f1f3a12a4f19a1e7e8baa44557106160267de7
      https://github.com/llvm/llvm-project/commit/54f1f3a12a4f19a1e7e8baa44557106160267de7
  Author: Yuxuan Chen <i at yuxuan.ch>
  Date:   2026-07-08 (Wed, 08 Jul 2026)

  Changed paths:
    M clang/include/clang/Lex/HeaderSearchOptions.h
    M clang/include/clang/Options/Options.td
    M clang/include/clang/Serialization/ASTReader.h
    M clang/include/clang/Serialization/ModuleFile.h
    M clang/lib/Serialization/ASTReader.cpp

  Log Message:
  -----------
  [clang][Serialization] Deduplicate loaded module SLoc ranges

Header units that textually include the same large header used to consume loaded SourceLocation address space independently, even when the serialized file contents were identical. Importing many such header units could therefore run out of the loaded SLoc region long before the translation unit was large in any meaningful source sense.

Add -cc1 -fdeduplicate-module-source-locations as the opt-in switch for the new reader path. The metadata is now always present in the PCM format, but the reader only changes allocation behavior when this flag is enabled, which keeps the default module-loading behavior unchanged while making the new path explicit and easy to exercise.

Use the eager SLoc dedup metadata to share already-loaded file ranges when a later PCM contains a matching dedupable file entry. The reader builds a per-module remapping from serialized local offsets to the global SourceManager offsets actually used for the import. Fresh entries are packed into the module's allocation, while duplicate file entries map back to the canonical FileID and offset range from the first loaded PCM.

This changes the flagged module-loading model from a single base-offset translation for every loaded location to a segmented translation for PCMs that participate in deduplication. Modules loaded without the flag keep the existing single-add fast path. The SourceManager loaded-entry table only receives slots for entries that are actually allocated, and serialized local FileIDs are remapped to their canonical loaded FileIDs when a file range is shared.

The result is that repeated textual headers in modules no longer multiply loaded SLoc address-space usage when the flag is enabled, while the default path continues to load as before.


Compare: https://github.com/llvm/llvm-project/compare/92f02148816f...54f1f3a12a4f

To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications