[Lldb-commits] [lldb] [lldb][NativePDB] Defer PdbIndex creation until the symbol file is queried (PR #211178)
via lldb-commits
lldb-commits at lists.llvm.org
Tue Jul 21 23:25:55 PDT 2026
https://github.com/cherleey created https://github.com/llvm/llvm-project/pull/211178
## Summary
On Windows, `symbols.load-on-demand` does not bound resident memory when adjacent PDBs are present:
`SymbolFileNativePDB::CalculateAbilities` eagerly builds the **PdbIndex** (parsing the type/symbol
record streams) for every candidate module at SymbolFile creation — *before* the on-demand wrapper
can defer anything. For a target that loads a few hundred modules with GB-scale engine PDBs, this
balloons to ~20 GB of anonymous heap and makes the session un-runnable on a 32 GB machine. This
patch defers `PdbIndex::create` to `GetOrCreateIndex()` (first real symbol query);
`CalculateAbilities` now reads only the DBI stream header for the stripped check, and rejects PDBs
lacking TPI/IPI so an unusable PDB does not win plugin selection.
PR branch: 4 files, +141/−23, all inside the NativePDB plugin (`PdbIndex.{cpp,h}`,
`SymbolFileNativePDB.{cpp,h}`), rebased onto current `main`. (The original validation patch
against llvmorg-22.1.8 is preserved as `lldb22-lazy-pdbindex.patch` in the repro repo below.)
## Motivation / root cause
Repro target: Unreal Engine 5.8 editor built with **engine debug symbols installed** (1134 PDBs;
~12.9 GB on disk on the 64 GB diagnosis bed, ~16 GB on the 32 GB target), debugged with lldb on Windows.
Diagnosis (measurements in the linked assets):
- With `load-on-demand true`, memory still balloons to ~19–20 GB. Removing the breakpoint entirely
(BP_MODE=none) produces the *identical* balloon → breakpoints are exonerated. Running the same
workload from `lldb.exe` (no lldb-dap) balloons identically → the cost is in **liblldb core**, not
the DAP layer.
- Enabling `log enable lldb symbol` shows `Loading <module>.pdb` **followed by**
`SymbolFileOnDemand::InitializeObject is skipped` — i.e. the bytes are read into the heap *before*
the on-demand wrapper skips indexing. load-on-demand only defers `InitializeObject` (type-stream
indexing); the PDB is already resident.
- Private bytes ≈ working set (≈12.1 GB vs ≈12.1 GB in one sample) → the balloon is anonymous heap
(not file-backed mmap), which is what exhausts a 32 GB machine's commit + physical memory.
Causal chain:
```
module load → SymbolVendor finds adjacent PDB → SymbolFile(NativePDB) CREATION reads the PDB into
private heap (~1.5x on-disk expansion) → load-on-demand only skips InitializeObject (indexing) —
bytes already resident → ~240 engine modules x GB-scale PDBs ≈ 20 GB anonymous heap
```
## Results (before / after)
Two beds: a 64 GB "clean" bed and the actual 32 GB target. Numbers are RSS of the lldb process.
| config | unpatched | patched (this PR) |
|---|---|---|
| launch, no breakpoint, LOD on | ~19.3 GB @115s (balloon; measured with a breakpoint set — removing the bp produced an identical balloon, but that run was terminated by a memory guard mid-settle, so no clean no-bp plateau figure exists) | **4.03 GB** |
| launch, breakpoint, LOD on | ~19.78 GB peak → thrash timeout | **6.34 GB** (64 GB and 32 GB beds match exactly) |
| attach (running editor) | 22–25 GB thrash + 1x OOM-kill | ramp 3.8 → **6.5 GB** peak |
On the 32 GB target, unpatched lldb cannot launch or attach this target at all (sustained thrash /
one OOM-kill); patched, the whole session (lldb 6.5 GB + full-GUI editor 3.42 GB + realistic
residents) fits with physical `avail` staying > 2 GB (commit-basis is tighter: available commit
touched ≈2.0 GB in the attach run and pagefile-availability dipped to 1.41/1.68 GB in two launch
runs). For reference, the MS stack (cdb/DbgHelp)
attaches the same target at ~0.16 GB, so there is still headroom above this PR (a queried module can
be hydrated alone; the project PDB is 61 MB) — a natural follow-up. For scale: Visual Studio 2022
itself attaching the same target peaks at ~9.6–10 GB debugger-side (devenv + msvsmon, n=5), so the
patched 6.5 GB is already below the platform-native IDE debugger.
## Test plan
- `ninja check-lldb` on Windows (Release, `LLDB_ENABLE_PYTHON=OFF` → lit Shell + Unit suites; the
NativePDB coverage relevant to this patch is Shell-based, so it runs in this configuration.
API/dotest suites need Python and were not run): **32,299 passed / 30 failed** out of 34,219
discovered. All 30 failures were a test-harness infrastructure issue — `helper/build.py`
requires `lld-link`, which the build lacked (`LLVM_ENABLE_PROJECTS=clang;lldb`). The same 30
fail identically on unpatched `main` on the same host. After adding `lld` to the build, the
**30 rerun individually: 30/30 pass** (including the 9 `SymbolFile/NativePDB` tests). No
failures attributable to this patch.
- Functional smoke on the patched build (5/5): breakpoint hit, stack frame names, `frame variable`
locals, single-step, and **native `std::vector` rendering** (size=5, elements correct) — confirming
the lazy `tpi()` type-materialization path still works.
- Memory: LOD-on UE launch + attach RSS timelines (the tables above).
## Build note / deviation
The validation binaries were initially built with `-DLLDB_ENABLE_PYTHON=OFF -DLLDB_ENABLE_LUA=OFF`
(SWIG was not available on the build host); lldb 22 renders MSVC STL natively, so data-formatter
results do not depend on embedded Python. The build was later reconfigured with
`LLDB_ENABLE_PYTHON=ON` (SWIG 4.3.1) and re-passed the same functional smoke (5/5); the attach-path
RSS difference was negligible (+0.03 GB, single-point cross-run comparison).
## Scope / disclosures (please read before crediting the memory numbers)
- Measurements are **n=1 per configuration**, physical-memory basis, and depend on concurrent
background residents; the test project (a minimal C++ UE project) is a **floor** — a real game
editor (8–12 GB) would need the budget re-measured, and fit is not claimed there.
- Separately, when this patched build launches the editor with LOD on, the **target deadlocks during
init** (all target threads self-Wait, no game tick). That wedge is filed as a **separate issue**
(llvm/llvm-project#211170) and is **not actively maintained by this patch's lazy hydration at
wedge time** (the lldb debug loop is idle at wedge time; n=2, 4 independent measurements). Caveats
carried from that issue: no unpatched-LOD-on control run exists on this bed (unpatched balloons
before reaching that stage), so this is a *mechanism refutation*, not a counterfactual proof; the
pre-wedge symbol-materialization I/O is a candidate contributor to the init slowdown; and hydration
completion (~t92 s) overlaps the wedge onset in time, which this instrumentation did not exclude as
a factor. The attach path does not hit this wedge and is the recommended workflow — conditionally:
post-attach dynamic module loads, exception storms (PIE/ensure), and GUI interaction were not
soak-tested, and an intermittent (1/8) liblldb AV on disconnect after an attached session is
reported separately (llvm/llvm-project#211171).
Links: `SUMMARY-32gbfit.md` (derivation, 64 GB bed) + `realbed/` (32 GB validation) in the
companion repro repo: https://github.com/cherleey/lldb-ue-pdb-repro
>From 8299688f2e272b85025aded7d5f3946a4cad3083 Mon Sep 17 00:00:00 2001
From: cherleey <cherleey2 at gmail.com>
Date: Wed, 22 Jul 2026 14:36:05 +0900
Subject: [PATCH] [lldb][NativePDB] Defer PdbIndex creation until the symbol
file is queried
CalculateAbilities eagerly built the PdbIndex (parsing the type/symbol
record streams) for every candidate module at SymbolFile creation, before
symbols.load-on-demand could defer anything. For targets that load a few
hundred modules with GB-scale adjacent PDBs (an Unreal Engine editor with
engine symbols), this balloons to ~20 GB of anonymous heap and makes the
session un-runnable on a 32 GB machine.
Defer PdbIndex::create to GetOrCreateIndex(), called on first real symbol
query; CalculateAbilities now reads only the DBI stream header for the
stripped check and rejects PDBs lacking TPI/IPI so an unusable PDB does
not win plugin selection. Launch RSS on the UE target drops 20.3 GB ->
4.0 GB (no bp) / 6.3 GB (source bp), attach 22-25 GB (thrash/OOM) ->
6.5 GB peak, with breakpoints, stepping, frame variables and native MSVC
STL rendering intact.
---
.../Plugins/SymbolFile/NativePDB/PdbIndex.cpp | 36 ++++++-
.../Plugins/SymbolFile/NativePDB/PdbIndex.h | 21 ++++-
.../NativePDB/SymbolFileNativePDB.cpp | 93 ++++++++++++++++---
.../NativePDB/SymbolFileNativePDB.h | 14 ++-
4 files changed, 141 insertions(+), 23 deletions(-)
diff --git a/lldb/source/Plugins/SymbolFile/NativePDB/PdbIndex.cpp b/lldb/source/Plugins/SymbolFile/NativePDB/PdbIndex.cpp
index ea778fc6cca67..b2a39d2344349 100644
--- a/lldb/source/Plugins/SymbolFile/NativePDB/PdbIndex.cpp
+++ b/lldb/source/Plugins/SymbolFile/NativePDB/PdbIndex.cpp
@@ -45,20 +45,50 @@ PdbIndex::create(llvm::pdb::PDBFile *file) {
std::unique_ptr<PdbIndex> result(new PdbIndex());
ASSIGN_PTR_OR_RETURN(result->m_dbi, file->getPDBDbiStream());
- ASSIGN_PTR_OR_RETURN(result->m_tpi, file->getPDBTpiStream());
- ASSIGN_PTR_OR_RETURN(result->m_ipi, file->getPDBIpiStream());
ASSIGN_PTR_OR_RETURN(result->m_info, file->getPDBInfoStream());
ASSIGN_PTR_OR_RETURN(result->m_publics, file->getPDBPublicsStream());
ASSIGN_PTR_OR_RETURN(result->m_globals, file->getPDBGlobalsStream());
ASSIGN_PTR_OR_RETURN(result->m_symrecords, file->getPDBSymbolStream());
- result->m_tpi->buildHashMap();
+ // The TPI/IPI (type) streams — usually the largest streams of a big PDB —
+ // are materialized lazily in tpi()/ipi(); symtab-only consumers (e.g.
+ // AddSymbols from the publics stream) never pay for them.
result->m_file = file;
return std::move(result);
}
+llvm::pdb::TpiStream &PdbIndex::tpi() {
+ // Symbol parsing can run on multiple threads; materialize exactly once.
+ // Stream *presence* is validated during abilities probing, so a failure
+ // here means a corrupt stream in a PDB we already committed to — callers
+ // hold references, so the only honest exits are success or fatal (the
+ // pre-lazy behavior rejected such a PDB wholesale in create()).
+ llvm::call_once(m_tpi_once, [this] {
+ auto expected_tpi = m_file->getPDBTpiStream();
+ if (!expected_tpi) {
+ llvm::consumeError(expected_tpi.takeError());
+ llvm::report_fatal_error("PdbIndex: failed to load TPI stream");
+ }
+ m_tpi = &*expected_tpi;
+ m_tpi->buildHashMap();
+ });
+ return *m_tpi;
+}
+
+llvm::pdb::TpiStream &PdbIndex::ipi() {
+ llvm::call_once(m_ipi_once, [this] {
+ auto expected_ipi = m_file->getPDBIpiStream();
+ if (!expected_ipi) {
+ llvm::consumeError(expected_ipi.takeError());
+ llvm::report_fatal_error("PdbIndex: failed to load IPI stream");
+ }
+ m_ipi = &*expected_ipi;
+ });
+ return *m_ipi;
+}
+
lldb::addr_t PdbIndex::MakeVirtualAddress(uint16_t segment,
uint32_t offset) const {
uint32_t max_section = dbi().getSectionHeaders().size();
diff --git a/lldb/source/Plugins/SymbolFile/NativePDB/PdbIndex.h b/lldb/source/Plugins/SymbolFile/NativePDB/PdbIndex.h
index 796aa4c8dfd1b..2a350d82f99c8 100644
--- a/lldb/source/Plugins/SymbolFile/NativePDB/PdbIndex.h
+++ b/lldb/source/Plugins/SymbolFile/NativePDB/PdbIndex.h
@@ -13,6 +13,7 @@
#include "llvm/ADT/IntervalMap.h"
#include "llvm/DebugInfo/PDB/Native/PDBFile.h"
#include "llvm/DebugInfo/PDB/PDBTypes.h"
+#include "llvm/Support/Threading.h"
#include "CompileUnitIndex.h"
#include "PdbSymUid.h"
@@ -63,6 +64,8 @@ class PdbIndex {
/// the other way around.
llvm::pdb::TpiStream *m_tpi = nullptr;
llvm::pdb::TpiStream *m_ipi = nullptr;
+ llvm::once_flag m_tpi_once;
+ llvm::once_flag m_ipi_once;
/// This is called the "PDB Stream" in the Microsoft reference implementation.
/// It contains information about the structure of the file, as well as fields
@@ -121,11 +124,19 @@ class PdbIndex {
llvm::pdb::DbiStream &dbi() { return *m_dbi; }
const llvm::pdb::DbiStream &dbi() const { return *m_dbi; }
- llvm::pdb::TpiStream &tpi() { return *m_tpi; }
- const llvm::pdb::TpiStream &tpi() const { return *m_tpi; }
-
- llvm::pdb::TpiStream &ipi() { return *m_ipi; }
- const llvm::pdb::TpiStream &ipi() const { return *m_ipi; }
+ /// The TPI/IPI (type) streams are the largest streams of a big PDB and are
+ /// only needed once debug info (types) is actually consumed — symtab
+ /// construction from publics does not touch them. They are therefore
+ /// materialized lazily on first access rather than in create().
+ llvm::pdb::TpiStream &tpi();
+ const llvm::pdb::TpiStream &tpi() const {
+ return const_cast<PdbIndex *>(this)->tpi();
+ }
+
+ llvm::pdb::TpiStream &ipi();
+ const llvm::pdb::TpiStream &ipi() const {
+ return const_cast<PdbIndex *>(this)->ipi();
+ }
llvm::pdb::InfoStream &info() { return *m_info; }
const llvm::pdb::InfoStream &info() const { return *m_info; }
diff --git a/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.cpp b/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.cpp
index ec6e89b10e776..214ba98d3446f 100644
--- a/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.cpp
+++ b/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.cpp
@@ -34,6 +34,7 @@
#include "llvm/DebugInfo/CodeView/SymbolDeserializer.h"
#include "llvm/DebugInfo/CodeView/SymbolRecordHelpers.h"
#include "llvm/DebugInfo/CodeView/TypeDeserializer.h"
+#include "llvm/DebugInfo/MSF/MappedBlockStream.h"
#include "llvm/DebugInfo/PDB/Native/DbiStream.h"
#include "llvm/DebugInfo/PDB/Native/GlobalsStream.h"
#include "llvm/DebugInfo/PDB/Native/InfoStream.h"
@@ -173,6 +174,34 @@ loadMatchingPDBFile(std::string exe_path, llvm::BumpPtrAllocator &allocator) {
return pdb;
}
+// Reads only the DBI stream header to answer isStripped(). Parsing the full
+// DBI stream (let alone PdbIndex::create, which also pulls in the type and
+// symbol-record streams) materializes most of a large PDB on the private
+// heap; abilities probing runs for every candidate module at attach/launch,
+// so it must stay O(header).
+static std::optional<bool> IsDbiStripped(llvm::pdb::PDBFile &pdb) {
+ using namespace llvm::pdb;
+ if (!pdb.hasPDBDbiStream())
+ return std::nullopt;
+ auto stream_or_err = pdb.safelyCreateIndexedStream(
+ static_cast<uint32_t>(SpecialStream::StreamDBI));
+ if (!stream_or_err) {
+ llvm::consumeError(stream_or_err.takeError());
+ return std::nullopt;
+ }
+ std::unique_ptr<llvm::msf::MappedBlockStream> stream =
+ std::move(*stream_or_err);
+ if (stream->getLength() < sizeof(DbiStreamHeader))
+ return std::nullopt;
+ llvm::BinaryStreamReader reader(*stream);
+ const DbiStreamHeader *header = nullptr;
+ if (auto ec = reader.readObject(header)) {
+ llvm::consumeError(std::move(ec));
+ return std::nullopt;
+ }
+ return (header->Flags & DbiFlags::FlagStrippedMask) != 0;
+}
+
static bool IsFunctionPrologue(const CompilandIndexItem &cci,
lldb::addr_t addr) {
// FIXME: Implement this.
@@ -383,7 +412,7 @@ uint32_t SymbolFileNativePDB::CalculateAbilities() {
if (!m_objfile_sp)
return 0;
- if (!m_index) {
+ if (!m_pdb_file) {
// Lazily load and match the PDB file, but only do this once.
PDBFile *pdb_file;
if (auto *pdb = llvm::dyn_cast<ObjectFilePDB>(m_objfile_sp.get())) {
@@ -402,26 +431,53 @@ uint32_t SymbolFileNativePDB::CalculateAbilities() {
pdb_file->getFilePath(),
m_objfile_sp->GetModule()->GetObjectFile()->GetFileSpec().GetPath());
- auto expected_index = PdbIndex::create(pdb_file);
- if (!expected_index) {
- llvm::consumeError(expected_index.takeError());
- return 0;
- }
- m_index = std::move(*expected_index);
+ m_pdb_file = pdb_file;
}
- if (!m_index)
- return 0;
// We don't especially have to be precise here. We only distinguish between
- // stripped and not stripped.
- abilities = kAllAbilities;
+ // stripped and not stripped. Building the PdbIndex here would eagerly parse
+ // the type and symbol-record streams of every candidate module, so the
+ // stripped check reads just the DBI stream header instead — the index is
+ // built on demand in GetOrCreateIndex().
+ //
+ // PdbIndex::create requires the DBI/TPI/IPI streams; reject PDBs lacking
+ // them here so an unusable PDB does not win plugin selection only to fail
+ // when the index is materialized later.
+ if (!m_pdb_file->hasPDBTpiStream() || !m_pdb_file->hasPDBIpiStream())
+ return 0;
- if (m_index->dbi().isStripped())
+ std::optional<bool> stripped = IsDbiStripped(*m_pdb_file);
+ if (!stripped)
+ return 0;
+
+ abilities = kAllAbilities;
+ if (*stripped)
abilities &= ~(Blocks | LocalVariables);
return abilities;
}
+PdbIndex *SymbolFileNativePDB::GetOrCreateIndex() {
+ if (m_index)
+ return m_index.get();
+ if (!m_pdb_file)
+ return nullptr;
+
+ LLDB_LOG(GetLog(LLDBLog::Symbols), "Building PDB index for {0}",
+ m_objfile_sp->GetFileSpec().GetPath());
+
+ auto expected_index = PdbIndex::create(m_pdb_file);
+ if (!expected_index) {
+ LLDB_LOG_ERROR(GetLog(LLDBLog::Symbols), expected_index.takeError(),
+ "Failed to build PDB index: {0}");
+ return nullptr;
+ }
+ m_index = std::move(*expected_index);
+ return m_index.get();
+}
+
void SymbolFileNativePDB::InitializeObject() {
+ if (!GetOrCreateIndex())
+ return;
m_obj_load_address = m_objfile_sp->GetModule()
->GetObjectFile()
->GetBaseAddress()
@@ -442,6 +498,8 @@ void SymbolFileNativePDB::InitializeObject() {
}
uint32_t SymbolFileNativePDB::CalculateNumCompileUnits() {
+ if (!GetOrCreateIndex())
+ return 0;
const DbiModuleList &modules = m_index->dbi().modules();
uint32_t count = modules.getModuleCount();
if (count == 0)
@@ -1245,6 +1303,11 @@ lldb::LanguageType SymbolFileNativePDB::ParseLanguage(CompileUnit &comp_unit) {
}
void SymbolFileNativePDB::AddSymbols(Symtab &symtab) {
+ // Symtab construction is reachable before InitializeObject (e.g. the
+ // on-demand wrapper serves pre-hydration lookups from the symtab), so the
+ // index must be materialized here as well.
+ if (!GetOrCreateIndex())
+ return;
auto *section_list =
m_objfile_sp->GetModule()->GetObjectFile()->GetSectionList();
if (!section_list)
@@ -2759,7 +2822,11 @@ SymbolFileNativePDB::GetTypeSystemForLanguage(lldb::LanguageType language) {
uint64_t SymbolFileNativePDB::GetDebugInfoSize(bool load_all_debug_info) {
// PDB files are a separate file that contains all debug info.
- return m_index->pdb().getFileSize();
+ // Reachable before the index is materialized (e.g. `statistics dump` with
+ // on-demand symbol loading), so use the PDB file directly.
+ if (!m_pdb_file)
+ return 0;
+ return m_pdb_file->getFileSize();
}
void SymbolFileNativePDB::BuildParentMap() {
diff --git a/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.h b/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.h
index 4d5d9fb58bcac..5707b1acc72b2 100644
--- a/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.h
+++ b/lldb/source/Plugins/SymbolFile/NativePDB/SymbolFileNativePDB.h
@@ -158,8 +158,8 @@ class SymbolFileNativePDB : public SymbolFileCommon {
llvm::StringRef GetPluginName() override { return GetPluginNameStatic(); }
- llvm::pdb::PDBFile &GetPDBFile() { return m_index->pdb(); }
- const llvm::pdb::PDBFile &GetPDBFile() const { return m_index->pdb(); }
+ llvm::pdb::PDBFile &GetPDBFile() { return *m_pdb_file; }
+ const llvm::pdb::PDBFile &GetPDBFile() const { return *m_pdb_file; }
PdbIndex &GetIndex() { return *m_index; };
@@ -301,7 +301,17 @@ class SymbolFileNativePDB : public SymbolFileCommon {
// pdb debug info.
lldb::user_id_t anonymous_id = LLDB_INVALID_UID - 1;
+ /// Builds m_index on first use. PdbIndex::create eagerly parses the DBI,
+ /// type (TPI/IPI) and symbol-record streams — for large PDBs that is most
+ /// of the file materialized on the private heap — so it must not run
+ /// during abilities probing, only when debug info is actually consumed
+ /// (InitializeObject / symtab construction).
+ PdbIndex *GetOrCreateIndex();
+
std::unique_ptr<llvm::pdb::PDBFile> m_file_up;
+ /// The matched PDB file (owned by m_file_up, or by the ObjectFilePDB when
+ /// the module's object file IS the PDB). Set by CalculateAbilities.
+ llvm::pdb::PDBFile *m_pdb_file = nullptr;
std::unique_ptr<PdbIndex> m_index;
llvm::DenseMap<lldb::user_id_t, lldb::VariableSP> m_global_vars;
More information about the lldb-commits
mailing list