[Lldb-commits] [lldb] [lldb] Add a DataFileCache for ObjectFile header bytes (PR #231344)
Jason Molenda via lldb-commits
lldb-commits at lists.llvm.org
Mon Oct 12 22:23:45 PDT 2026
https://github.com/jasonmolenda updated https://github.com/llvm/llvm-project/pull/231344
>From e67432b087ec7a83c70576d372da56be1d0de676 Mon Sep 17 00:00:00 2001
From: Jason Molenda <jmolenda at apple.com>
Date: Mon, 12 Oct 2026 18:09:04 -0700
Subject: [PATCH 1/2] [lldb] Add a DataFileCache for ObjectFile header bytes
When creating a memory-read module, lldb needs to read the binary's
header & load commands (Mach-O terminology) out of memory, and then
read the symbol table out of memory. In
https://github.com/llvm/llvm-project/pull/229305 I added a new
setting default-on which saves the symbol table for a memory-read
binary in the DataFileCache. But we're still reading the header
and load command bytes out of memory on every debug session, and
if there are a large number of binary images being read from memory,
this packet communication will be a real performance problem.
This patch enables a new DataFileCache file with the same
`symbols.enable-lldb-index-cache-memory-modules` default-on setting,
"headerbytes". After creating a memory-read binary, it saves the
bytes of the header to the DataFileCache. And before reading a new
memory-read binary, it looks in the DataFileCache to see if the
header bytes are already available. If so, it puts those bytes in
the Process memory-cache, and allows the ObjectFile plugin sublcass
to read them.
The symtab files in DataFileCache have a naming convention designed
to identify a unique file, including hashing the full path and binary
type and triple into a couple of 32-bit hashes in the name, e.g.
llvmcache-arm64-apple-macosx27.0.0-a.out-0x662f71f5-symtab-0x16017900
and then the UUID is stored in the contents of the symtab file, after
the version number This allows for recompilation/incremental
development, where the same file rebuilt & re-debugged will not result
in multiple files in the DataFileCache. The filepath/triple/etc will
hash to the same values, lldb will find a symtab with a non-matching
UUID, and replace the old one with the current one.
For a memory-read module, at the point we're creating them, we have
less information -- a filepath and a UUID, mostly. And incremental
development is not an expected use case for these. So For these
I'm using the filename alone and a UUID,
llvmcache-a.out-631A3D4F-311B-4F63-B9E3-10F83B4F97E3-headerbytes
If a.out is rebuilt here and we debug it again as a memory module,
it will have a new UUID and the previous headerbytes file will
persist, until the DataFileCache expiration settings take effect
for the file and remove it. I think the use use case for these
memory-read binaries is different enough that this is a reasonable
approach. The default policy is that these files will be removed
if not accessed for 7 days, the next time lldb runs and touches
the DataFileCache.
I added a new method to Process, ReadModuleFromMemoryWithCachedHeader()
which mimics the ReadModuleFromMemory() method, with the addition
of a UUID. I didn't want to replace Process::ReadModuleFromMemory()
because there are use cases (like corefile debugging) where we may
be reading from "memory" but creating a cache of the bytes would
be a waste. This is opt-in for a DynamicLoader that wants to use
it. DynamicLoaderDarwin is using it.
I enhanced my previous TestMemoryReadModuleSymtab test (which
attaches to a memory-read binary to create the DataCacheFiles,
detaches, then attaches with a new Target and looks at what memory
reads lldb made from the second session) to also look for any reads
of the binary header (the difference between the start of __TEXT
segment and the start of __text section).
rdar://188329546
---
lldb/include/lldb/Target/Process.h | 23 ++++++
.../MacOSX-DYLD/DynamicLoaderDarwin.cpp | 7 +-
lldb/source/Target/Process.cpp | 79 +++++++++++++++++++
.../TestMemoryReadModuleSymtab.py | 29 +++++--
4 files changed, 130 insertions(+), 8 deletions(-)
diff --git a/lldb/include/lldb/Target/Process.h b/lldb/include/lldb/Target/Process.h
index 226c58034dd65..cf29ca547e4ab 100644
--- a/lldb/include/lldb/Target/Process.h
+++ b/lldb/include/lldb/Target/Process.h
@@ -25,6 +25,7 @@
#include "lldb/Breakpoint/BreakpointSite.h"
#include "lldb/Breakpoint/StopPointSiteList.h"
#include "lldb/Breakpoint/WatchpointResource.h"
+#include "lldb/Core/DataFileCache.h"
#include "lldb/Core/LoadedModuleInfoList.h"
#include "lldb/Core/PluginInterface.h"
#include "lldb/Core/SourceManager.h"
@@ -2079,6 +2080,15 @@ class Process : public std::enable_shared_from_this<Process>,
ReadModuleFromMemory(const FileSpec &file_spec, lldb::addr_t header_addr,
size_t size_to_read = 512);
+ /// Read the DataFileCache 'headerbytes' cache before creating an
+ /// ObjectFile from live memory, if it is cached on the debug host
+ /// already. Create the 'headerbytes' cache file if it did not exist
+ /// yet.
+ llvm::Expected<lldb::ModuleSP>
+ ReadModuleFromMemoryWithCachedHeader(const FileSpec &file_spec, UUID uuid,
+ lldb::addr_t header_addr,
+ size_t size_to_read = 512);
+
/// Attempt to get the attributes for a region of memory in the process.
///
/// It may be possible for the remote debug server to inspect attributes for
@@ -3777,6 +3787,19 @@ void PruneThreadPlans();
lldb::EventSP CreateEventFromProcessState(uint32_t event_type);
+ /// Load the header bytes of an in-memory binary from the DataFileCache
+ /// 'headerbytes' file, if it exists, and put the bytes in the Process
+ /// memory cache.
+ bool LoadHeaderBytesFromCache(const FileSpec &file_spec, UUID uuid,
+ lldb::addr_t header_addr, size_t size_to_read,
+ lldb_private::DataFileCache *cache);
+
+ /// Save the header bytes of an in-memory binary into the DataFileCache
+ /// 'headerbytes' file.
+ void SaveHeaderBytesToCache(const FileSpec &file_spec, UUID uuid,
+ lldb::addr_t header_addr, size_t size_to_read,
+ lldb_private::DataFileCache *cache);
+
Process(const Process &) = delete;
const Process &operator=(const Process &) = delete;
};
diff --git a/lldb/source/Plugins/DynamicLoader/MacOSX-DYLD/DynamicLoaderDarwin.cpp b/lldb/source/Plugins/DynamicLoader/MacOSX-DYLD/DynamicLoaderDarwin.cpp
index 0016b5b450a65..280553154be58 100644
--- a/lldb/source/Plugins/DynamicLoader/MacOSX-DYLD/DynamicLoaderDarwin.cpp
+++ b/lldb/source/Plugins/DynamicLoader/MacOSX-DYLD/DynamicLoaderDarwin.cpp
@@ -179,9 +179,10 @@ ModuleSP DynamicLoaderDarwin::FindTargetModuleForImageInfo(
// We'll call Target::ModulesDidLoad after all the modules have been
// added to the target, don't let it be called for every one.
if (!module_sp || module_sp->GetObjectFile() == nullptr) {
- llvm::Expected<ModuleSP> module_sp_or_err = m_process->ReadModuleFromMemory(
- image_info.file_spec, image_info.address,
- image_info.mh_and_load_cmd_size);
+ llvm::Expected<ModuleSP> module_sp_or_err =
+ m_process->ReadModuleFromMemoryWithCachedHeader(
+ image_info.file_spec, image_info.uuid, image_info.address,
+ image_info.mh_and_load_cmd_size);
if (auto err = module_sp_or_err.takeError()) {
LLDB_LOG_ERROR(GetLog(LLDBLog::DynamicLoader), std::move(err),
"Failed to load module from memory: {0}");
diff --git a/lldb/source/Target/Process.cpp b/lldb/source/Target/Process.cpp
index c1343fa196f6e..a61264f9b4e9a 100644
--- a/lldb/source/Target/Process.cpp
+++ b/lldb/source/Target/Process.cpp
@@ -18,6 +18,7 @@
#include "lldb/Breakpoint/BreakpointLocation.h"
#include "lldb/Breakpoint/StoppointCallbackContext.h"
+#include "lldb/Core/DataFileCache.h"
#include "lldb/Core/Debugger.h"
#include "lldb/Core/Module.h"
#include "lldb/Core/ModuleSpec.h"
@@ -70,6 +71,7 @@
#include "lldb/Target/ThreadPlanStack.h"
#include "lldb/Target/UnixSignals.h"
#include "lldb/Utility/AddressableBits.h"
+#include "lldb/Utility/DataEncoder.h"
#include "lldb/Utility/Event.h"
#include "lldb/Utility/LLDBLog.h"
#include "lldb/Utility/Log.h"
@@ -2833,6 +2835,83 @@ Process::ReadModuleFromMemory(const FileSpec &file_spec,
return error.takeError();
}
+// Unlike the Symtab.cpp naming convention which encodes the
+// triple, filename, and filepath into hashes in the name
+// and puts the UUID in the file contents (which allows for
+// incremental rebuilding of a file without keeping old versions
+// of the file in the DataFileCache), for memory-read modules we
+// have less information at this point -- a filename and UUID only,
+// so we construct our key based on them.
+// If a developer is reading a binary under active development and
+// rebuilding, this means that the cache will keep all previous
+// caches of the header bytes in the DataFileCache, relying on the
+// DFC expiration policies to remove the previous versions.
+static std::string get_memory_binary_cache_key(const FileSpec &file_spec,
+ UUID uuid) {
+ std::string key;
+ llvm::raw_string_ostream strm(key);
+ strm << file_spec.GetFilename().str() << '-';
+ strm << uuid.GetAsString();
+ strm << "-headerbytes";
+ return key;
+}
+
+bool Process::LoadHeaderBytesFromCache(const FileSpec &file_spec, UUID uuid,
+ lldb::addr_t header_addr,
+ size_t size_to_read,
+ DataFileCache *cache) {
+ std::unique_ptr<llvm::MemoryBuffer> mem_buffer_up =
+ cache->GetCachedData(get_memory_binary_cache_key(file_spec, uuid));
+ if (!mem_buffer_up)
+ return false;
+
+ // Stuff the header bytes into the Process memory cache.
+ // The ObjectFile will read the bytes from there at this
+ // same stop.
+ WritableDataBufferSP databuf_sp = std::make_shared<DataBufferHeap>(
+ mem_buffer_up->getBufferStart(), mem_buffer_up->getBufferSize());
+ AddCacheData(header_addr, databuf_sp);
+ return true;
+}
+
+void Process::SaveHeaderBytesToCache(const FileSpec &file_spec, UUID uuid,
+ lldb::addr_t header_addr,
+ size_t size_to_read,
+ DataFileCache *cache) {
+ ProcessAddress header_address(header_addr);
+ std::vector<uint8_t> data(size_to_read);
+ Status error;
+ if (ReadMemory(header_address, data.data(), size_to_read, error) !=
+ size_to_read ||
+ !error.Success())
+ return;
+
+ const auto byte_order = endian::InlHostByteOrder();
+ DataEncoder file(byte_order, /*addr_size=*/8);
+ llvm::ArrayRef<uint8_t> data_ref(data.data(), size_to_read);
+ file.AppendData(data_ref);
+
+ cache->SetCachedData(get_memory_binary_cache_key(file_spec, uuid),
+ file.GetData());
+}
+
+llvm::Expected<ModuleSP> Process::ReadModuleFromMemoryWithCachedHeader(
+ const FileSpec &file_spec, UUID uuid, lldb::addr_t header_addr,
+ size_t size_to_read) {
+ DataFileCache *cache = Module::GetIndexCache(/* memory_module= */ true);
+ if (!cache)
+ return ReadModuleFromMemory(file_spec, header_addr, size_to_read);
+
+ bool cache_was_used = LoadHeaderBytesFromCache(file_spec, uuid, header_addr,
+ size_to_read, cache);
+ llvm::Expected<ModuleSP> module_sp_or_err =
+ ReadModuleFromMemory(file_spec, header_addr, size_to_read);
+ if (module_sp_or_err && *module_sp_or_err)
+ if (!cache_was_used)
+ SaveHeaderBytesToCache(file_spec, uuid, header_addr, size_to_read, cache);
+ return module_sp_or_err;
+}
+
bool Process::GetLoadAddressPermissions(lldb::addr_t load_addr,
uint32_t &permissions) {
MemoryRegionInfo range_info;
diff --git a/lldb/test/API/macosx/memory-read-module-symtab/TestMemoryReadModuleSymtab.py b/lldb/test/API/macosx/memory-read-module-symtab/TestMemoryReadModuleSymtab.py
index 7a428c45003fb..0d126a428ef1e 100644
--- a/lldb/test/API/macosx/memory-read-module-symtab/TestMemoryReadModuleSymtab.py
+++ b/lldb/test/API/macosx/memory-read-module-symtab/TestMemoryReadModuleSymtab.py
@@ -115,7 +115,7 @@ def test_memory_read_module_symtab(self):
self.dbg.MemoryPressureDetected()
# Enable logging of gdb-remote packets, to look for any
- # reads of a.out's __LINKEDIT in this second attach.
+ # reads of a.out's __TEXT or __LINKEDIT in this second attach.
logfile = self.packet_log("packets")
# Second attach: Read the DataFileCache for the MemoryModule.
@@ -127,9 +127,14 @@ def test_memory_read_module_symtab(self):
exe_filespec = lldb.SBFileSpec("a.out")
exe_module = target.FindModule(exe_filespec)
- print(exe_module)
+ exe_text = exe_module.FindSection("__TEXT")
+ self.assertTrue(exe_text.IsValid())
+ exe_text_text = exe_text.FindSubSection("__text")
+ self.assertTrue(exe_text_text.IsValid())
+ exe_header_start = exe_text.GetLoadAddress(target)
+ exe_header_end = exe_text_text.GetLoadAddress(target)
+
exe_linkedit = exe_module.FindSection("__LINKEDIT")
- print(exe_linkedit)
self.assertTrue(exe_linkedit.IsValid())
linkedit_startaddr = exe_linkedit.GetLoadAddress(target)
linkedit_endaddr = linkedit_startaddr + exe_linkedit.GetByteSize()
@@ -139,10 +144,24 @@ def test_memory_read_module_symtab(self):
# Iterate through the start & end address of every memory
# read packet, and confirm that none of them are within
- # the VM range of a.out's __LINKEDIT. The SymbolTable for
- # a.out should be gotten from the DataFileCache, not memory.
+ # 1. the VM range of a.out's Mach-O header and load commands
+ # (the bytes between the start of __TEXT and the start of __text)
+ # 2. the VM range of __LINKEDIT.
+ #
+ # The Mach-O header and load commands should have been read from
+ # the 'headerbytes' DataCacheFile, and the SymbolTable aka LINKEDIT
+ # should have been read from the 'symtab' DataCacheFile.
for addrs in memory_read_addrs:
for addr in addrs:
+ if addr >= exe_header_start and addr < exe_header_end:
+ print(
+ "Address 0x%x was read from a.out's Mach-O header and load commands. It should have been retrieved from the 'headerbytes' DataFileCache file."
+ % addr
+ )
+ if self.TraceOn():
+ self.runCmd("target modules dump sections a.out")
+ print("Read of address 0x%x is within Mach-O header" % addr)
+ self.assertFalse(addr >= linkedit_startaddr and addr < linkedit_endaddr)
if addr >= linkedit_startaddr and addr < linkedit_endaddr:
print(
"Address 0x%x was read from a.out's __LINKEDIT; all symbol table information should have been retrieved from DataFileCache of it."
>From 0491e43a9d14ba9bae70b3694f2496b4a80f95ff Mon Sep 17 00:00:00 2001
From: Jason Molenda <jmolenda at apple.com>
Date: Mon, 12 Oct 2026 22:23:25 -0700
Subject: [PATCH 2/2] test tweak.
---
.../TestMemoryReadModuleSymtab.py | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/lldb/test/API/macosx/memory-read-module-symtab/TestMemoryReadModuleSymtab.py b/lldb/test/API/macosx/memory-read-module-symtab/TestMemoryReadModuleSymtab.py
index 0d126a428ef1e..c8389a22f1729 100644
--- a/lldb/test/API/macosx/memory-read-module-symtab/TestMemoryReadModuleSymtab.py
+++ b/lldb/test/API/macosx/memory-read-module-symtab/TestMemoryReadModuleSymtab.py
@@ -153,6 +153,9 @@ def test_memory_read_module_symtab(self):
# should have been read from the 'symtab' DataCacheFile.
for addrs in memory_read_addrs:
for addr in addrs:
+
+ # Check that the address isn't in the range of the
+ # mach-o header and load commands.
if addr >= exe_header_start and addr < exe_header_end:
print(
"Address 0x%x was read from a.out's Mach-O header and load commands. It should have been retrieved from the 'headerbytes' DataFileCache file."
@@ -161,7 +164,10 @@ def test_memory_read_module_symtab(self):
if self.TraceOn():
self.runCmd("target modules dump sections a.out")
print("Read of address 0x%x is within Mach-O header" % addr)
- self.assertFalse(addr >= linkedit_startaddr and addr < linkedit_endaddr)
+ self.assertFalse(addr >= exe_header_start and addr < exe_header_end)
+
+ # Check that the address isn't in the range of the
+ # LINKEDIT segment.
if addr >= linkedit_startaddr and addr < linkedit_endaddr:
print(
"Address 0x%x was read from a.out's __LINKEDIT; all symbol table information should have been retrieved from DataFileCache of it."
More information about the lldb-commits
mailing list