[lld] [llvm] WIP - [DTLTO] Speed up temporary file removal "2" (PR #209423)
Ben Dunbobbin via llvm-commits
llvm-commits at lists.llvm.org
Tue Jul 14 02:58:36 PDT 2026
https://github.com/bd1976bris created https://github.com/llvm/llvm-project/pull/209423
Deleting the temporary files produced by the DTLTO pipeline can be expensive on Windows hosts. For a Clang link (Debug build with sanitizers and instrumentation) using an optimized toolchain (PGO non-LTO, llvmorg-22.1.0) on a Windows 11 Pro (Build 26200), AMD Family 25 @ ~4.5 GHz, 16 cores/32 threads, 64 GB RAM machine, the mean duration of the "Remove DTLTO temporary files" time trace scope was 1267.789 ms (measured over 10 runs).
This patch performs the deletions on a background thread, allowing them to overlap with the tail of the link to hide this cost.
This is a re-implementation of the asynchronous cleanup idea from https://github.com/llvm/llvm-project/pull/186988, which had to be reverted in https://github.com/llvm/llvm-project/pull/189043 because cleanup was not guaranteed to complete before LLD invoked timeTraceProfilerCleanup(). In certain cases timeTraceProfilerCleanup() was called before temporary file deletion had completed in LLD, which caused memory leaks that were flagged by sanitizer builds.
Note that the DTLTO implementation has been refactored heavily since https://github.com/llvm/llvm-project/pull/186988, so this is a re-implementation along the same lines.
To solve the ordering issue, LLD now destroys the LTO owner before time trace serialization and cleanup. The DTLTO destructor waits for any queued temporary file deletion task to finish. This preserves overlap with the tail of the link while avoiding shutdown-order dependence on ManagedStatic or process-exit cleanup.
>From 2fe2a0a2c6b32e341abb4f342099ed5f4d6c0af9 Mon Sep 17 00:00:00 2001
From: Dunbobbin <Ben.Dunbobbin at sony.com>
Date: Fri, 10 Jul 2026 21:55:24 +0100
Subject: [PATCH] [DTLTO] Speed up temporary file removal "2"
Deleting the temporary files produced by the DTLTO pipeline can be expensive on
Windows hosts. For a Clang link (Debug build with sanitizers and
instrumentation) using an optimized toolchain (PGO non-LTO, llvmorg-22.1.0) on
a Windows 11 Pro (Build 26200), AMD Family 25 @ ~4.5 GHz, 16 cores/32
threads, 64 GB RAM machine, the mean duration of the "Remove DTLTO temporary
files" time trace scope was 1267.789 ms (measured over 10 runs).
This patch performs the deletions on a background thread, allowing them to
overlap with the tail of the link to hide this cost.
This is a re-implementation of the asynchronous cleanup idea from
https://github.com/llvm/llvm-project/pull/186988, which had to be reverted in
https://github.com/llvm/llvm-project/pull/189043 because cleanup was not
guaranteed to complete before LLD invoked timeTraceProfilerCleanup(). In certain
cases timeTraceProfilerCleanup() was called before temporary file deletion had
completed in LLD, which caused memory leaks that were flagged by sanitizer
builds.
Note that the DTLTO implementation has been refactored heavily since
https://github.com/llvm/llvm-project/pull/186988, so this is a re-implementation
along the same lines.
To solve the ordering issue, LLD now destroys the LTO owner before time trace
serialization and cleanup. The DTLTO destructor waits for any queued temporary
file deletion task to finish. This preserves overlap with the tail of the link
while avoiding shutdown-order dependence on ManagedStatic or process-exit
cleanup.
---
cross-project-tests/dtlto/savetemps-lock.test | 14 ++--
lld/COFF/Driver.cpp | 1 +
lld/COFF/SymbolTable.cpp | 2 +
lld/COFF/SymbolTable.h | 2 +
lld/ELF/Config.h | 1 +
lld/ELF/Driver.cpp | 4 +
llvm/include/llvm/DTLTO/DTLTO.h | 12 ++-
llvm/lib/DTLTO/DTLTO.cpp | 73 +++++++++++++++----
8 files changed, 85 insertions(+), 24 deletions(-)
diff --git a/cross-project-tests/dtlto/savetemps-lock.test b/cross-project-tests/dtlto/savetemps-lock.test
index 17ada42a5ef14..fd076cb8bec76 100644
--- a/cross-project-tests/dtlto/savetemps-lock.test
+++ b/cross-project-tests/dtlto/savetemps-lock.test
@@ -28,14 +28,14 @@ RUN: | FileCheck %s --implicit-check-not=warning
# Sanity check for the expected test_temps.py behaviour.
CHECK-DAG: Lock any files in the output directory.
-CHECK-DAG: warning: could not remove the file 'locked{{/|\\}}t.[[#PID:]].dist-file.json': {{.*}}
+CHECK-DAG: warning: could not remove temporary DTLTO file 'locked{{/|\\}}t.[[#PID:]].dist-file.json': {{.*}}
# Filename composition: <archive>(<member> at <offset>).<task>.<pid>.<task>.<pid>.native.o.
-CHECK-DAG: warning: could not remove the file 'locked{{/|\\}}t.a(t1.o at [[#T1_OFFSET:]]).1.[[#%X,HEXPID:]].o': {{.*}}
-CHECK-DAG: warning: could not remove the file 'locked{{/|\\}}t.a(t1.o at [[#T1_OFFSET:]]).1.[[#%X,HEXPID:]].1.[[#PID]].native.o': {{.*}}
-CHECK-DAG: warning: could not remove the file 'locked{{/|\\}}t.a(t1.o at [[#T1_OFFSET]]).1.[[#%X,HEXPID]].1.[[#PID]].native.o.thinlto.bc': {{.*}}
-CHECK-DAG: warning: could not remove the file 'locked{{/|\\}}t.a(t2.o at [[#T2_OFFSET:]]).2.[[#%X,HEXPID]].o': {{.*}}
-CHECK-DAG: warning: could not remove the file 'locked{{/|\\}}t.a(t2.o at [[#T2_OFFSET:]]).2.[[#%X,HEXPID]].2.[[#PID]].native.o': {{.*}}
-CHECK-DAG: warning: could not remove the file 'locked{{/|\\}}t.a(t2.o at [[#T2_OFFSET]]).2.[[#%X,HEXPID]].2.[[#PID]].native.o.thinlto.bc': {{.*}}
+CHECK-DAG: warning: could not remove temporary DTLTO file 'locked{{/|\\}}t.a(t1.o at [[#T1_OFFSET:]]).1.[[#%X,HEXPID:]].o': {{.*}}
+CHECK-DAG: warning: could not remove temporary DTLTO file 'locked{{/|\\}}t.a(t1.o at [[#T1_OFFSET:]]).1.[[#%X,HEXPID:]].1.[[#PID]].native.o': {{.*}}
+CHECK-DAG: warning: could not remove temporary DTLTO file 'locked{{/|\\}}t.a(t1.o at [[#T1_OFFSET]]).1.[[#%X,HEXPID]].1.[[#PID]].native.o.thinlto.bc': {{.*}}
+CHECK-DAG: warning: could not remove temporary DTLTO file 'locked{{/|\\}}t.a(t2.o at [[#T2_OFFSET:]]).2.[[#%X,HEXPID]].o': {{.*}}
+CHECK-DAG: warning: could not remove temporary DTLTO file 'locked{{/|\\}}t.a(t2.o at [[#T2_OFFSET:]]).2.[[#%X,HEXPID]].2.[[#PID]].native.o': {{.*}}
+CHECK-DAG: warning: could not remove temporary DTLTO file 'locked{{/|\\}}t.a(t2.o at [[#T2_OFFSET]]).2.[[#%X,HEXPID]].2.[[#PID]].native.o.thinlto.bc': {{.*}}
#--- t1.c
__attribute__((retain)) int t1(int x) { return x; }
diff --git a/lld/COFF/Driver.cpp b/lld/COFF/Driver.cpp
index dc4903a5fe126..55d1097e3993b 100644
--- a/lld/COFF/Driver.cpp
+++ b/lld/COFF/Driver.cpp
@@ -2960,6 +2960,7 @@ void LinkerDriver::linkerMain(ArrayRef<const char *> argsArr) {
// Write the result.
writeResult(ctx);
+ ctx.forEachSymtab([](SymbolTable &symtab) { symtab.cleanupLTO(); });
// Stop early so we can print the results.
rootTimer.stop();
diff --git a/lld/COFF/SymbolTable.cpp b/lld/COFF/SymbolTable.cpp
index df540be800e41..a912718688f6d 100644
--- a/lld/COFF/SymbolTable.cpp
+++ b/lld/COFF/SymbolTable.cpp
@@ -1511,4 +1511,6 @@ void SymbolTable::compileBitcodeFiles() {
}
}
+void SymbolTable::cleanupLTO() { lto.reset(); }
+
} // namespace lld::coff
diff --git a/lld/COFF/SymbolTable.h b/lld/COFF/SymbolTable.h
index aadd366c7d39f..62bf762afad00 100644
--- a/lld/COFF/SymbolTable.h
+++ b/lld/COFF/SymbolTable.h
@@ -109,6 +109,8 @@ class SymbolTable {
// added and before the writer writes results to a file.
void compileBitcodeFiles();
+ void cleanupLTO();
+
// Creates an Undefined symbol and marks it as live.
Symbol *addGCRoot(StringRef sym, bool aliasEC = false);
diff --git a/lld/ELF/Config.h b/lld/ELF/Config.h
index 12b16ded61fca..ad186ba3fd67d 100644
--- a/lld/ELF/Config.h
+++ b/lld/ELF/Config.h
@@ -215,6 +215,7 @@ class LinkerDriver {
void createFiles(llvm::opt::InputArgList &args);
void loadFiles();
void inferMachineType();
+ void cleanupLTO();
template <class ELFT> void link(llvm::opt::InputArgList &args);
template <class ELFT> void compileBitcodeFiles(bool skipLinkedOutput);
// True if we are in --whole-archive and --no-whole-archive.
diff --git a/lld/ELF/Driver.cpp b/lld/ELF/Driver.cpp
index 7ec7dfcae6bca..2eeabfb212a18 100644
--- a/lld/ELF/Driver.cpp
+++ b/lld/ELF/Driver.cpp
@@ -625,6 +625,8 @@ constexpr const char *saveTempsValues[] = {
LinkerDriver::LinkerDriver(Ctx &ctx) : ctx(ctx) {}
+void LinkerDriver::cleanupLTO() { lto.reset(); }
+
void LinkerDriver::linkerMain(ArrayRef<const char *> argsArr) {
ELFOptTable parser;
opt::InputArgList args = parser.parse(ctx, argsArr.slice(1));
@@ -707,6 +709,8 @@ void LinkerDriver::linkerMain(ArrayRef<const char *> argsArr) {
invokeELFT(link, args);
}
+ cleanupLTO();
+
if (ctx.arg.timeTraceEnabled) {
checkError(ctx.e, timeTraceProfilerWrite(
args.getLastArgValue(OPT_time_trace_eq).str(),
diff --git a/llvm/include/llvm/DTLTO/DTLTO.h b/llvm/include/llvm/DTLTO/DTLTO.h
index 6b09b0e4794d9..879ae5d740f5e 100644
--- a/llvm/include/llvm/DTLTO/DTLTO.h
+++ b/llvm/include/llvm/DTLTO/DTLTO.h
@@ -19,10 +19,13 @@
#include "llvm/ADT/DenseSet.h"
#include "llvm/ADT/SmallString.h"
#include "llvm/LTO/LTO.h"
-#include "llvm/Support/ManagedStatic.h"
#include "llvm/Support/Signals.h"
+#include "llvm/Support/ThreadPool.h"
#include <functional>
+#include <mutex>
+#include <string>
+#include <utility>
#include <vector>
namespace llvm {
@@ -67,6 +70,8 @@ class LLVM_ABI DTLTO : public LTO {
LTOMode = lto::LTO::LTOKind::LTOK_UnifiedThin;
}
+ ~DTLTO() override;
+
// Create an instance of WriteIndexesBackend class.
static lto::ThinBackend writeIndexesBackendInstance() {
return lto::createWriteIndexesThinBackend(hardware_concurrency(), "", "",
@@ -359,6 +364,11 @@ class LLVM_ABI DTLTO : public LTO {
// Cleanup files list.
std::vector<std::string> CleanupList;
+ DefaultThreadPool TempFileDeletePool{hardware_concurrency(1)};
+ std::mutex TempFileDeleteWarningsMutex;
+ std::vector<std::pair<std::string, std::string>> TempFileDeleteWarnings;
+ void enqueueTempFileDeletes(std::vector<std::string> Paths);
+ void waitForTempFileDeleteTasks();
// Record a file for cleanup and register signal-time removal if requested.
void addToCleanup(StringRef Filename) {
diff --git a/llvm/lib/DTLTO/DTLTO.cpp b/llvm/lib/DTLTO/DTLTO.cpp
index 6805746f64978..bdb0ea5c986af 100644
--- a/llvm/lib/DTLTO/DTLTO.cpp
+++ b/llvm/lib/DTLTO/DTLTO.cpp
@@ -26,28 +26,69 @@
#include "llvm/Support/TimeProfiler.h"
#include "llvm/Support/raw_ostream.h"
+#include <mutex>
#include <string>
+#include <system_error>
+#include <utility>
+#include <vector>
using namespace llvm;
+lto::DTLTO::~DTLTO() { waitForTempFileDeleteTasks(); }
+
+void lto::DTLTO::enqueueTempFileDeletes(std::vector<std::string> Paths) {
+ if (Paths.empty())
+ return;
+
+ TempFileDeletePool.async([this, Paths = std::move(Paths)] {
+ if (LLVM_ENABLE_THREADS && Conf.TimeTraceEnabled)
+ timeTraceProfilerInitialize(Conf.TimeTraceGranularity,
+ "DTLTO temporary file cleanup");
+
+ {
+ TimeTraceScope JobScope("Remove DTLTO temporary files");
+ for (const std::string &Path : Paths) {
+ if (Path.empty())
+ continue;
+
+ std::error_code EC = sys::fs::remove(Path, true);
+ if (!EC ||
+ EC == std::make_error_code(std::errc::no_such_file_or_directory))
+ continue;
+
+ std::lock_guard<std::mutex> Guard(TempFileDeleteWarningsMutex);
+ TempFileDeleteWarnings.emplace_back(Path, EC.message());
+ }
+ }
+
+ if (LLVM_ENABLE_THREADS && Conf.TimeTraceEnabled)
+ timeTraceProfilerFinishThread();
+ });
+}
+
+void lto::DTLTO::waitForTempFileDeleteTasks() {
+ TempFileDeletePool.wait();
+
+ std::vector<std::pair<std::string, std::string>> Warnings;
+ {
+ std::lock_guard<std::mutex> Guard(TempFileDeleteWarningsMutex);
+ Warnings = std::move(TempFileDeleteWarnings);
+ TempFileDeleteWarnings.clear();
+ }
+
+ for (const auto &Warning : Warnings)
+ errs() << "warning: could not remove temporary DTLTO file '"
+ << Warning.first << "': " << Warning.second << "\n";
+}
+
// Remove temporary files created to enable distribution.
void lto::DTLTO::cleanup() {
- if (!SaveTemps) {
- // Remove one file, report error if any.
- auto removeFile = [](StringRef FileName) -> void {
- std::error_code EC = sys::fs::remove(FileName, true);
- if (EC &&
- EC != std::make_error_code(std::errc::no_such_file_or_directory))
- errs() << "warning: could not remove the file '" << FileName
- << "': " << EC.message() << "\n";
- };
-
- TimeTraceScope JobScope("Remove DTLTO temporary files");
- for (const auto &Name : CleanupList)
- removeFile(Name);
- // Clean the CleanupList for safety.
- CleanupList.clear();
- }
+ if (SaveTemps)
+ return;
+
+ enqueueTempFileDeletes(std::move(CleanupList));
+ // Clean the CleanupList for safety.
+ CleanupList.clear();
}
// Runs the DTLTO thin link phase, producing per-module summary indices,
More information about the llvm-commits
mailing list