[llvm] [Support] Introduce driver for running tools as daemons (PR #193706)
via llvm-commits
llvm-commits at lists.llvm.org
Thu Apr 23 03:02:12 PDT 2026
llvmbot wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-llvm-support
Author: Benjamin Stott (BStott6)
<details>
<summary>Changes</summary>
This PR introduces the general LLVM-side code for the [test daemonization project](https://discourse.llvm.org/t/rfc-reducing-process-creation-overhead-in-llvm-regression-tests/88612/9), including the implementation for the daemon IPC protocol. `lib/Support/DaemonDriver.cpp` implements the main function for the daemon, and `include/llvm/Support/ToolInterface.h` introduces the interface that tools can adopt to support daemonization.
Tools must implement the `LLVMTool` interface to support daemonization. A new utility target `ExampleDaemon` is also introduced, which is a faux tool used for testing the daemon IPC implementation - see `test/Support/DaemonDriver`.
See `llvm/docs/DaemonMode.rst` for more information. A working demo of this project for the `llvm/Transforms` tests can be found [here](https://github.com/BStott6/llvm-project/tree/new-daemonized-testing-demo).
---
Patch is 48.50 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/193706.diff
29 Files Affected:
- (modified) llvm/CMakeLists.txt (+1)
- (added) llvm/docs/DaemonMode.rst (+135)
- (added) llvm/include/llvm/Support/DaemonDriver.h (+35)
- (added) llvm/include/llvm/Support/ToolInterface.h (+99)
- (modified) llvm/lib/Support/CMakeLists.txt (+2)
- (added) llvm/lib/Support/DaemonDriver.cpp (+530)
- (added) llvm/lib/Support/ToolInterface.cpp (+68)
- (modified) llvm/test/CMakeLists.txt (+1)
- (added) llvm/test/Support/DaemonDriver/Inputs/change-directory.txt (+32)
- (added) llvm/test/Support/DaemonDriver/Inputs/file-input-1.txt (+1)
- (added) llvm/test/Support/DaemonDriver/Inputs/file-input-2.txt (+1)
- (added) llvm/test/Support/DaemonDriver/Inputs/file-input-3.txt (+1)
- (added) llvm/test/Support/DaemonDriver/Inputs/file-input.txt (+18)
- (added) llvm/test/Support/DaemonDriver/Inputs/redirect-stderr.txt (+14)
- (added) llvm/test/Support/DaemonDriver/Inputs/run-twice.txt (+7)
- (added) llvm/test/Support/DaemonDriver/Inputs/str-input.txt (+21)
- (added) llvm/test/Support/DaemonDriver/Inputs/tool-exits-process.txt (+5)
- (added) llvm/test/Support/DaemonDriver/change-directory.test (+23)
- (added) llvm/test/Support/DaemonDriver/error-status-pipe-handle-unix.test (+5)
- (added) llvm/test/Support/DaemonDriver/error-status-pipe-invalid-file.test (+4)
- (added) llvm/test/Support/DaemonDriver/file-input.test (+22)
- (added) llvm/test/Support/DaemonDriver/redirect-stderr.test (+18)
- (added) llvm/test/Support/DaemonDriver/run-twice.test (+7)
- (added) llvm/test/Support/DaemonDriver/str-input.test (+23)
- (added) llvm/test/Support/DaemonDriver/tool-exits-process.test (+5)
- (added) llvm/test/Support/DaemonDriver/verify-example-daemon.test (+9)
- (added) llvm/utils/ExampleDaemon/CMakeLists.txt (+6)
- (added) llvm/utils/ExampleDaemon/ExampleDaemon.cpp (+104)
- (added) llvm/utils/ExampleDaemon/README (+6)
``````````diff
diff --git a/llvm/CMakeLists.txt b/llvm/CMakeLists.txt
index 6001928f92e37..b4a46194ef903 100644
--- a/llvm/CMakeLists.txt
+++ b/llvm/CMakeLists.txt
@@ -1380,6 +1380,7 @@ add_subdirectory(include)
add_subdirectory(lib)
if( LLVM_INCLUDE_UTILS )
+ add_subdirectory(utils/ExampleDaemon)
add_subdirectory(utils/FileCheck)
add_subdirectory(utils/PerfectShuffle)
add_subdirectory(utils/count)
diff --git a/llvm/docs/DaemonMode.rst b/llvm/docs/DaemonMode.rst
new file mode 100644
index 0000000000000..625ff35ce1326
--- /dev/null
+++ b/llvm/docs/DaemonMode.rst
@@ -0,0 +1,135 @@
+============================
+Running Tools in Daemon Mode
+============================
+
+.. contents::
+ :local:
+
+Introduction
+============
+
+Process creation incurs a significant overhead, especially on Windows. This
+document describes the "daemon mode" execution model for LLVM tools, where
+tools run as persistent daemon processes that can perform many tasks, with
+different arguments and input, in the same process. The original motivation
+for this work is to reduce Lit testing time by replacing tool invocations with
+commands for the daemon - this means that the daemonized tool must be able to
+produce the exact same output as the regular version of the tool.
+
+Adding Daemon Support to a Tool
+===============================
+
+Tools can support daemon mode by implementing the ``LLVMTool`` interface and
+calling ``runWithDaemonSupport(Tool)`` in the main function.
+
+First, the tool's ``main`` function must be refactored into a class implementing
+the ``LLVMTool`` interface. This has two functions, ``run`` and ``resetState``.
+The majority of the body of main (everything aside from ``InitLLVM`` and other
+one-time initialization) must be moved to ``run``. This is the function that is
+called by the daemon for each invocation, and it is also invoked once when the
+tool is run normally. The handling of standard input must also be reworked so that
+the ``StandardInputSource`` provided to ``run`` is respected as standard input.
+For example, ``MemoryBuffer::getSTDIN`` should be replaced by
+``StandardInputSource::getInput()`` and ``MemoryBuffer::getFileOrStdin`` should
+be replaced by ``StandardInputSource::getFileOrInput()``. This is how the daemon
+provides input to the tool, as the input cannot be read from stdin normally
+because the pipe never closes.
+
+``resetState`` must also be implemented, which is responsible for resetting any
+application state, for example command line options, statistics and debug
+counters, for the next invocation. This is not called when the process is run
+normally. Any persistent state which may affect the output and is not reset by
+``resetState`` will cause flaky tests.
+
+Finally, the contents of main that were refactored into ``run`` can be replaced
+by ``runWithDaemonSupport(Tool)``, which will detect the daemon argument and
+either run the tool in daemon mode or run it normally by deferring to ``run``.
+
+Daemon IPC Protocol
+===================
+
+The communication protocol between the daemon and the Lit tester uses four
+pipes: the daemon's ``stdin``, ``stdout`` and ``stderr`` and another pipe,
+called the `status pipe`, for the daemon to communicate responses to Lit.
+
+When the daemon first starts it will send a ``ready`` response to indicate that
+it has initialized correctly and is ready to receive commands.
+
+The daemon accepts `commands` on ``stdin``. Commands must be separated by a
+newline (\n; \r is ignored). The following commands are accepted:
+
+* ``run``: This command takes no arguments. Upon receiving this command, the
+ daemon will run the tool, whose output will be sent on ``stdout`` and ``stderr``
+ as usual (unless ``stderr`` is redirected). When the tool returns, a
+ ``returned`` response indicating the exit code is sent.
+
+* ``arg``: This command is followed by an integer indicating a number of
+ bytes. The daemon will read this many bytes from ``stdin``; this string will
+ be appended to the list of arguments for the next invocation. Arguments are
+ framed in this way to avoid having to worry about escaping a separator char.
+
+* ``input_string``: This command is followed by an integer indicating a number of
+ bytes. The daemon will read this many bytes from ``stdin``; this string will
+ be used as standard input for the next invocation.
+
+* ``input_file``: This command is followed by a file name. The content of this
+ file will be used as standard input for the next invocation.
+
+* ``cd``: This command is followed by a directory name. The current directory
+ for the daemon is changed to the provided directory until the next invocation
+ finishes. This controls the working directory for the tool, and following
+ ``input_file`` and ``cd`` commands until the next ``run``. start from this
+ directory too.
+
+* ``redirect_stderr_to_stdout``: This causes writes to ``stderr`` to be
+ directed through ``stdout`` for the next invocation.
+
+* ``exit``: The daemon will exit.
+
+The daemon may send the following `responses` along the status
+pipe:
+
+* ``ready``: This is sent when the daemon finishes initialization, to
+ indicate it is ready to receive commands.
+
+* ``error``: This is sent when the daemon encounters an error, for example a
+ badly formed command. The daemon will exit after encountering an error. The
+ managing process should re-raise the error, as this indicates incorrect use of
+ the daemon. The response is followed by a string describing the error.
+
+* ``returned``. This is sent when a task finishes executing. The response is
+ followed by an integer representing the exit code from the task.
+
+If the daemon exits unexpectedly while running the tool, this means that the
+tool itself caused the process to exit. The exit code from the daemon should be
+taken as the exit code for the task, and the daemon should be restarted.
+
+Each command and response has a space before its argument. An example exchange
+may look like:
+
+* Command: ``arg 3`` followed by ``opt``
+
+* Command: ``arg 2`` followed by ``-S``
+
+* Command: ``arg 20`` followed by ``--passes=instcombine``
+
+* Command: ``input_file llvm/test/Transforms/InstCombine/range-check.ll``
+
+* Command: ``run``
+
+* (``stdout`` and ``stderr`` are sent as usual.)
+
+* Response: ``returned 0``
+
+* Command: ``bad command``
+
+* Response: ``error Unexpected command: bad command``
+
+Running a Daemon
+================
+
+Tools are invoked in daemon mode by passing ``--daemon`` as the first command
+line argument. Additionally, ``--daemon-status-pipe`` argument must be provided
+to set the file descriptor or Windows file handle on which the status messages
+are sent. This may take the form ``fd:{N}`` indicating a Unix/CRT file descriptor
+or ``handle:{N}`` indicating a Windows file handle.
diff --git a/llvm/include/llvm/Support/DaemonDriver.h b/llvm/include/llvm/Support/DaemonDriver.h
new file mode 100644
index 0000000000000..40fc24f4e19df
--- /dev/null
+++ b/llvm/include/llvm/Support/DaemonDriver.h
@@ -0,0 +1,35 @@
+//===- llvm/Support/DaemonDriver.h - Daemon driver interface ----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file provides the function `runWithDaemonSupport` to run a tool
+// which implements `LLVMTool` as a daemon, as described in
+// docs/DaemonDriver.rst.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_DAEMONDRIVER_H
+#define LLVM_SUPPORT_DAEMONDRIVER_H
+
+#include "llvm/Support/Compiler.h"
+#include "llvm/Support/ToolInterface.h"
+
+namespace llvm {
+/// This function checks for the `--daemon` command line option and, if it is
+/// present, runs the given tool in daemon mode, as described in
+/// `docs/DaemonMode.rst`. Otherwise, the tool is run
+/// with standard input as the input source - a normal invocation. One-time
+/// initialization logic, for example constructing `InitLLVM` and initializing
+/// passes, should be performed before this function is called. Per-invocation
+/// initialization logic, for example parsing command line options or setting up
+/// the pass pipeline, should be performed inside of the tool's `run` function.
+/// The invoke function is responsible for resetting all global state that may
+/// affect its output on the next invocation.
+LLVM_ABI int runWithDaemonSupport(LLVMTool &Tool, int Argc, char **Argv);
+} // namespace llvm
+
+#endif // LLVM_SUPPORT_DAEMONDRIVER_H
diff --git a/llvm/include/llvm/Support/ToolInterface.h b/llvm/include/llvm/Support/ToolInterface.h
new file mode 100644
index 0000000000000..eead2e4a3d6e5
--- /dev/null
+++ b/llvm/include/llvm/Support/ToolInterface.h
@@ -0,0 +1,99 @@
+//===- llvm/Support/ToolInterface.h - Abstract tool interface ---*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Abstract class for LLVM tools that can be re-invoked in the same process.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_TOOLINTERFACE_H
+#define LLVM_SUPPORT_TOOLINTERFACE_H
+
+#include "llvm/Support/ErrorOr.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include <memory>
+
+namespace llvm {
+
+/// Many tools operate on standard input, which is problematic for in-process
+/// execution/daemonization as the process may not want to close its standard
+/// input stream. This class allows the tools to function as if a different
+/// source, either an in-memory string or a file on disk, were standard input.
+/// Tool implementations should read from this class instead of standard input,
+/// for example using `StandardInputSource::getInput` instead of
+/// `MemoryBuffer::getSTDIN` and `StandardInputSource::getFileOrInput` instead
+/// of `MemoryBuffer::getFileOrSTDIN`.
+class LLVM_ABI StandardInputSource final {
+public:
+ /// Construct a `StandardInputSource` that draws input from the process's
+ /// standard input stream. This is used for regular tool invocations.
+ static StandardInputSource fromStdin() {
+ return StandardInputSource(Kind::Stdin, std::string());
+ }
+
+ /// Construct a `StandardInputSource` that draws input from a file.
+ static StandardInputSource fromFile(std::string &&Filename) {
+ return StandardInputSource(Kind::File, std::move(Filename));
+ }
+
+ /// Construct a `StandardInputSource` that pretends that the given string
+ /// is the contents of standard input.
+ static StandardInputSource fromString(std::string &&StringValue) {
+ return StandardInputSource(Kind::String, std::move(StringValue));
+ }
+
+ /// Replacement for `MemoryBuffer::getSTDIN`. The returned memory buffer
+ /// always has the identifier "<stdin>".
+ ErrorOr<std::unique_ptr<MemoryBuffer>> getInput() const;
+
+ /// Replacement for `MemoryBuffer::getFileOrInput`. Returns `getInput()` if
+ /// the file name is "-", otherwise returns a memory buffer of the file.
+ ErrorOr<std::unique_ptr<MemoryBuffer>> getFileOrInput(StringRef Filename,
+ bool IsText) const {
+ if (Filename == "-")
+ return getInput();
+
+ return MemoryBuffer::getFile(Filename, IsText);
+ }
+
+private:
+ enum class Kind {
+ Stdin,
+ File,
+ String,
+ };
+
+ StandardInputSource(Kind SourceKind, std::string &&StringValue)
+ : SourceKind(SourceKind), StringValue(StringValue) {}
+
+ Kind SourceKind;
+ std::string StringValue;
+};
+
+/// This class represents a shared interface for LLVM tools that can be
+/// re-invoked in the same process, for example when run by the daemon driver.
+class LLVM_ABI LLVMTool {
+public:
+ virtual ~LLVMTool() = default;
+
+ /// This is the function called to run the tool with a given input and set of
+ /// arguments. All code to run the tool except for one-time initialization and
+ /// finalization (for example `InitLLVM`) should be done in this function.
+ ///
+ /// Note that as `run` may use global state, it is not thread-safe.
+ virtual int run(int Argc, char **Argv,
+ const StandardInputSource &InputSource) = 0;
+
+ /// This function is called between `run` invocations to reset any persistent
+ /// state, for example static command line options, statistics and debug
+ /// counters, for the next invocation. This function is not called when the
+ /// tool is run normally.
+ virtual void resetState() = 0;
+};
+} // namespace llvm
+
+#endif // LLVM_SUPPORT_TOOLINTERFACE_H
diff --git a/llvm/lib/Support/CMakeLists.txt b/llvm/lib/Support/CMakeLists.txt
index e8d505f218b69..2808801c566d6 100644
--- a/llvm/lib/Support/CMakeLists.txt
+++ b/llvm/lib/Support/CMakeLists.txt
@@ -178,6 +178,7 @@ add_llvm_component_library(LLVMSupport
CrashRecoveryContext.cpp
CSKYAttributes.cpp
CSKYAttributeParser.cpp
+ DaemonDriver.cpp
DataExtractor.cpp
Debug.cpp
DebugCounter.cpp
@@ -269,6 +270,7 @@ add_llvm_component_library(LLVMSupport
ThreadPool.cpp
TimeProfiler.cpp
Timer.cpp
+ ToolInterface.cpp
ToolOutputFile.cpp
TrieRawHashMap.cpp
Twine.cpp
diff --git a/llvm/lib/Support/DaemonDriver.cpp b/llvm/lib/Support/DaemonDriver.cpp
new file mode 100644
index 0000000000000..199a1b0daacb7
--- /dev/null
+++ b/llvm/lib/Support/DaemonDriver.cpp
@@ -0,0 +1,530 @@
+//===- llvm/Support/DaemonDriver.cpp - Daemon driver interface ---- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the interface for tools to be run in "daemon mode",
+// following the IPC protocol as described in docs/DaemonMode.rst.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Support/DaemonDriver.h"
+
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Support/Program.h"
+#include "llvm/Support/raw_ostream.h"
+#include <cerrno>
+#include <cstdio>
+#include <cstdlib>
+#include <filesystem>
+#include <optional>
+#include <system_error>
+
+#if defined _WIN32
+#include <io.h>
+#else
+#include <unistd.h>
+#endif
+
+using namespace llvm;
+
+// Windows emulated POSIX functions are prefixed by `_`.
+#ifdef _WIN32
+#define CLOSE_FN ::_close
+#define DUP_FN ::_dup
+#define DUP2_FN ::_dup2
+#else
+#define CLOSE_FN ::close
+#define DUP_FN ::dup
+#define DUP2_FN ::dup2
+#endif
+
+// Standard stream fileno macros are not defined on Windows.
+#ifndef STDIN_FILENO
+#define STDIN_FILENO 0
+#endif
+#ifndef STDOUT_FILENO
+#define STDOUT_FILENO 1
+#endif
+#ifndef STDERR_FILENO
+#define STDERR_FILENO 2
+#endif
+
+namespace {
+namespace util {
+/// RAII mechanism to redirect a file descriptor to the file pointed to by a
+/// different file descriptor. The destructor will reset the file descriptor to
+/// its original file.
+class ScopedFileRedirect {
+public:
+ /// Redirect `FromFd` to the same file as `ToFd` for the lifetime of this
+ /// object.
+ ScopedFileRedirect(int FromFd, int ToFd) : FromFd(FromFd) {
+ // Store a copy of the original file so that we can restore the original fd
+ // to this copy.
+ CopyFd = DUP_FN(FromFd);
+
+ // Close the source fd and reopen it to the target fd.
+ DUP2_FN(ToFd, FromFd);
+ }
+
+ ~ScopedFileRedirect() {
+ if (Moved)
+ return;
+
+ // Close the source fd and reopen it to the original file.
+ DUP2_FN(CopyFd, FromFd);
+
+ // Close the copied file descriptor, as it's no longer needed.
+ CLOSE_FN(CopyFd);
+ }
+
+ ScopedFileRedirect(ScopedFileRedirect &&Other) {
+ *this = Other;
+ Other.Moved = true;
+ }
+ ScopedFileRedirect &operator=(ScopedFileRedirect &&Other) {
+ *this = Other;
+ Other.Moved = true;
+ return *this;
+ }
+
+private:
+ ScopedFileRedirect(const ScopedFileRedirect &Other) = default;
+ ScopedFileRedirect &operator=(const ScopedFileRedirect &Other) = default;
+
+ int FromFd;
+ int CopyFd;
+ bool Moved = false;
+};
+
+static ErrorOr<std::string> readNextLine(FILE *File) {
+ std::string Result;
+ raw_string_ostream ResultOS(Result);
+
+ constexpr size_t BufSize = 512;
+ char Buf[BufSize];
+
+ // Read from the file, appending to the result, until a new line character
+ // is found.
+ while (std::fgets(Buf, BufSize, File)) {
+ ResultOS << Buf;
+
+ if (std::strchr(Buf, '\n'))
+ break;
+ }
+
+ if (std::ferror(File))
+ return std::make_error_code(static_cast<std::errc>(errno));
+
+ return Result;
+}
+
+} // namespace util
+
+/// Status code returned if the daemon fails to initialize, for example due to
+/// incorrect command line arguments.
+constexpr int StatusInitError = 2;
+/// Status code returned if the daemon receives a malformed command.
+constexpr int StatusCommandError = 3;
+
+/// This should only be used before the status pipe is set up - after,
+/// errors are reported to the user via the status pipe.
+[[noreturn]] static void reportInitError(const Twine &Err) {
+ errs() << "[daemon] Error: " << Err << "\n";
+ std::exit(StatusInitError);
+}
+
+/// Returns true if ``--daemon`` is passed as the first command line argument.
+static bool detectDaemonArg(int Argc, char **Argv) {
+ // `--daemon` must be the first argument.
+ return Argc >= 2 && Argv[1] == StringRef("--daemon");
+}
+
+struct DaemonCommandLineOptions {
+ bool DaemonModeEnabled;
+ std::string StatusPipe;
+};
+
+// Creates the command line options for configuring the daemon. The returned
+// reference points to a static struct where the option values will be stored.
+static const DaemonCommandLineOptions &initializeDaemonCommandLineOptions() {
+ // This is declared as a static local variables in this function rather than
+ // a static global variable as is common in LLVM to avoid adding global
+ // constructors and destructors to the program, which is forbidden in the
+ // Support library.
+ static DaemonCommandLineOptions Options;
+
+ static cl::opt<bool, true> DaemonModeEnabledOpt(
+ "daemon", cl::location(Options.DaemonModeEnabled), cl::init(false));
+
+ static cl::opt<std::string, true> StatusPipeOpt(
+ "daemon-status-pipe",
+ cl::desc("File to which the daemon tool will send status messages. May "
+ "be 'path:{filepath}', 'fd:{file descriptor}' or "
+ "'handle:{Windows file handle}'"),
+ cl::location(Options.StatusPipe), cl::init(""));
+
+ return Options;
+}
+
+/// Command sent to invoke the tool.
+struct RunCommand {
+ static constexpr StringRef Prefix = "run";
+
+ static Expected<RunCommand> parse(StringRef Remaining) {
+ Remaining = Remaining.trim();
+
+ if (!Remaining.trim().empty())
+ return createStringError("Unexpected trailing characters in command");
+
+ return RunCommand();
+ }
+};
+
+/// Command providing a command line argument for the tool as a framed string.
+struct ArgCommand {
+ static constexpr StringRef Prefix = "arg";
+
+ static Expected<ArgCommand> parse(StringRef Remaining) {
+ Remaining = Remaining.trim();
+
+ size_t ExpectedLength;
+ if (Remaining.consumeInteger(10, ExpectedLength))
+ return createStringError("Expected integer");
+
+ if (!Remaining.empty())
+ return createStringError("Unexpected trailing characters in command");
+
+ return ArgCommand{ExpectedLength};
+ }
+
+ size_t ExpectedLength;
+};
+
+// Command providing standard input for the tool as a framed string.
+struct InputStringCommand {
+ static constexpr StringRef Prefix = "input_string";
+
+ static Expected<InputStringCommand> parse(StringRef Remaining) {
+ Remaining = Remaining.trim();
+
+ size_t ExpectedLength;
+ if (Remaining.consumeInteger(10, ExpectedLength))
+ return createStringError("Expected integer");
+
+ if (!Remaining.trim().empty())
+ return createStringError("Unexpected trailing chara...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/193706
More information about the llvm-commits
mailing list