[llvm] [ORC] Harmonize the SimpleRemoteEPC hangup protocol (PR #218586)
Lang Hames via llvm-commits
llvm-commits at lists.llvm.org
Mon Aug 24 22:55:21 PDT 2026
https://github.com/lhames created https://github.com/llvm/llvm-project/pull/218586
A Hangup message now always carries a serialized Error saying why the session is ending -- success for an orderly disconnect, otherwise the reason -- encoded and decoded through shared helpers so the two ends of the protocol cannot drift apart. SimpleRemoteEPC::disconnect sends one, and SimpleRemoteEPCServer decodes the payload it previously discarded.
Both ends now also report an error when the transport disconnects without either side having announced a hangup. Previously the executor exited successfully whether the controller hung up or vanished, so a crashed controller was indistinguishable from a clean run. The cause of such a disconnection isn't knowable, so the error states what was observed rather than attributing it.
This aligns SimpleRemoteEPC/SimpleRemoteEPCServer with upcoming ORC runtime patches that will introduce a SimpleRemoteEPCServer-compatible ControllerAccess implementation.
>From 5a85a68122947afb726ea5434f75405e91442554 Mon Sep 17 00:00:00 2001
From: Lang Hames <lhames at gmail.com>
Date: Fri, 21 Aug 2026 12:14:44 +1000
Subject: [PATCH] [ORC] Harmonize the SimpleRemoteEPC hangup protocol
A Hangup message now always carries a serialized Error saying why the
session is ending -- success for an orderly disconnect, otherwise the
reason -- encoded and decoded through shared helpers so the two ends of
the protocol cannot drift apart. SimpleRemoteEPC::disconnect sends one,
and SimpleRemoteEPCServer decodes the payload it previously discarded.
Both ends now also report an error when the transport disconnects
without either side having announced a hangup. Previously the executor
exited successfully whether the controller hung up or vanished, so a
crashed controller was indistinguishable from a clean run. The cause of
such a disconnection isn't knowable, so the error states what was
observed rather than attributing it.
This aligns SimpleRemoteEPC/SimpleRemoteEPCServer with upcoming ORC
runtime patches that will introduce a SimpleRemoteEPCServer-compatible
ControllerAccess implementation.
---
.../Orc/Shared/SimpleRemoteEPCUtils.h | 16 +++++
.../ExecutionEngine/Orc/SimpleRemoteEPC.h | 21 +++++++
.../Orc/TargetProcess/SimpleRemoteEPCServer.h | 6 ++
.../Orc/Shared/SimpleRemoteEPCUtils.cpp | 24 ++++++++
.../ExecutionEngine/Orc/SimpleRemoteEPC.cpp | 59 +++++++++++++++----
.../TargetProcess/SimpleRemoteEPCServer.cpp | 30 +++++++++-
6 files changed, 142 insertions(+), 14 deletions(-)
diff --git a/llvm/include/llvm/ExecutionEngine/Orc/Shared/SimpleRemoteEPCUtils.h b/llvm/include/llvm/ExecutionEngine/Orc/Shared/SimpleRemoteEPCUtils.h
index a81ffe975b876..83137eadf4461 100644
--- a/llvm/include/llvm/ExecutionEngine/Orc/Shared/SimpleRemoteEPCUtils.h
+++ b/llvm/include/llvm/ExecutionEngine/Orc/Shared/SimpleRemoteEPCUtils.h
@@ -51,6 +51,22 @@ struct SimpleRemoteEPCExecutorInfo {
StringMap<ExecutorAddr> BootstrapSymbols;
};
+/// Encode an Error as the payload of a Hangup message.
+///
+/// A Hangup always carries a serialized Error saying why the session is ending:
+/// a success value for an orderly disconnect, otherwise the reason. Both ends
+/// of the protocol encode and decode hangups through these two functions, so
+/// that their idea of the payload format cannot drift apart.
+LLVM_ABI shared::WrapperFunctionBuffer encodeHangupPayload(Error Err);
+
+/// Decode a Hangup payload produced by encodeHangupPayload.
+///
+/// Returns the encoded Error, or an Error describing the payload if it cannot
+/// be decoded -- including an empty payload, which is never valid. Both
+/// outcomes end the session with an error; they are distinguished only by the
+/// message.
+LLVM_ABI Error decodeHangupPayload(shared::WrapperFunctionBuffer Payload);
+
class LLVM_ABI SimpleRemoteEPCTransportClient {
public:
enum HandleMessageAction { ContinueSession, EndSession };
diff --git a/llvm/include/llvm/ExecutionEngine/Orc/SimpleRemoteEPC.h b/llvm/include/llvm/ExecutionEngine/Orc/SimpleRemoteEPC.h
index 7353e807bdc22..6e27cc2c65080 100644
--- a/llvm/include/llvm/ExecutionEngine/Orc/SimpleRemoteEPC.h
+++ b/llvm/include/llvm/ExecutionEngine/Orc/SimpleRemoteEPC.h
@@ -102,6 +102,27 @@ class LLVM_ABI SimpleRemoteEPC : public ExecutorProcessControl,
std::mutex SimpleRemoteEPCMutex;
std::condition_variable DisconnectCV;
bool Disconnected = false;
+
+ // Whether either side announced the end of the session. If the transport
+ // reports a disconnection and neither of these is set then the executor went
+ // away without saying so, which is reported as an error: see
+ // handleDisconnect.
+ //
+ // LocalHangup has to be shared state: disconnect() sets it on the calling
+ // thread, while handleDisconnect reads it on the transport's listener thread.
+ //
+ // RemoteHangup is shared state only because the read loop lives in the
+ // transport: the hangup is observed in handleMessage but needed in
+ // handleDisconnect, and the two are separate entry points on the
+ // SimpleRemoteEPCTransportClient interface with no call edge between them.
+ //
+ // TODO: Once the read loop is reshaped into a reactor (mirroring
+ // FDSimpleRemoteCA in the ORC runtime), RemoteHangup should fold into a stop
+ // reason returned from it, leaving only LocalHangup as state -- as
+ // FDSimpleRemoteCA does with ShutdownRequested.
+ bool LocalHangup = false;
+ bool RemoteHangup = false;
+
Error DisconnectErr = Error::success();
std::unique_ptr<SimpleRemoteEPCTransport> T;
diff --git a/llvm/include/llvm/ExecutionEngine/Orc/TargetProcess/SimpleRemoteEPCServer.h b/llvm/include/llvm/ExecutionEngine/Orc/TargetProcess/SimpleRemoteEPCServer.h
index 0aa040bc00138..74d3efd04bd49 100644
--- a/llvm/include/llvm/ExecutionEngine/Orc/TargetProcess/SimpleRemoteEPCServer.h
+++ b/llvm/include/llvm/ExecutionEngine/Orc/TargetProcess/SimpleRemoteEPCServer.h
@@ -180,6 +180,12 @@ class LLVM_ABI SimpleRemoteEPCServer : public SimpleRemoteEPCTransportClient {
std::mutex ServerStateMutex;
std::condition_variable ShutdownCV;
enum { ServerRunning, ServerShuttingDown, ServerShutDown } RunState;
+
+ // Whether the controller announced the end of the session. The server never
+ // initiates a disconnection, so a transport disconnection without this is the
+ // controller going away unexpectedly: see handleDisconnect.
+ bool RemoteHangup = false;
+
Error ShutdownErr = Error::success();
std::unique_ptr<SimpleRemoteEPCTransport> T;
std::unique_ptr<Dispatcher> D;
diff --git a/llvm/lib/ExecutionEngine/Orc/Shared/SimpleRemoteEPCUtils.cpp b/llvm/lib/ExecutionEngine/Orc/Shared/SimpleRemoteEPCUtils.cpp
index a8bf26d7da033..781e2fc8e23e8 100644
--- a/llvm/lib/ExecutionEngine/Orc/Shared/SimpleRemoteEPCUtils.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/Shared/SimpleRemoteEPCUtils.cpp
@@ -46,6 +46,30 @@ const char *DispatchFnName = "__llvm_orc_SimpleRemoteEPC_dispatch_fn";
} // end namespace SimpleRemoteEPCDefaultBootstrapSymbolNames
+shared::WrapperFunctionBuffer encodeHangupPayload(Error Err) {
+ using SPSSerialize = shared::SPSArgList<shared::SPSError>;
+ auto SE = shared::detail::toSPSSerializable(std::move(Err));
+ auto Payload =
+ shared::WrapperFunctionBuffer::allocate(SPSSerialize::size(SE));
+ shared::SPSOutputBuffer OB(Payload.data(), Payload.size());
+ bool Success = SPSSerialize::serialize(OB, SE);
+ (void)Success;
+ assert(Success && "Hangup payload serialization should not fail");
+ return Payload;
+}
+
+Error decodeHangupPayload(shared::WrapperFunctionBuffer Payload) {
+ assert(!Payload.getOutOfBandError() &&
+ "Hangup payload should not be an out-of-band error buffer");
+
+ shared::detail::SPSSerializableError Info;
+ shared::SPSInputBuffer IB(Payload.data(), Payload.size());
+ if (!shared::SPSArgList<shared::SPSError>::deserialize(IB, Info))
+ return make_error<StringError>("Could not deserialize hangup info",
+ inconvertibleErrorCode());
+ return shared::detail::fromSPSSerializable(std::move(Info));
+}
+
SimpleRemoteEPCTransportClient::~SimpleRemoteEPCTransportClient() = default;
SimpleRemoteEPCTransport::~SimpleRemoteEPCTransport() = default;
diff --git a/llvm/lib/ExecutionEngine/Orc/SimpleRemoteEPC.cpp b/llvm/lib/ExecutionEngine/Orc/SimpleRemoteEPC.cpp
index 0d4e889de202f..bf3d98f9d0312 100644
--- a/llvm/lib/ExecutionEngine/Orc/SimpleRemoteEPC.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/SimpleRemoteEPC.cpp
@@ -87,6 +87,28 @@ SimpleRemoteEPC::createDefaultMemoryAccess() {
}
Error SimpleRemoteEPC::disconnect() {
+ // disconnect is idempotent, so the first caller owns the hangup. There is
+ // also nothing to announce to an executor that has already announced its own
+ // departure.
+ bool SendHangup = false;
+ {
+ std::lock_guard<std::mutex> Lock(SimpleRemoteEPCMutex);
+ SendHangup = !LocalHangup && !RemoteHangup;
+ LocalHangup = true;
+ }
+
+ // Tell the executor we're going away, so that it can distinguish this from
+ // losing us unexpectedly. Best-effort: if the send fails there is nothing to
+ // do but tear down anyway, and the executor will report the disconnection as
+ // unexpected. A locally requested disconnect is orderly, so the hangup
+ // carries a success value.
+ if (SendHangup) {
+ auto Payload = encodeHangupPayload(Error::success());
+ if (auto Err = sendMessage(SimpleRemoteEPCOpcode::Hangup, 0, ExecutorAddr(),
+ {Payload.data(), Payload.size()}))
+ consumeError(std::move(Err));
+ }
+
T->disconnect();
D->shutdown();
std::unique_lock<std::mutex> Lock(SimpleRemoteEPCMutex);
@@ -137,6 +159,10 @@ SimpleRemoteEPC::handleMessage(SimpleRemoteEPCOpcode OpC, uint64_t SeqNo,
break;
case SimpleRemoteEPCOpcode::Hangup:
T->disconnect();
+ {
+ std::lock_guard<std::mutex> Lock(SimpleRemoteEPCMutex);
+ RemoteHangup = true;
+ }
if (auto Err = handleHangup(std::move(ArgBytes)))
return std::move(Err);
return EndSession;
@@ -169,7 +195,26 @@ void SimpleRemoteEPC::handleDisconnect(Error Err) {
shared::WrapperFunctionBuffer::createOutOfBandError("disconnecting"));
std::lock_guard<std::mutex> Lock(SimpleRemoteEPCMutex);
- DisconnectErr = joinErrors(std::move(DisconnectErr), std::move(Err));
+
+ // If the transport reported no error, but neither side announced the end of
+ // the session, then the executor went away without telling us. The cause is
+ // not knowable from here -- it may have crashed, been killed, or become
+ // unreachable -- so report what was observed rather than a cause.
+ //
+ // A missing hangup is evidence, not proof: a hangup can also be lost in
+ // transit, since closing a TCP socket with unread data queued sends an RST,
+ // which can discard bytes the peer had already delivered. We accept that
+ // rather than draining the read side before closing -- the cost is a
+ // misleading diagnostic on a session that is ending regardless, whereas a
+ // drain risks stalling teardown on a peer that never closes.
+ Error DisconnectReason =
+ (!Err && !LocalHangup && !RemoteHangup)
+ ? make_error<StringError>("Connection closed without hangup",
+ inconvertibleErrorCode())
+ : std::move(Err);
+
+ DisconnectErr =
+ joinErrors(std::move(DisconnectErr), std::move(DisconnectReason));
Disconnected = true;
DisconnectCV.notify_all();
}
@@ -346,17 +391,7 @@ void SimpleRemoteEPC::handleCallWrapper(
}
Error SimpleRemoteEPC::handleHangup(shared::WrapperFunctionBuffer ArgBytes) {
- using namespace llvm::orc::shared;
- auto WFR = WrapperFunctionBuffer::copyFrom(ArgBytes.data(), ArgBytes.size());
- if (const char *ErrMsg = WFR.getOutOfBandError())
- return make_error<StringError>(ErrMsg, inconvertibleErrorCode());
-
- orc::shared::detail::SPSSerializableError Info;
- SPSInputBuffer IB(WFR.data(), WFR.size());
- if (!SPSArgList<SPSError>::deserialize(IB, Info))
- return make_error<StringError>("Could not deserialize hangup info",
- inconvertibleErrorCode());
- return fromSPSSerializable(std::move(Info));
+ return decodeHangupPayload(std::move(ArgBytes));
}
} // end namespace orc
diff --git a/llvm/lib/ExecutionEngine/Orc/TargetProcess/SimpleRemoteEPCServer.cpp b/llvm/lib/ExecutionEngine/Orc/TargetProcess/SimpleRemoteEPCServer.cpp
index 5aa61526be848..99147192f3490 100644
--- a/llvm/lib/ExecutionEngine/Orc/TargetProcess/SimpleRemoteEPCServer.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/TargetProcess/SimpleRemoteEPCServer.cpp
@@ -100,8 +100,15 @@ SimpleRemoteEPCServer::handleMessage(SimpleRemoteEPCOpcode OpC, uint64_t SeqNo,
case SimpleRemoteEPCOpcode::Setup:
return make_error<StringError>("Unexpected Setup opcode",
inconvertibleErrorCode());
- case SimpleRemoteEPCOpcode::Hangup:
+ case SimpleRemoteEPCOpcode::Hangup: {
+ {
+ std::lock_guard<std::mutex> Lock(ServerStateMutex);
+ RemoteHangup = true;
+ }
+ if (auto Err = decodeHangupPayload(std::move(ArgBytes)))
+ return std::move(Err);
return SimpleRemoteEPCTransportClient::EndSession;
+ }
case SimpleRemoteEPCOpcode::Result:
if (auto Err = handleResult(SeqNo, TagAddr, std::move(ArgBytes)))
return std::move(Err);
@@ -144,7 +151,26 @@ void SimpleRemoteEPCServer::handleDisconnect(Error Err) {
}
std::lock_guard<std::mutex> Lock(ServerStateMutex);
- ShutdownErr = joinErrors(std::move(ShutdownErr), std::move(Err));
+
+ // The server never initiates a disconnection, so if the transport reported no
+ // error and no hangup arrived then the controller went away without telling
+ // us. The cause is not knowable from here -- it may have crashed, been
+ // killed, or become unreachable -- so report what was observed rather than a
+ // cause.
+ //
+ // A missing hangup is evidence, not proof: a hangup can also be lost in
+ // transit, since closing a TCP socket with unread data queued sends an RST,
+ // which can discard bytes the peer had already delivered. We accept that
+ // rather than draining the read side before closing -- the cost is a
+ // misleading diagnostic on a session that is ending regardless, whereas a
+ // drain risks stalling teardown on a peer that never closes.
+ Error DisconnectReason =
+ (!Err && !RemoteHangup)
+ ? make_error<StringError>("Connection closed without hangup",
+ inconvertibleErrorCode())
+ : std::move(Err);
+
+ ShutdownErr = joinErrors(std::move(ShutdownErr), std::move(DisconnectReason));
RunState = ServerShutDown;
ShutdownCV.notify_all();
}
More information about the llvm-commits
mailing list