[libc-commits] [PATCH] D150516: [libc] Cache ownership of the shared buffer in the port

[Joseph Huber via Phabricator via libc-commits]

jhuber6 created this revision.
jhuber6 added reviewers: jdoerfert, JonChesterfield, tianshilei1992, lntue, michaelrj, sivachandra.
Herald added subscribers: libc-commits, ecnelises, tschuett, yaxunl.
Herald added projects: libc-project, All.
jhuber6 requested review of this revision.

This patch adds another variable to cache cases where we know that we
own the buffer. This allows us to skip the atomic load on the inbox
because we already know its state. This is legal immediately after
opening a port, or when sending immediately after a recieve. This
caching nets a significant (~17%) speedup for the basic open, send,
recieve combination.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D150516

Files:
  libc/src/__support/RPC/rpc.h


Index: libc/src/__support/RPC/rpc.h
===================================================================
--- libc/src/__support/RPC/rpc.h
+++ libc/src/__support/RPC/rpc.h
@@ -83,7 +83,7 @@
 ///
 /// No process writes to its inbox. Each toggles the bit in the outbox to pass
 /// ownership to the other process.
-/// When inbox == outbox, the current state machine owns the buffer.
+/// When inbox == outbox, the current state machine owns_buffer the buffer.
 /// Initially the client is able to open any port as it will load 0 from both.
 /// The server inbox read is inverted, so it loads inbox==1, outbox==0 until
 /// the client has written to its outbox.
@@ -153,7 +153,7 @@
   /// Inverting the bits loaded from the inbox in exactly one of the pair of
   /// processes means that each can use the same state transitions.
   /// Whichever process has InvertInbox==false is the initial owner.
-  /// Inbox equal Outbox => current process owns the buffer
+  /// Inbox equal Outbox => current process owns_buffer the buffer
   /// Inbox difer Outbox => current process does not own the buffer
   /// At startup, memory is zero initialised and raw loads of either mailbox
   /// would return zero. Thus both would succeed in opening a port and data
@@ -300,7 +300,7 @@
   LIBC_INLINE Port(Process<T> &process, uint64_t lane_mask, uint64_t index,
                    uint32_t out)
       : process(process), lane_mask(lane_mask), index(index), out(out),
-        receive(false) {}
+        receive(false), owns_buffer(true) {}
   LIBC_INLINE ~Port() = default;
 
 private:
@@ -329,9 +329,9 @@
   }
 
   LIBC_INLINE void close() {
-    // If the server last did a receive it needs to exchange ownership before
-    // closing the port.
-    if (receive && T)
+    // The server is passive, if it owns_buffer the buffer when it closes we
+    // need to give ownership back to the client.
+    if (owns_buffer && T)
       out = process.invert_outbox(index, out);
     process.unlock(lane_mask, index);
   }
@@ -342,6 +342,7 @@
   uint64_t index;
   uint32_t out;
   bool receive;
+  bool owns_buffer;
 };
 
 /// The RPC client used to make requests to the server.
@@ -370,7 +371,7 @@
 
 /// Applies \p fill to the shared buffer and initiates a send operation.
 template <bool T> template <typename F> LIBC_INLINE void Port<T>::send(F fill) {
-  uint32_t in = process.load_inbox(index);
+  uint32_t in = owns_buffer ? out ^ T : process.load_inbox(index);
 
   // We need to wait until we own the buffer before sending.
   while (Process<T>::buffer_unavailable(in, out)) {
@@ -382,6 +383,7 @@
   process.invoke_rpc(fill, process.get_packet(index));
   atomic_thread_fence(cpp::MemoryOrder::RELEASE);
   out = process.invert_outbox(index, out);
+  owns_buffer = false;
   receive = false;
 }
 
@@ -389,10 +391,12 @@
 template <bool T> template <typename U> LIBC_INLINE void Port<T>::recv(U use) {
   // We only exchange ownership of the buffer during a receive if we are waiting
   // for a previous receive to finish.
-  if (receive)
+  if (receive) {
     out = process.invert_outbox(index, out);
+    owns_buffer = false;
+  }
 
-  uint32_t in = process.load_inbox(index);
+  uint32_t in = owns_buffer ? out ^ T : process.load_inbox(index);
 
   // We need to wait until we own the buffer before receiving.
   while (Process<T>::buffer_unavailable(in, out)) {
@@ -404,6 +408,7 @@
   // Apply the \p use function to read the memory out of the buffer.
   process.invoke_rpc(use, process.get_packet(index));
   receive = true;
+  owns_buffer = true;
 }
 
 /// Combines a send and receive into a single function.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D150516.521959.patch
Type: text/x-patch
Size: 3606 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/libc-commits/attachments/20230514/80cb57ed/attachment-0001.bin>