[libcxx-commits] [libcxx] [libc++] Implement P2545R4 Read-Copy Update (RCU) (PR #175451)

Fri Aug 21 11:10:48 PDT 2026

================
@@ -0,0 +1,426 @@
+====================
+``<rcu>`` Design
+====================
+
+.. contents::
+   :local:
+   :depth: 2
+
+
+Introduction
+============
+
+This is the C++ paper `Read-Copy Update (RCU) <https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2545r4.pdf>`__.
+There are three designs discussed in the paper `Supplementary Material for User-Level Implementations of
+Read-Copy Update <http://www.rdrop.com/users/paulmck/RCU/urcu-supp-accepted.2011.08.30a.pdf>`__
+
+- Quiescent-State-Based Reclamation RCU
+- General-Purpose RCU
+- Low-Overhead RCU Via Signal Handling
+
+libc++ adopted a variation that closely resembles the "General-Purpose RCU" design in the above paper.
+
+Synopsis
+========
+::
+
+  class rcu_domain {
+  public:
+    void lock() noexcept;
+    bool try_lock() noexcept;
+    void unlock() noexcept;
+  };
+
+  rcu_domain& rcu_default_domain() noexcept;
+
+  void rcu_synchronize(rcu_domain& dom = rcu_default_domain()) noexcept;
+  void rcu_barrier(rcu_domain& dom = rcu_default_domain()) noexcept;
+
+  template<class T, class D = default_delete<T>>
+  void rcu_retire(T* p, D d = D(), rcu_domain& dom = rcu_default_domain());
+
+  template<class T, class D = default_delete<T>>
+  class rcu_obj_base {
+  public:
+    void retire(D d = D(), rcu_domain& dom = rcu_default_domain()) noexcept;
+  };
+
+Background Information
+======================
+
+`rcu` is an alternative to read-write locks and it is suitable for read-mostly data structures.
+Here is an example usage ::
+
+  struct Data { /* members */ };
+  std::atomic<Data*> data_;
+
+  // ==============================================
+  template <typename Func>
+  Result reader_op(Func fn) {
+    std::scoped_lock l(std::rcu_default_domain());
+    Data* p = data_;
+    // fn should not block too long or call
+    // rcu_synchronize(), rcu_barrier(), or
+    // rcu_retire(), directly or indirectly
+    return fn(p);
+ }
+
+  // ==============================================
+  // May be called concurrently with reader_op
+  void update(Data* newdata) {
+    Data* olddata = data_.exchange(newdata);
+    std::rcu_retire(olddata); // reclaim *olddata when safe
+  }
+
+There are several key properties that an implementation of `rcu` must satisfy:
+
+- On the reader side, `rcu_domain::lock` and `rcu_domain::unlock` must not block the thread while the writer thread
+  is performing updates, or the collector thread is reclaiming the objects.
+
+- On the writer side, `retire`-ing an object in principle should not block at all. However, it is technically conforming to block (e.g. evaluate the `deleter`s directly inside `rcu_retire`).
+
+- On the collector side, `rcu_synchronize` should block until at least all the existing readers exit their critical sections
+  via `rcu_domain::unlock` . Note that this is a key difference between `rcu` and read-write locks:
+  In case a late reader enters the critical section after the collector thread has called `rcu_synchronize` and started waiting, the
+  collector thread does not need to wait for the late reader to exit the critical section. In contrast, a collector thread trying to acquire a read-write lock would have to wait until the late reader releases the lock before it can acquire it.
+
+- On the collector side, `rcu_barrier` should block until all the objects `retired` before the `rcu_barrier` call have been reclaimed.
+
+- The threads that are using `rcu` must be known by the `rcu` implementation states.
+
+
+Adopted Design
+==============
+
+The core idea of `rcu` can be described by this image from lwn.net
+
+.. image:: https://static.lwn.net/images/ns/kernel/rcu/GracePeriodGood.png
+
+- Each row is a thread. The last row is the collector thread and the rows above are the reader threads.
+- Each "Reader" block represents a critical section, which starts with `rcu_domain::lock` and ends with `rcu_domain::unlock` .
+- When `rcu_retire` is called from the writer thread, it starts the "Removal" block.
+- When `rcu_synchronize` is called from the collector thread, it starts the "Grace Period" block. We need to wait until all the 
+  "Reader" blocks that started before the "Grace Period" started, to exit via `rcu_domain::unlock`, then we can end the "Grace Period".
+  Note that the "Grace Period" ends after the 4th row's "Reader" block ends. Also note that the "Grace Period" does not need to wait
+  for the late "Reader" blocks.
+- After the "Grace Period" ends, we can start the "Reclaim" block to reclaim the retired objects.
+
+Libc++ adopted a design that closely resembles the "General-Purpose RCU" design in the above paper.
+
+Some key details of this design are:
+
+- There is a global state which has two phases and it flips between the two phases.
+- Each thread stores its state: whether there is a reader in the critical section, and which phase it was when it entered the critical section.
+- When `rcu_synchronize` is called, it
+
+  - flips the global state to the next phase
+  - Going through a grace period: waits until all the threads that are in the critical section with the previous phase to exit the critical section.
+  - flips the global state back to the original phase
+  - Going through another grace period: waits until all the threads that are in the critical section with the next phase to exit the critical section.
+
+When `rcu_synchronize` returns, we can be sure that all the readers that were in the critical section before `rcu_synchronize` are now out of the critical section.
+The paper explains why we need to wait two phases instead of just one phase in detail. The key point is that, if we only wait for the readers in the previous phase to exit,
+there might be a late reader that enters the critical section after we flip the global state and before we wait for the previous phase's readers to exit.
+
+Implementation Terminology
+---------------------------
+
+- reader: A thread that calls `rcu_domain::lock` and `rcu_domain::unlock` to enter and exit the critical section.
+
+- quiescent state: When a thread is *not* in an RCU read-side critical section (between `lock` and `unlock`),
+  it is in a quiescent state.
+
+- grace period: Any time period during which every thread has been in at least one quiescent state is a grace period.
+  The grace period cannot end until all pre-existing readers have exited their critical sections. (see the image above)
+
+Implementation Details
+----------------------
+
+`thread_local_container`
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+A helper class that manages thread local objects. It provides APIs to get the `thread_local` object for the current thread, and to iterate through all the `thread_local` objects from all threads.
+It only contains `static` member variables and functions.
+
+It uses `mutex` to protect the list of all object pointers. Since the list is only modified when a thread first calls `get_current_thread_instance` to register itself, and when a thread exits, 
+the contention on the `mutex` should be low.
+
+TODO: We need to replace `mutex` as all non allocating APIs in `rcu` are designed to be `noexcept` and `mutex` can throw.
+
+`reader_states`
+~~~~~~~~~~~~~~~
+
+It defines the state of each reader thread. The state is essentially a pair of `(phase, lock_nested_level)` merged into a single `uint16_t` . 
+
+- `phase` is a single bit as the global state only has two phases. This `phase` is the phase of the global state when the current thread enters the critical section.
+- `lock_nested_level` is the nested level of the `rcu_domain::lock` . As you can call `rcu_domain::lock` multiple times in the same thread, we need to keep track of the nested level.
+  When `lock_nested_level` is `0`, it means the current thread is in a quiescent state (not in a critical section).
+
+The `reader_states` class is also a `static` helper class which only provides static definitions and functions, instead of the wrapper of the `uint16_t` state. This is because we need
+the `atomic` integer API, e.g. `fetch_or`. An `atomic` of a wrapper struct won't have these APIs.
+
+`rcu_singly_list_view`
+~~~~~~~~~~~~~~~~~~~~~~
+
+A helper class that provides a view of the intrusive singly linked list of `__rcu_node` s. It also stores the back pointer of the list to make the push back operation more efficient.
+It provides APIs to push a node to the back of the list, splice from another list and to iterate through the list.
+
+For the derived class of `rcu_obj_base`, the `retire` is `noexcept`, which means that no memory allocation can happen when pushing it to the retired callback queue.
+
+`rcu_domain_impl`
+~~~~~~~~~~~~~~~~~
+
+This is the main class that implements the `rcu` logic. It contains
+
+- `std::atomic<reader_states::state_type> global_reader_phase_` : the global state that flips between two phases. The readers will record the phase when they enter the critical section, and the collector will flip the global state when it calls `rcu_synchronize` to start a new grace period.
+
+- `std::mutex grace_period_mutex_` : If we have multiple collector threads calling `rcu_synchronize` concurrenly, we need to make sure only one of them is performing the phase flipping and deleter queue draining.
+  TODO: `mutex` can throw, we need to consider how to replace it.
+
+- `std::atomic<bool> grace_period_waiting_flag_` : This flag is used to sleep/wake up the collector thread that is waiting for the grace period to end.
+
+- `std::mutex retire_queue_mutex_` and  `rcu_singly_list_view __retired_callback_queue_` : This queue stores all the retired callbacks that are waiting for the grace period to end.
+
+  - TODO: `mutex` can throw, we need to consider how to replace it.
+
+  - Design 2: The access pattern of this list in this implementation is very special. The writer threads are only going to push to this queue. And the collector threads are only going to splice the entire queue. It is easy to implement it with a single `atomic<pair<Head*, Tail*>>`. It will be efficient if the platform supports 16 bytes CAS. To reduce contention between multiple writer threads retiring objects, we let each thread have a thread local queue
+
+- `rcu_singly_list_view callbacks_phase_1_` and `rcu_singly_list_view callbacks_phase_2_` : These two queues are used to let the retired callbacks go through two grace periods before being invoked. No additional synchronization is needed for these two queues as they are only processed when the collector (?) thread is holding the `grace_period_mutex_` .
+
+The domain has few operations:
+
+`lock`
+^^^^^^
+
+- If the current thread was in quiescent state, record the current global phase and set the nested level to 1. 
+- If the current thread was already in a critical section, just increment the nested level.
+
+`unlock`
+^^^^^^^^
+
+- Decrement the nested level. 
+- If the nested level becomes 0, the thread is now in a quiescent state, we can notify the waiting collector thread if there is any.
+
+`retire`
+^^^^^^^^
+
+- We need to push the retired callback to the `__retired_callback_queue_`
+
+`synchronize`
+^^^^^^^^^^^^^
+
+- We need to go through two phases of grace period
+- For each phase of the grace period, we need to flip the phase, and wait until all the reading threads are either
+
+  - in a quiescent state, or
+  - in the critical section with the new phase.
+
+- After each phase ends, we can move the callbacks to the next stage's queue, and we can evaluate the callbacks that have gone through two stages.
+
+
+Design Questions
+================
+
+What thread(s) should the deleters run on
----------------
ldionne wrote:

Other design discussions we had just now, just to summarize:
- With the current design, there are concerns around users never calling `rcu_barrier`, and hence never performing collection. We could potentially solve this by having `rcu_retire` collect at most one element, that way it is still bounded, however this presents a challenge as `rcu_retire` is expected to be extremely fast and non-blocking (in my opinion, based on what I expect typical RCU usage to be). If we decided to do that, we discussed problems about contending on the global "collector queues" lock, and about contention on "popping back from the global stage2 queue" if we were using a lockfree queue instead. We discussed the possibility of moving the stage 2 queues to be thread local, which would solve the problem of contention. However, that still does not solve the problem that the elements have to migrate from stage to stage, and `rcu_barrier` has to be called for that to happen.
- If we create a libc++ owned collector thread that performs stage migration and potentially stage 2 collection, there is a challenge that creating that thread is something each platform would have to teach us how to do (e.g. embedded threads can't actually create a thread with the `std::thread` constructor). There's also no precedent for libc++ to create a thread on the user's behalf.

https://github.com/llvm/llvm-project/pull/175451
