[libcxx-commits] [libcxx] [libc++] Implement P2545R4 Read-Copy Update (RCU) (PR #175451)

[via libcxx-commits]

================
@@ -0,0 +1,423 @@
+====================
+``<rcu>`` Design
+====================
+
+.. contents::
+   :local:
+   :depth: 2
+
+
+Introduction
+============
+
+This is the C++ paper `Read-Copy Update (RCU) <https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2545r4.pdf>`__.
+There are three designs discussed in the paper `Supplementary Material for User-Level Implementations of
+Read-Copy Update <http://www.rdrop.com/users/paulmck/RCU/urcu-supp-accepted.2011.08.30a.pdf>`__
+
+- Quiescent-State-Based Reclamation RCU
+- General-Purpose RCU
+- Low-Overhead RCU Via Signal Handling
+
+libc++ adopted a variation that closely resembles the "General-Purpose RCU" design in the above paper.
+
+Synopsis
+========
+::
+
+  class rcu_domain {
+  public:
+    void lock() noexcept;
+    bool try_lock() noexcept;
+    void unlock() noexcept;
+  };
+
+  rcu_domain& rcu_default_domain() noexcept;
+
+  void rcu_synchronize(rcu_domain& dom = rcu_default_domain()) noexcept;
+  void rcu_barrier(rcu_domain& dom = rcu_default_domain()) noexcept;
+
+  template<class T, class D = default_delete<T>>
+  void rcu_retire(T* p, D d = D(), rcu_domain& dom = rcu_default_domain());
+
+  template<class T, class D = default_delete<T>>
+  class rcu_obj_base {
+  public:
+    void retire(D d = D(), rcu_domain& dom = rcu_default_domain()) noexcept;
+  };
+
+Background Information
+======================
+
+`rcu` is an alternative to read-write locks and it is suitable for read-mostly data structures.
+Here is an example usage ::
+
+  struct Data { /* members */ };
+  std::atomic<Data*> data_;
+
+  // ==============================================
+  template <typename Func>
+  Result reader_op(Func fn) {
+    std::scoped_lock l(std::rcu_default_domain());
+    Data* p = data_;
+    // fn should not block too long or call
+    // rcu_synchronize(), rcu_barrier(), or
+    // rcu_retire(), directly or indirectly
+    return fn(p);
+ }
+
+  // ==============================================
+  // May be called concurrently with reader_op
+  void update(Data* newdata) {
+    Data* olddata = data_.exchange(newdata);
+    std::rcu_retire(olddata); // reclaim *olddata when safe
+  }
+
+There are several key properties that an implementation of `rcu` must satisfy:
+
+- On the reader side, `rcu_domain::lock` and `rcu_domain::unlock` must not block the thread while the writer thread
+  is performing updates.
+
+- On the writer side, `retire`-ing an object in principle should not block at all. However, it is technically conforming to block (e.g. evaluate the `deleter`s directly inside `rcu_retire`).
+
+- On the writer side, `rcu_synchronize` should block until at least all the existing readers exit their critical sections
+  via `rcu_domain::unlock` . Note that this is a key difference between `rcu` and read-write locks:
+  In case a late reader enters the critical section after the writer thread has called `rcu_synchronize` and started waiting, the
+  writer thread does not need to wait for the late reader to exit the critical section. In contrast, a writer thread trying to acquire a read-write lock would have to wait until the late reader releases the lock before it can acquire it.
+
+- On the writer side, `rcu_barrier` should block until all the objects `retired` before the `rcu_barrier` call have been reclaimed.
+
+- The threads that are using `rcu` must be known by the `rcu` implementation states.
+
+
+Adopted Design
+==============
+
+The core idea of `rcu` can be described by this image from lwn.net
+
+.. image:: https://static.lwn.net/images/ns/kernel/rcu/GracePeriodGood.png
+
+- Each row is a thread. The last row is the writer thread and the rows above are the reader threads.
+- Each "Reader" block represents a critical section, which starts with `rcu_domain::lock` and ends with `rcu_domain::unlock` .
+- When `rcu_retire` is called from the writer thread, it starts the "Removal" block.
+- When `rcu_synchronize` is called from the writer thread, it starts the "Grace Period" block. We need to wait until all the 
+  "Reader" blocks that started before the "Grace Period" started, to exit via `rcu_domain::unlock`, then we can end the "Grace Period".
+  Note that the "Grace Period" ends after the 4th row's "Reader" block ends. Also note that the "Grace Period" does not need to wait
+  for the late "Reader" blocks.
+- After the "Grace Period" ends, we can start the "Reclaim" block to reclaim the retired objects.
+
+Libc++ adopted a design that closely resembles the "General-Purpose RCU" design in the above paper.
+
+Some key details of this design are:
+
+- There is a global state which has two phases and it flips between the two phases.
+- Each thread stores its state: whether there is a reader in the critical section, and which phase it was when it entered the critical section.
+- When `rcu_synchronize` is called, it
+
+  - flips the global state to the next phase
+  - Going through a grace period: waits until all the threads that are in the critical section with the previous phase to exit the critical section.
+  - flips the global state back to the original phase
+  - Going through another grace period: waits until all the threads that are in the critical section with the next phase to exit the critical section.
+
+When `rcu_synchronize` returns, we can be sure that all the readers that were in the critical section before `rcu_synchronize` are now out of the critical section.
+The paper explains why we need to wait two phases instead of just one phase in detail. The key point is that, if we only wait for the readers in the previous phase to exit,
+there might be a late reader that enters the critical section after we flip the global state and before we wait for the previous phase's readers to exit.
+
+Implementation Terminology
+---------------------------
+
+- reader: A thread that calls `rcu_domain::lock` and `rcu_domain::unlock` to enter and exit the critical section.
+
+- quiescent state: When a thread is *not* in an RCU read-side critical section (between `lock` and `unlock`),
+  it is in a quiescent state.
+
+- grace period: Any time period during which every thread has been in at least one quiescent state is a grace period.
+  The grace period cannot end until all pre-existing readers have exited their critical sections. (see the image above)
+
+Implementation Details
+----------------------
+
+`thread_local_container`
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+A helper class that manages thread local objects. It provides APIs to get the `thread_local` object for the current thread, and to iterate through all the `thread_local` objects from all threads.
+It only contains `static` member variables and functions.
+
+It uses `mutex` to protect the list of all object pointers. Since the list is only modified when a thread first calls `get_current_thread_instance` to register itself, and when a thread exits, 
+the contention on the `mutex` should be low.
+
+TODO: We need to replace `mutex` as all non allocating APIs in `rcu` are designed to be `noexcept` and `mutex` can throw.
+
+`reader_states`
+~~~~~~~~~~~~~~~
+
+It defines the state of each reader thread. The state is essentially a pair of `(phase, lock_nested_level)` merged into a single `uint16_t` . 
+
+- `phase` is a single bit as the global state only has two phases. This `phase` is the phase of the global state when the current thread enters the critical section.
+- `lock_nested_level` is the nested level of the `rcu_domain::lock` . As you can call `rcu_domain::lock` multiple times in the same thread, we need to keep track of the nested level.
+  When `lock_nested_level` is `0`, it means the current thread is in a quiescent state (not in a critical section).
+
+The `reader_states` class is also a `static` helper class which only provides static definitions and functions, instead of the wrapper of the `uint16_t` state. This is because we need
+the `atomic` integer API, e.g. `fetch_or`. An `atomic` of a wrapper struct won't have these APIs.
+
+`rcu_singly_list_view`
+~~~~~~~~~~~~~~~~~~~~~~
+
+A helper class that provides a view of the intrusive singly linked list of `__rcu_node` s. It also stores the back pointer of the list to make the push back operation more efficient.
+It provides APIs to push a node to the back of the list, splice from another list and to iterate through the list.
+
+For the derived class of `rcu_obj_base`, the `retire` is `noexcept`, which means that no memory allocation can happen when pushing it to the retired callback queue.
+
+`rcu_domain_impl`
+~~~~~~~~~~~~~~~~~
+
+This is the main class that implements the `rcu` logic. It contains
+
+- `std::atomic<reader_states::state_type> global_reader_phase_` : the global state that flips between two phases. The readers will record the phase when they enter the critical section, and the writer will flip the global state when it calls `rcu_synchronize` to start a new grace period.
+
+- `std::mutex grace_period_mutex_` : If we have multiple writer threads calling `rcu_synchronize` concurrenly, we need to make sure only one of them is performing the phase flipping and deleter queue draining.
+  TODO: `mutex` can throw, we need to consider how to replace it.
+
+- `std::atomic<bool> grace_period_waiting_flag_` : This flag is used to sleep/wake up the writer thread that is waiting for the grace period to end.
+
+- `std::mutex retire_queue_mutex_` and  `rcu_singly_list_view __retired_callback_queue_` : This queue stores all the retired callbacks that are waiting for the grace period to end.
+  TODO: `mutex` can throw, we need to consider how to replace it.
+
+- `rcu_singly_list_view callbacks_phase_1_` and `rcu_singly_list_view callbacks_phase_2_` : These two queues are used to let the queued retired callbacks to go through two grace periods before invocation. No additional synchronization is needed for these two queues as they are only processed when the writer thread is holding the `grace_period_mutex_` .
----------------
huixie90 wrote:

I have updated the doc to document these steps

https://github.com/llvm/llvm-project/pull/175451