[libcxx-commits] [libcxx] [libc++] Implement P2545R4 Read-Copy Update (RCU) (PR #175451)

[via libcxx-commits]

================
@@ -0,0 +1,400 @@
+====================
+``<rcu>`` Design
+====================
+
+.. contents::
+   :local:
+   :depth: 2
+
+
+Introduction
+============
+
+This is the C++ paper `Read-Copy Update (RCU) <https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2545r4.pdf>`__.
+There are three designs discussed in the paper `Supplementary Material for User-Level Implementations of
+Read-Copy Update <http://www.rdrop.com/users/paulmck/RCU/urcu-supp-accepted.2011.08.30a.pdf>`__
+
+- Quiescent-State-Based Reclamation RCU
+- General-Purpose RCU
+- Low-Overhead RCU Via Signal Handling
+
+libc++ adopted a variation that closely resembles the "General-Purpose RCU" design in the above paper.
+
+
+Background Information
+======================
+
+`rcu` is an alternative to read-write locks and it is suitable for read-mostly data structures.
+Here is an example usage ::
+
+  struct Data /* members */ ;
+  std::atomic<Data*> data_;
+
+  // ==============================================
+  template <typename Func>
+  Result reader_op(Func fn) {
+    std::scoped_lock l(std::rcu_default_domain());
+    Data* p = data_;
+    // fn should not block too long or call
+    // rcu_synchronize(), rcu_barrier(), or
+    // rcu_retire(), directly or indirectly
+    return fn(p);
+ }
+
+  // ==============================================
+  // May be called concurrently with reader_op
+  void update(Data* newdata) {
+    Data* olddata = data_.exchange(newdata);
+    std::rcu_retire(olddata); // reclaim *olddata when safe
+  }
+
+There are several key details that an implementation of `rcu` must address:
+
+- On the reader side, `rcu_domain::lock` and `rcu_domain::unlock` must not block the thread while the writer thread
+  is performing updates.
+
+- On the writer side, `retire` -ing an object in principle should not block at all. However, it is conforming to
+  evaluate the `deleter` s inside `run_retire` .
+
+- On the writer side, `rcu_synchronize` should block until at least all the existing readers exit their critical sections
+  via `rcu_domain::unlock` . Note that this is a key difference between `rcu` and read-write locks:
+  In case a late reader enters the critical section after the writer thread has called `rcu_synchronize` and waiting, the
+  writer thread does not need to wait for the late reader to exit the critical section.
+
+- On the writer side, `run_barrier` should block until all the `retired` objects that happen before the `run_barrier` call are reclaimed.
+
+- The threads that are using `rcu` must be known by the `rcu` implementation states.
+
+
+Adopted Design
+==============
+
+The core idea of `rcu` can be described in this simple image from lwn.net
+
+.. image:: https://static.lwn.net/images/ns/kernel/rcu/GracePeriodGood.png
+
+- Each row is a thread. The last row is the writer thread and the rows above are the reader threads.
+- Each "Reader" block represents a critical section, which starts with `rcu_domain::lock` and ends with `rcu_domain::unlock` .
+- When `run_retire` is called from the writer thread, it starts the "Removal" block.
+- When `run_synchronize` is called from the writer thread, it starts the "Grace Period" block. We need to wait until all the 
+  "Reader" blocks that started before the "Grace Period" started, to exit via `rcu_domain::unlock`, then we can end the "Grace Period".
+  Note that the "Grace Period" ends after the 4th row's "Reader" block ends. Also note that the "Grace Period" does not need to wait
+  for the late "Reader" blocks.
+- After the "Grace Period" ends, we can start the "Reclaim" block to reclaim the retired objects.
+
+Libc++ adopted a design that closely resembles the "General-Purpose RCU" design in the above paper.
+
+Some key details of this design are:
+
+- There is a global state which has two phases and it flips between the two phases.
+- Each thread stores its state: whether there is a reader in the critical section, and which phase it was when it entered the critical section.
+- When `run_synchronize` is called, it
+
+  - flips the global state to the next phase
+  - Going through a grace period: waits until all the threads that are in the critical section with the previous phase to exit the critical section.
+  - flips the global state back to the original phase
+  - Going through another grace period: waits until all the threads that are in the critical section ith the next phase to exit the critical section.
+
+When `run_synchronize` returns, we can be sure that all the readers that were in the critical section before `run_synchronize` are now out of the critical section.
+The paper explains why we need to wait two phases instead of just one phase in detail. The key point is that, if we only wait for the readers in the previous phase to exit,
+there might be a late reader that enters the critical section after we flip the global state and before we wait for the previous phase's readers to exit.
+
+Implementation Terminology
+---------------------------
+
+- reader: A thread that calls `rcu_domain::lock` and `rcu_domain::unlock` to enter and exit the critical section.
+
+- quiescent state: When a thread is *not* in an RCU read-side critical section (between `lock` and `unlock`),
+  it is in a quiescent state.
+
+- grace period: Any time period during which every thread has been in at least one quiescent state is a grace period.
+  The grace period cannot end until all pre-existing readers have exited their critical sections. (see the image above)
+
+Implementation Details
+----------------------
+
+`thread_local_container`
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+A helper class that manages thread local objects. It provides APIs to get the `thread_local` object for the current thread, and to iterate through all the `thread_local` objects from all threads.
+It only contains `static` member variables and functions.
+
+It uses `mutex` to protect the list of all object pointers. Since the list is only modified when a thread first calls `get_current_thread_instance` to register itself, and when a thread exits, 
+the contention on the `mutex` should be low.
+
+TODO: We need to replace `mutex` as all non allocating APIs in `rcu` are designed to be `noexcept` and `mutex` can throw.
+
+`reader_states`
+~~~~~~~~~~~~~~~
+
+It defines the state of each reader thread. The state is essentially a pair of `(phase, lock_nested_level)` merged into a single `uint16_t` . 
+
+- `phase` is a single bit as the global state only has two phases. This `phase` is the phase of the global state when the current thread enters the critical section.
+- `lock_nested_level` is the nested level of the `rcu_domain::lock` . As you can call `rcu_domain::lock` multiple times in the same thread, we need to keep track of the nested level.
+  When `lock_nested_level` is `0`, it means the current thread is in a quiescent state (not in a critical section).
+
+The `reader_states` class is also a `static` helper class which only provides static definitions and functions, instead of the wrapper of the `uint16_t` state. This is because we need
+the `atomic` integer API, e.g. `fetch_or`. An `atomic` of a wrapper struct won't have these APIs.
+
+`rcu_singly_list_view`
+~~~~~~~~~~~~~~~~~~~~~~
+
+A helper class that provides a view of the intrusive singly linked list of `__rcu_node` s. It also stores the back pointer of the list to make the push back operation more efficient.
+It provides APIs to push a node to the back of the list, splice from another list and to iterate through the list.
+
+For the derived class of `rcu_obj_base`, the `retire` is `noexcept`, which means that no memory allocation can happen when pushing it to the retired callback queue.
+
+`rcu_domain_impl`
+~~~~~~~~~~~~~~~~~
+
+This is the main class that implements the `rcu` logic. It contains
+
+- `std::atomic<reader_states::state_type> global_reader_phase_` : the global state that flips between two phases. The readers will record the phase when they enter the critical section, and the writer will flip the global state when it calls `run_synchronize` to start a new grace period.
+
+- `std::mutex grace_period_mutex_` : If we have multiple writer threads calling `run_synchronize` concurrenly, we need to make sure only one of them is performing the phase flipping and deleter queue draining.
----------------
huixie90 wrote:

`run_synchronize`  isn't just flipping the `atomic` flag, it also has other operations including draining the queue and flipping the flag again etc. It is very challenging to avoid this mutex. In any case, this `mutex` is protecting multiple writers and writer thread is not what is optimised for in RCU

https://github.com/llvm/llvm-project/pull/175451