[libcxx-commits] [libcxx] [libc++] Implement P2545R4 Read-Copy Update (RCU) (PR #175451)

[via libcxx-commits]

================
@@ -0,0 +1,464 @@
+====================
+``<rcu>`` Design
+====================
+
+.. contents::
+   :local:
+   :depth: 2
+
+
+Introduction
+============
+
+This is the C++ paper `Read-Copy Update (RCU) <https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2545r4.pdf>`__.
+There are three designs discussed in the paper `Supplementary Material for User-Level Implementations of
+Read-Copy Update <http://www.rdrop.com/users/paulmck/RCU/urcu-supp-accepted.2011.08.30a.pdf>`__
+
+- Quiescent-State-Based Reclamation RCU
+- General-Purpose RCU
+- Low-Overhead RCU Via Signal Handling
+
+libc++ adopted a variation that closely resembles the "General-Purpose RCU" design in the above paper.
+
+Synopsis
+========
+::
+
+  class rcu_domain {
+  public:
+    void lock() noexcept;
+    bool try_lock() noexcept;
+    void unlock() noexcept;
+  };
+
+  rcu_domain& rcu_default_domain() noexcept;
+
+  // blocks until all previous readers exit the critical section
+  void rcu_synchronize(rcu_domain& dom = rcu_default_domain()) noexcept;
+
+  // blocks until all deleters of previously retired objects run
+  void rcu_barrier(rcu_domain& dom = rcu_default_domain()) noexcept;
+
+  template<class T, class D = default_delete<T>>
+  void rcu_retire(T* p, D d = D(), rcu_domain& dom = rcu_default_domain());
+
+  template<class T, class D = default_delete<T>>
+  class rcu_obj_base {
+  public:
+    void retire(D d = D(), rcu_domain& dom = rcu_default_domain()) noexcept;
+  };
+
+Background Information
+======================
+
+`rcu` is an alternative to read-write locks and it is suitable for read-mostly data structures.
+Here is an example usage ::
+
+  struct Data { /* members */ };
+  std::atomic<Data*> data_;
+
+  // reader thread
+  // ==============================================
+  template <typename Func>
+  Result reader_op(Func fn) {
+    std::scoped_lock l(std::rcu_default_domain());
+    Data* p = data_;
+    // fn should not block too long or call
+    // rcu_synchronize(), rcu_barrier(), or
+    // rcu_retire(), directly or indirectly
+    return fn(p);
+ }
+
+  // writer thread
+  // ==============================================
+  // May be called concurrently with reader_op
+  void update(Data* newdata) {
+    Data* olddata = data_.exchange(newdata);
+    std::rcu_retire(olddata); // reclaim *olddata when safe
+  }
+
+  // collector thread 
+  // ==============================================
+  void collect_garbage(std::stop_token st) {
+    while (!st.stop_requested()) {
+      std::rcu_barrier();
+      std::this_thread::sleep(1s);
+    }
+  }
+
+There are several key properties that an implementation of `rcu` must satisfy:
+
+- On the reader side, `rcu_domain::lock` and `rcu_domain::unlock` must not block the thread while the writer thread is performing updates, other readers are `lock` ing the critical section, or the collector thread is reclaiming the objects.
+
+- On the writer side, our interpretation is that `retire`-ing an object should not block for quality of implementation reasons. However, it is technically conforming to block (e.g. evaluate the `deleter`s directly inside `rcu_retire`).
+
+- On the collector side, `rcu_synchronize` should block until at least all the existing readers exit their critical sections
+  via `rcu_domain::unlock`. Note that this is a key difference between `rcu` and read-write locks:
+  In case a late reader enters the critical section after the collector thread has called `rcu_synchronize` and started waiting, the
+  collector thread does not need to wait for the late reader to exit the critical section. In contrast, a collector thread trying to acquire a read-write lock would have to wait until the late reader releases the lock before it can acquire it.
+
+- On the collector side, `rcu_barrier` should block until all the objects `retired` before the `rcu_barrier` call have been reclaimed.
+
+- The threads that are participating in `rcu` must be known by the `rcu` implementation.
+
+
+Adopted Design
+==============
+
+The core idea of `rcu` can be described by this image from lwn.net
+
+.. image:: https://static.lwn.net/images/ns/kernel/rcu/GracePeriodGood.png
+
+- Each row is a thread. The last row is the collector thread and the rows above are the reader threads.
+- Each "Reader" block represents a critical section, which starts with `rcu_domain::lock` and ends with `rcu_domain::unlock` .
+- When `rcu_retire` is called from the writer thread, it starts the "Removal" block.
+- When `rcu_synchronize`/`rcu_barrier` is called from the collector thread, it starts the "Grace Period" block. We need to wait until all the 
+  "Reader" blocks that started before the "Grace Period" started, to exit via `rcu_domain::unlock`, then we can end the "Grace Period".
+  Note that the "Grace Period" ends after the 4th row's "Reader" block ends. Also note that the "Grace Period" does not need to wait
+  for the late "Reader" blocks.
+- After the "Grace Period" ends, we can start the "Reclaim" block to reclaim the retired objects.
+
+Libc++ adopted a design that closely resembles the "General-Purpose RCU" design in the above paper.
+
+Some key details of this design are:
+
+- There is a global state which has two phases and flips between the two.
+- Each thread stores its state: whether there is a reader in the critical section, and which phase was active when it entered the critical section.
+- Assume the two phases are `a` and `b`, and the current phase is `a`. When `rcu_synchronize`/`rcu_barrier` is called, it
+
+  - flips the global state to `b`
+  - goes through a grace period: waits until all the threads that are in the critical section with phase `a` have exited their critical section
+  - flips the global state back to phase `a`
+  - goes through another grace period: waits until all the threads that are in the critical section with phase `b` have exited their critical section
+
+When `rcu_synchronize`/`rcu_barrier` returns, we can be sure that all the readers that were in the critical section before the call are now out of the critical section.
+The paper explains why we need to wait two phases instead of just one phase in detail. The key point is that, if we only wait for the readers in the previous phase to exit,
+there might be a late reader that enters the critical section after we flip the global state and before we wait for the previous phase's readers to exit.
+
+Implementation Terminology
+---------------------------
+
+- reader: A thread that calls `rcu_domain::lock` and `rcu_domain::unlock` to enter and exit the critical section.
+
+- quiescent state: When a thread is *not* in an RCU read-side critical section (between `lock` and `unlock`),
+  it is in a quiescent state.
+
+- grace period: Any time period during which every thread has been in at least one quiescent state is a grace period.
+  The grace period cannot end until all pre-existing readers have exited their critical sections. (see the image above)
+
+Implementation Details
+----------------------
+
+`class thread_local_container`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A helper class that manages thread local objects. It provides APIs to get the `thread_local` object for the current thread, and to iterate through all the `thread_local` objects from all threads.
+It only contains `static` member variables and functions.
+
+It uses `mutex` to protect the list of all object pointers. Since the list is only modified when a thread first calls `get_current_thread_instance` to register itself, and when a thread exits, 
+the contention on the `mutex` should be low.
+
+TODO: We need to replace `mutex` as all non allocating APIs in `rcu` are designed to be `noexcept` and `mutex` can throw.
+
+`struct reader_states`
+~~~~~~~~~~~~~~~~~~~~~~
+
+It defines the state of each reader thread. The state is essentially a pair of `(phase, lock_nested_level)` merged into a single `uint16_t` . 
+
+- `phase` is a single bit as the global state only has two phases. This `phase` is the phase of the global state when the current thread enters the critical section.
+- `lock_nested_level` is the nested level of the `rcu_domain::lock` . As you can call `rcu_domain::lock` multiple times in the same thread, we need to keep track of the nested level.
+  When `lock_nested_level` is `0`, it means the current thread is in a quiescent state (not in a critical section).
+
+`class rcu_singly_list_view`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A helper class that provides a view of the intrusive singly linked list of `__rcu_node` s. It also stores the back pointer of the list to make the splice back operation more efficient.
+It provides APIs to splice from another `rcu_singly_list_view` or  `rcu_atomic_list_view`, and to iterate through the list.
+
+`class rcu_atomic_list_view`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A thread safe helper class that provides a view of the intrusive singly linked list of `__rcu_node` s. It provides APIs to push a node into the list, or to be spliced to another `rcu_singly_list_view`.
+
+`class rcu_domain_impl`
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+This is the main class that implements the `rcu` logic. It contains
+
+- `using per_thread_states = thread_local_container<reader_states>`. It contains each thread's `reader_states`. This is not a member but static thread_local objects.
+
+- `std::atomic<reader_states::state_type> global_reader_phase_` : the global state that flips between two phases. The readers will record the phase when they enter the critical section, and the collector will flip the global state when it calls `rcu_synchronize` to start a new grace period.
+
+- `std::mutex grace_period_mutex_` : If we have multiple collector threads calling `rcu_synchronize` concurrenly, we need to make sure only one of them is performing the phase flipping and deleter queue draining.
+  TODO: `mutex` can throw, we need to consider how to replace it.
----------------
huixie90 wrote:

I had a look at other implementations

- Folly: uses `std::mutex` for protecting multiple  `rcu_synchronize/rcu_barrier` calls. and at the same time, mark them `noexcept`. So in case it throws, we get `std::terminate`
- liburcu: it is a C library and it uses `pthread_mutex`. In case the lock has a error return code, it `abort()`

https://github.com/llvm/llvm-project/pull/175451