[libcxx-commits] [libcxx] [libc++][pstl] std_thread scheduler (PR #212121)
via libcxx-commits
libcxx-commits at lists.llvm.org
Wed Jul 29 08:06:53 PDT 2026
llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-libcxx
Author: Michael G. Kazakov (mikekazakov)
<details>
<summary>Changes</summary>
This PR makes the `std_thread` PSTL backend actually parallel.
The aim of this implementation is mostly an exploration of a potential MVP that can bring in the parallel execution and still be as simple as possible.
The header front-end part of the `std_thread` PSTL backend was entirely copied from the existing `libdispatch` backend, only with renames and using this scheduler instead of GCD.
The scheduler part is exposed via a single function `apply`:
```c++
_LIBCPP_EXPORTED_FROM_ABI void
__apply(size_t iterations, void* context, void (*func)(void* context, size_t iteration)) noexcept;
```
The scheduler by design supports almost unlimited nested parallelism.
In practice it can be limited by the stack size.
Internally it works approximately as follows:
- `hardware_concurrency * 2` queues are created.
The queues are SPMC work-stealing circular deques based on the Chase-Lev paper.
The queue entries are raw pointers to the task objects that live on a stack of the client thread.
- `hardware_concurrency - 1` worker threads are spawn.
They are spawn eagerly upon initialisation and live forever.
Each worker is assigned its queue upfront.
- The worker threads either pick a task from their own queues or steal one from other queues.
Once there's no more work to do, they go to sleep until other thread awakens them.
- The client thread enters via the `__apply` function.
It identifies itself via `std::thread::id` and acquires either a free queue or the same queue this thread already uses.
Because of nested parallelism, the client thread may be one of the worker threads.
Next, `workers_count` requests to participate in the parallel application are pushed into this queue.
The calling thread participates in working on the parallel application of its task and potentially steals others' work.
The worker threads steal the requests to participate from this queue and help with the applications.
Once the entire input task was processed, the function releases the queue and returns to the caller outside PSTL.
Some benchmarks of `std::for_each` with this backend (`-DLIBCXX_PSTL_BACKEND=std_thread`), measured on Apple M5 (4+6 cores), showing ~5x speedup:
```c++
std::for_each(policy, first, last, [](double& x) {
x = std::pow(std::exp(std::sqrt(std::sin(std::cos(x)) + 1.0)), 42.0);
});
```
```
| ------------------------------------------------------------------------------------------------------
| Benchmark Time CPU Iterations
| ------------------------------------------------------------------------------------------------------
| std::for_each(std::execution::seq, vector<double>)/65536 1516850 ns 1387649 ns 444
| std::for_each(std::execution::seq, vector<double>)/524288 10347158 ns 10340913 ns 69
| std::for_each(std::execution::seq, vector<double>)/4194304 84099875 ns 83784500 ns 10
| std::for_each(std::execution::seq, vector<double>)/33554432 594077042 ns 593392000 ns 1
| std::for_each(std::execution::par, vector<double>)/65536 286998 ns 278579 ns 2516
| std::for_each(std::execution::par, vector<double>)/524288 2104996 ns 2040708 ns 342
| std::for_each(std::execution::par, vector<double>)/4194304 16995701 ns 16120674 ns 43
| std::for_each(std::execution::par, vector<double>)/33554432 131717232 ns 126858857 ns 7
`-----------------------------
```
For comparison, here's the same benchmark executed with `libdispatch` backend (`-DLIBCXX_PSTL_BACKEND=libdispatch`):
```
| ------------------------------------------------------------------------------------------------------
| Benchmark Time CPU Iterations
| ------------------------------------------------------------------------------------------------------
| std::for_each(std::execution::seq, vector<double>)/65536 1571508 ns 1467843 ns 528
| std::for_each(std::execution::seq, vector<double>)/524288 11254936 ns 11192182 ns 66
| std::for_each(std::execution::seq, vector<double>)/4194304 87344940 ns 87267667 ns 9
| std::for_each(std::execution::seq, vector<double>)/33554432 622306875 ns 622191000 ns 1
| std::for_each(std::execution::par, vector<double>)/65536 358571 ns 348536 ns 2039
| std::for_each(std::execution::par, vector<double>)/524288 2705046 ns 2630560 ns 268
| std::for_each(std::execution::par, vector<double>)/4194304 20368309 ns 18077488 ns 41
| std::for_each(std::execution::par, vector<double>)/33554432 158312851 ns 139865000 ns 7
`-----------------------------
```
---
Patch is 34.83 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/212121.diff
5 Files Affected:
- (modified) libcxx/include/__pstl/backends/std_thread.h (+302-25)
- (modified) libcxx/src/CMakeLists.txt (+6)
- (added) libcxx/src/pstl/std_thread.cpp (+478)
- (added) libcxx/test/libcxx/algorithms/pstl.std_thread.apply.pass.cpp (+55)
- (modified) libcxx/utils/libcxx/test/features/libcxx_macros.py (+1)
``````````diff
diff --git a/libcxx/include/__pstl/backends/std_thread.h b/libcxx/include/__pstl/backends/std_thread.h
index 93935d22b9442..95dc8c1f77d5d 100644
--- a/libcxx/include/__pstl/backends/std_thread.h
+++ b/libcxx/include/__pstl/backends/std_thread.h
@@ -9,7 +9,24 @@
#ifndef _LIBCPP___PSTL_BACKENDS_STD_THREAD_H
#define _LIBCPP___PSTL_BACKENDS_STD_THREAD_H
+#include <__algorithm/inplace_merge.h>
+#include <__algorithm/lower_bound.h>
+#include <__algorithm/max.h>
+#include <__algorithm/merge.h>
+#include <__algorithm/move.h>
+#include <__algorithm/upper_bound.h>
+#include <__atomic/atomic.h>
#include <__config>
+#include <__cstddef/ptrdiff_t.h>
+#include <__exception/terminate.h>
+#include <__iterator/iterator_traits.h>
+#include <__iterator/move_iterator.h>
+#include <__memory/allocator.h>
+#include <__memory/construct_at.h>
+#include <__memory/destroy.h>
+#include <__memory/unique_ptr.h>
+#include <__new/exceptions.h>
+#include <__numeric/reduce.h>
#include <__pstl/backend_fwd.h>
#include <__pstl/cpu_algos/any_of.h>
#include <__pstl/cpu_algos/cpu_traits.h>
@@ -22,7 +39,9 @@
#include <__pstl/cpu_algos/transform.h>
#include <__pstl/cpu_algos/transform_reduce.h>
#include <__utility/empty.h>
+#include <__utility/exception_guard.h>
#include <__utility/move.h>
+#include <__utility/pair.h>
#include <optional>
#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
@@ -35,8 +54,47 @@ _LIBCPP_PUSH_MACROS
#if _LIBCPP_STD_VER >= 17
_LIBCPP_BEGIN_NAMESPACE_STD
+_LIBCPP_BEGIN_EXPLICIT_ABI_ANNOTATIONS
namespace __pstl {
+namespace __std_thread {
+
+_LIBCPP_EXPORTED_FROM_ABI void
+__apply(size_t __iterations, void* __context, void (*__func)(void* __context, size_t __iteration)) noexcept;
+
+struct __chunk_partitions {
+ ptrdiff_t __chunk_count_; // includes the first chunk
+ ptrdiff_t __chunk_size_;
+ ptrdiff_t __first_chunk_size_;
+};
+
+[[__gnu__::__const__]] _LIBCPP_EXPORTED_FROM_ABI __chunk_partitions __partition_chunks(ptrdiff_t __size) noexcept;
+
+template <class _Func>
+_LIBCPP_HIDE_FROM_ABI void __apply(size_t __chunk_count, _Func __func) noexcept {
+ __std_thread::__apply(__chunk_count, &__func, [](void* __context, size_t __chunk) {
+ (*static_cast<_Func*>(__context))(__chunk);
+ });
+}
+
+template <class _RandomAccessIterator, class _Functor>
+_LIBCPP_HIDE_FROM_ABI optional<__empty>
+__parallel_for(__chunk_partitions __partitions, _RandomAccessIterator __first, _Functor __func) {
+ // Perform the chunked execution.
+ __std_thread::__apply(__partitions.__chunk_count_, [&](size_t __chunk) {
+ auto __this_chunk_size = __chunk == 0 ? __partitions.__first_chunk_size_ : __partitions.__chunk_size_;
+ auto __index =
+ __chunk == 0
+ ? 0
+ : (__chunk * __partitions.__chunk_size_) + (__partitions.__first_chunk_size_ - __partitions.__chunk_size_);
+ __func(__first + __index, __first + __index + __this_chunk_size);
+ });
+
+ return __empty{};
+}
+
+} // namespace __std_thread
+
//
// This partial backend implementation is for testing purposes only and not meant for production use. This will be
// replaced by a proper implementation once the PSTL implementation is somewhat stable.
@@ -46,45 +104,263 @@ namespace __pstl {
template <>
struct __cpu_traits<__std_thread_backend_tag> {
- template <class _RandomAccessIterator, class _Fp>
+ template <class _RandomAccessIterator, class _Functor>
_LIBCPP_HIDE_FROM_ABI static optional<__empty>
- __for_each(_RandomAccessIterator __first, _RandomAccessIterator __last, _Fp __f) {
- __f(__first, __last);
- return __empty{};
+ __for_each(_RandomAccessIterator __first, _RandomAccessIterator __last, _Functor __func) {
+ return __std_thread::__parallel_for(
+ __std_thread::__partition_chunks(__last - __first), std::move(__first), std::move(__func));
}
- template <class _Index, class _UnaryOp, class _Tp, class _BinaryOp, class _Reduce>
- _LIBCPP_HIDE_FROM_ABI static optional<_Tp>
- __transform_reduce(_Index __first, _Index __last, _UnaryOp, _Tp __init, _BinaryOp, _Reduce __reduce) {
- return __reduce(std::move(__first), std::move(__last), std::move(__init));
- }
+ template <class _RandomAccessIterator1, class _RandomAccessIterator2, class _RandomAccessIteratorOut>
+ struct __merge_range {
+ _LIBCPP_HIDE_FROM_ABI
+ __merge_range(_RandomAccessIterator1 __mid1, _RandomAccessIterator2 __mid2, _RandomAccessIteratorOut __result)
+ : __mid1_(__mid1), __mid2_(__mid2), __result_(__result) {}
- template <class _RandomAccessIterator, class _Compare, class _LeafSort>
- _LIBCPP_HIDE_FROM_ABI static optional<__empty>
- __stable_sort(_RandomAccessIterator __first, _RandomAccessIterator __last, _Compare __comp, _LeafSort __leaf_sort) {
- __leaf_sort(__first, __last, __comp);
- return __empty{};
- }
+ _RandomAccessIterator1 __mid1_;
+ _RandomAccessIterator2 __mid2_;
+ _RandomAccessIteratorOut __result_;
+ };
- _LIBCPP_HIDE_FROM_ABI static void __cancel_execution() {}
-
- template <class _RandomAccessIterator1,
- class _RandomAccessIterator2,
- class _RandomAccessIterator3,
- class _Compare,
- class _LeafMerge>
+ template <typename _RandomAccessIterator1,
+ typename _RandomAccessIterator2,
+ typename _RandomAccessIterator3,
+ typename _Compare,
+ typename _LeafMerge>
_LIBCPP_HIDE_FROM_ABI static optional<__empty>
__merge(_RandomAccessIterator1 __first1,
_RandomAccessIterator1 __last1,
_RandomAccessIterator2 __first2,
_RandomAccessIterator2 __last2,
- _RandomAccessIterator3 __outit,
+ _RandomAccessIterator3 __result,
_Compare __comp,
- _LeafMerge __leaf_merge) {
- __leaf_merge(__first1, __last1, __first2, __last2, __outit, __comp);
+ _LeafMerge __leaf_merge) noexcept {
+ __std_thread::__chunk_partitions __partitions =
+ __std_thread::__partition_chunks(std::max<ptrdiff_t>(__last1 - __first1, __last2 - __first2));
+
+ if (__partitions.__chunk_count_ == 0)
+ return __empty{};
+
+ if (__partitions.__chunk_count_ == 1) {
+ __leaf_merge(__first1, __last1, __first2, __last2, __result, __comp);
+ return __empty{};
+ }
+
+ using __merge_range_t = __merge_range<_RandomAccessIterator1, _RandomAccessIterator2, _RandomAccessIterator3>;
+ auto const __n_ranges = __partitions.__chunk_count_ + 1;
+
+ // TODO: use __uninitialized_buffer
+ auto __destroy = [=](__merge_range_t* __ptr) {
+ std::destroy_n(__ptr, __n_ranges);
+ std::allocator<__merge_range_t>().deallocate(__ptr, __n_ranges);
+ };
+
+ unique_ptr<__merge_range_t[], decltype(__destroy)> __ranges(
+ [&]() -> __merge_range_t* {
+# if _LIBCPP_HAS_EXCEPTIONS
+ try {
+# endif
+ return std::allocator<__merge_range_t>().allocate(__n_ranges);
+# if _LIBCPP_HAS_EXCEPTIONS
+ } catch (const std::bad_alloc&) {
+ return nullptr;
+ }
+# endif
+ }(),
+ __destroy);
+
+ if (!__ranges)
+ return nullopt;
+
+ // TODO: Improve the case where the smaller range is merged into just a few (or even one) chunks of the larger case
+ __merge_range_t* __r = __ranges.get();
+ std::__construct_at(__r++, __first1, __first2, __result);
+
+ bool __iterate_first_range = __last1 - __first1 > __last2 - __first2;
+
+ auto __compute_chunk = [&](size_t __chunk_size) -> __merge_range_t {
+ auto [__mid1, __mid2] = [&] {
+ if (__iterate_first_range) {
+ auto __m1 = __first1 + __chunk_size;
+ auto __m2 = std::lower_bound(__first2, __last2, __m1[-1], __comp);
+ return std::make_pair(__m1, __m2);
+ } else {
+ auto __m2 = __first2 + __chunk_size;
+ auto __m1 = std::lower_bound(__first1, __last1, __m2[-1], __comp);
+ return std::make_pair(__m1, __m2);
+ }
+ }();
+
+ __result += (__mid1 - __first1) + (__mid2 - __first2);
+ __first1 = __mid1;
+ __first2 = __mid2;
+ return {std::move(__mid1), std::move(__mid2), __result};
+ };
+
+ // handle first chunk
+ std::__construct_at(__r++, __compute_chunk(__partitions.__first_chunk_size_));
+
+ // handle 2 -> N - 1 chunks
+ for (ptrdiff_t __i = 0; __i != __partitions.__chunk_count_ - 2; ++__i)
+ std::__construct_at(__r++, __compute_chunk(__partitions.__chunk_size_));
+
+ // handle last chunk
+ std::__construct_at(__r, __last1, __last2, __result);
+
+ __std_thread::__apply(__partitions.__chunk_count_, [&](size_t __index) {
+ auto __first_iters = __ranges[__index];
+ auto __last_iters = __ranges[__index + 1];
+ __leaf_merge(
+ __first_iters.__mid1_,
+ __last_iters.__mid1_,
+ __first_iters.__mid2_,
+ __last_iters.__mid2_,
+ __first_iters.__result_,
+ __comp);
+ });
+
+ return __empty{};
+ }
+
+ template <class _RandomAccessIterator, class _Transform, class _Value, class _Combiner, class _Reduction>
+ _LIBCPP_HIDE_FROM_ABI static optional<_Value> __transform_reduce(
+ _RandomAccessIterator __first,
+ _RandomAccessIterator __last,
+ _Transform __transform,
+ _Value __init,
+ _Combiner __combiner,
+ _Reduction __reduction) {
+ if (__first == __last)
+ return __init;
+
+ auto __partitions = __std_thread::__partition_chunks(__last - __first);
+
+ auto __destroy = [__count = __partitions.__chunk_count_](_Value* __ptr) {
+ std::destroy_n(__ptr, __count);
+ std::allocator<_Value>().deallocate(__ptr, __count);
+ };
+
+ // TODO: use __uninitialized_buffer
+ // TODO: allocate one element per worker instead of one element per chunk
+ unique_ptr<_Value[], decltype(__destroy)> __values(
+ std::allocator<_Value>().allocate(__partitions.__chunk_count_), __destroy);
+
+ // __apply is noexcept
+ __std_thread::__apply(__partitions.__chunk_count_, [&](size_t __chunk) {
+ auto __this_chunk_size = __chunk == 0 ? __partitions.__first_chunk_size_ : __partitions.__chunk_size_;
+ auto __index = __chunk == 0 ? 0
+ : (__chunk * __partitions.__chunk_size_) +
+ (__partitions.__first_chunk_size_ - __partitions.__chunk_size_);
+ if (__this_chunk_size != 1) {
+ std::__construct_at(
+ __values.get() + __chunk,
+ __reduction(__first + __index + 2,
+ __first + __index + __this_chunk_size,
+ __combiner(__transform(__first + __index), __transform(__first + __index + 1))));
+ } else {
+ std::__construct_at(__values.get() + __chunk, __transform(__first + __index));
+ }
+ });
+
+ return std::reduce(
+ std::make_move_iterator(__values.get()),
+ std::make_move_iterator(__values.get() + __partitions.__chunk_count_),
+ std::move(__init),
+ __combiner);
+ }
+
+ template <class _RandomAccessIterator, class _Comp, class _LeafSort>
+ _LIBCPP_HIDE_FROM_ABI static optional<__empty>
+ __stable_sort(_RandomAccessIterator __first, _RandomAccessIterator __last, _Comp __comp, _LeafSort __leaf_sort) {
+ const auto __size = __last - __first;
+ auto __partitions = __std_thread::__partition_chunks(__size);
+
+ if (__partitions.__chunk_count_ == 0)
+ return __empty{};
+
+ if (__partitions.__chunk_count_ == 1) {
+ __leaf_sort(__first, __last, __comp);
+ return __empty{};
+ }
+
+ using _Value = __iterator_value_type<_RandomAccessIterator>;
+
+ auto __destroy = [__size](_Value* __ptr) {
+ std::destroy_n(__ptr, __size);
+ std::allocator<_Value>().deallocate(__ptr, __size);
+ };
+
+ // TODO: use __uninitialized_buffer
+ unique_ptr<_Value[], decltype(__destroy)> __values(std::allocator<_Value>().allocate(__size), __destroy);
+
+ // Initialize all elements to a moved-from state
+ // TODO: Don't do this - this can be done in the first merge - see https://llvm.org/PR63928
+ std::__construct_at(__values.get(), std::move(*__first));
+ for (__iterator_difference_type<_RandomAccessIterator> __i = 1; __i != __size; ++__i) {
+ std::__construct_at(__values.get() + __i, std::move(__values.get()[__i - 1]));
+ }
+ *__first = std::move(__values.get()[__size - 1]);
+
+ __std_thread::__parallel_for(
+ __partitions,
+ __first,
+ [&__leaf_sort, &__comp](_RandomAccessIterator __chunk_first, _RandomAccessIterator __chunk_last) {
+ __leaf_sort(std::move(__chunk_first), std::move(__chunk_last), __comp);
+ });
+
+ bool __objects_are_in_buffer = false;
+ do {
+ const auto __old_chunk_size = __partitions.__chunk_size_;
+ if (__partitions.__chunk_count_ % 2 == 1) {
+ auto __inplace_merge_chunks = [&__comp, &__partitions](auto __first_chunk_begin) {
+ std::inplace_merge(
+ __first_chunk_begin,
+ __first_chunk_begin + __partitions.__first_chunk_size_,
+ __first_chunk_begin + __partitions.__first_chunk_size_ + __partitions.__chunk_size_,
+ __comp);
+ };
+ if (__objects_are_in_buffer)
+ __inplace_merge_chunks(__values.get());
+ else
+ __inplace_merge_chunks(__first);
+ __partitions.__first_chunk_size_ += 2 * __partitions.__chunk_size_;
+ } else {
+ __partitions.__first_chunk_size_ += __partitions.__chunk_size_;
+ }
+
+ __partitions.__chunk_size_ *= 2;
+ __partitions.__chunk_count_ /= 2;
+
+ auto __merge_chunks = [__partitions, __old_chunk_size, &__comp](auto __from_first, auto __to_first) {
+ __std_thread::__parallel_for(
+ __partitions,
+ __from_first,
+ [__old_chunk_size, &__from_first, &__to_first, &__comp](auto __chunk_first, auto __chunk_last) {
+ std::merge(std::make_move_iterator(__chunk_first),
+ std::make_move_iterator(__chunk_last - __old_chunk_size),
+ std::make_move_iterator(__chunk_last - __old_chunk_size),
+ std::make_move_iterator(__chunk_last),
+ __to_first + (__chunk_first - __from_first),
+ __comp);
+ });
+ };
+
+ if (__objects_are_in_buffer)
+ __merge_chunks(__values.get(), __first);
+ else
+ __merge_chunks(__first, __values.get());
+ __objects_are_in_buffer = !__objects_are_in_buffer;
+ } while (__partitions.__chunk_count_ > 1);
+
+ if (__objects_are_in_buffer) {
+ std::move(__values.get(), __values.get() + __size, __first);
+ }
+
return __empty{};
}
+ _LIBCPP_HIDE_FROM_ABI static void __cancel_execution() {}
+
static constexpr size_t __lane_size = 64;
};
@@ -135,6 +411,7 @@ struct __fill<__std_thread_backend_tag, _ExecutionPolicy>
: __cpu_parallel_fill<__std_thread_backend_tag, _ExecutionPolicy> {};
} // namespace __pstl
+_LIBCPP_END_EXPLICIT_ABI_ANNOTATIONS
_LIBCPP_END_NAMESPACE_STD
#endif // _LIBCPP_STD_VER >= 17
diff --git a/libcxx/src/CMakeLists.txt b/libcxx/src/CMakeLists.txt
index e029f9f29d5ea..472552afb6057 100644
--- a/libcxx/src/CMakeLists.txt
+++ b/libcxx/src/CMakeLists.txt
@@ -310,6 +310,12 @@ if (LIBCXX_PSTL_BACKEND STREQUAL "libdispatch")
)
endif()
+if (LIBCXX_PSTL_BACKEND STREQUAL "std_thread")
+ list(APPEND LIBCXX_EXPERIMENTAL_SOURCES
+ pstl/std_thread.cpp
+ )
+endif()
+
if (LIBCXX_ENABLE_LOCALIZATION AND LIBCXX_ENABLE_FILESYSTEM AND LIBCXX_ENABLE_TIME_ZONE_DATABASE)
list(APPEND LIBCXX_EXPERIMENTAL_SOURCES
experimental/include/tzdb/time_zone_private.h
diff --git a/libcxx/src/pstl/std_thread.cpp b/libcxx/src/pstl/std_thread.cpp
new file mode 100644
index 0000000000000..37a667bf243dc
--- /dev/null
+++ b/libcxx/src/pstl/std_thread.cpp
@@ -0,0 +1,478 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include <__algorithm/max.h>
+#include <__atomic/aliases.h>
+#include <__atomic/atomic.h>
+#include <__config>
+#include <__mutex/lock_guard.h>
+#include <__mutex/mutex.h>
+#include <__mutex/once_flag.h>
+#include <__mutex/unique_lock.h>
+#include <__new/interference_size.h>
+#include <__pstl/backends/std_thread.h>
+#include <__thread/this_thread.h>
+#include <__thread/thread.h>
+#include <__type_traits/is_trivial.h>
+#include <latch>
+
+// #define WITH_LOGGING 1
+#ifdef WITH_LOGGING
+# include <stdio.h>
+#endif
+
+_LIBCPP_BEGIN_NAMESPACE_STD
+_LIBCPP_BEGIN_EXPLICIT_ABI_ANNOTATIONS
+namespace __pstl::__std_thread {
+
+#if defined(_AIX) && !defined(__64BIT__)
+// on AIX (32-bit):
+// c++/v1/__atomic/support/c11.h:83:10: error: large atomic operation may incur significant performance penalty;
+// the access size (8 bytes) exceeds the max lock-free size (4 bytes) [-Werror,-Watomic-alignment]
+// Sequential dummy implementation for now.
+void __apply(size_t __iterations, void* __context, void (*__func)(void* __context, size_t __iteration)) noexcept {
+ for (std::size_t i = 0; i < __iterations; ++i) {
+ __func(__context, i);
+ }
+}
+#else
+
+# ifdef WITH_LOGGING
+[[clang::no_destroy]] static std::mutex LOG_MUTEX;
+# define LOG(...) \
+ do { \
+ std::lock_guard lock{LOG_MUTEX}; \
+ fprintf(stderr, __VA_ARGS__); \
+ } while (0)
+# else
+# define LOG(...) \
+ do { \
+ } while (0)
+# endif
+
+struct Task;
+
+// Implementation of "Dynamic Circular Work-Stealing Deque" by David Chase and Yossi Lev.
+// The element type is a pointer to Task, nullptr is not a valid value.
+struct alignas(std::hardware_destructive_interference_size) SPMCQueue {
+ static constexpr unsigned log_initial_size = 6; // 64 elements by default
+
+ SPMCQueue() noexcept : bottom(0), top(0), buffer(alloc_buffer(log_initial_size)) {}
+ SPMCQueue(const SPMCQueue&) = delete;
+ SPMCQueue& operator=(const SPMCQueue&) = delete;
+ ~SPMCQueue() { ::operator delete(buffer); }
+
+ // Pushes the new task at the bottom of the queue, only called by the owner.
+ // Can return false if the allocation of grown buffer fails.
+ bool push(Task* task) noexcept {
+ std::uint64_t b = bottom.load();
+ std::uint64_t t = top.load();
+ std::uint64_t size = b - t;
+ if (size >= buffer_size(buffer) - 1) {
+ Buffer* grown = grow_buffer(buffer, b, t);
+ if (grown == nullptr) {
+ return false;
+ }
+ Buffer* current;
+ {
+ std::lock_guard lock{mut};
+ current = std::exchange(buffer, grown);
+ }
+ if (current->ref_count.fetch_sub(1) == 1) {
+ ::operator delete(current);
+ }
+ }
+ put(buffer, b, task);
+ bottom.store(b + 1);
+ return true;
+ }
+
+ // Pops a task from the bottom of the queue, only called by the owner.
+ // Returns nullptr if the queue is empty.
+ Task* pop() noexcept {
+ std::uint64_t b = bottom.load();
+ if (b == 0)
+ return nullptr;
+ --b;
+ bottom.store(b);
+ std::uint64_t t = top.load();
+ if (t <= b) {
+ Task* task = get(buffer, b);
+ if (b > t)
+ return task;
+ if (top.compare_exchange_strong(t, t + 1)) {
+ bottom.store(t + 1);
+ return task;
+ } else {
+ bottom.store(t);
+ return nullptr;
+ }
+ } else {
+ bottom.store(t);
+ return nullptr;
+ }
+ }
+
+ // Steals a task from the top of the queue, can be called by any thread.
+ // Returns nullptr if the queue is empty or if ...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/212121
More information about the libcxx-commits
mailing list