[llvm-branch-commits] [libcxx] 555106a - Optimize 'construct at end' loops in vector

[Martijn Vels via llvm-branch-commits]

Author: Martijn Vels
Date: 2020-06-18T13:35:46-04:00
New Revision: 555106aa69355c1e91903a85682e81dc9db57478

URL: https://github.com/llvm/llvm-project/commit/555106aa69355c1e91903a85682e81dc9db57478
DIFF: https://github.com/llvm/llvm-project/commit/555106aa69355c1e91903a85682e81dc9db57478.diff

LOG: Optimize 'construct at end' loops in vector

Summary:
This change adds local 'end' and 'pos' variables for the main loop inmstead of using the ConstructTransaction variables directly.

We observed that not all vector initialization and resize operations got properly vectorized, i.e., (partially) unrolled into XMM stores for floats.

For example, `vector<int32_t> v(n, 1)` gets vectorized, but `vector<float> v(n, 1)`. It looks like the compiler assumes the state is leaked / aliased in the latter case (unclear how/why for float, but not for int32), and because of this fails to see vectorization optimization?

See https://gcc.godbolt.org/z/UWhiie

By using a local `__new_end_` (fixed), and local `__pos` (copied into __tx.__pos_ per iteration), we offer the compiler a clean loop for unrolling.

A demonstration can be seen in the isolated logic in https://gcc.godbolt.org/z/KoCNWv

The com

Reviewers: EricWF, #libc!

Subscribers: libcxx-commits

Tags: #libc

Differential Revision: https://reviews.llvm.org/D82111

Added: 
    

Modified: 
    libcxx/include/vector

Removed: 
    


################################################################################
diff  --git a/libcxx/include/vector b/libcxx/include/vector
index 7d0fec87c4fa..1007beeaafd0 100644
--- a/libcxx/include/vector
+++ b/libcxx/include/vector
@@ -1043,8 +1043,9 @@ void
 vector<_Tp, _Allocator>::__construct_at_end(size_type __n)
 {
     _ConstructTransaction __tx(*this, __n);
-    for (; __tx.__pos_ != __tx.__new_end_; ++__tx.__pos_) {
-      __alloc_traits::construct(this->__alloc(), _VSTD::__to_address(__tx.__pos_));
+    const_pointer __new_end = __tx.__new_end_;
+    for (pointer __pos = __tx.__pos_; __pos != __new_end; ++__pos, __tx.__pos_ = __pos) {
+        __alloc_traits::construct(this->__alloc(), _VSTD::__to_address(__pos));
     }
 }
 
@@ -1060,8 +1061,9 @@ void
 vector<_Tp, _Allocator>::__construct_at_end(size_type __n, const_reference __x)
 {
     _ConstructTransaction __tx(*this, __n);
-    for (; __tx.__pos_ != __tx.__new_end_; ++__tx.__pos_) {
-        __alloc_traits::construct(this->__alloc(), _VSTD::__to_address(__tx.__pos_), __x);
+    const_pointer __new_end = __tx.__new_end_;
+    for (pointer __pos = __tx.__pos_; __pos != __new_end; ++__pos, __tx.__pos_ = __pos) {
+        __alloc_traits::construct(this->__alloc(), _VSTD::__to_address(__pos), __x);
     }
 }
 
@@ -1753,9 +1755,10 @@ vector<_Tp, _Allocator>::__move_range(pointer __from_s, pointer __from_e, pointe
     {
       pointer __i = __from_s + __n;
       _ConstructTransaction __tx(*this, __from_e - __i);
-      for (; __i < __from_e; ++__i, ++__tx.__pos_) {
+      for (pointer __pos = __tx.__pos_; __i < __from_e;
+           ++__i, ++__pos, __tx.__pos_ = __pos) {
           __alloc_traits::construct(this->__alloc(),
-                                    _VSTD::__to_address(__tx.__pos_),
+                                    _VSTD::__to_address(__pos),
                                     _VSTD::move(*__i));
       }
     }