[llvm] [LSV] Enhance LoadStoreVectorizer to Handle Disjoint Flag in OR Instructions and Restore Vectorization Opportunities (PR #96495)
via llvm-commits
llvm-commits at lists.llvm.org
Mon Jun 24 07:45:53 PDT 2024
llvmbot wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-llvm-transforms
Author: Hao Li (LiHao217)
<details>
<summary>Changes</summary>
### Summary
This PR proposes an enhancement to the LoadStoreVectorizer (LSV) pass in LLVM to handle the recent changes in how LLVM treats or instructions with the disjoint flag. This change aims to restore vectorization opportunities that were lost due to a recent patch that introduced the disjoint flag for or instructions.
### Background
In the previous implementation, haveNoCommonBitsSet() was used to detect and convert certain or instructions to add instructions, a transformation that LoadStoreVectorizer relies on for vectorization. A recent patch changed this behavior by introducing a disjoint flag, which serves the same purpose but is more reliable and easier to revert.
However, this change resulted in LoadStoreVectorizer no longer recognizing these transformed or instructions, causing certain vectorization opportunities to be missed.
### Problem
Consider the following IR:
```
%linear_index3 = or i32 %linear_index_plus_base.fr, 3
%linear_index2 = or i32 %linear_index_plus_base.fr, 2
%linear_index1 = or i32 %linear_index_plus_base.fr, 1
```
In LLVM versions prior to patch [74467](https://github.com/llvm/llvm-project/pull/74467), LLVM used the haveNoCommonBitsSet() function to determine whether these or operations had no common bits. If they had no common bits, these or operations were converted to add operations, thus allowing the LoadStoreVectorizer (LSV) to vectorize them.
In LLVM versions after patch https://github.com/llvm/llvm-project/pull/74467, a new disjoint flag was introduced to replace the haveNoCommonBitsSet() function. However, in this new implementation, the or instructions were not recognized as disjoint and thus were not transformed into add operations. This led to a failure in the LSV to vectorize these instructions.
Specifically:
1.The previous implementation used haveNoCommonBitsSet() to detect and convert specific or instructions to add instructions. LSV could then recognize and vectorize these add instructions.
2.In the new implementation, or instructions were not correctly identified as disjoint, and thus were not converted to add instructions. This caused LSV to miss the opportunity to vectorize them.
To solve this problem, we need to identify if the or instructions in the IR are disjoint. If they are, the IR should be transformed to the following form to enable vectorization:
```
%linear_index3 = or disjoint i32 %linear_index_plus_base.fr, 3
%linear_index2 = or disjoint i32 %linear_index_plus_base.fr, 2
%linear_index1 = or disjoint i32 %linear_index_plus_base.fr, 1
```
This modification will allow LSV to recognize these instructions and restore vectorization opportunities.
### Solution
To resolve this, we propose updating the LoadStoreVectorizer pass to check for or instructions and apply the disjoint flag based on the haveNoCommonBitsSet() function if applicable. This will ensure that vectorization opportunities are restored.
### Implementation
In the Vectorizer::run function, we iterate over the basic blocks and check for or instructions. If an or instruction has no common bits set between its operands, we set the disjoint flag. This preprocessing step makes sure that LSV recognizes the pattern and can perform the necessary vectorizations.
```
bool Vectorizer::run() {
bool Changed = false;
for (BasicBlock *BB : post_order(&F)) {
assert(!BB->empty());
SmallVector<BasicBlock::iterator, 8> Barriers;
Barriers.push_back(BB->begin());
for (Instruction &I : *BB) {
if (auto *OrInst = dyn_cast<PossiblyDisjointInst>(&I)) {
Value *Op0 = OrInst->getOperand(0);
Value *Op1 = OrInst->getOperand(1);
if (haveNoCommonBitsSet(Op0, Op1, DL)) {
OrInst->setIsDisjoint(true);
}
}
if (!isGuaranteedToTransferExecutionToSuccessor(&I))
Barriers.push_back(I.getIterator());
}
Barriers.push_back(BB->end());
for (auto It = Barriers.begin(), End = std::prev(Barriers.end()); It != End;
++It)
Changed |= runOnPseudoBB(*It, *std::next(It));
for (Instruction *I : ToErase) {
auto *PtrOperand = getLoadStorePointerOperand(I);
if (I->use_empty())
I->eraseFromParent();
RecursivelyDeleteTriviallyDeadInstructions(PtrOperand);
}
ToErase.clear();
}
return Changed;
}
```
### Testing and Validation
The modified implementation has been tested with the existing suite of vectorization benchmarks and specific cases that regressed due to the introduction of the disjoint flag. The results showed that vectorization opportunities originally intended to be captured by LSV were successfully restored.
### Discussion
This enhancement is expected to bring back performance gains lost with the new disjoint flag handling. It is a minor change but will have a significant impact on GPU-targeted vectorization especially for AMD and NVIDIA architectures.
The community’s feedback on this approach is highly appreciated. We would particularly like to know if there are any potential edge cases or alternate methods to handle similar scenarios in the future.
---
Patch is 67.33 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/96495.diff
3 Files Affected:
- (modified) llvm/lib/Transforms/Vectorize/LoadStoreVectorizer.cpp (+10-1)
- (modified) llvm/test/Transforms/LoadStoreVectorizer/AMDGPU/adjust-alloca-alignment.ll (+4-4)
- (added) llvm/test/Transforms/LoadStoreVectorizer/NVPTX/disjoint_or_vectorizer.ll (+840)
``````````diff
diff --git a/llvm/lib/Transforms/Vectorize/LoadStoreVectorizer.cpp b/llvm/lib/Transforms/Vectorize/LoadStoreVectorizer.cpp
index 6ba278794b7fb..814b1d55e3a0b 100644
--- a/llvm/lib/Transforms/Vectorize/LoadStoreVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoadStoreVectorizer.cpp
@@ -437,9 +437,18 @@ bool Vectorizer::run() {
SmallVector<BasicBlock::iterator, 8> Barriers;
Barriers.push_back(BB->begin());
- for (Instruction &I : *BB)
+ for (Instruction &I : *BB){
+ if (auto *OrInst = dyn_cast<PossiblyDisjointInst>(&I)) {
+ Value *Op0 = OrInst->getOperand(0);
+ Value *Op1 = OrInst->getOperand(1);
+
+ if (haveNoCommonBitsSet(Op0, Op1, DL)) {
+ OrInst->setIsDisjoint(true);
+ }
+ }
if (!isGuaranteedToTransferExecutionToSuccessor(&I))
Barriers.push_back(I.getIterator());
+ }
Barriers.push_back(BB->end());
for (auto It = Barriers.begin(), End = std::prev(Barriers.end()); It != End;
diff --git a/llvm/test/Transforms/LoadStoreVectorizer/AMDGPU/adjust-alloca-alignment.ll b/llvm/test/Transforms/LoadStoreVectorizer/AMDGPU/adjust-alloca-alignment.ll
index caa53ba0e6674..0daa2737ed7f3 100644
--- a/llvm/test/Transforms/LoadStoreVectorizer/AMDGPU/adjust-alloca-alignment.ll
+++ b/llvm/test/Transforms/LoadStoreVectorizer/AMDGPU/adjust-alloca-alignment.ll
@@ -329,7 +329,7 @@ define i32 @private_load_2xi16_align2_not_alloca(ptr addrspace(5) %p) #0 {
; ALIGNED-NEXT: [[ZEXT_0:%.*]] = zext i16 [[P_0]] to i32
; ALIGNED-NEXT: [[ZEXT_1:%.*]] = zext i16 [[P_1]] to i32
; ALIGNED-NEXT: [[SHL_1:%.*]] = shl i32 [[ZEXT_1]], 16
-; ALIGNED-NEXT: [[OR:%.*]] = or i32 [[ZEXT_0]], [[SHL_1]]
+; ALIGNED-NEXT: [[OR:%.*]] = or disjoint i32 [[ZEXT_0]], [[SHL_1]]
; ALIGNED-NEXT: ret i32 [[OR]]
;
; UNALIGNED-LABEL: @private_load_2xi16_align2_not_alloca(
@@ -339,7 +339,7 @@ define i32 @private_load_2xi16_align2_not_alloca(ptr addrspace(5) %p) #0 {
; UNALIGNED-NEXT: [[ZEXT_0:%.*]] = zext i16 [[P_01]] to i32
; UNALIGNED-NEXT: [[ZEXT_1:%.*]] = zext i16 [[P_12]] to i32
; UNALIGNED-NEXT: [[SHL_1:%.*]] = shl i32 [[ZEXT_1]], 16
-; UNALIGNED-NEXT: [[OR:%.*]] = or i32 [[ZEXT_0]], [[SHL_1]]
+; UNALIGNED-NEXT: [[OR:%.*]] = or disjoint i32 [[ZEXT_0]], [[SHL_1]]
; UNALIGNED-NEXT: ret i32 [[OR]]
;
%gep.p = getelementptr i16, ptr addrspace(5) %p, i64 1
@@ -360,7 +360,7 @@ define i32 @private_load_2xi16_align1_not_alloca(ptr addrspace(5) %p) #0 {
; ALIGNED-NEXT: [[ZEXT_0:%.*]] = zext i16 [[P_0]] to i32
; ALIGNED-NEXT: [[ZEXT_1:%.*]] = zext i16 [[P_1]] to i32
; ALIGNED-NEXT: [[SHL_1:%.*]] = shl i32 [[ZEXT_1]], 16
-; ALIGNED-NEXT: [[OR:%.*]] = or i32 [[ZEXT_0]], [[SHL_1]]
+; ALIGNED-NEXT: [[OR:%.*]] = or disjoint i32 [[ZEXT_0]], [[SHL_1]]
; ALIGNED-NEXT: ret i32 [[OR]]
;
; UNALIGNED-LABEL: @private_load_2xi16_align1_not_alloca(
@@ -370,7 +370,7 @@ define i32 @private_load_2xi16_align1_not_alloca(ptr addrspace(5) %p) #0 {
; UNALIGNED-NEXT: [[ZEXT_0:%.*]] = zext i16 [[P_01]] to i32
; UNALIGNED-NEXT: [[ZEXT_1:%.*]] = zext i16 [[P_12]] to i32
; UNALIGNED-NEXT: [[SHL_1:%.*]] = shl i32 [[ZEXT_1]], 16
-; UNALIGNED-NEXT: [[OR:%.*]] = or i32 [[ZEXT_0]], [[SHL_1]]
+; UNALIGNED-NEXT: [[OR:%.*]] = or disjoint i32 [[ZEXT_0]], [[SHL_1]]
; UNALIGNED-NEXT: ret i32 [[OR]]
;
%gep.p = getelementptr i16, ptr addrspace(5) %p, i64 1
diff --git a/llvm/test/Transforms/LoadStoreVectorizer/NVPTX/disjoint_or_vectorizer.ll b/llvm/test/Transforms/LoadStoreVectorizer/NVPTX/disjoint_or_vectorizer.ll
new file mode 100644
index 0000000000000..4c45a93fb1ffd
--- /dev/null
+++ b/llvm/test/Transforms/LoadStoreVectorizer/NVPTX/disjoint_or_vectorizer.ll
@@ -0,0 +1,840 @@
+; RUN: opt -mtriple=nvptx64-nvidia-cuda -passes=load-store-vectorizer -S -o - %s | FileCheck %s
+
+; Function Attrs: mustprogress nocallback nofree nosync nounwind speculatable willreturn memory(none)
+declare noundef i32 @llvm.nvvm.read.ptx.sreg.ctaid.x()
+
+; Function Attrs: mustprogress nocallback nofree nosync nounwind speculatable willreturn memory(none)
+declare noundef i32 @llvm.nvvm.read.ptx.sreg.tid.x()
+
+; Function Attrs: nofree nosync nounwind memory(argmem: readwrite)
+define void @copy_fusion_1(i32 %workgroup_id_x, i32 %workitem_id_x, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg0, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg1, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg2, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg3, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg4, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg5, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg6, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg7, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg8, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg9, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg10, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg11, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg12, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg13, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg14, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg15, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg16, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg17, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg18, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg19, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg20, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg21, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg22, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg23, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg24, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg25, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg26, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg27, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg28, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg29, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg30, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg31, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg32, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg33, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg34, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg35, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg36, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg37, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg38, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg39, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg40, ptr noalias nocapture readonly align 16 dereferenceable(23592960) %arg41, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg42, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg43, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg44, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg45, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg46, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg47, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg48, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg49, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg50, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg51, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg52, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg53, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg54, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg55, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg56, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg57, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg58, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg59, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg60, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg61, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg62, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg63, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg64, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg65, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg66, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg67, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg68, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg69, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg70, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg71, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg72, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg73, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg74, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg75, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg76, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg77, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg78, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg79, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg80, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg81, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg82, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg83, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg84, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg85, ptr noalias nocapture writeonly align 128 dereferenceable(23592960) %arg86) local_unnamed_addr #5 {
+entry:
+ %arg86180 = addrspacecast ptr %arg86 to ptr addrspace(1)
+ %arg85178 = addrspacecast ptr %arg85 to ptr addrspace(1)
+ %arg84176 = addrspacecast ptr %arg84 to ptr addrspace(1)
+ %arg83174 = addrspacecast ptr %arg83 to ptr addrspace(1)
+ %arg82172 = addrspacecast ptr %arg82 to ptr addrspace(1)
+ %arg81170 = addrspacecast ptr %arg81 to ptr addrspace(1)
+ %arg80168 = addrspacecast ptr %arg80 to ptr addrspace(1)
+ %arg79166 = addrspacecast ptr %arg79 to ptr addrspace(1)
+ %arg78164 = addrspacecast ptr %arg78 to ptr addrspace(1)
+ %arg77162 = addrspacecast ptr %arg77 to ptr addrspace(1)
+ %arg76160 = addrspacecast ptr %arg76 to ptr addrspace(1)
+ %arg75158 = addrspacecast ptr %arg75 to ptr addrspace(1)
+ %arg74156 = addrspacecast ptr %arg74 to ptr addrspace(1)
+ %arg73154 = addrspacecast ptr %arg73 to ptr addrspace(1)
+ %arg72152 = addrspacecast ptr %arg72 to ptr addrspace(1)
+ %arg71150 = addrspacecast ptr %arg71 to ptr addrspace(1)
+ %arg70148 = addrspacecast ptr %arg70 to ptr addrspace(1)
+ %arg69146 = addrspacecast ptr %arg69 to ptr addrspace(1)
+ %arg68144 = addrspacecast ptr %arg68 to ptr addrspace(1)
+ %arg67142 = addrspacecast ptr %arg67 to ptr addrspace(1)
+ %arg66140 = addrspacecast ptr %arg66 to ptr addrspace(1)
+ %arg65138 = addrspacecast ptr %arg65 to ptr addrspace(1)
+ %arg64136 = addrspacecast ptr %arg64 to ptr addrspace(1)
+ %arg63134 = addrspacecast ptr %arg63 to ptr addrspace(1)
+ %arg62132 = addrspacecast ptr %arg62 to ptr addrspace(1)
+ %arg61130 = addrspacecast ptr %arg61 to ptr addrspace(1)
+ %arg60128 = addrspacecast ptr %arg60 to ptr addrspace(1)
+ %arg59126 = addrspacecast ptr %arg59 to ptr addrspace(1)
+ %arg58124 = addrspacecast ptr %arg58 to ptr addrspace(1)
+ %arg57122 = addrspacecast ptr %arg57 to ptr addrspace(1)
+ %arg56120 = addrspacecast ptr %arg56 to ptr addrspace(1)
+ %arg55118 = addrspacecast ptr %arg55 to ptr addrspace(1)
+ %arg54116 = addrspacecast ptr %arg54 to ptr addrspace(1)
+ %arg53114 = addrspacecast ptr %arg53 to ptr addrspace(1)
+ %arg52112 = addrspacecast ptr %arg52 to ptr addrspace(1)
+ %arg51110 = addrspacecast ptr %arg51 to ptr addrspace(1)
+ %arg50108 = addrspacecast ptr %arg50 to ptr addrspace(1)
+ %arg49106 = addrspacecast ptr %arg49 to ptr addrspace(1)
+ %arg48104 = addrspacecast ptr %arg48 to ptr addrspace(1)
+ %arg47102 = addrspacecast ptr %arg47 to ptr addrspace(1)
+ %arg46100 = addrspacecast ptr %arg46 to ptr addrspace(1)
+ %arg4598 = addrspacecast ptr %arg45 to ptr addrspace(1)
+ %arg4496 = addrspacecast ptr %arg44 to ptr addrspace(1)
+ %arg4394 = addrspacecast ptr %arg43 to ptr addrspace(1)
+ %arg4292 = addrspacecast ptr %arg42 to ptr addrspace(1)
+ %arg4190 = addrspacecast ptr %arg41 to ptr addrspace(1)
+ %arg4088 = addrspacecast ptr %arg40 to ptr addrspace(1)
+ %arg3986 = addrspacecast ptr %arg39 to ptr addrspace(1)
+ %arg3884 = addrspacecast ptr %arg38 to ptr addrspace(1)
+ %arg3782 = addrspacecast ptr %arg37 to ptr addrspace(1)
+ %arg3680 = addrspacecast ptr %arg36 to ptr addrspace(1)
+ %arg3578 = addrspacecast ptr %arg35 to ptr addrspace(1)
+ %arg3476 = addrspacecast ptr %arg34 to ptr addrspace(1)
+ %arg3374 = addrspacecast ptr %arg33 to ptr addrspace(1)
+ %arg3272 = addrspacecast ptr %arg32 to ptr addrspace(1)
+ %arg3170 = addrspacecast ptr %arg31 to ptr addrspace(1)
+ %arg3068 = addrspacecast ptr %arg30 to ptr addrspace(1)
+ %arg2966 = addrspacecast ptr %arg29 to ptr addrspace(1)
+ %arg2864 = addrspacecast ptr %arg28 to ptr addrspace(1)
+ %arg2762 = addrspacecast ptr %arg27 to ptr addrspace(1)
+ %arg2660 = addrspacecast ptr %arg26 to ptr addrspace(1)
+ %arg2558 = addrspacecast ptr %arg25 to ptr addrspace(1)
+ %arg2456 = addrspacecast ptr %arg24 to ptr addrspace(1)
+ %arg2354 = addrspacecast ptr %arg23 to ptr addrspace(1)
+ %arg2252 = addrspacecast ptr %arg22 to ptr addrspace(1)
+ %arg2150 = addrspacecast ptr %arg21 to ptr addrspace(1)
+ %arg2048 = addrspacecast ptr %arg20 to ptr addrspace(1)
+ %arg1946 = addrspacecast ptr %arg19 to ptr addrspace(1)
+ %arg1844 = addrspacecast ptr %arg18 to ptr addrspace(1)
+ %arg1742 = addrspacecast ptr %arg17 to ptr addrspace(1)
+ %arg1640 = addrspacecast ptr %arg16 to ptr addrspace(1)
+ %arg1538 = addrspacecast ptr %arg15 to ptr addrspace(1)
+ %arg1436 = addrspacecast ptr %arg14 to ptr addrspace(1)
+ %arg1334 = addrspacecast ptr %arg13 to ptr addrspace(1)
+ %arg1232 = addrspacecast ptr %arg12 to ptr addrspace(1)
+ %arg1130 = addrspacecast ptr %arg11 to ptr addrspace(1)
+ %arg1028 = addrspacecast ptr %arg10 to ptr addrspace(1)
+ %arg926 = addrspacecast ptr %arg9 to ptr addrspace(1)
+ %arg824 = addrspacecast ptr %arg8 to ptr addrspace(1)
+ %arg722 = addrspacecast ptr %arg7 to ptr addrspace(1)
+ %arg620 = addrspacecast ptr %arg6 to ptr addrspace(1)
+ %arg518 = addrspacecast ptr %arg5 to ptr addrspace(1)
+ %arg416 = addrspacecast ptr %arg4 to ptr addrspace(1)
+ %arg314 = addrspacecast ptr %arg3 to ptr addrspace(1)
+ %arg212 = addrspacecast ptr %arg2 to ptr addrspace(1)
+ %arg110 = addrspacecast ptr %arg1 to ptr addrspace(1)
+ %arg02 = addrspacecast ptr %arg0 to ptr addrspace(1)
+ %0 = tail call i32 @llvm.nvvm.read.ptx.sreg.ctaid.x()
+ %1 = tail call i32 @llvm.nvvm.read.ptx.sreg.tid.x()
+ %2 = shl i32 %0, 9
+ %3 = shl i32 %1, 2
+ %4 = and i32 %3, 60
+ %5 = zext i32 %4 to i64
+ %6 = add i32 %2, %3
+ br label %loop.loop_body
+
+loop.loop_body: ; preds = %entry, %copy_fusion_1.in_bounds-after
+ %lsr.iv = phi i32 [ -884736, %entry ], [ %lsr.iv.next, %copy_fusion_1.in_bounds-after ]
+ %7 = add i32 %6, %lsr.iv
+ %8 = add i32 %7, 884736
+ %linear_index_plus_base.fr = freeze i32 %8
+ %9 = icmp ult i32 %linear_index_plus_base.fr, 11796480
+ br i1 %9, label %copy_fusion_1.in_bounds-true, label %copy_fusion_1.in_bounds-after
+
+copy_fusion_1.in_bounds-after: ; preds = %copy_fusion_1.in_bounds-true, %loop.loop_body
+ %lsr.iv.next = add nsw i32 %lsr.iv, 884736
+ %10 = icmp ugt i32 %lsr.iv.next, 10911743
+ br i1 %10, label %loop.loop_exit, label %loop.loop_body
+
+loop.loop_exit: ; preds = %copy_fusion_1.in_bounds-after
+ ret void
+
+copy_fusion_1.in_bounds-true: ; preds = %loop.loop_body
+ %linear_index3 = or i32 %linear_index_plus_base.fr, 3
+ %11 = and i32 %linear_index3, 63
+ %linear_index2 = or i32 %linear_index_plus_base.fr, 2
+ %12 = and i32 %linear_index2, 62
+ %linear_index1 = or i32 %linear_index_plus_base.fr, 1
+ %13 = and i32 %linear_index1, 61
+ %14 = lshr i32 %linear_index_plus_base.fr, 6
+ %15 = zext i32 %14 to i64
+ %16 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg02, i64 0, i64 0, i64 %15, i64 %5
+ %17 = load i16, ptr addrspace(1) %16, align 8
+ %18 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg110, i64 0, i64 0, i64 %15, i64 %5
+ %19 = load i16, ptr addrspace(1) %18, align 8
+ %20 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg212, i64 0, i64 0, i64 %15, i64 %5
+ %21 = load i16, ptr addrspace(1) %20, align 8
+ %22 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg314, i64 0, i64 0, i64 %15, i64 %5
+ %23 = load i16, ptr addrspace(1) %22, align 8
+ %24 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg416, i64 0, i64 0, i64 %15, i64 %5
+ %25 = load i16, ptr addrspace(1) %24, align 8
+ %26 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg518, i64 0, i64 0, i64 %15, i64 %5
+ %27 = load i16, ptr addrspace(1) %26, align 8
+ %28 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg620, i64 0, i64 0, i64 %15, i64 %5
+ %29 = load i16, ptr addrspace(1) %28, align 8
+ %30 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg722, i64 0, i64 0, i64 %15, i64 %5
+ %31 = load i16, ptr addrspace(1) %30, align 8
+ %32 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg824, i64 0, i64 0, i64 %15, i64 %5
+ %33 = load i16, ptr addrspace(1) %32, align 8
+ %34 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg926, i64 0, i64 0, i64 %15, i64 %5
+ %35 = load i16, ptr addrspace(1) %34, align 8
+ %36 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg1028, i64 0, i64 0, i64 %15, i64 %5
+ %37 = load i16, ptr addrspace(1) %36, align 8
+ %38 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg1130, i64 0, i64 0, i64 %15, i64 %5
+ %39 = load i16, ptr addrspace(1) %38, align 8
+ %40 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg1232, i64 0, i64 0, i64 %15, i64 %5
+ %41 = load i16, ptr addrspace(1) %40, align 8
+ %42 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg1334, i64 0, i64 0, i64 %15, i64 %5
+ %43 = load i16, ptr addrspace(1) %42, align 8
+ %44 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg1436, i64 0, i64 0, i64 %15, i64 %5
+ %45 = load i16, ptr addrspace(1) %44, align 8
+ %46 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg1538, i64 0, i64 0, i64 %15, i64 %5
+ %47 = load i16, ptr addrspace(1) %46, align 8
+ %48 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg1640, i64 0, i64 0, i64 %15, i64 %5
+ %49 = load i16, ptr addrspace(1) %48, align 8
+ %50 = getelementptr inbounds [1 x [184320 x [64 x i16]]], ptr addrspace(1) %arg1742, i64 0, i64 0, i64 %15, i64 %5
+ %51 = load i16, ptr addrspace(1) %50, ali...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/96495
More information about the llvm-commits
mailing list