[llvm] [AMDGPU] Run early-cse<memssa> at the end of the full-LTO pipeline (PR #208461)
via llvm-commits
llvm-commits at lists.llvm.org
Thu Jul 9 07:11:02 PDT 2026
https://github.com/michaelselehov updated https://github.com/llvm/llvm-project/pull/208461
>From d9c1cffe6e1bbf89aad944bb753b137460e4bd5e Mon Sep 17 00:00:00 2001
From: mselehov <michael.selehov at amd.com>
Date: Thu, 9 Jul 2026 08:22:55 -0500
Subject: [PATCH 1/2] [AMDGPU] Run early-cse<memssa> at the end of the full-LTO
pipeline
The regular (non-LTO) and ThinLTO function simplification pipelines run an
EarlyCSE-with-MemorySSA pass near the start of the function pass sequence, but
the full-LTO postlink pipeline does not. Without it, a redundant load/store
round-trip can survive all the way to codegen; the later GVN/DSE in the LTO
pipeline do not catch this particular pattern. On AMDGPU this leaves extra
hazard s_nop padding in some kernels (observed ~6% throughput loss on a rocRAND
log-normal-double kernel).
Rather than touch the target-independent LTO pipeline, register
early-cse<memssa> through the AMDGPU FullLinkTimeOptimizationLast extension
point, so it runs at the end of the full-LTO middle-end (before codegen) only
for AMDGPU.
Assisted-by: Claude Opus
---
llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp | 11 +++++++++++
llvm/test/CodeGen/AMDGPU/print-pipeline-passes.ll | 2 ++
.../PhaseOrdering/AMDGPU/infer-address-space.ll | 4 +---
3 files changed, 14 insertions(+), 3 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
index 4835fad9f7897..67395966b0411 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
@@ -1111,6 +1111,17 @@ void AMDGPUTargetMachine::registerPassBuilderCallbacks(PassBuilder &PB) {
PB.registerFullLinkTimeOptimizationLastEPCallback(
[this](ModulePassManager &PM, OptimizationLevel Level) {
+ // The regular (non-LTO) and ThinLTO function simplification pipelines
+ // run early-cse with MemorySSA near the start of the function pass
+ // sequence, but the full-LTO postlink pipeline does not. Without it a
+ // redundant load/store round-trip can survive to codegen (the later
+ // GVN/DSE do not catch this pattern), which costs extra hazard s_nop
+ // padding in some kernels. Run it here, at the end of the full-LTO
+ // middle-end, before codegen.
+ if (Level != OptimizationLevel::O0)
+ PM.addPass(createModuleToFunctionPassAdaptor(
+ EarlyCSEPass(/*UseMemorySSA=*/true)));
+
// When we are using -fgpu-rdc, we can only run accelerator code
// selection after linking to prevent, otherwise we end up removing
// potentially reachable symbols that were exported as external in other
diff --git a/llvm/test/CodeGen/AMDGPU/print-pipeline-passes.ll b/llvm/test/CodeGen/AMDGPU/print-pipeline-passes.ll
index b1fc76f457ece..54d3428828add 100644
--- a/llvm/test/CodeGen/AMDGPU/print-pipeline-passes.ll
+++ b/llvm/test/CodeGen/AMDGPU/print-pipeline-passes.ll
@@ -9,8 +9,10 @@
; RUN: opt -mtriple=amdgcn--amdhsa -S -passes="lto-pre-link<O3>" -print-pipeline-passes -amdgpu-internalize-symbols %s -o - | FileCheck --check-prefix=PRE %s
+; CHECK: early-cse<memssa>
; CHECK: amdgpu-attributor
; O0-NOT: amdgpu-attributor
+; O0-NOT: early-cse<memssa>
; PRE-NOT: internalize
; PRE-NOT: amdgpu-attributor
diff --git a/llvm/test/Transforms/PhaseOrdering/AMDGPU/infer-address-space.ll b/llvm/test/Transforms/PhaseOrdering/AMDGPU/infer-address-space.ll
index 38ad7ed28dee4..df7ccb95ba0c8 100644
--- a/llvm/test/Transforms/PhaseOrdering/AMDGPU/infer-address-space.ll
+++ b/llvm/test/Transforms/PhaseOrdering/AMDGPU/infer-address-space.ll
@@ -42,9 +42,7 @@ define void @caller(ptr addrspace(1) %ptr_as1, i32 %value) {
; NO-INFER-NEXT: store i32 [[VALUE]], ptr addrspace(5) [[VAL_FIELD]], align 4
; NO-INFER-NEXT: [[GENERIC_INPUT:%.*]] = addrspacecast ptr addrspace(1) [[PTR_AS1]] to ptr
; NO-INFER-NEXT: store ptr [[GENERIC_INPUT]], ptr addrspace(5) [[DATA]], align 8
-; NO-INFER-NEXT: [[RETRIEVED_PTR:%.*]] = load ptr, ptr addrspace(5) [[DATA]], align 8
-; NO-INFER-NEXT: [[RETRIEVED_VAL:%.*]] = load i32, ptr addrspace(5) [[VAL_FIELD]], align 4
-; NO-INFER-NEXT: call void @callee(ptr [[RETRIEVED_PTR]], i32 [[RETRIEVED_VAL]])
+; NO-INFER-NEXT: call void @callee(ptr [[GENERIC_INPUT]], i32 [[VALUE]])
; NO-INFER-NEXT: ret void
;
%data = alloca %struct.data, align 8, addrspace(5)
>From 4545512a88885468ab3cb410055cabc9e16b051c Mon Sep 17 00:00:00 2001
From: mselehov <michael.selehov at amd.com>
Date: Thu, 9 Jul 2026 09:10:07 -0500
Subject: [PATCH 2/2] [AMDGPU] Shorten the early-cse<memssa> comment
---
llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp | 9 ++-------
1 file changed, 2 insertions(+), 7 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
index 67395966b0411..9e44720d546fa 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp
@@ -1111,13 +1111,8 @@ void AMDGPUTargetMachine::registerPassBuilderCallbacks(PassBuilder &PB) {
PB.registerFullLinkTimeOptimizationLastEPCallback(
[this](ModulePassManager &PM, OptimizationLevel Level) {
- // The regular (non-LTO) and ThinLTO function simplification pipelines
- // run early-cse with MemorySSA near the start of the function pass
- // sequence, but the full-LTO postlink pipeline does not. Without it a
- // redundant load/store round-trip can survive to codegen (the later
- // GVN/DSE do not catch this pattern), which costs extra hazard s_nop
- // padding in some kernels. Run it here, at the end of the full-LTO
- // middle-end, before codegen.
+ // Clean up redundant memory round-trips that the full-LTO pipeline,
+ // unlike the non-LTO/ThinLTO ones, otherwise leaves for codegen.
if (Level != OptimizationLevel::O0)
PM.addPass(createModuleToFunctionPassAdaptor(
EarlyCSEPass(/*UseMemorySSA=*/true)));
More information about the llvm-commits
mailing list