[Mlir-commits] [mlir] [mlir][gpu] Split target-specific passes out of MLIRGPUTransforms (PR #234903)
Markus Böck
llvmlistbot at llvm.org
Fri Oct 23 03:30:53 PDT 2026
https://github.com/zero9178 created https://github.com/llvm/llvm-project/pull/234903
MLIRGPUTransforms linked every GPU target: the AMDGPU, LLVM, ROCDL and XeVM dialects, plus MLIRNVVMTarget, MLIRROCDLTarget, MLIRSPIRVTarget and MLIRExecutionEngineUtils. So any user of any target-specific or target-independent GPU pass also pulled in every target's dialect and serialization library.
Move the target-specific passes and patterns into one library per target. Each has its own Passes.td and Passes.h and links only its own target dialect.
MLIRGPUTransforms no longer has any target dependencies at all.
>From 6af35e363e350b36d5c33d3f4ac4557b302d3f70 Mon Sep 17 00:00:00 2001
From: Markus Boeck <mboeck at nvidia.com>
Date: Fri, 23 Oct 2026 10:22:52 +0000
Subject: [PATCH] [mlir][gpu] Split target-specific passes out of
MLIRGPUTransforms
MLIRGPUTransforms linked every GPU target: the AMDGPU, LLVM, ROCDL and
XeVM dialects, plus MLIRNVVMTarget, MLIRROCDLTarget, MLIRSPIRVTarget
and MLIRExecutionEngineUtils. So any user of any target-specific or
target-independent GPU pass also pulled in every target's dialect
and serialization library.
Move the target-specific passes and patterns into one library per
target. Each has its own Passes.td and Passes.h and links only its
own target dialect.
MLIRGPUTransforms no longer has any target dependencies at all.
Co-authored-by: Claude <noreply at anthropic.com>
---
mlir/docs/Passes.md | 16 ++
mlir/docs/ReleaseNotes.md | 18 ++
mlir/include/mlir-c/Dialect/GPU.h | 4 +
.../Dialect/GPU/Transforms/CMakeLists.txt | 5 +
.../GPU/Transforms/NVVM/CMakeLists.txt | 7 +
.../mlir/Dialect/GPU/Transforms/NVVM/Passes.h | 30 +++
.../Dialect/GPU/Transforms/NVVM/Passes.td | 68 +++++
.../mlir/Dialect/GPU/Transforms/Passes.h | 19 --
.../mlir/Dialect/GPU/Transforms/Passes.td | 201 --------------
.../GPU/Transforms/ROCDL/CMakeLists.txt | 7 +
.../Dialect/GPU/Transforms/ROCDL/Passes.h | 50 ++++
.../Dialect/GPU/Transforms/ROCDL/Passes.td | 83 ++++++
.../GPU/Transforms/SPIRV/CMakeLists.txt | 7 +
.../Dialect/GPU/Transforms/SPIRV/Passes.h | 30 +++
.../Dialect/GPU/Transforms/SPIRV/Passes.td | 58 ++++
.../GPU/Transforms/XeVM/CMakeLists.txt | 7 +
.../mlir/Dialect/GPU/Transforms/XeVM/Passes.h | 30 +++
.../Dialect/GPU/Transforms/XeVM/Passes.td | 49 ++++
.../include/mlir/Dialect/GPU/Utils/GPUUtils.h | 13 +
mlir/lib/Bindings/Python/GPUPasses.cpp | 4 +
mlir/lib/CAPI/Dialect/CMakeLists.txt | 8 +
mlir/lib/CAPI/Dialect/GPUPasses.cpp | 12 +
mlir/lib/Conversion/GPUToROCDL/CMakeLists.txt | 1 +
.../GPUToROCDL/LowerGpuOpsToROCDLOps.cpp | 1 +
mlir/lib/Dialect/GPU/CMakeLists.txt | 85 +++++-
mlir/lib/Dialect/GPU/Pipelines/CMakeLists.txt | 4 +
.../GPU/Pipelines/GPUToNVVMPipeline.cpp | 1 +
.../GPU/Pipelines/GPUToROCDLPipeline.cpp | 1 +
.../GPU/Pipelines/GPUToXeVMPipeline.cpp | 1 +
.../Dialect/GPU/TransformOps/CMakeLists.txt | 1 +
.../GPU/TransformOps/GPUTransformOps.cpp | 1 +
.../{ => NVVM}/NVVMAttachTarget.cpp | 5 +-
.../{ => ROCDL}/PromoteShuffleToAMDGPU.cpp | 2 +-
.../{ => ROCDL}/ROCDLAttachTarget.cpp | 5 +-
.../Transforms/ROCDL/SubgroupReduceToDPP.cpp | 238 +++++++++++++++++
.../{ => SPIRV}/SPIRVAttachTarget.cpp | 5 +-
.../GPU/Transforms/SubgroupReduceLowering.cpp | 248 +-----------------
.../{ => XeVM}/XeVMAttachTarget.cpp | 5 +-
mlir/lib/Dialect/GPU/Utils/Utils.cpp | 25 ++
.../SparseTensor/Pipelines/CMakeLists.txt | 1 +
.../Pipelines/SparseTensorPipelines.cpp | 1 +
mlir/lib/RegisterAllPasses.cpp | 8 +
mlir/test/lib/Dialect/GPU/CMakeLists.txt | 1 +
mlir/test/lib/Dialect/GPU/TestGpuRewrite.cpp | 1 +
mlir/test/lib/Pass/CMakeLists.txt | 1 +
.../lib/Pass/TestVulkanRunnerPipeline.cpp | 1 +
46 files changed, 879 insertions(+), 490 deletions(-)
create mode 100644 mlir/include/mlir/Dialect/GPU/Transforms/NVVM/CMakeLists.txt
create mode 100644 mlir/include/mlir/Dialect/GPU/Transforms/NVVM/Passes.h
create mode 100644 mlir/include/mlir/Dialect/GPU/Transforms/NVVM/Passes.td
create mode 100644 mlir/include/mlir/Dialect/GPU/Transforms/ROCDL/CMakeLists.txt
create mode 100644 mlir/include/mlir/Dialect/GPU/Transforms/ROCDL/Passes.h
create mode 100644 mlir/include/mlir/Dialect/GPU/Transforms/ROCDL/Passes.td
create mode 100644 mlir/include/mlir/Dialect/GPU/Transforms/SPIRV/CMakeLists.txt
create mode 100644 mlir/include/mlir/Dialect/GPU/Transforms/SPIRV/Passes.h
create mode 100644 mlir/include/mlir/Dialect/GPU/Transforms/SPIRV/Passes.td
create mode 100644 mlir/include/mlir/Dialect/GPU/Transforms/XeVM/CMakeLists.txt
create mode 100644 mlir/include/mlir/Dialect/GPU/Transforms/XeVM/Passes.h
create mode 100644 mlir/include/mlir/Dialect/GPU/Transforms/XeVM/Passes.td
rename mlir/lib/Dialect/GPU/Transforms/{ => NVVM}/NVVMAttachTarget.cpp (96%)
rename mlir/lib/Dialect/GPU/Transforms/{ => ROCDL}/PromoteShuffleToAMDGPU.cpp (98%)
rename mlir/lib/Dialect/GPU/Transforms/{ => ROCDL}/ROCDLAttachTarget.cpp (97%)
create mode 100644 mlir/lib/Dialect/GPU/Transforms/ROCDL/SubgroupReduceToDPP.cpp
rename mlir/lib/Dialect/GPU/Transforms/{ => SPIRV}/SPIRVAttachTarget.cpp (96%)
rename mlir/lib/Dialect/GPU/Transforms/{ => XeVM}/XeVMAttachTarget.cpp (95%)
diff --git a/mlir/docs/Passes.md b/mlir/docs/Passes.md
index 9e70da8a3ec978..7ef03285a7ef6f 100644
--- a/mlir/docs/Passes.md
+++ b/mlir/docs/Passes.md
@@ -56,6 +56,22 @@ This document describes the available MLIR passes and their contracts.
[include "GPUPasses.md"]
+### NVVM-specific 'gpu' Dialect Passes
+
+[include "GPUNVVMPasses.md"]
+
+### ROCDL-specific 'gpu' Dialect Passes
+
+[include "GPUROCDLPasses.md"]
+
+### SPIR-V-specific 'gpu' Dialect Passes
+
+[include "GPUSPIRVPasses.md"]
+
+### XeVM-specific 'gpu' Dialect Passes
+
+[include "GPUXeVMPasses.md"]
+
## 'linalg' Dialect Passes
[include "LinalgPasses.md"]
diff --git a/mlir/docs/ReleaseNotes.md b/mlir/docs/ReleaseNotes.md
index 24a4a5f1b6ddd4..f57642308f4528 100644
--- a/mlir/docs/ReleaseNotes.md
+++ b/mlir/docs/ReleaseNotes.md
@@ -55,6 +55,24 @@ default rules, `attr-dict` contains only discardable attributes. See
manually setting a triple/chipset/features combination is a low-level
operation that should require a manual `rocdl-attach-target` call.
+### Target-Independent GPU Transforms
+
+`MLIRGPUTransforms` no longer links any target dialect or target library. The
+target-specific GPU passes and patterns moved into per-target libraries with
+their own pass headers; pass names and options are unchanged:
+
+| Library | Header | Contents |
+| --- | --- | --- |
+| `MLIRGPUNVVMTransforms` | `mlir/Dialect/GPU/Transforms/NVVM/Passes.h` | `nvvm-attach-target` |
+| `MLIRGPUROCDLTransforms` | `mlir/Dialect/GPU/Transforms/ROCDL/Passes.h` | `rocdl-attach-target`, `populateGpuPromoteShuffleToAMDGPUPatterns`, `populateGpuLowerSubgroupReduceToDPPPatterns`, `populateGpuLowerClusteredSubgroupReduceToDPPPatterns` |
+| `MLIRGPUSPIRVTransforms` | `mlir/Dialect/GPU/Transforms/SPIRV/Passes.h` | `spirv-attach-target` |
+| `MLIRGPUXeVMTransforms` | `mlir/Dialect/GPU/Transforms/XeVM/Passes.h` | `xevm-attach-target` |
+
+Each header provides its own registration function (for example
+`registerGPUNVVMPasses()`). The C API functions for these passes are renamed
+to match, e.g. `mlirCreateGPUGpuNVVMAttachTarget` is now
+`mlirCreateGPUNVVMGpuNVVMAttachTarget`.
+
## LLVM 21
### GPU/NVVM Changes
diff --git a/mlir/include/mlir-c/Dialect/GPU.h b/mlir/include/mlir-c/Dialect/GPU.h
index f87ac9924d355f..bc8d1dfc72f701 100644
--- a/mlir/include/mlir-c/Dialect/GPU.h
+++ b/mlir/include/mlir-c/Dialect/GPU.h
@@ -72,5 +72,9 @@ mlirGPUObjectAttrGetKernels(MlirAttribute mlirObjectAttr);
#endif
#include "mlir/Dialect/GPU/Transforms/Passes.capi.h.inc"
+#include "mlir/Dialect/GPU/Transforms/NVVM/Passes.capi.h.inc"
+#include "mlir/Dialect/GPU/Transforms/ROCDL/Passes.capi.h.inc"
+#include "mlir/Dialect/GPU/Transforms/SPIRV/Passes.capi.h.inc"
+#include "mlir/Dialect/GPU/Transforms/XeVM/Passes.capi.h.inc"
#endif // MLIR_C_DIALECT_GPU_H
diff --git a/mlir/include/mlir/Dialect/GPU/Transforms/CMakeLists.txt b/mlir/include/mlir/Dialect/GPU/Transforms/CMakeLists.txt
index 7cc5eb0bf9bc16..006ca6eea83475 100644
--- a/mlir/include/mlir/Dialect/GPU/Transforms/CMakeLists.txt
+++ b/mlir/include/mlir/Dialect/GPU/Transforms/CMakeLists.txt
@@ -5,3 +5,8 @@ mlir_tablegen(Passes.capi.cpp.inc -gen-pass-capi-impl --prefix GPU)
add_mlir_dialect_tablegen_target(MLIRGPUPassIncGen)
add_mlir_doc(Passes GPUPasses ./ -gen-pass-doc)
+
+add_subdirectory(NVVM)
+add_subdirectory(ROCDL)
+add_subdirectory(SPIRV)
+add_subdirectory(XeVM)
diff --git a/mlir/include/mlir/Dialect/GPU/Transforms/NVVM/CMakeLists.txt b/mlir/include/mlir/Dialect/GPU/Transforms/NVVM/CMakeLists.txt
new file mode 100644
index 00000000000000..4053e16db43811
--- /dev/null
+++ b/mlir/include/mlir/Dialect/GPU/Transforms/NVVM/CMakeLists.txt
@@ -0,0 +1,7 @@
+set(LLVM_TARGET_DEFINITIONS Passes.td)
+mlir_tablegen(Passes.h.inc -gen-pass-decls -name GPUNVVM)
+mlir_tablegen(Passes.capi.h.inc -gen-pass-capi-header --prefix GPUNVVM)
+mlir_tablegen(Passes.capi.cpp.inc -gen-pass-capi-impl --prefix GPUNVVM)
+add_mlir_dialect_tablegen_target(MLIRGPUNVVMPassIncGen)
+
+add_mlir_doc(Passes GPUNVVMPasses ./ -gen-pass-doc)
diff --git a/mlir/include/mlir/Dialect/GPU/Transforms/NVVM/Passes.h b/mlir/include/mlir/Dialect/GPU/Transforms/NVVM/Passes.h
new file mode 100644
index 00000000000000..0c79b7b03191fb
--- /dev/null
+++ b/mlir/include/mlir/Dialect/GPU/Transforms/NVVM/Passes.h
@@ -0,0 +1,30 @@
+//===- Passes.h - GPU NVVM pass entrypoints ---------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This header file defines prototypes of the NVVM-specific GPU passes and
+// patterns.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_GPU_TRANSFORMS_NVVM_PASSES_H_
+#define MLIR_DIALECT_GPU_TRANSFORMS_NVVM_PASSES_H_
+
+#include "mlir/Pass/Pass.h"
+
+namespace mlir {
+
+#define GEN_PASS_DECL
+#include "mlir/Dialect/GPU/Transforms/NVVM/Passes.h.inc"
+
+/// Generate the code for registering passes.
+#define GEN_PASS_REGISTRATION
+#include "mlir/Dialect/GPU/Transforms/NVVM/Passes.h.inc"
+
+} // namespace mlir
+
+#endif // MLIR_DIALECT_GPU_TRANSFORMS_NVVM_PASSES_H_
diff --git a/mlir/include/mlir/Dialect/GPU/Transforms/NVVM/Passes.td b/mlir/include/mlir/Dialect/GPU/Transforms/NVVM/Passes.td
new file mode 100644
index 00000000000000..3e685ae098d995
--- /dev/null
+++ b/mlir/include/mlir/Dialect/GPU/Transforms/NVVM/Passes.td
@@ -0,0 +1,68 @@
+//===-- Passes.td - GPU NVVM pass definition file ----------*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_GPU_TRANSFORMS_NVVM_PASSES
+#define MLIR_DIALECT_GPU_TRANSFORMS_NVVM_PASSES
+
+include "mlir/Pass/PassBase.td"
+
+def GpuNVVMAttachTarget: Pass<"nvvm-attach-target", ""> {
+ let summary = "Attaches an NVVM target attribute to a GPU Module.";
+ let description = [{
+ This pass searches for all GPU Modules in the immediate regions and attaches
+ an NVVM target if the module matches the name specified by the `module` argument.
+
+ Example:
+ ```
+ // File: in.mlir:
+ gpu.module @nvvm_module_1 {...}
+ gpu.module @nvvm_module_2 {...}
+ gpu.module @rocdl_module_1 {...}
+ // mlir-opt --nvvm-attach-target="module=nvvm.* chip=sm_90" in.mlir
+ gpu.module @nvvm_module_1 [#nvvm.target<chip = "sm_90">] {...}
+ gpu.module @nvvm_module_2 [#nvvm.target<chip = "sm_90">] {...}
+ gpu.module @rocdl_module_1 {...}
+ ```
+ }];
+ let options = [
+ Option<"moduleMatcher", "module", "std::string",
+ /*default=*/ [{""}],
+ "Regex used to identify the modules to attach the target to.">,
+ Option<"triple", "triple", "std::string",
+ /*default=*/ "\"nvptx64-nvidia-cuda\"",
+ "Target triple.">,
+ Option<"chip", "chip", "std::string",
+ /*default=*/"\"sm_75\"",
+ "Target chip.">,
+ Option<"features", "features", "std::string",
+ /*default=*/"\"\"",
+ "Target features.">,
+ Option<"optLevel", "O", "unsigned",
+ /*default=*/"2",
+ "Optimization level.">,
+ Option<"fastFlag", "fast", "bool",
+ /*default=*/"false",
+ "Enable fast math mode.">,
+ Option<"ftzFlag", "ftz", "bool",
+ /*default=*/"false",
+ "Enable flush to zero for denormals.">,
+ Option<"compilerDiagnosticsFlag", "collect-compiler-diagnostics", "bool",
+ /*default=*/"false",
+ "Enable collection of compiler diagnostics.">,
+ ListOption<"linkLibs", "l", "std::string",
+ "Extra bitcode libraries paths to link to.">,
+ Option<"cmdOptions", "ptxas-cmd-options", "std::string",
+ /*default=*/ [{""}],
+ "Command line options passed to downstream compiler">,
+ Option<"verifyTarget", "verify-target-arch", "bool",
+ /*default=*/"true",
+ "Enable verification of the target architecture">,
+ ];
+}
+
+#endif // MLIR_DIALECT_GPU_TRANSFORMS_NVVM_PASSES
diff --git a/mlir/include/mlir/Dialect/GPU/Transforms/Passes.h b/mlir/include/mlir/Dialect/GPU/Transforms/Passes.h
index 6657ff819dea5c..509d096d765a04 100644
--- a/mlir/include/mlir/Dialect/GPU/Transforms/Passes.h
+++ b/mlir/include/mlir/Dialect/GPU/Transforms/Passes.h
@@ -15,10 +15,8 @@
#include "mlir/Dialect/GPU/IR/GPUDialect.h"
#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
-#include "mlir/Dialect/LLVMIR/ROCDLTargetInfo.h"
#include "mlir/IR/PatternMatch.h"
#include "mlir/Pass/Pass.h"
-#include <optional>
namespace llvm {
class TargetMachine;
@@ -73,19 +71,6 @@ void populateGpuLowerClusteredSubgroupReduceToShufflePatterns(
RewritePatternSet &patterns, unsigned subgroupSize,
unsigned shuffleBitwidth = 32, PatternBenefit benefit = 1);
-/// Collect a set of patterns to lower `gpu.subgroup_reduce` into `amdgpu.dpp`
-/// ops over scalar types. Assumes that the subgroup has
-/// `subgroupSize` lanes. Applicable only to AMD GPUs.
-void populateGpuLowerSubgroupReduceToDPPPatterns(
- RewritePatternSet &patterns, unsigned subgroupSize,
- const ROCDL::TargetInfo &target, PatternBenefit benefit = 1);
-
-/// Disjoint counterpart of `populateGpuLowerSubgroupReduceToDPPPatterns`
-/// that only matches `gpu.subgroup_reduce` ops with a `cluster_size`.
-void populateGpuLowerClusteredSubgroupReduceToDPPPatterns(
- RewritePatternSet &patterns, unsigned subgroupSize,
- const ROCDL::TargetInfo &target, PatternBenefit benefit = 1);
-
/// Collect all patterns to rewrite ops within the GPU dialect.
inline void populateGpuRewritePatterns(RewritePatternSet &patterns) {
populateGpuAllReducePatterns(patterns);
@@ -112,10 +97,6 @@ void populateGpuDecomposeMemrefsPatterns(RewritePatternSet &patterns);
/// Erase barriers that do not enforce conflicting memory side effects.
void populateGpuEliminateBarriersPatterns(RewritePatternSet &patterns);
-/// Tries to promote `gpu.shuffle`s to specialized AMDGPU intrinsics.
-void populateGpuPromoteShuffleToAMDGPUPatterns(
- RewritePatternSet &patterns, std::optional<ROCDL::TargetInfo> target);
-
/// Generate the code for registering passes.
#define GEN_PASS_REGISTRATION
#include "mlir/Dialect/GPU/Transforms/Passes.h.inc"
diff --git a/mlir/include/mlir/Dialect/GPU/Transforms/Passes.td b/mlir/include/mlir/Dialect/GPU/Transforms/Passes.td
index 090bfdba3ce8f5..b67caaadaf4ddb 100644
--- a/mlir/include/mlir/Dialect/GPU/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/GPU/Transforms/Passes.td
@@ -115,205 +115,4 @@ def GpuModuleToBinaryPass
];
}
-def GpuNVVMAttachTarget: Pass<"nvvm-attach-target", ""> {
- let summary = "Attaches an NVVM target attribute to a GPU Module.";
- let description = [{
- This pass searches for all GPU Modules in the immediate regions and attaches
- an NVVM target if the module matches the name specified by the `module` argument.
-
- Example:
- ```
- // File: in.mlir:
- gpu.module @nvvm_module_1 {...}
- gpu.module @nvvm_module_2 {...}
- gpu.module @rocdl_module_1 {...}
- // mlir-opt --nvvm-attach-target="module=nvvm.* chip=sm_90" in.mlir
- gpu.module @nvvm_module_1 [#nvvm.target<chip = "sm_90">] {...}
- gpu.module @nvvm_module_2 [#nvvm.target<chip = "sm_90">] {...}
- gpu.module @rocdl_module_1 {...}
- ```
- }];
- let options = [
- Option<"moduleMatcher", "module", "std::string",
- /*default=*/ [{""}],
- "Regex used to identify the modules to attach the target to.">,
- Option<"triple", "triple", "std::string",
- /*default=*/ "\"nvptx64-nvidia-cuda\"",
- "Target triple.">,
- Option<"chip", "chip", "std::string",
- /*default=*/"\"sm_75\"",
- "Target chip.">,
- Option<"features", "features", "std::string",
- /*default=*/"\"\"",
- "Target features.">,
- Option<"optLevel", "O", "unsigned",
- /*default=*/"2",
- "Optimization level.">,
- Option<"fastFlag", "fast", "bool",
- /*default=*/"false",
- "Enable fast math mode.">,
- Option<"ftzFlag", "ftz", "bool",
- /*default=*/"false",
- "Enable flush to zero for denormals.">,
- Option<"compilerDiagnosticsFlag", "collect-compiler-diagnostics", "bool",
- /*default=*/"false",
- "Enable collection of compiler diagnostics.">,
- ListOption<"linkLibs", "l", "std::string",
- "Extra bitcode libraries paths to link to.">,
- Option<"cmdOptions", "ptxas-cmd-options", "std::string",
- /*default=*/ [{""}],
- "Command line options passed to downstream compiler">,
- Option<"verifyTarget", "verify-target-arch", "bool",
- /*default=*/"true",
- "Enable verification of the target architecture">,
- ];
-}
-
-def GpuROCDLAttachTarget: Pass<"rocdl-attach-target", ""> {
- let summary = "Attaches a ROCDL target attribute to a GPU Module.";
- let description = [{
- This pass searches for all GPU Modules in the immediate regions and attaches
- a ROCDL target if the module matches the name specified by the `module` argument.
-
- Example:
- ```
- // File: in.mlir:
- gpu.module @nvvm_module_1 {...}
- gpu.module @nvvm_module_2 {...}
- gpu.module @rocdl_module_1 {...}
- // mlir-opt --nvvm-attach-target="module=rocdl.* chip=gfx90a" in.mlir
- gpu.module @nvvm_module_1 {...}
- gpu.module @nvvm_module_2 {...}
- gpu.module @rocdl_module_1 [#rocdl.target<chip = "gfx90a">] {...}
- ```
- }];
- let options = [
- Option<"moduleMatcher", "module", "std::string",
- /*default=*/ [{""}],
- "Regex used to identify the modules to attach the target to.">,
- Option<"arch", "arch", "std::string",
- /*default=*/"\"\"",
- "Target architecture, in the spelling the conversion passes take "
- "(e.g. gfx942, gfx90a:xnack+, amdgpu9.4-amd-amdhsa, "
- "amdgcn-amd-amdhsa--gfx90a:xnack-). "
- "When non-empty this supersedes 'triple' and 'chip', and its "
- "target-ID modifiers become the module's 'rocdl.xnack' and "
- "'rocdl.sramecc' attributes rather than target features.">,
- Option<"triple", "triple", "std::string",
- /*default=*/ "\"amdgcn-amd-amdhsa\"",
- "Target triple. Ignored when 'arch' is given.">,
- Option<"chip", "chip", "std::string",
- /*default=*/"\"gfx900\"",
- "Target chip. Ignored when 'arch' is given.">,
- Option<"features", "features", "std::string",
- /*default=*/"\"\"",
- "Target features.">,
- Option<"abiVersion", "abi", "std::string",
- /*default=*/"\"600\"",
- "ABI version.">,
- Option<"optLevel", "O", "unsigned",
- /*default=*/"2",
- "Optimization level.">,
- Option<"waveSize", "wavesize", "unsigned",
- /*default=*/"0",
- "Wavefront size (32 or 64). 0 takes the size from the target when "
- "'arch' is given, and Wave64 otherwise.">,
- Option<"fastFlag", "fast", "bool",
- /*default=*/"false",
- "Enable fast relaxed math opt.">,
- Option<"dazFlag", "daz", "bool",
- /*default=*/"false",
- "Enable denormals are zero opt.">,
- Option<"finiteOnlyFlag", "finite-only", "bool",
- /*default=*/"false",
- "Enable finite only opt.">,
- Option<"unsafeMathFlag", "unsafe-math", "bool",
- /*default=*/"false",
- "Enable unsafe math opt.">,
- Option<"correctSqrtFlag", "correct-sqrt", "bool",
- /*default=*/"true",
- "Enable correct rounded sqrt.">,
- ListOption<"linkLibs", "l", "std::string",
- "Extra bitcode libraries paths to link to.">,
- ];
-}
-
-def GpuSPIRVAttachTarget: Pass<"spirv-attach-target", ""> {
- let summary = "Attaches an SPIR-V target attribute to a GPU Module.";
- let description = [{
- This pass searches for all GPU Modules in the immediate regions and attaches
- an SPIR-V target if the module matches the name specified by the `module` argument.
-
- Example:
- ```
- // Given the following file: in1.mlir:
- gpu.module @nvvm_module_1 {...}
- gpu.module @spirv_module_1 {...}
- // With
- // mlir-opt --spirv-attach-target="module=spirv.* ver=v1.0 caps=Kernel" in1.mlir
- // it will generate,
- gpu.module @nvvm_module_1 {...}
- gpu.module @spirv_module_1 [#spirv.target<#spirv.vce<v1.0, [Kernel], []>, #spirv.resource_limits<>>] {...}
- ```
- }];
- let options = [
- Option<"moduleMatcher", "module", "std::string",
- /*default=*/ [{""}],
- "Regex used to identify the modules to attach the target to.">,
- Option<"spirvVersion", "ver", "std::string",
- /*default=*/ "\"v1.0\"",
- "SPIR-V Version.">,
- ListOption<"spirvCapabilities", "caps", "std::string",
- "List of supported SPIR-V Capabilities">,
- ListOption<"spirvExtensions", "exts", "std::string",
- "List of supported SPIR-V Extensions">,
- Option<"clientApi", "client_api", "std::string",
- /*default=*/ "\"Unknown\"",
- "Client API">,
- Option<"deviceVendor", "vendor", "std::string",
- /*default=*/ "\"Unknown\"",
- "Device Vendor">,
- Option<"deviceType", "device_type", "std::string",
- /*default=*/ "\"Unknown\"",
- "Device Type">,
- Option<"deviceId", "device_id", "uint32_t",
- /*default=*/ "",
- "Device ID">,
- ];
-}
-
-def GpuXeVMAttachTarget : Pass<"xevm-attach-target", ""> {
- let summary = "Attaches a XeVM target attribute to a GPU Module.";
- let description = [{
- This pass searches for all GPU Modules in the immediate regions and attaches
- a XeVM target if the module matches the name specified by the `module` argument.
-
- Example:
- ```
- // File: in.mlir:
- gpu.module @nvvm_module_1 {...}
- gpu.module @rocdl_module_2 {...}
- gpu.module @xevm_module_3 {...}
- // mlir-opt --xevm-attach-target="module=xevm.* chip=pvc" in.mlir
- gpu.module @nvvm_module_1 {...}
- gpu.module @rocdl_module_2 {...}
- gpu.module @xevm_module_3 [#xevm.target<chip = "pvc">] {...}
- ```
- }];
- let options =
- [Option<"moduleMatcher", "module", "std::string",
- /*default=*/[{""}],
- "Regex used to identify the modules to attach the target to.">,
- Option<"triple", "triple", "std::string",
- /*default=*/"\"spirv64-unknown-unknown\"", "Target triple.">,
- Option<"chip", "chip", "std::string",
- /*default=*/"\"bmg\"", "Target chip.">,
- Option<"optLevel", "O", "unsigned",
- /*default=*/"2", "Optimization level.">,
- ListOption<"linkLibs", "l", "std::string",
- "Extra bitcode libraries paths to link to.">,
- Option<"cmdOptions", "cmd-options", "std::string",
- /*default=*/[{""}],
- "Command line options passed to downstream compiler">];
-}
#endif // MLIR_DIALECT_GPU_PASSES
diff --git a/mlir/include/mlir/Dialect/GPU/Transforms/ROCDL/CMakeLists.txt b/mlir/include/mlir/Dialect/GPU/Transforms/ROCDL/CMakeLists.txt
new file mode 100644
index 00000000000000..1d023895aa23af
--- /dev/null
+++ b/mlir/include/mlir/Dialect/GPU/Transforms/ROCDL/CMakeLists.txt
@@ -0,0 +1,7 @@
+set(LLVM_TARGET_DEFINITIONS Passes.td)
+mlir_tablegen(Passes.h.inc -gen-pass-decls -name GPUROCDL)
+mlir_tablegen(Passes.capi.h.inc -gen-pass-capi-header --prefix GPUROCDL)
+mlir_tablegen(Passes.capi.cpp.inc -gen-pass-capi-impl --prefix GPUROCDL)
+add_mlir_dialect_tablegen_target(MLIRGPUROCDLPassIncGen)
+
+add_mlir_doc(Passes GPUROCDLPasses ./ -gen-pass-doc)
diff --git a/mlir/include/mlir/Dialect/GPU/Transforms/ROCDL/Passes.h b/mlir/include/mlir/Dialect/GPU/Transforms/ROCDL/Passes.h
new file mode 100644
index 00000000000000..a3e596580a8019
--- /dev/null
+++ b/mlir/include/mlir/Dialect/GPU/Transforms/ROCDL/Passes.h
@@ -0,0 +1,50 @@
+//===- Passes.h - GPU ROCDL pass entrypoints --------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This header file defines prototypes of the ROCDL-specific GPU passes and
+// patterns.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_GPU_TRANSFORMS_ROCDL_PASSES_H_
+#define MLIR_DIALECT_GPU_TRANSFORMS_ROCDL_PASSES_H_
+
+#include "mlir/Dialect/LLVMIR/ROCDLTargetInfo.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Pass/Pass.h"
+#include <optional>
+
+namespace mlir {
+
+#define GEN_PASS_DECL
+#include "mlir/Dialect/GPU/Transforms/ROCDL/Passes.h.inc"
+
+/// Collect a set of patterns to lower `gpu.subgroup_reduce` into `amdgpu.dpp`
+/// ops over scalar types. Assumes that the subgroup has
+/// `subgroupSize` lanes. Applicable only to AMD GPUs.
+void populateGpuLowerSubgroupReduceToDPPPatterns(
+ RewritePatternSet &patterns, unsigned subgroupSize,
+ const ROCDL::TargetInfo &target, PatternBenefit benefit = 1);
+
+/// Disjoint counterpart of `populateGpuLowerSubgroupReduceToDPPPatterns`
+/// that only matches `gpu.subgroup_reduce` ops with a `cluster_size`.
+void populateGpuLowerClusteredSubgroupReduceToDPPPatterns(
+ RewritePatternSet &patterns, unsigned subgroupSize,
+ const ROCDL::TargetInfo &target, PatternBenefit benefit = 1);
+
+/// Tries to promote `gpu.shuffle`s to specialized AMDGPU intrinsics.
+void populateGpuPromoteShuffleToAMDGPUPatterns(
+ RewritePatternSet &patterns, std::optional<ROCDL::TargetInfo> target);
+
+/// Generate the code for registering passes.
+#define GEN_PASS_REGISTRATION
+#include "mlir/Dialect/GPU/Transforms/ROCDL/Passes.h.inc"
+
+} // namespace mlir
+
+#endif // MLIR_DIALECT_GPU_TRANSFORMS_ROCDL_PASSES_H_
diff --git a/mlir/include/mlir/Dialect/GPU/Transforms/ROCDL/Passes.td b/mlir/include/mlir/Dialect/GPU/Transforms/ROCDL/Passes.td
new file mode 100644
index 00000000000000..0ffd4d6f0501b1
--- /dev/null
+++ b/mlir/include/mlir/Dialect/GPU/Transforms/ROCDL/Passes.td
@@ -0,0 +1,83 @@
+//===-- Passes.td - GPU ROCDL pass definition file ---------*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_GPU_TRANSFORMS_ROCDL_PASSES
+#define MLIR_DIALECT_GPU_TRANSFORMS_ROCDL_PASSES
+
+include "mlir/Pass/PassBase.td"
+
+def GpuROCDLAttachTarget: Pass<"rocdl-attach-target", ""> {
+ let summary = "Attaches a ROCDL target attribute to a GPU Module.";
+ let description = [{
+ This pass searches for all GPU Modules in the immediate regions and attaches
+ a ROCDL target if the module matches the name specified by the `module` argument.
+
+ Example:
+ ```
+ // File: in.mlir:
+ gpu.module @nvvm_module_1 {...}
+ gpu.module @nvvm_module_2 {...}
+ gpu.module @rocdl_module_1 {...}
+ // mlir-opt --nvvm-attach-target="module=rocdl.* chip=gfx90a" in.mlir
+ gpu.module @nvvm_module_1 {...}
+ gpu.module @nvvm_module_2 {...}
+ gpu.module @rocdl_module_1 [#rocdl.target<chip = "gfx90a">] {...}
+ ```
+ }];
+ let options = [
+ Option<"moduleMatcher", "module", "std::string",
+ /*default=*/ [{""}],
+ "Regex used to identify the modules to attach the target to.">,
+ Option<"arch", "arch", "std::string",
+ /*default=*/"\"\"",
+ "Target architecture, in the spelling the conversion passes take "
+ "(e.g. gfx942, gfx90a:xnack+, amdgpu9.4-amd-amdhsa, "
+ "amdgcn-amd-amdhsa--gfx90a:xnack-). "
+ "When non-empty this supersedes 'triple' and 'chip', and its "
+ "target-ID modifiers become the module's 'rocdl.xnack' and "
+ "'rocdl.sramecc' attributes rather than target features.">,
+ Option<"triple", "triple", "std::string",
+ /*default=*/ "\"amdgcn-amd-amdhsa\"",
+ "Target triple. Ignored when 'arch' is given.">,
+ Option<"chip", "chip", "std::string",
+ /*default=*/"\"gfx900\"",
+ "Target chip. Ignored when 'arch' is given.">,
+ Option<"features", "features", "std::string",
+ /*default=*/"\"\"",
+ "Target features.">,
+ Option<"abiVersion", "abi", "std::string",
+ /*default=*/"\"600\"",
+ "ABI version.">,
+ Option<"optLevel", "O", "unsigned",
+ /*default=*/"2",
+ "Optimization level.">,
+ Option<"waveSize", "wavesize", "unsigned",
+ /*default=*/"0",
+ "Wavefront size (32 or 64). 0 takes the size from the target when "
+ "'arch' is given, and Wave64 otherwise.">,
+ Option<"fastFlag", "fast", "bool",
+ /*default=*/"false",
+ "Enable fast relaxed math opt.">,
+ Option<"dazFlag", "daz", "bool",
+ /*default=*/"false",
+ "Enable denormals are zero opt.">,
+ Option<"finiteOnlyFlag", "finite-only", "bool",
+ /*default=*/"false",
+ "Enable finite only opt.">,
+ Option<"unsafeMathFlag", "unsafe-math", "bool",
+ /*default=*/"false",
+ "Enable unsafe math opt.">,
+ Option<"correctSqrtFlag", "correct-sqrt", "bool",
+ /*default=*/"true",
+ "Enable correct rounded sqrt.">,
+ ListOption<"linkLibs", "l", "std::string",
+ "Extra bitcode libraries paths to link to.">,
+ ];
+}
+
+#endif // MLIR_DIALECT_GPU_TRANSFORMS_ROCDL_PASSES
diff --git a/mlir/include/mlir/Dialect/GPU/Transforms/SPIRV/CMakeLists.txt b/mlir/include/mlir/Dialect/GPU/Transforms/SPIRV/CMakeLists.txt
new file mode 100644
index 00000000000000..e355c14926d847
--- /dev/null
+++ b/mlir/include/mlir/Dialect/GPU/Transforms/SPIRV/CMakeLists.txt
@@ -0,0 +1,7 @@
+set(LLVM_TARGET_DEFINITIONS Passes.td)
+mlir_tablegen(Passes.h.inc -gen-pass-decls -name GPUSPIRV)
+mlir_tablegen(Passes.capi.h.inc -gen-pass-capi-header --prefix GPUSPIRV)
+mlir_tablegen(Passes.capi.cpp.inc -gen-pass-capi-impl --prefix GPUSPIRV)
+add_mlir_dialect_tablegen_target(MLIRGPUSPIRVPassIncGen)
+
+add_mlir_doc(Passes GPUSPIRVPasses ./ -gen-pass-doc)
diff --git a/mlir/include/mlir/Dialect/GPU/Transforms/SPIRV/Passes.h b/mlir/include/mlir/Dialect/GPU/Transforms/SPIRV/Passes.h
new file mode 100644
index 00000000000000..94b60d0cc4b8a5
--- /dev/null
+++ b/mlir/include/mlir/Dialect/GPU/Transforms/SPIRV/Passes.h
@@ -0,0 +1,30 @@
+//===- Passes.h - GPU SPIR-V pass entrypoints -------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This header file defines prototypes of the SPIR-V-specific GPU passes and
+// patterns.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_GPU_TRANSFORMS_SPIRV_PASSES_H_
+#define MLIR_DIALECT_GPU_TRANSFORMS_SPIRV_PASSES_H_
+
+#include "mlir/Pass/Pass.h"
+
+namespace mlir {
+
+#define GEN_PASS_DECL
+#include "mlir/Dialect/GPU/Transforms/SPIRV/Passes.h.inc"
+
+/// Generate the code for registering passes.
+#define GEN_PASS_REGISTRATION
+#include "mlir/Dialect/GPU/Transforms/SPIRV/Passes.h.inc"
+
+} // namespace mlir
+
+#endif // MLIR_DIALECT_GPU_TRANSFORMS_SPIRV_PASSES_H_
diff --git a/mlir/include/mlir/Dialect/GPU/Transforms/SPIRV/Passes.td b/mlir/include/mlir/Dialect/GPU/Transforms/SPIRV/Passes.td
new file mode 100644
index 00000000000000..dd1dd3f78dc895
--- /dev/null
+++ b/mlir/include/mlir/Dialect/GPU/Transforms/SPIRV/Passes.td
@@ -0,0 +1,58 @@
+//===-- Passes.td - GPU SPIR-V pass definition file --------*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_GPU_TRANSFORMS_SPIRV_PASSES
+#define MLIR_DIALECT_GPU_TRANSFORMS_SPIRV_PASSES
+
+include "mlir/Pass/PassBase.td"
+
+def GpuSPIRVAttachTarget: Pass<"spirv-attach-target", ""> {
+ let summary = "Attaches an SPIR-V target attribute to a GPU Module.";
+ let description = [{
+ This pass searches for all GPU Modules in the immediate regions and attaches
+ an SPIR-V target if the module matches the name specified by the `module` argument.
+
+ Example:
+ ```
+ // Given the following file: in1.mlir:
+ gpu.module @nvvm_module_1 {...}
+ gpu.module @spirv_module_1 {...}
+ // With
+ // mlir-opt --spirv-attach-target="module=spirv.* ver=v1.0 caps=Kernel" in1.mlir
+ // it will generate,
+ gpu.module @nvvm_module_1 {...}
+ gpu.module @spirv_module_1 [#spirv.target<#spirv.vce<v1.0, [Kernel], []>, #spirv.resource_limits<>>] {...}
+ ```
+ }];
+ let options = [
+ Option<"moduleMatcher", "module", "std::string",
+ /*default=*/ [{""}],
+ "Regex used to identify the modules to attach the target to.">,
+ Option<"spirvVersion", "ver", "std::string",
+ /*default=*/ "\"v1.0\"",
+ "SPIR-V Version.">,
+ ListOption<"spirvCapabilities", "caps", "std::string",
+ "List of supported SPIR-V Capabilities">,
+ ListOption<"spirvExtensions", "exts", "std::string",
+ "List of supported SPIR-V Extensions">,
+ Option<"clientApi", "client_api", "std::string",
+ /*default=*/ "\"Unknown\"",
+ "Client API">,
+ Option<"deviceVendor", "vendor", "std::string",
+ /*default=*/ "\"Unknown\"",
+ "Device Vendor">,
+ Option<"deviceType", "device_type", "std::string",
+ /*default=*/ "\"Unknown\"",
+ "Device Type">,
+ Option<"deviceId", "device_id", "uint32_t",
+ /*default=*/ "",
+ "Device ID">,
+ ];
+}
+
+#endif // MLIR_DIALECT_GPU_TRANSFORMS_SPIRV_PASSES
diff --git a/mlir/include/mlir/Dialect/GPU/Transforms/XeVM/CMakeLists.txt b/mlir/include/mlir/Dialect/GPU/Transforms/XeVM/CMakeLists.txt
new file mode 100644
index 00000000000000..182e56aa3432c5
--- /dev/null
+++ b/mlir/include/mlir/Dialect/GPU/Transforms/XeVM/CMakeLists.txt
@@ -0,0 +1,7 @@
+set(LLVM_TARGET_DEFINITIONS Passes.td)
+mlir_tablegen(Passes.h.inc -gen-pass-decls -name GPUXeVM)
+mlir_tablegen(Passes.capi.h.inc -gen-pass-capi-header --prefix GPUXeVM)
+mlir_tablegen(Passes.capi.cpp.inc -gen-pass-capi-impl --prefix GPUXeVM)
+add_mlir_dialect_tablegen_target(MLIRGPUXeVMPassIncGen)
+
+add_mlir_doc(Passes GPUXeVMPasses ./ -gen-pass-doc)
diff --git a/mlir/include/mlir/Dialect/GPU/Transforms/XeVM/Passes.h b/mlir/include/mlir/Dialect/GPU/Transforms/XeVM/Passes.h
new file mode 100644
index 00000000000000..26b51550c5a943
--- /dev/null
+++ b/mlir/include/mlir/Dialect/GPU/Transforms/XeVM/Passes.h
@@ -0,0 +1,30 @@
+//===- Passes.h - GPU XeVM pass entrypoints ---------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This header file defines prototypes of the XeVM-specific GPU passes and
+// patterns.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_GPU_TRANSFORMS_XEVM_PASSES_H_
+#define MLIR_DIALECT_GPU_TRANSFORMS_XEVM_PASSES_H_
+
+#include "mlir/Pass/Pass.h"
+
+namespace mlir {
+
+#define GEN_PASS_DECL
+#include "mlir/Dialect/GPU/Transforms/XeVM/Passes.h.inc"
+
+/// Generate the code for registering passes.
+#define GEN_PASS_REGISTRATION
+#include "mlir/Dialect/GPU/Transforms/XeVM/Passes.h.inc"
+
+} // namespace mlir
+
+#endif // MLIR_DIALECT_GPU_TRANSFORMS_XEVM_PASSES_H_
diff --git a/mlir/include/mlir/Dialect/GPU/Transforms/XeVM/Passes.td b/mlir/include/mlir/Dialect/GPU/Transforms/XeVM/Passes.td
new file mode 100644
index 00000000000000..9fe2b164abf5c3
--- /dev/null
+++ b/mlir/include/mlir/Dialect/GPU/Transforms/XeVM/Passes.td
@@ -0,0 +1,49 @@
+//===-- Passes.td - GPU XeVM pass definition file ----------*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_GPU_TRANSFORMS_XEVM_PASSES
+#define MLIR_DIALECT_GPU_TRANSFORMS_XEVM_PASSES
+
+include "mlir/Pass/PassBase.td"
+
+def GpuXeVMAttachTarget : Pass<"xevm-attach-target", ""> {
+ let summary = "Attaches a XeVM target attribute to a GPU Module.";
+ let description = [{
+ This pass searches for all GPU Modules in the immediate regions and attaches
+ a XeVM target if the module matches the name specified by the `module` argument.
+
+ Example:
+ ```
+ // File: in.mlir:
+ gpu.module @nvvm_module_1 {...}
+ gpu.module @rocdl_module_2 {...}
+ gpu.module @xevm_module_3 {...}
+ // mlir-opt --xevm-attach-target="module=xevm.* chip=pvc" in.mlir
+ gpu.module @nvvm_module_1 {...}
+ gpu.module @rocdl_module_2 {...}
+ gpu.module @xevm_module_3 [#xevm.target<chip = "pvc">] {...}
+ ```
+ }];
+ let options =
+ [Option<"moduleMatcher", "module", "std::string",
+ /*default=*/[{""}],
+ "Regex used to identify the modules to attach the target to.">,
+ Option<"triple", "triple", "std::string",
+ /*default=*/"\"spirv64-unknown-unknown\"", "Target triple.">,
+ Option<"chip", "chip", "std::string",
+ /*default=*/"\"bmg\"", "Target chip.">,
+ Option<"optLevel", "O", "unsigned",
+ /*default=*/"2", "Optimization level.">,
+ ListOption<"linkLibs", "l", "std::string",
+ "Extra bitcode libraries paths to link to.">,
+ Option<"cmdOptions", "cmd-options", "std::string",
+ /*default=*/[{""}],
+ "Command line options passed to downstream compiler">];
+}
+
+#endif // MLIR_DIALECT_GPU_TRANSFORMS_XEVM_PASSES
diff --git a/mlir/include/mlir/Dialect/GPU/Utils/GPUUtils.h b/mlir/include/mlir/Dialect/GPU/Utils/GPUUtils.h
index 073493971e6b7c..f5cb7fee392674 100644
--- a/mlir/include/mlir/Dialect/GPU/Utils/GPUUtils.h
+++ b/mlir/include/mlir/Dialect/GPU/Utils/GPUUtils.h
@@ -29,6 +29,19 @@ class LaunchOp;
/// Returns the matching vector combining kind.
vector::CombiningKind convertReductionKind(gpu::AllReduceOperation mode);
+
+/// Lane layout of the clusters reduced by a `gpu.subgroup_reduce` op.
+struct SubgroupReduceClusterInfo {
+ unsigned clusterStride;
+ unsigned clusterSize;
+ unsigned subgroupSize;
+};
+
+/// Returns the cluster layout of `op` in a subgroup of `subgroupSize` lanes,
+/// which must be a power of two. Emits an op error and fails if the cluster
+/// size exceeds the subgroup size or the cluster stride is not less than it.
+FailureOr<SubgroupReduceClusterInfo>
+getSubgroupReduceClusterInfo(SubgroupReduceOp op, unsigned subgroupSize);
} // namespace gpu
/// Get a gpu.func created from outlining the region of a gpu.launch op with the
diff --git a/mlir/lib/Bindings/Python/GPUPasses.cpp b/mlir/lib/Bindings/Python/GPUPasses.cpp
index be474edbe9639a..762531d5cfcc86 100644
--- a/mlir/lib/Bindings/Python/GPUPasses.cpp
+++ b/mlir/lib/Bindings/Python/GPUPasses.cpp
@@ -19,4 +19,8 @@ NB_MODULE(_mlirGPUPasses, m) {
// Register all GPU passes on load.
mlirRegisterGPUPasses();
+ mlirRegisterGPUNVVMPasses();
+ mlirRegisterGPUROCDLPasses();
+ mlirRegisterGPUSPIRVPasses();
+ mlirRegisterGPUXeVMPasses();
}
diff --git a/mlir/lib/CAPI/Dialect/CMakeLists.txt b/mlir/lib/CAPI/Dialect/CMakeLists.txt
index 769b2f44ccfbff..3062ecfa6c647a 100644
--- a/mlir/lib/CAPI/Dialect/CMakeLists.txt
+++ b/mlir/lib/CAPI/Dialect/CMakeLists.txt
@@ -158,10 +158,18 @@ add_mlir_upstream_c_api_library(MLIRCAPIGPU
PARTIAL_SOURCES_INTENDED
DEPENDS
MLIRGPUPassIncGen
+ MLIRGPUNVVMPassIncGen
+ MLIRGPUROCDLPassIncGen
+ MLIRGPUSPIRVPassIncGen
+ MLIRGPUXeVMPassIncGen
LINK_LIBS PUBLIC
MLIRCAPIIR
+ MLIRGPUNVVMTransforms
+ MLIRGPUROCDLTransforms
+ MLIRGPUSPIRVTransforms
MLIRGPUTransforms
+ MLIRGPUXeVMTransforms
MLIRPass
)
diff --git a/mlir/lib/CAPI/Dialect/GPUPasses.cpp b/mlir/lib/CAPI/Dialect/GPUPasses.cpp
index 5128c63ecb511f..f455e6d67317e4 100644
--- a/mlir/lib/CAPI/Dialect/GPUPasses.cpp
+++ b/mlir/lib/CAPI/Dialect/GPUPasses.cpp
@@ -7,11 +7,19 @@
//===----------------------------------------------------------------------===//
#include "mlir/CAPI/Pass.h"
+#include "mlir/Dialect/GPU/Transforms/NVVM/Passes.h"
#include "mlir/Dialect/GPU/Transforms/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/ROCDL/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/SPIRV/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/XeVM/Passes.h"
#include "mlir/Pass/Pass.h"
// Must include the declarations as they carry important visibility attributes.
+#include "mlir/Dialect/GPU/Transforms/NVVM/Passes.capi.h.inc"
#include "mlir/Dialect/GPU/Transforms/Passes.capi.h.inc"
+#include "mlir/Dialect/GPU/Transforms/ROCDL/Passes.capi.h.inc"
+#include "mlir/Dialect/GPU/Transforms/SPIRV/Passes.capi.h.inc"
+#include "mlir/Dialect/GPU/Transforms/XeVM/Passes.capi.h.inc"
using namespace mlir;
@@ -19,7 +27,11 @@ using namespace mlir;
extern "C" {
#endif
+#include "mlir/Dialect/GPU/Transforms/NVVM/Passes.capi.cpp.inc"
#include "mlir/Dialect/GPU/Transforms/Passes.capi.cpp.inc"
+#include "mlir/Dialect/GPU/Transforms/ROCDL/Passes.capi.cpp.inc"
+#include "mlir/Dialect/GPU/Transforms/SPIRV/Passes.capi.cpp.inc"
+#include "mlir/Dialect/GPU/Transforms/XeVM/Passes.capi.cpp.inc"
#ifdef __cplusplus
}
diff --git a/mlir/lib/Conversion/GPUToROCDL/CMakeLists.txt b/mlir/lib/Conversion/GPUToROCDL/CMakeLists.txt
index 945e3ccdfa87bb..27a4ff414e9641 100644
--- a/mlir/lib/Conversion/GPUToROCDL/CMakeLists.txt
+++ b/mlir/lib/Conversion/GPUToROCDL/CMakeLists.txt
@@ -17,6 +17,7 @@ add_mlir_conversion_library(MLIRGPUToROCDLTransforms
MLIRAMDGPUToROCDL
MLIRFuncToLLVM
MLIRGPUDialect
+ MLIRGPUROCDLTransforms
MLIRGPUToGPURuntimeTransforms
MLIRLLVMCommonConversion
MLIRLLVMDialect
diff --git a/mlir/lib/Conversion/GPUToROCDL/LowerGpuOpsToROCDLOps.cpp b/mlir/lib/Conversion/GPUToROCDL/LowerGpuOpsToROCDLOps.cpp
index 4c98fe247bd123..dfc872aedaacc0 100644
--- a/mlir/lib/Conversion/GPUToROCDL/LowerGpuOpsToROCDLOps.cpp
+++ b/mlir/lib/Conversion/GPUToROCDL/LowerGpuOpsToROCDLOps.cpp
@@ -31,6 +31,7 @@
#include "mlir/Dialect/Func/IR/FuncOps.h"
#include "mlir/Dialect/GPU/IR/GPUDialect.h"
#include "mlir/Dialect/GPU/Transforms/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/ROCDL/Passes.h"
#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
#include "mlir/Dialect/LLVMIR/ROCDLDialect.h"
#include "mlir/Dialect/MemRef/IR/MemRef.h"
diff --git a/mlir/lib/Dialect/GPU/CMakeLists.txt b/mlir/lib/Dialect/GPU/CMakeLists.txt
index 67cb2f1e6dacd0..cd4fc441f51f4b 100644
--- a/mlir/lib/Dialect/GPU/CMakeLists.txt
+++ b/mlir/lib/Dialect/GPU/CMakeLists.txt
@@ -66,15 +66,10 @@ add_mlir_dialect_library(MLIRGPUTransforms
Transforms/IndexedAccessOpInterfaceImpl.cpp
Transforms/MemoryPromotion.cpp
Transforms/ModuleToBinary.cpp
- Transforms/NVVMAttachTarget.cpp
Transforms/ParallelLoopMapper.cpp
- Transforms/PromoteShuffleToAMDGPU.cpp
- Transforms/ROCDLAttachTarget.cpp
- Transforms/SPIRVAttachTarget.cpp
Transforms/ShuffleRewriter.cpp
Transforms/SubgroupIdRewriter.cpp
Transforms/SubgroupReduceLowering.cpp
- Transforms/XeVMAttachTarget.cpp
OBJECT
@@ -86,30 +81,96 @@ add_mlir_dialect_library(MLIRGPUTransforms
MLIRParallelLoopMapperEnumsGen
LINK_LIBS PUBLIC
- MLIRAMDGPUDialect
MLIRAffineUtils
MLIRArithDialect
MLIRAsyncDialect
MLIRBufferizationDialect
MLIRDataLayoutInterfaces
MLIRDialectUtils
- MLIRExecutionEngineUtils
MLIRGPUDialect
MLIRGPUUtils
MLIRIR
MLIRIndexDialect
- MLIRLLVMDialect
MLIRMemRefDialect
- MLIRNVVMTarget
MLIRPass
- MLIRROCDLDialect
- MLIRROCDLTarget
MLIRSCFDialect
- MLIRSPIRVTarget
MLIRSideEffectInterfaces
MLIRSupport
MLIRTransformUtils
MLIRVectorDialect
+ )
+
+add_mlir_dialect_library(MLIRGPUNVVMTransforms
+ Transforms/NVVM/NVVMAttachTarget.cpp
+
+ ADDITIONAL_HEADER_DIRS
+ ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/GPU/Transforms/NVVM
+
+ DEPENDS
+ MLIRGPUNVVMPassIncGen
+
+ LINK_LIBS PUBLIC
+ MLIRGPUDialect
+ MLIRIR
+ MLIRNVVMDialect
+ MLIRPass
+ MLIRSupport
+ )
+
+add_mlir_dialect_library(MLIRGPUROCDLTransforms
+ Transforms/ROCDL/PromoteShuffleToAMDGPU.cpp
+ Transforms/ROCDL/ROCDLAttachTarget.cpp
+ Transforms/ROCDL/SubgroupReduceToDPP.cpp
+
+ ADDITIONAL_HEADER_DIRS
+ ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/GPU/Transforms/ROCDL
+
+ DEPENDS
+ MLIRGPUROCDLPassIncGen
+
+ LINK_LIBS PUBLIC
+ MLIRAMDGPUDialect
+ MLIRArithDialect
+ MLIRGPUDialect
+ MLIRGPUUtils
+ MLIRIR
+ MLIRPass
+ MLIRROCDLDialect
+ MLIRSupport
+ MLIRVectorDialect
+ )
+
+add_mlir_dialect_library(MLIRGPUSPIRVTransforms
+ Transforms/SPIRV/SPIRVAttachTarget.cpp
+
+ ADDITIONAL_HEADER_DIRS
+ ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/GPU/Transforms/SPIRV
+
+ DEPENDS
+ MLIRGPUSPIRVPassIncGen
+
+ LINK_LIBS PUBLIC
+ MLIRGPUDialect
+ MLIRIR
+ MLIRPass
+ MLIRSPIRVDialect
+ MLIRSupport
+ )
+
+add_mlir_dialect_library(MLIRGPUXeVMTransforms
+ Transforms/XeVM/XeVMAttachTarget.cpp
+
+ ADDITIONAL_HEADER_DIRS
+ ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/GPU/Transforms/XeVM
+
+ DEPENDS
+ MLIRGPUXeVMPassIncGen
+
+ LINK_LIBS PUBLIC
+ MLIRGPUDialect
+ MLIRIR
+ MLIRPass
+ MLIRSupport
MLIRXeVMDialect
)
diff --git a/mlir/lib/Dialect/GPU/Pipelines/CMakeLists.txt b/mlir/lib/Dialect/GPU/Pipelines/CMakeLists.txt
index 3a0cd7ef5c39d2..1a9f7bd9715906 100644
--- a/mlir/lib/Dialect/GPU/Pipelines/CMakeLists.txt
+++ b/mlir/lib/Dialect/GPU/Pipelines/CMakeLists.txt
@@ -17,9 +17,13 @@ add_mlir_dialect_library(MLIRGPUPipelines
MLIRArithToLLVM
MLIRArithTransforms
MLIRFuncToLLVM
+ MLIRGPUNVVMTransforms
+ MLIRGPUROCDLTransforms
MLIRGPUToLLVMSPV
MLIRGPUToNVVMTransforms
MLIRGPUToROCDLTransforms
+ MLIRGPUTransforms
+ MLIRGPUXeVMTransforms
MLIRIndexToLLVM
MLIRMathToLLVM
MLIRMathToXeVM
diff --git a/mlir/lib/Dialect/GPU/Pipelines/GPUToNVVMPipeline.cpp b/mlir/lib/Dialect/GPU/Pipelines/GPUToNVVMPipeline.cpp
index 5462cddd44718d..af145369db7e82 100644
--- a/mlir/lib/Dialect/GPU/Pipelines/GPUToNVVMPipeline.cpp
+++ b/mlir/lib/Dialect/GPU/Pipelines/GPUToNVVMPipeline.cpp
@@ -25,6 +25,7 @@
#include "mlir/Conversion/VectorToSCF/VectorToSCF.h"
#include "mlir/Dialect/GPU/IR/GPUDialect.h"
#include "mlir/Dialect/GPU/Pipelines/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/NVVM/Passes.h"
#include "mlir/Dialect/GPU/Transforms/Passes.h"
#include "mlir/Dialect/MemRef/Transforms/Passes.h"
#include "mlir/Pass/PassManager.h"
diff --git a/mlir/lib/Dialect/GPU/Pipelines/GPUToROCDLPipeline.cpp b/mlir/lib/Dialect/GPU/Pipelines/GPUToROCDLPipeline.cpp
index edbcaa1f8ca1c0..bc191d3f7591ff 100644
--- a/mlir/lib/Dialect/GPU/Pipelines/GPUToROCDLPipeline.cpp
+++ b/mlir/lib/Dialect/GPU/Pipelines/GPUToROCDLPipeline.cpp
@@ -28,6 +28,7 @@
#include "mlir/Dialect/GPU/IR/GPUDialect.h"
#include "mlir/Dialect/GPU/Pipelines/Passes.h"
#include "mlir/Dialect/GPU/Transforms/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/ROCDL/Passes.h"
#include "mlir/Dialect/LLVMIR/ROCDLTargetInfo.h"
#include "mlir/Dialect/MemRef/Transforms/Passes.h"
#include "mlir/Pass/PassManager.h"
diff --git a/mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp b/mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp
index 9af4624a53b253..3ab090aa82529d 100644
--- a/mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp
+++ b/mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp
@@ -29,6 +29,7 @@
#include "mlir/Dialect/GPU/IR/GPUDialect.h"
#include "mlir/Dialect/GPU/Pipelines/Passes.h"
#include "mlir/Dialect/GPU/Transforms/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/XeVM/Passes.h"
#include "mlir/Dialect/LLVMIR/Transforms/RequestCWrappers.h"
#include "mlir/Dialect/Math/Transforms/Passes.h"
#include "mlir/Dialect/MemRef/Transforms/Passes.h"
diff --git a/mlir/lib/Dialect/GPU/TransformOps/CMakeLists.txt b/mlir/lib/Dialect/GPU/TransformOps/CMakeLists.txt
index 2d3dfdbfc11368..2b341bb8f660e1 100644
--- a/mlir/lib/Dialect/GPU/TransformOps/CMakeLists.txt
+++ b/mlir/lib/Dialect/GPU/TransformOps/CMakeLists.txt
@@ -13,6 +13,7 @@ add_mlir_dialect_library(MLIRGPUTransformOps
LINK_LIBS PUBLIC
MLIRGPUDialect
+ MLIRGPUROCDLTransforms
MLIRGPUTransforms
MLIRIR
MLIRParser
diff --git a/mlir/lib/Dialect/GPU/TransformOps/GPUTransformOps.cpp b/mlir/lib/Dialect/GPU/TransformOps/GPUTransformOps.cpp
index a7858781a10792..1b2988a1abe9e3 100644
--- a/mlir/lib/Dialect/GPU/TransformOps/GPUTransformOps.cpp
+++ b/mlir/lib/Dialect/GPU/TransformOps/GPUTransformOps.cpp
@@ -18,6 +18,7 @@
#include "mlir/Dialect/GPU/IR/GPUDialect.h"
#include "mlir/Dialect/GPU/TransformOps/Utils.h"
#include "mlir/Dialect/GPU/Transforms/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/ROCDL/Passes.h"
#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
#include "mlir/Dialect/LLVMIR/ROCDLDialect.h"
#include "mlir/Dialect/MemRef/IR/MemRef.h"
diff --git a/mlir/lib/Dialect/GPU/Transforms/NVVMAttachTarget.cpp b/mlir/lib/Dialect/GPU/Transforms/NVVM/NVVMAttachTarget.cpp
similarity index 96%
rename from mlir/lib/Dialect/GPU/Transforms/NVVMAttachTarget.cpp
rename to mlir/lib/Dialect/GPU/Transforms/NVVM/NVVMAttachTarget.cpp
index daccb86e23f67b..05ffff5260358f 100644
--- a/mlir/lib/Dialect/GPU/Transforms/NVVMAttachTarget.cpp
+++ b/mlir/lib/Dialect/GPU/Transforms/NVVM/NVVMAttachTarget.cpp
@@ -11,18 +11,17 @@
//
//===----------------------------------------------------------------------===//
-#include "mlir/Dialect/GPU/Transforms/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/NVVM/Passes.h"
#include "mlir/Dialect/GPU/IR/GPUDialect.h"
#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
#include "mlir/IR/Builders.h"
#include "mlir/Pass/Pass.h"
-#include "mlir/Target/LLVM/NVVM/Target.h"
#include "llvm/Support/Regex.h"
namespace mlir {
#define GEN_PASS_DEF_GPUNVVMATTACHTARGET
-#include "mlir/Dialect/GPU/Transforms/Passes.h.inc"
+#include "mlir/Dialect/GPU/Transforms/NVVM/Passes.h.inc"
} // namespace mlir
using namespace mlir;
diff --git a/mlir/lib/Dialect/GPU/Transforms/PromoteShuffleToAMDGPU.cpp b/mlir/lib/Dialect/GPU/Transforms/ROCDL/PromoteShuffleToAMDGPU.cpp
similarity index 98%
rename from mlir/lib/Dialect/GPU/Transforms/PromoteShuffleToAMDGPU.cpp
rename to mlir/lib/Dialect/GPU/Transforms/ROCDL/PromoteShuffleToAMDGPU.cpp
index 6003e32b5d6afb..918f4a3f369939 100644
--- a/mlir/lib/Dialect/GPU/Transforms/PromoteShuffleToAMDGPU.cpp
+++ b/mlir/lib/Dialect/GPU/Transforms/ROCDL/PromoteShuffleToAMDGPU.cpp
@@ -11,7 +11,7 @@
//
//===----------------------------------------------------------------------===//
-#include "mlir/Dialect/GPU/Transforms/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/ROCDL/Passes.h"
#include "mlir/Dialect/LLVMIR/ROCDLTargetInfo.h"
#include "mlir/Dialect/AMDGPU/IR/AMDGPUDialect.h"
diff --git a/mlir/lib/Dialect/GPU/Transforms/ROCDLAttachTarget.cpp b/mlir/lib/Dialect/GPU/Transforms/ROCDL/ROCDLAttachTarget.cpp
similarity index 97%
rename from mlir/lib/Dialect/GPU/Transforms/ROCDLAttachTarget.cpp
rename to mlir/lib/Dialect/GPU/Transforms/ROCDL/ROCDLAttachTarget.cpp
index cef6deee5c3548..ad8ed4fbf03b38 100644
--- a/mlir/lib/Dialect/GPU/Transforms/ROCDLAttachTarget.cpp
+++ b/mlir/lib/Dialect/GPU/Transforms/ROCDL/ROCDLAttachTarget.cpp
@@ -11,19 +11,18 @@
//
//===----------------------------------------------------------------------===//
-#include "mlir/Dialect/GPU/Transforms/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/ROCDL/Passes.h"
#include "mlir/Dialect/GPU/IR/GPUDialect.h"
#include "mlir/Dialect/LLVMIR/ROCDLDialect.h"
#include "mlir/Dialect/LLVMIR/ROCDLTargetInfo.h"
#include "mlir/IR/Builders.h"
#include "mlir/Pass/Pass.h"
-#include "mlir/Target/LLVM/ROCDL/Target.h"
#include "llvm/Support/Regex.h"
namespace mlir {
#define GEN_PASS_DEF_GPUROCDLATTACHTARGET
-#include "mlir/Dialect/GPU/Transforms/Passes.h.inc"
+#include "mlir/Dialect/GPU/Transforms/ROCDL/Passes.h.inc"
} // namespace mlir
using namespace mlir;
diff --git a/mlir/lib/Dialect/GPU/Transforms/ROCDL/SubgroupReduceToDPP.cpp b/mlir/lib/Dialect/GPU/Transforms/ROCDL/SubgroupReduceToDPP.cpp
new file mode 100644
index 00000000000000..acf8b8091d1384
--- /dev/null
+++ b/mlir/lib/Dialect/GPU/Transforms/ROCDL/SubgroupReduceToDPP.cpp
@@ -0,0 +1,238 @@
+//===- SubgroupReduceToDPP.cpp - subgroup_reduce to AMDGPU DPP lowering ---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Implements lowering of `gpu.subgroup_reduce` ops to AMDGPU DPP operations.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/GPU/Transforms/ROCDL/Passes.h"
+
+#include "mlir/Dialect/AMDGPU/IR/AMDGPUDialect.h"
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/GPU/IR/GPUDialect.h"
+#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
+#include "mlir/Dialect/LLVMIR/ROCDLDialect.h"
+#include "mlir/Dialect/LLVMIR/ROCDLTargetInfo.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/IR/PatternMatch.h"
+#include "llvm/Support/FormatVariadic.h"
+#include <cassert>
+
+using namespace mlir;
+
+namespace {
+static FailureOr<Value>
+createSubgroupDPPReduction(PatternRewriter &rewriter, gpu::SubgroupReduceOp op,
+ Value input, gpu::AllReduceOperation mode,
+ const gpu::SubgroupReduceClusterInfo &ci,
+ const ROCDL::TargetInfo &target) {
+ Location loc = op.getLoc();
+ Value dpp;
+ Value res = input;
+ constexpr int allRows = 0xf;
+ constexpr int allBanks = 0xf;
+ const bool boundCtrl = true;
+ if (ci.clusterSize >= 2) {
+ // Perform reduction between all lanes N <-> N+1.
+ dpp = amdgpu::DPPOp::create(
+ rewriter, loc, res.getType(), res, res, amdgpu::DPPPerm::quad_perm,
+ rewriter.getI32ArrayAttr({1, 0, 3, 2}), allRows, allBanks, boundCtrl);
+ res = vector::makeArithReduction(rewriter, loc,
+ gpu::convertReductionKind(mode), res, dpp);
+ }
+
+ if (ci.clusterSize >= 4) {
+ // Perform reduction between all lanes N <-> N+2.
+ dpp = amdgpu::DPPOp::create(
+ rewriter, loc, res.getType(), res, res, amdgpu::DPPPerm::quad_perm,
+ rewriter.getI32ArrayAttr({2, 3, 0, 1}), allRows, allBanks, boundCtrl);
+ res = vector::makeArithReduction(rewriter, loc,
+ gpu::convertReductionKind(mode), res, dpp);
+ }
+ if (ci.clusterSize >= 8) {
+ // Perform reduction between all lanes N <-> 7-N,
+ // e.g lane[0] <-> lane[7], lane[1] <-> lane[6]..., lane[3] <-> lane[4].
+ dpp = amdgpu::DPPOp::create(rewriter, loc, res.getType(), res, res,
+ amdgpu::DPPPerm::row_half_mirror,
+ rewriter.getUnitAttr(), allRows, allBanks,
+ boundCtrl);
+ res = vector::makeArithReduction(rewriter, loc,
+ gpu::convertReductionKind(mode), res, dpp);
+ }
+ if (ci.clusterSize >= 16) {
+ // Perform reduction between all lanes N <-> 15-N,
+ // e.g lane[0] <-> lane[15], lane[1] <-> lane[14]..., lane[7] <-> lane[8].
+ dpp = amdgpu::DPPOp::create(
+ rewriter, loc, res.getType(), res, res, amdgpu::DPPPerm::row_mirror,
+ rewriter.getUnitAttr(), allRows, allBanks, boundCtrl);
+ res = vector::makeArithReduction(rewriter, loc,
+ gpu::convertReductionKind(mode), res, dpp);
+ }
+ if (ci.clusterSize >= 32) {
+ if (!target.has(llvm::AMDGPU::FEAT_GFX10_INSTS)) {
+ // Broadcast last value from each row to next row.
+ // Use row mask to avoid polluting row 0 (and row 2 if wave-64).
+ dpp = amdgpu::DPPOp::create(rewriter, loc, res.getType(), res, res,
+ amdgpu::DPPPerm::row_bcast_15,
+ rewriter.getUnitAttr(), 0xa, allBanks,
+ /*bound_ctrl*/ false);
+ res = vector::makeArithReduction(
+ rewriter, loc, gpu::convertReductionKind(mode), res, dpp);
+
+ // For subgroupSize = 64, at this point lanes [16, 32) contain the full
+ // reduction over lanes [0, 32), but lanes [0, 16) do not. Similarly,
+ // lanes [48, 64) contain the full reduction over lanes [32, 64), but
+ // lanes [32, 48) do not.
+ //
+ // If subgroup size is 64 and cluster size is 64, we don't need lanes [0,
+ // 16) and [32, 48) to have the correct cluster-32 reduction values at
+ // this point, because only lane 63's value will ultimately be read in
+ // this full-cluster case.
+ //
+ // If subgroup size is 64 and cluster size is 32, we need to ensure that
+ // lanes [0, 16) and [32, 48) have the correct final cluster-32 reduction
+ // values (subgroup_reduce guarantees that all lanes within each cluster
+ // contain the final reduction value). We do this by broadcasting lane
+ // 31's value to lanes [0, 16) and lanes 63's value to lanes [32, 48).
+ //
+ // See https://gpuopen.com/learn/amd-gcn-assembly-cross-lane-operations
+ // for an illustration of how this within-cluster broadcast works with a
+ // swizzle.
+ if (ci.subgroupSize == 64 && ci.clusterSize == 32) {
+ res =
+ amdgpu::SwizzleBitModeOp::create(rewriter, loc, res, /*and_mask=*/0,
+ /*or_mask=*/31,
+ /*xor_mask=*/0);
+ }
+ } else if (!target.has(llvm::AMDGPU::FEAT_GFX13_INSTS)) {
+ // Use a permute lane to cross rows (row 1 <-> row 0, row 3 <-> row 2).
+ Value uint32Max = arith::ConstantOp::create(
+ rewriter, loc, rewriter.getI32Type(), rewriter.getI32IntegerAttr(-1));
+ dpp = ROCDL::PermlaneX16Op::create(rewriter, loc, res.getType(), res, res,
+ uint32Max, uint32Max,
+ /*fi=*/true,
+ /*boundControl=*/false);
+ res = vector::makeArithReduction(
+ rewriter, loc, gpu::convertReductionKind(mode), res, dpp);
+ } else {
+ return rewriter.notifyMatchFailure(
+ op, "Subgroup reduce lowering to DPP not currently supported for "
+ "this device.");
+ }
+ if (ci.subgroupSize == 32) {
+ Value lane31 = arith::ConstantOp::create(
+ rewriter, loc, rewriter.getI32Type(), rewriter.getI32IntegerAttr(31));
+ res =
+ ROCDL::ReadlaneOp::create(rewriter, loc, res.getType(), res, lane31);
+ }
+ }
+ if (ci.clusterSize >= 64) {
+ if (!target.has(llvm::AMDGPU::FEAT_GFX10_INSTS)) {
+ // Broadcast 31st lane value to rows 2 and 3.
+ dpp = amdgpu::DPPOp::create(rewriter, loc, res.getType(), res, res,
+ amdgpu::DPPPerm::row_bcast_31,
+ rewriter.getUnitAttr(), 0xf, allBanks,
+ /*bound_ctrl*/ true);
+ res = vector::makeArithReduction(
+ rewriter, loc, gpu::convertReductionKind(mode), dpp, res);
+ // Obtain reduction from last rows, the previous rows are polluted.
+ Value lane63 = arith::ConstantOp::create(
+ rewriter, loc, rewriter.getI32Type(), rewriter.getI32IntegerAttr(63));
+ res =
+ ROCDL::ReadlaneOp::create(rewriter, loc, res.getType(), res, lane63);
+
+ } else if (!target.has(llvm::AMDGPU::FEAT_GFX13_INSTS)) {
+ // Assume reduction across 32 lanes has been done.
+ // Perform final reduction manually by summing values in lane 0 and
+ // lane 32.
+ Value lane31 = arith::ConstantOp::create(
+ rewriter, loc, rewriter.getI32Type(), rewriter.getI32IntegerAttr(31));
+ Value lane63 = arith::ConstantOp::create(
+ rewriter, loc, rewriter.getI32Type(), rewriter.getI32IntegerAttr(63));
+ lane31 =
+ ROCDL::ReadlaneOp::create(rewriter, loc, res.getType(), res, lane31);
+ lane63 =
+ ROCDL::ReadlaneOp::create(rewriter, loc, res.getType(), res, lane63);
+ res = vector::makeArithReduction(
+ rewriter, loc, gpu::convertReductionKind(mode), lane31, lane63);
+ } else {
+ return rewriter.notifyMatchFailure(
+ op, "Subgroup reduce lowering to DPP not currently supported for "
+ "this device.");
+ }
+ }
+ assert(res.getType() == input.getType());
+ return res;
+}
+
+/// Collect a set of patterns to lower `gpu.subgroup_reduce` into `amdgpu.dpp`
+/// ops over scalar types. Assumes that the subgroup has
+/// `subgroupSize` lanes. Applicable only to AMD GPUs.
+struct ScalarSubgroupReduceToDPP final
+ : OpRewritePattern<gpu::SubgroupReduceOp> {
+ ScalarSubgroupReduceToDPP(MLIRContext *ctx, unsigned subgroupSize,
+ bool matchClustered,
+ const ROCDL::TargetInfo &target,
+ PatternBenefit benefit)
+ : OpRewritePattern(ctx, benefit), subgroupSize(subgroupSize),
+ matchClustered(matchClustered), target(target) {}
+
+ LogicalResult matchAndRewrite(gpu::SubgroupReduceOp op,
+ PatternRewriter &rewriter) const override {
+ if (op.getClusterSize().has_value() != matchClustered) {
+ return rewriter.notifyMatchFailure(
+ op, llvm::formatv("op is {0}clustered but pattern is configured to "
+ "only match {1}clustered ops",
+ matchClustered ? "non-" : "",
+ matchClustered ? "" : "non-"));
+ }
+ auto ci = gpu::getSubgroupReduceClusterInfo(op, subgroupSize);
+ if (failed(ci))
+ return failure();
+
+ if (ci->clusterStride != 1)
+ return rewriter.notifyMatchFailure(
+ op, "Subgroup reductions using DPP are currently only available for "
+ "clusters of contiguous lanes.");
+
+ Type valueTy = op.getType();
+ if (!valueTy.isIntOrFloat())
+ return rewriter.notifyMatchFailure(
+ op, "Value type is not a compatible scalar.");
+
+ FailureOr<Value> dpp = createSubgroupDPPReduction(
+ rewriter, op, op.getValue(), op.getOp(), *ci, target);
+ if (failed(dpp))
+ return failure();
+
+ rewriter.replaceOp(op, dpp.value());
+ return success();
+ }
+
+private:
+ unsigned subgroupSize = 0;
+ bool matchClustered = false;
+ ROCDL::TargetInfo target;
+};
+} // namespace
+
+void mlir::populateGpuLowerSubgroupReduceToDPPPatterns(
+ RewritePatternSet &patterns, unsigned subgroupSize,
+ const ROCDL::TargetInfo &target, PatternBenefit benefit) {
+ patterns.add<ScalarSubgroupReduceToDPP>(patterns.getContext(), subgroupSize,
+ /*matchClustered=*/false, target,
+ benefit);
+}
+
+void mlir::populateGpuLowerClusteredSubgroupReduceToDPPPatterns(
+ RewritePatternSet &patterns, unsigned subgroupSize,
+ const ROCDL::TargetInfo &target, PatternBenefit benefit) {
+ patterns.add<ScalarSubgroupReduceToDPP>(patterns.getContext(), subgroupSize,
+ /*matchClustered=*/true, target,
+ benefit);
+}
diff --git a/mlir/lib/Dialect/GPU/Transforms/SPIRVAttachTarget.cpp b/mlir/lib/Dialect/GPU/Transforms/SPIRV/SPIRVAttachTarget.cpp
similarity index 96%
rename from mlir/lib/Dialect/GPU/Transforms/SPIRVAttachTarget.cpp
rename to mlir/lib/Dialect/GPU/Transforms/SPIRV/SPIRVAttachTarget.cpp
index e4468ed6d28845..5955a0e372c4b2 100644
--- a/mlir/lib/Dialect/GPU/Transforms/SPIRVAttachTarget.cpp
+++ b/mlir/lib/Dialect/GPU/Transforms/SPIRV/SPIRVAttachTarget.cpp
@@ -11,7 +11,7 @@
//
//===----------------------------------------------------------------------===//
-#include "mlir/Dialect/GPU/Transforms/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/SPIRV/Passes.h"
#include "mlir/Dialect/GPU/IR/GPUDialect.h"
#include "mlir/Dialect/SPIRV/IR/SPIRVAttributes.h"
@@ -19,12 +19,11 @@
#include "mlir/Dialect/SPIRV/IR/TargetAndABI.h"
#include "mlir/IR/Builders.h"
#include "mlir/Pass/Pass.h"
-#include "mlir/Target/SPIRV/Target.h"
#include "llvm/Support/Regex.h"
namespace mlir {
#define GEN_PASS_DEF_GPUSPIRVATTACHTARGET
-#include "mlir/Dialect/GPU/Transforms/Passes.h.inc"
+#include "mlir/Dialect/GPU/Transforms/SPIRV/Passes.h.inc"
} // namespace mlir
using namespace mlir;
diff --git a/mlir/lib/Dialect/GPU/Transforms/SubgroupReduceLowering.cpp b/mlir/lib/Dialect/GPU/Transforms/SubgroupReduceLowering.cpp
index 1018306fc3cf28..d1076e5d3c2441 100644
--- a/mlir/lib/Dialect/GPU/Transforms/SubgroupReduceLowering.cpp
+++ b/mlir/lib/Dialect/GPU/Transforms/SubgroupReduceLowering.cpp
@@ -10,13 +10,10 @@
//
//===----------------------------------------------------------------------===//
-#include "mlir/Dialect/AMDGPU/IR/AMDGPUDialect.h"
#include "mlir/Dialect/Arith/IR/Arith.h"
#include "mlir/Dialect/GPU/IR/GPUDialect.h"
#include "mlir/Dialect/GPU/Transforms/Passes.h"
#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
-#include "mlir/Dialect/LLVMIR/ROCDLDialect.h"
-#include "mlir/Dialect/LLVMIR/ROCDLTargetInfo.h"
#include "mlir/Dialect/Vector/IR/VectorOps.h"
#include "mlir/IR/BuiltinTypes.h"
#include "mlir/IR/Location.h"
@@ -149,35 +146,6 @@ struct ScalarizeSingleElementReduce final
}
};
-struct ClusterInfo {
- unsigned clusterStride;
- unsigned clusterSize;
- unsigned subgroupSize;
-};
-
-static FailureOr<ClusterInfo>
-getAndValidateClusterInfo(gpu::SubgroupReduceOp op, unsigned subgroupSize) {
- assert(llvm::isPowerOf2_32(subgroupSize));
-
- std::optional<uint32_t> clusterSize = op.getClusterSize();
- assert(!clusterSize ||
- llvm::isPowerOf2_32(*clusterSize)); // Verifier should've caught this.
- if (clusterSize && *clusterSize > subgroupSize)
- return op.emitOpError()
- << "cluster size " << *clusterSize
- << " is greater than subgroup size " << subgroupSize;
- unsigned effectiveClusterSize = clusterSize.value_or(subgroupSize);
-
- auto clusterStride = op.getClusterStride();
- assert(llvm::isPowerOf2_32(clusterStride)); // Verifier should've caught this.
- if (clusterStride >= subgroupSize)
- return op.emitOpError()
- << "cluster stride " << clusterStride
- << " is not less than subgroup size " << subgroupSize;
-
- return ClusterInfo{clusterStride, effectiveClusterSize, subgroupSize};
-}
-
/// Emits a subgroup reduction using a sequence of shuffles. Uses the `packFn`
/// and `unpackFn` to convert to the native shuffle type and to the reduction
/// type, respectively. For example, with `input` of type `f16`, `packFn` could
@@ -188,7 +156,7 @@ getAndValidateClusterInfo(gpu::SubgroupReduceOp op, unsigned subgroupSize) {
/// lanes within a cluster, reducing all lanes in each cluster in parallel.
Value createSubgroupShuffleReduction(OpBuilder &builder, Location loc,
Value input, gpu::AllReduceOperation mode,
- const ClusterInfo &ci,
+ const gpu::SubgroupReduceClusterInfo &ci,
function_ref<Value(Value)> packFn,
function_ref<Value(Value)> unpackFn) {
// Lane value always stays in the original type. We use it to perform arith
@@ -229,7 +197,7 @@ struct ScalarSubgroupReduceToShuffles final
matchClustered ? "" : "non-"));
}
- auto ci = getAndValidateClusterInfo(op, subgroupSize);
+ auto ci = gpu::getSubgroupReduceClusterInfo(op, subgroupSize);
if (failed(ci))
return failure();
@@ -296,7 +264,7 @@ struct VectorSubgroupReduceToShuffles final
matchClustered ? "" : "non-"));
}
- auto ci = getAndValidateClusterInfo(op, subgroupSize);
+ auto ci = gpu::getSubgroupReduceClusterInfo(op, subgroupSize);
if (failed(ci))
return failure();
@@ -366,200 +334,6 @@ struct VectorSubgroupReduceToShuffles final
unsigned shuffleBitwidth = 0;
bool matchClustered = false;
};
-
-static FailureOr<Value>
-createSubgroupDPPReduction(PatternRewriter &rewriter, gpu::SubgroupReduceOp op,
- Value input, gpu::AllReduceOperation mode,
- const ClusterInfo &ci,
- const ROCDL::TargetInfo &target) {
- Location loc = op.getLoc();
- Value dpp;
- Value res = input;
- constexpr int allRows = 0xf;
- constexpr int allBanks = 0xf;
- const bool boundCtrl = true;
- if (ci.clusterSize >= 2) {
- // Perform reduction between all lanes N <-> N+1.
- dpp = amdgpu::DPPOp::create(
- rewriter, loc, res.getType(), res, res, amdgpu::DPPPerm::quad_perm,
- rewriter.getI32ArrayAttr({1, 0, 3, 2}), allRows, allBanks, boundCtrl);
- res = vector::makeArithReduction(rewriter, loc,
- gpu::convertReductionKind(mode), res, dpp);
- }
-
- if (ci.clusterSize >= 4) {
- // Perform reduction between all lanes N <-> N+2.
- dpp = amdgpu::DPPOp::create(
- rewriter, loc, res.getType(), res, res, amdgpu::DPPPerm::quad_perm,
- rewriter.getI32ArrayAttr({2, 3, 0, 1}), allRows, allBanks, boundCtrl);
- res = vector::makeArithReduction(rewriter, loc,
- gpu::convertReductionKind(mode), res, dpp);
- }
- if (ci.clusterSize >= 8) {
- // Perform reduction between all lanes N <-> 7-N,
- // e.g lane[0] <-> lane[7], lane[1] <-> lane[6]..., lane[3] <-> lane[4].
- dpp = amdgpu::DPPOp::create(rewriter, loc, res.getType(), res, res,
- amdgpu::DPPPerm::row_half_mirror,
- rewriter.getUnitAttr(), allRows, allBanks,
- boundCtrl);
- res = vector::makeArithReduction(rewriter, loc,
- gpu::convertReductionKind(mode), res, dpp);
- }
- if (ci.clusterSize >= 16) {
- // Perform reduction between all lanes N <-> 15-N,
- // e.g lane[0] <-> lane[15], lane[1] <-> lane[14]..., lane[7] <-> lane[8].
- dpp = amdgpu::DPPOp::create(
- rewriter, loc, res.getType(), res, res, amdgpu::DPPPerm::row_mirror,
- rewriter.getUnitAttr(), allRows, allBanks, boundCtrl);
- res = vector::makeArithReduction(rewriter, loc,
- gpu::convertReductionKind(mode), res, dpp);
- }
- if (ci.clusterSize >= 32) {
- if (!target.has(llvm::AMDGPU::FEAT_GFX10_INSTS)) {
- // Broadcast last value from each row to next row.
- // Use row mask to avoid polluting row 0 (and row 2 if wave-64).
- dpp = amdgpu::DPPOp::create(rewriter, loc, res.getType(), res, res,
- amdgpu::DPPPerm::row_bcast_15,
- rewriter.getUnitAttr(), 0xa, allBanks,
- /*bound_ctrl*/ false);
- res = vector::makeArithReduction(
- rewriter, loc, gpu::convertReductionKind(mode), res, dpp);
-
- // For subgroupSize = 64, at this point lanes [16, 32) contain the full
- // reduction over lanes [0, 32), but lanes [0, 16) do not. Similarly,
- // lanes [48, 64) contain the full reduction over lanes [32, 64), but
- // lanes [32, 48) do not.
- //
- // If subgroup size is 64 and cluster size is 64, we don't need lanes [0,
- // 16) and [32, 48) to have the correct cluster-32 reduction values at
- // this point, because only lane 63's value will ultimately be read in
- // this full-cluster case.
- //
- // If subgroup size is 64 and cluster size is 32, we need to ensure that
- // lanes [0, 16) and [32, 48) have the correct final cluster-32 reduction
- // values (subgroup_reduce guarantees that all lanes within each cluster
- // contain the final reduction value). We do this by broadcasting lane
- // 31's value to lanes [0, 16) and lanes 63's value to lanes [32, 48).
- //
- // See https://gpuopen.com/learn/amd-gcn-assembly-cross-lane-operations
- // for an illustration of how this within-cluster broadcast works with a
- // swizzle.
- if (ci.subgroupSize == 64 && ci.clusterSize == 32) {
- res =
- amdgpu::SwizzleBitModeOp::create(rewriter, loc, res, /*and_mask=*/0,
- /*or_mask=*/31,
- /*xor_mask=*/0);
- }
- } else if (!target.has(llvm::AMDGPU::FEAT_GFX13_INSTS)) {
- // Use a permute lane to cross rows (row 1 <-> row 0, row 3 <-> row 2).
- Value uint32Max = arith::ConstantOp::create(
- rewriter, loc, rewriter.getI32Type(), rewriter.getI32IntegerAttr(-1));
- dpp = ROCDL::PermlaneX16Op::create(rewriter, loc, res.getType(), res, res,
- uint32Max, uint32Max,
- /*fi=*/true,
- /*boundControl=*/false);
- res = vector::makeArithReduction(
- rewriter, loc, gpu::convertReductionKind(mode), res, dpp);
- } else {
- return rewriter.notifyMatchFailure(
- op, "Subgroup reduce lowering to DPP not currently supported for "
- "this device.");
- }
- if (ci.subgroupSize == 32) {
- Value lane31 = arith::ConstantOp::create(
- rewriter, loc, rewriter.getI32Type(), rewriter.getI32IntegerAttr(31));
- res =
- ROCDL::ReadlaneOp::create(rewriter, loc, res.getType(), res, lane31);
- }
- }
- if (ci.clusterSize >= 64) {
- if (!target.has(llvm::AMDGPU::FEAT_GFX10_INSTS)) {
- // Broadcast 31st lane value to rows 2 and 3.
- dpp = amdgpu::DPPOp::create(rewriter, loc, res.getType(), res, res,
- amdgpu::DPPPerm::row_bcast_31,
- rewriter.getUnitAttr(), 0xf, allBanks,
- /*bound_ctrl*/ true);
- res = vector::makeArithReduction(
- rewriter, loc, gpu::convertReductionKind(mode), dpp, res);
- // Obtain reduction from last rows, the previous rows are polluted.
- Value lane63 = arith::ConstantOp::create(
- rewriter, loc, rewriter.getI32Type(), rewriter.getI32IntegerAttr(63));
- res =
- ROCDL::ReadlaneOp::create(rewriter, loc, res.getType(), res, lane63);
-
- } else if (!target.has(llvm::AMDGPU::FEAT_GFX13_INSTS)) {
- // Assume reduction across 32 lanes has been done.
- // Perform final reduction manually by summing values in lane 0 and
- // lane 32.
- Value lane31 = arith::ConstantOp::create(
- rewriter, loc, rewriter.getI32Type(), rewriter.getI32IntegerAttr(31));
- Value lane63 = arith::ConstantOp::create(
- rewriter, loc, rewriter.getI32Type(), rewriter.getI32IntegerAttr(63));
- lane31 =
- ROCDL::ReadlaneOp::create(rewriter, loc, res.getType(), res, lane31);
- lane63 =
- ROCDL::ReadlaneOp::create(rewriter, loc, res.getType(), res, lane63);
- res = vector::makeArithReduction(
- rewriter, loc, gpu::convertReductionKind(mode), lane31, lane63);
- } else {
- return rewriter.notifyMatchFailure(
- op, "Subgroup reduce lowering to DPP not currently supported for "
- "this device.");
- }
- }
- assert(res.getType() == input.getType());
- return res;
-}
-
-/// Collect a set of patterns to lower `gpu.subgroup_reduce` into `amdgpu.dpp`
-/// ops over scalar types. Assumes that the subgroup has
-/// `subgroupSize` lanes. Applicable only to AMD GPUs.
-struct ScalarSubgroupReduceToDPP final
- : OpRewritePattern<gpu::SubgroupReduceOp> {
- ScalarSubgroupReduceToDPP(MLIRContext *ctx, unsigned subgroupSize,
- bool matchClustered,
- const ROCDL::TargetInfo &target,
- PatternBenefit benefit)
- : OpRewritePattern(ctx, benefit), subgroupSize(subgroupSize),
- matchClustered(matchClustered), target(target) {}
-
- LogicalResult matchAndRewrite(gpu::SubgroupReduceOp op,
- PatternRewriter &rewriter) const override {
- if (op.getClusterSize().has_value() != matchClustered) {
- return rewriter.notifyMatchFailure(
- op, llvm::formatv("op is {0}clustered but pattern is configured to "
- "only match {1}clustered ops",
- matchClustered ? "non-" : "",
- matchClustered ? "" : "non-"));
- }
- auto ci = getAndValidateClusterInfo(op, subgroupSize);
- if (failed(ci))
- return failure();
-
- if (ci->clusterStride != 1)
- return rewriter.notifyMatchFailure(
- op, "Subgroup reductions using DPP are currently only available for "
- "clusters of contiguous lanes.");
-
- Type valueTy = op.getType();
- if (!valueTy.isIntOrFloat())
- return rewriter.notifyMatchFailure(
- op, "Value type is not a compatible scalar.");
-
- FailureOr<Value> dpp = createSubgroupDPPReduction(
- rewriter, op, op.getValue(), op.getOp(), *ci, target);
- if (failed(dpp))
- return failure();
-
- rewriter.replaceOp(op, dpp.value());
- return success();
- }
-
-private:
- unsigned subgroupSize = 0;
- bool matchClustered = false;
- ROCDL::TargetInfo target;
-};
} // namespace
void mlir::populateGpuBreakDownSubgroupReducePatterns(
@@ -570,22 +344,6 @@ void mlir::populateGpuBreakDownSubgroupReducePatterns(
patterns.add<ScalarizeSingleElementReduce>(patterns.getContext(), benefit);
}
-void mlir::populateGpuLowerSubgroupReduceToDPPPatterns(
- RewritePatternSet &patterns, unsigned subgroupSize,
- const ROCDL::TargetInfo &target, PatternBenefit benefit) {
- patterns.add<ScalarSubgroupReduceToDPP>(patterns.getContext(), subgroupSize,
- /*matchClustered=*/false, target,
- benefit);
-}
-
-void mlir::populateGpuLowerClusteredSubgroupReduceToDPPPatterns(
- RewritePatternSet &patterns, unsigned subgroupSize,
- const ROCDL::TargetInfo &target, PatternBenefit benefit) {
- patterns.add<ScalarSubgroupReduceToDPP>(patterns.getContext(), subgroupSize,
- /*matchClustered=*/true, target,
- benefit);
-}
-
void mlir::populateGpuLowerSubgroupReduceToShufflePatterns(
RewritePatternSet &patterns, unsigned subgroupSize,
unsigned shuffleBitwidth, PatternBenefit benefit) {
diff --git a/mlir/lib/Dialect/GPU/Transforms/XeVMAttachTarget.cpp b/mlir/lib/Dialect/GPU/Transforms/XeVM/XeVMAttachTarget.cpp
similarity index 95%
rename from mlir/lib/Dialect/GPU/Transforms/XeVMAttachTarget.cpp
rename to mlir/lib/Dialect/GPU/Transforms/XeVM/XeVMAttachTarget.cpp
index 6da76e9e7a331d..f7fea91b5ea99d 100644
--- a/mlir/lib/Dialect/GPU/Transforms/XeVMAttachTarget.cpp
+++ b/mlir/lib/Dialect/GPU/Transforms/XeVM/XeVMAttachTarget.cpp
@@ -11,18 +11,17 @@
//
//===----------------------------------------------------------------------===//
-#include "mlir/Dialect/GPU/Transforms/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/XeVM/Passes.h"
#include "mlir/Dialect/GPU/IR/GPUDialect.h"
#include "mlir/Dialect/LLVMIR/XeVMDialect.h"
#include "mlir/IR/Builders.h"
#include "mlir/Pass/Pass.h"
-#include "mlir/Target/LLVM/XeVM/Target.h"
#include "llvm/Support/Regex.h"
namespace mlir {
#define GEN_PASS_DEF_GPUXEVMATTACHTARGET
-#include "mlir/Dialect/GPU/Transforms/Passes.h.inc"
+#include "mlir/Dialect/GPU/Transforms/XeVM/Passes.h.inc"
} // namespace mlir
using namespace mlir;
diff --git a/mlir/lib/Dialect/GPU/Utils/Utils.cpp b/mlir/lib/Dialect/GPU/Utils/Utils.cpp
index 8edd1ae904b6b9..562d8a259ba5bc 100644
--- a/mlir/lib/Dialect/GPU/Utils/Utils.cpp
+++ b/mlir/lib/Dialect/GPU/Utils/Utils.cpp
@@ -12,6 +12,7 @@
#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
#include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/MathExtras.h"
namespace mlir::gpu {
@@ -43,4 +44,28 @@ vector::CombiningKind convertReductionKind(gpu::AllReduceOperation mode) {
llvm_unreachable("Vector and GPU reduction kinds should match 1:1");
}
+FailureOr<SubgroupReduceClusterInfo>
+getSubgroupReduceClusterInfo(SubgroupReduceOp op, unsigned subgroupSize) {
+ assert(llvm::isPowerOf2_32(subgroupSize));
+
+ std::optional<uint32_t> clusterSize = op.getClusterSize();
+ assert(!clusterSize ||
+ llvm::isPowerOf2_32(*clusterSize)); // Verifier should've caught this.
+ if (clusterSize && *clusterSize > subgroupSize)
+ return op.emitOpError()
+ << "cluster size " << *clusterSize
+ << " is greater than subgroup size " << subgroupSize;
+ unsigned effectiveClusterSize = clusterSize.value_or(subgroupSize);
+
+ auto clusterStride = op.getClusterStride();
+ assert(llvm::isPowerOf2_32(clusterStride)); // Verifier should've caught this.
+ if (clusterStride >= subgroupSize)
+ return op.emitOpError()
+ << "cluster stride " << clusterStride
+ << " is not less than subgroup size " << subgroupSize;
+
+ return SubgroupReduceClusterInfo{clusterStride, effectiveClusterSize,
+ subgroupSize};
+}
+
} // namespace mlir::gpu
diff --git a/mlir/lib/Dialect/SparseTensor/Pipelines/CMakeLists.txt b/mlir/lib/Dialect/SparseTensor/Pipelines/CMakeLists.txt
index c724b49762a1a3..ed82967c895380 100644
--- a/mlir/lib/Dialect/SparseTensor/Pipelines/CMakeLists.txt
+++ b/mlir/lib/Dialect/SparseTensor/Pipelines/CMakeLists.txt
@@ -12,6 +12,7 @@ add_mlir_dialect_library(MLIRSparseTensorPipelines
MLIRComplexToLibm
MLIRComplexToStandard
MLIRFuncTransforms
+ MLIRGPUNVVMTransforms
MLIRGPUToNVVMTransforms
MLIRGPUTransforms
MLIRLinalgTransforms
diff --git a/mlir/lib/Dialect/SparseTensor/Pipelines/SparseTensorPipelines.cpp b/mlir/lib/Dialect/SparseTensor/Pipelines/SparseTensorPipelines.cpp
index c95ecdf20dc220..abee1a20b1a89e 100644
--- a/mlir/lib/Dialect/SparseTensor/Pipelines/SparseTensorPipelines.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Pipelines/SparseTensorPipelines.cpp
@@ -22,6 +22,7 @@
#include "mlir/Dialect/Bufferization/Transforms/Passes.h"
#include "mlir/Dialect/Func/IR/FuncOps.h"
#include "mlir/Dialect/GPU/IR/GPUDialect.h"
+#include "mlir/Dialect/GPU/Transforms/NVVM/Passes.h"
#include "mlir/Dialect/GPU/Transforms/Passes.h"
#include "mlir/Dialect/Linalg/Passes.h"
#include "mlir/Dialect/MemRef/Transforms/Passes.h"
diff --git a/mlir/lib/RegisterAllPasses.cpp b/mlir/lib/RegisterAllPasses.cpp
index 76aac7ae1f1823..6fd72482425c49 100644
--- a/mlir/lib/RegisterAllPasses.cpp
+++ b/mlir/lib/RegisterAllPasses.cpp
@@ -25,7 +25,11 @@
#include "mlir/Dialect/EmitC/Transforms/Passes.h"
#include "mlir/Dialect/Func/Transforms/Passes.h"
#include "mlir/Dialect/GPU/Pipelines/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/NVVM/Passes.h"
#include "mlir/Dialect/GPU/Transforms/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/ROCDL/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/SPIRV/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/XeVM/Passes.h"
#include "mlir/Dialect/LLVMIR/Transforms/Passes.h"
#include "mlir/Dialect/Linalg/Passes.h"
#include "mlir/Dialect/MLProgram/Transforms/Passes.h"
@@ -73,6 +77,10 @@ void mlir::registerAllPasses() {
bufferization::registerBufferizationPasses();
func::registerFuncPasses();
registerGPUPasses();
+ registerGPUNVVMPasses();
+ registerGPUROCDLPasses();
+ registerGPUSPIRVPasses();
+ registerGPUXeVMPasses();
registerLinalgPasses();
registerNVGPUPasses();
NVVM::registerNVVMPasses();
diff --git a/mlir/test/lib/Dialect/GPU/CMakeLists.txt b/mlir/test/lib/Dialect/GPU/CMakeLists.txt
index 882d5abc2eeb84..0996604e80264d 100644
--- a/mlir/test/lib/Dialect/GPU/CMakeLists.txt
+++ b/mlir/test/lib/Dialect/GPU/CMakeLists.txt
@@ -9,6 +9,7 @@ set(LIBS
MLIRBuiltinToLLVMIRTranslation
MLIRFuncDialect
MLIRGPUDialect
+ MLIRGPUROCDLTransforms
MLIRGPUToGPURuntimeTransforms
MLIRGPUTransforms
MLIRIR
diff --git a/mlir/test/lib/Dialect/GPU/TestGpuRewrite.cpp b/mlir/test/lib/Dialect/GPU/TestGpuRewrite.cpp
index 4b620496d6b410..6dceaa7abb5645 100644
--- a/mlir/test/lib/Dialect/GPU/TestGpuRewrite.cpp
+++ b/mlir/test/lib/Dialect/GPU/TestGpuRewrite.cpp
@@ -14,6 +14,7 @@
#include "mlir/Dialect/Arith/IR/Arith.h"
#include "mlir/Dialect/Func/IR/FuncDialect.h"
#include "mlir/Dialect/GPU/Transforms/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/ROCDL/Passes.h"
#include "mlir/Dialect/Index/IR/IndexDialect.h"
#include "mlir/Dialect/LLVMIR/ROCDLDialectDecl.h"
#include "mlir/Dialect/MemRef/IR/MemRefDialect.h"
diff --git a/mlir/test/lib/Pass/CMakeLists.txt b/mlir/test/lib/Pass/CMakeLists.txt
index 04c91635def85c..5ed1cc4b94db92 100644
--- a/mlir/test/lib/Pass/CMakeLists.txt
+++ b/mlir/test/lib/Pass/CMakeLists.txt
@@ -18,6 +18,7 @@ add_mlir_library(MLIRTestPass
)
mlir_target_link_libraries(MLIRTestPass PUBLIC
${conversion_libs}
+ MLIRGPUSPIRVTransforms
MLIRIR
MLIRSPIRVTransforms
MLIRPass
diff --git a/mlir/test/lib/Pass/TestVulkanRunnerPipeline.cpp b/mlir/test/lib/Pass/TestVulkanRunnerPipeline.cpp
index e30c31693fae7a..83252a4f0c789c 100644
--- a/mlir/test/lib/Pass/TestVulkanRunnerPipeline.cpp
+++ b/mlir/test/lib/Pass/TestVulkanRunnerPipeline.cpp
@@ -17,6 +17,7 @@
#include "mlir/Dialect/Func/IR/FuncOps.h"
#include "mlir/Dialect/GPU/IR/GPUDialect.h"
#include "mlir/Dialect/GPU/Transforms/Passes.h"
+#include "mlir/Dialect/GPU/Transforms/SPIRV/Passes.h"
#include "mlir/Dialect/LLVMIR/Transforms/RequestCWrappers.h"
#include "mlir/Dialect/MemRef/Transforms/Passes.h"
#include "mlir/Dialect/SPIRV/IR/SPIRVOps.h"
More information about the Mlir-commits
mailing list