[llvm] 1b34722 - [AMDGPU] Fix computation of waves/EU maximum (#140921)

[via llvm-commits]

Author: Lucas Ramirez
Date: 2025-05-22T02:45:59+02:00
New Revision: 1b347223023ddbe6c52dc1d48ec90a52407c4ca0

URL: https://github.com/llvm/llvm-project/commit/1b347223023ddbe6c52dc1d48ec90a52407c4ca0
DIFF: https://github.com/llvm/llvm-project/commit/1b347223023ddbe6c52dc1d48ec90a52407c4ca0.diff

LOG: [AMDGPU] Fix computation of waves/EU maximum (#140921)

This fixes an issue in the waves/EU range calculation wherein, if the
`amdgpu-waves-per-eu` attribute exists and is valid, the entire
attribute may be spuriously and completely ignored if workgroup sizes
and LDS usage restrict the maximum achievable occupancy below the
subtarget maximum. In such cases, we should still honor the requested
minimum number of waves/EU, even if the requested maximum is higher than
the actually achievable maximum (but still within subtarget
specification).

As such, the added unit test `empty_at_least_2_lds_limited`'s waves/EU
range should be [2,4] after this patch, when it is currently [1,4] (i.e,
as if `amdgpu-waves-per-eu` was not specified at all).

Before e377dc4 the default maximum waves/EU was always set to the
subtarget maximum, trivially avoiding the issue.

Added: 
    

Modified: 
    llvm/lib/Target/AMDGPU/AMDGPUSubtarget.cpp
    llvm/test/CodeGen/AMDGPU/attr-amdgpu-waves-per-eu.ll

Removed: 
    


################################################################################
diff  --git a/llvm/lib/Target/AMDGPU/AMDGPUSubtarget.cpp b/llvm/lib/Target/AMDGPU/AMDGPUSubtarget.cpp
index 776cc6258dbcd..d095fc6cf9549 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUSubtarget.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUSubtarget.cpp
@@ -191,17 +191,17 @@ std::pair<unsigned, unsigned> AMDGPUSubtarget::getEffectiveWavesPerEU(
       getOccupancyWithWorkGroupSizes(LDSBytes, FlatWorkGroupSizes).second};
   Default.first = std::min(Default.first, Default.second);
 
-  // Make sure requested minimum is less than requested maximum.
-  if (RequestedWavesPerEU.second &&
-      RequestedWavesPerEU.first > RequestedWavesPerEU.second)
-    return Default;
-
-  // Make sure requested values do not violate subtarget's specifications and
-  // are compatible with values implied by minimum/maximum flat workgroup sizes.
+  // Make sure requested minimum is within the default range and lower than the
+  // requested maximum. The latter must not violate target specification.
   if (RequestedWavesPerEU.first < Default.first ||
-      RequestedWavesPerEU.second > Default.second)
+      RequestedWavesPerEU.first > Default.second ||
+      RequestedWavesPerEU.first > RequestedWavesPerEU.second ||
+      RequestedWavesPerEU.second > getMaxWavesPerEU())
     return Default;
 
+  // We cannot exceed maximum occupancy implied by flat workgroup size and LDS.
+  RequestedWavesPerEU.second =
+      std::min(RequestedWavesPerEU.second, Default.second);
   return RequestedWavesPerEU;
 }
 

diff  --git a/llvm/test/CodeGen/AMDGPU/attr-amdgpu-waves-per-eu.ll b/llvm/test/CodeGen/AMDGPU/attr-amdgpu-waves-per-eu.ll
index 4507fd5865989..e9fe4f3c618c7 100644
--- a/llvm/test/CodeGen/AMDGPU/attr-amdgpu-waves-per-eu.ll
+++ b/llvm/test/CodeGen/AMDGPU/attr-amdgpu-waves-per-eu.ll
@@ -200,3 +200,28 @@ entry:
   ret void
 }
 attributes #10 = {"amdgpu-flat-work-group-size"="256,256" "amdgpu-waves-per-eu"="2,2"}
+
+; Minimum 2 waves, maximum limited by LDS usage.
+; CHECK-LABEL: {{^}}empty_at_least_2_lds_limited:
+; CHECK: SGPRBlocks: 12
+; CHECK: VGPRBlocks: 12
+; CHECK: NumSGPRsForWavesPerEU: 102
+; CHECK: NumVGPRsForWavesPerEU: 49
+define amdgpu_kernel void @empty_at_least_2_lds_limited() #11 {
+entry:
+  ret void
+}
+attributes #11 = {"amdgpu-flat-work-group-size"="1,256" "amdgpu-waves-per-eu"="2" "amdgpu-lds-size"="16384"}
+
+; Minimum 2 waves, maximum limited by LDS usage. Requested maximum within spec
+; but above achievable occupancy has no effect.
+; CHECK-LABEL: {{^}}empty_at_least_2_lds_limited_max_above_achievable:
+; CHECK: SGPRBlocks: 12
+; CHECK: VGPRBlocks: 12
+; CHECK: NumSGPRsForWavesPerEU: 102
+; CHECK: NumVGPRsForWavesPerEU: 49
+define amdgpu_kernel void @empty_at_least_2_lds_limited_max_above_achievable() #12 {
+entry:
+  ret void
+}
+attributes #12 = {"amdgpu-flat-work-group-size"="1,256" "amdgpu-waves-per-eu"="2,10" "amdgpu-lds-size"="16384"}