<table border="1" cellspacing="0" cellpadding="8">
<tr>
<th>Issue</th>
<td>
<a href=https://github.com/llvm/llvm-project/issues/56323>56323</a>
</td>
</tr>
<tr>
<th>Summary</th>
<td>
--vector-to-rocdl broken on gfx1030 backend
</td>
</tr>
<tr>
<th>Labels</th>
<td>
bug,
backend:AMDGPU,
mlir:gpu,
mlir:vectorops,
mlir
</td>
</tr>
<tr>
<th>Assignees</th>
<td>
</td>
</tr>
<tr>
<th>Reporter</th>
<td>
krzysz00
</td>
</tr>
</table>
<pre>
# Symptoms
```
$ ninja check-mlir
[0/1] Running the MLIR regression tests
-- Testing: 1488 tests, 32 workers --
Testing: 0.. 10.. 20.. 30..
FAIL: MLIR :: Integration/GPU/ROCM/vector-transferops.mlir (91 of 1488)
******************** TEST 'MLIR :: Integration/GPU/ROCM/vector-transferops.mlir' FAILED ********************
Script:
--
: 'RUN: at line 1'; /home/kdrewnia/llvm-project/build/bin/mlir-opt /home/kdrewnia/llvm-project/mlir/test/Integration/GPU/ROCM/vector-transferops.mlir -convert-scf-to-cf -gpu-kernel-outlining -pass-pipeline='gpu.module(strip-debuginfo,convert-gpu-to-rocdl,gpu-to-hsaco{chip=gfx1030})' -gpu-to-llvm | /home/kdrewnia/llvm-project/build/bin/mlir-cpu-runner --shared-libs=/home/kdrewnia/llvm-project/build/lib/libmlir_rocm_runtime.so --shared-libs=/home/kdrewnia/llvm-project/build/lib/libmlir_runner_utils.so --entry-point-result=void | /home/kdrewnia/llvm-project/build/bin/FileCheck /home/kdrewnia/llvm-project/mlir/test/Integration/GPU/ROCM/vector-transferops.mlir
--
Exit Code: 1
Command Output (stderr):
--
/home/kdrewnia/llvm-project/mlir/test/Integration/GPU/ROCM/vector-transferops.mlir:82:12: error: CHECK: expected string not found in input
// CHECK: [1.23, 2.46, 2.46, 1.23]
^
<stdin>:1:1: note: scanning from here
Unranked Memref base@ = 0x75ad310 rank = 1 offset = 0 sizes = [4] strides = [1] data =
^
<stdin>:2:1: note: possible intended match here
[1.23, 1.23, 1.23, 1.23]
^
Input file: <stdin>
Check file: /home/kdrewnia/llvm-project/mlir/test/Integration/GPU/ROCM/vector-transferops.mlir
-dump-input=help explains the following input dump.
Input was:
<<<<<<
1: Unranked Memref base@ = 0x75ad310 rank = 1 offset = 0 sizes = [4] strides = [1] data =
check:82'0 X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
2: [1.23, 1.23, 1.23, 1.23]
check:82'0 ~~~~~~~~~~~~~~~~~~~~~~~~~
check:82'1 ? possible intended match
3: Unranked Memref base@ = 0x75ad310 rank = 1 offset = 0 sizes = [4] strides = [1] data =
check:82'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4: [1.23, 1.23, 1.23, 1.23]
check:82'0 ~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>
--
********************
Testing: 0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
********************
Failed Tests (1):
MLIR :: Integration/GPU/ROCM/vector-transferops.mlir
Testing Time: 26.89s
Unsupported: 181
Passed : 1365
Failed : 1
```
Note that only the transferops test fails - all other ROCm integration tests pass.
# The likely cause
In `--vector-to-rocdl`, the buffer description structure that is needed for certain memory instructions is defined using [certain magic constants](https://github.com/llvm/llvm-project/blob/main/mlir/lib/Conversion/VectorToROCDL/VectorToROCDL.cpp#L113)
These magic constants are correct for gfx9xx GPUs. However, the appropriate values of these constants is now backend-dependent. Specifically
- For gfx10xx chips (RDNA 2), bit 120 in the buffer descriptor must be 1. See [the RDNA manual, table 37 on page 74 (big PDF)](https://developer.amd.com/wp-content/resources/RDNA_Shader_ISA.pdf)
- For gfx9 chips, such as the CDNA line, bit 120 is reserved and must be 0. See [the CDNA v2, table 36 on page 74](https://developer.amd.com/wp-content/resources/CDNA2_Shader_ISA_18November2021.pdf)
This means that the intrinsic calls emitted by `--vector-to-rocdl` will be incorrect on one of the two backends (going off of my interpretation of the documentation as written).
Since `--vector-to-rocdl` doesn't know about the target GPU, there's no obvious way around this.
In the AMDGPU dialect, we use a `targetIsRDNA` attribute to determine whether to set or clear bit 120, but this isn't really a general solution. https://github.com/llvm/llvm-project/blob/main/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp#L166
# Potential fixes
1. The AMDGPU backend gains additional intrinsics that handle setting/clearing that bit at issue depending on the compilation target.
2. It turns out the documentation isn't quite correct and we could get away with setting the bit to 1 everywhere. This would require getting some clarity out of the hardware people though.
3. Remove this `--vector-to-rocdl` conversion.
# Configuration notes
**System:** Ubuntu 20.04, ROCm 5.2.0 installed from AMD repositories
**Build flags:**
``
`-DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++ -DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang -DLLVM_ENABLE_PROJECTS=mlir -DLLVM_TARGETS_TO_BUILD=X86\;AMDGPU -DLLVM_ENABLE_ASSERTIONS=ON -DMLIR_ENABLE_ROCM_RUNNER=ON -DMLIR_ENABLE_ROCM_CONVERSIONS=ON -DMLIR_ROCM_RUNNER_ENABLED=ON -DMLIR_INCLUDE_INTEGRATION_TESTS=ON -DCMAKE_BUILD_TYPE=Debug
```
```
$ rocm_agent_enumerator
gfx000
gfx1030
```
</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJztWcly4zgS_Rr5giCDi9aDD7YkV3vaW0hyRc1JAZKQxDZJcADQsurr-yVIavEy011VU1OHUdDgAiCRmXi5wZFMduedIGTzXV4ameuON-l4F52-11z1a9BlRVr8wVm8EfGTk2epYk1X79LrBFd-pzdhs6rAqDUzG8Fub65nTIm1ElqnsmBGaNMQdxy2wBtGdsIL5neHw6Y3GLMwYFupnoTSzHHq4Udjmee6zKcmoCakph50dXF9QyPssnig5-vCYH1usDw4_PTwiHZ2P77F7VnERirHKF7olVCy1G4tUzAc-UyuLFedYNSK_-0XW0znC9AdfB9nIMBIxumEfQ83tTzzWKWlIV6a_WjEBGdYZ_Z4R0_csCwtBPPxqRNeMoa-q43MBW5PiRLbIuV4zLLn3CmV_AN84zWq0iyhe0qSEeeOLM1fmlqLeUVQwO0bNg8_J5bFs1DG0fHKMdKJV_bruqwcYKoQmSMrA7EIpdRRcq2dMi0FidoJJ5AVY91cJlUGbofaQFNOIqJqnRYrCYS2CxBJLKBknGT43LxuNI9lZ3AZb9IS5NarF98Lvc5gQljCFrbMYChJD50Oxt-q1xh0FCxO1JI7esOVSJwsjbSV5K8TxZS6JbpLiJQvQdikuXC1_MHELcPLyqSZ3hMXhVE7p5RpYRz4iyoDNifPMk2-VT9XaSbG5Kl-CvBOrWj6kho2lomwzq2xLNuOZZ7zImH3lSkrsgnAKxFKETbe2OLP4Du8GAZofGoYGJH0iY1_m45_t19eSkwUCSMrgMEU0rCVrCBBWuCCDDWvpGRch4mICb4bhOTPA7fbP77b771JM_Hw6_SmrRMaQyvYxXBKrDV_tLZVqI55HWNWSuZsI5Sopz0WEO4JvN6KXIkVi7gWna4Hhzth3sugx5PQ9xiNsZ_Ix6-0MHU_0-lXoe0zWO9SKCORk8M3G94Sbnj9oYl877IcvGa5lIiAUSagMiOKBDzm3MSbI-aP9PXuvdXXYUXbXtMWsBWwbpV-xEUNN2sA--6fZggNkJMqL50aJeFkI7KS8JTxtNA2P1jJLJNb2kk7htFw9610W673tgEJX1-vYcSs5n8qGBo02MSotqhg4FlevnQG0_9fB8dSyAb51om83bvgle_4wBY-VviPZPztGn69RnjFPvp9YOlvJQ1_IZT-zwHyq1xvdqn7y-HRRpjT69jlOsdv318m_IfCq0tNj5o-NQNqhtSM0PwoHq444ldi60VNGZN_nCyx7yz3TtR1LDJbIPclmkHfHY50u9pjoauylAo5kVWKP_TbrgcUEuCz9hDI-8J-r-1qRGi7bJB6t8i27R2yBgRIVF-yyHY2VB6xbQtltgJF1MeMZxmTGKEY5Myt12kUUBfUjMob9xQUIVuAZJY-CVCPeaXFadBl4MdxWoW1BQ54BOCJmahagRUGR2OLSFoLjqeKTaUavlPNCiHI-62kYjGKJQR9lotcqh14rEdjnqaRiVih8kpYpUntMLb9eL5OY4ZqSxteGE05UDDcGFPafMAmnOvUbKrIjWXepDLvlAWZpOoj5_uyaV-RjG0hp2u0fLbyLiT0OLl5_e7GZQnF3fh-eDgSqOGyEVq8ZpWhUsKbUqBhVYAqcPTywoBI7bLf5FZg3VadvASzpUo5dv2ZZxX8t1xRjxZHFEmlcosIET8hrqAeLSm8FMZlc6To6SqNAYVd4wXYVb2m72FRKkSt4cwmd4CzLUPHLEKJ4gce5fHv7Cmm5xVgFqH4xwpC0L7QOEsDJUzFM8s_p1gXDoBUIG0t2KBLK0Xpmj1MrmipdzYtgfSZLIVyeZ40e7ctqW5HyKQtQwEoKxULTeaLBZfzDUeVtLyeX7hlstrvwF7OUS0kcaQrZBe8TjDHxKwt648F1gz0hXoG5KgUa-X0TuS0U5-DIxn7RzJ-t1REPjgSa-kP7-SzyCOhAi_wT6RscQbGc8Ft7gwTIyZh7ajKNCEPm6-ZyFNDxVq0-8iE2TaFw4hoagtPSCUL0UCOma1sMWYxs5ZklEg7aEC-sw5GlUqY2sc0sxIZVzmkrD9C-1tFnMCsRie-Z45lxYe8JVJoTBkY9kRQ55GsajkNV2tkPdafW5tB1RQMyCCYjJ5TWWFBvoPR2crUQFOvyghL5eJ2AgosSXlmPcOYbQWcDgyQOKrXuNaEN2KGG-g2qsgTS5gFpM7pKGy7Edbb4iMlYuTdMsFViy4LNMt1SiZbS6MEmSaWWYtCKJ4xLbOKNOWy_5o3q4U9eLOT97036_dfR4YHSWiFilA1vgh93A1HsDiosQEJW9tyjidJShJh3h6UDVA3MDLYD7RlM4ngyiqsPiNGNynORgxdAUfWq1nE1XsGVZRp1sQzu0PNzgYuu4aaK4XFW5icorDV_r-q1BycMVn8ll6rDLxjBzkhZwvFtxzW7hBcYYt9Ro56tyXEkfTY1K2dqQTIwsevmzkadTWQALnMzvLTGMaGq2RLwaAUsswoOspqvWlkCF02Q0h8FjVcPjKLeL-vb-I4tnyVrqsm4NNpQ7tjNn-a77QRuYWXPYd-jKrCVJS_eV1Cqk0Zem7gejYoA6UUselUBZsMGVHMpOAmPaV6SedsbJXxtT6QPkln2mdnMr69-H26HH_5shzf3z5c30xn9dGhLK1HlPER0utzOyiRUHKJi-3n_73ZmHdz8_l2Ob27uLyZLh9m9_-YjhdzzLVnxE3v4mL2abqYLxf3y8vH65sJur8M-53euBNeNhg_pXMxn09ni-v7O6J0f4deSj3bXkozl7PHuzvL5Pvd4_u7z9PZ_A2Jo7nN-MlJ__Xd-OZxMsV9Mf00uyAelvQ_hT2VWk1WjOXinw-oCiYTOrD-N0nmO__esae-CHCFWYoClgRYySZBRoz1PG__bE-0T8mcJedhMgpH_MykJhPnb6DMIiXhMciyGwqtDzmrVHb-tz2h9RgUS3v9MAjPNud-HI1Gg248jIUXRd1wlISi64ciWPliFA96ZxmPRKbPEeE7QUDaCWx8p5eaESxf7_yhpzkkXZfVm2-1fJLSjpMevPUmZ-k5Anng9UOE8143HLh-PBgKbzSMQhQGfTFCnS_gxjOX5HKlWp-pcysiONPozFKk74dOpPHpGkm1ZR_0eQVfos6f1Ned_up5Z1Yd51YXfwJubSG8">