<table border="1" cellspacing="0" cellpadding="8">
<tr>
<th>Issue</th>
<td>
<a href=https://github.com/llvm/llvm-project/issues/163882>163882</a>
</td>
</tr>
<tr>
<th>Summary</th>
<td>
[DirectX] Mismatched data layout causing validation errors about exceeding TGSM storage
</td>
</tr>
<tr>
<th>Labels</th>
<td>
new issue
</td>
</tr>
<tr>
<th>Assignees</th>
<td>
</td>
</tr>
<tr>
<th>Reporter</th>
<td>
Icohedron
</td>
</tr>
</table>
<pre>
99 DML shaders are [failing to validate](https://microsoft.visualstudio.com/WindowsAI/_build/results?buildId=131899824&view=ms.vss-test-web.build-test-results-tab) after #163587 with the error:
```
error: Total Thread Group Shared Memory storage is 43688, exceeded 32768.
Validation failed.
```
All 99 DML shaders have names of the form `QuantizedGemm*`. (e.g., `QuantizedGemm_20480_16_0_uint4_packed32_float16_native_accum32_0`)
## Minimal reproducible test case
```hlsl
// compile args: -T cs_6_7 -E CSMain -enable-16bit-types -Fo output.dat
groupshared float16_t smem[10240];
[numthreads(1, 1, 1)]
void CSMain() {
smem[0] = 0;
}
```
Comparing the dxil output before and after the PR commit c87e0e8fe0ea14dcd84e835c0f7b02c5b0edca70, the only difference is the data layout.
```
1c1
< target datalayout = "e-m:e-p:32:32-i1:32-i8:8-i16:16-i32:32-i64:64-f16:16-f32:32-f64:64-n8:16:32:64"
---
> target datalayout = "e-m:e-p:32:32-i1:8-i8:8-i16:32-i32:32-i64:64-f16:32-f32:32-f64:64-n8:16:32:64"
270c270
< !1 = !{!"clang version 22.0.0git (git@github.com:Icohedron/llvm-project.git 72c6e4b230ddb5ca85361e145e177245319b271e)"}
---
> !1 = !{!"clang version 22.0.0git (git@github.com:Icohedron/llvm-project.git c87e0e8fe0ea14dcd84e835c0f7b02c5b0edca70)"}
```
Compiling the same DML shader with DXC, DXC gives the shader a datalayout of
```
target datalayout = "e-m:e-p:32:32-i1:32-i8:8-i16:16-i32:32-i64:64-f16:16-f32:32-f64:64-n8:16:32:64"
```
which matches the data layout that Clang emitted before the PR.
</pre>
<img width="1" height="1" alt="" src="http://email.email.llvm.org/o/eJzMlm-L47YTx1-N8mSwkUb-lwd54E0ux8Fv4dfe0u6zIEvjWK1tBUvO3vbVFznJ7W5K4SgUCsHB0mj01cxnRlbe2-NItGH5A8t3KzWHzk2bL9p1ZCY3rhpnXjfrNewe_we-U4YmD2oiYPlDq2xvxyMEB2fVW6MCsXzHsOpCOHkma4Z7hvvB6sl514b0bP2seh9mY12q3cBw_6sdjXvx9ReG-0Mz294w3E_k5z54JvfLyBfD5E5IUa3XFWYMi7OlFyZ3g0_P3ieBfEheqEkX48vr1UMSVMNwDaoNNAFDKQqZVyW82NBB6AhomtwUlfKaFfz64_VtGJ5cUD08dRMpA58nN5_ga6cmMvBIg5tewQc3qSOB9ZDJoqoYboG-aSJDBiSWRZUyXv9yCY91I8SgkUnvNmS8rvse7uLcqTPBqAby4NpFb-umAVjBf5rVGOwfZD7TMDCMvlJgWFF6TKOEe5MD8qziB1Ec-GG2Y8gOJ6V_JyPx0PZOBVEcRhXsmQ5K63mQeFiE4fqijaFkKOHRjnZQPUx0mpyZtW16ghhv0MrT-yN1ve-XdREA0G442Z5ATceIBSRPoP2hOJSQfILt10dlR0hoVE1PiSgaG5LweiIPyd6Bm8NpDqlRgfH6GDPgLwm46Q7gBxpY_iA4ZjwCKB_i1vnDOA9hyZxnWIkYlttjzfIdMF6fnTVXAQyrSAor42K4-eSLodwBv3otd3eZ27rhpKalDDoC8832V8nQUOsmAjWaK3_R4P8_x2gMNoCuSuJUtcRJicxoU2VUyVzztmw46rzhZLQqeZQcV7qxfwVj25YmGvVC3LKjCgp69ermcA-V0CKOyC0ENR0pLLYX0-VMDJGSgcmakhOTtcTlkVhx_a-YrKvEioLJWhSJ_W5QZEzWRZa0t6n2NtXepsZqmbm5LTKGyHidJMki6dM_kFR9VBSH_0ZRFPKjirDkGkt-DRRDIa5CREQBBUPUvRqPcKbJxwJGTHnKjzbEejvawLL40s3N0tFk_b11Mtz3_XlITpP7jXRI45ISdUFZg5Ib0-RaVbksBIksJ1GWmOVSrBssBUVEES-0vcXs31b340i-qYO7NgbbpdhvBeHVQO962qXz7p63kerd8xaO9kwXkK8G6j0SroX7Lf47LH-U9dJZ3cGggu7oL5UJoVMBtkumaLAhkLm1h0tTSFdmI81artWKNqIseJFnoixW3SbPMiOQtFFYFcZUpcrVWqNSZcFJt2plN8gxF1wUiLIQRdpg1ZQZNzkpKvJCsYzToGyfxnynbjqurPczbUQhqwpXvWqo98v9jzjSCyyz8Yj5bjVtFkia-ehZxnvrg39zE2zolw-HnZ1Ih-fYKx-tv8TAfDi_VrOPSJzfLsLljvWgmjh_uTKjxdPnr4-3W3U1T_3m48fEO5gvAN9xzHC_6PcM99cDnjf4ZwAAAP__4AmjPw">