<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/145924>145924</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            [DirectX] Scalarizer is producing GEP chains that fail validation
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
            bug
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          Icohedron
      </td>
    </tr>
</table>

<pre>
    The validator doesn't like GEP chains where an array GEP is followed by a scalar GEP.

If you compile the following LLVM IR to DXIL using `llc -filetype=obj -mtriple=dxil-pc-shadermodel6.3-library`
```llvm
define i32 @gep_chain() #0 {

    %1 = alloca [4 x i32], align 4
    %2 = getelementptr inbounds nuw [4 x i32], ptr %1, i32 0, i32 2
    %3 = getelementptr inbounds nuw i32, ptr %2, i32 1
    %4 = load i32, ptr %3
    ret i32 %4
}

attributes #0 = { convergent norecurse nounwind "hlsl.export"}
```
and then validate the resulting dxil using dxv, you get the following validation error (#140417):
```
Function: gep_chain: error: Access to out-of-bounds memory is disallowed.
note: at '%3 = getelementptr inbounds i32, i32* %2, i32 1' in block '#0' of function 'gep_chain'.
Validation failed.
```

These GEP chains can appear when compiling HLSL shaders such as this one: https://godbolt.org/z/ExeGKb7MG
The scalarizer pass is responsible for emitting the scalar GEP in this case, which itself occurs after the dxil-flatten-arrays pass that had attempted to collapse GEP chains. Therefore, we need a later pass, such as the dxil-legalize pass, or a new pass to collapse the GEP chains again.

I propose that to solve this cleanly, we should shift the responsibility of collapsing GEP chains from the dxil-flatten-arrays pass over to the dxil-legalize pass.

Another reason to move GEP-chain-collapsing over to the dxil-legalize pass is to consolidate i8 GEP transformations as well. Currently i8 GEPs can appear during dxil-data-scalarization and dxil-flatten-arrays (see #145780), which occurs before the dxil-legalize pass that is responsible for i8 GEP legalization. By moving the GEP-chain-collapsing logic from the dxil-flatten-arrays pass over to the dxil-legalize pass, the dxil-legalize pass can simultaneously collapse GEP chains and legalize any i8 and scalar GEPs in those GEP chains.

The dxil-flatten-arrays pass' visitGetElementPtrInst can be simplified down to the following:
```c++
/// This visitor simply converts GEPs on multidimensional arrays into GEPs on
/// flattened arrays. A later "GEP chain collapser" pass will be used to combine
/// GEP chains into single flat GEPs, including cases of i8 GEPs, and scalar
/// GEPs introduced by the scalarizer.
bool DXILFlattenArraysVisitor::visitGetElementPtrInst(GetElementPtrInst &GEP) {
 if (!isMultiDimensionalArray(GEP.getSourceElementType())) {
    return false;
  }

  ArrayType *ArrType = cast<ArrayType>(GEP.getSourceElementType());
  IRBuilder<> Builder(&GEP);
 auto [TotalElements, BaseType] = getElementCountAndType(ArrType);
 ArrayType *FlattenedArrayType = ArrayType::get(BaseType, TotalElements);
 Value *PtrOperand = GEP.getPointerOperand();

  Value *FlattenedIndex;

 const DataLayout &DL = GEP.getDataLayout();
  unsigned BitWidth = DL.getIndexTypeSizeInBits(GEP.getType());
  APInt ConstantOffset(BitWidth, 0);
  SmallMapVector<Value *, APInt, 4> VariableOffsets;
  bool Success = GEP.collectOffset(DL, BitWidth, VariableOffsets, ConstantOffset);
 (void)Success;
  assert(Success && "Failed to collect offsets for GEP");

  // GEP.collectOffset returns the offset in bytes. So we need to divide its
  // offsets by the size in bytes of the BaseType
  unsigned BaseTypeSizeInBytes = BaseType->getPrimitiveSizeInBits() / 8;

  Value *ZeroIndex = Builder.getInt({BitWidth, 0});
  FlattenedIndex = Builder.getInt(ConstantOffset.udiv(BaseTypeSizeInBytes));
  for (auto [VarIndex, Multiplier] : VariableOffsets){
    Value *ConstIntMul = Builder.getInt(Multiplier.udiv(BaseTypeSizeInBytes));
    Value *MulVarIndex = Builder.CreateMul(VarIndex, ConstIntMul);
    FlattenedIndex = Builder.CreateAdd(FlattenedIndex, MulVarIndex);
  }

  Value *NewGEP = Builder.CreateGEP(
      FlattenedArrayType, PtrOperand, {ZeroIndex, FlattenedIndex},
      GEP.getName(), GEP.getNoWrapFlags());

 GEP.replaceAllUsesWith(NewGEP);
  GEP.eraseFromParent();
  return true;
}
```
This will flatten all GEPs, but not collapse any GEP chains. Again, the collapsing of GEP chains could be handled by the dxil-legalize pass (or a new pass).



</pre>
<img width="1" height="1" alt="" src="http://email.email.llvm.org/o/eJykWF1vIrnS_jXOTQnUcfN5kQsCYd7oTXajzZzs6twcudsFeNfYyHaTYX79UdnupiGZ2ZWOhETjLj_19VS5jPBebQ3iHRvfs_HqRjRhZ93dY213KJ01N5WVp7uvO4Sj0EqKYB1Ii94wPg2g1V8IXx5eoN4JZTy879AhCAPCOXGKb5SHjdXavqOE6gQCfC20cPRuyIoFKxaPGzjZBmq7PyiNEHaYdyizhaent2d4_A2ChdUfj0_QeFplk0LrGgYbpTGcDsjKla3-hME-OHXQ9FN-U3pwqAd-JyS6vZWoJ8NyoFXlhDuxSUG6J0X6aH3cs2IhcaMMgio5sFGxxcN_oluMzxifA-NlAWx6n4wGAGB8fAusXIHQ2tYC2Ph-BN9oPxuvGF-C0GprYHQW51F8iwE17tGEQ3CgTGUbIz2Y5v0jBEmQHnomw4r2gZ9Ry79DJbgzFm8hbs8QowihrZBXwmWWcRhSYPiY_GHTVYqDCMGpqgnoc4DKFQUJamuO6LZoAhjrsG6cRzC2Me_KSGCc77TXQ_x2sC4wzjNemxACNpKoYFreJWI49I0ORAFKcGaD_HYki4lEWwxXBMrblTWAzllyasZ4eTsqRrdTxuesXFxpXjemJnlWLuBMgnKR9tPDoq7Re-KkbcLAbgY51nvcW3ciykvlRSI9kdzYgLRPBGB8-ncZywmIX4urhPEpKAOVtvVfCaosaM1uYJOtpuUedaek_-0cg41QOhnVd5kVi6879Be1XFMZHw4oHJW1yfVJIf2_p9cnSIXlwTf1DoSHsFMerIl-7kI4eAosXzO-3lpZWR2G1m0ZX39nfP3wDb_8fzV9_pL05pagvqODg_CeAujQH6zxqtKUTAe4VyHmPXTyqb2YpLkWHilM7ztV70AFj3oDtibegdgEdHFj7AobLUJAM4g9yieNYScC7IQEerU_BJSU3dpqLQ4XYRnCV-pxG-uSOgSDKEGAFiGbT-vnqGSlGrdCq-_YSVgHAgy-Z_09ZbSnlwexFcq0rRIOzh5sFBKBdnmrj5hDoFEYfcpm-Z1ttAS_U5vQlk6KqNIqnIgyWSNFtadv4-z-58GyRwqn_YFz2VQi_Q4dOBTeGhLf22N0bBAVDXrafw5IdIjxMd7mVqBm0eLghPEb6_aR254C_o5aD2HZOIcm6FOWvGCzbFzbQAZSBDFo6ZcqhBrPZ64zPvOIEHvHeDorqHd0hMtMqyIxfuRHzNkn3M7eZOFoxRDuTxSwlvGfhk3brar_53yREz8wmILm1b7RQRi0jdenz0oiRqzbKEwMOq2dy9SnOrWXpdQ1nh-aTr3tqLwKXzA8pFb5Etyj8SHaViGZd9Bqo1CCtO-mdbPr_1ftvWb8nj5F7k2Mr-ErFU_UYl3CO-XjK_hkvDVAQVBS7dF4ZY3QkK1UJthW6AI1e0O9IUoOYZF7BOO8C0IXT8c4T0F_V1qTZ41vm9C-UgYvwHvBjwYQHYhMWoRoTDwwTK0bSTyh3uip4HMxxNGky881cIR0VjZ1mtjCRYOmpFXW6jiNrZOPi-jgW4ogBbxcfJ40xmcfE8n45MvDS5yw4nAFapPO6Fvlnynqq3PUoyZCeXgZbjG82sbVmPG-0hwYR7X0aeHS8NI4Ovu0R1am1W6CAYiotB0YXyycS4_ligIXWLns3rPy4R8pzyoef7tvlJboWLlk5QO0v0gu-5xFRRMszX5fbRA6Q8Y03QuPEXu8akeG_HppGxMWRmbN2ewz5IVT65aMvdVyBT3HKGlbpAx1KvkSruxpsd-EbiLuS3C_HtARmQgvR-bFKhOwfZPDUnaDc7e7s-rRSPx2lqBOH2AlgngSJ9tEiqye-hrO7_roAI2JtxkJ9yr8rmTYxU2rJ9oTlZBjr-o7Ppp7RR61yfw0f4uXRxNgSdYIE37dbHwKUMamABU98de90PpZHN6wjnWw7BwlyQhGDyOiwptwSlQaE6hvIWJlvTZpvGz9pRaB9Vn_6ikyo2fFNRpffrC6tZLx2dEqyfg8q2lVC--RRvFZp55PGJ9Qt1rHmbGdUbAOYJOieHpFIvPLFJ-7yaX1uRLTXJRA4jx7CuiH8Gq7eSpYkOqoJNIo14dsNbeNiY6cFoFaHC12DL6kRF7N6Y8bKMTt-oCVD0Rep_YqqOMlTdIFcA2zz3j8b3Q2sivhpSpPlIv8nN5fUma66rHmsgg-R7hM5rCR6tgr1J5Dl_zdpNtO21zehEuFxpcQG-tBK3Spsyw-cmh-7p-dp9GQRxOeG_25pWfcf2xlD_650a2RF_BLhyLgc6MZn_W96JlzAfiTmCaohaS2dNV_UljO-PNPjorO1F_wnQ7hj9CxHGbZkJ4p517Ll3Dum_SLTe87CtHvK7uILssOMDesX8S-a1jLbtH-7sRhrcXWXzUz2k5CDg9a1LjQ-l8e_e-KKDlLvvQcJkl0wuPa2f2LoFH6ss_mAzW4Jp-nH-_vcaqKs0yehEBo3Y0fVRPA2HAeJ2ls7N-yFtt4fU2jaf-qsLm4o8Y7ToWwE0bq87zyySzL-OziysX4PA-grFjcyLtSzsu5uMG72-m4mI_Hs9nkZnc3rUcCRVWV46qUoynON5NazsblXI64rKd4o-54wcfFhE9uJ-WsmAwnowIn89GkGBeiFrVkowL3Qumh1sc93YFvlPcN3t2OxnM-utGiQu3jv2-cV82WWul4dePuSHxQNVvPRoVWPvgzQFBBxz_sVsphHf6gEn49X6GVp1uibOqri128ftD9v_efyE3j9N3VdV2FXVMNa7tnfB3_GUtfg4Ozf2IdGF9HDzzj6-zE8Y7_NwAA__-8zZR-">