[llvm] [LLVM] [X86] Fix integer overflows in frame layout for huge frames (PR #101840)

[Wesley Wiser via llvm-commits]

================
@@ -945,11 +947,34 @@ X86RegisterInfo::eliminateFrameIndex(MachineBasicBlock::iterator II,
   }
 
   if (MI.getOperand(FIOperandNum+3).isImm()) {
-    // Offset is a 32-bit integer.
-    int Imm = (int)(MI.getOperand(FIOperandNum + 3).getImm());
-    int Offset = FIOffset + Imm;
-    assert((!Is64Bit || isInt<32>((long long)FIOffset + Imm)) &&
-           "Requesting 64-bit offset in 32-bit immediate!");
+    int64_t Imm = MI.getOperand(FIOperandNum + 3).getImm();
+    int64_t Offset = FIOffset + Imm;
+    bool FitsIn32Bits = isInt<32>(Offset);
+    // If the offset will not fit in a 32-bit displacement,
+    // then for 64-bit targets, scavenge a register to hold it.
+    // Otherwise, for 32-bit targets, this is a bug!
+    if (Is64Bit && !FitsIn32Bits) {
+      assert(RS && "RegisterScavenger was NULL");
+      const X86InstrInfo *TII = MF.getSubtarget<X86Subtarget>().getInstrInfo();
+      DebugLoc DL = MI.getDebugLoc();
+
+      RS->enterBasicBlockEnd(MBB);
+      RS->backward(std::next(II));
+
+      Register ScratchReg =
+          RS->scavengeRegisterBackwards(X86::GR64RegClass, II, false, 0, true);
+      assert(ScratchReg != 0 && "scratch reg was 0");
+      RS->setRegUsed(ScratchReg);
+
+      BuildMI(MBB, II, DL, TII->get(X86::MOV64ri), ScratchReg).addImm(Offset);
+
+      MI.getOperand(FIOperandNum + 3).setImm(0);
+      MI.getOperand(FIOperandNum + 2).setReg(ScratchReg);
+
+      return false;
+    } else if (!Is64Bit) {
+      assert(FitsIn32Bits && "Requesting 64-bit offset in 32-bit immediate!");
----------------
wesleywiser wrote:

The operand is i64 but that isn't valid on X86:

```
$ llc -mtriple=i386-unknown-linux-gnu -verify-machineinstrs llvm/test/CodeGen/X86/huge-stack.ll
*** Bad machine code: Displacement in address must fit into 32-bit signed integer ***
- function:    foo
- basic block: %bb.0  (0x56a1b68390f0)
- instruction: MOV8mi $esp, 1, $noreg, 4294967301, $noreg, 42 :: (store (s8) into %ir.3)
LLVM ERROR: Found 1 machine code errors.
```

https://github.com/llvm/llvm-project/pull/101840