[llvm] [X86] Check if signed value is too large for fixup under some conditions (PR #150976)

[via llvm-commits]

================
@@ -721,15 +722,56 @@ void X86AsmBackend::applyFixup(const MCFragment &F, const MCFixup &Fixup,
   assert(Fixup.getOffset() + Size <= Data.size() && "Invalid fixup offset!");
 
   int64_t SignedValue = static_cast<int64_t>(Value);
-  if (IsResolved && Fixup.isPCRel()) {
-    // check that PC relative fixup fits into the fixup size.
+
+  // If the fixup is resolved to an absolute value, and is explictly signed
+  // (either by its type or by being marked as PC-relative), we should check
+  // that it fits.
+  // Otherwise, be conservative because some users may rely on overflow, or
+  // being able to use a signed value where an unsigned value would normally be
+  // expected, so strict checks may break things.
+  bool CheckSignedVal = false;
+  if (IsResolved) {
+    switch (Fixup.getKind()) {
+    default:
+      llvm_unreachable("invalid fixup kind!");
+    case FK_NONE:
+    case FK_Data_1:
+    case FK_Data_2:
+    case FK_Data_4:
+    case FK_Data_8:
+    case FK_SecRel_1:
+    case FK_SecRel_2:
+    case FK_SecRel_4:
+    case FK_SecRel_8:
+      CheckSignedVal = false;
+      break;
+    case X86::reloc_riprel_4byte:
+    case X86::reloc_riprel_4byte_relax:
+    case X86::reloc_riprel_4byte_relax_rex:
+    case X86::reloc_riprel_4byte_relax_rex2:
+    case X86::reloc_riprel_4byte_movq_load:
+    case X86::reloc_riprel_4byte_movq_load_rex2:
+    case X86::reloc_riprel_4byte_relax_evex:
+    case X86::reloc_signed_4byte:
+    case X86::reloc_signed_4byte_relax:
+    case X86::reloc_global_offset_table:
+    case X86::reloc_branch_4byte_pcrel:
+      CheckSignedVal = true;
+      break;
+    }
+
+    if (Fixup.isPCRel())
+      CheckSignedVal = true;
----------------
Heath123 wrote:

Sometimes a PC-relative fixup uses one of the generic fixup types. I think this is because all of the architecture-specific ones are 4-byte, so for example a 1-byte PC-relative fixup will be emitted as a generic fixup, but with the PC-relative flag set to true

https://github.com/llvm/llvm-project/pull/150976