[llvm] [X86] Don't shrink VEX3 to VEX2 on a symbolic compare predicate (PR #213172)

[Zane Hambly via llvm-commits]

https://github.com/Zaneham created https://github.com/llvm/llvm-project/pull/213172

`llvm-mc` asserts on a VCMP predicate given as a symbol:

```asm
vcmpps $f0, %xmm0, %xmm1, %xmm2
```

```
Assertion failed: isImm() && "This is not an immediate", MCInst.h:85
```

`optimizeInstFromVEX3ToVEX2` reads the predicate with `getImm()` to decide whether the operands commute. A symbolic predicate is not known until link time, so that decision cannot be made here. Decline the shrink instead.

Unlike the sibling FPCLASS issues (#185364, #185365) this one is on the encoding path and asserts under `--filetype=obj` too, so a release build would be making the commutation decision on whatever `getImm()` returns for a non-immediate.

Literal predicates are unaffected and still commute to reach the two-byte VEX prefix. The test covers both.

Fixes #185355

>From 328a1afa35c03ee278c2029ab2336ff4da7cca9d Mon Sep 17 00:00:00 2001
From: Zane Hambly <zanehambly at gmail.com>
Date: Fri, 31 Jul 2026 02:19:03 +1200
Subject: [PATCH] [X86] Don't shrink VEX3 to VEX2 on a symbolic compare
 predicate

---
 .../MCTargetDesc/X86EncodingOptimization.cpp  |  4 ++++
 llvm/test/MC/X86/symbolic-vcmp-predicate.s    | 20 +++++++++++++++++++
 2 files changed, 24 insertions(+)
 create mode 100644 llvm/test/MC/X86/symbolic-vcmp-predicate.s

diff --git a/llvm/lib/Target/X86/MCTargetDesc/X86EncodingOptimization.cpp b/llvm/lib/Target/X86/MCTargetDesc/X86EncodingOptimization.cpp
index 58c5e329a38f6..d325932f54797 100644
--- a/llvm/lib/Target/X86/MCTargetDesc/X86EncodingOptimization.cpp
+++ b/llvm/lib/Target/X86/MCTargetDesc/X86EncodingOptimization.cpp
@@ -55,6 +55,10 @@ bool X86::optimizeInstFromVEX3ToVEX2(MCInst &MI, const MCInstrDesc &Desc) {
   case X86::VCMPPSYrri:
   case X86::VCMPSDrri:
   case X86::VCMPSSrri: {
+    // A symbolic predicate is not known until link time, so whether the
+    // operands commute cannot be decided here. Leave the encoding alone.
+    if (!MI.getOperand(3).isImm())
+      return false;
     switch (MI.getOperand(3).getImm() & 0x7) {
     default:
       return false;
diff --git a/llvm/test/MC/X86/symbolic-vcmp-predicate.s b/llvm/test/MC/X86/symbolic-vcmp-predicate.s
new file mode 100644
index 0000000000000..0a05d38a9bfdc
--- /dev/null
+++ b/llvm/test/MC/X86/symbolic-vcmp-predicate.s
@@ -0,0 +1,20 @@
+// RUN: llvm-mc -triple x86_64-unknown-unknown -mattr=+avx --show-encoding %s | FileCheck %s
+
+// The VCMP comparison predicate may be a symbol. The VEX3-to-VEX2 shrink used
+// to read it as a literal when deciding whether the operands commute, and
+// assert.
+
+// CHECK: vcmpps $f0, %xmm0, %xmm1, %xmm2
+// CHECK-SAME: encoding: [0xc5,0xf0,0xc2,0xd0,A]
+vcmpps $f0, %xmm0, %xmm1, %xmm2
+
+// A literal predicate still commutes to reach the two-byte VEX prefix. Without
+// the commute this would need the three-byte form.
+
+// CHECK: vcmpeqps %xmm1, %xmm8, %xmm2
+// CHECK-SAME: encoding: [0xc5,0xb8,0xc2,0xd1,0x00]
+vcmpeqps %xmm8, %xmm1, %xmm2
+
+// CHECK: vcmpltps %xmm8, %xmm1, %xmm2
+// CHECK-SAME: encoding: [0xc4,0xc1,0x70,0xc2,0xd0,0x01]
+vcmpps $1, %xmm8, %xmm1, %xmm2