[llvm] [X86][MC] Report error when the instruction length exceeds 15 bytes (PR #83708)

[Shengchen Kan via llvm-commits]

https://github.com/KanRobert created https://github.com/llvm/llvm-project/pull/83708

The instruction-size limit of 15 bytes still applies to APX
instructions.

Note that it is possible for an EVEX-encoded legacy instruction to reach
the 15-byte instruction length limit: 4
bytes of EVEX prefix + 1 byte of opcode + 1 byte of ModRM + 1 byte of
SIB + 4 bytes of displacement + 4 bytes of immediate = 15 bytes in total, e.g.

```
addq    $184, -96, %rax   # encoding:
[0x62,0xf4,0xfc,0x18,0x81,0x04,0x25,0xa0,0xff,0xff,0xff,0xb8,0x00,0x00,0x00]
```

If we added a segment prefix like fs, the length would be 16.

In such a case, no additional (ASIZE or segment override) prefix can be
used.

To help users find this issue earlier, we change the internal compiler
error to error in this patch.


>From 30893c5d1b65e8168584d0cdc6f41fcf61ea51f2 Mon Sep 17 00:00:00 2001
From: Shengchen Kan <shengchen.kan at intel.com>
Date: Sun, 3 Mar 2024 14:00:42 +0800
Subject: [PATCH] [X86][MC] Report error when the instruction length exceeds 15
 bytes

The instruction-size limit of 15 bytes still applies to APX
instructions.

Note that it is possible for an EVEX-encoded legacy instruction to reach
the 15-byte instruction length limit: 4
bytes of EVEX prefix + 1 byte of opcode + 1 byte of ModRM + 1 byte of
SIB + 4 bytes of displacement + 4 bytes of immediate = 15 bytes in total, e.g.

```
addq    $184, -96, %rax   # encoding:
[0x62,0xf4,0xfc,0x18,0x81,0x04,0x25,0xa0,0xff,0xff,0xff,0xb8,0x00,0x00,0x00]
```

If we added a segment prefix like fs, the length would be 16.

In such a case, no additional (ASIZE or segment override) prefix can be
used.

To help users find this issue earlier, we change the internal compiler
error to error in this patch.
---
 .../X86/MCTargetDesc/X86MCCodeEmitter.cpp     |  5 ++--
 llvm/test/MC/X86/apx/long-instruction-err.s   | 25 +++++++++++++++++++
 2 files changed, 28 insertions(+), 2 deletions(-)
 create mode 100644 llvm/test/MC/X86/apx/long-instruction-err.s

diff --git a/llvm/lib/Target/X86/MCTargetDesc/X86MCCodeEmitter.cpp b/llvm/lib/Target/X86/MCTargetDesc/X86MCCodeEmitter.cpp
index fdb11d1a408bb6..fd96a173af27e0 100644
--- a/llvm/lib/Target/X86/MCTargetDesc/X86MCCodeEmitter.cpp
+++ b/llvm/lib/Target/X86/MCTargetDesc/X86MCCodeEmitter.cpp
@@ -1924,8 +1924,9 @@ void X86MCCodeEmitter::encodeInstruction(const MCInst &MI,
   if ((TSFlags & X86II::OpMapMask) == X86II::ThreeDNow)
     emitByte(X86II::getBaseOpcodeFor(TSFlags), CB);
 
-  assert(CB.size() - StartByte <= 15 &&
-         "The size of instruction must be no longer than 15.");
+  if (CB.size() - StartByte > 15)
+    Ctx.reportError(MI.getLoc(),
+                    "The size of instruction must be no longer than 15.");
 #ifndef NDEBUG
   // FIXME: Verify.
   if (/*!Desc.isVariadic() &&*/ CurOp != NumOps) {
diff --git a/llvm/test/MC/X86/apx/long-instruction-err.s b/llvm/test/MC/X86/apx/long-instruction-err.s
new file mode 100644
index 00000000000000..a2210ecad81709
--- /dev/null
+++ b/llvm/test/MC/X86/apx/long-instruction-err.s
@@ -0,0 +1,25 @@
+# RUN: not llvm-mc -triple x86_64 -show-encoding %s 2>&1 | FileCheck %s
+
+# CHECK: error: The size of instruction must be no longer than 15.
+# CHECK: addq    $1234, %cs:-96, %rax
+addq    $1234, %cs:-96, %rax
+
+# CHECK: error: The size of instruction must be no longer than 15.
+# CHECK: subq    $1234, %fs:257(%rbx, %rcx), %rax
+subq    $1234, %fs:257(%rbx, %rcx), %rax
+
+# CHECK: error: The size of instruction must be no longer than 15.
+# CHECK: orq     $1234, 257(%ebx, %ecx), %rax
+orq     $1234, 257(%ebx, %ecx), %rax
+
+# CHECK: error: The size of instruction must be no longer than 15.
+# CHECK: xorq    $1234, %gs:257(%ebx), %rax
+xorq    $1234, %gs:257(%ebx), %rax
+
+# CHECK: error: The size of instruction must be no longer than 15.
+# CHECK: {nf} andq    $1234, %cs:-96
+{nf} andq    $1234, %cs:-96
+
+# CHECK: error: The size of instruction must be no longer than 15.
+# CHECK: {evex} adcq    $1234, %cs:-96
+{evex} adcq    $1234, %cs:-96