[llvm] [X86][MC] Reject out-of-range segment and debug registers encoded with APX (PR #82584)

[Timothy Herchen via llvm-commits]

https://github.com/anematode updated https://github.com/llvm/llvm-project/pull/82584

>From 2eb9be99510ba4004efa44a0fc4c8beb1d78c746 Mon Sep 17 00:00:00 2001
From: Timothy Herchen <timothy.herchen at gmail.com>
Date: Wed, 21 Feb 2024 22:19:00 -0800
Subject: [PATCH] [X86][MC] Reject out-of-range segment and debug registers
 encoded with APX

APX specification states that the high bits found in REX2 used to encode GPRs can also be used to encode segment and debug registers, although all of them will #UD. Therefore, when disassembling we reject attempts to create segment or debug registers with a value of 16 or more.
---
 llvm/lib/Target/X86/Disassembler/X86Disassembler.cpp | 4 ++++
 llvm/test/MC/Disassembler/X86/x86-64-err.txt         | 4 ++++
 2 files changed, 8 insertions(+)

diff --git a/llvm/lib/Target/X86/Disassembler/X86Disassembler.cpp b/llvm/lib/Target/X86/Disassembler/X86Disassembler.cpp
index 5f852613610664..dbc2cef39d8682 100644
--- a/llvm/lib/Target/X86/Disassembler/X86Disassembler.cpp
+++ b/llvm/lib/Target/X86/Disassembler/X86Disassembler.cpp
@@ -819,8 +819,12 @@ static int readModRM(struct InternalInstruction *insn) {
         *valid = 0;                                                            \
       return prefix##_ES + (index & 7);                                        \
     case TYPE_DEBUGREG:                                                        \
+      if (index > 15)                                                          \
+        *valid = 0;                                                            \
       return prefix##_DR0 + index;                                             \
     case TYPE_CONTROLREG:                                                      \
+      if (index > 15)                                                          \
+        *valid = 0;                                                            \
       return prefix##_CR0 + index;                                             \
     case TYPE_MVSIBX:                                                          \
       return prefix##_XMM0 + index;                                            \
diff --git a/llvm/test/MC/Disassembler/X86/x86-64-err.txt b/llvm/test/MC/Disassembler/X86/x86-64-err.txt
index 3eca239e60f5c7..2d6c3e86ceaba1 100644
--- a/llvm/test/MC/Disassembler/X86/x86-64-err.txt
+++ b/llvm/test/MC/Disassembler/X86/x86-64-err.txt
@@ -5,6 +5,10 @@
 # 32: into
 0xce
 
+# 64: invalid instruction encoding
+0xd5,0xc5,0x20,0xef
+# 64: invalid instruction encoding
+0xd5,0xc5,0x21,0xef
 # 64: invalid instruction encoding
 0xc4,0x62,0xf9,0x18,0x20
 # 64: invalid instruction encoding