[llvm] [X86][MC] Reject out-of-range segment and debug registers encoded with APX (PR #82584)

[via llvm-commits]

llvmbot wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-backend-x86

Author: Timothy Herchen (anematode)

<details>
<summary>Changes</summary>

Fixes #<!-- -->82557. APX specification states that the high bits found in REX2 used to encode GPRs can also be used to encode segment and debug registers, although all of them will #UD. Therefore, when disassembling we reject attempts to create segment or debug registers with a value of 16 or more.

See page 22 of the [specification](https://www.intel.com/content/www/us/en/developer/articles/technical/advanced-performance-extensions-apx.html):

> Note that the R, X and B register identifiers can also address non-GPR register types, such as vector registers, control registers and debug registers. When any of them does, the highest-order bits REX2.R4, REX2.X4 or REX2.B4 are generally ignored, except when the register being addressed is a control or debug register. [...] The exception is that REX2.R4 and REX2.R3 [*sic*] are not ignored when the R register identifier addresses a control or debug register. Furthermore, if any attempt is made to access a non-existent control register (CR*) or debug register (DR*) using the REX2 prefix and one of the following instructions:
“MOV CR*, r64”, “MOV r64, CR*”, “MOV DR*, r64”, “MOV r64, DR*”. #UD is raised.

The invalid encodings are 64-bit only because `0xd5` is a valid instruction in 32-bit mode.

---
Full diff: https://github.com/llvm/llvm-project/pull/82584.diff


2 Files Affected:

- (modified) llvm/lib/Target/X86/Disassembler/X86Disassembler.cpp (+4) 
- (modified) llvm/test/MC/Disassembler/X86/x86-64-err.txt (+4) 


``````````diff
diff --git a/llvm/lib/Target/X86/Disassembler/X86Disassembler.cpp b/llvm/lib/Target/X86/Disassembler/X86Disassembler.cpp
index 5f852613610664..dbc2cef39d8682 100644
--- a/llvm/lib/Target/X86/Disassembler/X86Disassembler.cpp
+++ b/llvm/lib/Target/X86/Disassembler/X86Disassembler.cpp
@@ -819,8 +819,12 @@ static int readModRM(struct InternalInstruction *insn) {
         *valid = 0;                                                            \
       return prefix##_ES + (index & 7);                                        \
     case TYPE_DEBUGREG:                                                        \
+      if (index > 15)                                                          \
+        *valid = 0;                                                            \
       return prefix##_DR0 + index;                                             \
     case TYPE_CONTROLREG:                                                      \
+      if (index > 15)                                                          \
+        *valid = 0;                                                            \
       return prefix##_CR0 + index;                                             \
     case TYPE_MVSIBX:                                                          \
       return prefix##_XMM0 + index;                                            \
diff --git a/llvm/test/MC/Disassembler/X86/x86-64-err.txt b/llvm/test/MC/Disassembler/X86/x86-64-err.txt
index 3eca239e60f5c7..bd744790fe33d5 100644
--- a/llvm/test/MC/Disassembler/X86/x86-64-err.txt
+++ b/llvm/test/MC/Disassembler/X86/x86-64-err.txt
@@ -13,3 +13,7 @@
 0xc4,0xe2,0xfd,0x1a,0x08
 # 64: invalid instruction encoding
 0xc4,0xe3,0xfd,0x39,0xc5,0x01
+# 64: invalid instruction encoding
+0xd5,0xc5,0x20,0xef
+# 64: invalid instruction encoding
+0xd5,0xc5,0x21,0xef
\ No newline at end of file

``````````

</details>


https://github.com/llvm/llvm-project/pull/82584