[llvm] [X86][MC][AsmParser] Reject H-byte regs with VEX/EVEX-encoded 8-bit RR (NDD) (PR #160039)

[Phoebe Wang via llvm-commits]

================
@@ -4044,6 +4044,24 @@ bool X86AsmParser::validateInstruction(MCInst &Inst, const OperandVector &Ops) {
     }
   }
 
+  unsigned Enc = TSFlags & X86II::EncodingMask;
+  if (Enc == X86II::VEX || Enc == X86II::EVEX || Enc == X86II::XOP) {
+    unsigned NumOps = Inst.getNumOperands();
+    for (unsigned i = 0; i != NumOps; ++i) {
+      const MCOperand &MO = Inst.getOperand(i);
+      if (!MO.isReg())
+        continue;
+      MCRegister Reg = MO.getReg();
+      if (Reg == X86::AH || Reg == X86::BH || Reg == X86::CH ||
+          Reg == X86::DH) {
+        StringRef RegName = X86IntelInstPrinter::getRegisterName(Reg);
----------------
phoebewang wrote:

I think Shengchen means combine the check with above code. E.g. changing line 4043 to:

```
      if (X86II::isX86_64NonExtLowByteReg(Reg) ||
          X86II::isX86_64ExtendedReg(Reg) ||
          (TSFlags & X86II::EncodingMask) == X86II::EVEX)
        UsesRex = true;
```

https://github.com/llvm/llvm-project/pull/160039