[llvm] [X86][MC] Report error when the instruction length exceeds 15 bytes (PR #83708)

[Shengchen Kan via llvm-commits]

KanRobert wrote:

> I haven't noticed that it was addressed somewhere. What about inline asm? It seems that now it bypasses all checks and fails only during encoding. Is it expected? Maybe we need to add additional checks?
> 
> P.S. This is almost irrelevant to the PR but is about _To help users find this issue earlier_

It's expected.
 
Short answer:
It' hard to implement a generic API to determine the size of instruction before encoding in machine IR.  We do not have special check for inline asm at machine IR stage.

Long answer:
We have some instruction size checks for machine IR #83578 though it's implemented specifically for OPmi_ND.
I proposed to implement `TargetInstrInfo::getInstSizeInBytes()` for X86 in https://discourse.llvm.org/t/rfc-support-long-instruction-fixup-for-x86/76539 to report an error when size_of(MachineInstr) > 15. But after investigation, I found it's hard b/c the complexity of

1. `X86_MC::needsAddressSizeOverride`
2.  `REX` -> `REX2`, `VEX2` -> `VEX3` promotion
3. The place where the register/immediate operand is encoded

So, I think for X86, checking the instruction size during encoding is good enough.


https://github.com/llvm/llvm-project/pull/83708