[llvm] [X86][MC] Report error when the instruction length exceeds 15 bytes (PR #83708)

[Phoebe Wang via llvm-commits]

phoebewang wrote:

> > I haven't noticed that it was addressed somewhere. What about inline asm? It seems that now it bypasses all checks and fails only during encoding. Is it expected? Maybe we need to add additional checks?
> > P.S. This is almost irrelevant to the PR but is about _To help users find this issue earlier_
> 
> It's expected.
> 
> Short answer: It' hard to implement a generic API to determine the size of instruction before encoding in machine IR. We do not have special check for inline asm at machine IR stage.
> 
> Long answer: We have some instruction size checks for machine IR #83578 though it's implemented specifically for OPmi_ND. I proposed to implement `TargetInstrInfo::getInstSizeInBytes()` for X86 in https://discourse.llvm.org/t/rfc-support-long-instruction-fixup-for-x86/76539 to report an error when size_of(MachineInstr) > 15. But after investigation, I found it's hard b/c the complexity of
> 
> 1. `X86_MC::needsAddressSizeOverride`
> 2. `REX` -> `REX2`, `VEX2` -> `VEX3` promotion
> 3. The place where the register/immediate operand is encoded
> 
> So, I think for X86, checking the instruction size during encoding is good enough.

Does it crash too if you compile it to object file?

https://github.com/llvm/llvm-project/pull/83708