[llvm-bugs] [Bug 42899] New: X86MCCodeEmitter crashes when encoding a VPERMIL2PS (XOP) with an invalid immediate

[via llvm-bugs]

https://bugs.llvm.org/show_bug.cgi?id=42899

            Bug ID: 42899
           Summary: X86MCCodeEmitter crashes when encoding a VPERMIL2PS
                    (XOP) with an invalid immediate
           Product: libraries
           Version: trunk
          Hardware: PC
                OS: Windows NT
            Status: NEW
          Severity: enhancement
          Priority: P
         Component: Backend: X86
          Assignee: unassignedbugs at nondot.org
          Reporter: andrea.dibiagio at gmail.com
                CC: craig.topper at gmail.com, llvm-bugs at lists.llvm.org,
                    llvm-dev at redking.me.uk, spatel+llvm at rotateright.com

The issue has to do with VPERMIL2P instructions with an invalid immediate
operand (the fifth operand - named M2Z).

The assembler uses this 2-bit value to encode M2Z in the instruction.
M2Z occupies bits [1:0] of an immediate byte. Bits [7:4] of the same byte are
used to select one of 16 YMM/XMM registers.

When VEX.W is 0, bits [7:4] are used to encode `src3`. ModRMr/m is used for
`src2`.

When VEX.W is 1, bits [7:4] are used to encode `src2`.  ModRMr/m is used for
`src3`.

If an invalid (out of range) immediate is used (for example $16), then
X86MCCodeEmitter::encodeInstruction() triggers an assertion failure.

Example (from test/tools/llvm-mca/X86/Generic.xop-super-registers-2.s:

   vpermil2pd $16, %xmm3, %xmm5, %xmm1, %xmm2



~/cllvm/llvm/test/tools/llvm-mca/X86/Generic$ llvm-mc -filetype=obj
xop-super-registers-2.s


llvm-mc:
/home/andrea/cllvm/llvm/lib/Target/X86/MCTargetDesc/X86MCCodeEmitter.cpp:1617:
virtual void {anonymous}::X86MCCodeEmitter::encodeInstruction(const
llvm::MCInst&, llvm::raw_ostream&, llvm::SmallVectorImpl<llvm::MCFixup>&, const
llvm::MCSubtargetInfo&) const: Assertion `Val < 16 && "Immediate operand value
out of range"' failed.
Stack dump:
0.      Program arguments: /home/andrea/cllvm/cllvm-debug-build/bin/llvm-mc
-filetype=obj xop-super-registers-2.s
 #0 0x000055c7b1b5069f llvm::sys::PrintStackTrace(llvm::raw_ostream&)
/home/andrea/cllvm/llvm/lib/Support/Unix/Signals.inc:533:0
 #1 0x000055c7b1b50732 PrintStackTraceSignalHandler(void*)
/home/andrea/cllvm/llvm/lib/Support/Unix/Signals.inc:594:0
 #2 0x000055c7b1b4e60c llvm::sys::RunSignalHandlers()
/home/andrea/cllvm/llvm/lib/Support/Signals.cpp:68:0
 #3 0x000055c7b1b50056 SignalHandler(int)
/home/andrea/cllvm/llvm/lib/Support/Unix/Signals.inc:385:0
 #4 0x00007eff3836a890 __restore_rt
(/lib/x86_64-linux-gnu/libpthread.so.0+0x12890)
 #5 0x00007eff3721fe97 raise
/build/glibc-OTsEL5/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c:51:0
 #6 0x00007eff37221801 abort /build/glibc-OTsEL5/glibc-2.27/stdlib/abort.c:81:0
 #7 0x00007eff3721139a __assert_fail_base
/build/glibc-OTsEL5/glibc-2.27/assert/assert.c:89:0
 #8 0x00007eff37211412 (/lib/x86_64-linux-gnu/libc.so.6+0x30412)
 #9 0x000055c7b17f80e0 (anonymous
namespace)::X86MCCodeEmitter::encodeInstruction(llvm::MCInst const&,
llvm::raw_ostream&, llvm::SmallVectorImpl<llvm::MCFixup>&,
llvm::MCSubtargetInfo const&) const
/home/andrea/cllvm/llvm/lib/Target/X86/MCTargetDesc/X86MCCodeEmitter.cpp:1618:0
#10 0x000055c7b19c47ab llvm::MCELFStreamer::EmitInstToData(llvm::MCInst const&,
llvm::MCSubtargetInfo const&)
/home/andrea/cllvm/llvm/lib/MC/MCELFStreamer.cpp:520:0
#11 0x000055c7b19df231 llvm::MCObjectStreamer::EmitInstructionImpl(llvm::MCInst
const&, llvm::MCSubtargetInfo const&)
/home/andrea/cllvm/llvm/lib/MC/MCObjectStreamer.cpp:338:0
#12 0x000055c7b19df0f5 llvm::MCObjectStreamer::EmitInstruction(llvm::MCInst
const&, llvm::MCSubtargetInfo const&)
/home/andrea/cllvm/llvm/lib/MC/MCObjectStreamer.cpp:320:0
#13 0x000055c7b14a73c4 (anonymous
namespace)::X86AsmParser::EmitInstruction(llvm::MCInst&,
llvm::SmallVectorImpl<std::unique_ptr<llvm::MCParsedAsmOperand,
std::default_delete<llvm::MCParsedAsmOperand> > >&, llvm::MCStreamer&)
/home/andrea/cllvm/llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp:3033:0
#14 0x000055c7b14a7e9c (anonymous
namespace)::X86AsmParser::MatchAndEmitATTInstruction(llvm::SMLoc, unsigned
int&, llvm::SmallVectorImpl<std::unique_ptr<llvm::MCParsedAsmOperand,
std::default_delete<llvm::MCParsedAsmOperand> > >&, llvm::MCStreamer&, unsigned
long&, bool)
/home/andrea/cllvm/llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp:3174:0
#15 0x000055c7b14a745a (anonymous
namespace)::X86AsmParser::MatchAndEmitInstruction(llvm::SMLoc, unsigned int&,
llvm::SmallVectorImpl<std::unique_ptr<llvm::MCParsedAsmOperand,
std::default_delete<llvm::MCParsedAsmOperand> > >&, llvm::MCStreamer&, unsigned
long&, bool)
/home/andrea/cllvm/llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp:3043:0
#16 0x000055c7b1a58b2c (anonymous
namespace)::AsmParser::parseStatement((anonymous
namespace)::ParseStatementInfo&, llvm::MCAsmParserSemaCallback*)
/home/andrea/cllvm/llvm/lib/MC/MCParser/AsmParser.cpp:2270:0
#17 0x000055c7b1a5287a (anonymous namespace)::AsmParser::Run(bool, bool)
/home/andrea/cllvm/llvm/lib/MC/MCParser/AsmParser.cpp:917:0
#18 0x000055c7b12900cc AssembleInput(char const*, llvm::Target const*,
llvm::SourceMgr&, llvm::MCContext&, llvm::MCStreamer&, llvm::MCAsmInfo&,
llvm::MCSubtargetInfo&, llvm::MCInstrInfo&, llvm::MCTargetOptions&)
/home/andrea/cllvm/llvm/tools/llvm-mc/llvm-mc.cpp:301:0
#19 0x000055c7b1291661 main
/home/andrea/cllvm/llvm/tools/llvm-mc/llvm-mc.cpp:504:0
#20 0x00007eff37202b97 __libc_start_main
/build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:344:0
#21 0x000055c7b128f57a _start
(/home/andrea/cllvm/cllvm-debug-build/bin/llvm-mc+0x46557a)
Aborted (core dumped)

---

Note: GAS errors out with the following message:

xop-super-registers-2.s: Assembler messages:
xop-super-registers-2.s:5: Error: constant doesn't fit in 4 bits for
`vpermil2pd'

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20190806/52c74dc8/attachment.html>