[llvm-dev] DAGLegalizer Issue

[pbrunet via llvm-dev]

Hello,

As I try to compile some generated llvm IR, I have an issue with the
shuffle instruction on FloatingPoint reaching an assert in the DAGLegalizer.
I try to produce a patch to fix it and I wonder what is the best way to
handle it.

Now the problem:
There is a shuffle transformation like:
; fold: (shuffle (bitcast (BINOP A, B)), Undef, <Mask>)
->                  
;       (shuffle (BINOP (bitcast A), (bitcast B)), Undef, <Mask>)

For X86 with BINOP in (FADD, FMUL, FSUB, ADD, MUL, SUB).
It seems OK and I don't want to change that but I have a case where
bitcast is done from float to int. In this case, we can't perform this
transformation
as FADD can't be apply on integers.

I could have fix it here and say : FADD, FMUL, FSUB can be switch with
float bitcast only and ADD/MUL/SUB with intergers bitcast  only. But few
lines above there is an if with:
TLI.isOperationLegal(Opcode, VT).
It means that Opcode == FADD and VT type == Integers return True (so is
Legal) and it doesn't looks correct to me.
So I wanted to registers op/type couple as illegal with something like

for (MVT VT : MVT::integer_valuetypes()) {
   setOperationAction(ISD::FADD, VT , Custom);
   setOperationAction(ISD::FSUB, VT , Custom);
   setOperationAction(ISD::FMUL, VT , Custom);
}

But "Custom" doesn't really means illegal so what should I use? I can
keep custom and fill the lowerOperation switch but as it is a corner
case where we perform transformation after legalizing the DAG, I am not
sure it is the good solution.
As I am certainly missing something, I rely on you to give me more
informations on the lowering process.

Thanks,
Pierrick