[PATCH] D36213: [InstCombine] Remove check for sext of vector icmp from shouldOptimizeCast

[Eli Friedman via Phabricator via llvm-commits]

efriedma added a comment.

Our handling of i1 masks in SelectionDAG is generally terrible.

x86 doesn't have this particular problem for `<4 x float>`, because it doesn't have 64-bit vectors, and it doesn't have this problem for `<8 x float>` if AVX is turned on because there's a target-specific DAGCombine to work around the issue (WidenMaskArithmetic), but it does show up in other cases.  For example, try the following on x86 without AVX:

  define <8 x i32> @testa(<8 x float> %a, <8 x float> %b) {
    %cmp = fcmp ult <8 x float> %a, zeroinitializer
    %cmp4 = fcmp ult <8 x float> %b, zeroinitializer
    %and1 = and <8 x i1> %cmp, %cmp4
    %and = sext <8 x i1> %and1 to <8 x i32>
    ret <8 x i32> %and
  }

Or the following with AVX (but not AVX512):

  define <16 x i32> @testb(<16 x float> %a, <16 x float> %b) {
    %cmp = fcmp ult <16 x float> %a, zeroinitializer
    %cmp4 = fcmp ult <16 x float> %b, zeroinitializer
    %and1 = and <16 x i1> %cmp, %cmp4
    %and = sext <16 x i1> %and1 to <16 x i32>
    ret <16 x i32> %and
  }


https://reviews.llvm.org/D36213