[llvm-dev] predicate optimization and NaN

[Heiko via llvm-dev]

Hello all,
for the AFL++ fuzzer project i wrote an llvm pass for transforming 
floating point
comparisons to equivalent integer comparisons which can be splitted into 
chains
of byte comparisons. This is a sort of deoptimizing for the purpose of 
getting
feedback during fuzzing.

While extending the pass for special values like NaN, I came about a strange
behaviour, that might be a bug in the compiler. I am using clang/llvm 
version 12.0.0.

My test case was like this:

double a,b;
a = NAN;
b = NAN;
assert(!(a <= b));

I expected my pass to encounter a FCMP::OLE instruction during 
instrumentation.
But with optimization -O3 i got a FCMP::OGT as if i had written

assert((a > b));

Sadly this optimization is not correct for NaN values, which don't 
compare to anything.
Without optimization (-O0) everything worked as expected with my pass.

Without my pass everything workedalso (with any optimization level), but 
I don't know why.

So my question is, what needs to be done to use optimization with my pass
and have the right (original) predicate delivered?

Thanks very much,
Heiko Eißfeldt