[llvm] [AMDGPU] Fix isCanonicalized for IEEE-mode-dependent sNaN quieting (PR #208733)

[Matt Arsenault via llvm-commits]

https://github.com/arsenm requested changes to this pull request.

There are a lot of problems in this area. They're large and definitional in nature, and I don't think this PR is moving in a wholistic direction to solving them. Solving them for real is a substantial undertaking

1. What "is canonical" means is platform dependent. An alternative framing we could make is that when IEEE=0, all nans are quiet nans and thus signaling nan bit patterns don't require quieting. This isn't necessarily a good idea, and has widespread consequences, but it would be internally consistent.
2. With IEEE=0, canonicalize doesn't actually quiet signaling nans. (but also, no nan can signal, so is it really a signaling nan?) @jwanggit86 has an open PR to do a software quieting of snans, though this has real code costs when the majority of use cases just want denormal flushing
3. The LLVM floating-point rules make performing canonicalize elision unsound. We've just been ignoring it here, but if we're going to fix something in this area I'd rather start doing something about this. My current preference would be to change the IR rules, at least in SDAG/GISel. Then at least we could soundly rely on doing normal combines and assuming operations behave as you would expect (and a combine can't drop almost-noop instructions). The alternative is to move this code into a machine pass, which would also be a large maintenance burden. We'd have to maintain yet another table of which instructions are known canonicalizing (which also has some subtarget variance, and mode dependency)


https://github.com/llvm/llvm-project/pull/208733