[llvm] [AMDGPU] always emit a soft wait even if it is trivially ~0 (PR #147257)

[Sameer Sahasrabuddhe via llvm-commits]

ssahasra wrote:

> > I would prefer that we keep it functionally equivalent to `S_WAITCNT`. I'd like to think that another pass could insert, say, `S_WAITCNT_soft expcnt(0)` and have `SIInsertWaitcnts` treat it as a wait for expcnt that can be relaxed or removed, _without_ getting extra unwanted behaviour to do with waiting for loads-to-LDS.
> 
> Aha! Now this kinda makes sense to me. And it brings out something that has been nagging me at the back of my mind. Note how there is a FIXME in the next patch which says that we need to wait for direct loads to LDS only at a release fence, but currently there is no way to detect that situation. What I would really like to have is an S_WAIT_FENCE_soft with extra parameters like address space and ordering. There would be no wait count parameters, because it is entirely up to SIInsertWaitcnts to compute what is needed. Again, this is not specifically about direct loads to LDS.

Uploaded a whole new attempt that introduces S_WAITCNT_FENCE_soft in #150167

https://github.com/llvm/llvm-project/pull/147257