[llvm] [AMDGPU] Move InferAddressSpacesPass to middle end optimization pipeline (PR #138604)

[Matt Arsenault via llvm-commits]

================
@@ -908,6 +924,12 @@ void AMDGPUTargetMachine::registerPassBuilderCallbacks(PassBuilder &PB) {
         if (EnableLowerModuleLDS)
           PM.addPass(AMDGPULowerModuleLDSPass(*this));
         if (Level != OptimizationLevel::O0) {
+          // We only want to run this with O2 or higher since inliner and SROA
+          // don't run in O1.
----------------
arsenm wrote:

This seems surprising 

https://github.com/llvm/llvm-project/pull/138604