[llvm] [X86] narrowBitOpRMW - allow additional uses of the BTC/R/S result (PR #166376)

[Phoebe Wang via llvm-commits]

================
@@ -53441,8 +53440,20 @@ static SDValue narrowBitOpRMW(StoreSDNode *St, const SDLoc &DL,
     Res = DAG.getNode(StoredVal.getOpcode(), DL, MVT::i32, X, Mask);
   }
 
-  return DAG.getStore(St->getChain(), DL, Res, NewPtr, St->getPointerInfo(),
-                      Align(), St->getMemOperand()->getFlags());
+  SDValue NewStore =
+      DAG.getStore(St->getChain(), DL, Res, NewPtr, St->getPointerInfo(),
+                   Align(), St->getMemOperand()->getFlags());
+
+  // If there are other uses of StoredVal, replace with a new load of the
+  // whole (updated) value and ensure that any chained dependencies on the
+  // original store are updated to come AFTER the new load.
+  if (!StoredVal.hasOneUse()) {
+    SDValue NewLoad =
+        DAG.getLoad(VT, DL, NewStore, Ld->getBasePtr(), Ld->getMemOperand());
+    DAG.ReplaceAllUsesWith(StoredVal, NewLoad);
+    DAG.ReplaceAllUsesWith(SDValue(St, 0), NewLoad.getValue(1));
----------------
phoebewang wrote:

The `NewLoad` has an input chain to `NewStore`. So (1) is not a problem, and load is guaranteed by data dependence, but store is dangerous. Maybe we can replace it with `NewStore` here?

`DAG.ReplaceAllUsesWith(SDValue(St, 0), NewStore.getValue(1));`

https://github.com/llvm/llvm-project/pull/166376