[llvm] r288418 - AArch64: fix 128-bit cmpxchg at -O0 (again, again).

[Tim Northover via llvm-commits]

Author: tnorthover
Date: Thu Dec  1 15:31:59 2016
New Revision: 288418

URL: http://llvm.org/viewvc/llvm-project?rev=288418&view=rev
Log:
AArch64: fix 128-bit cmpxchg at -O0 (again, again).

This time the issue is fortunately just a simple mistake rather than a horrible
design spectre. I thought SUBS/SBCS provided sufficient NZCV flags for
comparing two 64-bit values, but they don't.

The fix is slightly clunkier in AArch64 because we can't use conditional
execution to emit a pair of CMPs. Traditionally an "icmp ne i128" would map to
an EOR/EOR/ORR/CBNZ, but that uses more registers so it's easier to go with a
CSET/CINC/CBNZ combination. Slightly less efficient, but this is -O0 anyway.

Thanks to Anton Korobeynikov for pointing out the issue.

Modified:
    llvm/trunk/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp
    llvm/trunk/test/CodeGen/AArch64/cmpxchg-O0.ll

Modified: llvm/trunk/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp?rev=288418&r1=288417&r2=288418&view=diff
==============================================================================
--- llvm/trunk/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp (original)
+++ llvm/trunk/lib/Target/AArch64/AArch64ExpandPseudoInsts.cpp Thu Dec  1 15:31:59 2016
@@ -712,13 +712,21 @@ bool AArch64ExpandPseudo::expandCMP_SWAP
       .addReg(DestLo.getReg(), getKillRegState(DestLo.isDead()))
       .addOperand(DesiredLo)
       .addImm(0);
-  BuildMI(LoadCmpBB, DL, TII->get(AArch64::SBCSXr), AArch64::XZR)
+  BuildMI(LoadCmpBB, DL, TII->get(AArch64::CSINCWr), StatusReg)
+    .addUse(AArch64::WZR)
+    .addUse(AArch64::WZR)
+    .addImm(AArch64CC::EQ);
+  BuildMI(LoadCmpBB, DL, TII->get(AArch64::SUBSXrs), AArch64::XZR)
       .addReg(DestHi.getReg(), getKillRegState(DestHi.isDead()))
-      .addOperand(DesiredHi);
-  BuildMI(LoadCmpBB, DL, TII->get(AArch64::Bcc))
-      .addImm(AArch64CC::NE)
-      .addMBB(DoneBB)
-      .addReg(AArch64::NZCV, RegState::Implicit | RegState::Kill);
+      .addOperand(DesiredHi)
+      .addImm(0);
+  BuildMI(LoadCmpBB, DL, TII->get(AArch64::CSINCWr), StatusReg)
+      .addUse(StatusReg, RegState::Kill)
+      .addUse(StatusReg, RegState::Kill)
+      .addImm(AArch64CC::EQ);
+  BuildMI(LoadCmpBB, DL, TII->get(AArch64::CBNZW))
+      .addUse(StatusReg, RegState::Kill)
+      .addMBB(DoneBB);
   LoadCmpBB->addSuccessor(DoneBB);
   LoadCmpBB->addSuccessor(StoreBB);
 

Modified: llvm/trunk/test/CodeGen/AArch64/cmpxchg-O0.ll
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/CodeGen/AArch64/cmpxchg-O0.ll?rev=288418&r1=288417&r2=288418&view=diff
==============================================================================
--- llvm/trunk/test/CodeGen/AArch64/cmpxchg-O0.ll (original)
+++ llvm/trunk/test/CodeGen/AArch64/cmpxchg-O0.ll Thu Dec  1 15:31:59 2016
@@ -65,8 +65,10 @@ define { i128, i1 } @test_cmpxchg_128(i1
 ; CHECK: [[RETRY:.LBB[0-9]+_[0-9]+]]:
 ; CHECK:     ldaxp [[OLD_LO:x[0-9]+]], [[OLD_HI:x[0-9]+]], [x0]
 ; CHECK:     cmp [[OLD_LO]], x2
-; CHECK:     sbcs xzr, [[OLD_HI]], x3
-; CHECK:     b.ne [[DONE:.LBB[0-9]+_[0-9]+]]
+; CHECK:     cset [[CMP_TMP:w[0-9]+]], ne
+; CHECK:     cmp [[OLD_HI]], x3
+; CHECK:     cinc [[CMP:w[0-9]+]], [[CMP_TMP]], ne
+; CHECK:     cbnz [[CMP]], [[DONE:.LBB[0-9]+_[0-9]+]]
 ; CHECK:     stlxp [[STATUS:w[0-9]+]], x4, x5, [x0]
 ; CHECK:     cbnz [[STATUS]], [[RETRY]]
 ; CHECK: [[DONE]]:
@@ -88,8 +90,10 @@ define {i128, i1} @test_cmpxchg_128_unsp
 ; CHECK: [[RETRY:.LBB[0-9]+_[0-9]+]]:
 ; CHECK:     ldaxp [[OLD_LO:x[0-9]+]], [[OLD_HI:x[0-9]+]], [x0]
 ; CHECK:     cmp [[OLD_LO]], [[DESIRED_LO]]
-; CHECK:     sbcs xzr, [[OLD_HI]], [[DESIRED_HI]]
-; CHECK:     b.ne [[DONE:.LBB[0-9]+_[0-9]+]]
+; CHECK:     cset [[CMP_TMP:w[0-9]+]], ne
+; CHECK:     cmp [[OLD_HI]], [[DESIRED_HI]]
+; CHECK:     cinc [[CMP:w[0-9]+]], [[CMP_TMP]], ne
+; CHECK:     cbnz [[CMP]], [[DONE:.LBB[0-9]+_[0-9]+]]
 ; CHECK:     stlxp [[STATUS:w[0-9]+]], [[NEW_LO]], [[NEW_HI]], [x0]
 ; CHECK:     cbnz [[STATUS]], [[RETRY]]
 ; CHECK: [[DONE]]: