[llvm-branch-commits] [llvm] [SelectionDAG] Read callbr outputs at the top of indirect targets (PR #239115)

Bill Wendling via llvm-branch-commits llvm-branch-commits at lists.llvm.org
Fri Nov 20 02:42:58 PST 2026


https://github.com/isanbard updated https://github.com/llvm/llvm-project/pull/239115

>From 04e0f2fb629f385a44774056d2945012efb7c93a Mon Sep 17 00:00:00 2001
From: Bill Wendling <isanbard at gmail.com>
Date: Wed, 4 Nov 2026 21:13:33 -0800
Subject: [PATCH 1/5] [SelectionDAG] Read callbr outputs at the top of indirect
 targets

A callbr's indirect target reads the asm's outputs again, from the
registers that the INLINEASM_BR defines them in. When those are
physical registers, they have to be live into the indirect target, and
the read has to come before anything in the block clobbers them.
Neither was ensured.

Only the registers that an output was assigned were marked live-in. An
x86 flag output ("=@ccz") is assigned an unused GR32 virtual register,
and its lowering reads EFLAGS instead, so EFLAGS wasn't live-in, and
the verifier reported "Using an undefined physical register" (#60827).
AArch64 and SystemZ assign a flag output the flags register itself, so
they weren't affected. The read was also a CopyFromReg in the DAG,
which the scheduler could put after an instruction that clobbers the
register. At -O2, an indirect target that added the asm's carry flag to
a product ran the imul before the adc that reads the carry.

Read the registers the way an EH pad reads its exception registers.
Each physical register that an output reads is copied into a virtual
register at the top of the block, which also makes it live-in, and the
DAG reads the virtual register instead. MachineBasicBlock::addLiveIn()
does the copying, and it now allows an INLINEASM_BR indirect target.

A register that can't be copied directly, such as EFLAGS, NZCV, or CC,
is copied through the class that the target copies it to, and its use
restores it, which X86FlagsCopyLowering lowers on x86. When nothing
clobbers the register in between, the peephole optimizer drops the
restore, which leaves the copy dead, so this costs nothing at -O2. At
-O0, AArch64 gets an mrs/msr pair and SystemZ an ipm/tmlh pair.

Only the outputs that the indirect target uses are read, and the others
get an undefined value. An unused output would otherwise be live-in,
which the asm's block can contradict by clobbering the output's
register after the asm, and -O0 exports all of the outputs, used or
not. This also stops marking a register live-in once for each output in
it, such as NZCV for two AArch64 flag outputs, which addLiveIn() doesn't
allow.

Fixes #60827.

Assisted-by: Claude Opus 5.5
---
 llvm/lib/CodeGen/MachineBasicBlock.cpp        |   6 +-
 .../SelectionDAG/SelectionDAGBuilder.cpp      | 128 ++++++-
 .../callbr-asm-outputs-indirect-isel.ll       |   6 +-
 .../SystemZ/callbr-asm-flag-outputs.ll        |  33 ++
 .../CodeGen/X86/callbr-asm-flag-outputs.ll    | 347 ++++++++++++++++++
 .../callbr-asm-outputs-indirect-isel-m32.ll   |  28 +-
 .../X86/callbr-asm-outputs-indirect-isel.ll   |  75 +++-
 llvm/test/CodeGen/X86/callbr-asm-outputs.ll   |  11 +-
 8 files changed, 593 insertions(+), 41 deletions(-)
 create mode 100644 llvm/test/CodeGen/SystemZ/callbr-asm-flag-outputs.ll
 create mode 100644 llvm/test/CodeGen/X86/callbr-asm-flag-outputs.ll

diff --git a/llvm/lib/CodeGen/MachineBasicBlock.cpp b/llvm/lib/CodeGen/MachineBasicBlock.cpp
index c6cd6aaf4c904d..5ad5e4e28a1be4 100644
--- a/llvm/lib/CodeGen/MachineBasicBlock.cpp
+++ b/llvm/lib/CodeGen/MachineBasicBlock.cpp
@@ -709,8 +709,10 @@ MachineBasicBlock::addLiveIn(MCRegister PhysReg, const TargetRegisterClass *RC)
   assert(getParent() && "MBB must be inserted in function");
   assert(PhysReg.isPhysical() && "Expected physreg");
   assert(RC && "Register class is required");
-  assert((isEHPad() || this == &getParent()->front()) &&
-         "Only the entry block and landing pads can have physreg live ins");
+  assert((isEHPad() || isInlineAsmBrIndirectTarget() ||
+          this == &getParent()->front()) &&
+         "Only the entry block, landing pads, and INLINEASM_BR indirect "
+         "targets can have physreg live ins");
 
   bool LiveIn = isLiveIn(PhysReg);
   iterator I = SkipPHIsAndLabels(begin()), E = end();
diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index 7781613caf7db3..e2450095fa578d 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -16,6 +16,7 @@
 #include "llvm/ADT/APInt.h"
 #include "llvm/ADT/BitVector.h"
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SmallBitVector.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringRef.h"
@@ -13307,6 +13308,108 @@ void SelectionDAGBuilder::visitVectorSplice(const CallInst &I) {
   setValue(&I, DAG.getVectorShuffle(VT, DL, V1, V2, Mask));
 }
 
+/// Read each physical register that \p V copies from at the top of \p MBB, an
+/// indirect target of a callbr, and make \p V use those reads.
+///
+/// The registers carry the asm's outputs across the edge into \p MBB, so, like
+/// an EH pad's exception registers, they're live-in and have to be read before
+/// anything in the block clobbers them. A CopyFromReg in the DAG doesn't ensure
+/// that: the scheduler can put an instruction that sets EFLAGS, for example,
+/// ahead of the read of an x86 flag output. A COPY emitted at the top of the
+/// block, before the DAG's instructions, does.
+static void readPhysRegsAtBlockEntry(SDValue &V, MachineBasicBlock &MBB,
+                                     SelectionDAG &DAG, const SDLoc &DL) {
+  // Find the reads of physical registers that V depends on. These are reads of
+  // the registers an output is in, or of one that the target's lowering picks,
+  // such as EFLAGS for an x86 flag output ("=@ccz").
+  SmallVector<SDNode *, 4> Worklist = {V.getNode()};
+  SmallPtrSet<SDNode *, 4> Visited;
+  SmallVector<SDNode *, 4> PhysRegReads;
+  while (!Worklist.empty()) {
+    SDNode *N = Worklist.pop_back_val();
+    if (!Visited.insert(N).second)
+      continue;
+
+    if (N->getOpcode() == ISD::CopyFromReg) {
+      if (cast<RegisterSDNode>(N->getOperand(1))->getReg().isPhysical())
+        PhysRegReads.push_back(N);
+      continue;
+    }
+
+    // Follow only the operands that the value depends on. The chain leads to
+    // the rest of the block, not to what this output reads.
+    for (SDValue Op : N->op_values())
+      if (Op.getValueType() != MVT::Other)
+        Worklist.push_back(Op.getNode());
+  }
+
+  // V may itself be one of the reads, such as for an output in one register,
+  // so keep track of it while they're replaced.
+  HandleSDNode Handle(V);
+  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+  const TargetRegisterInfo *TRI = DAG.getSubtarget().getRegisterInfo();
+  // The walk found the reads last operand first. Copy them in operand order,
+  // which is the order of an output's registers.
+  for (SDNode *N : reverse(PhysRegReads)) {
+    assert(N->getNumOperands() == 2 && "Unexpected glue on a callbr output");
+    MCRegister PhysReg =
+        cast<RegisterSDNode>(N->getOperand(1))->getReg().asMCReg();
+    MVT VT = N->getSimpleValueType(0);
+
+    // Copy a register that can't be copied directly, such as EFLAGS, through
+    // the class that the target copies it to. The use of the copy then
+    // restores it, which the target lowers.
+    const TargetRegisterClass *RC = TRI->getMinimalPhysRegClass(PhysReg);
+    bool CrossCopy = RC->expensiveOrImpossibleToCopy();
+    if (CrossCopy)
+      RC = TRI->getCrossCopyRegClass(RC);
+    else if (const TargetRegisterClass *VTRC = TLI.getRegClassFor(VT);
+             VTRC->contains(PhysReg))
+      RC = VTRC;
+
+    // This marks PhysReg live-in, once, and reuses the copy if another output
+    // reads the same register, such as a second flag output.
+    Register VReg = MBB.addLiveIn(PhysReg, RC);
+
+    // The copy kills PhysReg. When nothing clobbers PhysReg before the
+    // restore, the peephole optimizer removes the restore, so the read after
+    // it then uses PhysReg past that kill.
+    if (CrossCopy)
+      DAG.getMachineFunction()
+          .getRegInfo()
+          .getVRegDef(VReg)
+          ->clearRegisterKills(PhysReg, TRI);
+
+    SDValue Read = DAG.getCopyFromReg(N->getOperand(0), DL, VReg, VT);
+    DAG.ReplaceAllUsesWith(N, Read.getNode());
+  }
+  V = Handle.getValue();
+}
+
+/// Return which of a callbr's \p NumOutputs outputs its landing pad \p I uses.
+static SmallBitVector getUsedCallBrOutputs(const CallInst &I,
+                                           unsigned NumOutputs) {
+  SmallBitVector Used(NumOutputs);
+  if (!I.getType()->isStructTy()) {
+    if (!I.use_empty())
+      Used.set();
+    return Used;
+  }
+
+  for (const User *U : I.users()) {
+    const auto *EV = dyn_cast<ExtractValueInst>(U);
+    if (!EV || EV->getIndices().empty() || EV->getIndices()[0] >= NumOutputs) {
+      Used.set();
+      return Used;
+    }
+
+    // Unoptimized IR can extract an output and not use it.
+    if (!EV->use_empty())
+      Used.set(EV->getIndices()[0]);
+  }
+  return Used;
+}
+
 // Read the outputs again from the registers that the INLINEASM_BR defines them
 // in, as visitInlineAsm() did, rather than with the simpler
 //   setValue(&I, getCopyFromRegs(CBR, CBR->getType()));
@@ -13318,17 +13421,26 @@ void SelectionDAGBuilder::visitCallBrLandingPad(const CallInst &I) {
   auto It = CallBrOutputs.find(CBR);
   assert(It != CallBrOutputs.end() && "callbr asm wasn't lowered");
 
-  SDValue Chain = DAG.getRoot();
+  // Read only the outputs that this block uses, which are the only ones live
+  // into it. The asm's block may clobber an unused one's registers after the
+  // asm, such as EFLAGS, which a live-in would contradict. An unused one can
+  // still be exported, such as at -O0, so it gets an undefined value instead.
+  SmallBitVector Used = getUsedCallBrOutputs(I, It->second.size());
+
   SmallVector<SDValue, 8> OutputValues;
-  for (const CallBrOutput &Output : It->second) {
+  for (auto [Idx, Output] : enumerate(It->second)) {
     RegsForValue Regs(Output.Regs, Output.RegVT, Output.ValueVT);
-    for (Register Reg : Regs.Regs)
-      if (Reg.isPhysical())
-        FuncInfo.MBB->addLiveIn(Reg);
-
+    // Each output's reads only follow the block's entry, so that an unused
+    // output's reads have no users left.
+    SDValue Chain = DAG.getRoot();
     SDValue Glue;
-    OutputValues.push_back(getInlineAsmOutput(
-        Output.OpInfo, Regs, *CBR, Chain, Glue, getCurSDLoc(), FuncInfo, DAG));
+    SDValue V = getInlineAsmOutput(Output.OpInfo, Regs, *CBR, Chain, Glue,
+                                   getCurSDLoc(), FuncInfo, DAG);
+    if (Used[Idx])
+      readPhysRegsAtBlockEntry(V, *FuncInfo.MBB, DAG, getCurSDLoc());
+    else
+      V = DAG.getUNDEF(V.getValueType());
+    OutputValues.push_back(V);
   }
 
   setValue(&I, getInlineAsmResult(*CBR, OutputValues, getCurSDLoc(), DAG));
diff --git a/llvm/test/CodeGen/AArch64/callbr-asm-outputs-indirect-isel.ll b/llvm/test/CodeGen/AArch64/callbr-asm-outputs-indirect-isel.ll
index 4cea4d2f493a79..9a4ce66bc96fe1 100644
--- a/llvm/test/CodeGen/AArch64/callbr-asm-outputs-indirect-isel.ll
+++ b/llvm/test/CodeGen/AArch64/callbr-asm-outputs-indirect-isel.ll
@@ -569,8 +569,10 @@ define i32 @flag_output() {
   ; CHECK-NEXT: bb.2.indirect (label-must-be-emitted, inlineasm-br-indirect-target):
   ; CHECK-NEXT:   liveins: $nzcv
   ; CHECK-NEXT: {{  $}}
-  ; CHECK-NEXT:   [[COPY2:%[0-9]+]]:gpr32 = COPY $wzr
-  ; CHECK-NEXT:   [[CSINCWr1:%[0-9]+]]:gpr32 = CSINCWr [[COPY2]], [[COPY2]], 1, implicit $nzcv
+  ; CHECK-NEXT:   [[COPY2:%[0-9]+]]:gpr64 = COPY $nzcv
+  ; CHECK-NEXT:   [[COPY3:%[0-9]+]]:gpr32 = COPY $wzr
+  ; CHECK-NEXT:   $nzcv = COPY [[COPY2]]
+  ; CHECK-NEXT:   [[CSINCWr1:%[0-9]+]]:gpr32 = CSINCWr [[COPY3]], [[COPY3]], 1, implicit $nzcv
   ; CHECK-NEXT:   $w0 = COPY [[CSINCWr1]]
   ; CHECK-NEXT:   RET_ReallyLR implicit $w0
 entry:
diff --git a/llvm/test/CodeGen/SystemZ/callbr-asm-flag-outputs.ll b/llvm/test/CodeGen/SystemZ/callbr-asm-flag-outputs.ll
new file mode 100644
index 00000000000000..708f92a7e54fd1
--- /dev/null
+++ b/llvm/test/CodeGen/SystemZ/callbr-asm-flag-outputs.ll
@@ -0,0 +1,33 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
+; RUN: llc -mtriple=s390x-linux-gnu -verify-machineinstrs < %s | FileCheck %s
+
+; A condition code output that both paths out of an asm goto read. CC is live
+; into the indirect target, which reads it again: the read after the asm
+; doesn't run when the asm branches there.
+define i32 @test_flag_output_on_both_paths() {
+; CHECK-LABEL: test_flag_output_on_both_paths:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    ipm %r2
+; CHECK-NEXT:  # %bb.1: # %ft
+; CHECK-NEXT:    srl %r2, 28
+; CHECK-NEXT:    br %r14
+; CHECK-NEXT:  .LBB0_2: # Inline asm indirect target
+; CHECK-NEXT:    # %ind
+; CHECK-NEXT:    # Label of block must be emitted
+; CHECK-NEXT:    ipm %r2
+; CHECK-NEXT:    srl %r2, 28
+; CHECK-NEXT:    ahi %r2, 7
+; CHECK-NEXT:    br %r14
+entry:
+  %r = callbr i32 asm "", "={@cc},!i"()
+          to label %ft [label %ind]
+
+ft:
+  ret i32 %r
+
+ind:
+  %s = add i32 %r, 7
+  ret i32 %s
+}
diff --git a/llvm/test/CodeGen/X86/callbr-asm-flag-outputs.ll b/llvm/test/CodeGen/X86/callbr-asm-flag-outputs.ll
new file mode 100644
index 00000000000000..cb873e02e07a75
--- /dev/null
+++ b/llvm/test/CodeGen/X86/callbr-asm-flag-outputs.ll
@@ -0,0 +1,347 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
+; RUN: llc -mtriple=x86_64-linux-gnu -O0 -verify-machineinstrs < %s | FileCheck %s --check-prefix=O0
+; RUN: llc -mtriple=x86_64-linux-gnu -O2 -verify-machineinstrs < %s | FileCheck %s --check-prefix=O2
+
+; An asm goto's indirect target reads the asm's flag outputs from the EFLAGS
+; that live into it from the asm. Nothing after the asm in its block runs on
+; the way there, so the indirect target can't use a flag read or an EFLAGS
+; copy from there, and it has to read EFLAGS before anything clobbers it.
+;
+; Each opt level hits problems that the other doesn't. At -O0, the fallthrough
+; path copies EFLAGS after the asm, and every output is exported. At -O2,
+; MachineCSE runs, and the scheduler can move a clobber ahead of a flag read.
+
+; A flag output that both paths out of an asm goto read (#60827). EFLAGS is
+; live into the indirect target, which computes its own sete: the one after
+; the asm doesn't run when the asm branches there.
+define i8 @test_flag_output_on_both_paths() {
+; O0-LABEL: test_flag_output_on_both_paths:
+; O0:       # %bb.0: # %entry
+; O0-NEXT:    #APP
+; O0-NEXT:    #NO_APP
+; O0-NEXT:    sete %al
+; O0-NEXT:    movb %al, {{[-0-9]+}}(%r{{[sb]}}p) # 1-byte Spill
+; O0-NEXT:    jmp .LBB0_1
+; O0-NEXT:  .LBB0_1: # %ft
+; O0-NEXT:    movb {{[-0-9]+}}(%r{{[sb]}}p), %al # 1-byte Reload
+; O0-NEXT:    retq
+; O0-NEXT:  .LBB0_2: # Inline asm indirect target
+; O0-NEXT:    # %ind
+; O0-NEXT:    # Label of block must be emitted
+; O0-NEXT:    sete %al
+; O0-NEXT:    addb $7, %al
+; O0-NEXT:    retq
+;
+; O2-LABEL: test_flag_output_on_both_paths:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    #APP
+; O2-NEXT:    #NO_APP
+; O2-NEXT:    sete %al
+; O2-NEXT:  # %bb.1: # %ft
+; O2-NEXT:    retq
+; O2-NEXT:  .LBB0_2: # Inline asm indirect target
+; O2-NEXT:    # %ind
+; O2-NEXT:    # Label of block must be emitted
+; O2-NEXT:    sete %al
+; O2-NEXT:    addb $7, %al
+; O2-NEXT:    retq
+entry:
+  %r = callbr i8 asm "", "={@ccz},!i,~{dirflag},~{fpsr},~{flags}"()
+          to label %ft [label %ind]
+
+ft:
+  ret i8 %r
+
+ind:
+  %s = add i8 %r, 7
+  ret i8 %s
+}
+
+; Two flag outputs, which the asm's fallthrough path reads through an EFLAGS
+; copy. That copy is after the asm, so the indirect target can't use it and
+; reads EFLAGS itself.
+define i8 @test_two_flag_outputs() {
+; O0-LABEL: test_two_flag_outputs:
+; O0:       # %bb.0: # %entry
+; O0-NEXT:    #APP
+; O0-NEXT:    #NO_APP
+; O0-NEXT:    sete %al
+; O0-NEXT:    movb %al, {{[-0-9]+}}(%r{{[sb]}}p) # 1-byte Spill
+; O0-NEXT:    setb %al
+; O0-NEXT:    movb %al, {{[-0-9]+}}(%r{{[sb]}}p) # 1-byte Spill
+; O0-NEXT:    jmp .LBB1_1
+; O0-NEXT:  .LBB1_1: # %ft
+; O0-NEXT:    movb {{[-0-9]+}}(%r{{[sb]}}p), %al # 1-byte Reload
+; O0-NEXT:    movb {{[-0-9]+}}(%r{{[sb]}}p), %cl # 1-byte Reload
+; O0-NEXT:    addb %cl, %al
+; O0-NEXT:    retq
+; O0-NEXT:  .LBB1_2: # Inline asm indirect target
+; O0-NEXT:    # %ind
+; O0-NEXT:    # Label of block must be emitted
+; O0-NEXT:    setb %cl
+; O0-NEXT:    sete %al
+; O0-NEXT:    subb %cl, %al
+; O0-NEXT:    retq
+;
+; O2-LABEL: test_two_flag_outputs:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    #APP
+; O2-NEXT:    #NO_APP
+; O2-NEXT:    sete %al
+; O2-NEXT:    setb %cl
+; O2-NEXT:  # %bb.1: # %ft
+; O2-NEXT:    addb %cl, %al
+; O2-NEXT:    retq
+; O2-NEXT:  .LBB1_2: # Inline asm indirect target
+; O2-NEXT:    # %ind
+; O2-NEXT:    # Label of block must be emitted
+; O2-NEXT:    sete %al
+; O2-NEXT:    sbbb $0, %al
+; O2-NEXT:    retq
+entry:
+  %r = callbr { i8, i8 } asm "", "={@ccz},={@ccc},!i,~{dirflag},~{fpsr},~{flags}"()
+          to label %ft [label %ind]
+
+ft:
+  %a = extractvalue { i8, i8 } %r, 0
+  %b = extractvalue { i8, i8 } %r, 1
+  %s = add i8 %a, %b
+  ret i8 %s
+
+ind:
+  %c = extractvalue { i8, i8 } %r, 0
+  %d = extractvalue { i8, i8 } %r, 1
+  %t = sub i8 %c, %d
+  ret i8 %t
+}
+
+; A flag output and a tied register output, as in the kernel's try_cmpxchg.
+define i64 @test_flag_and_tied_outputs(i64 %x) {
+; O0-LABEL: test_flag_and_tied_outputs:
+; O0:       # %bb.0: # %entry
+; O0-NEXT:    movq %rdi, %rax
+; O0-NEXT:    #APP
+; O0-NEXT:    #NO_APP
+; O0-NEXT:    sete %cl
+; O0-NEXT:    movb %cl, {{[-0-9]+}}(%r{{[sb]}}p) # 1-byte Spill
+; O0-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; O0-NEXT:    jmp .LBB2_1
+; O0-NEXT:  .LBB2_1: # %ft
+; O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rcx # 8-byte Reload
+; O0-NEXT:    movb {{[-0-9]+}}(%r{{[sb]}}p), %al # 1-byte Reload
+; O0-NEXT:    movzbl %al, %eax
+; O0-NEXT:    # kill: def $rax killed $eax
+; O0-NEXT:    addq %rcx, %rax
+; O0-NEXT:    retq
+; O0-NEXT:  .LBB2_2: # Inline asm indirect target
+; O0-NEXT:    # %ind
+; O0-NEXT:    # Label of block must be emitted
+; O0-NEXT:    sete %al
+; O0-NEXT:    # implicit-def: $rcx
+; O0-NEXT:    movzbl %al, %eax
+; O0-NEXT:    # kill: def $rax killed $eax
+; O0-NEXT:    retq
+;
+; O2-LABEL: test_flag_and_tied_outputs:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    xorl %eax, %eax
+; O2-NEXT:    #APP
+; O2-NEXT:    #NO_APP
+; O2-NEXT:    sete %cl
+; O2-NEXT:  # %bb.1: # %ft
+; O2-NEXT:    movb %cl, %al
+; O2-NEXT:    addq %rdi, %rax
+; O2-NEXT:    retq
+; O2-NEXT:  .LBB2_2: # Inline asm indirect target
+; O2-NEXT:    # %ind
+; O2-NEXT:    # Label of block must be emitted
+; O2-NEXT:    sete %al
+; O2-NEXT:    movzbl %al, %eax
+; O2-NEXT:    retq
+entry:
+  %r = callbr { i8, i64 } asm "", "={@ccz},=r,1,!i,~{dirflag},~{fpsr},~{flags}"(i64 %x)
+          to label %ft [label %ind]
+
+ft:
+  %a = extractvalue { i8, i64 } %r, 0
+  %b = extractvalue { i8, i64 } %r, 1
+  %az = zext i8 %a to i64
+  %s = add i64 %az, %b
+  ret i64 %s
+
+ind:
+  %c = extractvalue { i8, i64 } %r, 0
+  %cz = zext i8 %c to i64
+  ret i64 %cz
+}
+
+; Two flag outputs and a tied register output.
+define i8 @test_two_flags_and_tied_outputs(i64 %x) {
+; O0-LABEL: test_two_flags_and_tied_outputs:
+; O0:       # %bb.0: # %entry
+; O0-NEXT:    movq %rdi, %rax
+; O0-NEXT:    #APP
+; O0-NEXT:    #NO_APP
+; O0-NEXT:    sete %cl
+; O0-NEXT:    movb %cl, {{[-0-9]+}}(%r{{[sb]}}p) # 1-byte Spill
+; O0-NEXT:    setb %cl
+; O0-NEXT:    movb %cl, {{[-0-9]+}}(%r{{[sb]}}p) # 1-byte Spill
+; O0-NEXT:    jmp .LBB3_1
+; O0-NEXT:  .LBB3_1: # %ft
+; O0-NEXT:    movb {{[-0-9]+}}(%r{{[sb]}}p), %al # 1-byte Reload
+; O0-NEXT:    movb {{[-0-9]+}}(%r{{[sb]}}p), %cl # 1-byte Reload
+; O0-NEXT:    addb %cl, %al
+; O0-NEXT:    retq
+; O0-NEXT:  .LBB3_2: # Inline asm indirect target
+; O0-NEXT:    # %ind
+; O0-NEXT:    # Label of block must be emitted
+; O0-NEXT:    setb %cl
+; O0-NEXT:    sete %al
+; O0-NEXT:    # implicit-def: $rdx
+; O0-NEXT:    subb %cl, %al
+; O0-NEXT:    retq
+;
+; O2-LABEL: test_two_flags_and_tied_outputs:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    #APP
+; O2-NEXT:    #NO_APP
+; O2-NEXT:    sete %al
+; O2-NEXT:    setb %cl
+; O2-NEXT:  # %bb.1: # %ft
+; O2-NEXT:    addb %cl, %al
+; O2-NEXT:    retq
+; O2-NEXT:  .LBB3_2: # Inline asm indirect target
+; O2-NEXT:    # %ind
+; O2-NEXT:    # Label of block must be emitted
+; O2-NEXT:    sete %al
+; O2-NEXT:    sbbb $0, %al
+; O2-NEXT:    retq
+entry:
+  %r = callbr { i8, i8, i64 } asm "", "={@ccz},={@ccc},=r,2,!i,~{dirflag},~{fpsr},~{flags}"(i64 %x)
+          to label %ft [label %ind]
+
+ft:
+  %a = extractvalue { i8, i8, i64 } %r, 0
+  %b = extractvalue { i8, i8, i64 } %r, 1
+  %s = add i8 %a, %b
+  ret i8 %s
+
+ind:
+  %c = extractvalue { i8, i8, i64 } %r, 0
+  %d = extractvalue { i8, i8, i64 } %r, 1
+  %t = sub i8 %c, %d
+  ret i8 %t
+}
+
+
+; The indirect target computes a product, which clobbers EFLAGS, and adds the
+; asm's carry flag to it.
+define i32 @test_flag_output_read_after_clobber(i32 %x, i32 %y) {
+; O0-LABEL: test_flag_output_read_after_clobber:
+; O0:       # %bb.0: # %entry
+; O0-NEXT:    movl %edi, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; O0-NEXT:    movl %esi, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; O0-NEXT:    #APP
+; O0-NEXT:    #NO_APP
+; O0-NEXT:    setb %al
+; O0-NEXT:    jmp .LBB4_1
+; O0-NEXT:  .LBB4_1: # %ft
+; O0-NEXT:    xorl %eax, %eax
+; O0-NEXT:    retq
+; O0-NEXT:  .LBB4_2: # Inline asm indirect target
+; O0-NEXT:    # %ind
+; O0-NEXT:    # Label of block must be emitted
+; O0-NEXT:    movl {{[-0-9]+}}(%r{{[sb]}}p), %eax # 4-byte Reload
+; O0-NEXT:    movl {{[-0-9]+}}(%r{{[sb]}}p), %edx # 4-byte Reload
+; O0-NEXT:    setb %cl
+; O0-NEXT:    imull %edx, %eax
+; O0-NEXT:    movzbl %cl, %ecx
+; O0-NEXT:    addl %ecx, %eax
+; O0-NEXT:    retq
+;
+; O2-LABEL: test_flag_output_read_after_clobber:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    #APP
+; O2-NEXT:    #NO_APP
+; O2-NEXT:  # %bb.1: # %ft
+; O2-NEXT:    xorl %eax, %eax
+; O2-NEXT:    retq
+; O2-NEXT:  .LBB4_2: # Inline asm indirect target
+; O2-NEXT:    # %ind
+; O2-NEXT:    # Label of block must be emitted
+; O2-NEXT:    setb %al
+; O2-NEXT:    imull %esi, %edi
+; O2-NEXT:    addb $255, %al
+; O2-NEXT:    adcl $0, %edi
+; O2-NEXT:    movl %edi, %eax
+; O2-NEXT:    retq
+entry:
+  %r = callbr i8 asm "", "={@ccc},!i,~{dirflag},~{fpsr},~{flags}"()
+          to label %ft [label %ind]
+
+ft:
+  ret i32 0
+
+ind:
+  %p = mul i32 %x, %y
+  %rz = zext i8 %r to i32
+  %s = add i32 %p, %rz
+  ret i32 %s
+}
+
+; The indirect target doesn't use the flag output, so EFLAGS isn't live into
+; it. The fallthrough path clobbers EFLAGS after the asm, which an EFLAGS
+; live-in would contradict.
+define i64 @test_unused_flag_output(i64 %x) {
+; O0-LABEL: test_unused_flag_output:
+; O0:       # %bb.0: # %entry
+; O0-NEXT:    movq %rdi, %rax
+; O0-NEXT:    #APP
+; O0-NEXT:    #NO_APP
+; O0-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; O0-NEXT:    sete %cl
+; O0-NEXT:    movzbl %cl, %ecx
+; O0-NEXT:    # kill: def $rcx killed $ecx
+; O0-NEXT:    movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; O0-NEXT:    xorl %ecx, %ecx
+; O0-NEXT:    # kill: def $rcx killed $ecx
+; O0-NEXT:    jmp .LBB5_1
+; O0-NEXT:  .LBB5_1: # %ft
+; O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; O0-NEXT:    retq
+; O0-NEXT:  .LBB5_2: # Inline asm indirect target
+; O0-NEXT:    # %ind
+; O0-NEXT:    # Label of block must be emitted
+; O0-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; O0-NEXT:    # implicit-def: $rcx
+; O0-NEXT:    # implicit-def: $rcx
+; O0-NEXT:    retq
+;
+; O2-LABEL: test_unused_flag_output:
+; O2:       # %bb.0: # %entry
+; O2-NEXT:    xorl %eax, %eax
+; O2-NEXT:    #APP
+; O2-NEXT:    #NO_APP
+; O2-NEXT:    sete %cl
+; O2-NEXT:  # %bb.1: # %ft
+; O2-NEXT:    movb %cl, %al
+; O2-NEXT:    retq
+; O2-NEXT:  .LBB5_2: # Inline asm indirect target
+; O2-NEXT:    # %ind
+; O2-NEXT:    # Label of block must be emitted
+; O2-NEXT:    movq %rdi, %rax
+; O2-NEXT:    retq
+entry:
+  %r = callbr { i128, i64 } asm "", "={@ccz},=r,1,!i,~{dirflag},~{fpsr},~{flags}"(i64 %x)
+          to label %ft [label %ind]
+
+ft:
+  %a = extractvalue { i128, i64 } %r, 0
+  %t = trunc i128 %a to i64
+  ret i64 %t
+
+ind:
+  %b = extractvalue { i128, i64 } %r, 1
+  ret i64 %b
+}
diff --git a/llvm/test/CodeGen/X86/callbr-asm-outputs-indirect-isel-m32.ll b/llvm/test/CodeGen/X86/callbr-asm-outputs-indirect-isel-m32.ll
index 4bf0e64c1d82e5..4f88348fa987f5 100644
--- a/llvm/test/CodeGen/X86/callbr-asm-outputs-indirect-isel-m32.ll
+++ b/llvm/test/CodeGen/X86/callbr-asm-outputs-indirect-isel-m32.ll
@@ -1,8 +1,10 @@
 ; NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
-; FIXME(ndesaulniers): get this test to pass with -verify-machineinstrs
-; enabled. https://github.com/llvm/llvm-project/issues/60827
 ; RUN: llc -mtriple=i386-linux-gnu %s -o - -stop-after=finalize-isel \
-; RUN:   -verify-machineinstrs=0 -disable-mir-output-verify -start-before=x86-isel | FileCheck %s
+; RUN:   -verify-machineinstrs -start-before=x86-isel | FileCheck %s
+
+; The flag outputs are read from EFLAGS on both paths. Their asms clobber
+; EFLAGS ("~{flags}"), as every asm from Clang does on x86: a flag output
+; alone doesn't make the asm define EFLAGS.
 
 define i8 @emulator_cmpxchg_emulated() {
   ; CHECK-LABEL: name: emulator_cmpxchg_emulated
@@ -11,7 +13,7 @@ define i8 @emulator_cmpxchg_emulated() {
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   [[MOV32rm:%[0-9]+]]:gr32 = MOV32rm $noreg, 1, $noreg, 0, $noreg :: (load (s32) from `ptr null`, align 8)
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:gr32_norex2 = COPY [[MOV32rm]]
-  ; CHECK-NEXT:   INLINEASM_BR &"", maystore attdialect, regdef:GR32, def %2, regdef:GR32_NOREX2, def %3, reguse tiedto:$1, [[COPY]](tied-def 5), imm, %bb.2
+  ; CHECK-NEXT:   INLINEASM_BR &"", maystore attdialect, regdef:GR32, def %2, regdef:GR32_NOREX2, def %3, reguse tiedto:$1, [[COPY]](tied-def 5), imm, %bb.2, clobber, implicit-def early-clobber $eflags
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:gr32 = COPY $eflags
   ; CHECK-NEXT:   $eflags = COPY [[COPY1]]
   ; CHECK-NEXT:   [[SETCCr:%[0-9]+]]:gr8 = SETCCr 4, implicit $eflags
@@ -23,12 +25,16 @@ define i8 @emulator_cmpxchg_emulated() {
   ; CHECK-NEXT:   RET 0, $al
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.2.efaultu64.split (label-must-be-emitted, inlineasm-br-indirect-target):
+  ; CHECK-NEXT:   liveins: $eflags
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   [[COPY3:%[0-9]+]]:gr32 = COPY $eflags
+  ; CHECK-NEXT:   $eflags = COPY [[COPY3]]
   ; CHECK-NEXT:   [[SETCCr1:%[0-9]+]]:gr8 = SETCCr 4, implicit $eflags
   ; CHECK-NEXT:   $al = COPY [[SETCCr1]]
   ; CHECK-NEXT:   RET 0, $al
 entry:
   %0 = load i32, ptr null, align 8
-  %1 = callbr { i8, i32 } asm "", "={@ccz},=r,1,!i"(i32 %0)
+  %1 = callbr { i8, i32 } asm "", "={@ccz},=r,1,!i,~{flags}"(i32 %0)
           to label %asm.fallthrough [label %efaultu64.split]
 
 asm.fallthrough:
@@ -51,7 +57,7 @@ define i32 @emulator_cmpxchg_emulated2() {
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   [[MOV32rm:%[0-9]+]]:gr32 = MOV32rm $noreg, 1, $noreg, 0, $noreg :: (load (s32) from `ptr null`, align 8)
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:gr32_norex2 = COPY [[MOV32rm]]
-  ; CHECK-NEXT:   INLINEASM_BR &"", maystore attdialect, regdef:GR32, def %2, regdef:GR32_NOREX2, def %3, reguse tiedto:$1, [[COPY]](tied-def 5), imm, %bb.2
+  ; CHECK-NEXT:   INLINEASM_BR &"", maystore attdialect, regdef:GR32, def %2, regdef:GR32_NOREX2, def %3, reguse tiedto:$1, [[COPY]](tied-def 5), imm, %bb.2, clobber, implicit-def early-clobber $eflags
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:gr32 = COPY $eflags
   ; CHECK-NEXT:   $eflags = COPY [[COPY1]]
   ; CHECK-NEXT:   [[SETCCr:%[0-9]+]]:gr8 = SETCCr 4, implicit $eflags
@@ -67,7 +73,7 @@ define i32 @emulator_cmpxchg_emulated2() {
   ; CHECK-NEXT:   RET 0, $eax
 entry:
   %0 = load i32, ptr null, align 8
-  %1 = callbr { i8, i32 } asm "", "={@ccz},=r,1,!i"(i32 %0)
+  %1 = callbr { i8, i32 } asm "", "={@ccz},=r,1,!i,~{flags}"(i32 %0)
           to label %asm.fallthrough [label %efaultu64.split]
 
 asm.fallthrough:
@@ -102,10 +108,10 @@ define i64 @multireg() {
   ; CHECK-NEXT: bb.2.split (label-must-be-emitted, inlineasm-br-indirect-target):
   ; CHECK-NEXT:   liveins: $eax, $edx
   ; CHECK-NEXT: {{  $}}
-  ; CHECK-NEXT:   [[COPY4:%[0-9]+]]:gr32 = COPY $eax
-  ; CHECK-NEXT:   [[COPY5:%[0-9]+]]:gr32 = COPY $edx
-  ; CHECK-NEXT:   $eax = COPY [[COPY4]]
-  ; CHECK-NEXT:   $edx = COPY [[COPY5]]
+  ; CHECK-NEXT:   [[COPY4:%[0-9]+]]:gr32 = COPY killed $edx
+  ; CHECK-NEXT:   [[COPY5:%[0-9]+]]:gr32 = COPY killed $eax
+  ; CHECK-NEXT:   $eax = COPY [[COPY5]]
+  ; CHECK-NEXT:   $edx = COPY [[COPY4]]
   ; CHECK-NEXT:   RET 0, $eax, $edx
 entry:
   %0 = callbr i64 asm "", "=A,!i"() to label %ft [label %split]
diff --git a/llvm/test/CodeGen/X86/callbr-asm-outputs-indirect-isel.ll b/llvm/test/CodeGen/X86/callbr-asm-outputs-indirect-isel.ll
index 2c480d59dc0f15..c432d37af13868 100644
--- a/llvm/test/CodeGen/X86/callbr-asm-outputs-indirect-isel.ll
+++ b/llvm/test/CodeGen/X86/callbr-asm-outputs-indirect-isel.ll
@@ -1,8 +1,6 @@
 ; NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
-; FIXME(ndesaulniers): get this test to pass with -verify-machineinstrs
-; enabled. https://github.com/llvm/llvm-project/issues/60827
 ; RUN: llc -mtriple=x86_64-linux-gnu %s -o - -stop-after=finalize-isel \
-; RUN:   -verify-machineinstrs=0 -disable-mir-output-verify -start-before=x86-isel | FileCheck %s
+; RUN:   -verify-machineinstrs -start-before=x86-isel | FileCheck %s
 
 ; One virtual register, w/o phi
 define i32 @test0() {
@@ -116,7 +114,7 @@ define i32 @test3() {
   ; CHECK-NEXT:   successors: %bb.2(0x80000000)
   ; CHECK-NEXT:   liveins: $ebx
   ; CHECK-NEXT: {{  $}}
-  ; CHECK-NEXT:   [[COPY2:%[0-9]+]]:gr32 = COPY $ebx
+  ; CHECK-NEXT:   [[COPY2:%[0-9]+]]:gr32 = COPY killed $ebx
   ; CHECK-NEXT:   [[COPY3:%[0-9]+]]:gr32 = COPY [[COPY2]]
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.2.cleanup:
@@ -152,9 +150,9 @@ define i32 @test4() {
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.1.z.split (label-must-be-emitted, inlineasm-br-indirect-target):
   ; CHECK-NEXT:   successors: %bb.2(0x80000000)
-  ; CHECK-NEXT:   liveins: $ebx, $edx
+  ; CHECK-NEXT:   liveins: $ebx
   ; CHECK-NEXT: {{  $}}
-  ; CHECK-NEXT:   [[COPY4:%[0-9]+]]:gr32 = COPY $ebx
+  ; CHECK-NEXT:   [[COPY4:%[0-9]+]]:gr32 = COPY killed $ebx
   ; CHECK-NEXT:   [[COPY5:%[0-9]+]]:gr32 = COPY [[COPY4]]
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.2.cleanup:
@@ -190,7 +188,7 @@ define i32 @test5() {
   ; CHECK-NEXT: bb.1.cleanup (label-must-be-emitted, inlineasm-br-indirect-target):
   ; CHECK-NEXT:   liveins: $ebx
   ; CHECK-NEXT: {{  $}}
-  ; CHECK-NEXT:   [[COPY2:%[0-9]+]]:gr32 = COPY $ebx
+  ; CHECK-NEXT:   [[COPY2:%[0-9]+]]:gr32 = COPY killed $ebx
   ; CHECK-NEXT:   $eax = COPY [[COPY2]]
   ; CHECK-NEXT:   RET 0, $eax
 entry:
@@ -231,7 +229,7 @@ define i64 @test6() {
   ; CHECK-NEXT:   successors: %bb.2(0x80000000)
   ; CHECK-NEXT:   liveins: $rdx
   ; CHECK-NEXT: {{  $}}
-  ; CHECK-NEXT:   [[COPY4:%[0-9]+]]:gr64 = COPY $rdx
+  ; CHECK-NEXT:   [[COPY4:%[0-9]+]]:gr64 = COPY killed $rdx
   ; CHECK-NEXT:   [[COPY5:%[0-9]+]]:gr64 = COPY [[COPY4]]
   ; CHECK-NEXT:   JMP_1 %bb.2
   ; CHECK-NEXT: {{  $}}
@@ -239,7 +237,7 @@ define i64 @test6() {
   ; CHECK-NEXT:   successors: %bb.2(0x80000000)
   ; CHECK-NEXT:   liveins: $rbx
   ; CHECK-NEXT: {{  $}}
-  ; CHECK-NEXT:   [[COPY6:%[0-9]+]]:gr64 = COPY $rbx
+  ; CHECK-NEXT:   [[COPY6:%[0-9]+]]:gr64 = COPY killed $rbx
   ; CHECK-NEXT:   [[COPY7:%[0-9]+]]:gr64 = COPY [[COPY6]]
   ; CHECK-NEXT:   JMP_1 %bb.2
 entry:
@@ -290,7 +288,7 @@ define i32 @test7() {
   ; CHECK-NEXT:   successors: %bb.1(0x80000000)
   ; CHECK-NEXT:   liveins: $edx
   ; CHECK-NEXT: {{  $}}
-  ; CHECK-NEXT:   [[COPY3:%[0-9]+]]:gr32 = COPY $edx
+  ; CHECK-NEXT:   [[COPY3:%[0-9]+]]:gr32 = COPY killed $edx
   ; CHECK-NEXT:   [[COPY4:%[0-9]+]]:gr32 = COPY [[COPY3]]
   ; CHECK-NEXT:   JMP_1 %bb.1
 entry:
@@ -333,12 +331,15 @@ cleanup:
   ret i32 %indirect
 }
 
+; A flag output, which both targets read from EFLAGS. The asm clobbers EFLAGS
+; ("~{flags}"), as every asm from Clang does on x86: the flag output alone
+; doesn't make the asm define EFLAGS.
 define i64 @condition_code() {
   ; CHECK-LABEL: name: condition_code
   ; CHECK: bb.0 (%ir-block.0):
   ; CHECK-NEXT:   successors: %bb.1(0x80000000), %bb.2(0x00000000)
   ; CHECK-NEXT: {{  $}}
-  ; CHECK-NEXT:   INLINEASM_BR &"", maystore attdialect, regdef:GR32, def %1, imm, %bb.2
+  ; CHECK-NEXT:   INLINEASM_BR &"", maystore attdialect, regdef:GR32, def %1, imm, %bb.2, clobber, implicit-def early-clobber $eflags
   ; CHECK-NEXT:   [[SETCCr:%[0-9]+]]:gr8 = SETCCr 4, implicit $eflags
   ; CHECK-NEXT:   [[MOVZX32rr8_:%[0-9]+]]:gr32 = MOVZX32rr8 [[SETCCr]]
   ; CHECK-NEXT:   [[SUBREG_TO_REG:%[0-9]+]]:gr64 = SUBREG_TO_REG [[MOVZX32rr8_]], %subreg.sub_32bit
@@ -349,12 +350,16 @@ define i64 @condition_code() {
   ; CHECK-NEXT:   RET 0, $rax
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.2.c (label-must-be-emitted, inlineasm-br-indirect-target):
+  ; CHECK-NEXT:   liveins: $eflags
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   [[COPY:%[0-9]+]]:gr64 = COPY $eflags
+  ; CHECK-NEXT:   $eflags = COPY [[COPY]]
   ; CHECK-NEXT:   [[SETCCr1:%[0-9]+]]:gr8 = SETCCr 4, implicit $eflags
   ; CHECK-NEXT:   [[MOVZX32rr8_1:%[0-9]+]]:gr32 = MOVZX32rr8 [[SETCCr1]]
   ; CHECK-NEXT:   [[SUBREG_TO_REG1:%[0-9]+]]:gr64 = SUBREG_TO_REG [[MOVZX32rr8_1]], %subreg.sub_32bit
   ; CHECK-NEXT:   $rax = COPY [[SUBREG_TO_REG1]]
   ; CHECK-NEXT:   RET 0, $rax
-  %a = callbr i64 asm "", "={@ccz},!i"()
+  %a = callbr i64 asm "", "={@ccz},!i,~{flags}"()
        to label %b [label %c]
 
 b:
@@ -365,9 +370,53 @@ c:
   ret i64 %1
 }
 
+; Two flag outputs, which the indirect target both reads from EFLAGS. EFLAGS
+; is live into it once.
+define i8 @two_condition_codes() {
+  ; CHECK-LABEL: name: two_condition_codes
+  ; CHECK: bb.0 (%ir-block.0):
+  ; CHECK-NEXT:   successors: %bb.1(0x80000000), %bb.2(0x00000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   INLINEASM_BR &"", maystore attdialect, regdef:GR32, def %2, regdef:GR32, def %3, imm, %bb.2, clobber, implicit-def early-clobber $eflags
+  ; CHECK-NEXT:   [[COPY:%[0-9]+]]:gr32 = COPY $eflags
+  ; CHECK-NEXT:   $eflags = COPY [[COPY]]
+  ; CHECK-NEXT:   [[SETCCr:%[0-9]+]]:gr8 = SETCCr 4, implicit $eflags
+  ; CHECK-NEXT:   [[SETCCr1:%[0-9]+]]:gr8 = SETCCr 2, implicit $eflags
+  ; CHECK-NEXT:   JMP_1 %bb.1
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.1.b:
+  ; CHECK-NEXT:   $al = COPY [[SETCCr]]
+  ; CHECK-NEXT:   RET 0, $al
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.2.c (label-must-be-emitted, inlineasm-br-indirect-target):
+  ; CHECK-NEXT:   liveins: $eflags
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:gr64 = COPY $eflags
+  ; CHECK-NEXT:   $eflags = COPY [[COPY1]]
+  ; CHECK-NEXT:   [[SETCCr2:%[0-9]+]]:gr8 = SETCCr 4, implicit $eflags
+  ; CHECK-NEXT:   $eflags = COPY [[COPY1]]
+  ; CHECK-NEXT:   [[ADC8ri:%[0-9]+]]:gr8 = ADC8ri [[SETCCr2]], 0, implicit-def dead $eflags, implicit $eflags
+  ; CHECK-NEXT:   $al = COPY [[ADC8ri]]
+  ; CHECK-NEXT:   RET 0, $al
+  %a = callbr { i8, i8 } asm "", "={@ccz},={@ccc},!i,~{flags}"()
+       to label %b [label %c]
+
+b:
+  %z = extractvalue { i8, i8 } %a, 0
+  ret i8 %z
+
+c:
+  %1 = call { i8, i8 } @llvm.callbr.landingpad.sl_i8i8s({ i8, i8 } %a)
+  %z1 = extractvalue { i8, i8 } %1, 0
+  %c1 = extractvalue { i8, i8 } %1, 1
+  %r = add i8 %z1, %c1
+  ret i8 %r
+}
+
 declare i64 @llvm.callbr.landingpad.i64(i64)
 declare i32 @llvm.callbr.landingpad.i32(i32)
 declare { i32, i32 } @llvm.callbr.landingpad.sl_i32i32s({ i32, i32 })
+declare { i8, i8 } @llvm.callbr.landingpad.sl_i8i8s({ i8, i8 })
 
 ; A float in a general-purpose register, which the indirect target bitcasts
 ; from the register the INLINEASM_BR defines.
@@ -468,7 +517,7 @@ define float @float_in_eax(i32 %x) {
   ; CHECK-NEXT: bb.2.z.split (label-must-be-emitted, inlineasm-br-indirect-target):
   ; CHECK-NEXT:   liveins: $eax
   ; CHECK-NEXT: {{  $}}
-  ; CHECK-NEXT:   [[COPY3:%[0-9]+]]:gr32 = COPY $eax
+  ; CHECK-NEXT:   [[COPY3:%[0-9]+]]:gr32 = COPY killed $eax
   ; CHECK-NEXT:   [[MOVDI2SSrr1:%[0-9]+]]:fr32 = MOVDI2SSrr [[COPY3]]
   ; CHECK-NEXT:   $xmm0 = COPY [[MOVDI2SSrr1]]
   ; CHECK-NEXT:   RET 0, $xmm0
diff --git a/llvm/test/CodeGen/X86/callbr-asm-outputs.ll b/llvm/test/CodeGen/X86/callbr-asm-outputs.ll
index 6ee43099209d29..b9c8ebeec75800 100644
--- a/llvm/test/CodeGen/X86/callbr-asm-outputs.ll
+++ b/llvm/test/CodeGen/X86/callbr-asm-outputs.ll
@@ -119,25 +119,26 @@ define i32 @test3(i1 %cmp) nounwind {
 ; CHECK-NEXT:    #NO_APP
 ; CHECK-NEXT:  # %bb.2:
 ; CHECK-NEXT:    movl %edi, %eax
-; CHECK-NEXT:    jmp .LBB2_4
+; CHECK-NEXT:    jmp .LBB2_5
 ; CHECK-NEXT:  .LBB2_3: # %false
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    .short %eax
 ; CHECK-NEXT:    .short %edx
 ; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:  # %bb.4:
 ; CHECK-NEXT:    movl %edx, %eax
-; CHECK-NEXT:  .LBB2_4: # %asm.fallthrough
+; CHECK-NEXT:  .LBB2_5: # %asm.fallthrough
 ; CHECK-NEXT:    popl %esi
 ; CHECK-NEXT:    popl %edi
 ; CHECK-NEXT:    retl
-; CHECK-NEXT:  .LBB2_5: # Inline asm indirect target
+; CHECK-NEXT:  .LBB2_6: # Inline asm indirect target
 ; CHECK-NEXT:    # %true.indirect_crit_edge
 ; CHECK-NEXT:    # Label of block must be emitted
-; CHECK-NEXT:  .LBB2_6: # Inline asm indirect target
+; CHECK-NEXT:  .LBB2_7: # Inline asm indirect target
 ; CHECK-NEXT:    # %false.indirect_crit_edge
 ; CHECK-NEXT:    # Label of block must be emitted
 ; CHECK-NEXT:    movl $42, %eax
-; CHECK-NEXT:    jmp .LBB2_4
+; CHECK-NEXT:    jmp .LBB2_5
 entry:
   br i1 %cmp, label %true, label %false
 

>From a0bbf18ff026abc0a68500acf9f9f274536f94da Mon Sep 17 00:00:00 2001
From: Bill Wendling <isanbard at gmail.com>
Date: Wed, 4 Nov 2026 23:20:52 -0800
Subject: [PATCH 2/5] Add blank lines after control-flow blocks and before
 comments

---
 llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index e2450095fa578d..7b9f1e423c319f 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -13348,6 +13348,7 @@ static void readPhysRegsAtBlockEntry(SDValue &V, MachineBasicBlock &MBB,
   HandleSDNode Handle(V);
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
   const TargetRegisterInfo *TRI = DAG.getSubtarget().getRegisterInfo();
+
   // The walk found the reads last operand first. Copy them in operand order,
   // which is the order of an output's registers.
   for (SDNode *N : reverse(PhysRegReads)) {
@@ -13383,6 +13384,7 @@ static void readPhysRegsAtBlockEntry(SDValue &V, MachineBasicBlock &MBB,
     SDValue Read = DAG.getCopyFromReg(N->getOperand(0), DL, VReg, VT);
     DAG.ReplaceAllUsesWith(N, Read.getNode());
   }
+
   V = Handle.getValue();
 }
 
@@ -13393,6 +13395,7 @@ static SmallBitVector getUsedCallBrOutputs(const CallInst &I,
   if (!I.getType()->isStructTy()) {
     if (!I.use_empty())
       Used.set();
+
     return Used;
   }
 
@@ -13407,6 +13410,7 @@ static SmallBitVector getUsedCallBrOutputs(const CallInst &I,
     if (!EV->use_empty())
       Used.set(EV->getIndices()[0]);
   }
+
   return Used;
 }
 
@@ -13430,6 +13434,7 @@ void SelectionDAGBuilder::visitCallBrLandingPad(const CallInst &I) {
   SmallVector<SDValue, 8> OutputValues;
   for (auto [Idx, Output] : enumerate(It->second)) {
     RegsForValue Regs(Output.Regs, Output.RegVT, Output.ValueVT);
+
     // Each output's reads only follow the block's entry, so that an unused
     // output's reads have no users left.
     SDValue Chain = DAG.getRoot();
@@ -13440,6 +13445,7 @@ void SelectionDAGBuilder::visitCallBrLandingPad(const CallInst &I) {
       readPhysRegsAtBlockEntry(V, *FuncInfo.MBB, DAG, getCurSDLoc());
     else
       V = DAG.getUNDEF(V.getValueType());
+
     OutputValues.push_back(V);
   }
 

>From 61e8f04274a0b12017bf6c9bb667c31825247409 Mon Sep 17 00:00:00 2001
From: Bill Wendling <isanbard at gmail.com>
Date: Thu, 5 Nov 2026 18:48:31 -0800
Subject: [PATCH 3/5] Don't use VT's register class for an Untyped callbr
 output

An i128 output in an explicit register pair on SystemZ is Untyped,
which has no register class, so getRegClassFor() asserted when the
indirect target read the pair. Use VT's class only when VT has one.

Assisted-by: Claude Opus 5.5
---
 .../SelectionDAG/SelectionDAGBuilder.cpp      | 10 +++---
 .../CodeGen/SystemZ/callbr-asm-outputs.ll     | 33 +++++++++++++++++++
 2 files changed, 39 insertions(+), 4 deletions(-)
 create mode 100644 llvm/test/CodeGen/SystemZ/callbr-asm-outputs.ll

diff --git a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
index ac9e4f1192b4b8..b4b413304a0e2e 100644
--- a/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
@@ -13359,14 +13359,16 @@ static void readPhysRegsAtBlockEntry(SDValue &V, MachineBasicBlock &MBB,
 
     // Copy a register that can't be copied directly, such as EFLAGS, through
     // the class that the target copies it to. The use of the copy then
-    // restores it, which the target lowers.
+    // restores it, which the target lowers. Otherwise, use VT's class, which
+    // leaves the register allocator the most choice, if VT has one and it
+    // contains PhysReg. An Untyped register pair, such as an i128 on SystemZ,
+    // has none, so it keeps the pair's own class.
     const TargetRegisterClass *RC = TRI->getMinimalPhysRegClass(PhysReg);
     bool CrossCopy = RC->expensiveOrImpossibleToCopy();
     if (CrossCopy)
       RC = TRI->getCrossCopyRegClass(RC);
-    else if (const TargetRegisterClass *VTRC = TLI.getRegClassFor(VT);
-             VTRC->contains(PhysReg))
-      RC = VTRC;
+    else if (TLI.isTypeLegal(VT) && TLI.getRegClassFor(VT)->contains(PhysReg))
+      RC = TLI.getRegClassFor(VT);
 
     // This marks PhysReg live-in, once, and reuses the copy if another output
     // reads the same register, such as a second flag output.
diff --git a/llvm/test/CodeGen/SystemZ/callbr-asm-outputs.ll b/llvm/test/CodeGen/SystemZ/callbr-asm-outputs.ll
new file mode 100644
index 00000000000000..95ca5965f901e6
--- /dev/null
+++ b/llvm/test/CodeGen/SystemZ/callbr-asm-outputs.ll
@@ -0,0 +1,33 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
+; RUN: llc -mtriple=s390x-linux-gnu -verify-machineinstrs < %s | FileCheck %s
+
+; An i128 output in an explicit register pair. Its value type is Untyped, which
+; has no register class, so the indirect target has to read the pair in the
+; pair's own class. Both paths store both halves.
+define void @test_reg_pair_output(ptr %p, ptr %q) {
+; CHECK-LABEL: test_reg_pair_output:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:  # %bb.1: # %ft
+; CHECK-NEXT:    stg %r0, 0(%r2)
+; CHECK-NEXT:    stg %r1, 8(%r2)
+; CHECK-NEXT:    br %r14
+; CHECK-NEXT:  .LBB0_2: # Inline asm indirect target
+; CHECK-NEXT:    # %ind
+; CHECK-NEXT:    # Label of block must be emitted
+; CHECK-NEXT:    stg %r1, 8(%r3)
+; CHECK-NEXT:    stg %r0, 0(%r3)
+; CHECK-NEXT:    br %r14
+entry:
+  %r = callbr i128 asm "", "={r0},!i"()
+          to label %ft [label %ind]
+
+ft:
+  store i128 %r, ptr %p
+  ret void
+
+ind:
+  store i128 %r, ptr %q
+  ret void
+}

>From 6c5f3f5de48999f980069c67f35a4f5957b3ea43 Mon Sep 17 00:00:00 2001
From: Bill Wendling <isanbard at gmail.com>
Date: Thu, 5 Nov 2026 19:05:41 -0800
Subject: [PATCH 4/5] Remove -verify-machineinstrs flag

---
 llvm/test/CodeGen/SystemZ/callbr-asm-flag-outputs.ll | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/llvm/test/CodeGen/SystemZ/callbr-asm-flag-outputs.ll b/llvm/test/CodeGen/SystemZ/callbr-asm-flag-outputs.ll
index 708f92a7e54fd1..939c9a63e04348 100644
--- a/llvm/test/CodeGen/SystemZ/callbr-asm-flag-outputs.ll
+++ b/llvm/test/CodeGen/SystemZ/callbr-asm-flag-outputs.ll
@@ -1,5 +1,5 @@
 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
-; RUN: llc -mtriple=s390x-linux-gnu -verify-machineinstrs < %s | FileCheck %s
+; RUN: llc -mtriple=s390x-linux-gnu < %s | FileCheck %s
 
 ; A condition code output that both paths out of an asm goto read. CC is live
 ; into the indirect target, which reads it again: the read after the asm

>From 761afa99bf17f1fc5179dc6ad10f158bc156b34d Mon Sep 17 00:00:00 2001
From: Bill Wendling <isanbard at gmail.com>
Date: Thu, 5 Nov 2026 19:05:58 -0800
Subject: [PATCH 5/5] Apply suggestion from @arsenm

Co-authored-by: Matt Arsenault <Matthew.Arsenault at amd.com>
---
 llvm/test/CodeGen/X86/callbr-asm-flag-outputs.ll | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/llvm/test/CodeGen/X86/callbr-asm-flag-outputs.ll b/llvm/test/CodeGen/X86/callbr-asm-flag-outputs.ll
index cb873e02e07a75..b83eb201a2aceb 100644
--- a/llvm/test/CodeGen/X86/callbr-asm-flag-outputs.ll
+++ b/llvm/test/CodeGen/X86/callbr-asm-flag-outputs.ll
@@ -1,6 +1,6 @@
 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
-; RUN: llc -mtriple=x86_64-linux-gnu -O0 -verify-machineinstrs < %s | FileCheck %s --check-prefix=O0
-; RUN: llc -mtriple=x86_64-linux-gnu -O2 -verify-machineinstrs < %s | FileCheck %s --check-prefix=O2
+; RUN: llc -mtriple=x86_64-linux-gnu -O0 < %s | FileCheck %s --check-prefix=O0
+; RUN: llc -mtriple=x86_64-linux-gnu < %s | FileCheck %s --check-prefix=O2
 
 ; An asm goto's indirect target reads the asm's flag outputs from the EFLAGS
 ; that live into it from the asm. Nothing after the asm in its block runs on



More information about the llvm-branch-commits mailing list