<div dir="ltr">Probably caused by this CL: <span style="line-height:1.5"><a href="http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/11271" class="cremed">http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/11271</a></span><div><span style="color:rgb(0,0,0);font-family:'Courier New',courier,monotype,monospace;font-size:medium;line-height:normal"><br></span></div><div><span style="color:rgb(0,0,0);font-family:'Courier New',courier,monotype,monospace;font-size:medium;line-height:normal"><a href="http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/11271/steps/check-llvm%20msan/logs/stdio">http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/11271/steps/check-llvm%20msan/logs/stdio</a></span><br></div><div><font color="#000000" face="Courier New, courier, monotype, monospace" size="3"><span style="line-height:normal"><a href="http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/11271/steps/check-llvm%20asan/logs/stdio">http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/11271/steps/check-llvm%20asan/logs/stdio</a></span></font><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Mar 21, 2016 at 1:33 PM Nicolai Haehnle via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Author: nha<br>
Date: Mon Mar 21 15:28:33 2016<br>
New Revision: 263982<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=263982&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project?rev=263982&view=rev</a><br>
Log:<br>
AMDGPU: Add SIWholeQuadMode pass<br>
<br>
Summary:<br>
Whole quad mode is already enabled for pixel shaders that compute<br>
derivatives, but it must be suspended for instructions that cause a<br>
shader to have side effects (i.e. stores and atomics).<br>
<br>
This pass addresses the issue by storing the real (initial) live mask<br>
in a register, masking EXEC before instructions that require exact<br>
execution and (re-)enabling WQM where required.<br>
<br>
This pass is run before register coalescing so that we can use<br>
machine SSA for analysis.<br>
<br>
The changes in this patch expose a problem with the second machine<br>
scheduling pass: target independent instructions like COPY implicitly<br>
use EXEC when they operate on VGPRs, but this fact is not encoded in<br>
the MIR. This can lead to miscompilation because instructions are<br>
moved past changes to EXEC.<br>
<br>
This patch fixes the problem by adding use-implicit operands to<br>
target independent instructions. Some general codegen passes are<br>
relaxed to work with such implicit use operands.<br>
<br>
Reviewers: arsenm, tstellarAMD, mareko<br>
<br>
Subscribers: MatzeB, arsenm, llvm-commits<br>
<br>
Differential Revision: <a href="http://reviews.llvm.org/D18162" rel="noreferrer" target="_blank">http://reviews.llvm.org/D18162</a><br>
<br>
Added:<br>
llvm/trunk/lib/Target/AMDGPU/SIWholeQuadMode.cpp<br>
llvm/trunk/test/CodeGen/AMDGPU/wqm.ll<br>
Modified:<br>
llvm/trunk/lib/Target/AMDGPU/AMDGPU.h<br>
llvm/trunk/lib/Target/AMDGPU/AMDGPUInstrInfo.h<br>
llvm/trunk/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp<br>
llvm/trunk/lib/Target/AMDGPU/CMakeLists.txt<br>
llvm/trunk/lib/Target/AMDGPU/SIInstrInfo.cpp<br>
llvm/trunk/lib/Target/AMDGPU/SIInstrInfo.h<br>
llvm/trunk/lib/Target/AMDGPU/SILowerControlFlow.cpp<br>
llvm/trunk/lib/Target/AMDGPU/SIRegisterInfo.h<br>
<br>
Modified: llvm/trunk/lib/Target/AMDGPU/AMDGPU.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/AMDGPU.h?rev=263982&r1=263981&r2=263982&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/AMDGPU.h?rev=263982&r1=263981&r2=263982&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Target/AMDGPU/AMDGPU.h (original)<br>
+++ llvm/trunk/lib/Target/AMDGPU/AMDGPU.h Mon Mar 21 15:28:33 2016<br>
@@ -44,6 +44,7 @@ FunctionPass *createSIFoldOperandsPass()<br>
FunctionPass *createSILowerI1CopiesPass();<br>
FunctionPass *createSIShrinkInstructionsPass();<br>
FunctionPass *createSILoadStoreOptimizerPass(TargetMachine &tm);<br>
+FunctionPass *createSIWholeQuadModePass();<br>
FunctionPass *createSILowerControlFlowPass();<br>
FunctionPass *createSIFixControlFlowLiveIntervalsPass();<br>
FunctionPass *createSIFixSGPRCopiesPass();<br>
@@ -70,6 +71,9 @@ extern char &SILowerI1CopiesID;<br>
void initializeSILoadStoreOptimizerPass(PassRegistry &);<br>
extern char &SILoadStoreOptimizerID;<br>
<br>
+void initializeSIWholeQuadModePass(PassRegistry &);<br>
+extern char &SIWholeQuadModeID;<br>
+<br>
void initializeSILowerControlFlowPass(PassRegistry &);<br>
extern char &SILowerControlFlowPassID;<br>
<br>
<br>
Modified: llvm/trunk/lib/Target/AMDGPU/AMDGPUInstrInfo.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/AMDGPUInstrInfo.h?rev=263982&r1=263981&r2=263982&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/AMDGPUInstrInfo.h?rev=263982&r1=263981&r2=263982&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Target/AMDGPU/AMDGPUInstrInfo.h (original)<br>
+++ llvm/trunk/lib/Target/AMDGPU/AMDGPUInstrInfo.h Mon Mar 21 15:28:33 2016<br>
@@ -62,7 +62,6 @@ public:<br>
int64_t Offset1, int64_t Offset2,<br>
unsigned NumLoads) const override;<br>
<br>
-<br>
/// \brief Return a target-specific opcode if Opcode is a pseudo instruction.<br>
/// Return -1 if the target-specific opcode for the pseudo instruction does<br>
/// not exist. If Opcode is not a pseudo instruction, this is identity.<br>
<br>
Modified: llvm/trunk/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp?rev=263982&r1=263981&r2=263982&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp?rev=263982&r1=263981&r2=263982&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp (original)<br>
+++ llvm/trunk/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp Mon Mar 21 15:28:33 2016<br>
@@ -57,6 +57,7 @@ extern "C" void LLVMInitializeAMDGPUTarg<br>
initializeSIAnnotateControlFlowPass(*PR);<br>
initializeSIInsertNopsPass(*PR);<br>
initializeSIInsertWaitsPass(*PR);<br>
+ initializeSIWholeQuadModePass(*PR);<br>
initializeSILowerControlFlowPass(*PR);<br>
}<br>
<br>
@@ -346,6 +347,7 @@ void GCNPassConfig::addPreRegAlloc() {<br>
insertPass(&MachineSchedulerID, &RegisterCoalescerID);<br>
}<br>
addPass(createSIShrinkInstructionsPass(), false);<br>
+ addPass(createSIWholeQuadModePass());<br>
}<br>
<br>
void GCNPassConfig::addFastRegAlloc(FunctionPass *RegAllocPass) {<br>
<br>
Modified: llvm/trunk/lib/Target/AMDGPU/CMakeLists.txt<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/CMakeLists.txt?rev=263982&r1=263981&r2=263982&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/CMakeLists.txt?rev=263982&r1=263981&r2=263982&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Target/AMDGPU/CMakeLists.txt (original)<br>
+++ llvm/trunk/lib/Target/AMDGPU/CMakeLists.txt Mon Mar 21 15:28:33 2016<br>
@@ -63,6 +63,7 @@ add_llvm_target(AMDGPUCodeGen<br>
SIRegisterInfo.cpp<br>
SIShrinkInstructions.cpp<br>
SITypeRewriter.cpp<br>
+ SIWholeQuadMode.cpp<br>
)<br>
<br>
add_subdirectory(AsmParser)<br>
<br>
Modified: llvm/trunk/lib/Target/AMDGPU/SIInstrInfo.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/SIInstrInfo.cpp?rev=263982&r1=263981&r2=263982&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/SIInstrInfo.cpp?rev=263982&r1=263981&r2=263982&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Target/AMDGPU/SIInstrInfo.cpp (original)<br>
+++ llvm/trunk/lib/Target/AMDGPU/SIInstrInfo.cpp Mon Mar 21 15:28:33 2016<br>
@@ -1248,6 +1248,19 @@ MachineInstr *SIInstrInfo::convertToThre<br>
.addImm(0); // omod<br>
}<br>
<br>
+bool SIInstrInfo::isSchedulingBoundary(const MachineInstr *MI,<br>
+ const MachineBasicBlock *MBB,<br>
+ const MachineFunction &MF) const {<br>
+ // Target-independent instructions do not have an implicit-use of EXEC, even<br>
+ // when they operate on VGPRs. Treating EXEC modifications as scheduling<br>
+ // boundaries prevents incorrect movements of such instructions.<br>
+ const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo();<br>
+ if (MI->modifiesRegister(AMDGPU::EXEC, TRI))<br>
+ return true;<br>
+<br>
+ return AMDGPUInstrInfo::isSchedulingBoundary(MI, MBB, MF);<br>
+}<br>
+<br>
bool SIInstrInfo::isInlineConstant(const APInt &Imm) const {<br>
int64_t SVal = Imm.getSExtValue();<br>
if (SVal >= -16 && SVal <= 64)<br>
<br>
Modified: llvm/trunk/lib/Target/AMDGPU/SIInstrInfo.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/SIInstrInfo.h?rev=263982&r1=263981&r2=263982&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/SIInstrInfo.h?rev=263982&r1=263981&r2=263982&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Target/AMDGPU/SIInstrInfo.h (original)<br>
+++ llvm/trunk/lib/Target/AMDGPU/SIInstrInfo.h Mon Mar 21 15:28:33 2016<br>
@@ -149,6 +149,10 @@ public:<br>
MachineBasicBlock::iterator &MI,<br>
LiveVariables *LV) const override;<br>
<br>
+ bool isSchedulingBoundary(const MachineInstr *MI,<br>
+ const MachineBasicBlock *MBB,<br>
+ const MachineFunction &MF) const override;<br>
+<br>
static bool isSALU(const MachineInstr &MI) {<br>
return MI.getDesc().TSFlags & SIInstrFlags::SALU;<br>
}<br>
<br>
Modified: llvm/trunk/lib/Target/AMDGPU/SILowerControlFlow.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/SILowerControlFlow.cpp?rev=263982&r1=263981&r2=263982&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/SILowerControlFlow.cpp?rev=263982&r1=263981&r2=263982&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Target/AMDGPU/SILowerControlFlow.cpp (original)<br>
+++ llvm/trunk/lib/Target/AMDGPU/SILowerControlFlow.cpp Mon Mar 21 15:28:33 2016<br>
@@ -78,7 +78,7 @@ private:<br>
void SkipIfDead(MachineInstr &MI);<br>
<br>
void If(MachineInstr &MI);<br>
- void Else(MachineInstr &MI);<br>
+ void Else(MachineInstr &MI, bool ExecModified);<br>
void Break(MachineInstr &MI);<br>
void IfBreak(MachineInstr &MI);<br>
void ElseBreak(MachineInstr &MI);<br>
@@ -215,7 +215,7 @@ void SILowerControlFlow::If(MachineInstr<br>
MI.eraseFromParent();<br>
}<br>
<br>
-void SILowerControlFlow::Else(MachineInstr &MI) {<br>
+void SILowerControlFlow::Else(MachineInstr &MI, bool ExecModified) {<br>
MachineBasicBlock &MBB = *MI.getParent();<br>
DebugLoc DL = MI.getDebugLoc();<br>
unsigned Dst = MI.getOperand(0).getReg();<br>
@@ -225,6 +225,15 @@ void SILowerControlFlow::Else(MachineIns<br>
TII->get(AMDGPU::S_OR_SAVEEXEC_B64), Dst)<br>
.addReg(Src); // Saved EXEC<br>
<br>
+ if (ExecModified) {<br>
+ // Adjust the saved exec to account for the modifications during the flow<br>
+ // block that contains the ELSE. This can happen when WQM mode is switched<br>
+ // off.<br>
+ BuildMI(MBB, &MI, DL, TII->get(AMDGPU::S_AND_B64), Dst)<br>
+ .addReg(AMDGPU::EXEC)<br>
+ .addReg(Dst);<br>
+ }<br>
+<br>
BuildMI(MBB, &MI, DL, TII->get(AMDGPU::S_XOR_B64), AMDGPU::EXEC)<br>
.addReg(AMDGPU::EXEC)<br>
.addReg(Dst);<br>
@@ -488,7 +497,6 @@ bool SILowerControlFlow::runOnMachineFun<br>
SIMachineFunctionInfo *MFI = MF.getInfo<SIMachineFunctionInfo>();<br>
<br>
bool HaveKill = false;<br>
- bool NeedWQM = false;<br>
bool NeedFlat = false;<br>
unsigned Depth = 0;<br>
<br>
@@ -498,17 +506,24 @@ bool SILowerControlFlow::runOnMachineFun<br>
MachineBasicBlock *EmptyMBBAtEnd = NULL;<br>
MachineBasicBlock &MBB = *BI;<br>
MachineBasicBlock::iterator I, Next;<br>
+ bool ExecModified = false;<br>
+<br>
for (I = MBB.begin(); I != MBB.end(); I = Next) {<br>
Next = std::next(I);<br>
<br>
MachineInstr &MI = *I;<br>
- if (TII->isWQM(MI) || TII->isDS(MI))<br>
- NeedWQM = true;<br>
<br>
// Flat uses m0 in case it needs to access LDS.<br>
if (TII->isFLAT(MI))<br>
NeedFlat = true;<br>
<br>
+ for (const auto &Def : I->defs()) {<br>
+ if (Def.isReg() && Def.isDef() && Def.getReg() == AMDGPU::EXEC) {<br>
+ ExecModified = true;<br>
+ break;<br>
+ }<br>
+ }<br>
+<br>
switch (MI.getOpcode()) {<br>
default: break;<br>
case AMDGPU::SI_IF:<br>
@@ -517,7 +532,7 @@ bool SILowerControlFlow::runOnMachineFun<br>
break;<br>
<br>
case AMDGPU::SI_ELSE:<br>
- Else(MI);<br>
+ Else(MI, ExecModified);<br>
break;<br>
<br>
case AMDGPU::SI_BREAK:<br>
@@ -599,12 +614,6 @@ bool SILowerControlFlow::runOnMachineFun<br>
}<br>
}<br>
<br>
- if (NeedWQM && MFI->getShaderType() == ShaderType::PIXEL) {<br>
- MachineBasicBlock &MBB = MF.front();<br>
- BuildMI(MBB, MBB.getFirstNonPHI(), DebugLoc(), TII->get(AMDGPU::S_WQM_B64),<br>
- AMDGPU::EXEC).addReg(AMDGPU::EXEC);<br>
- }<br>
-<br>
if (NeedFlat && MFI->IsKernel) {<br>
// TODO: What to use with function calls?<br>
// We will need to Initialize the flat scratch register pair.<br>
<br>
Modified: llvm/trunk/lib/Target/AMDGPU/SIRegisterInfo.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/SIRegisterInfo.h?rev=263982&r1=263981&r2=263982&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/SIRegisterInfo.h?rev=263982&r1=263981&r2=263982&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Target/AMDGPU/SIRegisterInfo.h (original)<br>
+++ llvm/trunk/lib/Target/AMDGPU/SIRegisterInfo.h Mon Mar 21 15:28:33 2016<br>
@@ -72,9 +72,12 @@ public:<br>
}<br>
<br>
bool isSGPRReg(const MachineRegisterInfo &MRI, unsigned Reg) const {<br>
+ const TargetRegisterClass *RC;<br>
if (TargetRegisterInfo::isVirtualRegister(Reg))<br>
- return isSGPRClass(MRI.getRegClass(Reg));<br>
- return getPhysRegClass(Reg);<br>
+ RC = MRI.getRegClass(Reg);<br>
+ else<br>
+ RC = getPhysRegClass(Reg);<br>
+ return isSGPRClass(RC);<br>
}<br>
<br>
/// \returns true if this class contains VGPR registers.<br>
<br>
Added: llvm/trunk/lib/Target/AMDGPU/SIWholeQuadMode.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/SIWholeQuadMode.cpp?rev=263982&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/AMDGPU/SIWholeQuadMode.cpp?rev=263982&view=auto</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Target/AMDGPU/SIWholeQuadMode.cpp (added)<br>
+++ llvm/trunk/lib/Target/AMDGPU/SIWholeQuadMode.cpp Mon Mar 21 15:28:33 2016<br>
@@ -0,0 +1,465 @@<br>
+//===-- SIWholeQuadMode.cpp - enter and suspend whole quad mode -----------===//<br>
+//<br>
+// The LLVM Compiler Infrastructure<br>
+//<br>
+// This file is distributed under the University of Illinois Open Source<br>
+// License. See LICENSE.TXT for details.<br>
+//<br>
+//===----------------------------------------------------------------------===//<br>
+//<br>
+/// \file<br>
+/// \brief This pass adds instructions to enable whole quad mode for pixel<br>
+/// shaders.<br>
+///<br>
+/// Whole quad mode is required for derivative computations, but it interferes<br>
+/// with shader side effects (stores and atomics). This pass is run on the<br>
+/// scheduled machine IR but before register coalescing, so that machine SSA is<br>
+/// available for analysis. It ensures that WQM is enabled when necessary, but<br>
+/// disabled around stores and atomics.<br>
+///<br>
+/// When necessary, this pass creates a function prolog<br>
+///<br>
+/// S_MOV_B64 LiveMask, EXEC<br>
+/// S_WQM_B64 EXEC, EXEC<br>
+///<br>
+/// to enter WQM at the top of the function and surrounds blocks of Exact<br>
+/// instructions by<br>
+///<br>
+/// S_AND_SAVEEXEC_B64 Tmp, LiveMask<br>
+/// ...<br>
+/// S_MOV_B64 EXEC, Tmp<br>
+///<br>
+/// In order to avoid excessive switching during sequences of Exact<br>
+/// instructions, the pass first analyzes which instructions must be run in WQM<br>
+/// (aka which instructions produce values that lead to derivative<br>
+/// computations).<br>
+///<br>
+/// Basic blocks are always exited in WQM as long as some successor needs WQM.<br>
+///<br>
+/// There is room for improvement given better control flow analysis:<br>
+///<br>
+/// (1) at the top level (outside of control flow statements, and as long as<br>
+/// kill hasn't been used), one SGPR can be saved by recovering WQM from<br>
+/// the LiveMask (this is implemented for the entry block).<br>
+///<br>
+/// (2) when entire regions (e.g. if-else blocks or entire loops) only<br>
+/// consist of exact and don't-care instructions, the switch only has to<br>
+/// be done at the entry and exit points rather than potentially in each<br>
+/// block of the region.<br>
+///<br>
+//===----------------------------------------------------------------------===//<br>
+<br>
+#include "AMDGPU.h"<br>
+#include "AMDGPUSubtarget.h"<br>
+#include "SIInstrInfo.h"<br>
+#include "SIMachineFunctionInfo.h"<br>
+#include "llvm/CodeGen/MachineDominanceFrontier.h"<br>
+#include "llvm/CodeGen/MachineDominators.h"<br>
+#include "llvm/CodeGen/MachineFunction.h"<br>
+#include "llvm/CodeGen/MachineFunctionPass.h"<br>
+#include "llvm/CodeGen/MachineInstrBuilder.h"<br>
+#include "llvm/CodeGen/MachineRegisterInfo.h"<br>
+#include "llvm/IR/Constants.h"<br>
+<br>
+using namespace llvm;<br>
+<br>
+#define DEBUG_TYPE "si-wqm"<br>
+<br>
+namespace {<br>
+<br>
+enum {<br>
+ StateWQM = 0x1,<br>
+ StateExact = 0x2,<br>
+};<br>
+<br>
+struct InstrInfo {<br>
+ char Needs = 0;<br>
+ char OutNeeds = 0;<br>
+};<br>
+<br>
+struct BlockInfo {<br>
+ char Needs = 0;<br>
+ char InNeeds = 0;<br>
+ char OutNeeds = 0;<br>
+};<br>
+<br>
+struct WorkItem {<br>
+ const MachineBasicBlock *MBB = nullptr;<br>
+ const MachineInstr *MI = nullptr;<br>
+<br>
+ WorkItem() {}<br>
+ WorkItem(const MachineBasicBlock *MBB) : MBB(MBB) {}<br>
+ WorkItem(const MachineInstr *MI) : MI(MI) {}<br>
+};<br>
+<br>
+class SIWholeQuadMode : public MachineFunctionPass {<br>
+private:<br>
+ const SIInstrInfo *TII;<br>
+ const SIRegisterInfo *TRI;<br>
+ MachineRegisterInfo *MRI;<br>
+<br>
+ DenseMap<const MachineInstr *, InstrInfo> Instructions;<br>
+ DenseMap<const MachineBasicBlock *, BlockInfo> Blocks;<br>
+ SmallVector<const MachineInstr *, 2> ExecExports;<br>
+<br>
+ char scanInstructions(const MachineFunction &MF, std::vector<WorkItem>& Worklist);<br>
+ void propagateInstruction(const MachineInstr &MI, std::vector<WorkItem>& Worklist);<br>
+ void propagateBlock(const MachineBasicBlock &MBB, std::vector<WorkItem>& Worklist);<br>
+ char analyzeFunction(const MachineFunction &MF);<br>
+<br>
+ void toExact(MachineBasicBlock &MBB, MachineBasicBlock::iterator Before,<br>
+ unsigned SaveWQM, unsigned LiveMaskReg);<br>
+ void toWQM(MachineBasicBlock &MBB, MachineBasicBlock::iterator Before,<br>
+ unsigned SavedWQM);<br>
+ void processBlock(MachineBasicBlock &MBB, unsigned LiveMaskReg, bool isEntry);<br>
+<br>
+public:<br>
+ static char ID;<br>
+<br>
+ SIWholeQuadMode() :<br>
+ MachineFunctionPass(ID) { }<br>
+<br>
+ bool runOnMachineFunction(MachineFunction &MF) override;<br>
+<br>
+ const char *getPassName() const override {<br>
+ return "SI Whole Quad Mode";<br>
+ }<br>
+<br>
+ void getAnalysisUsage(AnalysisUsage &AU) const override {<br>
+ AU.setPreservesCFG();<br>
+ MachineFunctionPass::getAnalysisUsage(AU);<br>
+ }<br>
+};<br>
+<br>
+} // End anonymous namespace<br>
+<br>
+char SIWholeQuadMode::ID = 0;<br>
+<br>
+INITIALIZE_PASS_BEGIN(SIWholeQuadMode, DEBUG_TYPE,<br>
+ "SI Whole Quad Mode", false, false)<br>
+INITIALIZE_PASS_END(SIWholeQuadMode, DEBUG_TYPE,<br>
+ "SI Whole Quad Mode", false, false)<br>
+<br>
+char &llvm::SIWholeQuadModeID = SIWholeQuadMode::ID;<br>
+<br>
+FunctionPass *llvm::createSIWholeQuadModePass() {<br>
+ return new SIWholeQuadMode;<br>
+}<br>
+<br>
+// Scan instructions to determine which ones require an Exact execmask and<br>
+// which ones seed WQM requirements.<br>
+char SIWholeQuadMode::scanInstructions(const MachineFunction &MF,<br>
+ std::vector<WorkItem> &Worklist) {<br>
+ char GlobalFlags = 0;<br>
+<br>
+ for (auto BI = MF.begin(), BE = MF.end(); BI != BE; ++BI) {<br>
+ const MachineBasicBlock &MBB = *BI;<br>
+<br>
+ for (auto II = MBB.begin(), IE = MBB.end(); II != IE; ++II) {<br>
+ const MachineInstr &MI = *II;<br>
+ unsigned Opcode = MI.getOpcode();<br>
+ char Flags;<br>
+<br>
+ if (TII->isWQM(Opcode) || TII->isDS(Opcode)) {<br>
+ Flags = StateWQM;<br>
+ } else if (TII->get(Opcode).mayStore() &&<br>
+ (MI.getDesc().TSFlags & SIInstrFlags::VM_CNT)) {<br>
+ Flags = StateExact;<br>
+ } else {<br>
+ // Handle export instructions with the exec mask valid flag set<br>
+ if (Opcode == AMDGPU::EXP && MI.getOperand(4).getImm() != 0)<br>
+ ExecExports.push_back(&MI);<br>
+ continue;<br>
+ }<br>
+<br>
+ Instructions[&MI].Needs = Flags;<br>
+ Worklist.push_back(&MI);<br>
+ GlobalFlags |= Flags;<br>
+ }<br>
+ }<br>
+<br>
+ return GlobalFlags;<br>
+}<br>
+<br>
+void SIWholeQuadMode::propagateInstruction(const MachineInstr &MI,<br>
+ std::vector<WorkItem>& Worklist) {<br>
+ const MachineBasicBlock &MBB = *MI.getParent();<br>
+ InstrInfo &II = Instructions[&MI];<br>
+ BlockInfo &BI = Blocks[&MBB];<br>
+<br>
+ // Control flow-type instructions that are followed by WQM computations<br>
+ // must themselves be in WQM.<br>
+ if ((II.OutNeeds & StateWQM) && !(II.Needs & StateWQM) &&<br>
+ (MI.isBranch() || MI.isTerminator() || MI.getOpcode() == AMDGPU::SI_KILL))<br>
+ II.Needs = StateWQM;<br>
+<br>
+ // Propagate to block level<br>
+ BI.Needs |= II.Needs;<br>
+ if ((BI.InNeeds | II.Needs) != BI.InNeeds) {<br>
+ BI.InNeeds |= II.Needs;<br>
+ Worklist.push_back(&MBB);<br>
+ }<br>
+<br>
+ // Propagate backwards within block<br>
+ if (const MachineInstr *PrevMI = MI.getPrevNode()) {<br>
+ char InNeeds = II.Needs | II.OutNeeds;<br>
+ if (!PrevMI->isPHI()) {<br>
+ InstrInfo &PrevII = Instructions[PrevMI];<br>
+ if ((PrevII.OutNeeds | InNeeds) != PrevII.OutNeeds) {<br>
+ PrevII.OutNeeds |= InNeeds;<br>
+ Worklist.push_back(PrevMI);<br>
+ }<br>
+ }<br>
+ }<br>
+<br>
+ // Propagate WQM flag to instruction inputs<br>
+ assert(II.Needs != (StateWQM | StateExact));<br>
+ if (II.Needs != StateWQM)<br>
+ return;<br>
+<br>
+ for (const MachineOperand &Use : MI.uses()) {<br>
+ if (!Use.isReg() || !Use.isUse())<br>
+ continue;<br>
+<br>
+ // At this point, physical registers appear as inputs or outputs<br>
+ // and following them makes no sense (and would in fact be incorrect<br>
+ // when the same VGPR is used as both an output and an input that leads<br>
+ // to a NeedsWQM instruction).<br>
+ //<br>
+ // Note: VCC appears e.g. in 64-bit addition with carry - theoretically we<br>
+ // have to trace this, in practice it happens for 64-bit computations like<br>
+ // pointers where both dwords are followed already anyway.<br>
+ if (!TargetRegisterInfo::isVirtualRegister(Use.getReg()))<br>
+ continue;<br>
+<br>
+ for (const MachineOperand &Def : MRI->def_operands(Use.getReg())) {<br>
+ const MachineInstr *DefMI = Def.getParent();<br>
+ InstrInfo &DefII = Instructions[DefMI];<br>
+<br>
+ // Obviously skip if DefMI is already flagged as NeedWQM.<br>
+ //<br>
+ // The instruction might also be flagged as NeedExact. This happens when<br>
+ // the result of an atomic is used in a WQM computation. In this case,<br>
+ // the atomic must not run for helper pixels and the WQM result is<br>
+ // undefined.<br>
+ if (DefII.Needs != 0)<br>
+ continue;<br>
+<br>
+ DefII.Needs = StateWQM;<br>
+ Worklist.push_back(DefMI);<br>
+ }<br>
+ }<br>
+}<br>
+<br>
+void SIWholeQuadMode::propagateBlock(const MachineBasicBlock &MBB,<br>
+ std::vector<WorkItem>& Worklist) {<br>
+ BlockInfo &BI = Blocks[&MBB];<br>
+<br>
+ // Propagate through instructions<br>
+ if (!MBB.empty()) {<br>
+ const MachineInstr *LastMI = &*MBB.rbegin();<br>
+ InstrInfo &LastII = Instructions[LastMI];<br>
+ if ((LastII.OutNeeds | BI.OutNeeds) != LastII.OutNeeds) {<br>
+ LastII.OutNeeds |= BI.OutNeeds;<br>
+ Worklist.push_back(LastMI);<br>
+ }<br>
+ }<br>
+<br>
+ // Predecessor blocks must provide for our WQM/Exact needs.<br>
+ for (const MachineBasicBlock *Pred : MBB.predecessors()) {<br>
+ BlockInfo &PredBI = Blocks[Pred];<br>
+ if ((PredBI.OutNeeds | BI.InNeeds) == PredBI.OutNeeds)<br>
+ continue;<br>
+<br>
+ PredBI.OutNeeds |= BI.InNeeds;<br>
+ PredBI.InNeeds |= BI.InNeeds;<br>
+ Worklist.push_back(Pred);<br>
+ }<br>
+<br>
+ // All successors must be prepared to accept the same set of WQM/Exact<br>
+ // data.<br>
+ for (const MachineBasicBlock *Succ : MBB.successors()) {<br>
+ BlockInfo &SuccBI = Blocks[Succ];<br>
+ if ((SuccBI.InNeeds | BI.OutNeeds) == SuccBI.InNeeds)<br>
+ continue;<br>
+<br>
+ SuccBI.InNeeds |= BI.OutNeeds;<br>
+ Worklist.push_back(Succ);<br>
+ }<br>
+}<br>
+<br>
+char SIWholeQuadMode::analyzeFunction(const MachineFunction &MF) {<br>
+ std::vector<WorkItem> Worklist;<br>
+ char GlobalFlags = scanInstructions(MF, Worklist);<br>
+<br>
+ while (!Worklist.empty()) {<br>
+ WorkItem WI = Worklist.back();<br>
+ Worklist.pop_back();<br>
+<br>
+ if (WI.MI)<br>
+ propagateInstruction(*WI.MI, Worklist);<br>
+ else<br>
+ propagateBlock(*WI.MBB, Worklist);<br>
+ }<br>
+<br>
+ return GlobalFlags;<br>
+}<br>
+<br>
+void SIWholeQuadMode::toExact(MachineBasicBlock &MBB,<br>
+ MachineBasicBlock::iterator Before,<br>
+ unsigned SaveWQM, unsigned LiveMaskReg)<br>
+{<br>
+ if (SaveWQM) {<br>
+ BuildMI(MBB, Before, DebugLoc(), TII->get(AMDGPU::S_AND_SAVEEXEC_B64),<br>
+ SaveWQM)<br>
+ .addReg(LiveMaskReg);<br>
+ } else {<br>
+ BuildMI(MBB, Before, DebugLoc(), TII->get(AMDGPU::S_AND_B64),<br>
+ AMDGPU::EXEC)<br>
+ .addReg(AMDGPU::EXEC)<br>
+ .addReg(LiveMaskReg);<br>
+ }<br>
+}<br>
+<br>
+void SIWholeQuadMode::toWQM(MachineBasicBlock &MBB,<br>
+ MachineBasicBlock::iterator Before,<br>
+ unsigned SavedWQM)<br>
+{<br>
+ if (SavedWQM) {<br>
+ BuildMI(MBB, Before, DebugLoc(), TII->get(AMDGPU::COPY), AMDGPU::EXEC)<br>
+ .addReg(SavedWQM);<br>
+ } else {<br>
+ BuildMI(MBB, Before, DebugLoc(), TII->get(AMDGPU::S_WQM_B64),<br>
+ AMDGPU::EXEC)<br>
+ .addReg(AMDGPU::EXEC);<br>
+ }<br>
+}<br>
+<br>
+void SIWholeQuadMode::processBlock(MachineBasicBlock &MBB, unsigned LiveMaskReg,<br>
+ bool isEntry) {<br>
+ auto BII = Blocks.find(&MBB);<br>
+ if (BII == Blocks.end())<br>
+ return;<br>
+<br>
+ const BlockInfo &BI = BII->second;<br>
+<br>
+ if (!(BI.InNeeds & StateWQM))<br>
+ return;<br>
+<br>
+ // This is a non-entry block that is WQM throughout, so no need to do<br>
+ // anything.<br>
+ if (!isEntry && !(BI.Needs & StateExact) && BI.OutNeeds != StateExact)<br>
+ return;<br>
+<br>
+ unsigned SavedWQMReg = 0;<br>
+ bool WQMFromExec = isEntry;<br>
+ char State = isEntry ? StateExact : StateWQM;<br>
+<br>
+ auto II = MBB.getFirstNonPHI(), IE = MBB.end();<br>
+ while (II != IE) {<br>
+ MachineInstr &MI = *II;<br>
+ ++II;<br>
+<br>
+ // Skip instructions that are not affected by EXEC<br>
+ if (MI.getDesc().TSFlags & (SIInstrFlags::SALU | SIInstrFlags::SMRD) &&<br>
+ !MI.isBranch() && !MI.isTerminator())<br>
+ continue;<br>
+<br>
+ // Generic instructions such as COPY will either disappear by register<br>
+ // coalescing or be lowered to SALU or VALU instructions.<br>
+ if (TargetInstrInfo::isGenericOpcode(MI.getOpcode())) {<br>
+ if (MI.getNumExplicitOperands() >= 1) {<br>
+ const MachineOperand &Op = MI.getOperand(0);<br>
+ if (Op.isReg()) {<br>
+ if (TRI->isSGPRReg(*MRI, Op.getReg())) {<br>
+ // SGPR instructions are not affected by EXEC<br>
+ continue;<br>
+ }<br>
+ }<br>
+ }<br>
+ }<br>
+<br>
+ char Needs = 0;<br>
+ char OutNeeds = 0;<br>
+ auto InstrInfoIt = Instructions.find(&MI);<br>
+ if (InstrInfoIt != Instructions.end()) {<br>
+ Needs = InstrInfoIt->second.Needs;<br>
+ OutNeeds = InstrInfoIt->second.OutNeeds;<br>
+<br>
+ // Make sure to switch to Exact mode before the end of the block when<br>
+ // Exact and only Exact is needed further downstream.<br>
+ if (OutNeeds == StateExact && (MI.isBranch() || MI.isTerminator())) {<br>
+ assert(Needs == 0);<br>
+ Needs = StateExact;<br>
+ }<br>
+ }<br>
+<br>
+ // State switching<br>
+ if (Needs && State != Needs) {<br>
+ if (Needs == StateExact) {<br>
+ assert(!SavedWQMReg);<br>
+<br>
+ if (!WQMFromExec && (OutNeeds & StateWQM))<br>
+ SavedWQMReg = MRI->createVirtualRegister(&AMDGPU::SReg_64RegClass);<br>
+<br>
+ toExact(MBB, &MI, SavedWQMReg, LiveMaskReg);<br>
+ } else {<br>
+ assert(WQMFromExec == (SavedWQMReg == 0));<br>
+ toWQM(MBB, &MI, SavedWQMReg);<br>
+ SavedWQMReg = 0;<br>
+ }<br>
+<br>
+ State = Needs;<br>
+ }<br>
+<br>
+ if (MI.getOpcode() == AMDGPU::SI_KILL)<br>
+ WQMFromExec = false;<br>
+ }<br>
+<br>
+ if ((BI.OutNeeds & StateWQM) && State != StateWQM) {<br>
+ assert(WQMFromExec == (SavedWQMReg == 0));<br>
+ toWQM(MBB, MBB.end(), SavedWQMReg);<br>
+ } else if (BI.OutNeeds == StateExact && State != StateExact) {<br>
+ toExact(MBB, MBB.end(), 0, LiveMaskReg);<br>
+ }<br>
+}<br>
+<br>
+bool SIWholeQuadMode::runOnMachineFunction(MachineFunction &MF) {<br>
+ SIMachineFunctionInfo *MFI = MF.getInfo<SIMachineFunctionInfo>();<br>
+<br>
+ if (MFI->getShaderType() != ShaderType::PIXEL)<br>
+ return false;<br>
+<br>
+ Instructions.clear();<br>
+ Blocks.clear();<br>
+ ExecExports.clear();<br>
+<br>
+ TII = static_cast<const SIInstrInfo *>(MF.getSubtarget().getInstrInfo());<br>
+ TRI = static_cast<const SIRegisterInfo *>(MF.getSubtarget().getRegisterInfo());<br>
+ MRI = &MF.getRegInfo();<br>
+<br>
+ char GlobalFlags = analyzeFunction(MF);<br>
+ if (!(GlobalFlags & StateWQM))<br>
+ return false;<br>
+<br>
+ MachineBasicBlock &Entry = MF.front();<br>
+ MachineInstr *EntryMI = Entry.getFirstNonPHI();<br>
+<br>
+ if (GlobalFlags == StateWQM) {<br>
+ // For a shader that needs only WQM, we can just set it once.<br>
+ BuildMI(Entry, EntryMI, DebugLoc(), TII->get(AMDGPU::S_WQM_B64),<br>
+ AMDGPU::EXEC).addReg(AMDGPU::EXEC);<br>
+ return true;<br>
+ }<br>
+<br>
+ // Handle the general case<br>
+ unsigned LiveMaskReg = MRI->createVirtualRegister(&AMDGPU::SReg_64RegClass);<br>
+ BuildMI(Entry, EntryMI, DebugLoc(), TII->get(AMDGPU::COPY), LiveMaskReg)<br>
+ .addReg(AMDGPU::EXEC);<br>
+<br>
+ for (const auto &BII : Blocks)<br>
+ processBlock(const_cast<MachineBasicBlock &>(*BII.first), LiveMaskReg,<br>
+ BII.first == &*MF.begin());<br>
+<br>
+ return true;<br>
+}<br>
<br>
Added: llvm/trunk/test/CodeGen/AMDGPU/wqm.ll<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/CodeGen/AMDGPU/wqm.ll?rev=263982&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/CodeGen/AMDGPU/wqm.ll?rev=263982&view=auto</a><br>
==============================================================================<br>
--- llvm/trunk/test/CodeGen/AMDGPU/wqm.ll (added)<br>
+++ llvm/trunk/test/CodeGen/AMDGPU/wqm.ll Mon Mar 21 15:28:33 2016<br>
@@ -0,0 +1,348 @@<br>
+;RUN: llc < %s -march=amdgcn -mcpu=verde -verify-machineinstrs | FileCheck %s --check-prefix=CHECK --check-prefix=SI<br>
+;RUN: llc < %s -march=amdgcn -mcpu=tonga -verify-machineinstrs | FileCheck %s --check-prefix=CHECK --check-prefix=VI<br>
+<br>
+; Check that WQM isn't triggered by image load/store intrinsics.<br>
+;<br>
+;CHECK-LABEL: {{^}}test1:<br>
+;CHECK-NOT: s_wqm<br>
+define <4 x float> @test1(<8 x i32> inreg %rsrc, <4 x i32> %c) #0 {<br>
+main_body:<br>
+ %tex = call <4 x float> @llvm.amdgcn.image.load.v4i32(<4 x i32> %c, <8 x i32> %rsrc, i32 15, i1 0, i1 0, i1 0, i1 0)<br>
+ call void @llvm.amdgcn.image.store.v4i32(<4 x float> %tex, <4 x i32> %c, <8 x i32> %rsrc, i32 15, i1 0, i1 0, i1 0, i1 0)<br>
+ ret <4 x float> %tex<br>
+}<br>
+<br>
+; Check that WQM is triggered by image samples and left untouched for loads...<br>
+;<br>
+;CHECK-LABEL: {{^}}test2:<br>
+;CHECK-NEXT: ; %main_body<br>
+;CHECK-NEXT: s_wqm_b64 exec, exec<br>
+;CHECK: image_sample<br>
+;CHECK-NOT: exec<br>
+;CHECK: _load_dword v0,<br>
+define float @test2(<8 x i32> inreg %rsrc, <4 x i32> inreg %sampler, float addrspace(1)* inreg %ptr, <4 x i32> %c) #0 {<br>
+main_body:<br>
+ %c.1 = call <4 x float> @llvm.SI.image.sample.v4i32(<4 x i32> %c, <8 x i32> %rsrc, <4 x i32> %sampler, i32 15, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0)<br>
+ %c.2 = bitcast <4 x float> %c.1 to <4 x i32><br>
+ %c.3 = extractelement <4 x i32> %c.2, i32 0<br>
+ %gep = getelementptr float, float addrspace(1)* %ptr, i32 %c.3<br>
+ %data = load float, float addrspace(1)* %gep<br>
+ ret float %data<br>
+}<br>
+<br>
+; ... but disabled for stores (and, in this simple case, not re-enabled).<br>
+;<br>
+;CHECK-LABEL: {{^}}test3:<br>
+;CHECK-NEXT: ; %main_body<br>
+;CHECK-NEXT: s_mov_b64 [[ORIG:s\[[0-9]+:[0-9]+\]]], exec<br>
+;CHECK-NEXT: s_wqm_b64 exec, exec<br>
+;CHECK: image_sample<br>
+;CHECK: s_and_b64 exec, exec, [[ORIG]]<br>
+;CHECK: store<br>
+;CHECK-NOT: exec<br>
+define <4 x float> @test3(<8 x i32> inreg %rsrc, <4 x i32> inreg %sampler, float addrspace(1)* inreg %ptr, <4 x i32> %c) #0 {<br>
+main_body:<br>
+ %tex = call <4 x float> @llvm.SI.image.sample.v4i32(<4 x i32> %c, <8 x i32> %rsrc, <4 x i32> %sampler, i32 15, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0)<br>
+ %tex.1 = bitcast <4 x float> %tex to <4 x i32><br>
+ %tex.2 = extractelement <4 x i32> %tex.1, i32 0<br>
+ %gep = getelementptr float, float addrspace(1)* %ptr, i32 %tex.2<br>
+ %wr = extractelement <4 x float> %tex, i32 1<br>
+ store float %wr, float addrspace(1)* %gep<br>
+ ret <4 x float> %tex<br>
+}<br>
+<br>
+; Check that WQM is re-enabled when required.<br>
+;<br>
+;CHECK-LABEL: {{^}}test4:<br>
+;CHECK-NEXT: ; %main_body<br>
+;CHECK-NEXT: s_mov_b64 [[ORIG:s\[[0-9]+:[0-9]+\]]], exec<br>
+;CHECK-NEXT: s_wqm_b64 exec, exec<br>
+;CHECK: v_mul_lo_i32 [[MUL:v[0-9]+]], v0, v1<br>
+;CHECK: s_and_b64 exec, exec, [[ORIG]]<br>
+;CHECK: store<br>
+;CHECK: s_wqm_b64 exec, exec<br>
+;CHECK: image_sample v[0:3], [[MUL]], s[0:7], s[8:11] dmask:0xf<br>
+define <4 x float> @test4(<8 x i32> inreg %rsrc, <4 x i32> inreg %sampler, float addrspace(1)* inreg %ptr, i32 %c, i32 %d, float %data) #0 {<br>
+main_body:<br>
+ %c.1 = mul i32 %c, %d<br>
+ %gep = getelementptr float, float addrspace(1)* %ptr, i32 %c.1<br>
+ store float %data, float addrspace(1)* %gep<br>
+ %tex = call <4 x float> @llvm.SI.image.sample.i32(i32 %c.1, <8 x i32> %rsrc, <4 x i32> %sampler, i32 15, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0)<br>
+ ret <4 x float> %tex<br>
+}<br>
+<br>
+; Check a case of one branch of an if-else requiring WQM, the other requiring<br>
+; exact.<br>
+;<br>
+; Note: In this particular case, the save-and-restore could be avoided if the<br>
+; analysis understood that the two branches of the if-else are mutually<br>
+; exclusive.<br>
+;<br>
+;CHECK-LABEL: {{^}}test_control_flow_0:<br>
+;CHECK-NEXT: ; %main_body<br>
+;CHECK-NEXT: s_mov_b64 [[ORIG:s\[[0-9]+:[0-9]+\]]], exec<br>
+;CHECK-NEXT: s_wqm_b64 exec, exec<br>
+;CHECK: %ELSE<br>
+;CHECK: s_and_saveexec_b64 [[SAVED:s\[[0-9]+:[0-9]+\]]], [[ORIG]]<br>
+;CHECK: store<br>
+;CHECK: s_mov_b64 exec, [[SAVED]]<br>
+;CHECK: %IF<br>
+;CHECK: image_sample<br>
+define float @test_control_flow_0(<8 x i32> inreg %rsrc, <4 x i32> inreg %sampler, float addrspace(1)* inreg %ptr, i32 %c, i32 %z, float %data) #0 {<br>
+main_body:<br>
+ %cmp = icmp eq i32 %z, 0<br>
+ br i1 %cmp, label %IF, label %ELSE<br>
+<br>
+IF:<br>
+ %tex = call <4 x float> @llvm.SI.image.sample.i32(i32 %c, <8 x i32> %rsrc, <4 x i32> %sampler, i32 15, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0)<br>
+ %data.if = extractelement <4 x float> %tex, i32 0<br>
+ br label %END<br>
+<br>
+ELSE:<br>
+ %gep = getelementptr float, float addrspace(1)* %ptr, i32 %c<br>
+ store float %data, float addrspace(1)* %gep<br>
+ br label %END<br>
+<br>
+END:<br>
+ %r = phi float [ %data.if, %IF ], [ %data, %ELSE ]<br>
+ ret float %r<br>
+}<br>
+<br>
+; Reverse branch order compared to the previous test.<br>
+;<br>
+;CHECK-LABEL: {{^}}test_control_flow_1:<br>
+;CHECK-NEXT: ; %main_body<br>
+;CHECK-NEXT: s_mov_b64 [[ORIG:s\[[0-9]+:[0-9]+\]]], exec<br>
+;CHECK-NEXT: s_wqm_b64 exec, exec<br>
+;CHECK: %IF<br>
+;CHECK: image_sample<br>
+;CHECK: %Flow<br>
+;CHECK-NEXT: s_or_saveexec_b64 [[SAVED:s\[[0-9]+:[0-9]+\]]],<br>
+;CHECK-NEXT: s_and_b64 exec, exec, [[ORIG]]<br>
+;CHECK-NEXT: s_and_b64 [[SAVED]], exec, [[SAVED]]<br>
+;CHECK-NEXT: s_xor_b64 exec, exec, [[SAVED]]<br>
+;CHECK-NEXT: %ELSE<br>
+;CHECK: store<br>
+;CHECK: %END<br>
+define float @test_control_flow_1(<8 x i32> inreg %rsrc, <4 x i32> inreg %sampler, float addrspace(1)* inreg %ptr, i32 %c, i32 %z, float %data) #0 {<br>
+main_body:<br>
+ %cmp = icmp eq i32 %z, 0<br>
+ br i1 %cmp, label %ELSE, label %IF<br>
+<br>
+IF:<br>
+ %tex = call <4 x float> @llvm.SI.image.sample.i32(i32 %c, <8 x i32> %rsrc, <4 x i32> %sampler, i32 15, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0)<br>
+ %data.if = extractelement <4 x float> %tex, i32 0<br>
+ br label %END<br>
+<br>
+ELSE:<br>
+ %gep = getelementptr float, float addrspace(1)* %ptr, i32 %c<br>
+ store float %data, float addrspace(1)* %gep<br>
+ br label %END<br>
+<br>
+END:<br>
+ %r = phi float [ %data.if, %IF ], [ %data, %ELSE ]<br>
+ ret float %r<br>
+}<br>
+<br>
+; Check that branch conditions are properly marked as needing WQM...<br>
+;<br>
+;CHECK-LABEL: {{^}}test_control_flow_2:<br>
+;CHECK-NEXT: ; %main_body<br>
+;CHECK-NEXT: s_mov_b64 [[ORIG:s\[[0-9]+:[0-9]+\]]], exec<br>
+;CHECK-NEXT: s_wqm_b64 exec, exec<br>
+;CHECK: s_and_b64 exec, exec, [[ORIG]]<br>
+;CHECK: store<br>
+;CHECK: s_wqm_b64 exec, exec<br>
+;CHECK: load<br>
+;CHECK: s_and_b64 exec, exec, [[ORIG]]<br>
+;CHECK: store<br>
+;CHECK: s_wqm_b64 exec, exec<br>
+;CHECK: v_cmp<br>
+define <4 x float> @test_control_flow_2(<8 x i32> inreg %rsrc, <4 x i32> inreg %sampler, float addrspace(1)* inreg %ptr, <3 x i32> %idx, <2 x float> %data, i32 %coord) #0 {<br>
+main_body:<br>
+ %idx.1 = extractelement <3 x i32> %idx, i32 0<br>
+ %gep.1 = getelementptr float, float addrspace(1)* %ptr, i32 %idx.1<br>
+ %data.1 = extractelement <2 x float> %data, i32 0<br>
+ store float %data.1, float addrspace(1)* %gep.1<br>
+<br>
+ ; The load that determines the branch (and should therefore be WQM) is<br>
+ ; surrounded by stores that require disabled WQM.<br>
+ %idx.2 = extractelement <3 x i32> %idx, i32 1<br>
+ %gep.2 = getelementptr float, float addrspace(1)* %ptr, i32 %idx.2<br>
+ %z = load float, float addrspace(1)* %gep.2<br>
+<br>
+ %idx.3 = extractelement <3 x i32> %idx, i32 2<br>
+ %gep.3 = getelementptr float, float addrspace(1)* %ptr, i32 %idx.3<br>
+ %data.3 = extractelement <2 x float> %data, i32 1<br>
+ store float %data.3, float addrspace(1)* %gep.3<br>
+<br>
+ %cc = fcmp ogt float %z, 0.0<br>
+ br i1 %cc, label %IF, label %ELSE<br>
+<br>
+IF:<br>
+ %coord.IF = mul i32 %coord, 3<br>
+ br label %END<br>
+<br>
+ELSE:<br>
+ %coord.ELSE = mul i32 %coord, 4<br>
+ br label %END<br>
+<br>
+END:<br>
+ %coord.END = phi i32 [ %coord.IF, %IF ], [ %coord.ELSE, %ELSE ]<br>
+ %tex = call <4 x float> @llvm.SI.image.sample.i32(i32 %coord.END, <8 x i32> %rsrc, <4 x i32> %sampler, i32 15, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0)<br>
+ ret <4 x float> %tex<br>
+}<br>
+<br>
+; ... but only if they really do need it.<br>
+;<br>
+;CHECK-LABEL: {{^}}test_control_flow_3:<br>
+;CHECK-NEXT: ; %main_body<br>
+;CHECK-NEXT: s_mov_b64 [[ORIG:s\[[0-9]+:[0-9]+\]]], exec<br>
+;CHECK-NEXT: s_wqm_b64 exec, exec<br>
+;CHECK: image_sample<br>
+;CHECK: s_and_b64 exec, exec, [[ORIG]]<br>
+;CHECK: store<br>
+;CHECK: load<br>
+;CHECK: store<br>
+;CHECK: v_cmp<br>
+define float @test_control_flow_3(<8 x i32> inreg %rsrc, <4 x i32> inreg %sampler, float addrspace(1)* inreg %ptr, <3 x i32> %idx, <2 x float> %data, i32 %coord) #0 {<br>
+main_body:<br>
+ %tex = call <4 x float> @llvm.SI.image.sample.i32(i32 %coord, <8 x i32> %rsrc, <4 x i32> %sampler, i32 15, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0)<br>
+ %tex.1 = extractelement <4 x float> %tex, i32 0<br>
+<br>
+ %idx.1 = extractelement <3 x i32> %idx, i32 0<br>
+ %gep.1 = getelementptr float, float addrspace(1)* %ptr, i32 %idx.1<br>
+ %data.1 = extractelement <2 x float> %data, i32 0<br>
+ store float %data.1, float addrspace(1)* %gep.1<br>
+<br>
+ %idx.2 = extractelement <3 x i32> %idx, i32 1<br>
+ %gep.2 = getelementptr float, float addrspace(1)* %ptr, i32 %idx.2<br>
+ %z = load float, float addrspace(1)* %gep.2<br>
+<br>
+ %idx.3 = extractelement <3 x i32> %idx, i32 2<br>
+ %gep.3 = getelementptr float, float addrspace(1)* %ptr, i32 %idx.3<br>
+ %data.3 = extractelement <2 x float> %data, i32 1<br>
+ store float %data.3, float addrspace(1)* %gep.3<br>
+<br>
+ %cc = fcmp ogt float %z, 0.0<br>
+ br i1 %cc, label %IF, label %ELSE<br>
+<br>
+IF:<br>
+ %tex.IF = fmul float %tex.1, 3.0<br>
+ br label %END<br>
+<br>
+ELSE:<br>
+ %tex.ELSE = fmul float %tex.1, 4.0<br>
+ br label %END<br>
+<br>
+END:<br>
+ %tex.END = phi float [ %tex.IF, %IF ], [ %tex.ELSE, %ELSE ]<br>
+ ret float %tex.END<br>
+}<br>
+<br>
+; Another test that failed at some point because of terminator handling.<br>
+;<br>
+;CHECK-LABEL: {{^}}test_control_flow_4:<br>
+;CHECK-NEXT: ; %main_body<br>
+;CHECK-NEXT: s_mov_b64 [[ORIG:s\[[0-9]+:[0-9]+\]]], exec<br>
+;CHECK-NEXT: s_wqm_b64 exec, exec<br>
+;CHECK: %IF<br>
+;CHECK: load<br>
+;CHECK: s_and_saveexec_b64 [[SAVE:s\[[0-9]+:[0-9]+\]]], [[ORIG]]<br>
+;CHECK: store<br>
+;CHECK: s_mov_b64 exec, [[SAVE]]<br>
+;CHECK: %END<br>
+;CHECK: image_sample<br>
+define <4 x float> @test_control_flow_4(<8 x i32> inreg %rsrc, <4 x i32> inreg %sampler, float addrspace(1)* inreg %ptr, i32 %coord, i32 %y, float %z) #0 {<br>
+main_body:<br>
+ %cond = icmp eq i32 %y, 0<br>
+ br i1 %cond, label %IF, label %END<br>
+<br>
+IF:<br>
+ %data = load float, float addrspace(1)* %ptr<br>
+ %gep = getelementptr float, float addrspace(1)* %ptr, i32 1<br>
+ store float %data, float addrspace(1)* %gep<br>
+ br label %END<br>
+<br>
+END:<br>
+ %tex = call <4 x float> @llvm.SI.image.sample.i32(i32 %coord, <8 x i32> %rsrc, <4 x i32> %sampler, i32 15, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0)<br>
+ ret <4 x float> %tex<br>
+}<br>
+<br>
+; Kill is performed in WQM mode so that uniform kill behaves correctly ...<br>
+;<br>
+;CHECK-LABEL: {{^}}test_kill_0:<br>
+;CHECK-NEXT: ; %main_body<br>
+;CHECK-NEXT: s_mov_b64 [[ORIG:s\[[0-9]+:[0-9]+\]]], exec<br>
+;CHECK-NEXT: s_wqm_b64 exec, exec<br>
+;CHECK: image_sample<br>
+;CHECK: s_and_b64 exec, exec, [[ORIG]]<br>
+;SI: buffer_store_dword<br>
+;VI: flat_store_dword<br>
+;CHECK: s_wqm_b64 exec, exec<br>
+;CHECK: v_cmpx_<br>
+;CHECK: s_and_saveexec_b64 [[SAVE:s\[[0-9]+:[0-9]+\]]], [[ORIG]]<br>
+;SI: buffer_store_dword<br>
+;VI: flat_store_dword<br>
+;CHECK: s_mov_b64 exec, [[SAVE]]<br>
+;CHECK: image_sample<br>
+define <4 x float> @test_kill_0(<8 x i32> inreg %rsrc, <4 x i32> inreg %sampler, float addrspace(1)* inreg %ptr, <2 x i32> %idx, <2 x float> %data, i32 %coord, i32 %coord2, float %z) #0 {<br>
+main_body:<br>
+ %tex = call <4 x float> @llvm.SI.image.sample.i32(i32 %coord, <8 x i32> %rsrc, <4 x i32> %sampler, i32 15, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0)<br>
+<br>
+ %idx.0 = extractelement <2 x i32> %idx, i32 0<br>
+ %gep.0 = getelementptr float, float addrspace(1)* %ptr, i32 %idx.0<br>
+ %data.0 = extractelement <2 x float> %data, i32 0<br>
+ store float %data.0, float addrspace(1)* %gep.0<br>
+<br>
+ call void @llvm.AMDGPU.kill(float %z)<br>
+<br>
+ %idx.1 = extractelement <2 x i32> %idx, i32 1<br>
+ %gep.1 = getelementptr float, float addrspace(1)* %ptr, i32 %idx.1<br>
+ %data.1 = extractelement <2 x float> %data, i32 1<br>
+ store float %data.1, float addrspace(1)* %gep.1<br>
+<br>
+ %tex2 = call <4 x float> @llvm.SI.image.sample.i32(i32 %coord2, <8 x i32> %rsrc, <4 x i32> %sampler, i32 15, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0)<br>
+ %out = fadd <4 x float> %tex, %tex2<br>
+<br>
+ ret <4 x float> %out<br>
+}<br>
+<br>
+; ... but only if WQM is necessary.<br>
+;<br>
+;CHECK-LABEL: {{^}}test_kill_1:<br>
+;CHECK-NEXT: ; %main_body<br>
+;CHECK-NEXT: s_mov_b64 [[ORIG:s\[[0-9]+:[0-9]+\]]], exec<br>
+;CHECK-NEXT: s_wqm_b64 exec, exec<br>
+;CHECK: image_sample<br>
+;CHECK: s_and_b64 exec, exec, [[ORIG]]<br>
+;SI: buffer_store_dword<br>
+;VI: flat_store_dword<br>
+;CHECK-NOT: wqm<br>
+;CHECK: v_cmpx_<br>
+define <4 x float> @test_kill_1(<8 x i32> inreg %rsrc, <4 x i32> inreg %sampler, float addrspace(1)* inreg %ptr, i32 %idx, float %data, i32 %coord, i32 %coord2, float %z) #0 {<br>
+main_body:<br>
+ %tex = call <4 x float> @llvm.SI.image.sample.i32(i32 %coord, <8 x i32> %rsrc, <4 x i32> %sampler, i32 15, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0)<br>
+<br>
+ %gep = getelementptr float, float addrspace(1)* %ptr, i32 %idx<br>
+ store float %data, float addrspace(1)* %gep<br>
+<br>
+ call void @llvm.AMDGPU.kill(float %z)<br>
+<br>
+ ret <4 x float> %tex<br>
+}<br>
+<br>
+declare void @llvm.amdgcn.image.store.v4i32(<4 x float>, <4 x i32>, <8 x i32>, i32, i1, i1, i1, i1) #1<br>
+<br>
+declare <4 x float> @llvm.amdgcn.image.load.v4i32(<4 x i32>, <8 x i32>, i32, i1, i1, i1, i1) #2<br>
+<br>
+declare <4 x float> @llvm.SI.image.sample.i32(i32, <8 x i32>, <4 x i32>, i32, i32, i32, i32, i32, i32, i32, i32) #3<br>
+declare <4 x float> @llvm.SI.image.sample.v4i32(<4 x i32>, <8 x i32>, <4 x i32>, i32, i32, i32, i32, i32, i32, i32, i32) #3<br>
+<br>
+declare void @llvm.AMDGPU.kill(float)<br>
+declare void @llvm.SI.export(i32, i32, i32, i32, i32, float, float, float, float)<br>
+<br>
+attributes #0 = { "ShaderType"="0" }<br>
+attributes #1 = { nounwind }<br>
+attributes #2 = { nounwind readonly }<br>
+attributes #3 = { nounwind readnone }<br>
<br>
<br>
_______________________________________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br>
</blockquote></div>