<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">A revised version which fixes the reproducer was uploaded to D32871. Could you please test it pre-commit and let us know of any issues that may remain, or would
you prefer we re-commit?<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></a></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Chandler Carruth [mailto:chandlerc@gmail.com]
<br>
<b>Sent:</b> Monday, August 21, 2017 02:18<br>
<b>To:</b> Zaks, Ayal <ayal.zaks@intel.com>; llvm-commits@lists.llvm.org<br>
<b>Subject:</b> Re: [llvm] r311077 - [LV] Using VPlan to model the vectorized code and drive its transformation<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Sorry for the trouble, but this is crashing or timing out the compiler for lots of tests for us.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">libpng's png.c is enough to reproduce an assert failure and I've filed
<a href="http://llvm.org/PR34248">http://llvm.org/PR34248</a> with an attached test case.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I've reverted it for now in r311304.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Aug 17, 2017 at 2:30 AM Ayal Zaks via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">Author: ayalz<br>
Date: Thu Aug 17 02:29:59 2017<br>
New Revision: 311077<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=311077&view=rev" target="_blank">
http://llvm.org/viewvc/llvm-project?rev=311077&view=rev</a><br>
Log:<br>
[LV] Using VPlan to model the vectorized code and drive its transformation<br>
<br>
VPlan is an ongoing effort to refactor and extend the Loop Vectorizer. This<br>
patch introduces the VPlan model into LV and uses it to represent the vectorized<br>
code and drive the generation of vectorized IR.<br>
<br>
In this patch VPlan models the vectorized loop body: the vectorized control-flow<br>
is represented using VPlan's Hierarchical CFG, with predication refactored from<br>
being a post-vectorization-step into a vectorization planning step modeling<br>
if-then VPRegionBlocks, and generating code inline with non-predicated code. The<br>
vectorized code within each VPBasicBlock is represented as a sequence of<br>
Recipes, each responsible for modelling and generating a sequence of IR<br>
instructions. To keep the size of this commit manageable the Recipes in this<br>
patch are coarse-grained and capture large chunks of LV's code-generation logic.<br>
The constructed VPlans are dumped in dot format under -debug.<br>
<br>
This commit retains current vectorizer output, except for minor instruction<br>
reorderings; see associated modifications to lit tests.<br>
<br>
For further details on the VPlan model see docs/Proposals/VectorizationPlan.rst<br>
and its references.<br>
<br>
Authors: Gil Rapaport and Ayal Zaks<br>
<br>
Differential Revision: <a href="https://reviews.llvm.org/D32871" target="_blank">
https://reviews.llvm.org/D32871</a><br>
<br>
Added:<br>
llvm/trunk/lib/Transforms/Vectorize/VPlan.cpp<br>
llvm/trunk/lib/Transforms/Vectorize/VPlan.h<br>
Modified:<br>
llvm/trunk/lib/Transforms/Vectorize/CMakeLists.txt<br>
llvm/trunk/lib/Transforms/Vectorize/LoopVectorize.cpp<br>
llvm/trunk/test/Transforms/LoopVectorize/AArch64/aarch64-predication.ll<br>
llvm/trunk/test/Transforms/LoopVectorize/AArch64/predication_costs.ll<br>
llvm/trunk/test/Transforms/LoopVectorize/SystemZ/load-store-scalarization-cost.ll<br>
llvm/trunk/test/Transforms/LoopVectorize/first-order-recurrence.ll<br>
llvm/trunk/test/Transforms/LoopVectorize/if-pred-non-void.ll<br>
<br>
Modified: llvm/trunk/lib/Transforms/Vectorize/CMakeLists.txt<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Vectorize/CMakeLists.txt?rev=311077&r1=311076&r2=311077&view=diff" target="_blank">
http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Vectorize/CMakeLists.txt?rev=311077&r1=311076&r2=311077&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Transforms/Vectorize/CMakeLists.txt (original)<br>
+++ llvm/trunk/lib/Transforms/Vectorize/CMakeLists.txt Thu Aug 17 02:29:59 2017<br>
@@ -3,6 +3,7 @@ add_llvm_library(LLVMVectorize<br>
LoopVectorize.cpp<br>
SLPVectorizer.cpp<br>
Vectorize.cpp<br>
+ VPlan.cpp<br>
<br>
ADDITIONAL_HEADER_DIRS<br>
${LLVM_MAIN_INCLUDE_DIR}/llvm/Transforms<br>
<br>
Modified: llvm/trunk/lib/Transforms/Vectorize/LoopVectorize.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Vectorize/LoopVectorize.cpp?rev=311077&r1=311076&r2=311077&view=diff" target="_blank">
http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Vectorize/LoopVectorize.cpp?rev=311077&r1=311076&r2=311077&view=diff</a><br>
==============================================================================<br>
--- llvm/trunk/lib/Transforms/Vectorize/LoopVectorize.cpp (original)<br>
+++ llvm/trunk/lib/Transforms/Vectorize/LoopVectorize.cpp Thu Aug 17 02:29:59 2017<br>
@@ -47,6 +47,7 @@<br>
//===----------------------------------------------------------------------===//<br>
<br>
#include "llvm/Transforms/Vectorize/LoopVectorize.h"<br>
+#include "VPlan.h"<br>
#include "llvm/ADT/DenseMap.h"<br>
#include "llvm/ADT/Hashing.h"<br>
#include "llvm/ADT/MapVector.h"<br>
@@ -98,6 +99,7 @@<br>
#include "llvm/Transforms/Utils/LoopVersioning.h"<br>
#include "llvm/Transforms/Vectorize.h"<br>
#include <algorithm><br>
+#include <functional><br>
#include <map><br>
#include <tuple><br>
<br>
@@ -250,6 +252,10 @@ class LoopVectorizeHints;<br>
class LoopVectorizationLegality;<br>
class LoopVectorizationCostModel;<br>
class LoopVectorizationRequirements;<br>
+class VPInterleaveRecipe;<br>
+class VPReplicateRecipe;<br>
+class VPWidenIntOrFpInductionRecipe;<br>
+class VPWidenRecipe;<br>
<br>
/// Returns true if the given loop body has a cycle, excluding the loop<br>
/// itself.<br>
@@ -362,6 +368,9 @@ static Constant *getSignedIntOrFpConstan<br>
: ConstantFP::get(Ty, C);<br>
}<br>
<br>
+} // end anonymous namespace<br>
+<br>
+namespace llvm {<br>
/// InnerLoopVectorizer vectorizes loops which contain only one basic<br>
/// block to a specified vectorization factor (VF).<br>
/// This class performs the widening of scalars into vectors, or multiple<br>
@@ -393,10 +402,11 @@ public:<br>
AddedSafetyChecks(false) {}<br>
<br>
/// Create a new empty loop. Unlink the old loop and connect the new one.<br>
- void createVectorizedLoopSkeleton();<br>
+ /// Return the pre-header block of the new loop.<br>
+ BasicBlock *createVectorizedLoopSkeleton();<br>
<br>
- /// Vectorize a single instruction within the innermost loop.<br>
- void vectorizeInstruction(Instruction &I);<br>
+ /// Widen a single instruction within the innermost loop.<br>
+ void widenInstruction(Instruction &I);<br>
<br>
/// Fix the vectorized code, taking care of header phi's, live-outs, and more.<br>
void fixVectorizedLoop();<br>
@@ -406,15 +416,72 @@ public:<br>
<br>
virtual ~InnerLoopVectorizer() {}<br>
<br>
-protected:<br>
- /// A small list of PHINodes.<br>
- typedef SmallVector<PHINode *, 4> PhiVector;<br>
-<br>
/// A type for vectorized values in the new loop. Each value from the<br>
/// original loop, when vectorized, is represented by UF vector values in the<br>
/// new unrolled loop, where UF is the unroll factor.<br>
typedef SmallVector<Value *, 2> VectorParts;<br>
<br>
+ /// A helper function that computes the predicate of the block BB, assuming<br>
+ /// that the header block of the loop is set to True. It returns the *entry*<br>
+ /// mask for the block BB.<br>
+ VectorParts createBlockInMask(BasicBlock *BB);<br>
+<br>
+ /// Vectorize a single PHINode in a block. This method handles the induction<br>
+ /// variable canonicalization. It supports both VF = 1 for unrolled loops and<br>
+ /// arbitrary length vectors.<br>
+ void widenPHIInstruction(Instruction *PN, unsigned UF, unsigned VF);<br>
+<br>
+ /// A helper function to scalarize a single Instruction in the innermost loop.<br>
+ /// Generates a sequence of scalar instances for each lane between \p MinLane<br>
+ /// and \p MaxLane, times each part between \p MinPart and \p MaxPart,<br>
+ /// inclusive..<br>
+ void scalarizeInstruction(Instruction *Instr, const VPIteration &Instance,<br>
+ bool IfPredicateInstr);<br>
+<br>
+ /// Widen an integer or floating-point induction variable \p IV. If \p Trunc<br>
+ /// is provided, the integer induction variable will first be truncated to<br>
+ /// the corresponding type.<br>
+ void widenIntOrFpInduction(PHINode *IV, TruncInst *Trunc = nullptr);<br>
+<br>
+ /// getOrCreateVectorValue and getOrCreateScalarValue coordinate to generate a<br>
+ /// vector or scalar value on-demand if one is not yet available. When<br>
+ /// vectorizing a loop, we visit the definition of an instruction before its<br>
+ /// uses. When visiting the definition, we either vectorize or scalarize the<br>
+ /// instruction, creating an entry for it in the corresponding map. (In some<br>
+ /// cases, such as induction variables, we will create both vector and scalar<br>
+ /// entries.) Then, as we encounter uses of the definition, we derive values<br>
+ /// for each scalar or vector use unless such a value is already available.<br>
+ /// For example, if we scalarize a definition and one of its uses is vector,<br>
+ /// we build the required vector on-demand with an insertelement sequence<br>
+ /// when visiting the use. Otherwise, if the use is scalar, we can use the<br>
+ /// existing scalar definition.<br>
+ ///<br>
+ /// Return a value in the new loop corresponding to \p V from the original<br>
+ /// loop at unroll index \p Part. If the value has already been vectorized,<br>
+ /// the corresponding vector entry in VectorLoopValueMap is returned. If,<br>
+ /// however, the value has a scalar entry in VectorLoopValueMap, we construct<br>
+ /// a new vector value on-demand by inserting the scalar values into a vector<br>
+ /// with an insertelement sequence. If the value has been neither vectorized<br>
+ /// nor scalarized, it must be loop invariant, so we simply broadcast the<br>
+ /// value into a vector.<br>
+ Value *getOrCreateVectorValue(Value *V, unsigned Part);<br>
+<br>
+ /// Return a value in the new loop corresponding to \p V from the original<br>
+ /// loop at unroll and vector indices \p Instance. If the value has been<br>
+ /// vectorized but not scalarized, the necessary extractelement instruction<br>
+ /// will be generated.<br>
+ Value *getOrCreateScalarValue(Value *V, const VPIteration &Instance);<br>
+<br>
+ /// Construct the vector value of a scalarized value \p V one lane at a time.<br>
+ void packScalarIntoVectorValue(Value *V, const VPIteration &Instance);<br>
+<br>
+ /// Try to vectorize the interleaved access group that \p Instr belongs to.<br>
+ void vectorizeInterleaveGroup(Instruction *Instr);<br>
+<br>
+protected:<br>
+ /// A small list of PHINodes.<br>
+ typedef SmallVector<PHINode *, 4> PhiVector;<br>
+<br>
/// A type for scalarized values in the new loop. Each value from the<br>
/// original loop, when scalarized, is represented by UF x VF scalar values<br>
/// in the new unrolled loop, where UF is the unroll factor and VF is the<br>
@@ -457,37 +524,18 @@ protected:<br>
/// the block that was created for it.<br>
void sinkScalarOperands(Instruction *PredInst);<br>
<br>
- /// Predicate conditional instructions that require predication on their<br>
- /// respective conditions.<br>
- void predicateInstructions();<br>
-<br>
/// Shrinks vector element sizes to the smallest bitwidth they can be legally<br>
/// represented as.<br>
void truncateToMinimalBitwidths();<br>
<br>
- /// A helper function that computes the predicate of the block BB, assuming<br>
- /// that the header block of the loop is set to True. It returns the *entry*<br>
- /// mask for the block BB.<br>
- VectorParts createBlockInMask(BasicBlock *BB);<br>
/// A helper function that computes the predicate of the edge between SRC<br>
/// and DST.<br>
VectorParts createEdgeMask(BasicBlock *Src, BasicBlock *Dst);<br>
<br>
- /// Vectorize a single PHINode in a block. This method handles the induction<br>
- /// variable canonicalization. It supports both VF = 1 for unrolled loops and<br>
- /// arbitrary length vectors.<br>
- void widenPHIInstruction(Instruction *PN, unsigned UF, unsigned VF);<br>
-<br>
/// Insert the new loop to the loop hierarchy and pass manager<br>
/// and update the analysis passes.<br>
void updateAnalysis();<br>
<br>
- /// This instruction is un-vectorizable. Implement it as a sequence<br>
- /// of scalars. If \p IfPredicateInstr is true we need to 'hide' each<br>
- /// scalarized instruction behind an if block predicated on the control<br>
- /// dependence of the instruction.<br>
- void scalarizeInstruction(Instruction *Instr, bool IfPredicateInstr = false);<br>
-<br>
/// Vectorize Load and Store instructions,<br>
virtual void vectorizeMemoryInstruction(Instruction *Instr);<br>
<br>
@@ -521,11 +569,6 @@ protected:<br>
void createVectorIntOrFpInductionPHI(const InductionDescriptor &II,<br>
Value *Step, Instruction *EntryVal);<br>
<br>
- /// Widen an integer or floating-point induction variable \p IV. If \p Trunc<br>
- /// is provided, the integer induction variable will first be truncated to<br>
- /// the corresponding type.<br>
- void widenIntOrFpInduction(PHINode *IV, TruncInst *Trunc = nullptr);<br>
-<br>
/// Returns true if an instruction \p I should be scalarized instead of<br>
/// vectorized for the chosen vectorization factor.<br>
bool shouldScalarizeInstruction(Instruction *I) const;<br>
@@ -533,38 +576,6 @@ protected:<br>
/// Returns true if we should generate a scalar version of \p IV.<br>
bool needsScalarInduction(Instruction *IV) const;<br>
<br>
- /// getOrCreateVectorValue and getOrCreateScalarValue coordinate to generate a<br>
- /// vector or scalar value on-demand if one is not yet available. When<br>
- /// vectorizing a loop, we visit the definition of an instruction before its<br>
- /// uses. When visiting the definition, we either vectorize or scalarize the<br>
- /// instruction, creating an entry for it in the corresponding map. (In some<br>
- /// cases, such as induction variables, we will create both vector and scalar<br>
- /// entries.) Then, as we encounter uses of the definition, we derive values<br>
- /// for each scalar or vector use unless such a value is already available.<br>
- /// For example, if we scalarize a definition and one of its uses is vector,<br>
- /// we build the required vector on-demand with an insertelement sequence<br>
- /// when visiting the use. Otherwise, if the use is scalar, we can use the<br>
- /// existing scalar definition.<br>
- ///<br>
- /// Return a value in the new loop corresponding to \p V from the original<br>
- /// loop at unroll index \p Part. If the value has already been vectorized,<br>
- /// the corresponding vector entry in VectorLoopValueMap is returned. If,<br>
- /// however, the value has a scalar entry in VectorLoopValueMap, we construct<br>
- /// a new vector value on-demand by inserting the scalar values into a vector<br>
- /// with an insertelement sequence. If the value has been neither vectorized<br>
- /// nor scalarized, it must be loop invariant, so we simply broadcast the<br>
- /// value into a vector.<br>
- Value *getOrCreateVectorValue(Value *V, unsigned Part);<br>
-<br>
- /// Return a value in the new loop corresponding to \p V from the original<br>
- /// loop at unroll index \p Part and vector index \p Lane. If the value has<br>
- /// been vectorized but not scalarized, the necessary extractelement<br>
- /// instruction will be generated.<br>
- Value *getOrCreateScalarValue(Value *V, unsigned Part, unsigned Lane);<br>
-<br>
- /// Try to vectorize the interleaved access group that \p Instr belongs to.<br>
- void vectorizeInterleaveGroup(Instruction *Instr);<br>
-<br>
/// Generate a shuffle sequence that will reverse the vector Vec.<br>
virtual Value *reverseVector(Value *Vec);<br>
<br>
@@ -605,127 +616,6 @@ protected:<br>
/// the instruction.<br>
void setDebugLocFromInst(IRBuilder<> &B, const Value *Ptr);<br>
<br>
- /// This is a helper class for maintaining vectorization state. It's used for<br>
- /// mapping values from the original loop to their corresponding values in<br>
- /// the new loop. Two mappings are maintained: one for vectorized values and<br>
- /// one for scalarized values. Vectorized values are represented with UF<br>
- /// vector values in the new loop, and scalarized values are represented with<br>
- /// UF x VF scalar values in the new loop. UF and VF are the unroll and<br>
- /// vectorization factors, respectively.<br>
- ///<br>
- /// Entries can be added to either map with setVectorValue and setScalarValue,<br>
- /// which assert that an entry was not already added before. If an entry is to<br>
- /// replace an existing one, call resetVectorValue. This is currently needed<br>
- /// to modify the mapped values during "fix-up" operations that occur once the<br>
- /// first phase of widening is complete. These operations include type<br>
- /// truncation and the second phase of recurrence widening.<br>
- ///<br>
- /// Entries from either map can be retrieved using the getVectorValue and<br>
- /// getScalarValue functions, which assert that the desired value exists.<br>
-<br>
- struct ValueMap {<br>
-<br>
- /// Construct an empty map with the given unroll and vectorization factors.<br>
- ValueMap(unsigned UF, unsigned VF) : UF(UF), VF(VF) {}<br>
-<br>
- /// \return True if the map has any vector entry for \p Key.<br>
- bool hasAnyVectorValue(Value *Key) const {<br>
- return VectorMapStorage.count(Key);<br>
- }<br>
-<br>
- /// \return True if the map has a vector entry for \p Key and \p Part.<br>
- bool hasVectorValue(Value *Key, unsigned Part) const {<br>
- assert(Part < UF && "Queried Vector Part is too large.");<br>
- if (!hasAnyVectorValue(Key))<br>
- return false;<br>
- const VectorParts &Entry = VectorMapStorage.find(Key)->second;<br>
- assert(Entry.size() == UF && "VectorParts has wrong dimensions.");<br>
- return Entry[Part] != nullptr;<br>
- }<br>
-<br>
- /// \return True if the map has any scalar entry for \p Key.<br>
- bool hasAnyScalarValue(Value *Key) const {<br>
- return ScalarMapStorage.count(Key);<br>
- }<br>
-<br>
- /// \return True if the map has a scalar entry for \p Key, \p Part and<br>
- /// \p Part.<br>
- bool hasScalarValue(Value *Key, unsigned Part, unsigned Lane) const {<br>
- assert(Part < UF && "Queried Scalar Part is too large.");<br>
- assert(Lane < VF && "Queried Scalar Lane is too large.");<br>
- if (!hasAnyScalarValue(Key))<br>
- return false;<br>
- const ScalarParts &Entry = ScalarMapStorage.find(Key)->second;<br>
- assert(Entry.size() == UF && "ScalarParts has wrong dimensions.");<br>
- assert(Entry[Part].size() == VF && "ScalarParts has wrong dimensions.");<br>
- return Entry[Part][Lane] != nullptr;<br>
- }<br>
-<br>
- /// Retrieve the existing vector value that corresponds to \p Key and<br>
- /// \p Part.<br>
- Value *getVectorValue(Value *Key, unsigned Part) {<br>
- assert(hasVectorValue(Key, Part) && "Getting non-existent value.");<br>
- return VectorMapStorage[Key][Part];<br>
- }<br>
-<br>
- /// Retrieve the existing scalar value that corresponds to \p Key, \p Part<br>
- /// and \p Lane.<br>
- Value *getScalarValue(Value *Key, unsigned Part, unsigned Lane) {<br>
- assert(hasScalarValue(Key, Part, Lane) && "Getting non-existent value.");<br>
- return ScalarMapStorage[Key][Part][Lane];<br>
- }<br>
-<br>
- /// Set a vector value associated with \p Key and \p Part. Assumes such a<br>
- /// value is not already set. If it is, use resetVectorValue() instead.<br>
- void setVectorValue(Value *Key, unsigned Part, Value *Vector) {<br>
- assert(!hasVectorValue(Key, Part) && "Vector value already set for part");<br>
- if (!VectorMapStorage.count(Key)) {<br>
- VectorParts Entry(UF);<br>
- VectorMapStorage[Key] = Entry;<br>
- }<br>
- VectorMapStorage[Key][Part] = Vector;<br>
- }<br>
-<br>
- /// Set a scalar value associated with \p Key for \p Part and \p Lane.<br>
- /// Assumes such a value is not already set.<br>
- void setScalarValue(Value *Key, unsigned Part, unsigned Lane,<br>
- Value *Scalar) {<br>
- assert(!hasScalarValue(Key, Part, Lane) && "Scalar value already set");<br>
- if (!ScalarMapStorage.count(Key)) {<br>
- ScalarParts Entry(UF);<br>
- for (unsigned Part = 0; Part < UF; ++Part)<br>
- Entry[Part].resize(VF, nullptr);<br>
- // TODO: Consider storing uniform values only per-part, as they occupy<br>
- // lane 0 only, keeping the other VF-1 redundant entries null.<br>
- ScalarMapStorage[Key] = Entry;<br>
- }<br>
- ScalarMapStorage[Key][Part][Lane] = Scalar;<br>
- }<br>
-<br>
- /// Reset the vector value associated with \p Key for the given \p Part.<br>
- /// This function can be used to update values that have already been<br>
- /// vectorized. This is the case for "fix-up" operations including type<br>
- /// truncation and the second phase of recurrence vectorization.<br>
- void resetVectorValue(Value *Key, unsigned Part, Value *Vector) {<br>
- assert(hasVectorValue(Key, Part) && "Vector value not set for part");<br>
- VectorMapStorage[Key][Part] = Vector;<br>
- }<br>
-<br>
- private:<br>
- /// The unroll factor. Each entry in the vector map contains UF vector<br>
- /// values.<br>
- unsigned UF;<br>
-<br>
- /// The vectorization factor. Each entry in the scalar map contains UF x VF<br>
- /// scalar values.<br>
- unsigned VF;<br>
-<br>
- /// The vector and scalar map storage. We use std::map and not DenseMap<br>
- /// because insertions to DenseMap invalidate its iterators.<br>
- std::map<Value *, VectorParts> VectorMapStorage;<br>
- std::map<Value *, ScalarParts> ScalarMapStorage;<br>
- };<br>
-<br>
/// The original loop.<br>
Loop *OrigLoop;<br>
/// A wrapper around ScalarEvolution used to add runtime SCEV checks. Applies<br>
@@ -758,7 +648,6 @@ protected:<br>
/// vector elements.<br>
unsigned VF;<br>
<br>
-protected:<br>
/// The vectorization unroll factor to use. Each scalar is vectorized to this<br>
/// many different vector instructions.<br>
unsigned UF;<br>
@@ -792,11 +681,10 @@ protected:<br>
/// vectorized loop. A key value can map to either vector values, scalar<br>
/// values or both kinds of values, depending on whether the key was<br>
/// vectorized and scalarized.<br>
- ValueMap VectorLoopValueMap;<br>
+ VectorizerValueMap VectorLoopValueMap;<br>
<br>
- /// Store instructions that should be predicated, as a pair<br>
- /// <StoreInst, Predicate><br>
- SmallVector<std::pair<Instruction *, Value *>, 4> PredicatedInstructions;<br>
+ /// Store instructions that were predicated.<br>
+ SmallVector<Instruction *, 4> PredicatedInstructions;<br>
EdgeMaskCacheTy EdgeMaskCache;<br>
BlockMaskCacheTy BlockMaskCache;<br>
/// Trip count of the original loop.<br>
@@ -816,6 +704,8 @@ protected:<br>
// Holds the end values for each induction variable. We save the end values<br>
// so we can later fix-up the external users of the induction variables.<br>
DenseMap<PHINode *, Value *> IVEndValues;<br>
+<br>
+ friend class LoopVectorizationPlanner;<br>
};<br>
<br>
class InnerLoopUnroller : public InnerLoopVectorizer {<br>
@@ -831,7 +721,6 @@ public:<br>
UnrollFactor, LVL, CM) {}<br>
<br>
private:<br>
- void vectorizeMemoryInstruction(Instruction *Instr) override;<br>
Value *getBroadcastInstrs(Value *V) override;<br>
Value *getStepVector(Value *Val, int StartIdx, Value *Step,<br>
Instruction::BinaryOps Opcode =<br>
@@ -908,6 +797,10 @@ void InnerLoopVectorizer::addMetadata(Ar<br>
}<br>
}<br>
<br>
+} // namespace llvm<br>
+<br>
+namespace {<br>
+<br>
/// \brief The group of interleaved loads/stores sharing the same stride and<br>
/// close to each other.<br>
///<br>
@@ -1920,6 +1813,7 @@ public:<br>
/// \returns True if it is more profitable to scalarize instruction \p I for<br>
/// vectorization factor \p VF.<br>
bool isProfitableToScalarize(Instruction *I, unsigned VF) const {<br>
+ assert(VF > 1 && "Profitable to scalarize relevant only for VF > 1.");<br>
auto Scalars = InstsToScalarize.find(VF);<br>
assert(Scalars != InstsToScalarize.end() &&<br>
"VF not yet analyzed for scalarization profitability");<br>
@@ -2033,6 +1927,22 @@ public:<br>
return Legal->isInductionVariable(Op);<br>
}<br>
<br>
+ /// Collects the instructions to scalarize for each predicated instruction in<br>
+ /// the loop.<br>
+ void collectInstsToScalarize(unsigned VF);<br>
+<br>
+ /// Collect Uniform and Scalar values for the given \p VF.<br>
+ /// The sets depend on CM decision for Load/Store instructions<br>
+ /// that may be vectorized as interleave, gather-scatter or scalarized.<br>
+ void collectUniformsAndScalars(unsigned VF) {<br>
+ // Do the analysis once.<br>
+ if (VF == 1 || Uniforms.count(VF))<br>
+ return;<br>
+ setCostBasedWideningDecision(VF);<br>
+ collectLoopUniforms(VF);<br>
+ collectLoopScalars(VF);<br>
+ }<br>
+<br>
private:<br>
/// \return An upper bound for the vectorization factor, larger than zero.<br>
/// One is returned if vectorization should best be avoided due to cost.<br>
@@ -2134,10 +2044,6 @@ private:<br>
int computePredInstDiscount(Instruction *PredInst, ScalarCostsTy &ScalarCosts,<br>
unsigned VF);<br>
<br>
- /// Collects the instructions to scalarize for each predicated instruction in<br>
- /// the loop.<br>
- void collectInstsToScalarize(unsigned VF);<br>
-<br>
/// Collect the instructions that are uniform after vectorization. An<br>
/// instruction is uniform if we represent it with a single scalar value in<br>
/// the vectorized loop corresponding to each vector iteration. Examples of<br>
@@ -2156,18 +2062,6 @@ private:<br>
/// iteration of the original scalar loop.<br>
void collectLoopScalars(unsigned VF);<br>
<br>
- /// Collect Uniform and Scalar values for the given \p VF.<br>
- /// The sets depend on CM decision for Load/Store instructions<br>
- /// that may be vectorized as interleave, gather-scatter or scalarized.<br>
- void collectUniformsAndScalars(unsigned VF) {<br>
- // Do the analysis once.<br>
- if (VF == 1 || Uniforms.count(VF))<br>
- return;<br>
- setCostBasedWideningDecision(VF);<br>
- collectLoopUniforms(VF);<br>
- collectLoopScalars(VF);<br>
- }<br>
-<br>
/// Keeps cost model vectorization decision and cost for instructions.<br>
/// Right now it is used for memory instructions only.<br>
typedef DenseMap<std::pair<Instruction *, unsigned>,<br>
@@ -2205,23 +2099,71 @@ public:<br>
SmallPtrSet<const Value *, 16> VecValuesToIgnore;<br>
};<br>
<br>
+} // end anonymous namespace<br>
+<br>
+namespace llvm {<br>
+/// InnerLoopVectorizer vectorizes loops which contain only one basic<br>
/// LoopVectorizationPlanner - drives the vectorization process after having<br>
/// passed Legality checks.<br>
+/// The planner builds and optimizes the Vectorization Plans which record the<br>
+/// decisions how to vectorize the given loop. In particular, represent the<br>
+/// control-flow of the vectorized version, the replication of instructions that<br>
+/// are to be scalarized, and interleave access groups.<br>
class LoopVectorizationPlanner {<br>
+ /// The loop that we evaluate.<br>
+ Loop *OrigLoop;<br>
+<br>
+ /// Loop Info analysis.<br>
+ LoopInfo *LI;<br>
+<br>
+ /// Target Library Info.<br>
+ const TargetLibraryInfo *TLI;<br>
+<br>
+ /// Target Transform Info.<br>
+ const TargetTransformInfo *TTI;<br>
+<br>
+ /// The legality analysis.<br>
+ LoopVectorizationLegality *Legal;<br>
+<br>
+ /// The profitablity analysis.<br>
+ LoopVectorizationCostModel &CM;<br>
+<br>
+ SmallVector<VPlan *, 4> VPlans;<br>
+<br>
+ unsigned BestVF;<br>
+ unsigned BestUF;<br>
+<br>
public:<br>
- LoopVectorizationPlanner(Loop *OrigLoop, LoopInfo *LI,<br>
+ LoopVectorizationPlanner(Loop *L, LoopInfo *LI, const TargetLibraryInfo *TLI,<br>
+ const TargetTransformInfo *TTI,<br>
LoopVectorizationLegality *Legal,<br>
LoopVectorizationCostModel &CM)<br>
- : OrigLoop(OrigLoop), LI(LI), Legal(Legal), CM(CM) {}<br>
+ : OrigLoop(L), LI(LI), TLI(TLI), TTI(TTI), Legal(Legal), CM(CM),<br>
+ BestVF(0), BestUF(0) {}<br>
<br>
- ~LoopVectorizationPlanner() {}<br>
+ ~LoopVectorizationPlanner() {<br>
+ while (!VPlans.empty()) {<br>
+ VPlan *Plan = VPlans.back();<br>
+ VPlans.pop_back();<br>
+ delete Plan;<br>
+ }<br>
+ }<br>
<br>
/// Plan how to best vectorize, return the best VF and its cost.<br>
LoopVectorizationCostModel::VectorizationFactor plan(bool OptForSize,<br>
unsigned UserVF);<br>
<br>
- /// Generate the IR code for the vectorized loop.<br>
- void executePlan(InnerLoopVectorizer &ILV);<br>
+ /// Finalize the best decision and dispose of all other VPlans.<br>
+ void setBestPlan(unsigned VF, unsigned UF);<br>
+<br>
+ /// Generate the IR code for the body of the vectorized loop according to the<br>
+ /// best selected VPlan.<br>
+ void executePlan(InnerLoopVectorizer &LB, DominatorTree *DT);<br>
+<br>
+ void printPlans(raw_ostream &O) {<br>
+ for (VPlan *Plan : VPlans)<br>
+ O << *Plan;<br>
+ }<br>
<br>
protected:<br>
/// Collect the instructions from the original loop that would be trivially<br>
@@ -2229,20 +2171,77 @@ protected:<br>
void collectTriviallyDeadInstructions(<br>
SmallPtrSetImpl<Instruction *> &DeadInstructions);<br>
<br>
-private:<br>
- /// The loop that we evaluate.<br>
- Loop *OrigLoop;<br>
+ /// A range of powers-of-2 vectorization factors with fixed start and<br>
+ /// adjustable end. The range includes start and excludes end, e.g.,:<br>
+ /// [1, 9) = {1, 2, 4, 8}<br>
+ struct VFRange {<br>
+ const unsigned Start; // A power of 2.<br>
+ unsigned End; // Need not be a power of 2. If End <= Start range is empty.<br>
+ };<br>
<br>
- /// Loop Info analysis.<br>
- LoopInfo *LI;<br>
+ /// Test a \p Predicate on a \p Range of VF's. Return the value of applying<br>
+ /// \p Predicate on Range.Start, possibly decreasing Range.End such that the<br>
+ /// returned value holds for the entire \p Range.<br>
+ bool getDecisionAndClampRange(const std::function<bool(unsigned)> &Predicate,<br>
+ VFRange &Range);<br>
+<br>
+ /// Build VPlans for power-of-2 VF's between \p MinVF and \p MaxVF inclusive,<br>
+ /// according to the information gathered by Legal when it checked if it is<br>
+ /// legal to vectorize the loop.<br>
+ void buildVPlans(unsigned MinVF, unsigned MaxVF);<br>
<br>
- /// The legality analysis.<br>
- LoopVectorizationLegality *Legal;<br>
-<br>
- /// The profitablity analysis.<br>
- LoopVectorizationCostModel &CM;<br>
+private:<br>
+ /// Check if \I belongs to an Interleave Group within the given VF \p Range,<br>
+ /// \return true in the first returned value if so and false otherwise.<br>
+ /// Build a new VPInterleaveGroup Recipe if \I is the primary member of an IG<br>
+ /// for \p Range.Start, and provide it as the second returned value.<br>
+ /// Note that if \I is an adjunct member of an IG for \p Range.Start, the<br>
+ /// \return value is <true, nullptr>, as it is handled by another recipe.<br>
+ /// \p Range.End may be decreased to ensure same decision from \p Range.Start<br>
+ /// to \p Range.End.<br>
+ VPInterleaveRecipe *tryToInterleaveMemory(Instruction *I, VFRange &Range);<br>
+<br>
+ /// Check if an induction recipe should be constructed for \I within the given<br>
+ /// VF \p Range. If so build and return it. If not, return null. \p Range.End<br>
+ /// may be decreased to ensure same decision from \p Range.Start to<br>
+ /// \p Range.End.<br>
+ VPWidenIntOrFpInductionRecipe *tryToOptimizeInduction(Instruction *I,<br>
+ VFRange &Range);<br>
+<br>
+ /// Check if \I can be widened within the given VF \p Range. If \I can be<br>
+ /// widened for Range.Start, extend \p LastWidenRecipe to include \p I if<br>
+ /// possible or else build a new VPWidenRecipe for it, and return the<br>
+ /// VPWidenRecipe that includes \p I. If \p I cannot be widened for<br>
+ /// Range.Start \return null. Range.End may be decreased to ensure same<br>
+ /// decision from \p Range.Start to \p Range.End.<br>
+ VPWidenRecipe *tryToWiden(Instruction *I, VPWidenRecipe *LastWidenRecipe,<br>
+ VFRange &Range);<br>
+<br>
+ /// Build a VPReplicationRecipe for \p I and enclose it within a Region if it<br>
+ /// is predicated. \return \p VPBB augmented with this new recipe if \p I is<br>
+ /// not predicated, otherwise \return a new VPBasicBlock that succeeds the new<br>
+ /// Region. Update the packing decision of predicated instructions if they<br>
+ /// feed \p I. Range.End may be decreased to ensure same recipe behavior from<br>
+ /// \p Range.Start to \p Range.End.<br>
+ VPBasicBlock *handleReplication(<br>
+ Instruction *I, VFRange &Range, VPBasicBlock *VPBB,<br>
+ DenseMap<Instruction *, VPReplicateRecipe *> &PredInst2Recipe);<br>
+<br>
+ /// Create a replicating region for instruction \p I that requires<br>
+ /// predication. \p PredRecipe is a VPReplicateRecipe holding \p I.<br>
+ VPRegionBlock *createReplicateRegion(Instruction *I,<br>
+ VPRecipeBase *PredRecipe);<br>
+<br>
+ /// Build a VPlan according to the information gathered by Legal. \return a<br>
+ /// VPlan for vectorization factors \p Range.Start and up to \p Range.End<br>
+ /// exclusive, possibly decreasing \p Range.End.<br>
+ VPlan *buildVPlan(VFRange &Range);<br>
};<br>
<br>
+} // namespace llvm<br>
+<br>
+namespace {<br>
+<br>
/// \brief This holds vectorization requirements that must be verified late in<br>
/// the process. The requirements are set by legalize and costmodel. Once<br>
/// vectorization has been determined to be possible and profitable the<br>
@@ -2663,15 +2662,15 @@ void InnerLoopVectorizer::buildScalarSte<br>
// iteration. If EntryVal is uniform, we only need to generate the first<br>
// lane. Otherwise, we generate all VF values.<br>
unsigned Lanes =<br>
- Cost->isUniformAfterVectorization(cast<Instruction>(EntryVal), VF) ? 1 : VF;<br>
-<br>
+ Cost->isUniformAfterVectorization(cast<Instruction>(EntryVal), VF) ? 1<br>
+ : VF;<br>
// Compute the scalar steps and save the results in VectorLoopValueMap.<br>
for (unsigned Part = 0; Part < UF; ++Part) {<br>
for (unsigned Lane = 0; Lane < Lanes; ++Lane) {<br>
auto *StartIdx = getSignedIntOrFpConstant(ScalarIVTy, VF * Part + Lane);<br>
auto *Mul = addFastMathFlag(Builder.CreateBinOp(MulOp, StartIdx, Step));<br>
auto *Add = addFastMathFlag(Builder.CreateBinOp(AddOp, ScalarIV, Mul));<br>
- VectorLoopValueMap.setScalarValue(EntryVal, Part, Lane, Add);<br>
+ VectorLoopValueMap.setScalarValue(EntryVal, {Part, Lane}, Add);<br>
}<br>
}<br>
}<br>
@@ -2709,7 +2708,7 @@ Value *InnerLoopVectorizer::getOrCreateV<br>
// vector form, we will construct the vector values on demand.<br>
if (VectorLoopValueMap.hasAnyScalarValue(V)) {<br>
<br>
- Value *ScalarValue = VectorLoopValueMap.getScalarValue(V, Part, 0);<br>
+ Value *ScalarValue = VectorLoopValueMap.getScalarValue(V, {Part, 0});<br>
<br>
// If we've scalarized a value, that value should be an instruction.<br>
auto *I = cast<Instruction>(V);<br>
@@ -2726,8 +2725,8 @@ Value *InnerLoopVectorizer::getOrCreateV<br>
// of the Part unroll iteration. Otherwise, the last instruction is the one<br>
// we created for the last vector lane of the Part unroll iteration.<br>
unsigned LastLane = Cost->isUniformAfterVectorization(I, VF) ? 0 : VF - 1;<br>
- auto *LastInst =<br>
- cast<Instruction>(VectorLoopValueMap.getScalarValue(V, Part, LastLane));<br>
+ auto *LastInst = cast<Instruction>(<br>
+ VectorLoopValueMap.getScalarValue(V, {Part, LastLane}));<br>
<br>
// Set the insert point after the last scalarized instruction. This ensures<br>
// the insertelement sequence will directly follow the scalar definitions.<br>
@@ -2744,14 +2743,15 @@ Value *InnerLoopVectorizer::getOrCreateV<br>
Value *VectorValue = nullptr;<br>
if (Cost->isUniformAfterVectorization(I, VF)) {<br>
VectorValue = getBroadcastInstrs(ScalarValue);<br>
+ VectorLoopValueMap.setVectorValue(V, Part, VectorValue);<br>
} else {<br>
- VectorValue = UndefValue::get(VectorType::get(V->getType(), VF));<br>
+ // Initialize packing with insertelements to start from undef.<br>
+ Value *Undef = UndefValue::get(VectorType::get(V->getType(), VF));<br>
+ VectorLoopValueMap.setVectorValue(V, Part, Undef);<br>
for (unsigned Lane = 0; Lane < VF; ++Lane)<br>
- VectorValue = Builder.CreateInsertElement(<br>
- VectorValue, getOrCreateScalarValue(V, Part, Lane),<br>
- Builder.getInt32(Lane));<br>
+ packScalarIntoVectorValue(V, {Part, Lane});<br>
+ VectorValue = VectorLoopValueMap.getVectorValue(V, Part);<br>
}<br>
- VectorLoopValueMap.setVectorValue(V, Part, VectorValue);<br>
Builder.restoreIP(OldIP);<br>
return VectorValue;<br>
}<br>
@@ -2763,28 +2763,29 @@ Value *InnerLoopVectorizer::getOrCreateV<br>
return B;<br>
}<br>
<br>
-Value *InnerLoopVectorizer::getOrCreateScalarValue(Value *V, unsigned Part,<br>
- unsigned Lane) {<br>
-<br>
+Value *<br>
+InnerLoopVectorizer::getOrCreateScalarValue(Value *V,<br>
+ const VPIteration &Instance) {<br>
// If the value is not an instruction contained in the loop, it should<br>
// already be scalar.<br>
if (OrigLoop->isLoopInvariant(V))<br>
return V;<br>
<br>
- assert(Lane > 0 ? !Cost->isUniformAfterVectorization(cast<Instruction>(V), VF)<br>
- : true && "Uniform values only have lane zero");<br>
+ assert(Instance.Lane > 0<br>
+ ? !Cost->isUniformAfterVectorization(cast<Instruction>(V), VF)<br>
+ : true && "Uniform values only have lane zero");<br>
<br>
// If the value from the original loop has not been vectorized, it is<br>
// represented by UF x VF scalar values in the new loop. Return the requested<br>
// scalar value.<br>
- if (VectorLoopValueMap.hasScalarValue(V, Part, Lane))<br>
- return VectorLoopValueMap.getScalarValue(V, Part, Lane);<br>
+ if (VectorLoopValueMap.hasScalarValue(V, Instance))<br>
+ return VectorLoopValueMap.getScalarValue(V, Instance);<br>
<br>
// If the value has not been scalarized, get its entry in VectorLoopValueMap<br>
// for the given unroll part. If this entry is not a vector type (i.e., the<br>
// vectorization factor is one), there is no need to generate an<br>
// extractelement instruction.<br>
- auto *U = getOrCreateVectorValue(V, Part);<br>
+ auto *U = getOrCreateVectorValue(V, Instance.Part);<br>
if (!U->getType()->isVectorTy()) {<br>
assert(VF == 1 && "Value not scalarized has non-vector type");<br>
return U;<br>
@@ -2793,7 +2794,20 @@ Value *InnerLoopVectorizer::getOrCreateS<br>
// Otherwise, the value from the original loop has been vectorized and is<br>
// represented by UF vector values. Extract and return the requested scalar<br>
// value from the appropriate vector lane.<br>
- return Builder.CreateExtractElement(U, Builder.getInt32(Lane));<br>
+ return Builder.CreateExtractElement(U, Builder.getInt32(Instance.Lane));<br>
+}<br>
+<br>
+void InnerLoopVectorizer::packScalarIntoVectorValue(<br>
+ Value *V, const VPIteration &Instance) {<br>
+ assert(V != Induction && "The new induction variable should not be used.");<br>
+ assert(!V->getType()->isVectorTy() && "Can't pack a vector");<br>
+ assert(!V->getType()->isVoidTy() && "Type does not produce a value");<br>
+<br>
+ Value *ScalarInst = VectorLoopValueMap.getScalarValue(V, Instance);<br>
+ Value *VectorValue = VectorLoopValueMap.getVectorValue(V, Instance.Part);<br>
+ VectorValue = Builder.CreateInsertElement(VectorValue, ScalarInst,<br>
+ Builder.getInt32(Instance.Lane));<br>
+ VectorLoopValueMap.resetVectorValue(V, Instance.Part, VectorValue);<br>
}<br>
<br>
Value *InnerLoopVectorizer::reverseVector(Value *Vec) {<br>
@@ -2866,7 +2880,7 @@ void InnerLoopVectorizer::vectorizeInter<br>
Index += (VF - 1) * Group->getFactor();<br>
<br>
for (unsigned Part = 0; Part < UF; Part++) {<br>
- Value *NewPtr = getOrCreateScalarValue(Ptr, Part, 0);<br>
+ Value *NewPtr = getOrCreateScalarValue(Ptr, {Part, 0});<br>
<br>
// Notice current instruction could be any index. Need to adjust the address<br>
// to the member of index 0.<br>
@@ -2992,10 +3006,6 @@ void InnerLoopVectorizer::vectorizeMemor<br>
Alignment = DL.getABITypeAlignment(ScalarDataTy);<br>
unsigned AddressSpace = getMemInstAddressSpace(Instr);<br>
<br>
- // Scalarize the memory instruction if necessary.<br>
- if (Decision == LoopVectorizationCostModel::CM_Scalarize)<br>
- return scalarizeInstruction(Instr, Legal->isScalarWithPredication(Instr));<br>
-<br>
// Determine if the pointer operand of the access is either consecutive or<br>
// reverse consecutive.<br>
int ConsecutiveStride = Legal->isConsecutivePtr(Ptr);<br>
@@ -3010,7 +3020,7 @@ void InnerLoopVectorizer::vectorizeMemor<br>
<br>
// Handle consecutive loads/stores.<br>
if (ConsecutiveStride)<br>
- Ptr = getOrCreateScalarValue(Ptr, 0, 0);<br>
+ Ptr = getOrCreateScalarValue(Ptr, {0, 0});<br>
<br>
VectorParts Mask = createBlockInMask(Instr->getParent());<br>
// Handle Stores:<br>
@@ -3107,71 +3117,41 @@ void InnerLoopVectorizer::vectorizeMemor<br>
}<br>
<br>
void InnerLoopVectorizer::scalarizeInstruction(Instruction *Instr,<br>
+ const VPIteration &Instance,<br>
bool IfPredicateInstr) {<br>
assert(!Instr->getType()->isAggregateType() && "Can't handle vectors");<br>
- DEBUG(dbgs() << "LV: Scalarizing"<br>
- << (IfPredicateInstr ? " and predicating:" : ":") << *Instr<br>
- << '\n');<br>
- // Holds vector parameters or scalars, in case of uniform vals.<br>
- SmallVector<VectorParts, 4> Params;<br>
<br>
setDebugLocFromInst(Builder, Instr);<br>
<br>
// Does this instruction return a value ?<br>
bool IsVoidRetTy = Instr->getType()->isVoidTy();<br>
<br>
- VectorParts Cond;<br>
- if (IfPredicateInstr)<br>
- Cond = createBlockInMask(Instr->getParent());<br>
-<br>
- // Determine the number of scalars we need to generate for each unroll<br>
- // iteration. If the instruction is uniform, we only need to generate the<br>
- // first lane. Otherwise, we generate all VF values.<br>
- unsigned Lanes = Cost->isUniformAfterVectorization(Instr, VF) ? 1 : VF;<br>
-<br>
- // For each vector unroll 'part':<br>
- for (unsigned Part = 0; Part < UF; ++Part) {<br>
- // For each scalar that we create:<br>
- for (unsigned Lane = 0; Lane < Lanes; ++Lane) {<br>
+ Instruction *Cloned = Instr->clone();<br>
+ if (!IsVoidRetTy)<br>
+ Cloned->setName(Instr->getName() + ".cloned");<br>
+<br>
+ // Replace the operands of the cloned instructions with their scalar<br>
+ // equivalents in the new loop.<br>
+ for (unsigned op = 0, e = Instr->getNumOperands(); op != e; ++op) {<br>
+ auto *NewOp = getOrCreateScalarValue(Instr->getOperand(op), Instance);<br>
+ Cloned->setOperand(op, NewOp);<br>
+ }<br>
+ addNewMetadata(Cloned, Instr);<br>
+<br>
+ // Place the cloned scalar in the new loop.<br>
+ Builder.Insert(Cloned);<br>
+<br>
+ // Add the cloned scalar to the scalar map entry.<br>
+ VectorLoopValueMap.setScalarValue(Instr, Instance, Cloned);<br>
+<br>
+ // If we just cloned a new assumption, add it the assumption cache.<br>
+ if (auto *II = dyn_cast<IntrinsicInst>(Cloned))<br>
+ if (II->getIntrinsicID() == Intrinsic::assume)<br>
+ AC->registerAssumption(II);<br>
<br>
- // Start if-block.<br>
- Value *Cmp = nullptr;<br>
- if (IfPredicateInstr) {<br>
- Cmp = Cond[Part];<br>
- if (!Cmp) // Block in mask is all-one.<br>
- Cmp = Builder.getTrue();<br>
- else if (Cmp->getType()->isVectorTy())<br>
- Cmp = Builder.CreateExtractElement(Cmp, Builder.getInt32(Lane));<br>
- }<br>
-<br>
- Instruction *Cloned = Instr->clone();<br>
- if (!IsVoidRetTy)<br>
- Cloned->setName(Instr->getName() + ".cloned");<br>
-<br>
- // Replace the operands of the cloned instructions with their scalar<br>
- // equivalents in the new loop.<br>
- for (unsigned op = 0, e = Instr->getNumOperands(); op != e; ++op) {<br>
- auto *NewOp = getOrCreateScalarValue(Instr->getOperand(op), Part, Lane);<br>
- Cloned->setOperand(op, NewOp);<br>
- }<br>
- addNewMetadata(Cloned, Instr);<br>
-<br>
- // Place the cloned scalar in the new loop.<br>
- Builder.Insert(Cloned);<br>
-<br>
- // Add the cloned scalar to the scalar map entry.<br>
- VectorLoopValueMap.setScalarValue(Instr, Part, Lane, Cloned);<br>
-<br>
- // If we just cloned a new assumption, add it the assumption cache.<br>
- if (auto *II = dyn_cast<IntrinsicInst>(Cloned))<br>
- if (II->getIntrinsicID() == Intrinsic::assume)<br>
- AC->registerAssumption(II);<br>
-<br>
- // End if-block.<br>
- if (IfPredicateInstr)<br>
- PredicatedInstructions.push_back(std::make_pair(Cloned, Cmp));<br>
- }<br>
- }<br>
+ // End if-block.<br>
+ if (IfPredicateInstr)<br>
+ PredicatedInstructions.push_back(Cloned);<br>
}<br>
<br>
PHINode *InnerLoopVectorizer::createInductionVariable(Loop *L, Value *Start,<br>
@@ -3375,7 +3355,7 @@ void InnerLoopVectorizer::emitMemRuntime<br>
LVer->prepareNoAliasMetadata();<br>
}<br>
<br>
-void InnerLoopVectorizer::createVectorizedLoopSkeleton() {<br>
+BasicBlock *InnerLoopVectorizer::createVectorizedLoopSkeleton() {<br>
/*<br>
In this function we generate a new loop. The new loop will contain<br>
the vectorized instructions while the old loop will continue to run the<br>
@@ -3556,6 +3536,8 @@ void InnerLoopVectorizer::createVectoriz<br>
<br>
LoopVectorizeHints Hints(Lp, true, *ORE);<br>
Hints.setAlreadyVectorized();<br>
+<br>
+ return LoopVectorPreHeader;<br>
}<br>
<br>
// Fix up external users of the induction variable. At this point, we are<br>
@@ -3929,7 +3911,8 @@ void InnerLoopVectorizer::fixVectorizedL<br>
IVEndValues[Entry.first], LoopMiddleBlock);<br>
<br>
fixLCSSAPHIs();<br>
- predicateInstructions();<br>
+ for (Instruction *PI : PredicatedInstructions)<br>
+ sinkScalarOperands(&*PI);<br>
<br>
// Remove redundant induction instructions.<br>
cse(LoopVectorBody);<br>
@@ -4384,126 +4367,6 @@ void InnerLoopVectorizer::sinkScalarOper<br>
} while (Changed);<br>
}<br>
<br>
-void InnerLoopVectorizer::predicateInstructions() {<br>
-<br>
- // For each instruction I marked for predication on value C, split I into its<br>
- // own basic block to form an if-then construct over C. Since I may be fed by<br>
- // an extractelement instruction or other scalar operand, we try to<br>
- // iteratively sink its scalar operands into the predicated block. If I feeds<br>
- // an insertelement instruction, we try to move this instruction into the<br>
- // predicated block as well. For non-void types, a phi node will be created<br>
- // for the resulting value (either vector or scalar).<br>
- //<br>
- // So for some predicated instruction, e.g. the conditional sdiv in:<br>
- //<br>
- // for.body:<br>
- // ...<br>
- // %add = add nsw i32 %mul, %0<br>
- // %cmp5 = icmp sgt i32 %2, 7<br>
- // br i1 %cmp5, label %if.then, label %if.end<br>
- //<br>
- // if.then:<br>
- // %div = sdiv i32 %0, %1<br>
- // br label %if.end<br>
- //<br>
- // if.end:<br>
- // %x.0 = phi i32 [ %div, %if.then ], [ %add, %for.body ]<br>
- //<br>
- // the sdiv at this point is scalarized and if-converted using a select.<br>
- // The inactive elements in the vector are not used, but the predicated<br>
- // instruction is still executed for all vector elements, essentially:<br>
- //<br>
- // vector.body:<br>
- // ...<br>
- // %17 = add nsw <2 x i32> %16, %wide.load<o:p></o:p></p>
<p class="MsoNormal">- // %29 = extractelement <2 x i32> %wide.load, i32 0<br>
- // %30 = extractelement <2 x i32> %wide.load51, i32 0<br>
- // %31 = sdiv i32 %29, %30<br>
- // %32 = insertelement <2 x i32> undef, i32 %31, i32 0<br>
- // %35 = extractelement <2 x i32> %wide.load, i32 1<br>
- // %36 = extractelement <2 x i32> %wide.load51, i32 1<br>
- // %37 = sdiv i32 %35, %36<br>
- // %38 = insertelement <2 x i32> %32, i32 %37, i32 1<br>
- // %predphi = select <2 x i1> %26, <2 x i32> %38, <2 x i32> %17<br>
- //<br>
- // Predication will now re-introduce the original control flow to avoid false<br>
- // side-effects by the sdiv instructions on the inactive elements, yielding<br>
- // (after cleanup):<br>
- //<br>
- // vector.body:<br>
- // ...<br>
- // %5 = add nsw <2 x i32> %4, %wide.load<br>
- // %8 = icmp sgt <2 x i32> %wide.load52, <i32 7, i32 7><br>
- // %9 = extractelement <2 x i1> %8, i32 0<br>
- // br i1 %9, label %pred.sdiv.if, label %pred.sdiv.continue<br>
- //<br>
- // pred.sdiv.if:<br>
- // %10 = extractelement <2 x i32> %wide.load, i32 0<br>
- // %11 = extractelement <2 x i32> %wide.load51, i32 0<br>
- // %12 = sdiv i32 %10, %11<br>
- // %13 = insertelement <2 x i32> undef, i32 %12, i32 0<br>
- // br label %pred.sdiv.continue<br>
- //<br>
- // pred.sdiv.continue:<br>
- // %14 = phi <2 x i32> [ undef, %vector.body ], [ %13, %pred.sdiv.if ]<br>
- // %15 = extractelement <2 x i1> %8, i32 1<br>
- // br i1 %15, label %pred.sdiv.if54, label %pred.sdiv.continue55<br>
- //<br>
- // pred.sdiv.if54:<br>
- // %16 = extractelement <2 x i32> %wide.load, i32 1<br>
- // %17 = extractelement <2 x i32> %wide.load51, i32 1<br>
- // %18 = sdiv i32 %16, %17<br>
- // %19 = insertelement <2 x i32> %14, i32 %18, i32 1<br>
- // br label %pred.sdiv.continue55<br>
- //<br>
- // pred.sdiv.continue55:<br>
- // %20 = phi <2 x i32> [ %14, %pred.sdiv.continue ], [ %19, %pred.sdiv.if54 ]<br>
- // %predphi = select <2 x i1> %8, <2 x i32> %20, <2 x i32> %5<br>
-<br>
- for (auto KV : PredicatedInstructions) {<br>
- BasicBlock::iterator I(KV.first);<br>
- BasicBlock *Head = I->getParent();<br>
- auto *T = SplitBlockAndInsertIfThen(KV.second, &*I, /*Unreachable=*/false,<br>
- /*BranchWeights=*/nullptr, DT, LI);<br>
- I->moveBefore(T);<br>
- sinkScalarOperands(&*I);<br>
-<br>
- BasicBlock *PredicatedBlock = I->getParent();<br>
- Twine BBNamePrefix = Twine("pred.") + I->getOpcodeName();<br>
- PredicatedBlock->setName(BBNamePrefix + ".if");<br>
- PredicatedBlock->getSingleSuccessor()->setName(BBNamePrefix + ".continue");<br>
-<br>
- // If the instruction is non-void create a Phi node at reconvergence point.<br>
- if (!I->getType()->isVoidTy()) {<br>
- Value *IncomingTrue = nullptr;<br>
- Value *IncomingFalse = nullptr;<br>
-<br>
- if (I->hasOneUse() && isa<InsertElementInst>(*I->user_begin())) {<br>
- // If the predicated instruction is feeding an insert-element, move it<br>
- // into the Then block; Phi node will be created for the vector.<br>
- InsertElementInst *IEI = cast<InsertElementInst>(*I->user_begin());<br>
- IEI->moveBefore(T);<br>
- IncomingTrue = IEI; // the new vector with the inserted element.<br>
- IncomingFalse = IEI->getOperand(0); // the unmodified vector<br>
- } else {<br>
- // Phi node will be created for the scalar predicated instruction.<br>
- IncomingTrue = &*I;<br>
- IncomingFalse = UndefValue::get(I->getType());<br>
- }<br>
-<br>
- BasicBlock *PostDom = I->getParent()->getSingleSuccessor();<br>
- assert(PostDom && "Then block has multiple successors");<br>
- PHINode *Phi =<br>
- PHINode::Create(IncomingTrue->getType(), 2, "", &PostDom->front());<br>
- IncomingTrue->replaceAllUsesWith(Phi);<br>
- Phi->addIncoming(IncomingFalse, Head);<br>
- Phi->addIncoming(IncomingTrue, I->getParent());<br>
- }<br>
- }<br>
-<br>
- DEBUG(DT->verifyDomTree());<br>
-}<br>
-<br>
InnerLoopVectorizer::VectorParts<br>
InnerLoopVectorizer::createEdgeMask(BasicBlock *Src, BasicBlock *Dst) {<br>
assert(is_contained(predecessors(Dst), Src) && "Invalid edge");<br>
@@ -4647,7 +4510,7 @@ void InnerLoopVectorizer::widenPHIInstru<br>
llvm_unreachable("Unknown induction");<br>
case InductionDescriptor::IK_IntInduction:<br>
case InductionDescriptor::IK_FpInduction:<br>
- return widenIntOrFpInduction(P);<br>
+ llvm_unreachable("Integer/fp induction is handled elsewhere.");<br>
case InductionDescriptor::IK_PtrInduction: {<br>
// Handle the pointer induction variable case.<br>
assert(P->getType()->isPointerTy() && "Unexpected type.");<br>
@@ -4666,7 +4529,7 @@ void InnerLoopVectorizer::widenPHIInstru<br>
Value *GlobalIdx = Builder.CreateAdd(PtrInd, Idx);<br>
Value *SclrGep = II.transform(Builder, GlobalIdx, PSE.getSE(), DL);<br>
SclrGep->setName("next.gep");<br>
- VectorLoopValueMap.setScalarValue(P, Part, Lane, SclrGep);<br>
+ VectorLoopValueMap.setScalarValue(P, {Part, Lane}, SclrGep);<br>
}<br>
}<br>
return;<br>
@@ -4692,25 +4555,11 @@ static bool mayDivideByZero(Instruction<br>
return !CInt || CInt->isZero();<br>
}<br>
<br>
-void InnerLoopVectorizer::vectorizeInstruction(Instruction &I) {<br>
- // Scalarize instructions that should remain scalar after vectorization.<br>
- if (VF > 1 &&<br>
- !(isa<BranchInst>(&I) || isa<PHINode>(&I) || isa<DbgInfoIntrinsic>(&I)) &&<br>
- shouldScalarizeInstruction(&I)) {<br>
- scalarizeInstruction(&I, Legal->isScalarWithPredication(&I));<br>
- return;<br>
- }<br>
-<br>
+void InnerLoopVectorizer::widenInstruction(Instruction &I) {<br>
switch (I.getOpcode()) {<br>
case Instruction::Br:<br>
- // Nothing to do for PHIs and BR, since we already took care of the<br>
- // loop control flow instructions.<br>
- break;<br>
- case Instruction::PHI: {<br>
- // Vectorize PHINodes.<br>
- widenPHIInstruction(&I, UF, VF);<br>
- break;<br>
- } // End of PHI.<br>
+ case Instruction::PHI:<br>
+ llvm_unreachable("This instruction is handled by a different recipe.");<br>
case Instruction::GetElementPtr: {<br>
// Construct a vector GEP by widening the operands of the scalar GEP as<br>
// necessary. We mark the vector GEP 'inbounds' if appropriate. A GEP<br>
@@ -4783,13 +4632,6 @@ void InnerLoopVectorizer::vectorizeInstr<br>
case Instruction::SDiv:<br>
case Instruction::SRem:<br>
case Instruction::URem:<br>
- // Scalarize with predication if this instruction may divide by zero and<br>
- // block execution is conditional, otherwise fallthrough.<br>
- if (Legal->isScalarWithPredication(&I)) {<br>
- scalarizeInstruction(&I, true);<br>
- break;<br>
- }<br>
- LLVM_FALLTHROUGH;<br>
case Instruction::Add:<br>
case Instruction::FAdd:<br>
case Instruction::Sub:<br>
@@ -4837,7 +4679,7 @@ void InnerLoopVectorizer::vectorizeInstr<br>
// We have to take the 'vectorized' value and pick the first lane.<br>
// Instcombine will make this a no-op.<br>
<br>
- auto *ScalarCond = getOrCreateScalarValue(I.getOperand(0), 0, 0);<br>
+ auto *ScalarCond = getOrCreateScalarValue(I.getOperand(0), {0, 0});<br>
<br>
for (unsigned Part = 0; Part < UF; ++Part) {<br>
Value *Cond = getOrCreateVectorValue(I.getOperand(0), Part);<br>
@@ -4896,16 +4738,6 @@ void InnerLoopVectorizer::vectorizeInstr<br>
auto *CI = dyn_cast<CastInst>(&I);<br>
setDebugLocFromInst(Builder, CI);<br>
<br>
- // Optimize the special case where the source is a constant integer<br>
- // induction variable. Notice that we can only optimize the 'trunc' case<br>
- // because (a) FP conversions lose precision, (b) sext/zext may wrap, and<br>
- // (c) other casts depend on pointer size.<br>
- if (Cost->isOptimizableIVTruncate(CI, VF)) {<br>
- widenIntOrFpInduction(cast<PHINode>(CI->getOperand(0)),<br>
- cast<TruncInst>(CI));<br>
- break;<br>
- }<br>
-<br>
/// Vectorize casts.<br>
Type *DestTy =<br>
(VF == 1) ? CI->getType() : VectorType::get(CI->getType(), VF);<br>
@@ -4936,11 +4768,7 @@ void InnerLoopVectorizer::vectorizeInstr<br>
Tys.push_back(ToVectorTy(ArgOperand->getType(), VF));<br>
<br>
Intrinsic::ID ID = getVectorIntrinsicIDForCall(CI, TLI);<br>
- if (ID && (ID == Intrinsic::assume || ID == Intrinsic::lifetime_end ||<br>
- ID == Intrinsic::lifetime_start)) {<br>
- scalarizeInstruction(&I);<br>
- break;<br>
- }<br>
+<br>
// The flag shows whether we use Intrinsic or a usual Call for vectorized<br>
// version of the instruction.<br>
// Is it beneficial to perform intrinsic call compared to lib call?<br>
@@ -4948,10 +4776,8 @@ void InnerLoopVectorizer::vectorizeInstr<br>
unsigned CallCost = getVectorCallCost(CI, VF, *TTI, TLI, NeedToScalarize);<br>
bool UseVectorIntrinsic =<br>
ID && getVectorIntrinsicCost(CI, VF, *TTI, TLI) <= CallCost;<br>
- if (!UseVectorIntrinsic && NeedToScalarize) {<br>
- scalarizeInstruction(&I);<br>
- break;<br>
- }<br>
+ assert((UseVectorIntrinsic || !NeedToScalarize) &&<br>
+ "Instruction should be scalarized elsewhere.");<br>
<br>
for (unsigned Part = 0; Part < UF; ++Part) {<br>
SmallVector<Value *, 4> Args;<br>
@@ -5001,9 +4827,9 @@ void InnerLoopVectorizer::vectorizeInstr<br>
}<br>
<br>
default:<br>
- // All other instructions are unsupported. Scalarize them.<br>
- scalarizeInstruction(&I);<br>
- break;<br>
+ // All other instructions are scalarized.<br>
+ DEBUG(dbgs() << "LV: Found an unhandled instruction: " << I);<br>
+ llvm_unreachable("Unhandled instruction!");<br>
} // end of switch.<br>
}<br>
<br>
@@ -5015,14 +4841,11 @@ void InnerLoopVectorizer::updateAnalysis<br>
assert(DT->properlyDominates(LoopBypassBlocks.front(), LoopExitBlock) &&<br>
"Entry does not dominate exit.");<br>
<br>
- DT->addNewBlock(LI->getLoopFor(LoopVectorBody)->getHeader(),<br>
- LoopVectorPreHeader);<br>
DT->addNewBlock(LoopMiddleBlock,<br>
LI->getLoopFor(LoopVectorBody)->getLoopLatch());<br>
DT->addNewBlock(LoopScalarPreHeader, LoopBypassBlocks[0]);<br>
DT->changeImmediateDominator(LoopScalarBody, LoopScalarPreHeader);<br>
DT->changeImmediateDominator(LoopExitBlock, LoopBypassBlocks[0]);<br>
-<br>
DEBUG(DT->verifyDomTree());<br>
}<br>
<br>
@@ -6962,13 +6785,6 @@ LoopVectorizationCostModel::Vectorizatio<br>
LoopVectorizationCostModel::expectedCost(unsigned VF) {<br>
VectorizationCostTy Cost;<br>
<br>
- // Collect Uniform and Scalar instructions after vectorization with VF.<br>
- collectUniformsAndScalars(VF);<br>
-<br>
- // Collect the instructions (and their associated costs) that will be more<br>
- // profitable to scalarize.<br>
- collectInstsToScalarize(VF);<br>
-<br>
// For each block.<br>
for (BasicBlock *BB : TheLoop->blocks()) {<br>
VectorizationCostTy BlockCost;<br>
@@ -7626,11 +7442,26 @@ LoopVectorizationPlanner::plan(bool OptF<br>
// Collect the instructions (and their associated costs) that will be more<br>
// profitable to scalarize.<br>
CM.selectUserVectorizationFactor(UserVF);<br>
+ buildVPlans(UserVF, UserVF);<br>
+ DEBUG(printPlans(dbgs()));<br>
return {UserVF, 0};<br>
}<br>
<br>
unsigned MaxVF = MaybeMaxVF.getValue();<br>
assert(MaxVF != 0 && "MaxVF is zero.");<br>
+<br>
+ for (unsigned VF = 1; VF <= MaxVF; VF *= 2) {<br>
+ // Collect Uniform and Scalar instructions after vectorization with VF.<br>
+ CM.collectUniformsAndScalars(VF);<br>
+<br>
+ // Collect the instructions (and their associated costs) that will be more<br>
+ // profitable to scalarize.<br>
+ if (VF > 1)<br>
+ CM.collectInstsToScalarize(VF);<br>
+ }<br>
+<br>
+ buildVPlans(1, MaxVF);<br>
+ DEBUG(printPlans(dbgs()));<br>
if (MaxVF == 1)<br>
return NoVectorization;<br>
<br>
@@ -7638,11 +7469,31 @@ LoopVectorizationPlanner::plan(bool OptF<br>
return CM.selectVectorizationFactor(MaxVF);<br>
}<br>
<br>
-void LoopVectorizationPlanner::executePlan(InnerLoopVectorizer &ILV) {<br>
+void LoopVectorizationPlanner::setBestPlan(unsigned VF, unsigned UF) {<br>
+ DEBUG(dbgs() << "Setting best plan to VF=" << VF << ", UF=" << UF << '\n');<br>
+ BestVF = VF;<br>
+ BestUF = UF;<br>
+<br>
+ for (auto *VPlanIter = VPlans.begin(); VPlanIter != VPlans.end();) {<br>
+ VPlan *Plan = *VPlanIter;<br>
+ if (Plan->hasVF(VF))<br>
+ ++VPlanIter;<br>
+ else {<br>
+ VPlanIter = VPlans.erase(VPlanIter);<br>
+ delete Plan;<br>
+ }<br>
+ }<br>
+ assert(VPlans.size() == 1 && "Best VF has not a single VPlan.");<br>
+}<br>
+<br>
+void LoopVectorizationPlanner::executePlan(InnerLoopVectorizer &ILV,<br>
+ DominatorTree *DT) {<br>
// Perform the actual loop transformation.<br>
<br>
// 1. Create a new empty loop. Unlink the old loop and connect the new one.<br>
- ILV.createVectorizedLoopSkeleton();<br>
+ VPTransformState State{<br>
+ BestVF, BestUF, LI, DT, ILV.Builder, ILV.VectorLoopValueMap, &ILV};<br>
+ State.CFG.PrevBB = ILV.createVectorizedLoopSkeleton();<br>
<br>
//===------------------------------------------------===//<br>
//<br>
@@ -7653,36 +7504,9 @@ void LoopVectorizationPlanner::executePl<br>
//===------------------------------------------------===//<br>
<br>
// 2. Copy and widen instructions from the old loop into the new loop.<br>
-<br>
- // Move instructions to handle first-order recurrences.<br>
- DenseMap<Instruction *, Instruction *> &SinkAfter = Legal->getSinkAfter();<br>
- for (auto &Entry : SinkAfter) {<br>
- Entry.first->removeFromParent();<br>
- Entry.first->insertAfter(Entry.second);<br>
- DEBUG(dbgs() << "Sinking" << *Entry.first << " after" << *Entry.second<br>
- << " to vectorize a 1st order recurrence.\n");<br>
- }<br>
-<br>
- // Collect instructions from the original loop that will become trivially dead<br>
- // in the vectorized loop. We don't need to vectorize these instructions. For<br>
- // example, original induction update instructions can become dead because we<br>
- // separately emit induction "steps" when generating code for the new loop.<br>
- // Similarly, we create a new latch condition when setting up the structure<br>
- // of the new loop, so the old one can become dead.<br>
- SmallPtrSet<Instruction *, 4> DeadInstructions;<br>
- collectTriviallyDeadInstructions(DeadInstructions);<br>
-<br>
- // Scan the loop in a topological order to ensure that defs are vectorized<br>
- // before users.<br>
- LoopBlocksDFS DFS(OrigLoop);<br>
- DFS.perform(LI);<br>
-<br>
- // Vectorize all instructions in the original loop that will not become<br>
- // trivially dead when vectorized.<br>
- for (BasicBlock *BB : make_range(DFS.beginRPO(), DFS.endRPO()))<br>
- for (Instruction &I : *BB)<br>
- if (!DeadInstructions.count(&I))<br>
- ILV.vectorizeInstruction(I);<br>
+ assert(VPlans.size() == 1 && "Not a single VPlan to execute.");<br>
+ VPlan *Plan = *VPlans.begin();<br>
+ Plan->execute(&State);<br>
<br>
// 3. Fix the vectorized code: take care of header phi's, live-outs,<br>
// predication, updating analyses.<br>
@@ -7712,14 +7536,6 @@ void LoopVectorizationPlanner::collectTr<br>
DeadInstructions.insert(IndUpdate);<br>
}<br>
}<br>
-<br>
-void InnerLoopUnroller::vectorizeMemoryInstruction(Instruction *Instr) {<br>
- auto *SI = dyn_cast<StoreInst>(Instr);<br>
- bool IfPredicateInstr = (SI && Legal->blockNeedsPredication(SI->getParent()));<br>
-<br>
- return scalarizeInstruction(Instr, IfPredicateInstr);<br>
-}<br>
-<br>
Value *InnerLoopUnroller::reverseVector(Value *Vec) { return Vec; }<br>
<br>
Value *InnerLoopUnroller::getBroadcastInstrs(Value *V) { return V; }<br>
@@ -7775,6 +7591,725 @@ static void AddRuntimeUnrollDisableMetaD<br>
}<br>
}<br>
<br>
+namespace {<br>
+/// VPWidenRecipe is a recipe for producing a copy of vector type for each<br>
+/// Instruction in its ingredients independently, in order. This recipe covers<br>
+/// most of the traditional vectorization cases where each ingredient transforms<br>
+/// into a vectorized version of itself.<br>
+class VPWidenRecipe : public VPRecipeBase {<br>
+private:<br>
+ /// Hold the ingredients by pointing to their original BasicBlock location.<br>
+ BasicBlock::iterator Begin;<br>
+ BasicBlock::iterator End;<br>
+<br>
+public:<br>
+ VPWidenRecipe(Instruction *I) : VPRecipeBase(VPWidenSC) {<br>
+ End = I->getIterator();<br>
+ Begin = End++;<br>
+ }<br>
+<br>
+ ~VPWidenRecipe() {}<br>
+<br>
+ /// Method to support type inquiry through isa, cast, and dyn_cast.<br>
+ static inline bool classof(const VPRecipeBase *V) {<br>
+ return V->getVPRecipeID() == VPRecipeBase::VPWidenSC;<br>
+ }<br>
+<br>
+ /// Produce widened copies of all Ingredients.<br>
+ void execute(VPTransformState &State) override {<br>
+ for (auto &Instr : make_range(Begin, End))<br>
+ State.ILV->widenInstruction(Instr);<br>
+ }<br>
+<br>
+ /// Augment the recipe to include Instr, if it lies at its End.<br>
+ bool appendInstruction(Instruction *Instr) {<br>
+ if (End != Instr->getIterator())<br>
+ return false;<br>
+ End++;<br>
+ return true;<br>
+ }<br>
+<br>
+ /// Print the recipe.<br>
+ void print(raw_ostream &O, const Twine &Indent) const override {<br>
+ O << " +\n" << Indent << "\"WIDEN\\l\"";<br>
+ for (auto &Instr : make_range(Begin, End))<br>
+ O << " +\n" << Indent << "\" " << VPlanIngredient(&Instr) << "<a href="file:///\\l\">\\l\</a>"";<br>
+ }<br>
+};<br>
+<br>
+/// A recipe for handling phi nodes of integer and floating-point inductions,<br>
+/// producing their vector and scalar values.<br>
+class VPWidenIntOrFpInductionRecipe : public VPRecipeBase {<br>
+private:<br>
+ PHINode *IV;<br>
+ TruncInst *Trunc;<br>
+<br>
+public:<br>
+ VPWidenIntOrFpInductionRecipe(PHINode *IV, TruncInst *Trunc = nullptr)<br>
+ : VPRecipeBase(VPWidenIntOrFpInductionSC), IV(IV), Trunc(Trunc) {}<br>
+<br>
+ ~VPWidenIntOrFpInductionRecipe() {}<br>
+<br>
+ /// Method to support type inquiry through isa, cast, and dyn_cast.<br>
+ static inline bool classof(const VPRecipeBase *V) {<br>
+ return V->getVPRecipeID() == VPRecipeBase::VPWidenIntOrFpInductionSC;<br>
+ }<br>
+<br>
+ /// Generate the vectorized and scalarized versions of the phi node as<br>
+ /// needed by their users.<br>
+ void execute(VPTransformState &State) override {<br>
+ assert(!State.Instance && "Int or FP induction being replicated.");<br>
+ State.ILV->widenIntOrFpInduction(IV, Trunc);<br>
+ }<br>
+<br>
+ /// Print the recipe.<br>
+ void print(raw_ostream &O, const Twine &Indent) const override {<br>
+ O << " +\n" << Indent << "\"WIDEN-INDUCTION";<br>
+ if (Trunc) {<br>
+ O << "<a href="file:///\\l\">\\l\</a>"";<br>
+ O << " +\n" << Indent << "\" " << VPlanIngredient(IV) << "<a href="file:///\\l\">\\l\</a>"";<br>
+ O << " +\n" << Indent << "\" " << VPlanIngredient(Trunc) << "<a href="file:///\\l\">\\l\</a>"";<br>
+ } else<br>
+ O << " " << VPlanIngredient(IV) << "<a href="file:///\\l\">\\l\</a>"";<br>
+ }<br>
+};<br>
+<br>
+/// A recipe for handling all phi nodes except for integer and FP inductions.<br>
+class VPWidenPHIRecipe : public VPRecipeBase {<br>
+private:<br>
+ PHINode *Phi;<br>
+<br>
+public:<br>
+ VPWidenPHIRecipe(PHINode *Phi) : VPRecipeBase(VPWidenPHISC), Phi(Phi) {}<br>
+<br>
+ ~VPWidenPHIRecipe() {}<br>
+<br>
+ /// Method to support type inquiry through isa, cast, and dyn_cast.<br>
+ static inline bool classof(const VPRecipeBase *V) {<br>
+ return V->getVPRecipeID() == VPRecipeBase::VPWidenPHISC;<br>
+ }<br>
+<br>
+ /// Generate the phi/select nodes.<br>
+ void execute(VPTransformState &State) override {<br>
+ State.ILV->widenPHIInstruction(Phi, State.UF, State.VF);<br>
+ }<br>
+<br>
+ /// Print the recipe.<br>
+ void print(raw_ostream &O, const Twine &Indent) const override {<br>
+ O << " +\n" << Indent << "\"WIDEN-PHI " << VPlanIngredient(Phi) << "<a href="file:///\\l\">\\l\</a>"";<br>
+ }<br>
+};<br>
+<br>
+/// VPInterleaveRecipe is a recipe for transforming an interleave group of load<br>
+/// or stores into one wide load/store and shuffles.<br>
+class VPInterleaveRecipe : public VPRecipeBase {<br>
+private:<br>
+ const InterleaveGroup *IG;<br>
+<br>
+public:<br>
+ VPInterleaveRecipe(const InterleaveGroup *IG)<br>
+ : VPRecipeBase(VPInterleaveSC), IG(IG) {}<br>
+<br>
+ ~VPInterleaveRecipe() {}<br>
+<br>
+ /// Method to support type inquiry through isa, cast, and dyn_cast.<br>
+ static inline bool classof(const VPRecipeBase *V) {<br>
+ return V->getVPRecipeID() == VPRecipeBase::VPInterleaveSC;<br>
+ }<br>
+<br>
+ /// Generate the wide load or store, and shuffles.<br>
+ void execute(VPTransformState &State) override {<br>
+ assert(!State.Instance && "Interleave group being replicated.");<br>
+ State.ILV->vectorizeInterleaveGroup(IG->getInsertPos());<br>
+ }<br>
+<br>
+ /// Print the recipe.<br>
+ void print(raw_ostream &O, const Twine &Indent) const override;<br>
+<br>
+ const InterleaveGroup *getInterleaveGroup() { return IG; }<br>
+};<br>
+<br>
+/// VPReplicateRecipe replicates a given instruction producing multiple scalar<br>
+/// copies of the original scalar type, one per lane, instead of producing a<br>
+/// single copy of widened type for all lanes. If the instruction is known to be<br>
+/// uniform only one copy, per lane zero, will be generated.<br>
+class VPReplicateRecipe : public VPRecipeBase {<br>
+private:<br>
+ /// The instruction being replicated.<br>
+ Instruction *Ingredient;<br>
+<br>
+ /// Indicator if only a single replica per lane is needed.<br>
+ bool IsUniform;<br>
+<br>
+ /// Indicator if the replicas are also predicated.<br>
+ bool IsPredicated;<br>
+<br>
+ /// Indicator if the scalar values should also be packed into a vector.<br>
+ bool AlsoPack;<br>
+<br>
+public:<br>
+ VPReplicateRecipe(Instruction *I, bool IsUniform, bool IsPredicated = false)<br>
+ : VPRecipeBase(VPReplicateSC), Ingredient(I), IsUniform(IsUniform),<br>
+ IsPredicated(IsPredicated) {<br>
+ // Retain the previous behavior of predicateInstructions(), where an<br>
+ // insert-element of a predicated instruction got hoisted into the<br>
+ // predicated basic block iff it was its only user. This is achieved by<br>
+ // having predicated instructions also pack their values into a vector by<br>
+ // default unless they have a replicated user which uses their scalar value.<br>
+ AlsoPack = IsPredicated && !I->use_empty();<br>
+ }<br>
+<br>
+ ~VPReplicateRecipe() {}<br>
+<br>
+ /// Method to support type inquiry through isa, cast, and dyn_cast.<br>
+ static inline bool classof(const VPRecipeBase *V) {<br>
+ return V->getVPRecipeID() == VPRecipeBase::VPReplicateSC;<br>
+ }<br>
+<br>
+ /// Generate replicas of the desired Ingredient. Replicas will be generated<br>
+ /// for all parts and lanes unless a specific part and lane are specified in<br>
+ /// the \p State.<br>
+ void execute(VPTransformState &State) override;<br>
+<br>
+ void setAlsoPack(bool Pack) { AlsoPack = Pack; }<br>
+<br>
+ /// Print the recipe.<br>
+ void print(raw_ostream &O, const Twine &Indent) const override {<br>
+ O << " +\n"<br>
+ << Indent << "\"" << (IsUniform ? "CLONE " : "REPLICATE ")<br>
+ << VPlanIngredient(Ingredient);<br>
+ if (AlsoPack)<br>
+ O << " (S->V)";<br>
+ O << "<a href="file:///\\l\">\\l\</a>"";<br>
+ }<br>
+};<br>
+<br>
+/// A recipe for generating conditional branches on the bits of a mask.<br>
+class VPBranchOnMaskRecipe : public VPRecipeBase {<br>
+private:<br>
+ /// The input IR basic block used to obtain the mask providing the condition<br>
+ /// bits for the branch.<br>
+ BasicBlock *MaskedBasicBlock;<br>
+<br>
+public:<br>
+ VPBranchOnMaskRecipe(BasicBlock *BB)<br>
+ : VPRecipeBase(VPBranchOnMaskSC), MaskedBasicBlock(BB) {}<br>
+<br>
+ /// Method to support type inquiry through isa, cast, and dyn_cast.<br>
+ static inline bool classof(const VPRecipeBase *V) {<br>
+ return V->getVPRecipeID() == VPRecipeBase::VPBranchOnMaskSC;<br>
+ }<br>
+<br>
+ /// Generate the extraction of the appropriate bit from the block mask and the<br>
+ /// conditional branch.<br>
+ void execute(VPTransformState &State) override;<br>
+<br>
+ /// Print the recipe.<br>
+ void print(raw_ostream &O, const Twine &Indent) const override {<br>
+ O << " +\n"<br>
+ << Indent << "\"BRANCH-ON-MASK-OF " << MaskedBasicBlock->getName()<br>
+ << "<a href="file:///\\l\">\\l\</a>"";<br>
+ }<br>
+};<br>
+<br>
+/// VPPredInstPHIRecipe is a recipe for generating the phi nodes needed when<br>
+/// control converges back from a Branch-on-Mask. The phi nodes are needed in<br>
+/// order to merge values that are set under such a branch and feed their uses.<br>
+/// The phi nodes can be scalar or vector depending on the users of the value.<br>
+/// This recipe works in concert with VPBranchOnMaskRecipe.<br>
+class VPPredInstPHIRecipe : public VPRecipeBase {<br>
+private:<br>
+ Instruction *PredInst;<br>
+<br>
+public:<br>
+ /// Construct a VPPredInstPHIRecipe given \p PredInst whose value needs a phi<br>
+ /// nodes after merging back from a Branch-on-Mask.<br>
+ VPPredInstPHIRecipe(Instruction *PredInst)<br>
+ : VPRecipeBase(VPPredInstPHISC), PredInst(PredInst) {}<br>
+<br>
+ ~VPPredInstPHIRecipe() {}<br>
+<br>
+ /// Method to support type inquiry through isa, cast, and dyn_cast.<br>
+ static inline bool classof(const VPRecipeBase *V) {<br>
+ return V->getVPRecipeID() == VPRecipeBase::VPPredInstPHISC;<br>
+ }<br>
+<br>
+ /// Generates phi nodes for live-outs as needed to retain SSA form.<br>
+ void execute(VPTransformState &State) override;<br>
+<br>
+ /// Print the recipe.<br>
+ void print(raw_ostream &O, const Twine &Indent) const override {<br>
+ O << " +\n"<br>
+ << Indent << "\"PHI-PREDICATED-INSTRUCTION " << VPlanIngredient(PredInst)<br>
+ << "<a href="file:///\\l\">\\l\</a>"";<br>
+ }<br>
+};<br>
+} // end anonymous namespace<br>
+<br>
+bool LoopVectorizationPlanner::getDecisionAndClampRange(<br>
+ const std::function<bool(unsigned)> &Predicate, VFRange &Range) {<br>
+ assert(Range.End > Range.Start && "Trying to test an empty VF range.");<br>
+ bool PredicateAtRangeStart = Predicate(Range.Start);<br>
+<br>
+ for (unsigned TmpVF = Range.Start * 2; TmpVF < Range.End; TmpVF *= 2)<br>
+ if (Predicate(TmpVF) != PredicateAtRangeStart) {<br>
+ Range.End = TmpVF;<br>
+ break;<br>
+ }<br>
+<br>
+ return PredicateAtRangeStart;<br>
+}<br>
+<br>
+/// Build VPlans for the full range of feasible VF's = {\p MinVF, 2 * \p MinVF,<br>
+/// 4 * \p MinVF, ..., \p MaxVF} by repeatedly building a VPlan for a sub-range<br>
+/// of VF's starting at a given VF and extending it as much as possible. Each<br>
+/// vectorization decision can potentially shorten this sub-range during<br>
+/// buildVPlan().<br>
+void LoopVectorizationPlanner::buildVPlans(unsigned MinVF, unsigned MaxVF) {<br>
+ for (unsigned VF = MinVF; VF < MaxVF + 1;) {<br>
+ VFRange SubRange = {VF, MaxVF + 1};<br>
+ VPlan *Plan = buildVPlan(SubRange);<br>
+ VPlans.push_back(Plan);<br>
+ VF = SubRange.End;<br>
+ }<br>
+}<br>
+<br>
+VPInterleaveRecipe *<br>
+LoopVectorizationPlanner::tryToInterleaveMemory(Instruction *I,<br>
+ VFRange &Range) {<br>
+ const InterleaveGroup *IG = Legal->getInterleavedAccessGroup(I);<br>
+ if (!IG)<br>
+ return nullptr;<br>
+<br>
+ // Now check if IG is relevant for VF's in the given range.<br>
+ auto isIGMember = [&](Instruction *I) -> std::function<bool(unsigned)> {<br>
+ return [=](unsigned VF) -> bool {<br>
+ return (VF >= 2 && // Query is illegal for VF == 1<br>
+ CM.getWideningDecision(I, VF) ==<br>
+ LoopVectorizationCostModel::CM_Interleave);<br>
+ };<br>
+ };<br>
+ if (!getDecisionAndClampRange(isIGMember(I), Range))<br>
+ return nullptr;<br>
+<br>
+ // I is a member of an InterleaveGroup for VF's in the (possibly trimmed)<br>
+ // range. If it's the primary member of the IG construct a VPInterleaveRecipe.<br>
+ // Otherwise, it's an adjunct member of the IG, do not construct any Recipe.<br>
+ assert(I == IG->getInsertPos() &&<br>
+ "Generating a recipe for an adjunct member of an interleave group");<br>
+<br>
+ return new VPInterleaveRecipe(IG);<br>
+}<br>
+<br>
+VPWidenIntOrFpInductionRecipe *<br>
+LoopVectorizationPlanner::tryToOptimizeInduction(Instruction *I,<br>
+ VFRange &Range) {<br>
+ if (PHINode *Phi = dyn_cast<PHINode>(I)) {<br>
+ // Check if this is an integer or fp induction. If so, build the recipe that<br>
+ // produces its scalar and vector values.<br>
+ InductionDescriptor II = Legal->getInductionVars()->lookup(Phi);<br>
+ if (II.getKind() == InductionDescriptor::IK_IntInduction ||<br>
+ II.getKind() == InductionDescriptor::IK_FpInduction)<br>
+ return new VPWidenIntOrFpInductionRecipe(Phi);<br>
+<br>
+ return nullptr;<br>
+ }<br>
+<br>
+ // Optimize the special case where the source is a constant integer<br>
+ // induction variable. Notice that we can only optimize the 'trunc' case<br>
+ // because (a) FP conversions lose precision, (b) sext/zext may wrap, and<br>
+ // (c) other casts depend on pointer size.<br>
+<br>
+ // Determine whether \p K is a truncation based on an induction variable that<br>
+ // can be optimized.<br>
+ auto isOptimizableIVTruncate =<br>
+ [&](Instruction *K) -> std::function<bool(unsigned)> {<br>
+ return<br>
+ [=](unsigned VF) -> bool { return CM.isOptimizableIVTruncate(K, VF); };<br>
+ };<br>
+<br>
+ if (isa<TruncInst>(I) &&<br>
+ getDecisionAndClampRange(isOptimizableIVTruncate(I), Range))<br>
+ return new VPWidenIntOrFpInductionRecipe(cast<PHINode>(I->getOperand(0)),<br>
+ cast<TruncInst>(I));<br>
+ return nullptr;<br>
+}<br>
+<br>
+VPWidenRecipe *LoopVectorizationPlanner::tryToWiden(<br>
+ Instruction *I, VPWidenRecipe *LastWidenRecipe, VFRange &Range) {<br>
+<br>
+ if (Legal->isScalarWithPredication(I))<br>
+ return nullptr;<br>
+<br>
+ static DenseSet<unsigned> VectorizableOpcodes = {<br>
+ Instruction::Br, Instruction::PHI, Instruction::GetElementPtr,<br>
+ Instruction::UDiv, Instruction::SDiv, Instruction::SRem,<br>
+ Instruction::URem, Instruction::Add, Instruction::FAdd,<br>
+ Instruction::Sub, Instruction::FSub, Instruction::Mul,<br>
+ Instruction::FMul, Instruction::FDiv, <o:p></o:p></p>
</blockquote>
</div>
</div>
<p>---------------------------------------------------------------------<br>
Intel Israel (74) Limited</p>
<p>This e-mail and any attachments may contain confidential material for<br>
the sole use of the intended recipient(s). Any review or distribution<br>
by others is strictly prohibited. If you are not the intended<br>
recipient, please contact the sender and delete all copies.</p></body>
</html>