[llvm] 585c594 - Move delinearization logic out of SCEV [NFC]
Philip Reames via llvm-commits
llvm-commits at lists.llvm.org
Wed Sep 8 16:57:20 PDT 2021
Done, e741fab
Philip
On 9/8/21 4:20 PM, Philip Reames wrote:
> Michael,
>
> Your first paragraph comes off as rather dismissive of both my
> judgement and the value of incrementalism in general. I'm going to
> assume that wasn't your intent, just pointing it out for your reference.
>
> I will move getIndexExpressionsFromGEP as requested.
>
> On the second point, feel free to contribute your own time towards the
> proposed enhancement. I don't disagree with anything you said, I'm
> just not motivated to do it.
>
> Philip
>
> On 9/8/21 4:12 PM, Michael Kruse wrote:
>> If there was a reason to move delinearize() out of ScEv, the same
>> reason applies to getIndexExpressionsFromGEP ("delinearize using
>> GetElementPtr as hint") as well. I would prefer to keep them together.
>>
>> getElementSize on the other side might have general usefulness as
>> well. However, I think it is badly named (better: "getAccessedSize")
>> and not only loads and stores access memory, some intrinsics do as
>> well and it is up to the client what that can support. For instance,
>> Polly also supports memset, memcpy, memove.
>>
>> Michael
>>
>>
>> Am Mi., 8. Sept. 2021 um 16:50 Uhr schrieb Philip Reames
>> <listmail at philipreames.com>:
>>> They weren't part of the tree rooted in delinearize. We could move
>>> them
>>> or not, either seems reasonable.
>>>
>>> Philip
>>>
>>> On 9/8/21 1:05 PM, Michael Kruse wrote:
>>>> You left getIndexExpressionsFromGEP inside ScalarEvolution. Don't you
>>>> consider it part of Delinearization? getElementSize is also only used
>>>> in delinearization use cases/was introduced for it.
>>>>
>>>> Michael
>>>>
>>>> Am Mi., 8. Sept. 2021 um 14:28 Uhr schrieb Philip Reames via
>>>> llvm-commits <llvm-commits at lists.llvm.org>:
>>>>> Author: Philip Reames
>>>>> Date: 2021-09-08T12:28:35-07:00
>>>>> New Revision: 585c594d749a2a88150b63804587af85abdabeaa
>>>>>
>>>>> URL:
>>>>> https://github.com/llvm/llvm-project/commit/585c594d749a2a88150b63804587af85abdabeaa
>>>>> DIFF:
>>>>> https://github.com/llvm/llvm-project/commit/585c594d749a2a88150b63804587af85abdabeaa.diff
>>>>>
>>>>> LOG: Move delinearization logic out of SCEV [NFC]
>>>>>
>>>>> None of this logic has anything to do with SCEV's internals, it
>>>>> just uses the existing public APIs. As a result, we can move the
>>>>> code from ScalarEvolution.cpp/hpp to Delinearization.cpp/hpp with
>>>>> only minor changes.
>>>>>
>>>>> This was discussed in advance on today's loop opt call. It turned
>>>>> out to be easy as hoped.
>>>>>
>>>>> Added:
>>>>>
>>>>>
>>>>> Modified:
>>>>> llvm/include/llvm/Analysis/Delinearization.h
>>>>> llvm/include/llvm/Analysis/ScalarEvolution.h
>>>>> llvm/lib/Analysis/Delinearization.cpp
>>>>> llvm/lib/Analysis/DependenceAnalysis.cpp
>>>>> llvm/lib/Analysis/LoopCacheAnalysis.cpp
>>>>> llvm/lib/Analysis/ScalarEvolution.cpp
>>>>>
>>>>> Removed:
>>>>>
>>>>>
>>>>>
>>>>> ################################################################################
>>>>>
>>>>> diff --git a/llvm/include/llvm/Analysis/Delinearization.h
>>>>> b/llvm/include/llvm/Analysis/Delinearization.h
>>>>> index 2658b6bbc80c0..dec3fd9b08a9f 100644
>>>>> --- a/llvm/include/llvm/Analysis/Delinearization.h
>>>>> +++ b/llvm/include/llvm/Analysis/Delinearization.h
>>>>> @@ -16,10 +16,100 @@
>>>>> #ifndef LLVM_ANALYSIS_DELINEARIZATION_H
>>>>> #define LLVM_ANALYSIS_DELINEARIZATION_H
>>>>>
>>>>> +#include "llvm/ADT/SmallVector.h"
>>>>> #include "llvm/IR/PassManager.h"
>>>>> #include "llvm/Support/raw_ostream.h"
>>>>>
>>>>> namespace llvm {
>>>>> +class ScalarEvolution;
>>>>> +class SCEV;
>>>>> +
>>>>> +/// Compute the array dimensions Sizes from the set of Terms
>>>>> extracted from
>>>>> +/// the memory access function of this SCEVAddRecExpr (second
>>>>> step of
>>>>> +/// delinearization).
>>>>> +void findArrayDimensions(ScalarEvolution &SE,
>>>>> + SmallVectorImpl<const SCEV *> &Terms,
>>>>> + SmallVectorImpl<const SCEV *> &Sizes,
>>>>> + const SCEV *ElementSize);
>>>>> +
>>>>> +/// Collect parametric terms occurring in step expressions (first
>>>>> step of
>>>>> +/// delinearization).
>>>>> +void collectParametricTerms(ScalarEvolution &SE, const SCEV *Expr,
>>>>> + SmallVectorImpl<const SCEV *> &Terms);
>>>>> +
>>>>> +/// Return in Subscripts the access functions for each dimension
>>>>> in Sizes
>>>>> +/// (third step of delinearization).
>>>>> +void computeAccessFunctions(ScalarEvolution &SE, const SCEV *Expr,
>>>>> + SmallVectorImpl<const SCEV *>
>>>>> &Subscripts,
>>>>> + SmallVectorImpl<const SCEV *> &Sizes);
>>>>> +/// Split this SCEVAddRecExpr into two vectors of SCEVs
>>>>> representing the
>>>>> +/// subscripts and sizes of an array access.
>>>>> +///
>>>>> +/// The delinearization is a 3 step process: the first two steps
>>>>> compute the
>>>>> +/// sizes of each subscript and the third step computes the
>>>>> access functions
>>>>> +/// for the delinearized array:
>>>>> +///
>>>>> +/// 1. Find the terms in the step functions
>>>>> +/// 2. Compute the array size
>>>>> +/// 3. Compute the access function: divide the SCEV by the array
>>>>> size
>>>>> +/// starting with the innermost dimensions found in step 2.
>>>>> The Quotient
>>>>> +/// is the SCEV to be divided in the next step of the
>>>>> recursion. The
>>>>> +/// Remainder is the subscript of the innermost dimension.
>>>>> Loop over all
>>>>> +/// array dimensions computed in step 2.
>>>>> +///
>>>>> +/// To compute a uniform array size for several memory accesses
>>>>> to the same
>>>>> +/// object, one can collect in step 1 all the step terms for all
>>>>> the memory
>>>>> +/// accesses, and compute in step 2 a unique array shape. This
>>>>> guarantees
>>>>> +/// that the array shape will be the same across all memory
>>>>> accesses.
>>>>> +///
>>>>> +/// FIXME: We could derive the result of steps 1 and 2 from a
>>>>> description of
>>>>> +/// the array shape given in metadata.
>>>>> +///
>>>>> +/// Example:
>>>>> +///
>>>>> +/// A[][n][m]
>>>>> +///
>>>>> +/// for i
>>>>> +/// for j
>>>>> +/// for k
>>>>> +/// A[j+k][2i][5i] =
>>>>> +///
>>>>> +/// The initial SCEV:
>>>>> +///
>>>>> +/// A[{{{0,+,2*m+5}_i, +, n*m}_j, +, n*m}_k]
>>>>> +///
>>>>> +/// 1. Find the
>>>>> diff erent terms in the step functions:
>>>>> +/// -> [2*m, 5, n*m, n*m]
>>>>> +///
>>>>> +/// 2. Compute the array size: sort and unique them
>>>>> +/// -> [n*m, 2*m, 5]
>>>>> +/// find the GCD of all the terms = 1
>>>>> +/// divide by the GCD and erase constant terms
>>>>> +/// -> [n*m, 2*m]
>>>>> +/// GCD = m
>>>>> +/// divide by GCD -> [n, 2]
>>>>> +/// remove constant terms
>>>>> +/// -> [n]
>>>>> +/// size of the array is A[unknown][n][m]
>>>>> +///
>>>>> +/// 3. Compute the access function
>>>>> +/// a. Divide {{{0,+,2*m+5}_i, +, n*m}_j, +, n*m}_k by the
>>>>> innermost size m
>>>>> +/// Quotient: {{{0,+,2}_i, +, n}_j, +, n}_k
>>>>> +/// Remainder: {{{0,+,5}_i, +, 0}_j, +, 0}_k
>>>>> +/// The remainder is the subscript of the innermost array
>>>>> dimension: [5i].
>>>>> +///
>>>>> +/// b. Divide Quotient: {{{0,+,2}_i, +, n}_j, +, n}_k by next
>>>>> outer size n
>>>>> +/// Quotient: {{{0,+,0}_i, +, 1}_j, +, 1}_k
>>>>> +/// Remainder: {{{0,+,2}_i, +, 0}_j, +, 0}_k
>>>>> +/// The Remainder is the subscript of the next array dimension:
>>>>> [2i].
>>>>> +///
>>>>> +/// The subscript of the outermost dimension is the Quotient: [j+k].
>>>>> +///
>>>>> +/// Overall, we have: A[][n][m], and the access function:
>>>>> A[j+k][2i][5i].
>>>>> +void delinearize(ScalarEvolution &SE, const SCEV *Expr,
>>>>> + SmallVectorImpl<const SCEV *> &Subscripts,
>>>>> + SmallVectorImpl<const SCEV *> &Sizes, const SCEV
>>>>> *ElementSize);
>>>>> +
>>>>> struct DelinearizationPrinterPass
>>>>> : public PassInfoMixin<DelinearizationPrinterPass> {
>>>>> explicit DelinearizationPrinterPass(raw_ostream &OS);
>>>>>
>>>>> diff --git a/llvm/include/llvm/Analysis/ScalarEvolution.h
>>>>> b/llvm/include/llvm/Analysis/ScalarEvolution.h
>>>>> index 6ae865e011cf0..445a925c70e83 100644
>>>>> --- a/llvm/include/llvm/Analysis/ScalarEvolution.h
>>>>> +++ b/llvm/include/llvm/Analysis/ScalarEvolution.h
>>>>> @@ -1099,29 +1099,11 @@ class ScalarEvolution {
>>>>> /// Return the size of an element read or written by Inst.
>>>>> const SCEV *getElementSize(Instruction *Inst);
>>>>>
>>>>> - /// Compute the array dimensions Sizes from the set of Terms
>>>>> extracted from
>>>>> - /// the memory access function of this SCEVAddRecExpr (second
>>>>> step of
>>>>> - /// delinearization).
>>>>> - void findArrayDimensions(SmallVectorImpl<const SCEV *> &Terms,
>>>>> - SmallVectorImpl<const SCEV *> &Sizes,
>>>>> - const SCEV *ElementSize);
>>>>> -
>>>>> void print(raw_ostream &OS) const;
>>>>> void verify() const;
>>>>> bool invalidate(Function &F, const PreservedAnalyses &PA,
>>>>> FunctionAnalysisManager::Invalidator &Inv);
>>>>>
>>>>> - /// Collect parametric terms occurring in step expressions
>>>>> (first step of
>>>>> - /// delinearization).
>>>>> - void collectParametricTerms(const SCEV *Expr,
>>>>> - SmallVectorImpl<const SCEV *> &Terms);
>>>>> -
>>>>> - /// Return in Subscripts the access functions for each
>>>>> dimension in Sizes
>>>>> - /// (third step of delinearization).
>>>>> - void computeAccessFunctions(const SCEV *Expr,
>>>>> - SmallVectorImpl<const SCEV *>
>>>>> &Subscripts,
>>>>> - SmallVectorImpl<const SCEV *> &Sizes);
>>>>> -
>>>>> /// Gathers the individual index expressions from a GEP
>>>>> instruction.
>>>>> ///
>>>>> /// This function optimistically assumes the GEP references
>>>>> into a fixed size
>>>>> @@ -1135,74 +1117,6 @@ class ScalarEvolution {
>>>>> SmallVectorImpl<const SCEV *> &Subscripts,
>>>>> SmallVectorImpl<int> &Sizes);
>>>>>
>>>>> - /// Split this SCEVAddRecExpr into two vectors of SCEVs
>>>>> representing the
>>>>> - /// subscripts and sizes of an array access.
>>>>> - ///
>>>>> - /// The delinearization is a 3 step process: the first two
>>>>> steps compute the
>>>>> - /// sizes of each subscript and the third step computes the
>>>>> access functions
>>>>> - /// for the delinearized array:
>>>>> - ///
>>>>> - /// 1. Find the terms in the step functions
>>>>> - /// 2. Compute the array size
>>>>> - /// 3. Compute the access function: divide the SCEV by the
>>>>> array size
>>>>> - /// starting with the innermost dimensions found in step 2.
>>>>> The Quotient
>>>>> - /// is the SCEV to be divided in the next step of the
>>>>> recursion. The
>>>>> - /// Remainder is the subscript of the innermost dimension.
>>>>> Loop over all
>>>>> - /// array dimensions computed in step 2.
>>>>> - ///
>>>>> - /// To compute a uniform array size for several memory accesses
>>>>> to the same
>>>>> - /// object, one can collect in step 1 all the step terms for
>>>>> all the memory
>>>>> - /// accesses, and compute in step 2 a unique array shape. This
>>>>> guarantees
>>>>> - /// that the array shape will be the same across all memory
>>>>> accesses.
>>>>> - ///
>>>>> - /// FIXME: We could derive the result of steps 1 and 2 from a
>>>>> description of
>>>>> - /// the array shape given in metadata.
>>>>> - ///
>>>>> - /// Example:
>>>>> - ///
>>>>> - /// A[][n][m]
>>>>> - ///
>>>>> - /// for i
>>>>> - /// for j
>>>>> - /// for k
>>>>> - /// A[j+k][2i][5i] =
>>>>> - ///
>>>>> - /// The initial SCEV:
>>>>> - ///
>>>>> - /// A[{{{0,+,2*m+5}_i, +, n*m}_j, +, n*m}_k]
>>>>> - ///
>>>>> - /// 1. Find the
>>>>> diff erent terms in the step functions:
>>>>> - /// -> [2*m, 5, n*m, n*m]
>>>>> - ///
>>>>> - /// 2. Compute the array size: sort and unique them
>>>>> - /// -> [n*m, 2*m, 5]
>>>>> - /// find the GCD of all the terms = 1
>>>>> - /// divide by the GCD and erase constant terms
>>>>> - /// -> [n*m, 2*m]
>>>>> - /// GCD = m
>>>>> - /// divide by GCD -> [n, 2]
>>>>> - /// remove constant terms
>>>>> - /// -> [n]
>>>>> - /// size of the array is A[unknown][n][m]
>>>>> - ///
>>>>> - /// 3. Compute the access function
>>>>> - /// a. Divide {{{0,+,2*m+5}_i, +, n*m}_j, +, n*m}_k by the
>>>>> innermost size m
>>>>> - /// Quotient: {{{0,+,2}_i, +, n}_j, +, n}_k
>>>>> - /// Remainder: {{{0,+,5}_i, +, 0}_j, +, 0}_k
>>>>> - /// The remainder is the subscript of the innermost array
>>>>> dimension: [5i].
>>>>> - ///
>>>>> - /// b. Divide Quotient: {{{0,+,2}_i, +, n}_j, +, n}_k by next
>>>>> outer size n
>>>>> - /// Quotient: {{{0,+,0}_i, +, 1}_j, +, 1}_k
>>>>> - /// Remainder: {{{0,+,2}_i, +, 0}_j, +, 0}_k
>>>>> - /// The Remainder is the subscript of the next array dimension:
>>>>> [2i].
>>>>> - ///
>>>>> - /// The subscript of the outermost dimension is the Quotient:
>>>>> [j+k].
>>>>> - ///
>>>>> - /// Overall, we have: A[][n][m], and the access function:
>>>>> A[j+k][2i][5i].
>>>>> - void delinearize(const SCEV *Expr, SmallVectorImpl<const SCEV
>>>>> *> &Subscripts,
>>>>> - SmallVectorImpl<const SCEV *> &Sizes,
>>>>> - const SCEV *ElementSize);
>>>>> -
>>>>> /// Return the DataLayout associated with the module this
>>>>> SCEV instance is
>>>>> /// operating on.
>>>>> const DataLayout &getDataLayout() const {
>>>>>
>>>>> diff --git a/llvm/lib/Analysis/Delinearization.cpp
>>>>> b/llvm/lib/Analysis/Delinearization.cpp
>>>>> index 448e970e9bcc9..8219d59b8f40e 100644
>>>>> --- a/llvm/lib/Analysis/Delinearization.cpp
>>>>> +++ b/llvm/lib/Analysis/Delinearization.cpp
>>>>> @@ -17,6 +17,7 @@
>>>>> #include "llvm/Analysis/LoopInfo.h"
>>>>> #include "llvm/Analysis/Passes.h"
>>>>> #include "llvm/Analysis/ScalarEvolution.h"
>>>>> +#include "llvm/Analysis/ScalarEvolutionDivision.h"
>>>>> #include "llvm/Analysis/ScalarEvolutionExpressions.h"
>>>>> #include "llvm/IR/Constants.h"
>>>>> #include "llvm/IR/DerivedTypes.h"
>>>>> @@ -36,6 +37,454 @@ using namespace llvm;
>>>>> #define DL_NAME "delinearize"
>>>>> #define DEBUG_TYPE DL_NAME
>>>>>
>>>>> +// Return true when S contains at least an undef value.
>>>>> +static inline bool containsUndefs(const SCEV *S) {
>>>>> + return SCEVExprContains(S, [](const SCEV *S) {
>>>>> + if (const auto *SU = dyn_cast<SCEVUnknown>(S))
>>>>> + return isa<UndefValue>(SU->getValue());
>>>>> + return false;
>>>>> + });
>>>>> +}
>>>>> +
>>>>> +namespace {
>>>>> +
>>>>> +// Collect all steps of SCEV expressions.
>>>>> +struct SCEVCollectStrides {
>>>>> + ScalarEvolution &SE;
>>>>> + SmallVectorImpl<const SCEV *> &Strides;
>>>>> +
>>>>> + SCEVCollectStrides(ScalarEvolution &SE, SmallVectorImpl<const
>>>>> SCEV *> &S)
>>>>> + : SE(SE), Strides(S) {}
>>>>> +
>>>>> + bool follow(const SCEV *S) {
>>>>> + if (const SCEVAddRecExpr *AR = dyn_cast<SCEVAddRecExpr>(S))
>>>>> + Strides.push_back(AR->getStepRecurrence(SE));
>>>>> + return true;
>>>>> + }
>>>>> +
>>>>> + bool isDone() const { return false; }
>>>>> +};
>>>>> +
>>>>> +// Collect all SCEVUnknown and SCEVMulExpr expressions.
>>>>> +struct SCEVCollectTerms {
>>>>> + SmallVectorImpl<const SCEV *> &Terms;
>>>>> +
>>>>> + SCEVCollectTerms(SmallVectorImpl<const SCEV *> &T) : Terms(T) {}
>>>>> +
>>>>> + bool follow(const SCEV *S) {
>>>>> + if (isa<SCEVUnknown>(S) || isa<SCEVMulExpr>(S) ||
>>>>> + isa<SCEVSignExtendExpr>(S)) {
>>>>> + if (!containsUndefs(S))
>>>>> + Terms.push_back(S);
>>>>> +
>>>>> + // Stop recursion: once we collected a term, do not walk
>>>>> its operands.
>>>>> + return false;
>>>>> + }
>>>>> +
>>>>> + // Keep looking.
>>>>> + return true;
>>>>> + }
>>>>> +
>>>>> + bool isDone() const { return false; }
>>>>> +};
>>>>> +
>>>>> +// Check if a SCEV contains an AddRecExpr.
>>>>> +struct SCEVHasAddRec {
>>>>> + bool &ContainsAddRec;
>>>>> +
>>>>> + SCEVHasAddRec(bool &ContainsAddRec) :
>>>>> ContainsAddRec(ContainsAddRec) {
>>>>> + ContainsAddRec = false;
>>>>> + }
>>>>> +
>>>>> + bool follow(const SCEV *S) {
>>>>> + if (isa<SCEVAddRecExpr>(S)) {
>>>>> + ContainsAddRec = true;
>>>>> +
>>>>> + // Stop recursion: once we collected a term, do not walk
>>>>> its operands.
>>>>> + return false;
>>>>> + }
>>>>> +
>>>>> + // Keep looking.
>>>>> + return true;
>>>>> + }
>>>>> +
>>>>> + bool isDone() const { return false; }
>>>>> +};
>>>>> +
>>>>> +// Find factors that are multiplied with an expression that
>>>>> (possibly as a
>>>>> +// subexpression) contains an AddRecExpr. In the expression:
>>>>> +//
>>>>> +// 8 * (100 + %p * %q * (%a + {0, +, 1}_loop))
>>>>> +//
>>>>> +// "%p * %q" are factors multiplied by the expression "(%a + {0,
>>>>> +, 1}_loop)"
>>>>> +// that contains the AddRec {0, +, 1}_loop. %p * %q are likely to
>>>>> be array size
>>>>> +// parameters as they form a product with an induction variable.
>>>>> +//
>>>>> +// This collector expects all array size parameters to be in the
>>>>> same MulExpr.
>>>>> +// It might be necessary to later add support for collecting
>>>>> parameters that are
>>>>> +// spread over
>>>>> diff erent nested MulExpr.
>>>>> +struct SCEVCollectAddRecMultiplies {
>>>>> + SmallVectorImpl<const SCEV *> &Terms;
>>>>> + ScalarEvolution &SE;
>>>>> +
>>>>> + SCEVCollectAddRecMultiplies(SmallVectorImpl<const SCEV *> &T,
>>>>> + ScalarEvolution &SE)
>>>>> + : Terms(T), SE(SE) {}
>>>>> +
>>>>> + bool follow(const SCEV *S) {
>>>>> + if (auto *Mul = dyn_cast<SCEVMulExpr>(S)) {
>>>>> + bool HasAddRec = false;
>>>>> + SmallVector<const SCEV *, 0> Operands;
>>>>> + for (auto Op : Mul->operands()) {
>>>>> + const SCEVUnknown *Unknown = dyn_cast<SCEVUnknown>(Op);
>>>>> + if (Unknown && !isa<CallInst>(Unknown->getValue())) {
>>>>> + Operands.push_back(Op);
>>>>> + } else if (Unknown) {
>>>>> + HasAddRec = true;
>>>>> + } else {
>>>>> + bool ContainsAddRec = false;
>>>>> + SCEVHasAddRec ContiansAddRec(ContainsAddRec);
>>>>> + visitAll(Op, ContiansAddRec);
>>>>> + HasAddRec |= ContainsAddRec;
>>>>> + }
>>>>> + }
>>>>> + if (Operands.size() == 0)
>>>>> + return true;
>>>>> +
>>>>> + if (!HasAddRec)
>>>>> + return false;
>>>>> +
>>>>> + Terms.push_back(SE.getMulExpr(Operands));
>>>>> + // Stop recursion: once we collected a term, do not walk
>>>>> its operands.
>>>>> + return false;
>>>>> + }
>>>>> +
>>>>> + // Keep looking.
>>>>> + return true;
>>>>> + }
>>>>> +
>>>>> + bool isDone() const { return false; }
>>>>> +};
>>>>> +
>>>>> +} // end anonymous namespace
>>>>> +
>>>>> +/// Find parametric terms in this SCEVAddRecExpr. We first for
>>>>> parameters in
>>>>> +/// two places:
>>>>> +/// 1) The strides of AddRec expressions.
>>>>> +/// 2) Unknowns that are multiplied with AddRec expressions.
>>>>> +void llvm::collectParametricTerms(ScalarEvolution &SE, const SCEV
>>>>> *Expr,
>>>>> + SmallVectorImpl<const SCEV *> &Terms) {
>>>>> + SmallVector<const SCEV *, 4> Strides;
>>>>> + SCEVCollectStrides StrideCollector(SE, Strides);
>>>>> + visitAll(Expr, StrideCollector);
>>>>> +
>>>>> + LLVM_DEBUG({
>>>>> + dbgs() << "Strides:\n";
>>>>> + for (const SCEV *S : Strides)
>>>>> + dbgs() << *S << "\n";
>>>>> + });
>>>>> +
>>>>> + for (const SCEV *S : Strides) {
>>>>> + SCEVCollectTerms TermCollector(Terms);
>>>>> + visitAll(S, TermCollector);
>>>>> + }
>>>>> +
>>>>> + LLVM_DEBUG({
>>>>> + dbgs() << "Terms:\n";
>>>>> + for (const SCEV *T : Terms)
>>>>> + dbgs() << *T << "\n";
>>>>> + });
>>>>> +
>>>>> + SCEVCollectAddRecMultiplies MulCollector(Terms, SE);
>>>>> + visitAll(Expr, MulCollector);
>>>>> +}
>>>>> +
>>>>> +static bool findArrayDimensionsRec(ScalarEvolution &SE,
>>>>> + SmallVectorImpl<const SCEV *> &Terms,
>>>>> + SmallVectorImpl<const SCEV *> &Sizes) {
>>>>> + int Last = Terms.size() - 1;
>>>>> + const SCEV *Step = Terms[Last];
>>>>> +
>>>>> + // End of recursion.
>>>>> + if (Last == 0) {
>>>>> + if (const SCEVMulExpr *M = dyn_cast<SCEVMulExpr>(Step)) {
>>>>> + SmallVector<const SCEV *, 2> Qs;
>>>>> + for (const SCEV *Op : M->operands())
>>>>> + if (!isa<SCEVConstant>(Op))
>>>>> + Qs.push_back(Op);
>>>>> +
>>>>> + Step = SE.getMulExpr(Qs);
>>>>> + }
>>>>> +
>>>>> + Sizes.push_back(Step);
>>>>> + return true;
>>>>> + }
>>>>> +
>>>>> + for (const SCEV *&Term : Terms) {
>>>>> + // Normalize the terms before the next call to
>>>>> findArrayDimensionsRec.
>>>>> + const SCEV *Q, *R;
>>>>> + SCEVDivision::divide(SE, Term, Step, &Q, &R);
>>>>> +
>>>>> + // Bail out when GCD does not evenly divide one of the terms.
>>>>> + if (!R->isZero())
>>>>> + return false;
>>>>> +
>>>>> + Term = Q;
>>>>> + }
>>>>> +
>>>>> + // Remove all SCEVConstants.
>>>>> + erase_if(Terms, [](const SCEV *E) { return
>>>>> isa<SCEVConstant>(E); });
>>>>> +
>>>>> + if (Terms.size() > 0)
>>>>> + if (!findArrayDimensionsRec(SE, Terms, Sizes))
>>>>> + return false;
>>>>> +
>>>>> + Sizes.push_back(Step);
>>>>> + return true;
>>>>> +}
>>>>> +
>>>>> +// Returns true when one of the SCEVs of Terms contains a
>>>>> SCEVUnknown parameter.
>>>>> +static inline bool containsParameters(SmallVectorImpl<const SCEV
>>>>> *> &Terms) {
>>>>> + for (const SCEV *T : Terms)
>>>>> + if (SCEVExprContains(T, [](const SCEV *S) { return
>>>>> isa<SCEVUnknown>(S); }))
>>>>> + return true;
>>>>> +
>>>>> + return false;
>>>>> +}
>>>>> +
>>>>> +// Return the number of product terms in S.
>>>>> +static inline int numberOfTerms(const SCEV *S) {
>>>>> + if (const SCEVMulExpr *Expr = dyn_cast<SCEVMulExpr>(S))
>>>>> + return Expr->getNumOperands();
>>>>> + return 1;
>>>>> +}
>>>>> +
>>>>> +static const SCEV *removeConstantFactors(ScalarEvolution &SE,
>>>>> const SCEV *T) {
>>>>> + if (isa<SCEVConstant>(T))
>>>>> + return nullptr;
>>>>> +
>>>>> + if (isa<SCEVUnknown>(T))
>>>>> + return T;
>>>>> +
>>>>> + if (const SCEVMulExpr *M = dyn_cast<SCEVMulExpr>(T)) {
>>>>> + SmallVector<const SCEV *, 2> Factors;
>>>>> + for (const SCEV *Op : M->operands())
>>>>> + if (!isa<SCEVConstant>(Op))
>>>>> + Factors.push_back(Op);
>>>>> +
>>>>> + return SE.getMulExpr(Factors);
>>>>> + }
>>>>> +
>>>>> + return T;
>>>>> +}
>>>>> +
>>>>> +void llvm::findArrayDimensions(ScalarEvolution &SE,
>>>>> + SmallVectorImpl<const SCEV *> &Terms,
>>>>> + SmallVectorImpl<const SCEV *> &Sizes,
>>>>> + const SCEV *ElementSize) {
>>>>> + if (Terms.size() < 1 || !ElementSize)
>>>>> + return;
>>>>> +
>>>>> + // Early return when Terms do not contain parameters: we do not
>>>>> delinearize
>>>>> + // non parametric SCEVs.
>>>>> + if (!containsParameters(Terms))
>>>>> + return;
>>>>> +
>>>>> + LLVM_DEBUG({
>>>>> + dbgs() << "Terms:\n";
>>>>> + for (const SCEV *T : Terms)
>>>>> + dbgs() << *T << "\n";
>>>>> + });
>>>>> +
>>>>> + // Remove duplicates.
>>>>> + array_pod_sort(Terms.begin(), Terms.end());
>>>>> + Terms.erase(std::unique(Terms.begin(), Terms.end()), Terms.end());
>>>>> +
>>>>> + // Put larger terms first.
>>>>> + llvm::sort(Terms, [](const SCEV *LHS, const SCEV *RHS) {
>>>>> + return numberOfTerms(LHS) > numberOfTerms(RHS);
>>>>> + });
>>>>> +
>>>>> + // Try to divide all terms by the element size. If term is not
>>>>> divisible by
>>>>> + // element size, proceed with the original term.
>>>>> + for (const SCEV *&Term : Terms) {
>>>>> + const SCEV *Q, *R;
>>>>> + SCEVDivision::divide(SE, Term, ElementSize, &Q, &R);
>>>>> + if (!Q->isZero())
>>>>> + Term = Q;
>>>>> + }
>>>>> +
>>>>> + SmallVector<const SCEV *, 4> NewTerms;
>>>>> +
>>>>> + // Remove constant factors.
>>>>> + for (const SCEV *T : Terms)
>>>>> + if (const SCEV *NewT = removeConstantFactors(SE, T))
>>>>> + NewTerms.push_back(NewT);
>>>>> +
>>>>> + LLVM_DEBUG({
>>>>> + dbgs() << "Terms after sorting:\n";
>>>>> + for (const SCEV *T : NewTerms)
>>>>> + dbgs() << *T << "\n";
>>>>> + });
>>>>> +
>>>>> + if (NewTerms.empty() || !findArrayDimensionsRec(SE, NewTerms,
>>>>> Sizes)) {
>>>>> + Sizes.clear();
>>>>> + return;
>>>>> + }
>>>>> +
>>>>> + // The last element to be pushed into Sizes is the size of an
>>>>> element.
>>>>> + Sizes.push_back(ElementSize);
>>>>> +
>>>>> + LLVM_DEBUG({
>>>>> + dbgs() << "Sizes:\n";
>>>>> + for (const SCEV *S : Sizes)
>>>>> + dbgs() << *S << "\n";
>>>>> + });
>>>>> +}
>>>>> +
>>>>> +void llvm::computeAccessFunctions(ScalarEvolution &SE, const SCEV
>>>>> *Expr,
>>>>> + SmallVectorImpl<const SCEV *> &Subscripts,
>>>>> + SmallVectorImpl<const SCEV *> &Sizes) {
>>>>> + // Early exit in case this SCEV is not an affine multivariate
>>>>> function.
>>>>> + if (Sizes.empty())
>>>>> + return;
>>>>> +
>>>>> + if (auto *AR = dyn_cast<SCEVAddRecExpr>(Expr))
>>>>> + if (!AR->isAffine())
>>>>> + return;
>>>>> +
>>>>> + const SCEV *Res = Expr;
>>>>> + int Last = Sizes.size() - 1;
>>>>> + for (int i = Last; i >= 0; i--) {
>>>>> + const SCEV *Q, *R;
>>>>> + SCEVDivision::divide(SE, Res, Sizes[i], &Q, &R);
>>>>> +
>>>>> + LLVM_DEBUG({
>>>>> + dbgs() << "Res: " << *Res << "\n";
>>>>> + dbgs() << "Sizes[i]: " << *Sizes[i] << "\n";
>>>>> + dbgs() << "Res divided by Sizes[i]:\n";
>>>>> + dbgs() << "Quotient: " << *Q << "\n";
>>>>> + dbgs() << "Remainder: " << *R << "\n";
>>>>> + });
>>>>> +
>>>>> + Res = Q;
>>>>> +
>>>>> + // Do not record the last subscript corresponding to the size
>>>>> of elements in
>>>>> + // the array.
>>>>> + if (i == Last) {
>>>>> +
>>>>> + // Bail out if the remainder is too complex.
>>>>> + if (isa<SCEVAddRecExpr>(R)) {
>>>>> + Subscripts.clear();
>>>>> + Sizes.clear();
>>>>> + return;
>>>>> + }
>>>>> +
>>>>> + continue;
>>>>> + }
>>>>> +
>>>>> + // Record the access function for the current subscript.
>>>>> + Subscripts.push_back(R);
>>>>> + }
>>>>> +
>>>>> + // Also push in last position the remainder of the last
>>>>> division: it will be
>>>>> + // the access function of the innermost dimension.
>>>>> + Subscripts.push_back(Res);
>>>>> +
>>>>> + std::reverse(Subscripts.begin(), Subscripts.end());
>>>>> +
>>>>> + LLVM_DEBUG({
>>>>> + dbgs() << "Subscripts:\n";
>>>>> + for (const SCEV *S : Subscripts)
>>>>> + dbgs() << *S << "\n";
>>>>> + });
>>>>> +}
>>>>> +
>>>>> +/// Splits the SCEV into two vectors of SCEVs representing the
>>>>> subscripts and
>>>>> +/// sizes of an array access. Returns the remainder of the
>>>>> delinearization that
>>>>> +/// is the offset start of the array. The SCEV->delinearize
>>>>> algorithm computes
>>>>> +/// the multiples of SCEV coefficients: that is a pattern
>>>>> matching of sub
>>>>> +/// expressions in the stride and base of a SCEV corresponding to
>>>>> the
>>>>> +/// computation of a GCD (greatest common divisor) of base and
>>>>> stride. When
>>>>> +/// SCEV->delinearize fails, it returns the SCEV unchanged.
>>>>> +///
>>>>> +/// For example: when analyzing the memory access A[i][j][k] in
>>>>> this loop nest
>>>>> +///
>>>>> +/// void foo(long n, long m, long o, double A[n][m][o]) {
>>>>> +///
>>>>> +/// for (long i = 0; i < n; i++)
>>>>> +/// for (long j = 0; j < m; j++)
>>>>> +/// for (long k = 0; k < o; k++)
>>>>> +/// A[i][j][k] = 1.0;
>>>>> +/// }
>>>>> +///
>>>>> +/// the delinearization input is the following AddRec SCEV:
>>>>> +///
>>>>> +/// AddRec: {{{%A,+,(8 * %m * %o)}<%for.i>,+,(8 *
>>>>> %o)}<%for.j>,+,8}<%for.k>
>>>>> +///
>>>>> +/// From this SCEV, we are able to say that the base offset of
>>>>> the access is %A
>>>>> +/// because it appears as an offset that does not divide any of
>>>>> the strides in
>>>>> +/// the loops:
>>>>> +///
>>>>> +/// CHECK: Base offset: %A
>>>>> +///
>>>>> +/// and then SCEV->delinearize determines the size of some of the
>>>>> dimensions of
>>>>> +/// the array as these are the multiples by which the strides are
>>>>> happening:
>>>>> +///
>>>>> +/// CHECK: ArrayDecl[UnknownSize][%m][%o] with elements of
>>>>> sizeof(double)
>>>>> +/// bytes.
>>>>> +///
>>>>> +/// Note that the outermost dimension remains of UnknownSize
>>>>> because there are
>>>>> +/// no strides that would help identifying the size of the last
>>>>> dimension: when
>>>>> +/// the array has been statically allocated, one could compute
>>>>> the size of that
>>>>> +/// dimension by dividing the overall size of the array by the
>>>>> size of the known
>>>>> +/// dimensions: %m * %o * 8.
>>>>> +///
>>>>> +/// Finally delinearize provides the access functions for the
>>>>> array reference
>>>>> +/// that does correspond to A[i][j][k] of the above C testcase:
>>>>> +///
>>>>> +/// CHECK:
>>>>> ArrayRef[{0,+,1}<%for.i>][{0,+,1}<%for.j>][{0,+,1}<%for.k>]
>>>>> +///
>>>>> +/// The testcases are checking the output of a function pass:
>>>>> +/// DelinearizationPass that walks through all loads and stores
>>>>> of a function
>>>>> +/// asking for the SCEV of the memory access with respect to all
>>>>> enclosing
>>>>> +/// loops, calling SCEV->delinearize on that and printing the
>>>>> results.
>>>>> +void llvm::delinearize(ScalarEvolution &SE, const SCEV *Expr,
>>>>> + SmallVectorImpl<const SCEV *> &Subscripts,
>>>>> + SmallVectorImpl<const SCEV *> &Sizes,
>>>>> + const SCEV *ElementSize) {
>>>>> + // First step: collect parametric terms.
>>>>> + SmallVector<const SCEV *, 4> Terms;
>>>>> + collectParametricTerms(SE, Expr, Terms);
>>>>> +
>>>>> + if (Terms.empty())
>>>>> + return;
>>>>> +
>>>>> + // Second step: find subscript sizes.
>>>>> + findArrayDimensions(SE, Terms, Sizes, ElementSize);
>>>>> +
>>>>> + if (Sizes.empty())
>>>>> + return;
>>>>> +
>>>>> + // Third step: compute the access functions for each subscript.
>>>>> + computeAccessFunctions(SE, Expr, Subscripts, Sizes);
>>>>> +
>>>>> + if (Subscripts.empty())
>>>>> + return;
>>>>> +
>>>>> + LLVM_DEBUG({
>>>>> + dbgs() << "succeeded to delinearize " << *Expr << "\n";
>>>>> + dbgs() << "ArrayDecl[UnknownSize]";
>>>>> + for (const SCEV *S : Sizes)
>>>>> + dbgs() << "[" << *S << "]";
>>>>> +
>>>>> + dbgs() << "\nArrayRef";
>>>>> + for (const SCEV *S : Subscripts)
>>>>> + dbgs() << "[" << *S << "]";
>>>>> + dbgs() << "\n";
>>>>> + });
>>>>> +}
>>>>> +
>>>>> namespace {
>>>>>
>>>>> class Delinearization : public FunctionPass {
>>>>> @@ -84,7 +533,7 @@ void printDelinearization(raw_ostream &O,
>>>>> Function *F, LoopInfo *LI,
>>>>> O << "AccessFunction: " << *AccessFn << "\n";
>>>>>
>>>>> SmallVector<const SCEV *, 3> Subscripts, Sizes;
>>>>> - SE->delinearize(AccessFn, Subscripts, Sizes,
>>>>> SE->getElementSize(&Inst));
>>>>> + delinearize(*SE, AccessFn, Subscripts, Sizes,
>>>>> SE->getElementSize(&Inst));
>>>>> if (Subscripts.size() == 0 || Sizes.size() == 0 ||
>>>>> Subscripts.size() != Sizes.size()) {
>>>>> O << "failed to delinearize\n";
>>>>>
>>>>> diff --git a/llvm/lib/Analysis/DependenceAnalysis.cpp
>>>>> b/llvm/lib/Analysis/DependenceAnalysis.cpp
>>>>> index 9c09894f3a451..fa9c672119455 100644
>>>>> --- a/llvm/lib/Analysis/DependenceAnalysis.cpp
>>>>> +++ b/llvm/lib/Analysis/DependenceAnalysis.cpp
>>>>> @@ -53,6 +53,7 @@
>>>>> #include "llvm/ADT/STLExtras.h"
>>>>> #include "llvm/ADT/Statistic.h"
>>>>> #include "llvm/Analysis/AliasAnalysis.h"
>>>>> +#include "llvm/Analysis/Delinearization.h"
>>>>> #include "llvm/Analysis/LoopInfo.h"
>>>>> #include "llvm/Analysis/ScalarEvolution.h"
>>>>> #include "llvm/Analysis/ScalarEvolutionExpressions.h"
>>>>> @@ -3439,16 +3440,16 @@ bool
>>>>> DependenceInfo::tryDelinearizeParametricSize(
>>>>>
>>>>> // First step: collect parametric terms in both array
>>>>> references.
>>>>> SmallVector<const SCEV *, 4> Terms;
>>>>> - SE->collectParametricTerms(SrcAR, Terms);
>>>>> - SE->collectParametricTerms(DstAR, Terms);
>>>>> + collectParametricTerms(*SE, SrcAR, Terms);
>>>>> + collectParametricTerms(*SE, DstAR, Terms);
>>>>>
>>>>> // Second step: find subscript sizes.
>>>>> SmallVector<const SCEV *, 4> Sizes;
>>>>> - SE->findArrayDimensions(Terms, Sizes, ElementSize);
>>>>> + findArrayDimensions(*SE, Terms, Sizes, ElementSize);
>>>>>
>>>>> // Third step: compute the access functions for each subscript.
>>>>> - SE->computeAccessFunctions(SrcAR, SrcSubscripts, Sizes);
>>>>> - SE->computeAccessFunctions(DstAR, DstSubscripts, Sizes);
>>>>> + computeAccessFunctions(*SE, SrcAR, SrcSubscripts, Sizes);
>>>>> + computeAccessFunctions(*SE, DstAR, DstSubscripts, Sizes);
>>>>>
>>>>> // Fail when there is only a subscript: that's a linearized
>>>>> access function.
>>>>> if (SrcSubscripts.size() < 2 || DstSubscripts.size() < 2 ||
>>>>>
>>>>> diff --git a/llvm/lib/Analysis/LoopCacheAnalysis.cpp
>>>>> b/llvm/lib/Analysis/LoopCacheAnalysis.cpp
>>>>> index 772e4eb239a9c..75bb8b02286c1 100644
>>>>> --- a/llvm/lib/Analysis/LoopCacheAnalysis.cpp
>>>>> +++ b/llvm/lib/Analysis/LoopCacheAnalysis.cpp
>>>>> @@ -30,6 +30,7 @@
>>>>> #include "llvm/ADT/Sequence.h"
>>>>> #include "llvm/ADT/SmallVector.h"
>>>>> #include "llvm/Analysis/AliasAnalysis.h"
>>>>> +#include "llvm/Analysis/Delinearization.h"
>>>>> #include "llvm/Analysis/DependenceAnalysis.h"
>>>>> #include "llvm/Analysis/LoopInfo.h"
>>>>> #include "llvm/Analysis/ScalarEvolutionExpressions.h"
>>>>> @@ -344,8 +345,8 @@ bool IndexedReference::delinearize(const
>>>>> LoopInfo &LI) {
>>>>> LLVM_DEBUG(dbgs().indent(2) << "In Loop '" << L->getName()
>>>>> << "', AccessFn: " << *AccessFn
>>>>> << "\n");
>>>>>
>>>>> - SE.delinearize(AccessFn, Subscripts, Sizes,
>>>>> - SE.getElementSize(&StoreOrLoadInst));
>>>>> + llvm::delinearize(SE, AccessFn, Subscripts, Sizes,
>>>>> + SE.getElementSize(&StoreOrLoadInst));
>>>>>
>>>>> if (Subscripts.empty() || Sizes.empty() ||
>>>>> Subscripts.size() != Sizes.size()) {
>>>>>
>>>>> diff --git a/llvm/lib/Analysis/ScalarEvolution.cpp
>>>>> b/llvm/lib/Analysis/ScalarEvolution.cpp
>>>>> index 50901f3524d1b..151883ab67367 100644
>>>>> --- a/llvm/lib/Analysis/ScalarEvolution.cpp
>>>>> +++ b/llvm/lib/Analysis/ScalarEvolution.cpp
>>>>> @@ -12187,237 +12187,6 @@ static inline bool containsUndefs(const
>>>>> SCEV *S) {
>>>>> });
>>>>> }
>>>>>
>>>>> -namespace {
>>>>> -
>>>>> -// Collect all steps of SCEV expressions.
>>>>> -struct SCEVCollectStrides {
>>>>> - ScalarEvolution &SE;
>>>>> - SmallVectorImpl<const SCEV *> &Strides;
>>>>> -
>>>>> - SCEVCollectStrides(ScalarEvolution &SE, SmallVectorImpl<const
>>>>> SCEV *> &S)
>>>>> - : SE(SE), Strides(S) {}
>>>>> -
>>>>> - bool follow(const SCEV *S) {
>>>>> - if (const SCEVAddRecExpr *AR = dyn_cast<SCEVAddRecExpr>(S))
>>>>> - Strides.push_back(AR->getStepRecurrence(SE));
>>>>> - return true;
>>>>> - }
>>>>> -
>>>>> - bool isDone() const { return false; }
>>>>> -};
>>>>> -
>>>>> -// Collect all SCEVUnknown and SCEVMulExpr expressions.
>>>>> -struct SCEVCollectTerms {
>>>>> - SmallVectorImpl<const SCEV *> &Terms;
>>>>> -
>>>>> - SCEVCollectTerms(SmallVectorImpl<const SCEV *> &T) : Terms(T) {}
>>>>> -
>>>>> - bool follow(const SCEV *S) {
>>>>> - if (isa<SCEVUnknown>(S) || isa<SCEVMulExpr>(S) ||
>>>>> - isa<SCEVSignExtendExpr>(S)) {
>>>>> - if (!containsUndefs(S))
>>>>> - Terms.push_back(S);
>>>>> -
>>>>> - // Stop recursion: once we collected a term, do not walk
>>>>> its operands.
>>>>> - return false;
>>>>> - }
>>>>> -
>>>>> - // Keep looking.
>>>>> - return true;
>>>>> - }
>>>>> -
>>>>> - bool isDone() const { return false; }
>>>>> -};
>>>>> -
>>>>> -// Check if a SCEV contains an AddRecExpr.
>>>>> -struct SCEVHasAddRec {
>>>>> - bool &ContainsAddRec;
>>>>> -
>>>>> - SCEVHasAddRec(bool &ContainsAddRec) :
>>>>> ContainsAddRec(ContainsAddRec) {
>>>>> - ContainsAddRec = false;
>>>>> - }
>>>>> -
>>>>> - bool follow(const SCEV *S) {
>>>>> - if (isa<SCEVAddRecExpr>(S)) {
>>>>> - ContainsAddRec = true;
>>>>> -
>>>>> - // Stop recursion: once we collected a term, do not walk
>>>>> its operands.
>>>>> - return false;
>>>>> - }
>>>>> -
>>>>> - // Keep looking.
>>>>> - return true;
>>>>> - }
>>>>> -
>>>>> - bool isDone() const { return false; }
>>>>> -};
>>>>> -
>>>>> -// Find factors that are multiplied with an expression that
>>>>> (possibly as a
>>>>> -// subexpression) contains an AddRecExpr. In the expression:
>>>>> -//
>>>>> -// 8 * (100 + %p * %q * (%a + {0, +, 1}_loop))
>>>>> -//
>>>>> -// "%p * %q" are factors multiplied by the expression "(%a + {0,
>>>>> +, 1}_loop)"
>>>>> -// that contains the AddRec {0, +, 1}_loop. %p * %q are likely to
>>>>> be array size
>>>>> -// parameters as they form a product with an induction variable.
>>>>> -//
>>>>> -// This collector expects all array size parameters to be in the
>>>>> same MulExpr.
>>>>> -// It might be necessary to later add support for collecting
>>>>> parameters that are
>>>>> -// spread over
>>>>> diff erent nested MulExpr.
>>>>> -struct SCEVCollectAddRecMultiplies {
>>>>> - SmallVectorImpl<const SCEV *> &Terms;
>>>>> - ScalarEvolution &SE;
>>>>> -
>>>>> - SCEVCollectAddRecMultiplies(SmallVectorImpl<const SCEV *> &T,
>>>>> ScalarEvolution &SE)
>>>>> - : Terms(T), SE(SE) {}
>>>>> -
>>>>> - bool follow(const SCEV *S) {
>>>>> - if (auto *Mul = dyn_cast<SCEVMulExpr>(S)) {
>>>>> - bool HasAddRec = false;
>>>>> - SmallVector<const SCEV *, 0> Operands;
>>>>> - for (auto Op : Mul->operands()) {
>>>>> - const SCEVUnknown *Unknown = dyn_cast<SCEVUnknown>(Op);
>>>>> - if (Unknown && !isa<CallInst>(Unknown->getValue())) {
>>>>> - Operands.push_back(Op);
>>>>> - } else if (Unknown) {
>>>>> - HasAddRec = true;
>>>>> - } else {
>>>>> - bool ContainsAddRec = false;
>>>>> - SCEVHasAddRec ContiansAddRec(ContainsAddRec);
>>>>> - visitAll(Op, ContiansAddRec);
>>>>> - HasAddRec |= ContainsAddRec;
>>>>> - }
>>>>> - }
>>>>> - if (Operands.size() == 0)
>>>>> - return true;
>>>>> -
>>>>> - if (!HasAddRec)
>>>>> - return false;
>>>>> -
>>>>> - Terms.push_back(SE.getMulExpr(Operands));
>>>>> - // Stop recursion: once we collected a term, do not walk
>>>>> its operands.
>>>>> - return false;
>>>>> - }
>>>>> -
>>>>> - // Keep looking.
>>>>> - return true;
>>>>> - }
>>>>> -
>>>>> - bool isDone() const { return false; }
>>>>> -};
>>>>> -
>>>>> -} // end anonymous namespace
>>>>> -
>>>>> -/// Find parametric terms in this SCEVAddRecExpr. We first for
>>>>> parameters in
>>>>> -/// two places:
>>>>> -/// 1) The strides of AddRec expressions.
>>>>> -/// 2) Unknowns that are multiplied with AddRec expressions.
>>>>> -void ScalarEvolution::collectParametricTerms(const SCEV *Expr,
>>>>> - SmallVectorImpl<const SCEV *> &Terms) {
>>>>> - SmallVector<const SCEV *, 4> Strides;
>>>>> - SCEVCollectStrides StrideCollector(*this, Strides);
>>>>> - visitAll(Expr, StrideCollector);
>>>>> -
>>>>> - LLVM_DEBUG({
>>>>> - dbgs() << "Strides:\n";
>>>>> - for (const SCEV *S : Strides)
>>>>> - dbgs() << *S << "\n";
>>>>> - });
>>>>> -
>>>>> - for (const SCEV *S : Strides) {
>>>>> - SCEVCollectTerms TermCollector(Terms);
>>>>> - visitAll(S, TermCollector);
>>>>> - }
>>>>> -
>>>>> - LLVM_DEBUG({
>>>>> - dbgs() << "Terms:\n";
>>>>> - for (const SCEV *T : Terms)
>>>>> - dbgs() << *T << "\n";
>>>>> - });
>>>>> -
>>>>> - SCEVCollectAddRecMultiplies MulCollector(Terms, *this);
>>>>> - visitAll(Expr, MulCollector);
>>>>> -}
>>>>> -
>>>>> -static bool findArrayDimensionsRec(ScalarEvolution &SE,
>>>>> - SmallVectorImpl<const SCEV *> &Terms,
>>>>> - SmallVectorImpl<const SCEV *> &Sizes) {
>>>>> - int Last = Terms.size() - 1;
>>>>> - const SCEV *Step = Terms[Last];
>>>>> -
>>>>> - // End of recursion.
>>>>> - if (Last == 0) {
>>>>> - if (const SCEVMulExpr *M = dyn_cast<SCEVMulExpr>(Step)) {
>>>>> - SmallVector<const SCEV *, 2> Qs;
>>>>> - for (const SCEV *Op : M->operands())
>>>>> - if (!isa<SCEVConstant>(Op))
>>>>> - Qs.push_back(Op);
>>>>> -
>>>>> - Step = SE.getMulExpr(Qs);
>>>>> - }
>>>>> -
>>>>> - Sizes.push_back(Step);
>>>>> - return true;
>>>>> - }
>>>>> -
>>>>> - for (const SCEV *&Term : Terms) {
>>>>> - // Normalize the terms before the next call to
>>>>> findArrayDimensionsRec.
>>>>> - const SCEV *Q, *R;
>>>>> - SCEVDivision::divide(SE, Term, Step, &Q, &R);
>>>>> -
>>>>> - // Bail out when GCD does not evenly divide one of the terms.
>>>>> - if (!R->isZero())
>>>>> - return false;
>>>>> -
>>>>> - Term = Q;
>>>>> - }
>>>>> -
>>>>> - // Remove all SCEVConstants.
>>>>> - erase_if(Terms, [](const SCEV *E) { return
>>>>> isa<SCEVConstant>(E); });
>>>>> -
>>>>> - if (Terms.size() > 0)
>>>>> - if (!findArrayDimensionsRec(SE, Terms, Sizes))
>>>>> - return false;
>>>>> -
>>>>> - Sizes.push_back(Step);
>>>>> - return true;
>>>>> -}
>>>>> -
>>>>> -// Returns true when one of the SCEVs of Terms contains a
>>>>> SCEVUnknown parameter.
>>>>> -static inline bool containsParameters(SmallVectorImpl<const SCEV
>>>>> *> &Terms) {
>>>>> - for (const SCEV *T : Terms)
>>>>> - if (SCEVExprContains(T, [](const SCEV *S) { return
>>>>> isa<SCEVUnknown>(S); }))
>>>>> - return true;
>>>>> -
>>>>> - return false;
>>>>> -}
>>>>> -
>>>>> -// Return the number of product terms in S.
>>>>> -static inline int numberOfTerms(const SCEV *S) {
>>>>> - if (const SCEVMulExpr *Expr = dyn_cast<SCEVMulExpr>(S))
>>>>> - return Expr->getNumOperands();
>>>>> - return 1;
>>>>> -}
>>>>> -
>>>>> -static const SCEV *removeConstantFactors(ScalarEvolution &SE,
>>>>> const SCEV *T) {
>>>>> - if (isa<SCEVConstant>(T))
>>>>> - return nullptr;
>>>>> -
>>>>> - if (isa<SCEVUnknown>(T))
>>>>> - return T;
>>>>> -
>>>>> - if (const SCEVMulExpr *M = dyn_cast<SCEVMulExpr>(T)) {
>>>>> - SmallVector<const SCEV *, 2> Factors;
>>>>> - for (const SCEV *Op : M->operands())
>>>>> - if (!isa<SCEVConstant>(Op))
>>>>> - Factors.push_back(Op);
>>>>> -
>>>>> - return SE.getMulExpr(Factors);
>>>>> - }
>>>>> -
>>>>> - return T;
>>>>> -}
>>>>> -
>>>>> /// Return the size of an element read or written by Inst.
>>>>> const SCEV *ScalarEvolution::getElementSize(Instruction *Inst) {
>>>>> Type *Ty;
>>>>> @@ -12432,211 +12201,6 @@ const SCEV
>>>>> *ScalarEvolution::getElementSize(Instruction *Inst) {
>>>>> return getSizeOfExpr(ETy, Ty);
>>>>> }
>>>>>
>>>>> -void ScalarEvolution::findArrayDimensions(SmallVectorImpl<const
>>>>> SCEV *> &Terms,
>>>>> - SmallVectorImpl<const SCEV *> &Sizes,
>>>>> - const SCEV *ElementSize) {
>>>>> - if (Terms.size() < 1 || !ElementSize)
>>>>> - return;
>>>>> -
>>>>> - // Early return when Terms do not contain parameters: we do not
>>>>> delinearize
>>>>> - // non parametric SCEVs.
>>>>> - if (!containsParameters(Terms))
>>>>> - return;
>>>>> -
>>>>> - LLVM_DEBUG({
>>>>> - dbgs() << "Terms:\n";
>>>>> - for (const SCEV *T : Terms)
>>>>> - dbgs() << *T << "\n";
>>>>> - });
>>>>> -
>>>>> - // Remove duplicates.
>>>>> - array_pod_sort(Terms.begin(), Terms.end());
>>>>> - Terms.erase(std::unique(Terms.begin(), Terms.end()), Terms.end());
>>>>> -
>>>>> - // Put larger terms first.
>>>>> - llvm::sort(Terms, [](const SCEV *LHS, const SCEV *RHS) {
>>>>> - return numberOfTerms(LHS) > numberOfTerms(RHS);
>>>>> - });
>>>>> -
>>>>> - // Try to divide all terms by the element size. If term is not
>>>>> divisible by
>>>>> - // element size, proceed with the original term.
>>>>> - for (const SCEV *&Term : Terms) {
>>>>> - const SCEV *Q, *R;
>>>>> - SCEVDivision::divide(*this, Term, ElementSize, &Q, &R);
>>>>> - if (!Q->isZero())
>>>>> - Term = Q;
>>>>> - }
>>>>> -
>>>>> - SmallVector<const SCEV *, 4> NewTerms;
>>>>> -
>>>>> - // Remove constant factors.
>>>>> - for (const SCEV *T : Terms)
>>>>> - if (const SCEV *NewT = removeConstantFactors(*this, T))
>>>>> - NewTerms.push_back(NewT);
>>>>> -
>>>>> - LLVM_DEBUG({
>>>>> - dbgs() << "Terms after sorting:\n";
>>>>> - for (const SCEV *T : NewTerms)
>>>>> - dbgs() << *T << "\n";
>>>>> - });
>>>>> -
>>>>> - if (NewTerms.empty() || !findArrayDimensionsRec(*this,
>>>>> NewTerms, Sizes)) {
>>>>> - Sizes.clear();
>>>>> - return;
>>>>> - }
>>>>> -
>>>>> - // The last element to be pushed into Sizes is the size of an
>>>>> element.
>>>>> - Sizes.push_back(ElementSize);
>>>>> -
>>>>> - LLVM_DEBUG({
>>>>> - dbgs() << "Sizes:\n";
>>>>> - for (const SCEV *S : Sizes)
>>>>> - dbgs() << *S << "\n";
>>>>> - });
>>>>> -}
>>>>> -
>>>>> -void ScalarEvolution::computeAccessFunctions(
>>>>> - const SCEV *Expr, SmallVectorImpl<const SCEV *> &Subscripts,
>>>>> - SmallVectorImpl<const SCEV *> &Sizes) {
>>>>> - // Early exit in case this SCEV is not an affine multivariate
>>>>> function.
>>>>> - if (Sizes.empty())
>>>>> - return;
>>>>> -
>>>>> - if (auto *AR = dyn_cast<SCEVAddRecExpr>(Expr))
>>>>> - if (!AR->isAffine())
>>>>> - return;
>>>>> -
>>>>> - const SCEV *Res = Expr;
>>>>> - int Last = Sizes.size() - 1;
>>>>> - for (int i = Last; i >= 0; i--) {
>>>>> - const SCEV *Q, *R;
>>>>> - SCEVDivision::divide(*this, Res, Sizes[i], &Q, &R);
>>>>> -
>>>>> - LLVM_DEBUG({
>>>>> - dbgs() << "Res: " << *Res << "\n";
>>>>> - dbgs() << "Sizes[i]: " << *Sizes[i] << "\n";
>>>>> - dbgs() << "Res divided by Sizes[i]:\n";
>>>>> - dbgs() << "Quotient: " << *Q << "\n";
>>>>> - dbgs() << "Remainder: " << *R << "\n";
>>>>> - });
>>>>> -
>>>>> - Res = Q;
>>>>> -
>>>>> - // Do not record the last subscript corresponding to the size
>>>>> of elements in
>>>>> - // the array.
>>>>> - if (i == Last) {
>>>>> -
>>>>> - // Bail out if the remainder is too complex.
>>>>> - if (isa<SCEVAddRecExpr>(R)) {
>>>>> - Subscripts.clear();
>>>>> - Sizes.clear();
>>>>> - return;
>>>>> - }
>>>>> -
>>>>> - continue;
>>>>> - }
>>>>> -
>>>>> - // Record the access function for the current subscript.
>>>>> - Subscripts.push_back(R);
>>>>> - }
>>>>> -
>>>>> - // Also push in last position the remainder of the last
>>>>> division: it will be
>>>>> - // the access function of the innermost dimension.
>>>>> - Subscripts.push_back(Res);
>>>>> -
>>>>> - std::reverse(Subscripts.begin(), Subscripts.end());
>>>>> -
>>>>> - LLVM_DEBUG({
>>>>> - dbgs() << "Subscripts:\n";
>>>>> - for (const SCEV *S : Subscripts)
>>>>> - dbgs() << *S << "\n";
>>>>> - });
>>>>> -}
>>>>> -
>>>>> -/// Splits the SCEV into two vectors of SCEVs representing the
>>>>> subscripts and
>>>>> -/// sizes of an array access. Returns the remainder of the
>>>>> delinearization that
>>>>> -/// is the offset start of the array. The SCEV->delinearize
>>>>> algorithm computes
>>>>> -/// the multiples of SCEV coefficients: that is a pattern
>>>>> matching of sub
>>>>> -/// expressions in the stride and base of a SCEV corresponding to
>>>>> the
>>>>> -/// computation of a GCD (greatest common divisor) of base and
>>>>> stride. When
>>>>> -/// SCEV->delinearize fails, it returns the SCEV unchanged.
>>>>> -///
>>>>> -/// For example: when analyzing the memory access A[i][j][k] in
>>>>> this loop nest
>>>>> -///
>>>>> -/// void foo(long n, long m, long o, double A[n][m][o]) {
>>>>> -///
>>>>> -/// for (long i = 0; i < n; i++)
>>>>> -/// for (long j = 0; j < m; j++)
>>>>> -/// for (long k = 0; k < o; k++)
>>>>> -/// A[i][j][k] = 1.0;
>>>>> -/// }
>>>>> -///
>>>>> -/// the delinearization input is the following AddRec SCEV:
>>>>> -///
>>>>> -/// AddRec: {{{%A,+,(8 * %m * %o)}<%for.i>,+,(8 *
>>>>> %o)}<%for.j>,+,8}<%for.k>
>>>>> -///
>>>>> -/// From this SCEV, we are able to say that the base offset of
>>>>> the access is %A
>>>>> -/// because it appears as an offset that does not divide any of
>>>>> the strides in
>>>>> -/// the loops:
>>>>> -///
>>>>> -/// CHECK: Base offset: %A
>>>>> -///
>>>>> -/// and then SCEV->delinearize determines the size of some of the
>>>>> dimensions of
>>>>> -/// the array as these are the multiples by which the strides are
>>>>> happening:
>>>>> -///
>>>>> -/// CHECK: ArrayDecl[UnknownSize][%m][%o] with elements of
>>>>> sizeof(double) bytes.
>>>>> -///
>>>>> -/// Note that the outermost dimension remains of UnknownSize
>>>>> because there are
>>>>> -/// no strides that would help identifying the size of the last
>>>>> dimension: when
>>>>> -/// the array has been statically allocated, one could compute
>>>>> the size of that
>>>>> -/// dimension by dividing the overall size of the array by the
>>>>> size of the known
>>>>> -/// dimensions: %m * %o * 8.
>>>>> -///
>>>>> -/// Finally delinearize provides the access functions for the
>>>>> array reference
>>>>> -/// that does correspond to A[i][j][k] of the above C testcase:
>>>>> -///
>>>>> -/// CHECK:
>>>>> ArrayRef[{0,+,1}<%for.i>][{0,+,1}<%for.j>][{0,+,1}<%for.k>]
>>>>> -///
>>>>> -/// The testcases are checking the output of a function pass:
>>>>> -/// DelinearizationPass that walks through all loads and stores
>>>>> of a function
>>>>> -/// asking for the SCEV of the memory access with respect to all
>>>>> enclosing
>>>>> -/// loops, calling SCEV->delinearize on that and printing the
>>>>> results.
>>>>> -void ScalarEvolution::delinearize(const SCEV *Expr,
>>>>> - SmallVectorImpl<const SCEV *>
>>>>> &Subscripts,
>>>>> - SmallVectorImpl<const SCEV *>
>>>>> &Sizes,
>>>>> - const SCEV *ElementSize) {
>>>>> - // First step: collect parametric terms.
>>>>> - SmallVector<const SCEV *, 4> Terms;
>>>>> - collectParametricTerms(Expr, Terms);
>>>>> -
>>>>> - if (Terms.empty())
>>>>> - return;
>>>>> -
>>>>> - // Second step: find subscript sizes.
>>>>> - findArrayDimensions(Terms, Sizes, ElementSize);
>>>>> -
>>>>> - if (Sizes.empty())
>>>>> - return;
>>>>> -
>>>>> - // Third step: compute the access functions for each subscript.
>>>>> - computeAccessFunctions(Expr, Subscripts, Sizes);
>>>>> -
>>>>> - if (Subscripts.empty())
>>>>> - return;
>>>>> -
>>>>> - LLVM_DEBUG({
>>>>> - dbgs() << "succeeded to delinearize " << *Expr << "\n";
>>>>> - dbgs() << "ArrayDecl[UnknownSize]";
>>>>> - for (const SCEV *S : Sizes)
>>>>> - dbgs() << "[" << *S << "]";
>>>>> -
>>>>> - dbgs() << "\nArrayRef";
>>>>> - for (const SCEV *S : Subscripts)
>>>>> - dbgs() << "[" << *S << "]";
>>>>> - dbgs() << "\n";
>>>>> - });
>>>>> -}
>>>>> -
>>>>> bool ScalarEvolution::getIndexExpressionsFromGEP(
>>>>> const GetElementPtrInst *GEP, SmallVectorImpl<const SCEV *>
>>>>> &Subscripts,
>>>>> SmallVectorImpl<int> &Sizes) {
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> llvm-commits mailing list
>>>>> llvm-commits at lists.llvm.org
>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
More information about the llvm-commits
mailing list