[llvm] 585c594 - Move delinearization logic out of SCEV [NFC]
Philip Reames via llvm-commits
llvm-commits at lists.llvm.org
Wed Sep 8 16:20:29 PDT 2021
Michael,
Your first paragraph comes off as rather dismissive of both my judgement
and the value of incrementalism in general. I'm going to assume that
wasn't your intent, just pointing it out for your reference.
I will move getIndexExpressionsFromGEP as requested.
On the second point, feel free to contribute your own time towards the
proposed enhancement. I don't disagree with anything you said, I'm just
not motivated to do it.
Philip
On 9/8/21 4:12 PM, Michael Kruse wrote:
> If there was a reason to move delinearize() out of ScEv, the same
> reason applies to getIndexExpressionsFromGEP ("delinearize using
> GetElementPtr as hint") as well. I would prefer to keep them together.
>
> getElementSize on the other side might have general usefulness as
> well. However, I think it is badly named (better: "getAccessedSize")
> and not only loads and stores access memory, some intrinsics do as
> well and it is up to the client what that can support. For instance,
> Polly also supports memset, memcpy, memove.
>
> Michael
>
>
> Am Mi., 8. Sept. 2021 um 16:50 Uhr schrieb Philip Reames
> <listmail at philipreames.com>:
>> They weren't part of the tree rooted in delinearize. We could move them
>> or not, either seems reasonable.
>>
>> Philip
>>
>> On 9/8/21 1:05 PM, Michael Kruse wrote:
>>> You left getIndexExpressionsFromGEP inside ScalarEvolution. Don't you
>>> consider it part of Delinearization? getElementSize is also only used
>>> in delinearization use cases/was introduced for it.
>>>
>>> Michael
>>>
>>> Am Mi., 8. Sept. 2021 um 14:28 Uhr schrieb Philip Reames via
>>> llvm-commits <llvm-commits at lists.llvm.org>:
>>>> Author: Philip Reames
>>>> Date: 2021-09-08T12:28:35-07:00
>>>> New Revision: 585c594d749a2a88150b63804587af85abdabeaa
>>>>
>>>> URL: https://github.com/llvm/llvm-project/commit/585c594d749a2a88150b63804587af85abdabeaa
>>>> DIFF: https://github.com/llvm/llvm-project/commit/585c594d749a2a88150b63804587af85abdabeaa.diff
>>>>
>>>> LOG: Move delinearization logic out of SCEV [NFC]
>>>>
>>>> None of this logic has anything to do with SCEV's internals, it just uses the existing public APIs. As a result, we can move the code from ScalarEvolution.cpp/hpp to Delinearization.cpp/hpp with only minor changes.
>>>>
>>>> This was discussed in advance on today's loop opt call. It turned out to be easy as hoped.
>>>>
>>>> Added:
>>>>
>>>>
>>>> Modified:
>>>> llvm/include/llvm/Analysis/Delinearization.h
>>>> llvm/include/llvm/Analysis/ScalarEvolution.h
>>>> llvm/lib/Analysis/Delinearization.cpp
>>>> llvm/lib/Analysis/DependenceAnalysis.cpp
>>>> llvm/lib/Analysis/LoopCacheAnalysis.cpp
>>>> llvm/lib/Analysis/ScalarEvolution.cpp
>>>>
>>>> Removed:
>>>>
>>>>
>>>>
>>>> ################################################################################
>>>> diff --git a/llvm/include/llvm/Analysis/Delinearization.h b/llvm/include/llvm/Analysis/Delinearization.h
>>>> index 2658b6bbc80c0..dec3fd9b08a9f 100644
>>>> --- a/llvm/include/llvm/Analysis/Delinearization.h
>>>> +++ b/llvm/include/llvm/Analysis/Delinearization.h
>>>> @@ -16,10 +16,100 @@
>>>> #ifndef LLVM_ANALYSIS_DELINEARIZATION_H
>>>> #define LLVM_ANALYSIS_DELINEARIZATION_H
>>>>
>>>> +#include "llvm/ADT/SmallVector.h"
>>>> #include "llvm/IR/PassManager.h"
>>>> #include "llvm/Support/raw_ostream.h"
>>>>
>>>> namespace llvm {
>>>> +class ScalarEvolution;
>>>> +class SCEV;
>>>> +
>>>> +/// Compute the array dimensions Sizes from the set of Terms extracted from
>>>> +/// the memory access function of this SCEVAddRecExpr (second step of
>>>> +/// delinearization).
>>>> +void findArrayDimensions(ScalarEvolution &SE,
>>>> + SmallVectorImpl<const SCEV *> &Terms,
>>>> + SmallVectorImpl<const SCEV *> &Sizes,
>>>> + const SCEV *ElementSize);
>>>> +
>>>> +/// Collect parametric terms occurring in step expressions (first step of
>>>> +/// delinearization).
>>>> +void collectParametricTerms(ScalarEvolution &SE, const SCEV *Expr,
>>>> + SmallVectorImpl<const SCEV *> &Terms);
>>>> +
>>>> +/// Return in Subscripts the access functions for each dimension in Sizes
>>>> +/// (third step of delinearization).
>>>> +void computeAccessFunctions(ScalarEvolution &SE, const SCEV *Expr,
>>>> + SmallVectorImpl<const SCEV *> &Subscripts,
>>>> + SmallVectorImpl<const SCEV *> &Sizes);
>>>> +/// Split this SCEVAddRecExpr into two vectors of SCEVs representing the
>>>> +/// subscripts and sizes of an array access.
>>>> +///
>>>> +/// The delinearization is a 3 step process: the first two steps compute the
>>>> +/// sizes of each subscript and the third step computes the access functions
>>>> +/// for the delinearized array:
>>>> +///
>>>> +/// 1. Find the terms in the step functions
>>>> +/// 2. Compute the array size
>>>> +/// 3. Compute the access function: divide the SCEV by the array size
>>>> +/// starting with the innermost dimensions found in step 2. The Quotient
>>>> +/// is the SCEV to be divided in the next step of the recursion. The
>>>> +/// Remainder is the subscript of the innermost dimension. Loop over all
>>>> +/// array dimensions computed in step 2.
>>>> +///
>>>> +/// To compute a uniform array size for several memory accesses to the same
>>>> +/// object, one can collect in step 1 all the step terms for all the memory
>>>> +/// accesses, and compute in step 2 a unique array shape. This guarantees
>>>> +/// that the array shape will be the same across all memory accesses.
>>>> +///
>>>> +/// FIXME: We could derive the result of steps 1 and 2 from a description of
>>>> +/// the array shape given in metadata.
>>>> +///
>>>> +/// Example:
>>>> +///
>>>> +/// A[][n][m]
>>>> +///
>>>> +/// for i
>>>> +/// for j
>>>> +/// for k
>>>> +/// A[j+k][2i][5i] =
>>>> +///
>>>> +/// The initial SCEV:
>>>> +///
>>>> +/// A[{{{0,+,2*m+5}_i, +, n*m}_j, +, n*m}_k]
>>>> +///
>>>> +/// 1. Find the
>>>> diff erent terms in the step functions:
>>>> +/// -> [2*m, 5, n*m, n*m]
>>>> +///
>>>> +/// 2. Compute the array size: sort and unique them
>>>> +/// -> [n*m, 2*m, 5]
>>>> +/// find the GCD of all the terms = 1
>>>> +/// divide by the GCD and erase constant terms
>>>> +/// -> [n*m, 2*m]
>>>> +/// GCD = m
>>>> +/// divide by GCD -> [n, 2]
>>>> +/// remove constant terms
>>>> +/// -> [n]
>>>> +/// size of the array is A[unknown][n][m]
>>>> +///
>>>> +/// 3. Compute the access function
>>>> +/// a. Divide {{{0,+,2*m+5}_i, +, n*m}_j, +, n*m}_k by the innermost size m
>>>> +/// Quotient: {{{0,+,2}_i, +, n}_j, +, n}_k
>>>> +/// Remainder: {{{0,+,5}_i, +, 0}_j, +, 0}_k
>>>> +/// The remainder is the subscript of the innermost array dimension: [5i].
>>>> +///
>>>> +/// b. Divide Quotient: {{{0,+,2}_i, +, n}_j, +, n}_k by next outer size n
>>>> +/// Quotient: {{{0,+,0}_i, +, 1}_j, +, 1}_k
>>>> +/// Remainder: {{{0,+,2}_i, +, 0}_j, +, 0}_k
>>>> +/// The Remainder is the subscript of the next array dimension: [2i].
>>>> +///
>>>> +/// The subscript of the outermost dimension is the Quotient: [j+k].
>>>> +///
>>>> +/// Overall, we have: A[][n][m], and the access function: A[j+k][2i][5i].
>>>> +void delinearize(ScalarEvolution &SE, const SCEV *Expr,
>>>> + SmallVectorImpl<const SCEV *> &Subscripts,
>>>> + SmallVectorImpl<const SCEV *> &Sizes, const SCEV *ElementSize);
>>>> +
>>>> struct DelinearizationPrinterPass
>>>> : public PassInfoMixin<DelinearizationPrinterPass> {
>>>> explicit DelinearizationPrinterPass(raw_ostream &OS);
>>>>
>>>> diff --git a/llvm/include/llvm/Analysis/ScalarEvolution.h b/llvm/include/llvm/Analysis/ScalarEvolution.h
>>>> index 6ae865e011cf0..445a925c70e83 100644
>>>> --- a/llvm/include/llvm/Analysis/ScalarEvolution.h
>>>> +++ b/llvm/include/llvm/Analysis/ScalarEvolution.h
>>>> @@ -1099,29 +1099,11 @@ class ScalarEvolution {
>>>> /// Return the size of an element read or written by Inst.
>>>> const SCEV *getElementSize(Instruction *Inst);
>>>>
>>>> - /// Compute the array dimensions Sizes from the set of Terms extracted from
>>>> - /// the memory access function of this SCEVAddRecExpr (second step of
>>>> - /// delinearization).
>>>> - void findArrayDimensions(SmallVectorImpl<const SCEV *> &Terms,
>>>> - SmallVectorImpl<const SCEV *> &Sizes,
>>>> - const SCEV *ElementSize);
>>>> -
>>>> void print(raw_ostream &OS) const;
>>>> void verify() const;
>>>> bool invalidate(Function &F, const PreservedAnalyses &PA,
>>>> FunctionAnalysisManager::Invalidator &Inv);
>>>>
>>>> - /// Collect parametric terms occurring in step expressions (first step of
>>>> - /// delinearization).
>>>> - void collectParametricTerms(const SCEV *Expr,
>>>> - SmallVectorImpl<const SCEV *> &Terms);
>>>> -
>>>> - /// Return in Subscripts the access functions for each dimension in Sizes
>>>> - /// (third step of delinearization).
>>>> - void computeAccessFunctions(const SCEV *Expr,
>>>> - SmallVectorImpl<const SCEV *> &Subscripts,
>>>> - SmallVectorImpl<const SCEV *> &Sizes);
>>>> -
>>>> /// Gathers the individual index expressions from a GEP instruction.
>>>> ///
>>>> /// This function optimistically assumes the GEP references into a fixed size
>>>> @@ -1135,74 +1117,6 @@ class ScalarEvolution {
>>>> SmallVectorImpl<const SCEV *> &Subscripts,
>>>> SmallVectorImpl<int> &Sizes);
>>>>
>>>> - /// Split this SCEVAddRecExpr into two vectors of SCEVs representing the
>>>> - /// subscripts and sizes of an array access.
>>>> - ///
>>>> - /// The delinearization is a 3 step process: the first two steps compute the
>>>> - /// sizes of each subscript and the third step computes the access functions
>>>> - /// for the delinearized array:
>>>> - ///
>>>> - /// 1. Find the terms in the step functions
>>>> - /// 2. Compute the array size
>>>> - /// 3. Compute the access function: divide the SCEV by the array size
>>>> - /// starting with the innermost dimensions found in step 2. The Quotient
>>>> - /// is the SCEV to be divided in the next step of the recursion. The
>>>> - /// Remainder is the subscript of the innermost dimension. Loop over all
>>>> - /// array dimensions computed in step 2.
>>>> - ///
>>>> - /// To compute a uniform array size for several memory accesses to the same
>>>> - /// object, one can collect in step 1 all the step terms for all the memory
>>>> - /// accesses, and compute in step 2 a unique array shape. This guarantees
>>>> - /// that the array shape will be the same across all memory accesses.
>>>> - ///
>>>> - /// FIXME: We could derive the result of steps 1 and 2 from a description of
>>>> - /// the array shape given in metadata.
>>>> - ///
>>>> - /// Example:
>>>> - ///
>>>> - /// A[][n][m]
>>>> - ///
>>>> - /// for i
>>>> - /// for j
>>>> - /// for k
>>>> - /// A[j+k][2i][5i] =
>>>> - ///
>>>> - /// The initial SCEV:
>>>> - ///
>>>> - /// A[{{{0,+,2*m+5}_i, +, n*m}_j, +, n*m}_k]
>>>> - ///
>>>> - /// 1. Find the
>>>> diff erent terms in the step functions:
>>>> - /// -> [2*m, 5, n*m, n*m]
>>>> - ///
>>>> - /// 2. Compute the array size: sort and unique them
>>>> - /// -> [n*m, 2*m, 5]
>>>> - /// find the GCD of all the terms = 1
>>>> - /// divide by the GCD and erase constant terms
>>>> - /// -> [n*m, 2*m]
>>>> - /// GCD = m
>>>> - /// divide by GCD -> [n, 2]
>>>> - /// remove constant terms
>>>> - /// -> [n]
>>>> - /// size of the array is A[unknown][n][m]
>>>> - ///
>>>> - /// 3. Compute the access function
>>>> - /// a. Divide {{{0,+,2*m+5}_i, +, n*m}_j, +, n*m}_k by the innermost size m
>>>> - /// Quotient: {{{0,+,2}_i, +, n}_j, +, n}_k
>>>> - /// Remainder: {{{0,+,5}_i, +, 0}_j, +, 0}_k
>>>> - /// The remainder is the subscript of the innermost array dimension: [5i].
>>>> - ///
>>>> - /// b. Divide Quotient: {{{0,+,2}_i, +, n}_j, +, n}_k by next outer size n
>>>> - /// Quotient: {{{0,+,0}_i, +, 1}_j, +, 1}_k
>>>> - /// Remainder: {{{0,+,2}_i, +, 0}_j, +, 0}_k
>>>> - /// The Remainder is the subscript of the next array dimension: [2i].
>>>> - ///
>>>> - /// The subscript of the outermost dimension is the Quotient: [j+k].
>>>> - ///
>>>> - /// Overall, we have: A[][n][m], and the access function: A[j+k][2i][5i].
>>>> - void delinearize(const SCEV *Expr, SmallVectorImpl<const SCEV *> &Subscripts,
>>>> - SmallVectorImpl<const SCEV *> &Sizes,
>>>> - const SCEV *ElementSize);
>>>> -
>>>> /// Return the DataLayout associated with the module this SCEV instance is
>>>> /// operating on.
>>>> const DataLayout &getDataLayout() const {
>>>>
>>>> diff --git a/llvm/lib/Analysis/Delinearization.cpp b/llvm/lib/Analysis/Delinearization.cpp
>>>> index 448e970e9bcc9..8219d59b8f40e 100644
>>>> --- a/llvm/lib/Analysis/Delinearization.cpp
>>>> +++ b/llvm/lib/Analysis/Delinearization.cpp
>>>> @@ -17,6 +17,7 @@
>>>> #include "llvm/Analysis/LoopInfo.h"
>>>> #include "llvm/Analysis/Passes.h"
>>>> #include "llvm/Analysis/ScalarEvolution.h"
>>>> +#include "llvm/Analysis/ScalarEvolutionDivision.h"
>>>> #include "llvm/Analysis/ScalarEvolutionExpressions.h"
>>>> #include "llvm/IR/Constants.h"
>>>> #include "llvm/IR/DerivedTypes.h"
>>>> @@ -36,6 +37,454 @@ using namespace llvm;
>>>> #define DL_NAME "delinearize"
>>>> #define DEBUG_TYPE DL_NAME
>>>>
>>>> +// Return true when S contains at least an undef value.
>>>> +static inline bool containsUndefs(const SCEV *S) {
>>>> + return SCEVExprContains(S, [](const SCEV *S) {
>>>> + if (const auto *SU = dyn_cast<SCEVUnknown>(S))
>>>> + return isa<UndefValue>(SU->getValue());
>>>> + return false;
>>>> + });
>>>> +}
>>>> +
>>>> +namespace {
>>>> +
>>>> +// Collect all steps of SCEV expressions.
>>>> +struct SCEVCollectStrides {
>>>> + ScalarEvolution &SE;
>>>> + SmallVectorImpl<const SCEV *> &Strides;
>>>> +
>>>> + SCEVCollectStrides(ScalarEvolution &SE, SmallVectorImpl<const SCEV *> &S)
>>>> + : SE(SE), Strides(S) {}
>>>> +
>>>> + bool follow(const SCEV *S) {
>>>> + if (const SCEVAddRecExpr *AR = dyn_cast<SCEVAddRecExpr>(S))
>>>> + Strides.push_back(AR->getStepRecurrence(SE));
>>>> + return true;
>>>> + }
>>>> +
>>>> + bool isDone() const { return false; }
>>>> +};
>>>> +
>>>> +// Collect all SCEVUnknown and SCEVMulExpr expressions.
>>>> +struct SCEVCollectTerms {
>>>> + SmallVectorImpl<const SCEV *> &Terms;
>>>> +
>>>> + SCEVCollectTerms(SmallVectorImpl<const SCEV *> &T) : Terms(T) {}
>>>> +
>>>> + bool follow(const SCEV *S) {
>>>> + if (isa<SCEVUnknown>(S) || isa<SCEVMulExpr>(S) ||
>>>> + isa<SCEVSignExtendExpr>(S)) {
>>>> + if (!containsUndefs(S))
>>>> + Terms.push_back(S);
>>>> +
>>>> + // Stop recursion: once we collected a term, do not walk its operands.
>>>> + return false;
>>>> + }
>>>> +
>>>> + // Keep looking.
>>>> + return true;
>>>> + }
>>>> +
>>>> + bool isDone() const { return false; }
>>>> +};
>>>> +
>>>> +// Check if a SCEV contains an AddRecExpr.
>>>> +struct SCEVHasAddRec {
>>>> + bool &ContainsAddRec;
>>>> +
>>>> + SCEVHasAddRec(bool &ContainsAddRec) : ContainsAddRec(ContainsAddRec) {
>>>> + ContainsAddRec = false;
>>>> + }
>>>> +
>>>> + bool follow(const SCEV *S) {
>>>> + if (isa<SCEVAddRecExpr>(S)) {
>>>> + ContainsAddRec = true;
>>>> +
>>>> + // Stop recursion: once we collected a term, do not walk its operands.
>>>> + return false;
>>>> + }
>>>> +
>>>> + // Keep looking.
>>>> + return true;
>>>> + }
>>>> +
>>>> + bool isDone() const { return false; }
>>>> +};
>>>> +
>>>> +// Find factors that are multiplied with an expression that (possibly as a
>>>> +// subexpression) contains an AddRecExpr. In the expression:
>>>> +//
>>>> +// 8 * (100 + %p * %q * (%a + {0, +, 1}_loop))
>>>> +//
>>>> +// "%p * %q" are factors multiplied by the expression "(%a + {0, +, 1}_loop)"
>>>> +// that contains the AddRec {0, +, 1}_loop. %p * %q are likely to be array size
>>>> +// parameters as they form a product with an induction variable.
>>>> +//
>>>> +// This collector expects all array size parameters to be in the same MulExpr.
>>>> +// It might be necessary to later add support for collecting parameters that are
>>>> +// spread over
>>>> diff erent nested MulExpr.
>>>> +struct SCEVCollectAddRecMultiplies {
>>>> + SmallVectorImpl<const SCEV *> &Terms;
>>>> + ScalarEvolution &SE;
>>>> +
>>>> + SCEVCollectAddRecMultiplies(SmallVectorImpl<const SCEV *> &T,
>>>> + ScalarEvolution &SE)
>>>> + : Terms(T), SE(SE) {}
>>>> +
>>>> + bool follow(const SCEV *S) {
>>>> + if (auto *Mul = dyn_cast<SCEVMulExpr>(S)) {
>>>> + bool HasAddRec = false;
>>>> + SmallVector<const SCEV *, 0> Operands;
>>>> + for (auto Op : Mul->operands()) {
>>>> + const SCEVUnknown *Unknown = dyn_cast<SCEVUnknown>(Op);
>>>> + if (Unknown && !isa<CallInst>(Unknown->getValue())) {
>>>> + Operands.push_back(Op);
>>>> + } else if (Unknown) {
>>>> + HasAddRec = true;
>>>> + } else {
>>>> + bool ContainsAddRec = false;
>>>> + SCEVHasAddRec ContiansAddRec(ContainsAddRec);
>>>> + visitAll(Op, ContiansAddRec);
>>>> + HasAddRec |= ContainsAddRec;
>>>> + }
>>>> + }
>>>> + if (Operands.size() == 0)
>>>> + return true;
>>>> +
>>>> + if (!HasAddRec)
>>>> + return false;
>>>> +
>>>> + Terms.push_back(SE.getMulExpr(Operands));
>>>> + // Stop recursion: once we collected a term, do not walk its operands.
>>>> + return false;
>>>> + }
>>>> +
>>>> + // Keep looking.
>>>> + return true;
>>>> + }
>>>> +
>>>> + bool isDone() const { return false; }
>>>> +};
>>>> +
>>>> +} // end anonymous namespace
>>>> +
>>>> +/// Find parametric terms in this SCEVAddRecExpr. We first for parameters in
>>>> +/// two places:
>>>> +/// 1) The strides of AddRec expressions.
>>>> +/// 2) Unknowns that are multiplied with AddRec expressions.
>>>> +void llvm::collectParametricTerms(ScalarEvolution &SE, const SCEV *Expr,
>>>> + SmallVectorImpl<const SCEV *> &Terms) {
>>>> + SmallVector<const SCEV *, 4> Strides;
>>>> + SCEVCollectStrides StrideCollector(SE, Strides);
>>>> + visitAll(Expr, StrideCollector);
>>>> +
>>>> + LLVM_DEBUG({
>>>> + dbgs() << "Strides:\n";
>>>> + for (const SCEV *S : Strides)
>>>> + dbgs() << *S << "\n";
>>>> + });
>>>> +
>>>> + for (const SCEV *S : Strides) {
>>>> + SCEVCollectTerms TermCollector(Terms);
>>>> + visitAll(S, TermCollector);
>>>> + }
>>>> +
>>>> + LLVM_DEBUG({
>>>> + dbgs() << "Terms:\n";
>>>> + for (const SCEV *T : Terms)
>>>> + dbgs() << *T << "\n";
>>>> + });
>>>> +
>>>> + SCEVCollectAddRecMultiplies MulCollector(Terms, SE);
>>>> + visitAll(Expr, MulCollector);
>>>> +}
>>>> +
>>>> +static bool findArrayDimensionsRec(ScalarEvolution &SE,
>>>> + SmallVectorImpl<const SCEV *> &Terms,
>>>> + SmallVectorImpl<const SCEV *> &Sizes) {
>>>> + int Last = Terms.size() - 1;
>>>> + const SCEV *Step = Terms[Last];
>>>> +
>>>> + // End of recursion.
>>>> + if (Last == 0) {
>>>> + if (const SCEVMulExpr *M = dyn_cast<SCEVMulExpr>(Step)) {
>>>> + SmallVector<const SCEV *, 2> Qs;
>>>> + for (const SCEV *Op : M->operands())
>>>> + if (!isa<SCEVConstant>(Op))
>>>> + Qs.push_back(Op);
>>>> +
>>>> + Step = SE.getMulExpr(Qs);
>>>> + }
>>>> +
>>>> + Sizes.push_back(Step);
>>>> + return true;
>>>> + }
>>>> +
>>>> + for (const SCEV *&Term : Terms) {
>>>> + // Normalize the terms before the next call to findArrayDimensionsRec.
>>>> + const SCEV *Q, *R;
>>>> + SCEVDivision::divide(SE, Term, Step, &Q, &R);
>>>> +
>>>> + // Bail out when GCD does not evenly divide one of the terms.
>>>> + if (!R->isZero())
>>>> + return false;
>>>> +
>>>> + Term = Q;
>>>> + }
>>>> +
>>>> + // Remove all SCEVConstants.
>>>> + erase_if(Terms, [](const SCEV *E) { return isa<SCEVConstant>(E); });
>>>> +
>>>> + if (Terms.size() > 0)
>>>> + if (!findArrayDimensionsRec(SE, Terms, Sizes))
>>>> + return false;
>>>> +
>>>> + Sizes.push_back(Step);
>>>> + return true;
>>>> +}
>>>> +
>>>> +// Returns true when one of the SCEVs of Terms contains a SCEVUnknown parameter.
>>>> +static inline bool containsParameters(SmallVectorImpl<const SCEV *> &Terms) {
>>>> + for (const SCEV *T : Terms)
>>>> + if (SCEVExprContains(T, [](const SCEV *S) { return isa<SCEVUnknown>(S); }))
>>>> + return true;
>>>> +
>>>> + return false;
>>>> +}
>>>> +
>>>> +// Return the number of product terms in S.
>>>> +static inline int numberOfTerms(const SCEV *S) {
>>>> + if (const SCEVMulExpr *Expr = dyn_cast<SCEVMulExpr>(S))
>>>> + return Expr->getNumOperands();
>>>> + return 1;
>>>> +}
>>>> +
>>>> +static const SCEV *removeConstantFactors(ScalarEvolution &SE, const SCEV *T) {
>>>> + if (isa<SCEVConstant>(T))
>>>> + return nullptr;
>>>> +
>>>> + if (isa<SCEVUnknown>(T))
>>>> + return T;
>>>> +
>>>> + if (const SCEVMulExpr *M = dyn_cast<SCEVMulExpr>(T)) {
>>>> + SmallVector<const SCEV *, 2> Factors;
>>>> + for (const SCEV *Op : M->operands())
>>>> + if (!isa<SCEVConstant>(Op))
>>>> + Factors.push_back(Op);
>>>> +
>>>> + return SE.getMulExpr(Factors);
>>>> + }
>>>> +
>>>> + return T;
>>>> +}
>>>> +
>>>> +void llvm::findArrayDimensions(ScalarEvolution &SE,
>>>> + SmallVectorImpl<const SCEV *> &Terms,
>>>> + SmallVectorImpl<const SCEV *> &Sizes,
>>>> + const SCEV *ElementSize) {
>>>> + if (Terms.size() < 1 || !ElementSize)
>>>> + return;
>>>> +
>>>> + // Early return when Terms do not contain parameters: we do not delinearize
>>>> + // non parametric SCEVs.
>>>> + if (!containsParameters(Terms))
>>>> + return;
>>>> +
>>>> + LLVM_DEBUG({
>>>> + dbgs() << "Terms:\n";
>>>> + for (const SCEV *T : Terms)
>>>> + dbgs() << *T << "\n";
>>>> + });
>>>> +
>>>> + // Remove duplicates.
>>>> + array_pod_sort(Terms.begin(), Terms.end());
>>>> + Terms.erase(std::unique(Terms.begin(), Terms.end()), Terms.end());
>>>> +
>>>> + // Put larger terms first.
>>>> + llvm::sort(Terms, [](const SCEV *LHS, const SCEV *RHS) {
>>>> + return numberOfTerms(LHS) > numberOfTerms(RHS);
>>>> + });
>>>> +
>>>> + // Try to divide all terms by the element size. If term is not divisible by
>>>> + // element size, proceed with the original term.
>>>> + for (const SCEV *&Term : Terms) {
>>>> + const SCEV *Q, *R;
>>>> + SCEVDivision::divide(SE, Term, ElementSize, &Q, &R);
>>>> + if (!Q->isZero())
>>>> + Term = Q;
>>>> + }
>>>> +
>>>> + SmallVector<const SCEV *, 4> NewTerms;
>>>> +
>>>> + // Remove constant factors.
>>>> + for (const SCEV *T : Terms)
>>>> + if (const SCEV *NewT = removeConstantFactors(SE, T))
>>>> + NewTerms.push_back(NewT);
>>>> +
>>>> + LLVM_DEBUG({
>>>> + dbgs() << "Terms after sorting:\n";
>>>> + for (const SCEV *T : NewTerms)
>>>> + dbgs() << *T << "\n";
>>>> + });
>>>> +
>>>> + if (NewTerms.empty() || !findArrayDimensionsRec(SE, NewTerms, Sizes)) {
>>>> + Sizes.clear();
>>>> + return;
>>>> + }
>>>> +
>>>> + // The last element to be pushed into Sizes is the size of an element.
>>>> + Sizes.push_back(ElementSize);
>>>> +
>>>> + LLVM_DEBUG({
>>>> + dbgs() << "Sizes:\n";
>>>> + for (const SCEV *S : Sizes)
>>>> + dbgs() << *S << "\n";
>>>> + });
>>>> +}
>>>> +
>>>> +void llvm::computeAccessFunctions(ScalarEvolution &SE, const SCEV *Expr,
>>>> + SmallVectorImpl<const SCEV *> &Subscripts,
>>>> + SmallVectorImpl<const SCEV *> &Sizes) {
>>>> + // Early exit in case this SCEV is not an affine multivariate function.
>>>> + if (Sizes.empty())
>>>> + return;
>>>> +
>>>> + if (auto *AR = dyn_cast<SCEVAddRecExpr>(Expr))
>>>> + if (!AR->isAffine())
>>>> + return;
>>>> +
>>>> + const SCEV *Res = Expr;
>>>> + int Last = Sizes.size() - 1;
>>>> + for (int i = Last; i >= 0; i--) {
>>>> + const SCEV *Q, *R;
>>>> + SCEVDivision::divide(SE, Res, Sizes[i], &Q, &R);
>>>> +
>>>> + LLVM_DEBUG({
>>>> + dbgs() << "Res: " << *Res << "\n";
>>>> + dbgs() << "Sizes[i]: " << *Sizes[i] << "\n";
>>>> + dbgs() << "Res divided by Sizes[i]:\n";
>>>> + dbgs() << "Quotient: " << *Q << "\n";
>>>> + dbgs() << "Remainder: " << *R << "\n";
>>>> + });
>>>> +
>>>> + Res = Q;
>>>> +
>>>> + // Do not record the last subscript corresponding to the size of elements in
>>>> + // the array.
>>>> + if (i == Last) {
>>>> +
>>>> + // Bail out if the remainder is too complex.
>>>> + if (isa<SCEVAddRecExpr>(R)) {
>>>> + Subscripts.clear();
>>>> + Sizes.clear();
>>>> + return;
>>>> + }
>>>> +
>>>> + continue;
>>>> + }
>>>> +
>>>> + // Record the access function for the current subscript.
>>>> + Subscripts.push_back(R);
>>>> + }
>>>> +
>>>> + // Also push in last position the remainder of the last division: it will be
>>>> + // the access function of the innermost dimension.
>>>> + Subscripts.push_back(Res);
>>>> +
>>>> + std::reverse(Subscripts.begin(), Subscripts.end());
>>>> +
>>>> + LLVM_DEBUG({
>>>> + dbgs() << "Subscripts:\n";
>>>> + for (const SCEV *S : Subscripts)
>>>> + dbgs() << *S << "\n";
>>>> + });
>>>> +}
>>>> +
>>>> +/// Splits the SCEV into two vectors of SCEVs representing the subscripts and
>>>> +/// sizes of an array access. Returns the remainder of the delinearization that
>>>> +/// is the offset start of the array. The SCEV->delinearize algorithm computes
>>>> +/// the multiples of SCEV coefficients: that is a pattern matching of sub
>>>> +/// expressions in the stride and base of a SCEV corresponding to the
>>>> +/// computation of a GCD (greatest common divisor) of base and stride. When
>>>> +/// SCEV->delinearize fails, it returns the SCEV unchanged.
>>>> +///
>>>> +/// For example: when analyzing the memory access A[i][j][k] in this loop nest
>>>> +///
>>>> +/// void foo(long n, long m, long o, double A[n][m][o]) {
>>>> +///
>>>> +/// for (long i = 0; i < n; i++)
>>>> +/// for (long j = 0; j < m; j++)
>>>> +/// for (long k = 0; k < o; k++)
>>>> +/// A[i][j][k] = 1.0;
>>>> +/// }
>>>> +///
>>>> +/// the delinearization input is the following AddRec SCEV:
>>>> +///
>>>> +/// AddRec: {{{%A,+,(8 * %m * %o)}<%for.i>,+,(8 * %o)}<%for.j>,+,8}<%for.k>
>>>> +///
>>>> +/// From this SCEV, we are able to say that the base offset of the access is %A
>>>> +/// because it appears as an offset that does not divide any of the strides in
>>>> +/// the loops:
>>>> +///
>>>> +/// CHECK: Base offset: %A
>>>> +///
>>>> +/// and then SCEV->delinearize determines the size of some of the dimensions of
>>>> +/// the array as these are the multiples by which the strides are happening:
>>>> +///
>>>> +/// CHECK: ArrayDecl[UnknownSize][%m][%o] with elements of sizeof(double)
>>>> +/// bytes.
>>>> +///
>>>> +/// Note that the outermost dimension remains of UnknownSize because there are
>>>> +/// no strides that would help identifying the size of the last dimension: when
>>>> +/// the array has been statically allocated, one could compute the size of that
>>>> +/// dimension by dividing the overall size of the array by the size of the known
>>>> +/// dimensions: %m * %o * 8.
>>>> +///
>>>> +/// Finally delinearize provides the access functions for the array reference
>>>> +/// that does correspond to A[i][j][k] of the above C testcase:
>>>> +///
>>>> +/// CHECK: ArrayRef[{0,+,1}<%for.i>][{0,+,1}<%for.j>][{0,+,1}<%for.k>]
>>>> +///
>>>> +/// The testcases are checking the output of a function pass:
>>>> +/// DelinearizationPass that walks through all loads and stores of a function
>>>> +/// asking for the SCEV of the memory access with respect to all enclosing
>>>> +/// loops, calling SCEV->delinearize on that and printing the results.
>>>> +void llvm::delinearize(ScalarEvolution &SE, const SCEV *Expr,
>>>> + SmallVectorImpl<const SCEV *> &Subscripts,
>>>> + SmallVectorImpl<const SCEV *> &Sizes,
>>>> + const SCEV *ElementSize) {
>>>> + // First step: collect parametric terms.
>>>> + SmallVector<const SCEV *, 4> Terms;
>>>> + collectParametricTerms(SE, Expr, Terms);
>>>> +
>>>> + if (Terms.empty())
>>>> + return;
>>>> +
>>>> + // Second step: find subscript sizes.
>>>> + findArrayDimensions(SE, Terms, Sizes, ElementSize);
>>>> +
>>>> + if (Sizes.empty())
>>>> + return;
>>>> +
>>>> + // Third step: compute the access functions for each subscript.
>>>> + computeAccessFunctions(SE, Expr, Subscripts, Sizes);
>>>> +
>>>> + if (Subscripts.empty())
>>>> + return;
>>>> +
>>>> + LLVM_DEBUG({
>>>> + dbgs() << "succeeded to delinearize " << *Expr << "\n";
>>>> + dbgs() << "ArrayDecl[UnknownSize]";
>>>> + for (const SCEV *S : Sizes)
>>>> + dbgs() << "[" << *S << "]";
>>>> +
>>>> + dbgs() << "\nArrayRef";
>>>> + for (const SCEV *S : Subscripts)
>>>> + dbgs() << "[" << *S << "]";
>>>> + dbgs() << "\n";
>>>> + });
>>>> +}
>>>> +
>>>> namespace {
>>>>
>>>> class Delinearization : public FunctionPass {
>>>> @@ -84,7 +533,7 @@ void printDelinearization(raw_ostream &O, Function *F, LoopInfo *LI,
>>>> O << "AccessFunction: " << *AccessFn << "\n";
>>>>
>>>> SmallVector<const SCEV *, 3> Subscripts, Sizes;
>>>> - SE->delinearize(AccessFn, Subscripts, Sizes, SE->getElementSize(&Inst));
>>>> + delinearize(*SE, AccessFn, Subscripts, Sizes, SE->getElementSize(&Inst));
>>>> if (Subscripts.size() == 0 || Sizes.size() == 0 ||
>>>> Subscripts.size() != Sizes.size()) {
>>>> O << "failed to delinearize\n";
>>>>
>>>> diff --git a/llvm/lib/Analysis/DependenceAnalysis.cpp b/llvm/lib/Analysis/DependenceAnalysis.cpp
>>>> index 9c09894f3a451..fa9c672119455 100644
>>>> --- a/llvm/lib/Analysis/DependenceAnalysis.cpp
>>>> +++ b/llvm/lib/Analysis/DependenceAnalysis.cpp
>>>> @@ -53,6 +53,7 @@
>>>> #include "llvm/ADT/STLExtras.h"
>>>> #include "llvm/ADT/Statistic.h"
>>>> #include "llvm/Analysis/AliasAnalysis.h"
>>>> +#include "llvm/Analysis/Delinearization.h"
>>>> #include "llvm/Analysis/LoopInfo.h"
>>>> #include "llvm/Analysis/ScalarEvolution.h"
>>>> #include "llvm/Analysis/ScalarEvolutionExpressions.h"
>>>> @@ -3439,16 +3440,16 @@ bool DependenceInfo::tryDelinearizeParametricSize(
>>>>
>>>> // First step: collect parametric terms in both array references.
>>>> SmallVector<const SCEV *, 4> Terms;
>>>> - SE->collectParametricTerms(SrcAR, Terms);
>>>> - SE->collectParametricTerms(DstAR, Terms);
>>>> + collectParametricTerms(*SE, SrcAR, Terms);
>>>> + collectParametricTerms(*SE, DstAR, Terms);
>>>>
>>>> // Second step: find subscript sizes.
>>>> SmallVector<const SCEV *, 4> Sizes;
>>>> - SE->findArrayDimensions(Terms, Sizes, ElementSize);
>>>> + findArrayDimensions(*SE, Terms, Sizes, ElementSize);
>>>>
>>>> // Third step: compute the access functions for each subscript.
>>>> - SE->computeAccessFunctions(SrcAR, SrcSubscripts, Sizes);
>>>> - SE->computeAccessFunctions(DstAR, DstSubscripts, Sizes);
>>>> + computeAccessFunctions(*SE, SrcAR, SrcSubscripts, Sizes);
>>>> + computeAccessFunctions(*SE, DstAR, DstSubscripts, Sizes);
>>>>
>>>> // Fail when there is only a subscript: that's a linearized access function.
>>>> if (SrcSubscripts.size() < 2 || DstSubscripts.size() < 2 ||
>>>>
>>>> diff --git a/llvm/lib/Analysis/LoopCacheAnalysis.cpp b/llvm/lib/Analysis/LoopCacheAnalysis.cpp
>>>> index 772e4eb239a9c..75bb8b02286c1 100644
>>>> --- a/llvm/lib/Analysis/LoopCacheAnalysis.cpp
>>>> +++ b/llvm/lib/Analysis/LoopCacheAnalysis.cpp
>>>> @@ -30,6 +30,7 @@
>>>> #include "llvm/ADT/Sequence.h"
>>>> #include "llvm/ADT/SmallVector.h"
>>>> #include "llvm/Analysis/AliasAnalysis.h"
>>>> +#include "llvm/Analysis/Delinearization.h"
>>>> #include "llvm/Analysis/DependenceAnalysis.h"
>>>> #include "llvm/Analysis/LoopInfo.h"
>>>> #include "llvm/Analysis/ScalarEvolutionExpressions.h"
>>>> @@ -344,8 +345,8 @@ bool IndexedReference::delinearize(const LoopInfo &LI) {
>>>> LLVM_DEBUG(dbgs().indent(2) << "In Loop '" << L->getName()
>>>> << "', AccessFn: " << *AccessFn << "\n");
>>>>
>>>> - SE.delinearize(AccessFn, Subscripts, Sizes,
>>>> - SE.getElementSize(&StoreOrLoadInst));
>>>> + llvm::delinearize(SE, AccessFn, Subscripts, Sizes,
>>>> + SE.getElementSize(&StoreOrLoadInst));
>>>>
>>>> if (Subscripts.empty() || Sizes.empty() ||
>>>> Subscripts.size() != Sizes.size()) {
>>>>
>>>> diff --git a/llvm/lib/Analysis/ScalarEvolution.cpp b/llvm/lib/Analysis/ScalarEvolution.cpp
>>>> index 50901f3524d1b..151883ab67367 100644
>>>> --- a/llvm/lib/Analysis/ScalarEvolution.cpp
>>>> +++ b/llvm/lib/Analysis/ScalarEvolution.cpp
>>>> @@ -12187,237 +12187,6 @@ static inline bool containsUndefs(const SCEV *S) {
>>>> });
>>>> }
>>>>
>>>> -namespace {
>>>> -
>>>> -// Collect all steps of SCEV expressions.
>>>> -struct SCEVCollectStrides {
>>>> - ScalarEvolution &SE;
>>>> - SmallVectorImpl<const SCEV *> &Strides;
>>>> -
>>>> - SCEVCollectStrides(ScalarEvolution &SE, SmallVectorImpl<const SCEV *> &S)
>>>> - : SE(SE), Strides(S) {}
>>>> -
>>>> - bool follow(const SCEV *S) {
>>>> - if (const SCEVAddRecExpr *AR = dyn_cast<SCEVAddRecExpr>(S))
>>>> - Strides.push_back(AR->getStepRecurrence(SE));
>>>> - return true;
>>>> - }
>>>> -
>>>> - bool isDone() const { return false; }
>>>> -};
>>>> -
>>>> -// Collect all SCEVUnknown and SCEVMulExpr expressions.
>>>> -struct SCEVCollectTerms {
>>>> - SmallVectorImpl<const SCEV *> &Terms;
>>>> -
>>>> - SCEVCollectTerms(SmallVectorImpl<const SCEV *> &T) : Terms(T) {}
>>>> -
>>>> - bool follow(const SCEV *S) {
>>>> - if (isa<SCEVUnknown>(S) || isa<SCEVMulExpr>(S) ||
>>>> - isa<SCEVSignExtendExpr>(S)) {
>>>> - if (!containsUndefs(S))
>>>> - Terms.push_back(S);
>>>> -
>>>> - // Stop recursion: once we collected a term, do not walk its operands.
>>>> - return false;
>>>> - }
>>>> -
>>>> - // Keep looking.
>>>> - return true;
>>>> - }
>>>> -
>>>> - bool isDone() const { return false; }
>>>> -};
>>>> -
>>>> -// Check if a SCEV contains an AddRecExpr.
>>>> -struct SCEVHasAddRec {
>>>> - bool &ContainsAddRec;
>>>> -
>>>> - SCEVHasAddRec(bool &ContainsAddRec) : ContainsAddRec(ContainsAddRec) {
>>>> - ContainsAddRec = false;
>>>> - }
>>>> -
>>>> - bool follow(const SCEV *S) {
>>>> - if (isa<SCEVAddRecExpr>(S)) {
>>>> - ContainsAddRec = true;
>>>> -
>>>> - // Stop recursion: once we collected a term, do not walk its operands.
>>>> - return false;
>>>> - }
>>>> -
>>>> - // Keep looking.
>>>> - return true;
>>>> - }
>>>> -
>>>> - bool isDone() const { return false; }
>>>> -};
>>>> -
>>>> -// Find factors that are multiplied with an expression that (possibly as a
>>>> -// subexpression) contains an AddRecExpr. In the expression:
>>>> -//
>>>> -// 8 * (100 + %p * %q * (%a + {0, +, 1}_loop))
>>>> -//
>>>> -// "%p * %q" are factors multiplied by the expression "(%a + {0, +, 1}_loop)"
>>>> -// that contains the AddRec {0, +, 1}_loop. %p * %q are likely to be array size
>>>> -// parameters as they form a product with an induction variable.
>>>> -//
>>>> -// This collector expects all array size parameters to be in the same MulExpr.
>>>> -// It might be necessary to later add support for collecting parameters that are
>>>> -// spread over
>>>> diff erent nested MulExpr.
>>>> -struct SCEVCollectAddRecMultiplies {
>>>> - SmallVectorImpl<const SCEV *> &Terms;
>>>> - ScalarEvolution &SE;
>>>> -
>>>> - SCEVCollectAddRecMultiplies(SmallVectorImpl<const SCEV *> &T, ScalarEvolution &SE)
>>>> - : Terms(T), SE(SE) {}
>>>> -
>>>> - bool follow(const SCEV *S) {
>>>> - if (auto *Mul = dyn_cast<SCEVMulExpr>(S)) {
>>>> - bool HasAddRec = false;
>>>> - SmallVector<const SCEV *, 0> Operands;
>>>> - for (auto Op : Mul->operands()) {
>>>> - const SCEVUnknown *Unknown = dyn_cast<SCEVUnknown>(Op);
>>>> - if (Unknown && !isa<CallInst>(Unknown->getValue())) {
>>>> - Operands.push_back(Op);
>>>> - } else if (Unknown) {
>>>> - HasAddRec = true;
>>>> - } else {
>>>> - bool ContainsAddRec = false;
>>>> - SCEVHasAddRec ContiansAddRec(ContainsAddRec);
>>>> - visitAll(Op, ContiansAddRec);
>>>> - HasAddRec |= ContainsAddRec;
>>>> - }
>>>> - }
>>>> - if (Operands.size() == 0)
>>>> - return true;
>>>> -
>>>> - if (!HasAddRec)
>>>> - return false;
>>>> -
>>>> - Terms.push_back(SE.getMulExpr(Operands));
>>>> - // Stop recursion: once we collected a term, do not walk its operands.
>>>> - return false;
>>>> - }
>>>> -
>>>> - // Keep looking.
>>>> - return true;
>>>> - }
>>>> -
>>>> - bool isDone() const { return false; }
>>>> -};
>>>> -
>>>> -} // end anonymous namespace
>>>> -
>>>> -/// Find parametric terms in this SCEVAddRecExpr. We first for parameters in
>>>> -/// two places:
>>>> -/// 1) The strides of AddRec expressions.
>>>> -/// 2) Unknowns that are multiplied with AddRec expressions.
>>>> -void ScalarEvolution::collectParametricTerms(const SCEV *Expr,
>>>> - SmallVectorImpl<const SCEV *> &Terms) {
>>>> - SmallVector<const SCEV *, 4> Strides;
>>>> - SCEVCollectStrides StrideCollector(*this, Strides);
>>>> - visitAll(Expr, StrideCollector);
>>>> -
>>>> - LLVM_DEBUG({
>>>> - dbgs() << "Strides:\n";
>>>> - for (const SCEV *S : Strides)
>>>> - dbgs() << *S << "\n";
>>>> - });
>>>> -
>>>> - for (const SCEV *S : Strides) {
>>>> - SCEVCollectTerms TermCollector(Terms);
>>>> - visitAll(S, TermCollector);
>>>> - }
>>>> -
>>>> - LLVM_DEBUG({
>>>> - dbgs() << "Terms:\n";
>>>> - for (const SCEV *T : Terms)
>>>> - dbgs() << *T << "\n";
>>>> - });
>>>> -
>>>> - SCEVCollectAddRecMultiplies MulCollector(Terms, *this);
>>>> - visitAll(Expr, MulCollector);
>>>> -}
>>>> -
>>>> -static bool findArrayDimensionsRec(ScalarEvolution &SE,
>>>> - SmallVectorImpl<const SCEV *> &Terms,
>>>> - SmallVectorImpl<const SCEV *> &Sizes) {
>>>> - int Last = Terms.size() - 1;
>>>> - const SCEV *Step = Terms[Last];
>>>> -
>>>> - // End of recursion.
>>>> - if (Last == 0) {
>>>> - if (const SCEVMulExpr *M = dyn_cast<SCEVMulExpr>(Step)) {
>>>> - SmallVector<const SCEV *, 2> Qs;
>>>> - for (const SCEV *Op : M->operands())
>>>> - if (!isa<SCEVConstant>(Op))
>>>> - Qs.push_back(Op);
>>>> -
>>>> - Step = SE.getMulExpr(Qs);
>>>> - }
>>>> -
>>>> - Sizes.push_back(Step);
>>>> - return true;
>>>> - }
>>>> -
>>>> - for (const SCEV *&Term : Terms) {
>>>> - // Normalize the terms before the next call to findArrayDimensionsRec.
>>>> - const SCEV *Q, *R;
>>>> - SCEVDivision::divide(SE, Term, Step, &Q, &R);
>>>> -
>>>> - // Bail out when GCD does not evenly divide one of the terms.
>>>> - if (!R->isZero())
>>>> - return false;
>>>> -
>>>> - Term = Q;
>>>> - }
>>>> -
>>>> - // Remove all SCEVConstants.
>>>> - erase_if(Terms, [](const SCEV *E) { return isa<SCEVConstant>(E); });
>>>> -
>>>> - if (Terms.size() > 0)
>>>> - if (!findArrayDimensionsRec(SE, Terms, Sizes))
>>>> - return false;
>>>> -
>>>> - Sizes.push_back(Step);
>>>> - return true;
>>>> -}
>>>> -
>>>> -// Returns true when one of the SCEVs of Terms contains a SCEVUnknown parameter.
>>>> -static inline bool containsParameters(SmallVectorImpl<const SCEV *> &Terms) {
>>>> - for (const SCEV *T : Terms)
>>>> - if (SCEVExprContains(T, [](const SCEV *S) { return isa<SCEVUnknown>(S); }))
>>>> - return true;
>>>> -
>>>> - return false;
>>>> -}
>>>> -
>>>> -// Return the number of product terms in S.
>>>> -static inline int numberOfTerms(const SCEV *S) {
>>>> - if (const SCEVMulExpr *Expr = dyn_cast<SCEVMulExpr>(S))
>>>> - return Expr->getNumOperands();
>>>> - return 1;
>>>> -}
>>>> -
>>>> -static const SCEV *removeConstantFactors(ScalarEvolution &SE, const SCEV *T) {
>>>> - if (isa<SCEVConstant>(T))
>>>> - return nullptr;
>>>> -
>>>> - if (isa<SCEVUnknown>(T))
>>>> - return T;
>>>> -
>>>> - if (const SCEVMulExpr *M = dyn_cast<SCEVMulExpr>(T)) {
>>>> - SmallVector<const SCEV *, 2> Factors;
>>>> - for (const SCEV *Op : M->operands())
>>>> - if (!isa<SCEVConstant>(Op))
>>>> - Factors.push_back(Op);
>>>> -
>>>> - return SE.getMulExpr(Factors);
>>>> - }
>>>> -
>>>> - return T;
>>>> -}
>>>> -
>>>> /// Return the size of an element read or written by Inst.
>>>> const SCEV *ScalarEvolution::getElementSize(Instruction *Inst) {
>>>> Type *Ty;
>>>> @@ -12432,211 +12201,6 @@ const SCEV *ScalarEvolution::getElementSize(Instruction *Inst) {
>>>> return getSizeOfExpr(ETy, Ty);
>>>> }
>>>>
>>>> -void ScalarEvolution::findArrayDimensions(SmallVectorImpl<const SCEV *> &Terms,
>>>> - SmallVectorImpl<const SCEV *> &Sizes,
>>>> - const SCEV *ElementSize) {
>>>> - if (Terms.size() < 1 || !ElementSize)
>>>> - return;
>>>> -
>>>> - // Early return when Terms do not contain parameters: we do not delinearize
>>>> - // non parametric SCEVs.
>>>> - if (!containsParameters(Terms))
>>>> - return;
>>>> -
>>>> - LLVM_DEBUG({
>>>> - dbgs() << "Terms:\n";
>>>> - for (const SCEV *T : Terms)
>>>> - dbgs() << *T << "\n";
>>>> - });
>>>> -
>>>> - // Remove duplicates.
>>>> - array_pod_sort(Terms.begin(), Terms.end());
>>>> - Terms.erase(std::unique(Terms.begin(), Terms.end()), Terms.end());
>>>> -
>>>> - // Put larger terms first.
>>>> - llvm::sort(Terms, [](const SCEV *LHS, const SCEV *RHS) {
>>>> - return numberOfTerms(LHS) > numberOfTerms(RHS);
>>>> - });
>>>> -
>>>> - // Try to divide all terms by the element size. If term is not divisible by
>>>> - // element size, proceed with the original term.
>>>> - for (const SCEV *&Term : Terms) {
>>>> - const SCEV *Q, *R;
>>>> - SCEVDivision::divide(*this, Term, ElementSize, &Q, &R);
>>>> - if (!Q->isZero())
>>>> - Term = Q;
>>>> - }
>>>> -
>>>> - SmallVector<const SCEV *, 4> NewTerms;
>>>> -
>>>> - // Remove constant factors.
>>>> - for (const SCEV *T : Terms)
>>>> - if (const SCEV *NewT = removeConstantFactors(*this, T))
>>>> - NewTerms.push_back(NewT);
>>>> -
>>>> - LLVM_DEBUG({
>>>> - dbgs() << "Terms after sorting:\n";
>>>> - for (const SCEV *T : NewTerms)
>>>> - dbgs() << *T << "\n";
>>>> - });
>>>> -
>>>> - if (NewTerms.empty() || !findArrayDimensionsRec(*this, NewTerms, Sizes)) {
>>>> - Sizes.clear();
>>>> - return;
>>>> - }
>>>> -
>>>> - // The last element to be pushed into Sizes is the size of an element.
>>>> - Sizes.push_back(ElementSize);
>>>> -
>>>> - LLVM_DEBUG({
>>>> - dbgs() << "Sizes:\n";
>>>> - for (const SCEV *S : Sizes)
>>>> - dbgs() << *S << "\n";
>>>> - });
>>>> -}
>>>> -
>>>> -void ScalarEvolution::computeAccessFunctions(
>>>> - const SCEV *Expr, SmallVectorImpl<const SCEV *> &Subscripts,
>>>> - SmallVectorImpl<const SCEV *> &Sizes) {
>>>> - // Early exit in case this SCEV is not an affine multivariate function.
>>>> - if (Sizes.empty())
>>>> - return;
>>>> -
>>>> - if (auto *AR = dyn_cast<SCEVAddRecExpr>(Expr))
>>>> - if (!AR->isAffine())
>>>> - return;
>>>> -
>>>> - const SCEV *Res = Expr;
>>>> - int Last = Sizes.size() - 1;
>>>> - for (int i = Last; i >= 0; i--) {
>>>> - const SCEV *Q, *R;
>>>> - SCEVDivision::divide(*this, Res, Sizes[i], &Q, &R);
>>>> -
>>>> - LLVM_DEBUG({
>>>> - dbgs() << "Res: " << *Res << "\n";
>>>> - dbgs() << "Sizes[i]: " << *Sizes[i] << "\n";
>>>> - dbgs() << "Res divided by Sizes[i]:\n";
>>>> - dbgs() << "Quotient: " << *Q << "\n";
>>>> - dbgs() << "Remainder: " << *R << "\n";
>>>> - });
>>>> -
>>>> - Res = Q;
>>>> -
>>>> - // Do not record the last subscript corresponding to the size of elements in
>>>> - // the array.
>>>> - if (i == Last) {
>>>> -
>>>> - // Bail out if the remainder is too complex.
>>>> - if (isa<SCEVAddRecExpr>(R)) {
>>>> - Subscripts.clear();
>>>> - Sizes.clear();
>>>> - return;
>>>> - }
>>>> -
>>>> - continue;
>>>> - }
>>>> -
>>>> - // Record the access function for the current subscript.
>>>> - Subscripts.push_back(R);
>>>> - }
>>>> -
>>>> - // Also push in last position the remainder of the last division: it will be
>>>> - // the access function of the innermost dimension.
>>>> - Subscripts.push_back(Res);
>>>> -
>>>> - std::reverse(Subscripts.begin(), Subscripts.end());
>>>> -
>>>> - LLVM_DEBUG({
>>>> - dbgs() << "Subscripts:\n";
>>>> - for (const SCEV *S : Subscripts)
>>>> - dbgs() << *S << "\n";
>>>> - });
>>>> -}
>>>> -
>>>> -/// Splits the SCEV into two vectors of SCEVs representing the subscripts and
>>>> -/// sizes of an array access. Returns the remainder of the delinearization that
>>>> -/// is the offset start of the array. The SCEV->delinearize algorithm computes
>>>> -/// the multiples of SCEV coefficients: that is a pattern matching of sub
>>>> -/// expressions in the stride and base of a SCEV corresponding to the
>>>> -/// computation of a GCD (greatest common divisor) of base and stride. When
>>>> -/// SCEV->delinearize fails, it returns the SCEV unchanged.
>>>> -///
>>>> -/// For example: when analyzing the memory access A[i][j][k] in this loop nest
>>>> -///
>>>> -/// void foo(long n, long m, long o, double A[n][m][o]) {
>>>> -///
>>>> -/// for (long i = 0; i < n; i++)
>>>> -/// for (long j = 0; j < m; j++)
>>>> -/// for (long k = 0; k < o; k++)
>>>> -/// A[i][j][k] = 1.0;
>>>> -/// }
>>>> -///
>>>> -/// the delinearization input is the following AddRec SCEV:
>>>> -///
>>>> -/// AddRec: {{{%A,+,(8 * %m * %o)}<%for.i>,+,(8 * %o)}<%for.j>,+,8}<%for.k>
>>>> -///
>>>> -/// From this SCEV, we are able to say that the base offset of the access is %A
>>>> -/// because it appears as an offset that does not divide any of the strides in
>>>> -/// the loops:
>>>> -///
>>>> -/// CHECK: Base offset: %A
>>>> -///
>>>> -/// and then SCEV->delinearize determines the size of some of the dimensions of
>>>> -/// the array as these are the multiples by which the strides are happening:
>>>> -///
>>>> -/// CHECK: ArrayDecl[UnknownSize][%m][%o] with elements of sizeof(double) bytes.
>>>> -///
>>>> -/// Note that the outermost dimension remains of UnknownSize because there are
>>>> -/// no strides that would help identifying the size of the last dimension: when
>>>> -/// the array has been statically allocated, one could compute the size of that
>>>> -/// dimension by dividing the overall size of the array by the size of the known
>>>> -/// dimensions: %m * %o * 8.
>>>> -///
>>>> -/// Finally delinearize provides the access functions for the array reference
>>>> -/// that does correspond to A[i][j][k] of the above C testcase:
>>>> -///
>>>> -/// CHECK: ArrayRef[{0,+,1}<%for.i>][{0,+,1}<%for.j>][{0,+,1}<%for.k>]
>>>> -///
>>>> -/// The testcases are checking the output of a function pass:
>>>> -/// DelinearizationPass that walks through all loads and stores of a function
>>>> -/// asking for the SCEV of the memory access with respect to all enclosing
>>>> -/// loops, calling SCEV->delinearize on that and printing the results.
>>>> -void ScalarEvolution::delinearize(const SCEV *Expr,
>>>> - SmallVectorImpl<const SCEV *> &Subscripts,
>>>> - SmallVectorImpl<const SCEV *> &Sizes,
>>>> - const SCEV *ElementSize) {
>>>> - // First step: collect parametric terms.
>>>> - SmallVector<const SCEV *, 4> Terms;
>>>> - collectParametricTerms(Expr, Terms);
>>>> -
>>>> - if (Terms.empty())
>>>> - return;
>>>> -
>>>> - // Second step: find subscript sizes.
>>>> - findArrayDimensions(Terms, Sizes, ElementSize);
>>>> -
>>>> - if (Sizes.empty())
>>>> - return;
>>>> -
>>>> - // Third step: compute the access functions for each subscript.
>>>> - computeAccessFunctions(Expr, Subscripts, Sizes);
>>>> -
>>>> - if (Subscripts.empty())
>>>> - return;
>>>> -
>>>> - LLVM_DEBUG({
>>>> - dbgs() << "succeeded to delinearize " << *Expr << "\n";
>>>> - dbgs() << "ArrayDecl[UnknownSize]";
>>>> - for (const SCEV *S : Sizes)
>>>> - dbgs() << "[" << *S << "]";
>>>> -
>>>> - dbgs() << "\nArrayRef";
>>>> - for (const SCEV *S : Subscripts)
>>>> - dbgs() << "[" << *S << "]";
>>>> - dbgs() << "\n";
>>>> - });
>>>> -}
>>>> -
>>>> bool ScalarEvolution::getIndexExpressionsFromGEP(
>>>> const GetElementPtrInst *GEP, SmallVectorImpl<const SCEV *> &Subscripts,
>>>> SmallVectorImpl<int> &Sizes) {
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> llvm-commits mailing list
>>>> llvm-commits at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
More information about the llvm-commits
mailing list