[llvm] 63afb70 - [RFC][GlobalISel] Overhauled MIR Patterns Support for Combiners
via llvm-commits
llvm-commits at lists.llvm.org
Thu Aug 10 23:55:02 PDT 2023
Author: pvanhout
Date: 2023-08-11T08:54:56+02:00
New Revision: 63afb70503bf254bb3cdd8acc6bd1ad25c66e0d8
URL: https://github.com/llvm/llvm-project/commit/63afb70503bf254bb3cdd8acc6bd1ad25c66e0d8
DIFF: https://github.com/llvm/llvm-project/commit/63afb70503bf254bb3cdd8acc6bd1ad25c66e0d8.diff
LOG: [RFC][GlobalISel] Overhauled MIR Patterns Support for Combiners
See https://discourse.llvm.org/t/rfc-overhauled-mir-patterns-for-globalisel-combiners/72264
This is a complete overrhaul of the recently-added GlobalISel Match Table backend which adds
support for MIR patterns for both match and apply patterns.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D156315
Added:
llvm/docs/GlobalISel/MIRPatterns.rst
llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-imms.td
llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-operand-types.td
llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-patfrag-root.td
llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-permutations.td
llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-variadics.td
llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/operand-types.td
llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/patfrag-errors.td
llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/pattern-errors.td
Modified:
llvm/docs/GlobalISel/index.rst
llvm/docs/UserGuides.rst
llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutor.h
llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutorImpl.h
llvm/include/llvm/Target/GlobalISel/Combine.td
llvm/lib/CodeGen/GlobalISel/GIMatchTableExecutor.cpp
llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table.td
llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/pattern-parsing.td
llvm/test/TableGen/GlobalISelEmitter.td
llvm/utils/TableGen/CodeGenTarget.cpp
llvm/utils/TableGen/CodeGenTarget.h
llvm/utils/TableGen/GlobalISel/CodeExpansions.h
llvm/utils/TableGen/GlobalISel/CombinerUtils.h
llvm/utils/TableGen/GlobalISelCombinerMatchTableEmitter.cpp
llvm/utils/TableGen/GlobalISelEmitter.cpp
llvm/utils/TableGen/GlobalISelMatchTable.h
llvm/utils/TableGen/GlobalISelMatchTableExecutorEmitter.cpp
llvm/utils/TableGen/GlobalISelMatchTableExecutorEmitter.h
Removed:
llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/pattern-parsing-errors.td
################################################################################
diff --git a/llvm/docs/GlobalISel/MIRPatterns.rst b/llvm/docs/GlobalISel/MIRPatterns.rst
new file mode 100644
index 00000000000000..715fd60d12eff9
--- /dev/null
+++ b/llvm/docs/GlobalISel/MIRPatterns.rst
@@ -0,0 +1,391 @@
+
+.. _tblgen-mirpats:
+
+========================
+MIR Patterns in TableGen
+========================
+
+.. contents::
+ :local:
+
+
+User's Guide
+============
+
+This section is intended for developers who want to use MIR patterns in their
+TableGen files.
+
+``NOTE``:
+This feature is still in active development. This document may become outdated
+over time. If you see something that's incorrect, please update it.
+
+Use Cases
+---------
+
+MIR patterns are supported in the following places:
+
+* GlobalISel ``GICombineRule``
+* GlobalISel ``GICombinePatFrag``
+
+Syntax
+------
+
+MIR patterns use the DAG datatype in TableGen.
+
+.. code-block:: text
+
+ (inst operand0, operand1, ...)
+
+``inst`` must be a def which inherits from ``Instruction`` (e.g. ``G_FADD``)
+or ``GICombinePatFrag``.
+
+Operands essentially fall into one of two categories:
+
+* immediates
+
+ * untyped, unnamed: ``0``
+ * untyped, named: ``0:$y``
+ * typed, unnamed: ``(i32 0)``
+ * typed, named: ``(i32 0):$y``
+
+* machine operands
+
+ * untyped: ``$x``
+ * typed: ``i32:$x``
+
+Semantics:
+
+* A typed operand always adds an operand type check to the matcher.
+* There is a trivial type inference system to propagate types.
+
+ * e.g. You only need to use ``i32:$x`` once in any pattern of a
+ ``GICombinePatFrag`` alternative or ``GICombineRule``, then all
+ other patterns in that rule/alternative can simply use ``$x``
+ (``i32:$x`` is redundant).
+
+* A nammed operand's behavior depends on whether the name has been seen before.
+
+ * For match patterns, reusing an operand name checks that the operands
+ are identical (see example 2 below).
+ * For apply patterns, reusing an operand name simply copies that operand into
+ the new instruction (see example 2 below).
+
+Operands are ordered just like they would be in a MachineInstr: the defs (outs)
+come first, then the uses (ins).
+
+Patterns are generally grouped into another DAG datatype with a dummy operator
+such as ``match``, ``apply`` or ``pattern``.
+
+Finally, any DAG datatype in TableGen can be named. This also holds for
+patterns. e.g. the following is valid: ``(G_FOO $root, (i32 0):$cst):$mypat``.
+This may also be helpful to debug issues. Patterns are *always* named, and if
+they don't have a name, an "anonymous" one is given to them. If you're trying
+to debug an error related to a MIR pattern, but the error mentions an anonymous
+pattern, you can try naming your patterns to see exactly where the issue is.
+
+.. code-block:: text
+ :caption: Pattern Example 1
+
+ // Match
+ // %imp = G_IMPLICIT_DEF
+ // %root = G_MUL %x, %imp
+ (match (G_IMPLICIT_DEF $imp),
+ (G_MUL $root, $x, $imp))
+
+.. code-block:: text
+ :caption: Pattern Example 2
+
+ // using $x twice here checks that the operand 1 and 2 of the G_AND are
+ // identical.
+ (match (G_AND $root, $x, $x))
+ // using $x again here copies operand 1 from G_AND into the new inst.
+ (apply (COPY $root, $x))
+
+
+Limitations
+-----------
+
+This a non-exhaustive list of known issues with MIR patterns at this time.
+
+* Matching intrinsics is not yet possible.
+* Using ``GICombinePatFrag`` within another ``GICombinePatFrag`` is not
+ supported.
+* Using ``GICombinePatFrag`` in the ``apply`` pattern of a ``GICombineRule``
+ is not supported.
+* Deleting the matched pattern in a ``GICombineRule`` needs to be done using
+ ``G_IMPLICIT_DEF`` or C++.
+* Replacing the root of a pattern with another instruction needs to be done
+ using COPY.
+* We cannot rewrite a matched instruction other than the root.
+* Matching/creating a (CImm) immediate >64 bits is not supported
+ (see comment in ``GIM_CheckConstantInt``)
+* There is currently no way to constrain two register/immediate types to
+ match. e.g. if a pattern needs to work on both i32 and i64, you either
+ need to leave it untyped and check the type in C++, or duplicate the
+ pattern.
+
+GICombineRule
+-------------
+
+MIR patterns can appear in the ``match`` or ``apply`` patterns of a
+``GICombineRule``.
+
+The ``root`` of the rule can either be a def of an instruction, or a
+named pattern. The latter is helpful when the instruction you want
+to match has no defs. The former is generally preferred because
+it's less verbose.
+
+.. code-block:: text
+ :caption: Combine Rule root is a def
+
+ // Fold x op 1 -> x
+ def right_identity_one: GICombineRule<
+ (defs root:$dst),
+ (match (G_MUL $dst, $x, 1)),
+ // Note: Patterns always need to create something, we can't just replace $dst with $x, so we need a COPY.
+ (apply (COPY $dst, $x))
+ >;
+
+.. code-block:: text
+ :caption: Combine Rule root is a named pattern
+
+ def Foo : GICombineRule<
+ (defs root:$root),
+ (match (G_ZEXT $tmp, (i32 0)),
+ (G_STORE $tmp, $ptr):$root),
+ (apply (G_STORE (i32 0), $ptr):$root)>;
+
+
+Combine Rules also allow mixing C++ code with MIR patterns, so that you
+may perform additional checks when matching, or run additional code after
+rewriting a pattern.
+
+The following expansions are available for MIR patterns:
+
+* operand names (``MachineOperand &``)
+* pattern names (``MachineInstr *`` for ``match``,
+ ``MachineInstrBuilder &`` for apply)
+
+.. code-block:: text
+ :caption: Example C++ Expansions
+
+ def Foo : GICombineRule<
+ (defs root:$root),
+ (match (G_ZEXT $root, $src):$mi),
+ (apply "foobar(${root}.getReg(), ${src}.getReg(), ${mi}->hasImplicitDef())")>;
+
+Common Pattern #1: Replace a Register with Another
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The 'apply' pattern must always redefine its root.
+It cannot just replace it with something else directly.
+A simple workaround is to just use a COPY that'll be eliminated later.
+
+.. code-block:: text
+
+ def Foo : GICombineRule<
+ (defs root:$dst),
+ (match (G_FNEG $tmp, $src), (G_FNEG $dst, $tmp)),
+ (apply (COPY $dst, $src))>;
+
+Common Pattern #2: Erasing a Pattern
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As said before, we must always emit something in the 'apply' pattern.
+If we wish to delete the matched instruction, we can simply replace its
+definition with a ``G_IMPLICIT_DEF``.
+
+.. code-block:: text
+
+ def Foo : GICombineRule<
+ (defs root:$dst),
+ (match (G_FOO $tmp, $src), (G_BAR $dst, $tmp)),
+ (apply (G_IMPLICIT_DEF $dst))>;
+
+If the instruction has no definition, like ``G_STORE``, we cannot use
+an instruction pattern in 'apply' - C++ has to be used.
+
+Common Pattern #3: Emitting a Constant Value
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When an immediate operand appears in an 'apply' pattern, the behavior
+depends on whether it's typed or not.
+
+* If the immediate is typed, a ``G_CONSTANT`` is implicitly emitted
+ (= a register operand is added to the instruction).
+* If the immediate is untyped, a simple immediate is added
+ (``MachineInstrBuilder::addImm``).
+
+There is of course a special case for ``G_CONSTANT``. Immediates for
+``G_CONSTANT`` must always be typed, and a CImm is added
+(``MachineInstrBuilder::addCImm``).
+
+.. code-block:: text
+ :caption: Constant Emission Examples:
+
+ // Example output:
+ // %0 = G_CONSTANT i32 0
+ // %dst = COPY %0
+ def Foo : GICombineRule<
+ (defs root:$dst),
+ (match (G_FOO $dst, $src)),
+ (apply (COPY $dst, (i32 0)))>;
+
+ // Example output:
+ // %dst = COPY 0
+ // Note that this would be ill-formed because COPY
+ // expects a register operand!
+ def Bar : GICombineRule<
+ (defs root:$dst),
+ (match (G_FOO $dst, $src)),
+ (apply (COPY $dst, (i32 0)))>;
+
+ // Example output:
+ // %dst = G_CONSTANT i32 0
+ def Bux : GICombineRule<
+ (defs root:$dst),
+ (match (G_FOO $dst, $src)),
+ (apply (G_CONSTANT $dst, (i32 0)))>;
+
+GICombinePatFrag
+----------------
+
+``GICombinePatFrag`` is an equivalent of ``PatFrags`` for MIR patterns.
+They have two main usecases:
+
+* Reduce repetition by creating a ``GICombinePatFrag`` for common
+ patterns (see example 1).
+* Implicitly duplicate a CombineRule for multiple variants of a
+ pattern (see example 2).
+
+A ``GICombinePatFrag`` is composed of three elements:
+
+* zero or more ``in`` (def) parameter
+* zero or more ``out`` parameter
+* A list of MIR patterns that can match.
+
+ * When a ``GICombinePatFrag`` is used within a pattern, the pattern is
+ cloned once for each alternative that can match.
+
+Parameters can have the following types:
+
+* ``gi_mo``, which is the implicit default (no type = ``gi_mo``).
+
+ * Refers to any operand of an instruction (register, BB ref, imm, etc.).
+ * Can be used in both ``in`` and ``out`` parameters.
+ * Users of the PatFrag can only use an operand name for this
+ parameter (e.g. ``(my_pat_frag $foo)``).
+
+* ``root``
+
+ * This is identical to ``gi_mo``.
+ * Can only be used in ``out`` parameters to declare the root of the
+ pattern.
+ * Non-empty ``out`` parameter lists must always have exactly one ``root``.
+
+* ``gi_imm``
+
+ * Refers to an (potentially typed) immediate.
+ * Can only be used in ``in`` parameters.
+ * Users of the PatFrag can only use an immediate for this parameter
+ (e.g. ``(my_pat_frag 0)`` or ``(my_pat_frag (i32 0))``)
+
+``out`` operands can only be empty if the ``GICombinePatFrag`` only contains
+C++ code. If the fragment contains instruction patterns, it has to have at
+least one ``out`` operand of type ``root``.
+
+``in`` operands are less restricted, but there is one important concept to
+remember: you can pass "unbound" operand names, but only if the
+``GICombinePatFrag`` binds it. See example 3 below.
+
+``GICombinePatFrag`` are used just like any other instructions.
+Note that the ``out`` operands are defs, so they come first in the list
+of operands.
+
+.. code-block:: text
+ :caption: Example 1: Reduce Repetition
+
+ def zext_cst : GICombinePatFrag<(outs root:$dst, $cst), (ins gi_imm:$val),
+ [(pattern (G_CONSTANT $cst, $val),
+ (G_ZEXT $dst, $cst))]
+ >;
+
+ def foo_to_impdef : GICombineRule<
+ (defs root:$dst),
+ (match (zext_cst $y, $cst, (i32 0))
+ (G_FOO $dst, $y)),
+ (apply (G_IMPLICIT_DEF $dst))>;
+
+ def store_ext_zero : GICombineRule<
+ (defs root:$root),
+ (match (zext_cst $y, $cst, (i32 0))
+ (G_STORE $y, $ptr):$root),
+ (apply (G_STORE $cst, $ptr):$root)>;
+
+.. code-block:: text
+ :caption: Example 2: Generate Multiple Rules at Once
+
+ // Fold (freeze (freeze x)) -> (freeze x).
+ // Fold (fabs (fabs x)) -> (fabs x).
+ // Fold (fcanonicalize (fcanonicalize x)) -> (fcanonicalize x).
+ def idempotent_prop_frags : GICombinePatFrag<(outs root:$dst, $src), (ins),
+ [
+ (pattern (G_FREEZE $dst, $src), (G_FREEZE $src, $x)),
+ (pattern (G_FABS $dst, $src), (G_FABS $src, $x)),
+ (pattern (G_FCANONICALIZE $dst, $src), (G_FCANONICALIZE $src, $x))
+ ]
+ >;
+
+ def idempotent_prop : GICombineRule<
+ (defs root:$dst),
+ (match (idempotent_prop_frags $dst, $src)),
+ (apply (COPY $dst, $src))>;
+
+
+
+.. code-block:: text
+ :caption: Example 3: Unbound Operand Names
+
+ // This fragment binds $x to an operand in all of its
+ // alternative patterns.
+ def always_binds : GICombinePatFrag<
+ (outs root:$dst), (ins $x),
+ [
+ (pattern (G_FREEZE $dst, $x)),
+ (pattern (G_FABS $dst, $x)),
+ ]
+ >;
+
+ // This fragment does not bind $x to an operand in any
+ // of its alternative patterns.
+ def does_not_bind : GICombinePatFrag<
+ (outs root:$dst), (ins $x),
+ [
+ (pattern (G_FREEZE $dst, $x)), // binds $x
+ (pattern (G_FOO $dst (i32 0))), // does not bind $x
+ (pattern "return myCheck(${x}.getReg())"), // does not bind $x
+ ]
+ >;
+
+ // Here we pass $x, which is unbound, to always_binds.
+ // This works because if $x is unbound, always_binds will bind it for us.
+ def test0 : GICombineRule<
+ (defs root:$dst),
+ (match (always_binds $dst, $x)),
+ (apply (COPY $dst, $x))>;
+
+ // Here we pass $x, which is unbound, to does_not_bind.
+ // This cannot work because $x may not have been initialized in 'apply'.
+ // error: operand 'x' (for parameter 'src' of 'does_not_bind') cannot be unbound
+ def test1 : GICombineRule<
+ (defs root:$dst),
+ (match (does_not_bind $dst, $x)),
+ (apply (COPY $dst, $x))>;
+
+ // Here we pass $x, which is bound, to does_not_bind.
+ // This is fine because $x will always be bound when emitting does_not_bind
+ def test2 : GICombineRule<
+ (defs root:$dst),
+ (match (does_not_bind $tmp, $x)
+ (G_MUL $dst, $x, $tmp)),
+ (apply (COPY $dst, $x))>;
diff --git a/llvm/docs/GlobalISel/index.rst b/llvm/docs/GlobalISel/index.rst
index 78afc1f5a2bc2d..2b639706afdc2f 100644
--- a/llvm/docs/GlobalISel/index.rst
+++ b/llvm/docs/GlobalISel/index.rst
@@ -51,6 +51,7 @@ the following sections.
GMIR
GenericOpcode
+ MIRPatterns
Pipeline
Porting
Resources
diff --git a/llvm/docs/UserGuides.rst b/llvm/docs/UserGuides.rst
index d7335911abe1c2..cc5f983b4454c2 100644
--- a/llvm/docs/UserGuides.rst
+++ b/llvm/docs/UserGuides.rst
@@ -35,6 +35,7 @@ intermediate LLVM representation.
FatLTO
ExtendingLLVM
GoldPlugin
+ GlobalISel/MIRPatterns
HowToBuildOnARM
HowToBuildWithPGO
HowToBuildWindowsItaniumPrograms
@@ -190,6 +191,13 @@ Code Generation
Describes the TableGen tool, which is used heavily by the LLVM code
generator.
+==========
+GlobalISel
+==========
+
+:doc:`MIRPatterns <GlobalISel/MIRPatterns>`
+ Describes the design of MIR Patterns and how to use them.
+
===
JIT
===
@@ -261,4 +269,3 @@ Additional Topics
:doc:`RISCVUsage`
This document describes using the RISCV-V target.
-
diff --git a/llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutor.h b/llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutor.h
index 76b510c5c09a6a..b9f6fec9f46427 100644
--- a/llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutor.h
+++ b/llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutor.h
@@ -368,6 +368,12 @@ enum {
/// - Imm - The immediate to add
GIR_AddImm,
+ /// Add an CImm to the specified instruction
+ /// - InsnID - Instruction ID to modify
+ /// - Ty - Type of the constant immediate.
+ /// - Imm - The immediate to add
+ GIR_AddCImm,
+
/// Render complex operands to the specified instruction
/// - InsnID - Instruction ID to modify
/// - RendererID - The renderer to call
@@ -562,7 +568,8 @@ class GIMatchTableExecutor {
const int64_t *MatchTable, const TargetInstrInfo &TII,
MachineRegisterInfo &MRI, const TargetRegisterInfo &TRI,
const RegisterBankInfo &RBI, const PredicateBitset &AvailableFeatures,
- CodeGenCoverage *CoverageInfo) const;
+ CodeGenCoverage *CoverageInfo,
+ GISelChangeObserver *Observer = nullptr) const;
virtual const int64_t *getMatchTable() const {
llvm_unreachable("Should have been overridden by tablegen if used");
@@ -590,12 +597,14 @@ class GIMatchTableExecutor {
llvm_unreachable("Subclass does not implement testSimplePredicate!");
}
- virtual void runCustomAction(unsigned, const MatcherState &State) const {
+ virtual void runCustomAction(unsigned, const MatcherState &State,
+ NewMIVector &OutMIs) const {
llvm_unreachable("Subclass does not implement runCustomAction!");
}
bool isOperandImmEqual(const MachineOperand &MO, int64_t Value,
- const MachineRegisterInfo &MRI) const;
+ const MachineRegisterInfo &MRI,
+ bool Splat = false) const;
/// Return true if the specified operand is a G_PTR_ADD with a G_CONSTANT on
/// the right-hand side. GlobalISel's separation of pointer and integer types
diff --git a/llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutorImpl.h b/llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutorImpl.h
index 533eb4cb0d7a68..e662e75ec4424d 100644
--- a/llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutorImpl.h
+++ b/llvm/include/llvm/CodeGen/GlobalISel/GIMatchTableExecutorImpl.h
@@ -17,6 +17,7 @@
#include "llvm/ADT/SmallVector.h"
#include "llvm/CodeGen/GlobalISel/GIMatchTableExecutor.h"
+#include "llvm/CodeGen/GlobalISel/GISelChangeObserver.h"
#include "llvm/CodeGen/GlobalISel/Utils.h"
#include "llvm/CodeGen/MachineInstrBuilder.h"
#include "llvm/CodeGen/MachineOperand.h"
@@ -27,6 +28,7 @@
#include "llvm/CodeGen/TargetRegisterInfo.h"
#include "llvm/IR/Constants.h"
#include "llvm/IR/DataLayout.h"
+#include "llvm/IR/Type.h"
#include "llvm/Support/CodeGenCoverage.h"
#include "llvm/Support/Debug.h"
#include "llvm/Support/ErrorHandling.h"
@@ -46,7 +48,7 @@ bool GIMatchTableExecutor::executeMatchTable(
const int64_t *MatchTable, const TargetInstrInfo &TII,
MachineRegisterInfo &MRI, const TargetRegisterInfo &TRI,
const RegisterBankInfo &RBI, const PredicateBitset &AvailableFeatures,
- CodeGenCoverage *CoverageInfo) const {
+ CodeGenCoverage *CoverageInfo, GISelChangeObserver *Observer) const {
uint64_t CurrentIdx = 0;
SmallVector<uint64_t, 4> OnFailResumeAt;
@@ -299,8 +301,7 @@ bool GIMatchTableExecutor::executeMatchTable(
assert(State.MIs[InsnID] != nullptr && "Used insn before defined");
assert(State.MIs[InsnID]->getOpcode() == TargetOpcode::G_CONSTANT &&
"Expected G_CONSTANT");
- assert(Predicate > GICXXPred_Invalid &&
- "Expected a valid predicate");
+ assert(Predicate > GICXXPred_Invalid && "Expected a valid predicate");
if (!State.MIs[InsnID]->getOperand(1).isCImm())
llvm_unreachable("Expected Imm or CImm operand");
@@ -323,8 +324,7 @@ bool GIMatchTableExecutor::executeMatchTable(
"Expected G_FCONSTANT");
assert(State.MIs[InsnID]->getOperand(1).isFPImm() &&
"Expected FPImm operand");
- assert(Predicate > GICXXPred_Invalid &&
- "Expected a valid predicate");
+ assert(Predicate > GICXXPred_Invalid && "Expected a valid predicate");
const APFloat &Value =
State.MIs[InsnID]->getOperand(1).getFPImm()->getValueAPF();
@@ -727,9 +727,16 @@ bool GIMatchTableExecutor::executeMatchTable(
if (MO.isReg()) {
// isOperandImmEqual() will sign-extend to 64-bits, so should we.
LLT Ty = MRI.getType(MO.getReg());
- Value = SignExtend64(Value, Ty.getSizeInBits());
+ // If the type is > 64 bits, it can't be a constant int, so we bail
+ // early because SignExtend64 will assert otherwise.
+ if (Ty.getScalarSizeInBits() > 64) {
+ if (handleReject() == RejectAndGiveUp)
+ return false;
+ break;
+ }
- if (!isOperandImmEqual(MO, Value, MRI)) {
+ Value = SignExtend64(Value, Ty.getScalarSizeInBits());
+ if (!isOperandImmEqual(MO, Value, MRI, /*Splat=*/true)) {
if (handleReject() == RejectAndGiveUp)
return false;
}
@@ -1022,6 +1029,23 @@ bool GIMatchTableExecutor::executeMatchTable(
break;
}
+ case GIR_AddCImm: {
+ int64_t InsnID = MatchTable[CurrentIdx++];
+ int64_t TypeID = MatchTable[CurrentIdx++];
+ int64_t Imm = MatchTable[CurrentIdx++];
+ assert(OutMIs[InsnID] && "Attempted to add to undefined instruction");
+
+ unsigned Width = ExecInfo.TypeObjects[TypeID].getScalarSizeInBits();
+ LLVMContext &Ctx = MF->getFunction().getContext();
+ OutMIs[InsnID].addCImm(
+ ConstantInt::get(IntegerType::get(Ctx, Width), Imm, /*signed*/ true));
+ DEBUG_WITH_TYPE(TgtExecutor::getName(),
+ dbgs() << CurrentIdx << ": GIR_AddCImm(OutMIs[" << InsnID
+ << "], TypeID=" << TypeID << ", Imm=" << Imm
+ << ")\n");
+ break;
+ }
+
case GIR_ComplexRenderer: {
int64_t InsnID = MatchTable[CurrentIdx++];
int64_t RendererID = MatchTable[CurrentIdx++];
@@ -1121,7 +1145,7 @@ bool GIMatchTableExecutor::executeMatchTable(
dbgs() << CurrentIdx << ": GIR_CustomAction(FnID=" << FnID
<< ")\n");
assert(FnID > GICXXCustomAction_Invalid && "Expected a valid FnID");
- runCustomAction(FnID, State);
+ runCustomAction(FnID, State, OutMIs);
break;
}
case GIR_CustomOperandRenderer: {
@@ -1191,12 +1215,14 @@ bool GIMatchTableExecutor::executeMatchTable(
case GIR_EraseFromParent: {
int64_t InsnID = MatchTable[CurrentIdx++];
- assert(State.MIs[InsnID] &&
- "Attempted to erase an undefined instruction");
- State.MIs[InsnID]->eraseFromParent();
+ MachineInstr *MI = State.MIs[InsnID];
+ assert(MI && "Attempted to erase an undefined instruction");
DEBUG_WITH_TYPE(TgtExecutor::getName(),
dbgs() << CurrentIdx << ": GIR_EraseFromParent(MIs["
<< InsnID << "])\n");
+ if (Observer)
+ Observer->erasingInstr(*MI);
+ MI->eraseFromParent();
break;
}
@@ -1226,6 +1252,10 @@ bool GIMatchTableExecutor::executeMatchTable(
case GIR_Done:
DEBUG_WITH_TYPE(TgtExecutor::getName(),
dbgs() << CurrentIdx << ": GIR_Done\n");
+ if (Observer) {
+ for (MachineInstr *MI : OutMIs)
+ Observer->createdInstr(*MI);
+ }
propagateFlags(OutMIs);
return true;
default:
diff --git a/llvm/include/llvm/Target/GlobalISel/Combine.td b/llvm/include/llvm/Target/GlobalISel/Combine.td
index 08cbfc02b6bfa1..5c3d382f210212 100644
--- a/llvm/include/llvm/Target/GlobalISel/Combine.td
+++ b/llvm/include/llvm/Target/GlobalISel/Combine.td
@@ -66,10 +66,11 @@ class GICombineRule<dag defs, dag match, dag apply> : GICombine {
/// is called. Targets can use this to, for instance, check Subtarget
/// features.
list<Predicate> Predicates = [];
-}
-/// The operator at the root of a GICombineRule.Defs dag.
-def defs;
+ // Maximum number of permutations of this rule that can be emitted.
+ // Set to -1 to disable the limit.
+ int MaxPermutations = 16;
+}
/// All arguments of the defs operator must be subclasses of GIDefKind or
/// sub-dags whose operator is GIDefKindWithArgs.
@@ -82,6 +83,30 @@ class GIDefKindWithArgs;
/// is incorrect.
def root : GIDefKind;
+def gi_mo;
+def gi_imm;
+def pattern;
+
+// This is an equivalent of PatFrags but for MIR Patterns.
+//
+// GICombinePatFrags can be used in place of instructions for 'match' patterns.
+// Much like normal instructions, the defs (outs) come first, and the ins second
+//
+// Out operands can only be of type "root" or "gi_mo", and they must be defined
+// by an instruction pattern in all alternatives.
+//
+// In operands can be gi_imm or gi_mo. They cannot be redefined in any alternative
+// pattern and may only appear in the C++ code, or in the output operand of an
+// instruction pattern.
+class GICombinePatFrag<dag outs, dag ins, list<dag> alts> {
+ dag InOperands = ins;
+ dag OutOperands = outs;
+ list<dag> Alternatives = alts;
+}
+
+/// The operator at the root of a GICombineRule.Defs dag.
+def defs;
+
/// Declares data that is passed from the match stage to the apply stage.
class GIDefMatchData<string type> : GIDefKind {
/// A C++ type name indicating the storage type.
diff --git a/llvm/lib/CodeGen/GlobalISel/GIMatchTableExecutor.cpp b/llvm/lib/CodeGen/GlobalISel/GIMatchTableExecutor.cpp
index d747cbf5aadc84..26752369a7711a 100644
--- a/llvm/lib/CodeGen/GlobalISel/GIMatchTableExecutor.cpp
+++ b/llvm/lib/CodeGen/GlobalISel/GIMatchTableExecutor.cpp
@@ -26,12 +26,19 @@ GIMatchTableExecutor::MatcherState::MatcherState(unsigned MaxRenderers)
GIMatchTableExecutor::GIMatchTableExecutor() = default;
-bool GIMatchTableExecutor::isOperandImmEqual(
- const MachineOperand &MO, int64_t Value,
- const MachineRegisterInfo &MRI) const {
- if (MO.isReg() && MO.getReg())
+bool GIMatchTableExecutor::isOperandImmEqual(const MachineOperand &MO,
+ int64_t Value,
+ const MachineRegisterInfo &MRI,
+ bool Splat) const {
+ if (MO.isReg() && MO.getReg()) {
if (auto VRegVal = getIConstantVRegValWithLookThrough(MO.getReg(), MRI))
return VRegVal->Value.getSExtValue() == Value;
+
+ if (Splat) {
+ if (auto VRegVal = getIConstantSplatVal(MO.getReg(), MRI))
+ return VRegVal->getSExtValue() == Value;
+ }
+ }
return false;
}
diff --git a/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-imms.td b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-imms.td
new file mode 100644
index 00000000000000..703075d7874ded
--- /dev/null
+++ b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-imms.td
@@ -0,0 +1,86 @@
+// RUN: llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
+// RUN: -combiners=MyCombiner %s | \
+// RUN: FileCheck %s
+
+include "llvm/Target/Target.td"
+include "llvm/Target/GlobalISel/Combine.td"
+
+def MyTargetISA : InstrInfo;
+def MyTarget : Target { let InstructionSet = MyTargetISA; }
+
+// emits a imm
+def InstTest0 : GICombineRule<
+ (defs root:$a),
+ (match (COPY $a, (i32 0))),
+ (apply (COPY $a, 0))>;
+
+// implicitly emits a G_CONSTANT
+def InstTest1 : GICombineRule<
+ (defs root:$a),
+ (match (G_ZEXT $a, 0)),
+ (apply (COPY $a, (i32 0)))>;
+
+// emits a CImm because G_CONSTANT is a special case.
+def CImmInstTest1 : GICombineRule<
+ (defs root:$a),
+ (match (G_CONSTANT $a, (i32 0))),
+ (apply (G_CONSTANT $a, (i32 42)))>;
+
+def MyCombiner: GICombinerHelper<"GenMyCombiner", [
+ InstTest0,
+ InstTest1,
+ CImmInstTest1
+]>;
+
+// CHECK: const int64_t *GenMyCombiner::getMatchTable() const {
+// CHECK-NEXT: constexpr static int64_t MatchTable0[] = {
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 0*/ 28, // Rule ID 0 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::COPY,
+// CHECK-NEXT: GIM_CheckType, /*MI*/0, /*Op*/1, /*Type*/GILLT_s32,
+// CHECK-NEXT: // MIs[0] a
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: GIM_CheckConstantInt, /*MI*/0, /*Op*/1, 0,
+// CHECK-NEXT: // Combiner Rule #0: InstTest0
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // a
+// CHECK-NEXT: GIR_AddImm, /*InsnID*/0, /*Imm*/0,
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 0: @28
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 1*/ 67, // Rule ID 1 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule1Enabled,
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_ZEXT,
+// CHECK-NEXT: // MIs[0] a
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] Operand 1
+// CHECK-NEXT: GIM_CheckConstantInt, /*MI*/0, /*Op*/1, 0,
+// CHECK-NEXT: GIR_MakeTempReg, /*TempRegID*/0, /*TypeID*/GILLT_s32,
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/1, /*Opcode*/TargetOpcode::G_CONSTANT,
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/1, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_AddCImm, /*InsnID*/1, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT: // Combiner Rule #1: InstTest1
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // a
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/0, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 1: @67
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 2*/ 96, // Rule ID 2 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule2Enabled,
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_CONSTANT,
+// CHECK-NEXT: GIM_CheckType, /*MI*/0, /*Op*/1, /*Type*/GILLT_s32,
+// CHECK-NEXT: // MIs[0] a
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: GIM_CheckLiteralInt, /*MI*/0, /*Op*/1, 0,
+// CHECK-NEXT: // Combiner Rule #2: CImmInstTest1
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::G_CONSTANT,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // a
+// CHECK-NEXT: GIR_AddCImm, /*InsnID*/0, /*Type*/GILLT_s32, /*Imm*/42,
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 2: @96
+// CHECK-NEXT: GIM_Reject,
+// CHECK-NEXT: };
+// CHECK-NEXT: return MatchTable0;
+// CHECK-NEXT: }
diff --git a/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-operand-types.td b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-operand-types.td
new file mode 100644
index 00000000000000..d33fddbc376812
--- /dev/null
+++ b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-operand-types.td
@@ -0,0 +1,52 @@
+// RUN: llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
+// RUN: -combiners=MyCombiner %s | \
+// RUN: FileCheck %s
+
+include "llvm/Target/Target.td"
+include "llvm/Target/GlobalISel/Combine.td"
+
+def MyTargetISA : InstrInfo;
+def MyTarget : Target { let InstructionSet = MyTargetISA; }
+
+def InstTest0 : GICombineRule<
+ (defs root:$a),
+ (match (G_MUL i32:$x, i32:$b, i32:$c),
+ (G_MUL $a, i32:$b, i32:$x)),
+ (apply (G_ADD i64:$tmp, $b, i32:$c),
+ (G_ADD i8:$a, $b, i64:$tmp))>;
+
+def MyCombiner: GICombinerHelper<"GenMyCombiner", [
+ InstTest0,
+]>;
+
+// CHECK: const int64_t *GenMyCombiner::getMatchTable() const {
+// CHECK-NEXT: constexpr static int64_t MatchTable0[] = {
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 0*/ 73, // Rule ID 0 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_MUL,
+// CHECK-NEXT: GIM_CheckType, /*MI*/0, /*Op*/0, /*Type*/GILLT_s8,
+// CHECK-NEXT: GIM_CheckType, /*MI*/0, /*Op*/1, /*Type*/GILLT_s32,
+// CHECK-NEXT: GIM_CheckType, /*MI*/0, /*Op*/2, /*Type*/GILLT_s32,
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/2, // MIs[1]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_MUL,
+// CHECK-NEXT: GIM_CheckType, /*MI*/1, /*Op*/2, /*Type*/GILLT_s32,
+// CHECK-NEXT: // MIs[1] b
+// CHECK-NEXT: GIM_CheckIsSameOperandIgnoreCopies, /*MI*/1, /*OpIdx*/1, /*OtherMI*/0, /*OtherOpIdx*/1,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT: GIR_MakeTempReg, /*TempRegID*/0, /*TypeID*/GILLT_s64,
+// CHECK-NEXT: // Combiner Rule #0: InstTest0
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::G_ADD,
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/0, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/1, // b
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/1, /*OpIdx*/2, // c
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/1, /*Opcode*/TargetOpcode::G_ADD,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/1, /*OldInsnID*/0, /*OpIdx*/0, // a
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/1, /*OldInsnID*/0, /*OpIdx*/1, // b
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/1, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 0: @73
+// CHECK-NEXT: GIM_Reject,
+// CHECK-NEXT: };
+// CHECK-NEXT: return MatchTable0;
+// CHECK-NEXT: }
diff --git a/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-patfrag-root.td b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-patfrag-root.td
new file mode 100644
index 00000000000000..28eb4920a6b713
--- /dev/null
+++ b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-patfrag-root.td
@@ -0,0 +1,92 @@
+// RUN: llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
+// RUN: -combiners=MyCombiner -gicombiner-matchtable-debug-cxxpreds %s | \
+// RUN: FileCheck %s
+
+include "llvm/Target/Target.td"
+include "llvm/Target/GlobalISel/Combine.td"
+
+def MyTargetISA : InstrInfo;
+def MyTarget : Target { let InstructionSet = MyTargetISA; }
+
+def MatchFooPerms: GICombinePatFrag<
+ (outs root:$foo),
+ (ins gi_imm:$cst),
+ [
+ (pattern (G_ZEXT $foo, $b), (G_TRUNC $b, $x):$dbg0),
+ (pattern (G_TRUNC $foo, $z):$dbg1),
+ (pattern (G_FPEXT $foo, $z):$dbg1)
+ ]>;
+
+def Test0 : GICombineRule<
+ (defs root:$root),
+ (match (MatchFooPerms $root, (i32 10))),
+ (apply (COPY $root, (i32 0)))>;
+
+def MyCombiner: GICombinerHelper<"GenMyCombiner", [
+ Test0
+]>;
+
+// CHECK: const int64_t *GenMyCombiner::getMatchTable() const {
+// CHECK-NEXT: constexpr static int64_t MatchTable0[] = {
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 0*/ 44, // Rule ID 0 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_ZEXT,
+// CHECK-NEXT: // MIs[0] root
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] __Test0_match_0.b
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[1] __Test0_match_0.x
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT: GIR_MakeTempReg, /*TempRegID*/0, /*TypeID*/GILLT_s32,
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/1, /*Opcode*/TargetOpcode::G_CONSTANT,
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/1, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_AddCImm, /*InsnID*/1, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT: // Combiner Rule #0: Test0 @ [__Test0_match_0[0]]
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // root
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/0, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 0: @44
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 1*/ 79, // Rule ID 1 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[0] root
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] __Test0_match_0.z
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: GIR_MakeTempReg, /*TempRegID*/0, /*TypeID*/GILLT_s32,
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/1, /*Opcode*/TargetOpcode::G_CONSTANT,
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/1, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_AddCImm, /*InsnID*/1, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT: // Combiner Rule #0: Test0 @ [__Test0_match_0[1]]
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // root
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/0, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 1: @79
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 2*/ 114, // Rule ID 2 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_FPEXT,
+// CHECK-NEXT: // MIs[0] root
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] __Test0_match_0.z
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: GIR_MakeTempReg, /*TempRegID*/0, /*TypeID*/GILLT_s32,
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/1, /*Opcode*/TargetOpcode::G_CONSTANT,
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/1, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_AddCImm, /*InsnID*/1, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT: // Combiner Rule #0: Test0 @ [__Test0_match_0[2]]
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // root
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/0, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 2: @114
+// CHECK-NEXT: GIM_Reject,
+// CHECK-NEXT: };
+// CHECK-NEXT: return MatchTable0;
+// CHECK-NEXT: }
diff --git a/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-permutations.td b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-permutations.td
new file mode 100644
index 00000000000000..513a14849218de
--- /dev/null
+++ b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-permutations.td
@@ -0,0 +1,545 @@
+// RUN: llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
+// RUN: -combiners=MyCombiner -gicombiner-matchtable-debug-cxxpreds %s | \
+// RUN: FileCheck %s
+
+include "llvm/Target/Target.td"
+include "llvm/Target/GlobalISel/Combine.td"
+
+def MyTargetISA : InstrInfo;
+def MyTarget : Target { let InstructionSet = MyTargetISA; }
+
+def MatchFooPerms: GICombinePatFrag<
+ (outs root:$foo),
+ (ins gi_imm:$cst),
+ [
+ (pattern (G_ZEXT $foo, $b), (G_TRUNC $b, $x):$dbg0, "return foo(${x}, ${cst})"),
+ (pattern (G_TRUNC $foo, $z):$dbg1, "return bar(${foo}, ${cst})")
+ ]>;
+
+def Test0 : GICombineRule<
+ (defs root:$dst),
+ (match (G_AND $dst, $cst0, $tmp),
+ (G_AND $tmp, $cst1, $cst2),
+ (MatchFooPerms $cst0, (i32 10)):$a,
+ (MatchFooPerms $cst1, (i32 20)):$b,
+ (MatchFooPerms $cst2, (i32 30)):$c
+ ),
+ (apply (COPY $dst, (i32 0)), "APPLY ${cst0}")>;
+
+def MyCombiner: GICombinerHelper<"GenMyCombiner", [
+ Test0
+]>;
+
+// CHECK: bool GenMyCombiner::testMIPredicate_MI(unsigned PredicateID, const MachineInstr & MI, const MatcherState &State) const {
+// CHECK-NEXT: switch (PredicateID) {
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner0: {
+// CHECK-NEXT: // Pattern Alternatives: [a[0], b[0], c[0]]
+// CHECK-NEXT: return foo(State.MIs[2]->getOperand(1), 10)
+// CHECK-NEXT: llvm_unreachable("GICombiner0 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner1: {
+// CHECK-NEXT: // Pattern Alternatives: [a[0], b[0], c[0]]
+// CHECK-NEXT: return foo(State.MIs[5]->getOperand(1), 20)
+// CHECK-NEXT: llvm_unreachable("GICombiner1 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner2: {
+// CHECK-NEXT: // Pattern Alternatives: [a[0], b[0], c[0]]
+// CHECK-NEXT: return foo(State.MIs[7]->getOperand(1), 30)
+// CHECK-NEXT: llvm_unreachable("GICombiner2 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner3: {
+// CHECK-NEXT: // Pattern Alternatives: [a[0], b[0], c[1]]
+// CHECK-NEXT: return foo(State.MIs[2]->getOperand(1), 10)
+// CHECK-NEXT: llvm_unreachable("GICombiner3 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner4: {
+// CHECK-NEXT: // Pattern Alternatives: [a[0], b[0], c[1]]
+// CHECK-NEXT: return foo(State.MIs[5]->getOperand(1), 20)
+// CHECK-NEXT: llvm_unreachable("GICombiner4 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner5: {
+// CHECK-NEXT: // Pattern Alternatives: [a[0], b[0], c[1]]
+// CHECK-NEXT: return bar(State.MIs[3]->getOperand(2), 30)
+// CHECK-NEXT: llvm_unreachable("GICombiner5 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner6: {
+// CHECK-NEXT: // Pattern Alternatives: [a[0], b[1], c[0]]
+// CHECK-NEXT: return foo(State.MIs[2]->getOperand(1), 10)
+// CHECK-NEXT: llvm_unreachable("GICombiner6 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner7: {
+// CHECK-NEXT: // Pattern Alternatives: [a[0], b[1], c[0]]
+// CHECK-NEXT: return bar(State.MIs[3]->getOperand(1), 20)
+// CHECK-NEXT: llvm_unreachable("GICombiner7 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner8: {
+// CHECK-NEXT: // Pattern Alternatives: [a[0], b[1], c[0]]
+// CHECK-NEXT: return foo(State.MIs[6]->getOperand(1), 30)
+// CHECK-NEXT: llvm_unreachable("GICombiner8 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner9: {
+// CHECK-NEXT: // Pattern Alternatives: [a[0], b[1], c[1]]
+// CHECK-NEXT: return foo(State.MIs[2]->getOperand(1), 10)
+// CHECK-NEXT: llvm_unreachable("GICombiner9 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner10: {
+// CHECK-NEXT: // Pattern Alternatives: [a[0], b[1], c[1]]
+// CHECK-NEXT: return bar(State.MIs[3]->getOperand(1), 20)
+// CHECK-NEXT: llvm_unreachable("GICombiner10 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner11: {
+// CHECK-NEXT: // Pattern Alternatives: [a[0], b[1], c[1]]
+// CHECK-NEXT: return bar(State.MIs[3]->getOperand(2), 30)
+// CHECK-NEXT: llvm_unreachable("GICombiner11 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner12: {
+// CHECK-NEXT: // Pattern Alternatives: [a[1], b[0], c[0]]
+// CHECK-NEXT: return bar(State.MIs[0]->getOperand(1), 10)
+// CHECK-NEXT: llvm_unreachable("GICombiner12 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner13: {
+// CHECK-NEXT: // Pattern Alternatives: [a[1], b[0], c[0]]
+// CHECK-NEXT: return foo(State.MIs[4]->getOperand(1), 20)
+// CHECK-NEXT: llvm_unreachable("GICombiner13 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner14: {
+// CHECK-NEXT: // Pattern Alternatives: [a[1], b[0], c[0]]
+// CHECK-NEXT: return foo(State.MIs[6]->getOperand(1), 30)
+// CHECK-NEXT: llvm_unreachable("GICombiner14 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner15: {
+// CHECK-NEXT: // Pattern Alternatives: [a[1], b[0], c[1]]
+// CHECK-NEXT: return bar(State.MIs[0]->getOperand(1), 10)
+// CHECK-NEXT: llvm_unreachable("GICombiner15 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner16: {
+// CHECK-NEXT: // Pattern Alternatives: [a[1], b[0], c[1]]
+// CHECK-NEXT: return foo(State.MIs[4]->getOperand(1), 20)
+// CHECK-NEXT: llvm_unreachable("GICombiner16 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner17: {
+// CHECK-NEXT: // Pattern Alternatives: [a[1], b[0], c[1]]
+// CHECK-NEXT: return bar(State.MIs[3]->getOperand(2), 30)
+// CHECK-NEXT: llvm_unreachable("GICombiner17 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner18: {
+// CHECK-NEXT: // Pattern Alternatives: [a[1], b[1], c[0]]
+// CHECK-NEXT: return bar(State.MIs[0]->getOperand(1), 10)
+// CHECK-NEXT: llvm_unreachable("GICombiner18 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner19: {
+// CHECK-NEXT: // Pattern Alternatives: [a[1], b[1], c[0]]
+// CHECK-NEXT: return bar(State.MIs[3]->getOperand(1), 20)
+// CHECK-NEXT: llvm_unreachable("GICombiner19 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner20: {
+// CHECK-NEXT: // Pattern Alternatives: [a[1], b[1], c[0]]
+// CHECK-NEXT: return foo(State.MIs[5]->getOperand(1), 30)
+// CHECK-NEXT: llvm_unreachable("GICombiner20 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner21: {
+// CHECK-NEXT: // Pattern Alternatives: [a[1], b[1], c[1]]
+// CHECK-NEXT: return bar(State.MIs[0]->getOperand(1), 10)
+// CHECK-NEXT: llvm_unreachable("GICombiner21 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner22: {
+// CHECK-NEXT: // Pattern Alternatives: [a[1], b[1], c[1]]
+// CHECK-NEXT: return bar(State.MIs[3]->getOperand(1), 20)
+// CHECK-NEXT: llvm_unreachable("GICombiner22 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner23: {
+// CHECK-NEXT: // Pattern Alternatives: [a[1], b[1], c[1]]
+// CHECK-NEXT: return bar(State.MIs[3]->getOperand(2), 30)
+// CHECK-NEXT: llvm_unreachable("GICombiner23 should have returned");
+// CHECK-NEXT: }
+// CHECK-NEXT: }
+// CHECK-NEXT: llvm_unreachable("Unknown predicate");
+// CHECK-NEXT: return false;
+// CHECK-NEXT: }
+
+// CHECK: const int64_t *GenMyCombiner::getMatchTable() const {
+// CHECK-NEXT: constexpr static int64_t MatchTable0[] = {
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 0*/ 746,
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_AND,
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 1*/ 111, // Rule ID 0 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT: // MIs[0] dst
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] cst0
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_ZEXT,
+// CHECK-NEXT: // MIs[1] a.b
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/2, /*MI*/1, /*OpIdx*/1, // MIs[2]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[2] a.x
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] tmp
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/3, /*MI*/0, /*OpIdx*/2, // MIs[3]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/3, TargetOpcode::G_AND,
+// CHECK-NEXT: // MIs[3] cst1
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/4, /*MI*/3, /*OpIdx*/1, // MIs[4]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/4, TargetOpcode::G_ZEXT,
+// CHECK-NEXT: // MIs[4] b.b
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/5, /*MI*/4, /*OpIdx*/1, // MIs[5]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/5, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[5] b.x
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[3] cst2
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/6, /*MI*/3, /*OpIdx*/2, // MIs[6]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/6, TargetOpcode::G_ZEXT,
+// CHECK-NEXT: // MIs[6] c.b
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/7, /*MI*/6, /*OpIdx*/1, // MIs[7]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/7, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[7] c.x
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner0,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner1,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner2,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/2,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/3,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/4,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/5,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/6,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/7,
+// CHECK-NEXT: GIR_MakeTempReg, /*TempRegID*/0, /*TypeID*/GILLT_s32,
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/1, /*Opcode*/TargetOpcode::G_CONSTANT,
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/1, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_AddCImm, /*InsnID*/1, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT: // Combiner Rule #0: Test0 @ [a[0], b[0], c[0]]
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/0, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 1: @111
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 2*/ 208, // Rule ID 1 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT: // MIs[0] dst
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] cst0
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_ZEXT,
+// CHECK-NEXT: // MIs[1] a.b
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/2, /*MI*/1, /*OpIdx*/1, // MIs[2]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[2] a.x
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] tmp
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/3, /*MI*/0, /*OpIdx*/2, // MIs[3]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/3, TargetOpcode::G_AND,
+// CHECK-NEXT: // MIs[3] cst1
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/4, /*MI*/3, /*OpIdx*/1, // MIs[4]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/4, TargetOpcode::G_ZEXT,
+// CHECK-NEXT: // MIs[4] b.b
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/5, /*MI*/4, /*OpIdx*/1, // MIs[5]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/5, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[5] b.x
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[3] cst2
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/6, /*MI*/3, /*OpIdx*/2, // MIs[6]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/6, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[6] c.z
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner3,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner4,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner5,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/2,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/3,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/4,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/5,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/6,
+// CHECK-NEXT: GIR_MakeTempReg, /*TempRegID*/0, /*TypeID*/GILLT_s32,
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/1, /*Opcode*/TargetOpcode::G_CONSTANT,
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/1, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_AddCImm, /*InsnID*/1, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT: // Combiner Rule #0: Test0 @ [a[0], b[0], c[1]]
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/0, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 2: @208
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 3*/ 305, // Rule ID 2 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT: // MIs[0] dst
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] cst0
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_ZEXT,
+// CHECK-NEXT: // MIs[1] a.b
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/2, /*MI*/1, /*OpIdx*/1, // MIs[2]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[2] a.x
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] tmp
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/3, /*MI*/0, /*OpIdx*/2, // MIs[3]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/3, TargetOpcode::G_AND,
+// CHECK-NEXT: // MIs[3] cst1
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/4, /*MI*/3, /*OpIdx*/1, // MIs[4]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/4, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[4] b.z
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[3] cst2
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/5, /*MI*/3, /*OpIdx*/2, // MIs[5]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/5, TargetOpcode::G_ZEXT,
+// CHECK-NEXT: // MIs[5] c.b
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/6, /*MI*/5, /*OpIdx*/1, // MIs[6]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/6, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[6] c.x
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner6,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner7,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner8,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/2,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/3,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/4,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/5,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/6,
+// CHECK-NEXT: GIR_MakeTempReg, /*TempRegID*/0, /*TypeID*/GILLT_s32,
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/1, /*Opcode*/TargetOpcode::G_CONSTANT,
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/1, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_AddCImm, /*InsnID*/1, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT: // Combiner Rule #0: Test0 @ [a[0], b[1], c[0]]
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/0, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 3: @305
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 4*/ 393, // Rule ID 3 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT: // MIs[0] dst
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] cst0
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_ZEXT,
+// CHECK-NEXT: // MIs[1] a.b
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/2, /*MI*/1, /*OpIdx*/1, // MIs[2]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[2] a.x
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] tmp
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/3, /*MI*/0, /*OpIdx*/2, // MIs[3]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/3, TargetOpcode::G_AND,
+// CHECK-NEXT: // MIs[3] cst1
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/4, /*MI*/3, /*OpIdx*/1, // MIs[4]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/4, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[4] b.z
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[3] cst2
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/5, /*MI*/3, /*OpIdx*/2, // MIs[5]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/5, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[5] c.z
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner9,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner10,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner11,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/2,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/3,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/4,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/5,
+// CHECK-NEXT: GIR_MakeTempReg, /*TempRegID*/0, /*TypeID*/GILLT_s32,
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/1, /*Opcode*/TargetOpcode::G_CONSTANT,
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/1, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_AddCImm, /*InsnID*/1, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT: // Combiner Rule #0: Test0 @ [a[0], b[1], c[1]]
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/0, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 4: @393
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 5*/ 490, // Rule ID 4 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT: // MIs[0] dst
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] cst0
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[1] a.z
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] tmp
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/2, /*MI*/0, /*OpIdx*/2, // MIs[2]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_AND,
+// CHECK-NEXT: // MIs[2] cst1
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/3, /*MI*/2, /*OpIdx*/1, // MIs[3]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/3, TargetOpcode::G_ZEXT,
+// CHECK-NEXT: // MIs[3] b.b
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/4, /*MI*/3, /*OpIdx*/1, // MIs[4]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/4, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[4] b.x
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[2] cst2
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/5, /*MI*/2, /*OpIdx*/2, // MIs[5]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/5, TargetOpcode::G_ZEXT,
+// CHECK-NEXT: // MIs[5] c.b
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/6, /*MI*/5, /*OpIdx*/1, // MIs[6]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/6, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[6] c.x
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner12,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner13,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner14,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/2,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/3,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/4,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/5,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/6,
+// CHECK-NEXT: GIR_MakeTempReg, /*TempRegID*/0, /*TypeID*/GILLT_s32,
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/1, /*Opcode*/TargetOpcode::G_CONSTANT,
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/1, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_AddCImm, /*InsnID*/1, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT: // Combiner Rule #0: Test0 @ [a[1], b[0], c[0]]
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/0, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 5: @490
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 6*/ 578, // Rule ID 5 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT: // MIs[0] dst
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] cst0
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[1] a.z
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] tmp
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/2, /*MI*/0, /*OpIdx*/2, // MIs[2]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_AND,
+// CHECK-NEXT: // MIs[2] cst1
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/3, /*MI*/2, /*OpIdx*/1, // MIs[3]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/3, TargetOpcode::G_ZEXT,
+// CHECK-NEXT: // MIs[3] b.b
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/4, /*MI*/3, /*OpIdx*/1, // MIs[4]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/4, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[4] b.x
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[2] cst2
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/5, /*MI*/2, /*OpIdx*/2, // MIs[5]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/5, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[5] c.z
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner15,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner16,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner17,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/2,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/3,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/4,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/5,
+// CHECK-NEXT: GIR_MakeTempReg, /*TempRegID*/0, /*TypeID*/GILLT_s32,
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/1, /*Opcode*/TargetOpcode::G_CONSTANT,
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/1, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_AddCImm, /*InsnID*/1, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT: // Combiner Rule #0: Test0 @ [a[1], b[0], c[1]]
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/0, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 6: @578
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 7*/ 666, // Rule ID 6 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT: // MIs[0] dst
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] cst0
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[1] a.z
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] tmp
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/2, /*MI*/0, /*OpIdx*/2, // MIs[2]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_AND,
+// CHECK-NEXT: // MIs[2] cst1
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/3, /*MI*/2, /*OpIdx*/1, // MIs[3]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/3, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[3] b.z
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[2] cst2
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/4, /*MI*/2, /*OpIdx*/2, // MIs[4]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/4, TargetOpcode::G_ZEXT,
+// CHECK-NEXT: // MIs[4] c.b
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/5, /*MI*/4, /*OpIdx*/1, // MIs[5]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/5, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[5] c.x
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner18,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner19,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner20,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/2,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/3,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/4,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/5,
+// CHECK-NEXT: GIR_MakeTempReg, /*TempRegID*/0, /*TypeID*/GILLT_s32,
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/1, /*Opcode*/TargetOpcode::G_CONSTANT,
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/1, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_AddCImm, /*InsnID*/1, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT: // Combiner Rule #0: Test0 @ [a[1], b[1], c[0]]
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/0, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 7: @666
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 8*/ 745, // Rule ID 7 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT: // MIs[0] dst
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] cst0
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[1] a.z
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] tmp
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/2, /*MI*/0, /*OpIdx*/2, // MIs[2]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/2, TargetOpcode::G_AND,
+// CHECK-NEXT: // MIs[2] cst1
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/3, /*MI*/2, /*OpIdx*/1, // MIs[3]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/3, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[3] b.z
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[2] cst2
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/4, /*MI*/2, /*OpIdx*/2, // MIs[4]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/4, TargetOpcode::G_TRUNC,
+// CHECK-NEXT: // MIs[4] c.z
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner21,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner22,
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner23,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/2,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/3,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/4,
+// CHECK-NEXT: GIR_MakeTempReg, /*TempRegID*/0, /*TypeID*/GILLT_s32,
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/1, /*Opcode*/TargetOpcode::G_CONSTANT,
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/1, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_AddCImm, /*InsnID*/1, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT: // Combiner Rule #0: Test0 @ [a[1], b[1], c[1]]
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/0, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 8: @745
+// CHECK-NEXT: GIM_Reject,
+// CHECK-NEXT: // Label 0: @746
+// CHECK-NEXT: GIM_Reject,
+// CHECK-NEXT: };
+// CHECK-NEXT: return MatchTable0;
+// CHECK-NEXT: }
diff --git a/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-variadics.td b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-variadics.td
new file mode 100644
index 00000000000000..534e15c8b6fcbe
--- /dev/null
+++ b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table-variadics.td
@@ -0,0 +1,103 @@
+// RUN: llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
+// RUN: -combiners=MyCombiner %s | \
+// RUN: FileCheck %s
+
+include "llvm/Target/Target.td"
+include "llvm/Target/GlobalISel/Combine.td"
+
+def MyTargetISA : InstrInfo;
+def MyTarget : Target { let InstructionSet = MyTargetISA; }
+
+def InstTest0 : GICombineRule<
+ (defs root:$a),
+ (match (G_BUILD_VECTOR $a, $b, $c, $d)),
+ (apply [{ APPLY }])>;
+
+def InstTest1 : GICombineRule<
+ (defs root:$a),
+ (match (G_BUILD_VECTOR $a, $b)),
+ (apply [{ APPLY }])>;
+
+def InstTest2 : GICombineRule<
+ (defs root:$a),
+ (match (G_UNMERGE_VALUES $a, $b)),
+ (apply [{ APPLY }])>;
+
+def InstTest3 : GICombineRule<
+ (defs root:$a),
+ (match (G_UNMERGE_VALUES $a, $b, $c, $d)),
+ (apply [{ APPLY }])>;
+
+def MyCombiner: GICombinerHelper<"GenMyCombiner", [
+ InstTest0,
+ InstTest1,
+ InstTest2,
+ InstTest3
+]>;
+
+// CHECK: const int64_t *GenMyCombiner::getMatchTable() const {
+// CHECK-NEXT: constexpr static int64_t MatchTable0[] = {
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 0*/ 26,
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_BUILD_VECTOR,
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 1*/ 15, // Rule ID 1 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule1Enabled,
+// CHECK-NEXT: GIM_CheckNumOperands, /*MI*/0, /*Expected*/2,
+// CHECK-NEXT: // MIs[0] a
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] b
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // Combiner Rule #1: InstTest1
+// CHECK-NEXT: GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 1: @15
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 2*/ 25, // Rule ID 0 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
+// CHECK-NEXT: GIM_CheckNumOperands, /*MI*/0, /*Expected*/4,
+// CHECK-NEXT: // MIs[0] a
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] b
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] c
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] d
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // Combiner Rule #0: InstTest0
+// CHECK-NEXT: GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 2: @25
+// CHECK-NEXT: GIM_Reject,
+// CHECK-NEXT: // Label 0: @26
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 3*/ 52,
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_UNMERGE_VALUES,
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 4*/ 41, // Rule ID 2 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule2Enabled,
+// CHECK-NEXT: GIM_CheckNumOperands, /*MI*/0, /*Expected*/2,
+// CHECK-NEXT: // MIs[0] a
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] b
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // Combiner Rule #2: InstTest2
+// CHECK-NEXT: GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 4: @41
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 5*/ 51, // Rule ID 3 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule3Enabled,
+// CHECK-NEXT: GIM_CheckNumOperands, /*MI*/0, /*Expected*/4,
+// CHECK-NEXT: // MIs[0] a
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] b
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] c
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] d
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // Combiner Rule #3: InstTest3
+// CHECK-NEXT: GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 5: @51
+// CHECK-NEXT: GIM_Reject,
+// CHECK-NEXT: // Label 3: @52
+// CHECK-NEXT: GIM_Reject,
+// CHECK-NEXT: };
+// CHECK-NEXT: return MatchTable0;
+// CHECK-NEXT: }
diff --git a/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table.td b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table.td
index 3b29a4e8bbc67b..4c5b572e277849 100644
--- a/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table.td
+++ b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/match-table.td
@@ -10,52 +10,68 @@ def MyTarget : Target { let InstructionSet = MyTargetISA; }
def dummy;
-def R0 : Register<"r0"> { let Namespace = "MyTarget"; }
-def GPR32 : RegisterClass<"MyTarget", [i32], 32, (add R0)>;
-class I<dag OOps, dag IOps, list<dag> Pat>
- : Instruction {
- let Namespace = "MyTarget";
- let OutOperandList = OOps;
- let InOperandList = IOps;
- let Pattern = Pat;
-}
-def MOV : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def TRUNC : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def ZEXT : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def SEXT : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-
def HasAnswerToEverything : Predicate<"Subtarget->getAnswerToUniverse() == 42 && Subtarget->getAnswerToLife() == 42">;
def WipOpcodeTest0 : GICombineRule<
(defs root:$d),
- (match (wip_match_opcode TRUNC):$d),
+ (match (wip_match_opcode G_TRUNC):$d),
(apply [{ APPLY }])>;
def WipOpcodeTest1 : GICombineRule<
(defs root:$d),
- (match (wip_match_opcode TRUNC, SEXT):$d),
+ (match (wip_match_opcode G_TRUNC, G_SEXT):$d),
(apply [{ APPLY }])>;
// Note: also checks that spaces in the type name are removed.
def reg_matchinfo : GIDefMatchData<"Register ">;
def InstTest0 : GICombineRule<
(defs root:$d, reg_matchinfo:$r0, reg_matchinfo:$r1),
- (match (MOV $a, $b):$d),
+ (match (COPY $a, $b):$d),
(apply [{ APPLY ${r0}, ${r1} }])>;
let Predicates = [HasAnswerToEverything] in
def InstTest1 : GICombineRule<
(defs root:$d, reg_matchinfo:$r0),
- (match (MOV $a, $b):$d,
- (ZEXT $b, $c),
+ (match (COPY $a, $b):$d,
+ (G_ZEXT $b, $c),
[{ return CHECK ${a}, ${b}, ${c}, ${d} }]),
(apply [{ APPLY }])>;
+def InOutInstTest0 : GICombineRule<
+ (defs root:$root),
+ (match (G_ZEXT $tmp, $ext),
+ (G_STORE $tmp, $ptr):$root),
+ (apply (G_STORE $ext, $ptr):$root, "APPLY ${ext} ${ptr} ${root}")>;
+
+// Imm operand of G_CONSTANT should match a literal int, while the second
+// should match a constant.
+def InOutInstTest1 : GICombineRule<
+ (defs root:$dst),
+ (match
+ (G_CONSTANT $x, -42:$z),
+ (G_AND $dst, $x, (i32 43))),
+ (apply (G_TRUNC $dst, $z))>;
+
+def MatchICst: GICombinePatFrag<
+ (outs),
+ (ins gi_mo:$foo, gi_imm:$cst),
+ [(pattern "return matchIConstant(${foo}, ${cst});")]>;
+
+def PatFragTest0 : GICombineRule<
+ (defs root:$dst),
+ (match (G_ZEXT $dst, $cst), (MatchICst $cst, (i32 0))),
+ (apply (COPY $dst, (i32 0)))>;
+
+// TODO: add test with temp reg use
+
def MyCombiner: GICombinerHelper<"GenMyCombiner", [
WipOpcodeTest0,
WipOpcodeTest1,
InstTest0,
- InstTest1
+ InstTest1,
+ InOutInstTest0,
+ InOutInstTest1,
+ PatFragTest0
]>;
// We have at most 2 registers used by one rule at a time, so we should only have 2 registers MDInfos.
@@ -69,6 +85,9 @@ def MyCombiner: GICombinerHelper<"GenMyCombiner", [
// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner0: {
// CHECK-NEXT: return CHECK State.MIs[0]->getOperand(0), State.MIs[0]->getOperand(1), State.MIs[1]->getOperand(1), State.MIs[0]
// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXPred_MI_Predicate_GICombiner1: {
+// CHECK-NEXT: return matchIConstant(State.MIs[0]->getOperand(1), 0);
+// CHECK-NEXT: }
// Verify we reset MatchData on each tryCombineAll
// CHECK: bool GenMyCombiner::tryCombineAll(MachineInstr &I) const {
@@ -79,28 +98,51 @@ def MyCombiner: GICombinerHelper<"GenMyCombiner", [
// CHECK-NEXT: State.MIs.push_back(&I);
// CHECK-NEXT: MatchInfos = MatchInfosTy();
// CHECK-EMPTY:
-// CHECK-NEXT: if (executeMatchTable(*this, OutMIs, State, ExecInfo, getMatchTable(), *ST.getInstrInfo(), MRI, *MRI.getTargetRegisterInfo(), *ST.getRegBankInfo(), AvailableFeatures, /*CoverageInfo*/ nullptr))
+// CHECK-NEXT: if (executeMatchTable(*this, OutMIs, State, ExecInfo, getMatchTable(), *ST.getInstrInfo(), MRI, *MRI.getTargetRegisterInfo(), *ST.getRegBankInfo(), AvailableFeatures, /*CoverageInfo*/ nullptr, &Observer))
// CHECK-NEXT: return true;
// CHECK-NEXT: }
// CHECK-EMPTY:
// CHECK-NEXT: return false;
// CHECK-NEXT: }
+// Check apply
+// CHECK: enum {
+// CHECK-NEXT: GICXXCustomAction_CombineApplyGICombiner0 = GICXXCustomAction_Invalid + 1,
+// CHECK-NEXT: GICXXCustomAction_CombineApplyGICombiner1,
+// CHECK-NEXT: GICXXCustomAction_CombineApplyGICombiner2,
+// CHECK-NEXT: };
+// CHECK-NEXT: void GenMyCombiner::runCustomAction(unsigned ApplyID, const MatcherState &State, NewMIVector &OutMIs) const {
+// CHECK-NEXT: switch(ApplyID) {
+// CHECK-NEXT: case GICXXCustomAction_CombineApplyGICombiner0:{
+// CHECK-NEXT: APPLY
+// CHECK-NEXT: return;
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXCustomAction_CombineApplyGICombiner1:{
+// CHECK-NEXT: APPLY MatchInfos.MDInfo0, MatchInfos.MDInfo1
+// CHECK-NEXT: return;
+// CHECK-NEXT: }
+// CHECK-NEXT: case GICXXCustomAction_CombineApplyGICombiner2:{
+// CHECK-NEXT: APPLY State.MIs[1]->getOperand(1) State.MIs[0]->getOperand(1) OutMIs[0]
+// CHECK-NEXT: return;
+// CHECK-NEXT: }
+// CHECK-NEXT: }
+// CHECK-NEXT: llvm_unreachable("Unknown Apply Action");
+// CHECK-NEXT: }
// Verify match table.
// CHECK: const int64_t *GenMyCombiner::getMatchTable() const {
// CHECK-NEXT: constexpr static int64_t MatchTable0[] = {
// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 0*/ 20,
-// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, MyTarget::TRUNC,
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_TRUNC,
// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 1*/ 12, // Rule ID 0 //
// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule0Enabled,
-// CHECK-NEXT: // Combiner Rule #0: WipOpcodeTest0; wip_match_opcode alternative 'TRUNC'
+// CHECK-NEXT: // Combiner Rule #0: WipOpcodeTest0; wip_match_opcode 'G_TRUNC'
// CHECK-NEXT: GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
// CHECK-NEXT: GIR_Done,
// CHECK-NEXT: // Label 1: @12
// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 2*/ 19, // Rule ID 1 //
// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule1Enabled,
-// CHECK-NEXT: // Combiner Rule #1: WipOpcodeTest1; wip_match_opcode alternative 'TRUNC'
+// CHECK-NEXT: // Combiner Rule #1: WipOpcodeTest1; wip_match_opcode 'G_TRUNC'
// CHECK-NEXT: GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
// CHECK-NEXT: GIR_Done,
// CHECK-NEXT: // Label 2: @19
@@ -108,13 +150,13 @@ def MyCombiner: GICombinerHelper<"GenMyCombiner", [
// CHECK-NEXT: // Label 0: @20
// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 3*/ 30, // Rule ID 2 //
// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule1Enabled,
-// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, MyTarget::SEXT,
-// CHECK-NEXT: // Combiner Rule #1: WipOpcodeTest1; wip_match_opcode alternative 'SEXT'
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_SEXT,
+// CHECK-NEXT: // Combiner Rule #1: WipOpcodeTest1; wip_match_opcode 'G_SEXT'
// CHECK-NEXT: GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner0,
// CHECK-NEXT: GIR_Done,
// CHECK-NEXT: // Label 3: @30
// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 4*/ 64,
-// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, MyTarget::MOV,
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::COPY,
// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 5*/ 42, // Rule ID 3 //
// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule2Enabled,
// CHECK-NEXT: // MIs[0] a
@@ -131,8 +173,8 @@ def MyCombiner: GICombinerHelper<"GenMyCombiner", [
// CHECK-NEXT: // MIs[0] a
// CHECK-NEXT: // No operand predicates
// CHECK-NEXT: // MIs[0] b
-// CHECK-NEXT: GIM_RecordInsn, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
-// CHECK-NEXT: GIM_CheckOpcode, /*MI*/1, MyTarget::ZEXT,
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_ZEXT,
// CHECK-NEXT: // MIs[1] c
// CHECK-NEXT: // No operand predicates
// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner0,
@@ -143,6 +185,65 @@ def MyCombiner: GICombinerHelper<"GenMyCombiner", [
// CHECK-NEXT: // Label 6: @63
// CHECK-NEXT: GIM_Reject,
// CHECK-NEXT: // Label 4: @64
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 7*/ 101, // Rule ID 5 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule4Enabled,
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_STORE,
+// CHECK-NEXT: // MIs[0] tmp
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/0, // MIs[1]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_ZEXT,
+// CHECK-NEXT: // MIs[1] ext
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] ptr
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT: // Combiner Rule #4: InOutInstTest0
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::G_STORE,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/1, /*OpIdx*/1, // ext
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/1, // ptr
+// CHECK-NEXT: GIR_MergeMemOperands, /*InsnID*/0, /*MergeInsnID's*/0, 1, GIU_MergeMemOperands_EndOfList,
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_CustomAction, GICXXCustomAction_CombineApplyGICombiner2,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 7: @101
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 8*/ 143, // Rule ID 6 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule5Enabled,
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_AND,
+// CHECK-NEXT: GIM_CheckType, /*MI*/0, /*Op*/2, /*Type*/GILLT_s32,
+// CHECK-NEXT: // MIs[0] dst
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] x
+// CHECK-NEXT: GIM_RecordInsnIgnoreCopies, /*DefineMI*/1, /*MI*/0, /*OpIdx*/1, // MIs[1]
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/1, TargetOpcode::G_CONSTANT,
+// CHECK-NEXT: // MIs[1] z
+// CHECK-NEXT: GIM_CheckLiteralInt, /*MI*/1, /*Op*/1, -42,
+// CHECK-NEXT: GIM_CheckConstantInt, /*MI*/0, /*Op*/2, 43,
+// CHECK-NEXT: GIM_CheckIsSafeToFold, /*InsnID*/1,
+// CHECK-NEXT: // Combiner Rule #5: InOutInstTest1
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::G_TRUNC,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/1, /*OpIdx*/1, // z
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 8: @143
+// CHECK-NEXT: GIM_Try, /*On fail goto*//*Label 9*/ 181, // Rule ID 7 //
+// CHECK-NEXT: GIM_CheckSimplePredicate, GICXXPred_Simple_IsRule6Enabled,
+// CHECK-NEXT: GIM_CheckOpcode, /*MI*/0, TargetOpcode::G_ZEXT,
+// CHECK-NEXT: // MIs[0] dst
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: // MIs[0] cst
+// CHECK-NEXT: // No operand predicates
+// CHECK-NEXT: GIM_CheckCxxInsnPredicate, /*MI*/0, /*FnId*/GICXXPred_MI_Predicate_GICombiner1,
+// CHECK-NEXT: GIR_MakeTempReg, /*TempRegID*/0, /*TypeID*/GILLT_s32,
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/1, /*Opcode*/TargetOpcode::G_CONSTANT,
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/1, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_AddCImm, /*InsnID*/1, /*Type*/GILLT_s32, /*Imm*/0,
+// CHECK-NEXT: // Combiner Rule #6: PatFragTest0 @ [__PatFragTest0_match_1[0]]
+// CHECK-NEXT: GIR_BuildMI, /*InsnID*/0, /*Opcode*/TargetOpcode::COPY,
+// CHECK-NEXT: GIR_Copy, /*NewInsnID*/0, /*OldInsnID*/0, /*OpIdx*/0, // dst
+// CHECK-NEXT: GIR_AddTempRegister, /*InsnID*/0, /*TempRegID*/0, /*TempRegFlags*/0,
+// CHECK-NEXT: GIR_EraseFromParent, /*InsnID*/0,
+// CHECK-NEXT: GIR_Done,
+// CHECK-NEXT: // Label 9: @181
// CHECK-NEXT: GIM_Reject,
// CHECK-NEXT: };
// CHECK-NEXT: return MatchTable0;
diff --git a/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/operand-types.td b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/operand-types.td
new file mode 100644
index 00000000000000..af588a787eb055
--- /dev/null
+++ b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/operand-types.td
@@ -0,0 +1,85 @@
+// RUN: llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
+// RUN: -gicombiner-stop-after-parse -combiners=MyCombiner %s | \
+// RUN: FileCheck %s
+
+include "llvm/Target/Target.td"
+include "llvm/Target/GlobalISel/Combine.td"
+
+def MyTargetISA : InstrInfo;
+def MyTarget : Target { let InstructionSet = MyTargetISA; }
+
+// CHECK: (CombineRule name:InstTest0 id:0 root:x
+// CHECK-NEXT: (MatchPats
+// CHECK-NEXT: __InstTest0_match_0:(CodeGenInstructionPattern G_BUILD_VECTOR operands:[<def>i64:$z, i32:$y, i16:$w])
+// CHECK-NEXT: <match_root>__InstTest0_match_1:(CodeGenInstructionPattern G_BUILD_VECTOR operands:[<def>i64:$x, i32:$y, i64:$z])
+// CHECK-NEXT: )
+// CHECK-NEXT: (ApplyPats
+// CHECK-NEXT: <apply_root>__InstTest0_apply_0:(CodeGenInstructionPattern G_MUL operands:[<def>i64:$x, i32:$y, i16:$w])
+// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable MatchPats
+// CHECK-NEXT: w -> <live-in>
+// CHECK-NEXT: x -> __InstTest0_match_1
+// CHECK-NEXT: y -> <live-in>
+// CHECK-NEXT: z -> __InstTest0_match_0
+// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable ApplyPats
+// CHECK-NEXT: w -> <live-in>
+// CHECK-NEXT: x -> __InstTest0_apply_0
+// CHECK-NEXT: y -> <live-in>
+// CHECK-NEXT: )
+// CHECK-NEXT: )
+def InstTest0 : GICombineRule<
+ (defs root:$x),
+ (match
+ (G_BUILD_VECTOR i64:$z, $y, $w),
+ (G_BUILD_VECTOR $x, i32:$y, $z)),
+ (apply (G_MUL i64:$x, $y, i16:$w))>;
+
+// CHECK: (CombineRule name:PatFragTest0 id:1 root:dst
+// CHECK-NEXT: (PatFrags
+// CHECK-NEXT: (PatFrag name:FooPF
+// CHECK-NEXT: (outs [cst:root])
+// CHECK-NEXT: (alternatives [
+// CHECK-NEXT: [
+// CHECK-NEXT: (CodeGenInstructionPattern name:__FooPF_alt0_pattern_0 G_BUILD_VECTOR operands:[<def>i64:$cst, i8:$y, $x]),
+// CHECK-NEXT: (CodeGenInstructionPattern name:__FooPF_alt0_pattern_1 G_BUILD_VECTOR operands:[<def>$x, i8:$y, $w]),
+// CHECK-NEXT: ],
+// CHECK-NEXT: [
+// CHECK-NEXT: (CodeGenInstructionPattern name:__FooPF_alt1_pattern_0 G_BUILD_VECTOR operands:[<def>i32:$cst, i32:$y, i16:$x]),
+// CHECK-NEXT: (CodeGenInstructionPattern name:__FooPF_alt1_pattern_1 G_BUILD_VECTOR operands:[<def>i16:$x, i32:$y, $w]),
+// CHECK-NEXT: ],
+// CHECK-NEXT: ])
+// CHECK-NEXT: )
+// CHECK-NEXT: )
+// CHECK-NEXT: (MatchPats
+// CHECK-NEXT: <match_root>__PatFragTest0_match_0:(PatFragPattern FooPF operands:[<def>$dst])
+// CHECK-NEXT: )
+// CHECK-NEXT: (ApplyPats
+// CHECK-NEXT: <apply_root>__PatFragTest0_apply_0:(CodeGenInstructionPattern COPY operands:[<def>$dst, (i32 0)])
+// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable MatchPats
+// CHECK-NEXT: dst -> __PatFragTest0_match_0
+// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable ApplyPats
+// CHECK-NEXT: dst -> __PatFragTest0_apply_0
+// CHECK-NEXT: )
+// CHECK-NEXT: (PermutationsToEmit
+// CHECK-NEXT: [__PatFragTest0_match_0[0]],
+// CHECK-NEXT: [__PatFragTest0_match_0[1]],
+// CHECK-NEXT: )
+// CHECK-NEXT: )
+def FooPF: GICombinePatFrag<
+ (outs root:$cst),(ins),
+ [
+ (pattern (G_BUILD_VECTOR i64:$cst, $y, $x), (G_BUILD_VECTOR $x, i8:$y, $w)),
+ (pattern (G_BUILD_VECTOR i32:$cst, $y, i16:$x), (G_BUILD_VECTOR $x, i32:$y, $w)),
+ ]>;
+def PatFragTest0 : GICombineRule<
+ (defs root:$dst),
+ (match (FooPF $dst)),
+ (apply (COPY $dst, (i32 0)))>;
+
+def MyCombiner: GICombinerHelper<"GenMyCombiner", [
+ InstTest0,
+ PatFragTest0
+]>;
diff --git a/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/patfrag-errors.td b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/patfrag-errors.td
new file mode 100644
index 00000000000000..fba501cf220cb0
--- /dev/null
+++ b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/patfrag-errors.td
@@ -0,0 +1,282 @@
+// RUN: not llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
+// RUN: -combiners=MyCombiner %s 2>&1| \
+// RUN: FileCheck %s -implicit-check-not=error:
+
+include "llvm/Target/Target.td"
+include "llvm/Target/GlobalISel/Combine.td"
+
+def MyTargetISA : InstrInfo;
+def MyTarget : Target { let InstructionSet = MyTargetISA; }
+
+def dummy;
+
+def MatchFooPerms: GICombinePatFrag<
+ (outs),
+ (ins gi_mo:$foo, gi_imm:$cst),
+ [
+ (pattern "return foo(${foo}, ${cst})"),
+ (pattern "return bar(${foo}, ${cst})"),
+ (pattern "return bux(${foo}, ${cst})"),
+ ]>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: cannot emit rule 'too_many_perms'; 27 permutations would be emitted, but the max is 16
+let MaxPermutations = 16 in
+def too_many_perms : GICombineRule<
+ (defs root:$dst),
+ (match (G_ZEXT $dst, $cst),
+ (MatchFooPerms $cst, (i32 0)):$a,
+ (MatchFooPerms $cst, (i32 0)):$b,
+ (MatchFooPerms $cst, (i32 0)):$c
+ ),
+ (apply (COPY $dst, (i32 0)), "APPLY ${src}")>;
+
+def DummyCXXPF: GICombinePatFrag<
+ (outs),
+ (ins gi_mo:$in),
+ [
+ (pattern "return foo()"),
+ ]>;
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: operand 'foo' (for parameter 'in' of 'DummyCXXPF') cannot be unbound
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: note: one or more alternatives of 'DummyCXXPF' do not bind 'in' to an instruction operand; either use a bound operand or ensure 'DummyCXXPF' binds 'in' in all alternatives
+def undef_livein : GICombineRule<
+ (defs root:$dst),
+ (match (G_ZEXT $dst, $bar),
+ (DummyCXXPF $foo)
+ ),
+ (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: GICombinePatFrag must have one root in its 'out' operands
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'OutMustBeRoot'
+def OutMustBeRoot: GICombinePatFrag<
+ (outs $foo, $bar),
+ (ins),
+ [
+ (pattern (G_ZEXT $foo, $bar), (G_FPEXT $bar, $y)),
+ ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(OutMustBeRoot ?:$bar, ?:$foo)'
+def out_must_be_root : GICombineRule<
+ (defs root:$dst),
+ (match (G_ZEXT $dst, $bar),
+ (OutMustBeRoot $bar, $foo)
+ ),
+ (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: output parameter 'bar' must be 'root' or 'gi_mo'
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'BadOutType'
+def BadOutType: GICombinePatFrag<
+ (outs root:$foo, gi_imm:$bar),
+ (ins),
+ [
+ (pattern (G_ZEXT $foo, $bar), (G_FPEXT $bar, $y)),
+ ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(BadOutType ?:$bar, ?:$foo)'
+def bad_out_type : GICombineRule<
+ (defs root:$dst),
+ (match (G_ZEXT $dst, $bar),
+ (BadOutType $bar, $foo)
+ ),
+ (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: pattern 'dbg' ('G_FPEXT') is unreachable from the pattern root!
+def UnreachablePat: GICombinePatFrag<
+ (outs root:$foo, $bar),
+ (ins),
+ [
+ (pattern (G_ZEXT $foo, $x), (G_FPEXT $bar, $y):$dbg),
+ ]>;
+def unreachable_pat : GICombineRule<
+ (defs root:$dst),
+ (match (G_ZEXT $dst, $bar),
+ (UnreachablePat $bar, $foo)
+ ),
+ (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: wip_match_opcode cannot be used in GICombinePatFrag
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'WipMatchOpcodePatFrag'
+def WipMatchOpcodePatFrag: GICombinePatFrag<
+ (outs),
+ (ins),
+ [
+ (pattern (wip_match_opcode G_ZEXT)),
+ ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(WipMatchOpcodePatFrag)'
+def wip_match_opcode_patfrag : GICombineRule<
+ (defs root:$dst),
+ (match (WipMatchOpcodePatFrag)),
+ (apply (COPY $dst, (i32 0)))>;
+
+def DummyPF: GICombinePatFrag<(outs),(ins),[]>;
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: nested GICombinePatFrag are not supported
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'NestingPatFrag'
+def NestingPatFrag: GICombinePatFrag<
+ (outs),
+ (ins),
+ [
+ (pattern (DummyPF)),
+ ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(NestingPatFrag)'
+def nest_pat_frag : GICombineRule<
+ (defs root:$dst),
+ (match (NestingPatFrag)),
+ (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: input parameter 'k' cannot be redefined!
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'DupParamIn'
+def DupParamIn: GICombinePatFrag<
+ (outs),
+ (ins gi_mo:$k, gi_mo:$k),
+ [
+ (pattern (G_ZEXT $k, $x)),
+ ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(DupParamIn ?:$dst, ?:$bar)'
+def dup_params_in : GICombineRule<
+ (defs root:$dst),
+ (match (DupParamIn $dst, $bar)),
+ (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: duplicate parameter 'k'
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'DupParamOut'
+def DupParamOut: GICombinePatFrag<
+ (outs root:$k, root:$k),
+ (ins),
+ [
+ (pattern (G_ZEXT $k, $x)),
+ ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(DupParamOut ?:$dst, ?:$bar)'
+def dup_params_out : GICombineRule<
+ (defs root:$dst),
+ (match (DupParamOut $dst, $bar)),
+ (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: input parameter 'k' cannot be redefined!
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'DupParamInOut'
+def DupParamInOut: GICombinePatFrag<
+ (outs root:$k),
+ (ins gi_mo:$k),
+ [
+ (pattern (G_ZEXT $k, $x)),
+ ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(DupParamInOut ?:$dst, ?:$bar)'
+def dup_params_inout : GICombineRule<
+ (defs root:$dst),
+ (match (DupParamInOut $dst, $bar)),
+ (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: output parameter 'k' must be defined by all alternative patterns in 'DefByAllAlts'
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'DefByAllAlts'
+def DefByAllAlts: GICombinePatFrag<
+ (outs root:$k),
+ (ins),
+ [
+ (pattern (G_ZEXT $k, $x)),
+ (pattern (G_FPEXT $z, $k))
+ ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(DefByAllAlts ?:$dst)'
+def def_by_all_alts : GICombineRule<
+ (defs root:$dst),
+ (match (DefByAllAlts $dst)),
+ (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: Operand 'x' is defined multiple times in patterns of alternative #1
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Could not parse GICombinePatFrag 'MultiDefInPat'
+def MultiDefInPat: GICombinePatFrag<
+ (outs root:$k),
+ (ins),
+ [
+ (pattern (G_ZEXT $k, $a)),
+ (pattern (G_ZEXT $k, $x), (G_ZEXT $x, $foo), (G_FPEXT $x, $foo)),
+ ]>;
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(MultiDefInPat ?:$dst)'
+def multi_def_in_pat : GICombineRule<
+ (defs root:$dst),
+ (match (MultiDefInPat $dst)),
+ (apply (COPY $dst, (i32 0)))>;
+
+def ExpectedImm: GICombinePatFrag<
+ (outs root:$k),
+ (ins gi_imm:$i),
+ [
+ (pattern (G_ZEXT $k, $i)),
+ ]>;
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: expected operand 1 of 'ExpectedImm' to be an immediate; got MachineOperand $z
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(ExpectedImm ?:$dst, ?:$z)'
+def expected_imm : GICombineRule<
+ (defs root:$dst),
+ (match (ExpectedImm $dst, $z)),
+ (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: operand 1 of 'ExpectedImm' cannot be a named immediate
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(ExpectedImm ?:$dst, (i32 0):$z)'
+def expected_imm_namedimm : GICombineRule<
+ (defs root:$dst),
+ (match (ExpectedImm $dst, (i32 0):$z)),
+ (apply (COPY $dst, (i32 0)))>;
+
+def ExpectedMO: GICombinePatFrag<
+ (outs root:$k),
+ (ins gi_mo:$i),
+ [
+ (pattern (G_ZEXT $k, $i)),
+ ]>;
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: expected operand 1 of 'ExpectedMO' to be a MachineOperand; got imm 0
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(ExpectedMO ?:$dst, (i32 0))'
+def expected_mo : GICombineRule<
+ (defs root:$dst),
+ (match (ExpectedMO $dst, (i32 0))),
+ (apply (COPY $dst, (i32 0)))>;
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: expected operand 1 of 'ExpectedMO' to be a MachineOperand; got imm 0:$z
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(ExpectedMO ?:$dst, (i32 0):$z)'
+def expected_mo_namedimm : GICombineRule<
+ (defs root:$dst),
+ (match (ExpectedMO $dst, (i32 0):$z)),
+ (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: 'dum': using GICombinePatFrag is not supported in apply patterns
+def patfrag_in_apply : GICombineRule<
+ (defs root:$dst),
+ (match (COPY $dst, (i32 0))),
+ (apply (DummyPF):$dum)>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: cannot use 'DummyPF as match root
+def patfrag_cannot_be_root : GICombineRule<
+ (defs root:$root),
+ (match (DummyPF):$root),
+ (apply (COPY $dst, (i32 0)):$root)>;
+
+def TypedParams: GICombinePatFrag<
+ (outs root:$k),
+ (ins gi_mo:$i),
+ [
+ (pattern (G_ZEXT $k, i32:$i)),
+ ]>;
+// CHECK: :[[@LINE+3]]:{{[0-9]+}}: warning: impossible type constraints: operand 1 of 'broken' has type 'i64', but 'TypedParams' constrains it to 'i32'
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: note: operand 1 of 'broken' is 'k'
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: note: argument 1 of 'TypedParams' is 'i'
+def inconsistent_arg_type : GICombineRule<
+ (defs root:$dst),
+ (match (TypedParams $dst, i64:$k):$broken),
+ (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: error: Failed to parse one or more rules
+
+def MyCombiner: GICombinerHelper<"GenMyCombiner", [
+ too_many_perms,
+ undef_livein,
+ out_must_be_root,
+ bad_out_type,
+ unreachable_pat,
+ wip_match_opcode_patfrag,
+ nest_pat_frag,
+ dup_params_in,
+ dup_params_out,
+ dup_params_inout,
+ def_by_all_alts,
+ multi_def_in_pat,
+ expected_imm,
+ expected_imm_namedimm,
+ expected_mo,
+ expected_mo_namedimm,
+ patfrag_in_apply,
+ patfrag_cannot_be_root,
+ inconsistent_arg_type
+]>;
diff --git a/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/pattern-errors.td b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/pattern-errors.td
new file mode 100644
index 00000000000000..0a484a5b1c1ecf
--- /dev/null
+++ b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/pattern-errors.td
@@ -0,0 +1,256 @@
+// RUN: not llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
+// RUN: -combiners=MyCombiner %s 2>&1| \
+// RUN: FileCheck %s -implicit-check-not=error:
+
+include "llvm/Target/Target.td"
+include "llvm/Target/GlobalISel/Combine.td"
+
+def MyTargetISA : InstrInfo;
+def MyTarget : Target { let InstructionSet = MyTargetISA; }
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Cannot find root 'missing' in match patterns!
+def root_not_found : GICombineRule<
+ (defs root:$missing),
+ (match (COPY $a, $b):$d),
+ (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: apply pattern 'd' is supposed to be a root but it does not redefine any of the defs of the match root
+def misleading_root : GICombineRule<
+ (defs root:$d),
+ (match (COPY $a, $b):$d),
+ (apply (COPY $tmp, $b):$d,
+ (COPY $a, $tmp))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: C++ code cannot be the root of a rule!
+def cxx_root : GICombineRule<
+ (defs root:$a),
+ (match "return MATCH":$a),
+ (apply [{ APPLY }]:$z)>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Cannot use live-in operand 'b' as match pattern root!
+def livein_root : GICombineRule<
+ (defs root:$b),
+ (match (COPY $a, $b)),
+ (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: 'COPY' expected 2 operands, got 1
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(COPY ?:$a)'
+def not_enough_operands : GICombineRule<
+ (defs root:$d),
+ (match (COPY $a):$d),
+ (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: 'COPY' expected 2 operands, got 3
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(COPY ?:$a, ?:$b, ?:$c)'
+def too_many_operands : GICombineRule<
+ (defs root:$d),
+ (match (COPY $a, $b, $c):$d),
+ (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Operand 'd' is defined multiple times in the 'match' patterns
+def multi_defs : GICombineRule<
+ (defs root:$d),
+ (match (COPY $d, $b), (COPY $d, $x)),
+ (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Operand 'd' is defined multiple times in the 'match' patterns
+def multi_defs_2 : GICombineRule<
+ (defs root:$d),
+ (match (G_UNMERGE_VALUES $d, $d, $b)),
+ (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: pattern 'foo' ('COPY') is unreachable from the pattern root!
+def unreachable_pat : GICombineRule<
+ (defs root:$d),
+ (match (COPY $a, $b):$d, (COPY $z, $k):$foo),
+ (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: 'applytest': wip_match_opcode is not supported in apply patterns
+def wip_match_opcode_in_apply : GICombineRule<
+ (defs root:$d),
+ (match (COPY $a, $b):$d, (wip_match_opcode G_ZEXT)),
+ (apply (wip_match_opcode G_ZEXT):$applytest)>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: wip_match_opcode can not be used with instruction patterns!
+def wip_match_opcode_with_inst_pat : GICombineRule<
+ (defs root:$d),
+ (match (COPY $a, $b):$d, (wip_match_opcode G_ZEXT)),
+ (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: wip_opcode_match can only be present once
+def multiple_wip_match_opcode : GICombineRule<
+ (defs root:$d),
+ (match (wip_match_opcode COPY):$d, (wip_match_opcode G_ZEXT)),
+ (apply [{ APPLY }])>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Operand 'a' is defined multiple times in the 'apply' patterns
+def multiple_def_in_apply : GICombineRule<
+ (defs root:$d),
+ (match (COPY $a, $b):$d),
+ (apply (COPY $a, $b), (COPY $a, $b))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: 'd' match pattern defined more than once!
+def redef_match : GICombineRule<
+ (defs root:$d),
+ (match (COPY $a, $b):$d, (COPY $b, $z):$d),
+ (apply (COPY $a, $b))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: 'o' apply pattern defined more than once!
+def redef_apply: GICombineRule<
+ (defs root:$d),
+ (match (COPY $a, $b):$d),
+ (apply (COPY $a, $x):$o, (COPY $x, $b):$o)>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: redefining an instruction other than the root is not supported (operand 'b'
+def redef_nonroot : GICombineRule<
+ (defs root:$a),
+ (match (COPY $a, $b), (COPY $b, $z)),
+ (apply (COPY $a, $b), (G_ZEXT $b, (i32 0)))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Cannot define live-in operand 'b' in the 'apply' pattern
+def cannot_def_match_livein : GICombineRule<
+ (defs root:$d),
+ (match (COPY $a, $b):$d),
+ (apply (COPY $a, $b), (COPY $b, $b))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: invalid output operand 'x': operand is not a live-in of the match pattern, and it has no definition
+def undef_in_apply : GICombineRule<
+ (defs root:$d),
+ (match (COPY $a, $b):$d),
+ (apply (COPY $a, $x))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: def of pattern root 'a' is not redefined in the apply pattern!
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: note: match pattern root is 'foo'
+def no_redef_in_apply : GICombineRule<
+ (defs root:$a),
+ (match (COPY $a, $b):$foo),
+ (apply (COPY $x, $b))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: def of pattern root 'b' is not redefined in the apply pattern!
+def no_redef_in_apply_multidefroot : GICombineRule<
+ (defs root:$a),
+ (match (G_UNMERGE_VALUES $a, $b, $c)),
+ (apply (COPY $a, $c))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: cannot use wip_match_opcode in combination with apply instruction patterns
+def instpat_with_wipmatch : GICombineRule<
+ (defs root:$a),
+ (match (wip_match_opcode COPY):$d),
+ (apply (COPY $x, $b):$d)>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: cannot parse operand '(i32)'
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(COPY ?:$x, (i32))
+def bad_imm_noarg : GICombineRule<
+ (defs root:$a),
+ (match (COPY $x, (i32)):$d),
+ (apply (COPY $x, $b):$d)>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: cannot parse operand '(i32 0, 0)'
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(COPY ?:$x, (i32 0, 0))
+def bad_imm_too_many_args : GICombineRule<
+ (defs root:$a),
+ (match (COPY $x, (i32 0, 0)):$d),
+ (apply (COPY $x, $b):$d)>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: cannot parse immediate '(COPY 0)', 'COPY' is not a ValueType
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(COPY ?:$x, (COPY 0))
+def bad_imm_not_a_valuetype : GICombineRule<
+ (defs root:$a),
+ (match (COPY $x, (COPY 0)):$d),
+ (apply (COPY $x, $b):$d)>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: invalid output operand 'imm': output immediates cannot be named
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: note: while emitting pattern 'd' (COPY)
+def output_imm_cannot_be_named : GICombineRule<
+ (defs root:$x),
+ (match (COPY $x, (i32 0)):$d),
+ (apply (COPY $x, (i32 0):$imm):$d)>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: 'G_CONSTANT' immediate must be typed!
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: note: while emitting pattern 'd' (G_CONSTANT)
+def output_imm_must_be_typed : GICombineRule<
+ (defs root:$x),
+ (match (COPY $x, (i32 0)):$d),
+ (apply (G_CONSTANT $x, 0):$d)>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: 'G_BUILD_VECTOR' expected at least 2 operands, got 1
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(G_BUILD_VECTOR ?:$x)'
+def too_few_ops_for_variadic : GICombineRule<
+ (defs root:$x),
+ (match (G_BUILD_VECTOR $x)),
+ (apply (COPY $x, 0))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: expected an operand name after 'i32'
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Failed to parse pattern: '(G_FNEG ?:$x, i32)'
+def expected_op_name : GICombineRule<
+ (defs root:$x),
+ (match (G_FNEG $x, i32)),
+ (apply (COPY $x, (i32 0)))>;
+
+// CHECK: :[[@LINE+3]]:{{[0-9]+}}: error: invalid operand type: 'not_a_type' is not a ValueType
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: Failed to parse pattern: '(G_FNEG ?:$x, not_a_type:$y)'
+def not_a_type;
+def bad_mo_type_not_a_valuetype : GICombineRule<
+ (defs root:$x),
+ (match (G_FNEG $x, not_a_type:$y)),
+ (apply (COPY $x, (i32 0)))>;
+
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: def of a new register 'newreg' in the apply patterns must have a type
+def untyped_new_reg_in_apply : GICombineRule<
+ (defs root:$x),
+ (match (G_FNEG $x, $y)),
+ (apply (COPY $newreg, (i32 0)),
+ (COPY $x, $newreg))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: 'y' is a named immediate, it cannot be defined by another instruction
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: note: 'y' is defined by 'foo'
+def def_named_imm_match : GICombineRule<
+ (defs root:$x),
+ (match (G_SEXT $y, $z):$foo,
+ (G_FNEG $x, (i32 0):$y)),
+ (apply (COPY $x, (i32 0)))>;
+
+// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: invalid output operand 'tmp': output immediates cannot be named
+// CHECK: :[[@LINE+1]]:{{[0-9]+}}: note: while emitting pattern 'foo' (COPY)
+def def_named_imm_apply : GICombineRule<
+ (defs root:$x),
+ (match (G_FNEG $x, $y)),
+ (apply (COPY i32:$tmp, $y),
+ (COPY $x, (i32 0):$tmp):$foo)>;
+
+// CHECK: error: Failed to parse one or more rules
+
+def MyCombiner: GICombinerHelper<"GenMyCombiner", [
+ root_not_found,
+ misleading_root,
+ cxx_root,
+ livein_root,
+ not_enough_operands,
+ too_many_operands,
+ multi_defs,
+ multi_defs_2,
+ unreachable_pat,
+ wip_match_opcode_in_apply,
+ wip_match_opcode_with_inst_pat,
+ multiple_wip_match_opcode,
+ multiple_def_in_apply,
+ redef_match,
+ redef_apply,
+ redef_nonroot,
+ cannot_def_match_livein,
+ undef_in_apply,
+ no_redef_in_apply,
+ no_redef_in_apply_multidefroot,
+ instpat_with_wipmatch,
+ bad_imm_noarg,
+ bad_imm_too_many_args,
+ bad_imm_not_a_valuetype,
+ output_imm_cannot_be_named,
+ output_imm_must_be_typed,
+ too_few_ops_for_variadic,
+ expected_op_name,
+ bad_mo_type_not_a_valuetype,
+ untyped_new_reg_in_apply,
+ def_named_imm_match,
+ def_named_imm_apply
+]>;
diff --git a/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/pattern-parsing-errors.td b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/pattern-parsing-errors.td
deleted file mode 100644
index a693c162c3e6f8..00000000000000
--- a/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/pattern-parsing-errors.td
+++ /dev/null
@@ -1,92 +0,0 @@
-// RUN: not llvm-tblgen -I %p/../../../include -gen-global-isel-combiner-matchtable \
-// RUN: -combiners=MyCombiner %s 2>&1| \
-// RUN: FileCheck %s -implicit-check-not=error:
-
-include "llvm/Target/Target.td"
-include "llvm/Target/GlobalISel/Combine.td"
-
-def MyTargetISA : InstrInfo;
-def MyTarget : Target { let InstructionSet = MyTargetISA; }
-
-def dummy;
-
-def R0 : Register<"r0"> { let Namespace = "MyTarget"; }
-def GPR32 : RegisterClass<"MyTarget", [i32], 32, (add R0)>;
-class I<dag OOps, dag IOps, list<dag> Pat>
- : Instruction {
- let Namespace = "MyTarget";
- let OutOperandList = OOps;
- let InOperandList = IOps;
- let Pattern = Pat;
-}
-def MOV : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def ADD : I<(outs GPR32:$dst), (ins GPR32:$src1, GPR32:$src2), []>;
-def SUB : I<(outs GPR32:$dst), (ins GPR32:$src1, GPR32:$src2), []>;
-def MUL : I<(outs GPR32:$dst), (ins GPR32:$src1, GPR32:$src2), []>;
-def TRUNC : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def SEXT : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def ZEXT : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def ICMP : I<(outs GPR32:$dst), (ins GPR32:$tst, GPR32:$src1, GPR32:$src2), []>;
-
-// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Cannot find root 'missing' in match patterns!
-def root_not_found : GICombineRule<
- (defs root:$missing),
- (match (MOV $a, $b):$d),
- (apply [{ APPLY }])>;
-
-// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Cannot use live-in operand 'b' as match pattern root!
-def livein_root : GICombineRule<
- (defs root:$b),
- (match (MOV $a, $b)),
- (apply [{ APPLY }])>;
-
-// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: 'MOV' expected 2 operands, got 1
-// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Expected a subclass of GIMatchKind or a sub-dag whose operator is either of a GIMatchKindWithArgs or Instruction
-def not_enough_operands : GICombineRule<
- (defs root:$d),
- (match (MOV $a):$d),
- (apply [{ APPLY }])>;
-
-// CHECK: :[[@LINE+2]]:{{[0-9]+}}: error: 'MOV' expected 2 operands, got 3
-// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Expected a subclass of GIMatchKind or a sub-dag whose operator is either of a GIMatchKindWithArgs or Instruction
-def too_many_operands : GICombineRule<
- (defs root:$d),
- (match (MOV $a, $b, $c):$d),
- (apply [{ APPLY }])>;
-
-// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Operand 'd' is defined multiple times in the 'match' patterns
-def multi_defs : GICombineRule<
- (defs root:$d),
- (match (MOV $d, $b), (MOV $d, $x)),
- (apply [{ APPLY }])>;
-
-// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: Instruction pattern '__anon_pat_match_5_0' is unreachable from the pattern root!
-def unreachable_pat : GICombineRule<
- (defs root:$d),
- (match (MOV $a, $b):$d, (MOV $z, $k)),
- (apply [{ APPLY }])>;
-
-// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: wip_match_opcode can not be used with instruction patterns!
-def wip_match_opcode_with_inst_pat : GICombineRule<
- (defs root:$d),
- (match (MOV $a, $b):$d, (wip_match_opcode SEXT)),
- (apply [{ APPLY }])>;
-
-// CHECK: :[[@LINE+1]]:{{[0-9]+}}: error: wip_opcode_match can only be present once
-def multiple_wip_match_opcode : GICombineRule<
- (defs root:$d),
- (match (wip_match_opcode MOV):$d, (wip_match_opcode SEXT)),
- (apply [{ APPLY }])>;
-
-// CHECK: error: Failed to parse one or more rules
-
-def MyCombiner: GICombinerHelper<"GenMyCombiner", [
- root_not_found,
- livein_root,
- not_enough_operands,
- too_many_operands,
- multi_defs,
- unreachable_pat,
- wip_match_opcode_with_inst_pat,
- multiple_wip_match_opcode
-]>;
diff --git a/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/pattern-parsing.td b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/pattern-parsing.td
index 15208e7f26d97e..b4a22675c6f17a 100644
--- a/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/pattern-parsing.td
+++ b/llvm/test/TableGen/GlobalISelCombinerMatchTableEmitter/pattern-parsing.td
@@ -10,69 +10,55 @@ def MyTarget : Target { let InstructionSet = MyTargetISA; }
def dummy;
-def R0 : Register<"r0"> { let Namespace = "MyTarget"; }
-def GPR32 : RegisterClass<"MyTarget", [i32], 32, (add R0)>;
-class I<dag OOps, dag IOps, list<dag> Pat>
- : Instruction {
- let Namespace = "MyTarget";
- let OutOperandList = OOps;
- let InOperandList = IOps;
- let Pattern = Pat;
-}
-def MOV : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def TRUNC : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def ZEXT : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-def SEXT : I<(outs GPR32:$dst), (ins GPR32:$src1), []>;
-
def HasAnswerToEverything : Predicate<"Subtarget->getAnswerToUniverse() == 42 && Subtarget->getAnswerToLife() == 42">;
def reg_matchinfo : GIDefMatchData<"Register">;
// CHECK: (CombineRule name:WipOpcodeTest0 id:0 root:d
-// CHECK-NEXT: (MatchDatas <empty>)
// CHECK-NEXT: (MatchPats
-// CHECK-NEXT: <root>d:(AnyOpcodePattern [TRUNC])
+// CHECK-NEXT: <match_root>d:(AnyOpcodePattern [G_TRUNC])
// CHECK-NEXT: )
// CHECK-NEXT: (ApplyPats
-// CHECK-NEXT: __anon_pat_apply_0_0:(CXXPattern apply code:"APPLY")
+// CHECK-NEXT: __WipOpcodeTest0_apply_0:(CXXPattern apply code:"APPLY")
// CHECK-NEXT: )
-// CHECK-NEXT: (OperandTable <empty>)
+// CHECK-NEXT: (OperandTable MatchPats <empty>)
+// CHECK-NEXT: (OperandTable ApplyPats <empty>)
// CHECK-NEXT: )
def WipOpcodeTest0 : GICombineRule<
(defs root:$d),
- (match (wip_match_opcode TRUNC):$d),
+ (match (wip_match_opcode G_TRUNC):$d),
(apply [{ APPLY }])>;
// CHECK: (CombineRule name:WipOpcodeTest1 id:1 root:d
-// CHECK-NEXT: (MatchDatas <empty>)
// CHECK-NEXT: (MatchPats
-// CHECK-NEXT: <root>d:(AnyOpcodePattern [TRUNC, SEXT])
+// CHECK-NEXT: <match_root>d:(AnyOpcodePattern [G_TRUNC, G_SEXT])
// CHECK-NEXT: )
// CHECK-NEXT: (ApplyPats
-// CHECK-NEXT: __anon_pat_apply_1_0:(CXXPattern apply code:"APPLY")
+// CHECK-NEXT: __WipOpcodeTest1_apply_0:(CXXPattern apply code:"APPLY")
// CHECK-NEXT: )
-// CHECK-NEXT: (OperandTable <empty>)
+// CHECK-NEXT: (OperandTable MatchPats <empty>)
+// CHECK-NEXT: (OperandTable ApplyPats <empty>)
// CHECK-NEXT: )
def WipOpcodeTest1 : GICombineRule<
(defs root:$d),
- (match (wip_match_opcode TRUNC, SEXT):$d),
+ (match (wip_match_opcode G_TRUNC, G_SEXT):$d),
(apply [{ APPLY }])>;
// CHECK: (CombineRule name:InstTest0 id:2 root:d
-// CHECK-NEXT: (MatchDatas <empty>)
// CHECK-NEXT: (MatchPats
-// CHECK-NEXT: <root>d:(InstructionPattern inst:MOV operands:[<def>a, b])
+// CHECK-NEXT: <match_root>d:(CodeGenInstructionPattern COPY operands:[<def>$a, $b])
// CHECK-NEXT: )
// CHECK-NEXT: (ApplyPats
-// CHECK-NEXT: __anon_pat_apply_2_0:(CXXPattern apply code:"APPLY")
+// CHECK-NEXT: __InstTest0_apply_0:(CXXPattern apply code:"APPLY")
// CHECK-NEXT: )
-// CHECK-NEXT: (OperandTable
-// CHECK-NEXT: [a match_pat:d]
-// CHECK-NEXT: [b live-in]
+// CHECK-NEXT: (OperandTable MatchPats
+// CHECK-NEXT: a -> d
+// CHECK-NEXT: b -> <live-in>
// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable ApplyPats <empty>)
// CHECK-NEXT: )
def InstTest0 : GICombineRule<
(defs root:$d),
- (match (MOV $a, $b):$d),
+ (match (COPY $a, $b):$d),
(apply [{ APPLY }])>;
// CHECK: (CombineRule name:InstTest1 id:3 root:d
@@ -80,28 +66,246 @@ def InstTest0 : GICombineRule<
// CHECK-NEXT: (MatchDataInfo pattern_symbol:r0 type:'Register' var_name:MDInfo0)
// CHECK-NEXT: )
// CHECK-NEXT: (MatchPats
-// CHECK-NEXT: <root>d:(InstructionPattern inst:MOV operands:[<def>a, b])
-// CHECK-NEXT: __anon_pat_match_3_0:(InstructionPattern inst:ZEXT operands:[<def>x, a])
+// CHECK-NEXT: <match_root>d:(CodeGenInstructionPattern COPY operands:[<def>$a, i32:$b])
+// CHECK-NEXT: __InstTest1_match_1:(CodeGenInstructionPattern G_ZEXT operands:[<def>$x, 0])
// CHECK-NEXT: )
// CHECK-NEXT: (ApplyPats
-// CHECK-NEXT: __anon_pat_apply_3_1:(CXXPattern apply code:"APPLY")
+// CHECK-NEXT: __InstTest1_apply_0:(CXXPattern apply code:"APPLY")
// CHECK-NEXT: )
-// CHECK-NEXT: (OperandTable
-// CHECK-NEXT: [a match_pat:d]
-// CHECK-NEXT: [b live-in]
-// CHECK-NEXT: [x match_pat:__anon_pat_match_3_0]
+// CHECK-NEXT: (OperandTable MatchPats
+// CHECK-NEXT: a -> d
+// CHECK-NEXT: b -> <live-in>
+// CHECK-NEXT: x -> __InstTest1_match_1
// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable ApplyPats <empty>)
// CHECK-NEXT: )
let Predicates = [HasAnswerToEverything] in
def InstTest1 : GICombineRule<
(defs root:$d, reg_matchinfo:$r0),
- (match (MOV $a, $b):$d,
- (ZEXT $x, $a)),
+ (match (COPY $a, i32:$b):$d,
+ (G_ZEXT $x, 0)),
(apply [{ APPLY }])>;
+// CHECK: (CombineRule name:InstTest2 id:4 root:d
+// CHECK-NEXT: (MatchDatas
+// CHECK-NEXT: (MatchDataInfo pattern_symbol:r0 type:'Register' var_name:MDInfo0)
+// CHECK-NEXT: )
+// CHECK-NEXT: (MatchPats
+// CHECK-NEXT: <match_root>__InstTest2_match_0:(CodeGenInstructionPattern COPY operands:[<def>$d, (i32 0):$x])
+// CHECK-NEXT: )
+// CHECK-NEXT: (ApplyPats
+// CHECK-NEXT: __InstTest2_apply_0:(CXXPattern apply code:"APPLY")
+// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable MatchPats
+// CHECK-NEXT: d -> __InstTest2_match_0
+// CHECK-NEXT: x -> <live-in>
+// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable ApplyPats <empty>)
+// CHECK-NEXT: )
+def InstTest2 : GICombineRule<
+ (defs root:$d, reg_matchinfo:$r0),
+ (match (COPY $d, (i32 0):$x)),
+ (apply [{ APPLY }])>;
+
+// CHECK: (CombineRule name:InOutInstTest0 id:5 root:dst
+// CHECK-NEXT: (MatchPats
+// CHECK-NEXT: <match_root>__InOutInstTest0_match_0:(CodeGenInstructionPattern COPY operands:[<def>$dst, $tmp])
+// CHECK-NEXT: __InOutInstTest0_match_1:(CodeGenInstructionPattern G_ZEXT operands:[<def>$tmp, $src])
+// CHECK-NEXT: )
+// CHECK-NEXT: (ApplyPats
+// CHECK-NEXT: <apply_root>__InOutInstTest0_apply_0:(CodeGenInstructionPattern G_TRUNC operands:[<def>$dst, $src])
+// CHECK-NEXT: __InOutInstTest0_apply_1:(CXXPattern apply code:"APPLY ${src}")
+// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable MatchPats
+// CHECK-NEXT: dst -> __InOutInstTest0_match_0
+// CHECK-NEXT: src -> <live-in>
+// CHECK-NEXT: tmp -> __InOutInstTest0_match_1
+// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable ApplyPats
+// CHECK-NEXT: dst -> __InOutInstTest0_apply_0
+// CHECK-NEXT: src -> <live-in>
+// CHECK-NEXT: )
+// CHECK-NEXT: )
+def InOutInstTest0 : GICombineRule<
+ (defs root:$dst),
+ (match (COPY $dst, $tmp),
+ (G_ZEXT $tmp, $src)),
+ (apply (G_TRUNC $dst, $src), "APPLY ${src}")>;
+
+def MatchICst: GICombinePatFrag<
+ (outs),
+ (ins gi_mo:$foo, gi_imm:$cst),
+ [(pattern "return matchIConstant(${foo}, ${cst})")]>;
+
+// CHECK: (CombineRule name:PatFragTest0 id:6 root:dst
+// CHECK-NEXT: (PatFrags
+// CHECK-NEXT: (PatFrag name:MatchICst
+// CHECK-NEXT: (ins [foo:machine_operand, cst:imm])
+// CHECK-NEXT: (alternatives [
+// CHECK-NEXT: [
+// CHECK-NEXT: (CXXPattern name:__MatchICst_alt0_pattern_0 match code:"return matchIConstant(${foo}, ${cst})"),
+// CHECK-NEXT: ],
+// CHECK-NEXT: ])
+// CHECK-NEXT: )
+// CHECK-NEXT: )
+// CHECK-NEXT: (MatchPats
+// CHECK-NEXT: <match_root>__PatFragTest0_match_0:(CodeGenInstructionPattern G_ZEXT operands:[<def>$dst, $cst])
+// CHECK-NEXT: __PatFragTest0_match_1:(PatFragPattern MatchICst operands:[$cst, (i32 0)])
+// CHECK-NEXT: )
+// CHECK-NEXT: (ApplyPats
+// CHECK-NEXT: <apply_root>__PatFragTest0_apply_0:(CodeGenInstructionPattern COPY operands:[<def>$dst, (i32 0)])
+// CHECK-NEXT: __PatFragTest0_apply_1:(CXXPattern apply code:"APPLY ${src}")
+// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable MatchPats
+// CHECK-NEXT: cst -> <live-in>
+// CHECK-NEXT: dst -> __PatFragTest0_match_0
+// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable ApplyPats
+// CHECK-NEXT: dst -> __PatFragTest0_apply_0
+// CHECK-NEXT: )
+// CHECK-NEXT: )
+def PatFragTest0 : GICombineRule<
+ (defs root:$dst),
+ (match (G_ZEXT $dst, $cst), (MatchICst $cst, (i32 0))),
+ (apply (COPY $dst, (i32 0)), "APPLY ${src}")>;
+
+def MatchFooPerms: GICombinePatFrag<
+ (outs),
+ (ins gi_mo:$foo, gi_imm:$cst),
+ [
+ (pattern "return foo(${foo}, ${cst})"),
+ (pattern "return bar(${foo}, ${cst})"),
+ (pattern "return bux(${foo}, ${cst})"),
+ ]>;
+
+// CHECK: (CombineRule name:PatFragTest1 id:7 root:dst
+// CHECK-NEXT: (PatFrags
+// CHECK-NEXT: (PatFrag name:MatchFooPerms
+// CHECK-NEXT: (ins [foo:machine_operand, cst:imm])
+// CHECK-NEXT: (alternatives [
+// CHECK-NEXT: [
+// CHECK-NEXT: (CXXPattern name:__MatchFooPerms_alt0_pattern_0 match code:"return foo(${foo}, ${cst})"),
+// CHECK-NEXT: ],
+// CHECK-NEXT: [
+// CHECK-NEXT: (CXXPattern name:__MatchFooPerms_alt1_pattern_0 match code:"return bar(${foo}, ${cst})"),
+// CHECK-NEXT: ],
+// CHECK-NEXT: [
+// CHECK-NEXT: (CXXPattern name:__MatchFooPerms_alt2_pattern_0 match code:"return bux(${foo}, ${cst})"),
+// CHECK-NEXT: ],
+// CHECK-NEXT: ])
+// CHECK-NEXT: )
+// CHECK-NEXT: )
+// CHECK-NEXT: (MatchPats
+// CHECK-NEXT: <match_root>__PatFragTest1_match_0:(CodeGenInstructionPattern G_ZEXT operands:[<def>$dst, $cst])
+// CHECK-NEXT: a:(PatFragPattern MatchFooPerms operands:[$cst, (i32 0)])
+// CHECK-NEXT: b:(PatFragPattern MatchFooPerms operands:[$cst, (i32 0)])
+// CHECK-NEXT: c:(PatFragPattern MatchFooPerms operands:[$cst, (i32 0)])
+// CHECK-NEXT: )
+// CHECK-NEXT: (ApplyPats
+// CHECK-NEXT: <apply_root>__PatFragTest1_apply_0:(CodeGenInstructionPattern COPY operands:[<def>$dst, (i32 0)])
+// CHECK-NEXT: __PatFragTest1_apply_1:(CXXPattern apply code:"APPLY ${src}")
+// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable MatchPats
+// CHECK-NEXT: cst -> <live-in>
+// CHECK-NEXT: dst -> __PatFragTest1_match_0
+// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable ApplyPats
+// CHECK-NEXT: dst -> __PatFragTest1_apply_0
+// CHECK-NEXT: )
+// CHECK-NEXT: (PermutationsToEmit
+// CHECK-NEXT: [a[0], b[0], c[0]],
+// CHECK-NEXT: [a[0], b[0], c[1]],
+// CHECK-NEXT: [a[0], b[0], c[2]],
+// CHECK-NEXT: [a[0], b[1], c[0]],
+// CHECK-NEXT: [a[0], b[1], c[1]],
+// CHECK-NEXT: [a[0], b[1], c[2]],
+// CHECK-NEXT: [a[0], b[2], c[0]],
+// CHECK-NEXT: [a[0], b[2], c[1]],
+// CHECK-NEXT: [a[0], b[2], c[2]],
+// CHECK-NEXT: [a[1], b[0], c[0]],
+// CHECK-NEXT: [a[1], b[0], c[1]],
+// CHECK-NEXT: [a[1], b[0], c[2]],
+// CHECK-NEXT: [a[1], b[1], c[0]],
+// CHECK-NEXT: [a[1], b[1], c[1]],
+// CHECK-NEXT: [a[1], b[1], c[2]],
+// CHECK-NEXT: [a[1], b[2], c[0]],
+// CHECK-NEXT: [a[1], b[2], c[1]],
+// CHECK-NEXT: [a[1], b[2], c[2]],
+// CHECK-NEXT: [a[2], b[0], c[0]],
+// CHECK-NEXT: [a[2], b[0], c[1]],
+// CHECK-NEXT: [a[2], b[0], c[2]],
+// CHECK-NEXT: [a[2], b[1], c[0]],
+// CHECK-NEXT: [a[2], b[1], c[1]],
+// CHECK-NEXT: [a[2], b[1], c[2]],
+// CHECK-NEXT: [a[2], b[2], c[0]],
+// CHECK-NEXT: [a[2], b[2], c[1]],
+// CHECK-NEXT: [a[2], b[2], c[2]],
+// CHECK-NEXT: )
+// CHECK-NEXT: )
+let MaxPermutations = -1 in
+def PatFragTest1 : GICombineRule<
+ (defs root:$dst),
+ (match (G_ZEXT $dst, $cst),
+ (MatchFooPerms $cst, (i32 0)):$a,
+ (MatchFooPerms $cst, (i32 0)):$b,
+ (MatchFooPerms $cst, (i32 0)):$c
+ ),
+ (apply (COPY $dst, (i32 0)), "APPLY ${src}")>;
+
+// CHECK: (CombineRule name:VariadicsInTest id:8 root:dst
+// CHECK-NEXT: (MatchPats
+// CHECK-NEXT: <match_root>__VariadicsInTest_match_0:(CodeGenInstructionPattern G_BUILD_VECTOR operands:[<def>$dst, $a, $b])
+// CHECK-NEXT: )
+// CHECK-NEXT: (ApplyPats
+// CHECK-NEXT: <apply_root>__VariadicsInTest_apply_0:(CodeGenInstructionPattern COPY operands:[<def>$dst, (i32 0)])
+// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable MatchPats
+// CHECK-NEXT: a -> <live-in>
+// CHECK-NEXT: b -> <live-in>
+// CHECK-NEXT: dst -> __VariadicsInTest_match_0
+// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable ApplyPats
+// CHECK-NEXT: dst -> __VariadicsInTest_apply_0
+// CHECK-NEXT: )
+// CHECK-NEXT: )
+def VariadicsInTest : GICombineRule<
+ (defs root:$dst),
+ (match (G_BUILD_VECTOR $dst, $a, $b)),
+ (apply (COPY $dst, (i32 0)))>;
+
+// CHECK: (CombineRule name:VariadicsOutTest id:9 root:a
+// CHECK-NEXT: (MatchPats
+// CHECK-NEXT: <match_root>__VariadicsOutTest_match_0:(CodeGenInstructionPattern G_UNMERGE_VALUES operands:[<def>$a, <def>$b, $src])
+// CHECK-NEXT: )
+// CHECK-NEXT: (ApplyPats
+// CHECK-NEXT: <apply_root>__VariadicsOutTest_apply_0:(CodeGenInstructionPattern COPY operands:[<def>$a, (i32 0)])
+// CHECK-NEXT: <apply_root>__VariadicsOutTest_apply_1:(CodeGenInstructionPattern COPY operands:[<def>$b, (i32 0)])
+// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable MatchPats
+// CHECK-NEXT: a -> __VariadicsOutTest_match_0
+// CHECK-NEXT: b -> __VariadicsOutTest_match_0
+// CHECK-NEXT: src -> <live-in>
+// CHECK-NEXT: )
+// CHECK-NEXT: (OperandTable ApplyPats
+// CHECK-NEXT: a -> __VariadicsOutTest_apply_0
+// CHECK-NEXT: b -> __VariadicsOutTest_apply_1
+// CHECK-NEXT: )
+// CHECK-NEXT: )
+def VariadicsOutTest : GICombineRule<
+ (defs root:$a),
+ (match (G_UNMERGE_VALUES $a, $b, $src)),
+ (apply (COPY $a, (i32 0)),
+ (COPY $b, (i32 0)))>;
+
def MyCombiner: GICombinerHelper<"GenMyCombiner", [
WipOpcodeTest0,
WipOpcodeTest1,
InstTest0,
- InstTest1
+ InstTest1,
+ InstTest2,
+ InOutInstTest0,
+ PatFragTest0,
+ PatFragTest1,
+ VariadicsInTest,
+ VariadicsOutTest
]>;
diff --git a/llvm/test/TableGen/GlobalISelEmitter.td b/llvm/test/TableGen/GlobalISelEmitter.td
index 24ebe807d27b0d..f00eea3970894a 100644
--- a/llvm/test/TableGen/GlobalISelEmitter.td
+++ b/llvm/test/TableGen/GlobalISelEmitter.td
@@ -82,7 +82,7 @@ def HasC : Predicate<"Subtarget->hasC()"> { let RecomputePerFunction = 1; }
// CHECK-NEXT: const int64_t *getMatchTable() const override;
// CHECK-NEXT: bool testMIPredicate_MI(unsigned PredicateID, const MachineInstr &MI, const MatcherState &State) const override;
// CHECK-NEXT: bool testSimplePredicate(unsigned PredicateID) const override;
-// CHECK-NEXT: void runCustomAction(unsigned FnID, const MatcherState &State) const override;
+// CHECK-NEXT: void runCustomAction(unsigned FnID, const MatcherState &State, NewMIVector &OutMIs) const override;
// CHECK-NEXT: #endif // ifdef GET_GLOBALISEL_TEMPORARIES_DECL
// CHECK-LABEL: #ifdef GET_GLOBALISEL_TEMPORARIES_INIT
diff --git a/llvm/utils/TableGen/CodeGenTarget.cpp b/llvm/utils/TableGen/CodeGenTarget.cpp
index fbdc0499a8cfbe..53efa66f9dfc1b 100644
--- a/llvm/utils/TableGen/CodeGenTarget.cpp
+++ b/llvm/utils/TableGen/CodeGenTarget.cpp
@@ -43,7 +43,7 @@ static cl::opt<unsigned>
/// getValueType - Return the MVT::SimpleValueType that the specified TableGen
/// record corresponds to.
-MVT::SimpleValueType llvm::getValueType(Record *Rec) {
+MVT::SimpleValueType llvm::getValueType(const Record *Rec) {
return (MVT::SimpleValueType)Rec->getValueAsInt("Value");
}
diff --git a/llvm/utils/TableGen/CodeGenTarget.h b/llvm/utils/TableGen/CodeGenTarget.h
index 2ba3af724d36dd..a2b559d53b19c2 100644
--- a/llvm/utils/TableGen/CodeGenTarget.h
+++ b/llvm/utils/TableGen/CodeGenTarget.h
@@ -43,7 +43,7 @@ class CodeGenSubRegIndex;
/// getValueType - Return the MVT::SimpleValueType that the specified TableGen
/// record corresponds to.
-MVT::SimpleValueType getValueType(Record *Rec);
+MVT::SimpleValueType getValueType(const Record *Rec);
StringRef getName(MVT::SimpleValueType T);
StringRef getEnumName(MVT::SimpleValueType T);
diff --git a/llvm/utils/TableGen/GlobalISel/CodeExpansions.h b/llvm/utils/TableGen/GlobalISel/CodeExpansions.h
index f536e801b27f39..b82c3257b32135 100644
--- a/llvm/utils/TableGen/GlobalISel/CodeExpansions.h
+++ b/llvm/utils/TableGen/GlobalISel/CodeExpansions.h
@@ -29,6 +29,10 @@ class CodeExpansions {
Expansions.try_emplace(Name, Expansion);
}
+ void redeclare(StringRef Name, StringRef Expansion) {
+ Expansions[Name] = Expansion;
+ }
+
std::string lookup(StringRef Variable) const {
return Expansions.lookup(Variable);
}
diff --git a/llvm/utils/TableGen/GlobalISel/CombinerUtils.h b/llvm/utils/TableGen/GlobalISel/CombinerUtils.h
index 394c43e3fa833b..3d34eb450ed2d2 100644
--- a/llvm/utils/TableGen/GlobalISel/CombinerUtils.h
+++ b/llvm/utils/TableGen/GlobalISel/CombinerUtils.h
@@ -61,10 +61,9 @@ inline const DagInit *getDagWithSpecificOperator(const Init &N,
inline const DagInit *getDagWithOperatorOfSubClass(const Init &N,
StringRef Cls) {
if (const DagInit *I = dyn_cast<DagInit>(&N))
- if (I->getNumArgs() > 0)
- if (const DefInit *OpI = dyn_cast<DefInit>(I->getOperator()))
- if (OpI->getDef()->isSubClassOf(Cls))
- return I;
+ if (const DefInit *OpI = dyn_cast<DefInit>(I->getOperator()))
+ if (OpI->getDef()->isSubClassOf(Cls))
+ return I;
return nullptr;
}
} // namespace llvm
diff --git a/llvm/utils/TableGen/GlobalISelCombinerMatchTableEmitter.cpp b/llvm/utils/TableGen/GlobalISelCombinerMatchTableEmitter.cpp
index b4a16a34f88e28..53bb54850c9150 100644
--- a/llvm/utils/TableGen/GlobalISelCombinerMatchTableEmitter.cpp
+++ b/llvm/utils/TableGen/GlobalISelCombinerMatchTableEmitter.cpp
@@ -9,6 +9,21 @@
/// \file Generate a combiner implementation for GlobalISel from a declarative
/// syntax using GlobalISelMatchTable.
///
+/// Usually, TableGen backends use "assert is an error" as a means to report
+/// invalid input. They try to diagnose common case but don't try very hard and
+/// crashes can be common. This backend aims to behave closer to how a language
+/// compiler frontend would behave: we try extra hard to diagnose invalid inputs
+/// early, and any crash should be considered a bug (= a feature or diagnostic
+/// is missing).
+///
+/// While this can make the backend a bit more complex than it needs to be, it
+/// pays off because MIR patterns can get complicated. Giving useful error
+/// messages to combine writers can help boost their productivity.
+///
+/// As with anything, a good balance has to be found. We also don't want to
+/// write hundreds of lines of code to detect edge cases. In practice, crashing
+/// very occasionally, or giving poor errors in some rare instances, is fine.
+///
//===----------------------------------------------------------------------===//
#include "CodeGenInstruction.h"
@@ -19,12 +34,14 @@
#include "GlobalISelMatchTable.h"
#include "GlobalISelMatchTableExecutorEmitter.h"
#include "SubtargetFeatureInfo.h"
+#include "llvm/ADT/APInt.h"
#include "llvm/ADT/Hashing.h"
#include "llvm/ADT/MapVector.h"
#include "llvm/ADT/Statistic.h"
#include "llvm/ADT/StringSet.h"
#include "llvm/Support/CommandLine.h"
#include "llvm/Support/Debug.h"
+#include "llvm/Support/PrettyStackTrace.h"
#include "llvm/Support/ScopedPrinter.h"
#include "llvm/TableGen/Error.h"
#include "llvm/TableGen/Record.h"
@@ -39,26 +56,58 @@ using namespace llvm::gi;
extern cl::list<std::string> SelectedCombiners;
extern cl::opt<bool> StopAfterParse;
+extern cl::OptionCategory GICombinerEmitterCat;
+static cl::opt<bool> DebugCXXPreds(
+ "gicombiner-matchtable-debug-cxxpreds",
+ cl::desc("Add Contextual/Debug comments to all C++ predicates"),
+ cl::cat(GICombinerEmitterCat));
namespace {
constexpr StringLiteral CXXApplyPrefix = "GICXXCustomAction_CombineApply";
constexpr StringLiteral CXXPredPrefix = "GICXXPred_MI_Predicate_";
+constexpr StringLiteral PatFragClassName = "GICombinePatFrag";
std::string getIsEnabledPredicateEnumName(unsigned CombinerRuleID) {
return "GICXXPred_Simple_IsRule" + to_string(CombinerRuleID) + "Enabled";
}
+/// Copies a StringRef into a static pool to make sure it has a static lifetime.
+StringRef insertStrRef(StringRef S) {
+ if (S.empty())
+ return {};
+
+ static StringSet<> Pool;
+ auto [It, Inserted] = Pool.insert(S);
+ return It->getKey();
+}
+
void declareInstExpansion(CodeExpansions &CE, const InstructionMatcher &IM,
StringRef Name) {
CE.declare(Name, "State.MIs[" + to_string(IM.getInsnVarID()) + "]");
}
+void declareInstExpansion(CodeExpansions &CE, const BuildMIAction &A,
+ StringRef Name) {
+ // Note: we use redeclare here because this may overwrite a matcher inst
+ // expansion.
+ CE.redeclare(Name, "OutMIs[" + to_string(A.getInsnID()) + "]");
+}
+
void declareOperandExpansion(CodeExpansions &CE, const OperandMatcher &OM,
StringRef Name) {
CE.declare(Name, "State.MIs[" + to_string(OM.getInsnVarID()) +
"]->getOperand(" + to_string(OM.getOpIdx()) + ")");
}
+void declareTempRegExpansion(CodeExpansions &CE, unsigned TempRegID,
+ StringRef Name) {
+ CE.declare(Name, "State.TempRegisters[" + to_string(TempRegID) + "]");
+}
+
+std::string makeAnonPatName(StringRef Prefix, unsigned Idx) {
+ return ("__" + Prefix + "_" + Twine(Idx)).str();
+}
+
template <typename Container> auto keys(Container &&C) {
return map_range(C, [](auto &Entry) -> auto & { return Entry.first; });
}
@@ -67,6 +116,11 @@ template <typename Container> auto values(Container &&C) {
return map_range(C, [](auto &Entry) -> auto & { return Entry.second; });
}
+LLTCodeGen getLLTCodeGenFromRecord(const Record *Ty) {
+ assert(Ty->isSubClassOf("ValueType"));
+ return LLTCodeGen(*MVTToLLT(getValueType(Ty)));
+}
+
//===- MatchData Handling -------------------------------------------------===//
/// Represents MatchData defined by the match stage and required by the apply
@@ -135,32 +189,84 @@ void AssignMatchDataVariables(MutableArrayRef<MatchDataInfo> Infos) {
static unsigned NextVarID = 0;
StringMap<unsigned> SeenTypes;
- for (auto &I : Infos) {
- unsigned &NumSeen = SeenTypes[I.getType()];
- auto &ExistingVars = AllMatchDataVars[I.getType()];
+ for (auto &Info : Infos) {
+ unsigned &NumSeen = SeenTypes[Info.getType()];
+ auto &ExistingVars = AllMatchDataVars[Info.getType()];
if (NumSeen == ExistingVars.size())
ExistingVars.push_back("MDInfo" + to_string(NextVarID++));
- I.setVariableName(ExistingVars[NumSeen++]);
+ Info.setVariableName(ExistingVars[NumSeen++]);
}
}
//===- C++ Predicates Handling --------------------------------------------===//
-/// Entry into the static pool of all CXX Predicate code. This contains the
+/// Entry into the static pool of all CXX Predicate code. This contains
/// fully expanded C++ code.
///
-/// Each CXXPattern creates a new entry in the pool to store its data, even
-/// after the pattern is destroyed.
+/// The static pool is hidden inside the object and can be accessed through
+/// getAllMatchCode/getAllApplyCode
///
/// Note that CXXPattern trims C++ code, so the Code is already expected to be
/// free of leading/trailing whitespace.
-struct CXXPredicateCode {
+class CXXPredicateCode {
+ using CXXPredicateCodePool =
+ DenseMap<hash_code, std::unique_ptr<CXXPredicateCode>>;
+ static CXXPredicateCodePool AllCXXMatchCode;
+ static CXXPredicateCodePool AllCXXApplyCode;
+
+ /// Sorts a `CXXPredicateCodePool` by their IDs and returns it.
+ static std::vector<const CXXPredicateCode *>
+ getSorted(const CXXPredicateCodePool &Pool) {
+ std::vector<const CXXPredicateCode *> Out;
+ std::transform(Pool.begin(), Pool.end(), std::back_inserter(Out),
+ [&](auto &Elt) { return Elt.second.get(); });
+ sort(Out, [](const auto *A, const auto *B) { return A->ID < B->ID; });
+ return Out;
+ }
+
+ /// Gets an instance of `CXXPredicateCode` for \p Code, or returns an already
+ /// existing one.
+ static const CXXPredicateCode &get(CXXPredicateCodePool &Pool,
+ std::string Code) {
+ // Check if we already have an identical piece of code, if not, create an
+ // entry in the pool.
+ const auto CodeHash = hash_value(Code);
+ if (auto It = Pool.find(CodeHash); It != Pool.end())
+ return *It->second;
+
+ const auto ID = Pool.size();
+ auto OwnedData = std::unique_ptr<CXXPredicateCode>(
+ new CXXPredicateCode(std::move(Code), ID));
+ const auto &DataRef = *OwnedData;
+ Pool[CodeHash] = std::move(OwnedData);
+ return DataRef;
+ }
+
CXXPredicateCode(std::string Code, unsigned ID)
: Code(Code), ID(ID), BaseEnumName("GICombiner" + to_string(ID)) {
- assert(StringRef(Code).trim() == Code &&
- "Code was expected to be trimmed!");
+ // Don't assert if ErrorsPrinted is set. This may mean CodeExpander failed,
+ // and it may add spaces in such cases.
+ assert(ErrorsPrinted || StringRef(Code).trim() == Code &&
+ "Code was expected to be trimmed!");
+ }
+
+public:
+ static const CXXPredicateCode &getMatchCode(std::string Code) {
+ return get(AllCXXMatchCode, std::move(Code));
+ }
+
+ static const CXXPredicateCode &getApplyCode(std::string Code) {
+ return get(AllCXXApplyCode, std::move(Code));
+ }
+
+ static std::vector<const CXXPredicateCode *> getAllMatchCode() {
+ return getSorted(AllCXXMatchCode);
+ }
+
+ static std::vector<const CXXPredicateCode *> getAllApplyCode() {
+ return getSorted(AllCXXApplyCode);
}
const std::string Code;
@@ -176,48 +282,29 @@ struct CXXPredicateCode {
}
};
-using CXXPredicateCodePool =
- DenseMap<hash_code, std::unique_ptr<CXXPredicateCode>>;
-CXXPredicateCodePool AllCXXMatchCode;
-CXXPredicateCodePool AllCXXApplyCode;
-
-/// Gets an instance of `CXXPredicateCode` for \p Code, or returns an already
-/// existing one.
-const CXXPredicateCode &getOrInsert(CXXPredicateCodePool &Pool,
- std::string Code) {
- // Check if we already have an identical piece of code, if not, create an
- // entry in the pool.
- const auto CodeHash = hash_value(Code);
- if (auto It = Pool.find(CodeHash); It != Pool.end())
- return *It->second;
-
- const auto ID = Pool.size();
- auto OwnedData = std::make_unique<CXXPredicateCode>(std::move(Code), ID);
- const auto &DataRef = *OwnedData;
- Pool[CodeHash] = std::move(OwnedData);
- return DataRef;
-}
-
-/// Sorts a `CXXPredicateCodePool` by their IDs and returns it.
-std::vector<const CXXPredicateCode *>
-getSorted(const CXXPredicateCodePool &Pool) {
- std::vector<const CXXPredicateCode *> Out;
- std::transform(Pool.begin(), Pool.end(), std::back_inserter(Out),
- [&](auto &Elt) { return Elt.second.get(); });
- sort(Out, [](const auto *A, const auto *B) { return A->ID < B->ID; });
- return Out;
-}
+CXXPredicateCode::CXXPredicateCodePool CXXPredicateCode::AllCXXMatchCode;
+CXXPredicateCode::CXXPredicateCodePool CXXPredicateCode::AllCXXApplyCode;
//===- Pattern Base Class -------------------------------------------------===//
-// An abstract pattern found in a combine rule. This can be an apply or match
-// pattern.
+/// Base class for all patterns that can be written in an `apply`, `match` or
+/// `pattern` DAG operator.
+///
+/// For example:
+///
+/// (apply (G_ZEXT $x, $y), (G_ZEXT $y, $z), "return isFoo(${z})")
+///
+/// Creates 3 Pattern objects:
+/// - Two CodeGenInstruction Patterns
+/// - A CXXPattern
class Pattern {
public:
enum {
K_AnyOpcode,
- K_Inst,
K_CXX,
+
+ K_CodeGenInstruction,
+ K_PatFrag,
};
virtual ~Pattern() = default;
@@ -232,7 +319,8 @@ class Pattern {
void dump() const { return print(dbgs()); }
protected:
- Pattern(unsigned Kind, StringRef Name) : Kind(Kind), Name(Name.str()) {
+ Pattern(unsigned Kind, StringRef Name)
+ : Kind(Kind), Name(insertStrRef(Name)) {
assert(!Name.empty() && "unnamed pattern!");
}
@@ -241,21 +329,19 @@ class Pattern {
private:
unsigned Kind;
-
- // Note: if this ever changes to a StringRef (e.g. allocated in a pool or
- // something), CombineRuleBuilder::verify() needs to be updated as well.
- // It currently checks that the StringRef in the PatternMap references this.
- std::string Name;
+ StringRef Name;
};
const char *Pattern::getKindName() const {
switch (Kind) {
case K_AnyOpcode:
return "AnyOpcodePattern";
- case K_Inst:
- return "InstructionPattern";
case K_CXX:
return "CXXPattern";
+ case K_CodeGenInstruction:
+ return "CodeGenInstructionPattern";
+ case K_PatFrag:
+ return "PatFragPattern";
}
llvm_unreachable("unknown pattern kind!");
@@ -275,6 +361,9 @@ void Pattern::printImpl(raw_ostream &OS, bool PrintName,
/// `wip_match_opcode` patterns.
/// This matches one or more opcodes, and does not check any operands
/// whatsoever.
+///
+/// TODO: Long-term, this needs to be removed. It's a hack around MIR
+/// pattern matching limitations.
class AnyOpcodePattern : public Pattern {
public:
AnyOpcodePattern(StringRef Name) : Pattern(K_AnyOpcode, Name) {}
@@ -300,77 +389,9 @@ void AnyOpcodePattern::print(raw_ostream &OS, bool PrintName) const {
});
}
-//===- InstructionPattern -------------------------------------------------===//
-
-/// Matches an instruction, e.g. `G_ADD $x, $y, $z`.
-///
-/// This pattern is simply CodeGenInstruction + a list of operands.
-class InstructionPattern : public Pattern {
-public:
- struct Operand {
- std::string Name;
- bool IsDef = false;
- };
-
- InstructionPattern(const CodeGenInstruction &I, StringRef Name)
- : Pattern(K_Inst, Name), I(I) {}
-
- static bool classof(const Pattern *P) { return P->getKind() == K_Inst; }
-
- const auto &operands() const { return Operands; }
- void addOperand(StringRef Name);
- unsigned getNumDefs() const { return I.Operands.NumDefs; }
-
- const CodeGenInstruction &getInst() const { return I; }
- StringRef getInstName() const { return I.TheDef->getName(); }
-
- void reportUnreachable(ArrayRef<SMLoc> Locs) const;
- bool checkSemantics(ArrayRef<SMLoc> Loc) const;
-
- void print(raw_ostream &OS, bool PrintName = true) const override;
-
-private:
- const CodeGenInstruction &I;
- SmallVector<Operand, 4> Operands;
-};
-
-void InstructionPattern::addOperand(StringRef Name) {
- const bool IsDef = Operands.size() < getNumDefs();
- Operands.emplace_back(Operand{Name.str(), IsDef});
-}
-
-void InstructionPattern::reportUnreachable(ArrayRef<SMLoc> Locs) const {
- PrintError(Locs, "Instruction pattern '" + getName() +
- "' is unreachable from the pattern root!");
-}
-
-bool InstructionPattern::checkSemantics(ArrayRef<SMLoc> Loc) const {
- unsigned NumExpectedOperands = I.Operands.size();
- if (NumExpectedOperands != Operands.size()) {
-
- PrintError(Loc, "'" + getInstName() + "' expected " +
- Twine(NumExpectedOperands) + " operands, got " +
- Twine(Operands.size()));
- return false;
- }
- return true;
-}
-
-void InstructionPattern::print(raw_ostream &OS, bool PrintName) const {
- printImpl(OS, PrintName, [&OS, this]() {
- OS << "inst:" << I.TheDef->getName() << " operands:["
- << join(map_range(Operands,
- [](const auto &O) {
- return (O.IsDef ? "<def>" : "") + O.Name;
- }),
- ", ")
- << "]";
- });
-}
-
//===- CXXPattern ---------------------------------------------------------===//
-/// Raw C++ code which may need some expansions.
+/// Represents raw C++ code which may need some expansions.
///
/// e.g. [{ return isFooBux(${src}.getReg()); }]
///
@@ -392,15 +413,15 @@ void InstructionPattern::print(raw_ostream &OS, bool PrintName) const {
/// `.startswith("return")` trivially without worrying about spaces.
class CXXPattern : public Pattern {
public:
- CXXPattern(const StringInit &Code, StringRef Name, bool IsApply)
- : CXXPattern(Code.getAsUnquotedString(), Name, IsApply) {}
+ CXXPattern(const StringInit &Code, StringRef Name)
+ : CXXPattern(Code.getAsUnquotedString(), Name) {}
- CXXPattern(StringRef Code, StringRef Name, bool IsApply)
- : Pattern(K_CXX, Name), IsApply(IsApply), RawCode(Code.trim().str()) {}
+ CXXPattern(StringRef Code, StringRef Name)
+ : Pattern(K_CXX, Name), RawCode(Code.trim().str()) {}
static bool classof(const Pattern *P) { return P->getKind() == K_CXX; }
- bool isApply() const { return IsApply; }
+ void setIsApply(bool Value = true) { IsApply = Value; }
StringRef getRawCode() const { return RawCode; }
/// Expands raw code, replacing things such as `${foo}` with their
@@ -409,27 +430,37 @@ class CXXPattern : public Pattern {
/// \param CE Map of Code Expansions
/// \param Locs SMLocs for the Code Expander, in case it needs to emit
/// diagnostics.
+ /// \param AddComment If DebugCXXPreds is enabled, this is called to emit a
+ /// comment before the expanded code.
+ ///
/// \return A CXXPredicateCode object that contains the expanded code. Note
/// that this may or may not insert a new object. All CXXPredicateCode objects
/// are held in a set to avoid emitting duplicate C++ code.
- const CXXPredicateCode &expandCode(const CodeExpansions &CE,
- ArrayRef<SMLoc> Locs) const;
+ const CXXPredicateCode &
+ expandCode(const CodeExpansions &CE, ArrayRef<SMLoc> Locs,
+ function_ref<void(raw_ostream &)> AddComment = {}) const;
void print(raw_ostream &OS, bool PrintName = true) const override;
private:
- bool IsApply;
+ bool IsApply = false;
std::string RawCode;
};
-const CXXPredicateCode &CXXPattern::expandCode(const CodeExpansions &CE,
- ArrayRef<SMLoc> Locs) const {
+const CXXPredicateCode &
+CXXPattern::expandCode(const CodeExpansions &CE, ArrayRef<SMLoc> Locs,
+ function_ref<void(raw_ostream &)> AddComment) const {
std::string Result;
raw_string_ostream OS(Result);
+
+ if (DebugCXXPreds && AddComment)
+ AddComment(OS);
+
CodeExpander Expander(RawCode, CE, Locs, /*ShowExpansions*/ false);
Expander.emit(OS);
- return getOrInsert(IsApply ? AllCXXApplyCode : AllCXXMatchCode,
- std::move(Result));
+ if (IsApply)
+ return CXXPredicateCode::getApplyCode(std::move(Result));
+ return CXXPredicateCode::getMatchCode(std::move(Result));
}
void CXXPattern::print(raw_ostream &OS, bool PrintName) const {
@@ -440,253 +471,1236 @@ void CXXPattern::print(raw_ostream &OS, bool PrintName) const {
});
}
-//===- CombineRuleBuilder -------------------------------------------------===//
-
-/// Helper for CombineRuleBuilder.
-///
-/// Represents information about an operand.
-/// Operands with no MatchPat are considered live-in to the pattern.
-struct OperandTableEntry {
- // The matcher pattern that defines this operand.
- // null for live-ins.
- InstructionPattern *MatchPat = nullptr;
- // The apply pattern that (re)defines this operand.
- // This can only be non-null if MatchPat is.
- InstructionPattern *ApplyPat = nullptr;
-
- bool isLiveIn() const { return !MatchPat; }
-};
+//===- InstructionPattern ---------------------------------------------===//
-/// Parses combine rule and builds a small intermediate representation to tie
-/// patterns together and emit RuleMatchers to match them. This may emit more
-/// than one RuleMatcher, e.g. for `wip_match_opcode`.
-///
-/// Memory management for `Pattern` objects is done through `std::unique_ptr`.
-/// In most cases, there are two stages to a pattern's lifetime:
-/// - Creation in a `parse` function
-/// - The unique_ptr is stored in a variable, and may be destroyed if the
-/// pattern is found to be semantically invalid.
-/// - Ownership transfer into a `PatternMap`
-/// - Once a pattern is moved into either the map of Match or Apply
-/// patterns, it is known to be valid and it never moves back.
-class CombineRuleBuilder {
+/// Base class for CodeGenInstructionPattern & PatFragPattern, which handles all
+/// the boilerplate for patterns that have a list of operands for some (pseudo)
+/// instruction.
+class InstructionPattern : public Pattern {
public:
- using PatternMap = MapVector<StringRef, std::unique_ptr<Pattern>>;
+ class Operand;
- CombineRuleBuilder(const CodeGenTarget &CGT,
- SubtargetFeatureInfoMap &SubtargetFeatures,
- Record &RuleDef, unsigned ID,
- std::vector<RuleMatcher> &OutRMs)
- : CGT(CGT), SubtargetFeatures(SubtargetFeatures), RuleDef(RuleDef),
- RuleID(ID), OutRMs(OutRMs) {}
+ virtual ~InstructionPattern();
- /// Parses all fields in the RuleDef record.
- bool parseAll();
+ static bool classof(const Pattern *P) {
+ return P->getKind() == K_CodeGenInstruction || P->getKind() == K_PatFrag;
+ }
- /// Emits all RuleMatchers into the vector of RuleMatchers passed in the
- /// constructor.
- bool emitRuleMatchers();
+ template <typename... Ty> void addOperand(Ty &&...Init) {
+ Operands.emplace_back(std::forward<Ty>(Init)...);
+ }
- void print(raw_ostream &OS) const;
- void dump() const { print(dbgs()); }
+ auto &operands() { return Operands; }
+ const auto &operands() const { return Operands; }
+ unsigned operands_size() const { return Operands.size(); }
+ Operand &getOperand(unsigned K) { return Operands[K]; }
+ const Operand &getOperand(unsigned K) const { return Operands[K]; }
- /// Debug-only verification of invariants.
-#ifndef NDEBUG
- void verify() const;
-#endif
+ auto named_operands() {
+ return make_filter_range(Operands,
+ [&](auto &O) { return O.isNamedOperand(); });
+ }
-private:
- void PrintError(Twine Msg) const { ::PrintError(RuleDef.getLoc(), Msg); }
+ auto named_operands() const {
+ return make_filter_range(Operands,
+ [&](auto &O) { return O.isNamedOperand(); });
+ }
- /// Adds the expansions from \see MatchDatas to \p CE.
- void declareAllMatchDatasExpansions(CodeExpansions &CE) const;
+ virtual bool isVariadic() const { return false; }
+ virtual unsigned getNumInstOperands() const = 0;
+ virtual unsigned getNumInstDefs() const = 0;
- /// Adds \p P to \p IM, expanding its code using \p CE.
- void addCXXPredicate(InstructionMatcher &IM, const CodeExpansions &CE,
- const CXXPattern &P);
+ bool hasAllDefs() const { return operands_size() >= getNumInstDefs(); }
- /// Generates a name for anonymous patterns.
- ///
- /// e.g. (G_ADD $x, $y, $z):$foo is a pattern named "foo", but if ":$foo" is
- /// absent, then the pattern is anonymous and this is used to assign it a
- /// name.
- std::string makeAnonPatName(StringRef Prefix) const;
- mutable unsigned AnonIDCnt = 0;
+ virtual StringRef getInstName() const = 0;
- /// Creates a new RuleMatcher with some boilerplate
- /// settings/actions/predicates, and and adds it to \p OutRMs.
- /// \see addFeaturePredicates too.
- ///
- /// \param AdditionalComment Comment string to be added to the
- /// `DebugCommentAction`.
- RuleMatcher &addRuleMatcher(Twine AdditionalComment = "");
- bool addFeaturePredicates(RuleMatcher &M);
+ void reportUnreachable(ArrayRef<SMLoc> Locs) const;
+ virtual bool checkSemantics(ArrayRef<SMLoc> Loc);
- bool findRoots();
- bool buildOperandsTable();
+ void print(raw_ostream &OS, bool PrintName = true) const override;
- bool parseDefs(DagInit &Def);
- bool parseMatch(DagInit &Match);
- bool parseApply(DagInit &Apply);
+protected:
+ InstructionPattern(unsigned K, StringRef Name) : Pattern(K, Name) {}
- std::unique_ptr<Pattern> parseInstructionMatcher(const Init &Arg,
- StringRef PatName);
- std::unique_ptr<Pattern> parseWipMatchOpcodeMatcher(const Init &Arg,
- StringRef PatName);
+ // std::vector is used here because we can instantiate it using a forward
+ // declaration.
+ std::vector<Operand> Operands;
+};
- bool emitMatchPattern(CodeExpansions &CE, const InstructionPattern &IP);
- bool emitMatchPattern(CodeExpansions &CE, const AnyOpcodePattern &AOP);
+/// An operand for an InstructionPattern.
+///
+/// Operands are composed of three elements:
+/// - (Optional) Value
+/// - (Optional) Name
+/// - (Optional) Type
+///
+/// Some examples:
+/// (i32 0):$x -> V=int(0), Name='x', Type=i32
+/// 0:$x -> V=int(0), Name='x'
+/// $x -> Name='x'
+/// i32:$x -> Name='x', Type = i32
+class InstructionPattern::Operand {
+public:
+ using IntImmTy = int64_t;
- bool emitApplyPatterns(CodeExpansions &CE, RuleMatcher &M);
+ Operand(IntImmTy Imm, StringRef Name, const Record *Type)
+ : Value(Imm), Name(insertStrRef(Name)), Type(Type) {
+ assert(!Type || Type->isSubClassOf("ValueType"));
+ }
- // Recursively visits InstructionPattern from P to build up the
- // RuleMatcher/InstructionMatcher. May create new InstructionMatchers as
- // needed.
- bool emitInstructionMatchPattern(CodeExpansions &CE, RuleMatcher &M,
- InstructionMatcher &IM,
- const InstructionPattern &P,
- DenseSet<const Pattern *> &SeenPats);
+ Operand(StringRef Name, const Record *Type)
+ : Name(insertStrRef(Name)), Type(Type) {}
- const CodeGenTarget &CGT;
- SubtargetFeatureInfoMap &SubtargetFeatures;
- Record &RuleDef;
- const unsigned RuleID;
- std::vector<RuleMatcher> &OutRMs;
+ bool isNamedImmediate() const { return hasImmValue() && isNamedOperand(); }
- // For InstructionMatcher::addOperand
- unsigned AllocatedTemporariesBaseID = 0;
+ bool hasImmValue() const { return Value.has_value(); }
+ IntImmTy getImmValue() const { return *Value; }
- /// The root of the pattern.
- StringRef RootName;
+ bool isNamedOperand() const { return !Name.empty(); }
+ StringRef getOperandName() const {
+ assert(isNamedOperand() && "Operand is unnamed");
+ return Name;
+ }
- /// These maps have ownership of the actual Pattern objects.
- /// They both map a Pattern's name to the Pattern instance.
- PatternMap MatchPats;
- PatternMap ApplyPats;
+ Operand withNewName(StringRef NewName) const {
+ Operand Result = *this;
+ Result.Name = insertStrRef(NewName);
+ return Result;
+ }
- /// Set by findRoots.
- Pattern *MatchRoot = nullptr;
+ void setIsDef(bool Value = true) { Def = Value; }
+ bool isDef() const { return Def; }
- MapVector<StringRef, OperandTableEntry> OperandTable;
- SmallVector<MatchDataInfo, 2> MatchDatas;
+ void setType(const Record *R) {
+ assert((!Type || (Type == R)) && "Overwriting type!");
+ Type = R;
+ }
+ const Record *getType() const { return Type; }
+
+ std::string describe() const {
+ if (!hasImmValue())
+ return "MachineOperand $" + getOperandName().str() + "";
+ std::string Str = "imm " + to_string(getImmValue());
+ if (isNamedImmediate())
+ Str += ":$" + getOperandName().str() + "";
+ return Str;
+ }
+
+ void print(raw_ostream &OS) const {
+ if (isDef())
+ OS << "<def>";
+
+ bool NeedsColon = true;
+ if (const Record *Ty = getType()) {
+ if (hasImmValue())
+ OS << "(" << Ty->getName() << " " << getImmValue() << ")";
+ else
+ OS << Ty->getName();
+ } else if (hasImmValue())
+ OS << getImmValue();
+ else
+ NeedsColon = false;
+
+ if (isNamedOperand())
+ OS << (NeedsColon ? ":" : "") << "$" << getOperandName();
+ }
+
+ void dump() const { return print(dbgs()); }
+
+private:
+ std::optional<int64_t> Value;
+ StringRef Name;
+ const Record *Type = nullptr;
+ bool Def = false;
};
-bool CombineRuleBuilder::parseAll() {
- if (!parseDefs(*RuleDef.getValueAsDag("Defs")))
- return false;
- if (!parseMatch(*RuleDef.getValueAsDag("Match")))
- return false;
- if (!parseApply(*RuleDef.getValueAsDag("Apply")))
- return false;
- if (!buildOperandsTable())
- return false;
- if (!findRoots())
- return false;
- LLVM_DEBUG(verify());
- return true;
+InstructionPattern::~InstructionPattern() = default;
+
+void InstructionPattern::reportUnreachable(ArrayRef<SMLoc> Locs) const {
+ PrintError(Locs, "pattern '" + getName() + "' ('" + getInstName() +
+ "') is unreachable from the pattern root!");
}
-bool CombineRuleBuilder::emitRuleMatchers() {
- assert(MatchRoot);
- CodeExpansions CE;
- declareAllMatchDatasExpansions(CE);
+bool InstructionPattern::checkSemantics(ArrayRef<SMLoc> Loc) {
+ unsigned NumExpectedOperands = getNumInstOperands();
- switch (MatchRoot->getKind()) {
- case Pattern::K_AnyOpcode: {
- if (!emitMatchPattern(CE, *cast<AnyOpcodePattern>(MatchRoot)))
- return false;
- break;
- }
- case Pattern::K_Inst:
- if (!emitMatchPattern(CE, *cast<InstructionPattern>(MatchRoot)))
+ if (isVariadic()) {
+ if (Operands.size() < NumExpectedOperands) {
+ PrintError(Loc, +"'" + getInstName() + "' expected at least " +
+ Twine(NumExpectedOperands) + " operands, got " +
+ Twine(Operands.size()));
return false;
- break;
- case Pattern::K_CXX:
- PrintError("C++ code cannot be the root of a pattern!");
+ }
+ } else if (NumExpectedOperands != Operands.size()) {
+ PrintError(Loc, +"'" + getInstName() + "' expected " +
+ Twine(NumExpectedOperands) + " operands, got " +
+ Twine(Operands.size()));
return false;
- default:
- llvm_unreachable("unknown pattern kind!");
}
+ unsigned OpIdx = 0;
+ unsigned NumDefs = getNumInstDefs();
+ for (auto &Op : Operands)
+ Op.setIsDef(OpIdx++ < NumDefs);
+
return true;
}
-void CombineRuleBuilder::print(raw_ostream &OS) const {
- OS << "(CombineRule name:" << RuleDef.getName() << " id:" << RuleID
- << " root:" << RootName << "\n";
+void InstructionPattern::print(raw_ostream &OS, bool PrintName) const {
+ printImpl(OS, PrintName, [&OS, this] {
+ OS << getInstName() << " operands:[";
+ StringRef Sep = "";
+ for (const auto &Op : Operands) {
+ OS << Sep;
+ Op.print(OS);
+ Sep = ", ";
+ }
+ OS << "]";
+ });
+}
- OS << " (MatchDatas ";
- if (MatchDatas.empty())
- OS << "<empty>)\n";
- else {
- OS << "\n";
- for (const auto &MD : MatchDatas) {
- OS << " ";
- MD.print(OS);
- OS << "\n";
+//===- OperandTable -------------------------------------------------------===//
+
+/// Maps InstructionPattern operands to their definitions. This allows us to tie
+///
diff erent patterns of a (apply), (match) or (patterns) set of patterns
+/// together.
+template <typename DefTy = InstructionPattern> class OperandTable {
+public:
+ static_assert(std::is_base_of_v<InstructionPattern, DefTy>,
+ "DefTy should be a derived class from InstructionPattern");
+
+ bool addPattern(DefTy *P, function_ref<void(StringRef)> DiagnoseRedef) {
+ for (const auto &Op : P->named_operands()) {
+ StringRef OpName = Op.getOperandName();
+
+ // We always create an entry in the OperandTable, even for uses.
+ // Uses of operands that don't have a def (= live-ins) will remain with a
+ // nullptr as the Def.
+ //
+ // This allows us tell whether an operand exists in a pattern or not. If
+ // there is no entry for it, it doesn't exist, if there is an entry, it's
+ // used/def'd at least once.
+ auto &Def = Table[OpName];
+
+ if (!Op.isDef())
+ continue;
+
+ if (Def) {
+ DiagnoseRedef(OpName);
+ return false;
+ }
+
+ Def = P;
}
- OS << " )\n";
+
+ return true;
}
- const auto DumpPats = [&](StringRef Name, const PatternMap &Pats) {
- OS << " (" << Name << " ";
- if (Pats.empty()) {
+ struct LookupResult {
+ LookupResult() = default;
+ LookupResult(DefTy *Def) : Found(true), Def(Def) {}
+
+ bool Found = false;
+ DefTy *Def = nullptr;
+
+ bool isLiveIn() const { return Found && !Def; }
+ };
+
+ LookupResult lookup(StringRef OpName) const {
+ if (auto It = Table.find(OpName); It != Table.end())
+ return LookupResult(It->second);
+ return LookupResult();
+ }
+
+ DefTy *getDef(StringRef OpName) const { return lookup(OpName).Def; }
+
+ void print(raw_ostream &OS, StringRef Name = "",
+ StringRef Indent = "") const {
+ OS << Indent << "(OperandTable ";
+ if (!Name.empty())
+ OS << Name << " ";
+ if (Table.empty()) {
OS << "<empty>)\n";
return;
}
- OS << "\n";
- for (const auto &[Name, Pat] : Pats) {
- OS << " ";
- if (Pat.get() == MatchRoot)
- OS << "<root>";
- OS << Name << ":";
- Pat->print(OS, /*PrintName=*/false);
- OS << "\n";
- }
- OS << " )\n";
- };
-
- DumpPats("MatchPats", MatchPats);
- DumpPats("ApplyPats", ApplyPats);
+ SmallVector<StringRef, 0> Keys(Table.keys());
+ sort(Keys);
- OS << " (OperandTable ";
- if (OperandTable.empty())
- OS << "<empty>)\n";
- else {
OS << "\n";
- for (const auto &[Key, Val] : OperandTable) {
- OS << " [" << Key;
- if (const auto *P = Val.MatchPat)
- OS << " match_pat:" << P->getName();
- if (const auto *P = Val.ApplyPat)
- OS << " apply_pat:" << P->getName();
- if (Val.isLiveIn())
- OS << " live-in";
- OS << "]\n";
+ for (const auto &Key : Keys) {
+ const auto *Def = Table.at(Key);
+ OS << Indent << " " << Key << " -> "
+ << (Def ? Def->getName() : "<live-in>") << "\n";
}
- OS << " )\n";
+ OS << Indent << ")\n";
}
- OS << ")\n";
-}
+ auto begin() const { return Table.begin(); }
+ auto end() const { return Table.end(); }
-#ifndef NDEBUG
-void CombineRuleBuilder::verify() const {
- const auto VerifyPats = [&](const PatternMap &Pats) {
- for (const auto &[Name, Pat] : Pats) {
- if (!Pat)
- PrintFatalError("null pattern in pattern map!");
+ void dump() const { print(dbgs()); }
- if (Name != Pat->getName()) {
- Pat->dump();
- PrintFatalError("Pattern name mismatch! Map name: " + Name +
+private:
+ StringMap<DefTy *> Table;
+};
+
+//===- CodeGenInstructionPattern ------------------------------------------===//
+
+/// Matches an instruction, e.g. `G_ADD $x, $y, $z`.
+class CodeGenInstructionPattern : public InstructionPattern {
+public:
+ CodeGenInstructionPattern(const CodeGenInstruction &I, StringRef Name)
+ : InstructionPattern(K_CodeGenInstruction, Name), I(I) {}
+
+ static bool classof(const Pattern *P) {
+ return P->getKind() == K_CodeGenInstruction;
+ }
+
+ bool is(StringRef OpcodeName) const {
+ return I.TheDef->getName() == OpcodeName;
+ }
+
+ bool hasVariadicDefs() const;
+ bool isVariadic() const override { return I.Operands.isVariadic; }
+ unsigned getNumInstDefs() const override;
+ unsigned getNumInstOperands() const override;
+
+ const CodeGenInstruction &getInst() const { return I; }
+ StringRef getInstName() const override { return I.TheDef->getName(); }
+
+private:
+ const CodeGenInstruction &I;
+};
+
+bool CodeGenInstructionPattern::hasVariadicDefs() const {
+ // Note: we cannot use variadicOpsAreDefs, it's not set for
+ // GenericInstructions.
+ if (!isVariadic())
+ return false;
+
+ if (I.variadicOpsAreDefs)
+ return true;
+
+ DagInit *OutOps = I.TheDef->getValueAsDag("OutOperandList");
+ if (OutOps->arg_empty())
+ return false;
+
+ auto *LastArgTy = dyn_cast<DefInit>(OutOps->getArg(OutOps->arg_size() - 1));
+ return LastArgTy && LastArgTy->getDef()->getName() == "variable_ops";
+}
+
+unsigned CodeGenInstructionPattern::getNumInstDefs() const {
+ if (!isVariadic() || !hasVariadicDefs())
+ return I.Operands.NumDefs;
+ unsigned NumOuts = I.Operands.size() - I.Operands.NumDefs;
+ assert(Operands.size() > NumOuts);
+ return std::max<unsigned>(I.Operands.NumDefs, Operands.size() - NumOuts);
+}
+
+unsigned CodeGenInstructionPattern::getNumInstOperands() const {
+ unsigned NumCGIOps = I.Operands.size();
+ return isVariadic() ? std::max<unsigned>(NumCGIOps, Operands.size())
+ : NumCGIOps;
+}
+
+//===- OperandTypeChecker -------------------------------------------------===//
+
+/// This is a trivial type checker for all operands in a set of
+/// InstructionPatterns.
+///
+/// It infers the type of each operand, check it's consistent with the known
+/// type of the operand, and then sets all of the types in all operands in
+/// setAllOperandTypes.
+class OperandTypeChecker {
+public:
+ OperandTypeChecker(ArrayRef<SMLoc> DiagLoc) : DiagLoc(DiagLoc) {}
+
+ bool check(InstructionPattern *P);
+
+ void setAllOperandTypes();
+
+private:
+ struct OpTypeInfo {
+ const Record *Type = nullptr;
+ InstructionPattern *TypeSrc = nullptr;
+ };
+
+ ArrayRef<SMLoc> DiagLoc;
+ StringMap<OpTypeInfo> Types;
+
+ SmallVector<InstructionPattern *, 16> Pats;
+};
+
+bool OperandTypeChecker::check(InstructionPattern *P) {
+ Pats.push_back(P);
+
+ for (auto &Op : P->named_operands()) {
+ const Record *Ty = Op.getType();
+ if (!Ty)
+ continue;
+
+ auto &Info = Types[Op.getOperandName()];
+
+ if (!Info.Type) {
+ Info.Type = Ty;
+ Info.TypeSrc = P;
+ continue;
+ }
+
+ if (Info.Type != Ty) {
+ PrintError(DiagLoc, "conflicting types for operand '" +
+ Op.getOperandName() + "': first seen with '" +
+ Info.Type->getName() + "' in '" +
+ Info.TypeSrc->getName() + ", now seen with '" +
+ Ty->getName() + "' in '" + P->getName() + "'");
+ return false;
+ }
+ }
+
+ return true;
+}
+
+void OperandTypeChecker::setAllOperandTypes() {
+ for (auto *Pat : Pats) {
+ for (auto &Op : Pat->named_operands()) {
+ if (auto &Info = Types[Op.getOperandName()]; Info.Type)
+ Op.setType(Info.Type);
+ }
+ }
+}
+
+//===- PatFrag ------------------------------------------------------------===//
+
+/// Represents a parsed GICombinePatFrag. This can be thought of as the
+/// equivalent of a CodeGenInstruction, but for PatFragPatterns.
+///
+/// PatFrags are made of 3 things:
+/// - Out parameters (defs)
+/// - In parameters
+/// - A set of pattern lists (alternatives).
+///
+/// If the PatFrag uses instruction patterns, the root must be one of the defs.
+///
+/// Note that this DOES NOT represent the use of the PatFrag, only its
+/// definition. The use of the PatFrag in a Pattern is represented by
+/// PatFragPattern.
+///
+/// PatFrags use the term "parameter" instead of operand because they're
+/// essentially macros, and using that name avoids confusion. Other than that,
+/// they're structured similarly to a MachineInstruction - all parameters
+/// (operands) are in the same list, with defs at the start. This helps mapping
+/// parameters to values, because, param N of a PatFrag is always operand N of a
+/// PatFragPattern.
+class PatFrag {
+public:
+ enum ParamKind {
+ PK_Root,
+ PK_MachineOperand,
+ PK_Imm,
+ };
+
+ struct Param {
+ StringRef Name;
+ ParamKind Kind;
+ };
+
+ using ParamVec = SmallVector<Param, 4>;
+ using ParamIt = ParamVec::const_iterator;
+
+ /// Represents an alternative of the PatFrag. When parsing a GICombinePatFrag,
+ /// this is created from its "Alternatives" list. Each alternative is a list
+ /// of patterns written wrapped in a `(pattern ...)` dag init.
+ ///
+ /// Each argument to the `pattern` DAG operator is parsed into a Pattern
+ /// instance.
+ struct Alternative {
+ OperandTable<> OpTable;
+ SmallVector<std::unique_ptr<Pattern>, 4> Pats;
+ };
+
+ PatFrag(const Record &Def) : Def(Def) {}
+
+ static StringRef getParamKindStr(ParamKind OK);
+
+ StringRef getName() const { return Def.getName(); }
+
+ const Record &getDef() const { return Def; }
+ ArrayRef<SMLoc> getLoc() const { return Def.getLoc(); }
+
+ Alternative &addAlternative() { return Alts.emplace_back(); }
+ const Alternative &getAlternative(unsigned K) const { return Alts[K]; }
+ unsigned num_alternatives() const { return Alts.size(); }
+
+ void addInParam(StringRef Name, ParamKind Kind);
+ iterator_range<ParamIt> in_params() const;
+ unsigned num_in_params() const { return Params.size() - NumOutParams; }
+
+ void addOutParam(StringRef Name, ParamKind Kind);
+ iterator_range<ParamIt> out_params() const;
+ unsigned num_out_params() const { return NumOutParams; }
+
+ unsigned num_roots() const;
+ unsigned num_params() const { return num_in_params() + num_out_params(); }
+
+ /// Finds the operand \p Name and returns its index or -1 if not found.
+ /// Remember that all params are part of the same list, with out params at the
+ /// start. This means that the index returned can be used to access operands
+ /// of InstructionPatterns.
+ unsigned getParamIdx(StringRef Name) const;
+ const Param &getParam(unsigned K) const { return Params[K]; }
+
+ bool canBeMatchRoot() const { return num_roots() == 1; }
+
+ void print(raw_ostream &OS, StringRef Indent = "") const;
+ void dump() const { print(dbgs()); }
+
+ /// Checks if the in-param \p ParamName can be unbound or not.
+ /// \p ArgName is the name of the argument passed to the PatFrag.
+ ///
+ /// An argument can be unbound only if, for all alternatives:
+ /// - There is no CXX pattern, OR:
+ /// - There is an InstructionPattern that binds the parameter.
+ ///
+ /// e.g. in (MyPatFrag $foo), if $foo has never been seen before (= it's
+ /// unbound), this checks if MyPatFrag supports it or not.
+ bool handleUnboundInParam(StringRef ParamName, StringRef ArgName,
+ ArrayRef<SMLoc> DiagLoc) const;
+
+ bool checkSemantics();
+ bool buildOperandsTables();
+
+private:
+ static void printParamsList(raw_ostream &OS, iterator_range<ParamIt> Params);
+
+ void PrintError(Twine Msg) const { ::PrintError(&Def, Msg); }
+
+ const Record &Def;
+ unsigned NumOutParams = 0;
+ ParamVec Params;
+ SmallVector<Alternative, 2> Alts;
+};
+
+StringRef PatFrag::getParamKindStr(ParamKind OK) {
+ switch (OK) {
+ case PK_Root:
+ return "root";
+ case PK_MachineOperand:
+ return "machine_operand";
+ case PK_Imm:
+ return "imm";
+ }
+
+ llvm_unreachable("Unknown operand kind!");
+}
+
+void PatFrag::addInParam(StringRef Name, ParamKind Kind) {
+ Params.emplace_back(Param{insertStrRef(Name), Kind});
+}
+
+iterator_range<PatFrag::ParamIt> PatFrag::in_params() const {
+ return {Params.begin() + NumOutParams, Params.end()};
+}
+
+void PatFrag::addOutParam(StringRef Name, ParamKind Kind) {
+ assert(NumOutParams == Params.size() &&
+ "Adding out-param after an in-param!");
+ Params.emplace_back(Param{insertStrRef(Name), Kind});
+ ++NumOutParams;
+}
+
+iterator_range<PatFrag::ParamIt> PatFrag::out_params() const {
+ return {Params.begin(), Params.begin() + NumOutParams};
+}
+
+unsigned PatFrag::num_roots() const {
+ return count_if(out_params(),
+ [&](const auto &P) { return P.Kind == PK_Root; });
+}
+
+unsigned PatFrag::getParamIdx(StringRef Name) const {
+ for (const auto &[Idx, Op] : enumerate(Params)) {
+ if (Op.Name == Name)
+ return Idx;
+ }
+
+ return -1;
+}
+
+bool PatFrag::checkSemantics() {
+ for (const auto &Alt : Alts) {
+ for (const auto &Pat : Alt.Pats) {
+ switch (Pat->getKind()) {
+ case Pattern::K_AnyOpcode:
+ PrintError("wip_match_opcode cannot be used in " + PatFragClassName);
+ return false;
+ case Pattern::K_CXX:
+ case Pattern::K_CodeGenInstruction:
+ continue;
+ case Pattern::K_PatFrag:
+ // TODO: It's just that the emitter doesn't handle it but technically
+ // there is no reason why we can't. We just have to be careful with
+ // operand mappings, it could get complex.
+ PrintError("nested " + PatFragClassName + " are not supported");
+ return false;
+ }
+ }
+ }
+
+ StringSet<> SeenOps;
+ for (const auto &Op : in_params()) {
+ if (SeenOps.count(Op.Name)) {
+ PrintError("duplicate parameter '" + Op.Name + "'");
+ return false;
+ }
+
+ // Check this operand is NOT defined in any alternative's patterns.
+ for (const auto &Alt : Alts) {
+ if (Alt.OpTable.lookup(Op.Name).Def) {
+ PrintError("input parameter '" + Op.Name + "' cannot be redefined!");
+ return false;
+ }
+ }
+
+ if (Op.Kind == PK_Root) {
+ PrintError("input parameterr '" + Op.Name + "' cannot be a root!");
+ return false;
+ }
+
+ SeenOps.insert(Op.Name);
+ }
+
+ for (const auto &Op : out_params()) {
+ if (Op.Kind != PK_Root && Op.Kind != PK_MachineOperand) {
+ PrintError("output parameter '" + Op.Name +
+ "' must be 'root' or 'gi_mo'");
+ return false;
+ }
+
+ if (SeenOps.count(Op.Name)) {
+ PrintError("duplicate parameter '" + Op.Name + "'");
+ return false;
+ }
+
+ // Check this operand is defined in all alternative's patterns.
+ for (const auto &Alt : Alts) {
+ if (!Alt.OpTable.getDef(Op.Name)) {
+ PrintError("output parameter '" + Op.Name +
+ "' must be defined by all alternative patterns in '" +
+ Def.getName() + "'");
+ return false;
+ }
+ }
+
+ SeenOps.insert(Op.Name);
+ }
+
+ if (num_out_params() != 0 && num_roots() == 0) {
+ PrintError(PatFragClassName + " must have one root in its 'out' operands");
+ return false;
+ }
+
+ if (num_roots() > 1) {
+ PrintError(PatFragClassName + " can only have one root");
+ return false;
+ }
+
+ // TODO: find unused params
+
+ // Now, typecheck all alternatives.
+ for (auto &Alt : Alts) {
+ OperandTypeChecker OTC(Def.getLoc());
+ for (auto &Pat : Alt.Pats) {
+ if (auto *IP = dyn_cast<InstructionPattern>(Pat.get())) {
+ if (!OTC.check(IP))
+ return false;
+ }
+ }
+ OTC.setAllOperandTypes();
+ }
+
+ return true;
+}
+
+bool PatFrag::handleUnboundInParam(StringRef ParamName, StringRef ArgName,
+ ArrayRef<SMLoc> DiagLoc) const {
+ // The parameter must be a live-in of all alternatives for this to work.
+ // Otherwise, we risk having unbound parameters being used (= crashes).
+ //
+ // Examples:
+ //
+ // in (ins $y), (patterns (G_FNEG $dst, $y), "return matchFnegOp(${y})")
+ // even if $y is unbound, we'll lazily bind it when emitting the G_FNEG.
+ //
+ // in (ins $y), (patterns "return matchFnegOp(${y})")
+ // if $y is unbound when this fragment is emitted, C++ code expansion will
+ // fail.
+ for (const auto &Alt : Alts) {
+ auto &OT = Alt.OpTable;
+ if (!OT.lookup(ParamName).Found) {
+ ::PrintError(DiagLoc, "operand '" + ArgName + "' (for parameter '" +
+ ParamName + "' of '" + getName() +
+ "') cannot be unbound");
+ PrintNote(
+ DiagLoc,
+ "one or more alternatives of '" + getName() + "' do not bind '" +
+ ParamName +
+ "' to an instruction operand; either use a bound operand or "
+ "ensure '" +
+ Def.getName() + "' binds '" + ParamName +
+ "' in all alternatives");
+ return false;
+ }
+ }
+
+ return true;
+}
+
+bool PatFrag::buildOperandsTables() {
+ // enumerate(...) doesn't seem to allow lvalues so we need to count the old
+ // way.
+ unsigned Idx = 0;
+
+ const auto DiagnoseRedef = [this, &Idx](StringRef OpName) {
+ PrintError("Operand '" + OpName +
+ "' is defined multiple times in patterns of alternative #" +
+ to_string(Idx));
+ };
+
+ for (auto &Alt : Alts) {
+ for (auto &Pat : Alt.Pats) {
+ auto *IP = dyn_cast<InstructionPattern>(Pat.get());
+ if (!IP)
+ continue;
+
+ if (!Alt.OpTable.addPattern(IP, DiagnoseRedef))
+ return false;
+ }
+
+ ++Idx;
+ }
+
+ return true;
+}
+
+void PatFrag::print(raw_ostream &OS, StringRef Indent) const {
+ OS << Indent << "(PatFrag name:" << getName() << "\n";
+ if (!in_params().empty()) {
+ OS << Indent << " (ins ";
+ printParamsList(OS, in_params());
+ OS << ")\n";
+ }
+
+ if (!out_params().empty()) {
+ OS << Indent << " (outs ";
+ printParamsList(OS, out_params());
+ OS << ")\n";
+ }
+
+ // TODO: Dump OperandTable as well.
+ OS << Indent << " (alternatives [\n";
+ for (const auto &Alt : Alts) {
+ OS << Indent << " [\n";
+ for (const auto &Pat : Alt.Pats) {
+ OS << Indent << " ";
+ Pat->print(OS, /*PrintName=*/true);
+ OS << ",\n";
+ }
+ OS << Indent << " ],\n";
+ }
+ OS << Indent << " ])\n";
+
+ OS << Indent << ')';
+}
+
+void PatFrag::printParamsList(raw_ostream &OS, iterator_range<ParamIt> Params) {
+ OS << '['
+ << join(map_range(Params,
+ [](auto &O) {
+ return (O.Name + ":" + getParamKindStr(O.Kind)).str();
+ }),
+ ", ")
+ << ']';
+}
+
+//===- PatFragPattern -----------------------------------------------------===//
+
+class PatFragPattern : public InstructionPattern {
+public:
+ PatFragPattern(const PatFrag &PF, StringRef Name)
+ : InstructionPattern(K_PatFrag, Name), PF(PF) {}
+
+ static bool classof(const Pattern *P) { return P->getKind() == K_PatFrag; }
+
+ const PatFrag &getPatFrag() const { return PF; }
+ StringRef getInstName() const override { return PF.getName(); }
+
+ unsigned getNumInstDefs() const override { return PF.num_out_params(); }
+ unsigned getNumInstOperands() const override { return PF.num_params(); }
+
+ bool checkSemantics(ArrayRef<SMLoc> DiagLoc) override;
+
+ /// Before emitting the patterns inside the PatFrag, add all necessary code
+ /// expansions to \p PatFragCEs imported from \p ParentCEs.
+ ///
+ /// For a MachineOperand PatFrag parameter, this will fetch the expansion for
+ /// that operand from \p ParentCEs and add it to \p PatFragCEs. Errors can be
+ /// emitted if the MachineOperand reference is unbound.
+ ///
+ /// For an Immediate PatFrag parameter this simply adds the integer value to
+ /// \p PatFragCEs as an expansion.
+ ///
+ /// \param ParentCEs Contains all of the code expansions declared by the other
+ /// patterns emitted so far in the pattern list containing
+ /// this PatFragPattern.
+ /// \param PatFragCEs Output Code Expansions (usually empty)
+ /// \param DiagLoc Diagnostic loc in case an error occurs.
+ /// \return `true` on success, `false` on failure.
+ bool mapInputCodeExpansions(const CodeExpansions &ParentCEs,
+ CodeExpansions &PatFragCEs,
+ ArrayRef<SMLoc> DiagLoc) const;
+
+private:
+ const PatFrag &PF;
+};
+
+bool PatFragPattern::checkSemantics(ArrayRef<SMLoc> DiagLoc) {
+ InstructionPattern::checkSemantics(DiagLoc);
+
+ for (const auto &[Idx, Op] : enumerate(Operands)) {
+ switch (PF.getParam(Idx).Kind) {
+ case PatFrag::PK_Imm:
+ if (!Op.hasImmValue()) {
+ PrintError(DiagLoc, "expected operand " + to_string(Idx) + " of '" +
+ getInstName() + "' to be an immediate; got " +
+ Op.describe());
+ return false;
+ }
+ if (Op.isNamedImmediate()) {
+ PrintError(DiagLoc, "operand " + to_string(Idx) + " of '" +
+ getInstName() +
+ "' cannot be a named immediate");
+ return false;
+ }
+ break;
+ case PatFrag::PK_Root:
+ case PatFrag::PK_MachineOperand:
+ if (!Op.isNamedOperand() || Op.isNamedImmediate()) {
+ PrintError(DiagLoc, "expected operand " + to_string(Idx) + " of '" +
+ getInstName() +
+ "' to be a MachineOperand; got " +
+ Op.describe());
+ return false;
+ }
+ break;
+ }
+ }
+
+ return true;
+}
+
+bool PatFragPattern::mapInputCodeExpansions(const CodeExpansions &ParentCEs,
+ CodeExpansions &PatFragCEs,
+ ArrayRef<SMLoc> DiagLoc) const {
+ for (const auto &[Idx, Op] : enumerate(operands())) {
+ StringRef ParamName = PF.getParam(Idx).Name;
+
+ // Operands to a PFP can only be named, or be an immediate, but not a named
+ // immediate.
+ assert(!Op.isNamedImmediate());
+
+ if (Op.isNamedOperand()) {
+ StringRef ArgName = Op.getOperandName();
+ // Map it only if it's been defined.
+ auto It = ParentCEs.find(ArgName);
+ if (It == ParentCEs.end()) {
+ if (!PF.handleUnboundInParam(ParamName, ArgName, DiagLoc))
+ return false;
+ } else
+ PatFragCEs.declare(ParamName, It->second);
+ continue;
+ }
+
+ if (Op.hasImmValue()) {
+ PatFragCEs.declare(ParamName, to_string(Op.getImmValue()));
+ continue;
+ }
+
+ llvm_unreachable("Unknown Operand Type!");
+ }
+
+ return true;
+}
+
+//===- PrettyStackTrace Helpers ------------------------------------------===//
+
+class PrettyStackTraceParse : public PrettyStackTraceEntry {
+ const Record &Def;
+
+public:
+ PrettyStackTraceParse(const Record &Def) : Def(Def) {}
+
+ void print(raw_ostream &OS) const override {
+ if (Def.isSubClassOf("GICombineRule"))
+ OS << "Parsing GICombineRule '" << Def.getName() << "'";
+ else if (Def.isSubClassOf(PatFragClassName))
+ OS << "Parsing " << PatFragClassName << " '" << Def.getName() << "'";
+ else
+ OS << "Parsing '" << Def.getName() << "'";
+ OS << "\n";
+ }
+};
+
+class PrettyStackTraceEmit : public PrettyStackTraceEntry {
+ const Record &Def;
+ const Pattern *Pat = nullptr;
+
+public:
+ PrettyStackTraceEmit(const Record &Def, const Pattern *Pat = nullptr)
+ : Def(Def), Pat(Pat) {}
+
+ void print(raw_ostream &OS) const override {
+ if (Def.isSubClassOf("GICombineRule"))
+ OS << "Emitting GICombineRule '" << Def.getName() << "'";
+ else if (Def.isSubClassOf(PatFragClassName))
+ OS << "Emitting " << PatFragClassName << " '" << Def.getName() << "'";
+ else
+ OS << "Emitting '" << Def.getName() << "'";
+
+ if (Pat)
+ OS << " [" << Pat->getKindName() << " '" << Pat->getName() << "']";
+ OS << "\n";
+ }
+};
+
+//===- CombineRuleBuilder -------------------------------------------------===//
+
+/// Parses combine rule and builds a small intermediate representation to tie
+/// patterns together and emit RuleMatchers to match them. This may emit more
+/// than one RuleMatcher, e.g. for `wip_match_opcode`.
+///
+/// Memory management for `Pattern` objects is done through `std::unique_ptr`.
+/// In most cases, there are two stages to a pattern's lifetime:
+/// - Creation in a `parse` function
+/// - The unique_ptr is stored in a variable, and may be destroyed if the
+/// pattern is found to be semantically invalid.
+/// - Ownership transfer into a `PatternMap`
+/// - Once a pattern is moved into either the map of Match or Apply
+/// patterns, it is known to be valid and it never moves back.
+class CombineRuleBuilder {
+public:
+ using PatternMap = MapVector<StringRef, std::unique_ptr<Pattern>>;
+ using PatternAlternatives = DenseMap<const Pattern *, unsigned>;
+
+ CombineRuleBuilder(const CodeGenTarget &CGT,
+ SubtargetFeatureInfoMap &SubtargetFeatures,
+ Record &RuleDef, unsigned ID,
+ std::vector<RuleMatcher> &OutRMs)
+ : CGT(CGT), SubtargetFeatures(SubtargetFeatures), RuleDef(RuleDef),
+ RuleID(ID), OutRMs(OutRMs) {}
+
+ /// Parses all fields in the RuleDef record.
+ bool parseAll();
+
+ /// Emits all RuleMatchers into the vector of RuleMatchers passed in the
+ /// constructor.
+ bool emitRuleMatchers();
+
+ void print(raw_ostream &OS) const;
+ void dump() const { print(dbgs()); }
+
+ /// Debug-only verification of invariants.
+#ifndef NDEBUG
+ void verify() const;
+#endif
+
+private:
+ const CodeGenInstruction &getGConstant() const {
+ return CGT.getInstruction(RuleDef.getRecords().getDef("G_CONSTANT"));
+ }
+
+ void PrintError(Twine Msg) const { ::PrintError(&RuleDef, Msg); }
+ void PrintWarning(Twine Msg) const { ::PrintWarning(RuleDef.getLoc(), Msg); }
+ void PrintNote(Twine Msg) const { ::PrintNote(RuleDef.getLoc(), Msg); }
+
+ void print(raw_ostream &OS, const PatternAlternatives &Alts) const;
+
+ bool addApplyPattern(std::unique_ptr<Pattern> Pat);
+ bool addMatchPattern(std::unique_ptr<Pattern> Pat);
+
+ /// Adds the expansions from \see MatchDatas to \p CE.
+ void declareAllMatchDatasExpansions(CodeExpansions &CE) const;
+
+ /// Adds a matcher \p P to \p IM, expanding its code using \p CE.
+ /// Note that the predicate is added on the last InstructionMatcher.
+ ///
+ /// \p Alts is only used if DebugCXXPreds is enabled.
+ void addCXXPredicate(RuleMatcher &M, const CodeExpansions &CE,
+ const CXXPattern &P, const PatternAlternatives &Alts);
+
+ /// Adds an apply \p P to \p IM, expanding its code using \p CE.
+ void addCXXAction(RuleMatcher &M, const CodeExpansions &CE,
+ const CXXPattern &P);
+
+ bool hasOnlyCXXApplyPatterns() const;
+ bool hasAnyOpcodeMatchPattern() const {
+ return any_of(MatchPats, [&](const auto &E) {
+ return isa<AnyOpcodePattern>(E.second.get());
+ });
+ }
+
+ // Infer machine operand types and check their consistency.
+ bool typecheckPatterns();
+
+ /// For all PatFragPatterns, add a new entry in PatternAlternatives for each
+ /// PatternList it contains. This is multiplicative, so if we have 2
+ /// PatFrags with 3 alternatives each, we get 2*3 permutations added to
+ /// PermutationsToEmit. The "MaxPermutations" field controls how many
+ /// permutations are allowed before an error is emitted and this function
+ /// returns false. This is a simple safeguard to prevent combination of
+ /// PatFrags from generating enormous amounts of rules.
+ bool buildPermutationsToEmit();
+
+ /// Creates a new RuleMatcher with some boilerplate
+ /// settings/actions/predicates, and and adds it to \p OutRMs.
+ /// \see addFeaturePredicates too.
+ ///
+ /// \param Alts Current set of alternatives, for debug comment.
+ /// \param AdditionalComment Comment string to be added to the
+ /// `DebugCommentAction`.
+ RuleMatcher &addRuleMatcher(const PatternAlternatives &Alts,
+ Twine AdditionalComment = "");
+ bool addFeaturePredicates(RuleMatcher &M);
+
+ bool findRoots();
+ bool buildRuleOperandsTable();
+
+ bool parseDefs(const DagInit &Def);
+ bool
+ parsePatternList(const DagInit &List,
+ function_ref<bool(std::unique_ptr<Pattern>)> ParseAction,
+ StringRef Operator, ArrayRef<SMLoc> DiagLoc,
+ StringRef AnonPatNamePrefix) const;
+
+ std::unique_ptr<Pattern> parseInstructionPattern(const Init &Arg,
+ StringRef PatName) const;
+ std::unique_ptr<Pattern> parseWipMatchOpcodeMatcher(const Init &Arg,
+ StringRef PatName) const;
+ bool parseInstructionPatternOperand(InstructionPattern &IP,
+ const Init *OpInit,
+ const StringInit *OpName) const;
+ std::unique_ptr<PatFrag> parsePatFragImpl(const Record *Def) const;
+ bool parsePatFragParamList(
+ ArrayRef<SMLoc> DiagLoc, const DagInit &OpsList,
+ function_ref<bool(StringRef, PatFrag::ParamKind)> ParseAction) const;
+ const PatFrag *parsePatFrag(const Record *Def) const;
+
+ bool emitMatchPattern(CodeExpansions &CE, const PatternAlternatives &Alts,
+ const InstructionPattern &IP);
+ bool emitMatchPattern(CodeExpansions &CE, const PatternAlternatives &Alts,
+ const AnyOpcodePattern &AOP);
+
+ bool emitPatFragMatchPattern(CodeExpansions &CE,
+ const PatternAlternatives &Alts, RuleMatcher &RM,
+ InstructionMatcher *IM,
+ const PatFragPattern &PFP,
+ DenseSet<const Pattern *> &SeenPats);
+
+ bool emitApplyPatterns(CodeExpansions &CE, RuleMatcher &M);
+
+ // Recursively visits CodeGenInstructionPattern from P to build up the
+ // RuleMatcher actions.
+ bool
+ emitCodeGenInstructionApplyPattern(CodeExpansions &CE, RuleMatcher &M,
+ const CodeGenInstructionPattern &P,
+ DenseSet<const Pattern *> &SeenPats,
+ StringMap<unsigned> &OperandToTempRegID);
+
+ bool
+ emitCodeGenInstructionApplyImmOperand(RuleMatcher &M, BuildMIAction &DstMI,
+ const CodeGenInstructionPattern &P,
+ const InstructionPattern::Operand &O);
+
+ // Recursively visits CodeGenInstructionPattern from P to build up the
+ // RuleMatcher/InstructionMatcher. May create new InstructionMatchers as
+ // needed.
+ using OperandMapperFnRef = function_ref<InstructionPattern::Operand(
+ const InstructionPattern::Operand &)>;
+ using OperandDefLookupFn =
+ function_ref<const InstructionPattern *(StringRef)>;
+ bool emitCodeGenInstructionMatchPattern(
+ CodeExpansions &CE, const PatternAlternatives &Alts, RuleMatcher &M,
+ InstructionMatcher &IM, const CodeGenInstructionPattern &P,
+ DenseSet<const Pattern *> &SeenPats, OperandDefLookupFn LookupOperandDef,
+ OperandMapperFnRef OperandMapper = [](const auto &O) { return O; });
+
+ const CodeGenTarget &CGT;
+ SubtargetFeatureInfoMap &SubtargetFeatures;
+ Record &RuleDef;
+ const unsigned RuleID;
+ std::vector<RuleMatcher> &OutRMs;
+
+ // For InstructionMatcher::addOperand
+ unsigned AllocatedTemporariesBaseID = 0;
+
+ /// The root of the pattern.
+ StringRef RootName;
+
+ /// These maps have ownership of the actual Pattern objects.
+ /// They both map a Pattern's name to the Pattern instance.
+ PatternMap MatchPats;
+ PatternMap ApplyPats;
+
+ /// Operand tables to tie match/apply patterns together.
+ OperandTable<> MatchOpTable;
+ OperandTable<CodeGenInstructionPattern> ApplyOpTable;
+
+ /// Set by findRoots.
+ Pattern *MatchRoot = nullptr;
+ SmallDenseSet<InstructionPattern *, 2> ApplyRoots;
+
+ SmallVector<MatchDataInfo, 2> MatchDatas;
+ SmallVector<PatternAlternatives, 1> PermutationsToEmit;
+
+ // print()/debug-only members.
+ mutable SmallPtrSet<const PatFrag *, 2> SeenPatFrags;
+};
+
+bool CombineRuleBuilder::parseAll() {
+ auto StackTrace = PrettyStackTraceParse(RuleDef);
+
+ if (!parseDefs(*RuleDef.getValueAsDag("Defs")))
+ return false;
+
+ if (!parsePatternList(
+ *RuleDef.getValueAsDag("Match"),
+ [this](auto Pat) { return addMatchPattern(std::move(Pat)); }, "match",
+ RuleDef.getLoc(), (RuleDef.getName() + "_match").str()))
+ return false;
+
+ if (!parsePatternList(
+ *RuleDef.getValueAsDag("Apply"),
+ [this](auto Pat) { return addApplyPattern(std::move(Pat)); }, "apply",
+ RuleDef.getLoc(), (RuleDef.getName() + "_apply").str()))
+ return false;
+
+ if (!buildRuleOperandsTable() || !typecheckPatterns() || !findRoots() ||
+ !buildPermutationsToEmit())
+ return false;
+ LLVM_DEBUG(verify());
+ return true;
+}
+
+bool CombineRuleBuilder::emitRuleMatchers() {
+ auto StackTrace = PrettyStackTraceEmit(RuleDef);
+
+ assert(MatchRoot);
+ CodeExpansions CE;
+ declareAllMatchDatasExpansions(CE);
+
+ assert(!PermutationsToEmit.empty());
+ for (const auto &Alts : PermutationsToEmit) {
+ switch (MatchRoot->getKind()) {
+ case Pattern::K_AnyOpcode: {
+ if (!emitMatchPattern(CE, Alts, *cast<AnyOpcodePattern>(MatchRoot)))
+ return false;
+ break;
+ }
+ case Pattern::K_PatFrag:
+ case Pattern::K_CodeGenInstruction:
+ if (!emitMatchPattern(CE, Alts, *cast<InstructionPattern>(MatchRoot)))
+ return false;
+ break;
+ case Pattern::K_CXX:
+ PrintError("C++ code cannot be the root of a rule!");
+ return false;
+ default:
+ llvm_unreachable("unknown pattern kind!");
+ }
+ }
+
+ return true;
+}
+
+void CombineRuleBuilder::print(raw_ostream &OS) const {
+ OS << "(CombineRule name:" << RuleDef.getName() << " id:" << RuleID
+ << " root:" << RootName << "\n";
+
+ if (!MatchDatas.empty()) {
+ OS << " (MatchDatas\n";
+ for (const auto &MD : MatchDatas) {
+ OS << " ";
+ MD.print(OS);
+ OS << "\n";
+ }
+ OS << " )\n";
+ }
+
+ if (!SeenPatFrags.empty()) {
+ OS << " (PatFrags\n";
+ for (const auto *PF : SeenPatFrags) {
+ PF->print(OS, /*Indent=*/" ");
+ OS << "\n";
+ }
+ OS << " )\n";
+ }
+
+ const auto DumpPats = [&](StringRef Name, const PatternMap &Pats) {
+ OS << " (" << Name << " ";
+ if (Pats.empty()) {
+ OS << "<empty>)\n";
+ return;
+ }
+
+ OS << "\n";
+ for (const auto &[Name, Pat] : Pats) {
+ OS << " ";
+ if (Pat.get() == MatchRoot)
+ OS << "<match_root>";
+ if (isa<InstructionPattern>(Pat.get()) &&
+ ApplyRoots.contains(cast<InstructionPattern>(Pat.get())))
+ OS << "<apply_root>";
+ OS << Name << ":";
+ Pat->print(OS, /*PrintName=*/false);
+ OS << "\n";
+ }
+ OS << " )\n";
+ };
+
+ DumpPats("MatchPats", MatchPats);
+ DumpPats("ApplyPats", ApplyPats);
+
+ MatchOpTable.print(OS, "MatchPats", /*Indent*/ " ");
+ ApplyOpTable.print(OS, "ApplyPats", /*Indent*/ " ");
+
+ if (PermutationsToEmit.size() > 1) {
+ OS << " (PermutationsToEmit\n";
+ for (const auto &Perm : PermutationsToEmit) {
+ OS << " ";
+ print(OS, Perm);
+ OS << ",\n";
+ }
+ OS << " )\n";
+ }
+
+ OS << ")\n";
+}
+
+#ifndef NDEBUG
+void CombineRuleBuilder::verify() const {
+ const auto VerifyPats = [&](const PatternMap &Pats) {
+ for (const auto &[Name, Pat] : Pats) {
+ if (!Pat)
+ PrintFatalError("null pattern in pattern map!");
+
+ if (Name != Pat->getName()) {
+ Pat->dump();
+ PrintFatalError("Pattern name mismatch! Map name: " + Name +
", Pat name: " + Pat->getName());
}
- // As an optimization, the PatternMaps don't re-allocate the PatternName
- // string. They simply reference the std::string inside Pattern. Ensure
- // this is the case to avoid memory issues.
+ // Sanity check: the map should point to the same data as the Pattern.
+ // Both strings are allocated in the pool using insertStrRef.
if (Name.data() != Pat->getName().data()) {
dbgs() << "Map StringRef: '" << Name << "' @ "
<< (const void *)Name.data() << "\n";
@@ -701,41 +1715,91 @@ void CombineRuleBuilder::verify() const {
VerifyPats(MatchPats);
VerifyPats(ApplyPats);
- for (const auto &[Name, Op] : OperandTable) {
- if (Op.ApplyPat && !Op.MatchPat) {
- dump();
- PrintFatalError("Operand " + Name +
- " has an apply pattern, but no match pattern!");
- }
+ // Check there are no wip_match_opcode patterns in the "apply" patterns.
+ if (any_of(ApplyPats,
+ [&](auto &E) { return isa<AnyOpcodePattern>(E.second.get()); })) {
+ dump();
+ PrintFatalError(
+ "illegal wip_match_opcode pattern in the 'apply' patterns!");
+ }
+
+ // Check there are no nullptrs in ApplyRoots.
+ if (ApplyRoots.contains(nullptr)) {
+ PrintFatalError(
+ "CombineRuleBuilder's ApplyRoots set contains a null pointer!");
}
}
#endif
-bool CombineRuleBuilder::addFeaturePredicates(RuleMatcher &M) {
- if (!RuleDef.getValue("Predicates"))
- return true;
+void CombineRuleBuilder::print(raw_ostream &OS,
+ const PatternAlternatives &Alts) const {
+ SmallVector<std::string, 1> Strings(
+ map_range(Alts, [](const auto &PatAndPerm) {
+ return PatAndPerm.first->getName().str() + "[" +
+ to_string(PatAndPerm.second) + "]";
+ }));
+ // Sort so output is deterministic for tests. Otherwise it's sorted by pointer
+ // values.
+ sort(Strings);
+ OS << "[" << join(Strings, ", ") << "]";
+}
- ListInit *Preds = RuleDef.getValueAsListInit("Predicates");
- for (Init *I : Preds->getValues()) {
- if (DefInit *Pred = dyn_cast<DefInit>(I)) {
- Record *Def = Pred->getDef();
- if (!Def->isSubClassOf("Predicate")) {
- ::PrintError(Def->getLoc(), "Unknown 'Predicate' Type");
- return false;
- }
+bool CombineRuleBuilder::addApplyPattern(std::unique_ptr<Pattern> Pat) {
+ StringRef Name = Pat->getName();
+ if (ApplyPats.contains(Name)) {
+ PrintError("'" + Name + "' apply pattern defined more than once!");
+ return false;
+ }
- if (Def->getValueAsString("CondString").empty())
- continue;
+ if (isa<AnyOpcodePattern>(Pat.get())) {
+ PrintError("'" + Name +
+ "': wip_match_opcode is not supported in apply patterns");
+ return false;
+ }
- if (SubtargetFeatures.count(Def) == 0) {
- SubtargetFeatures.emplace(
- Def, SubtargetFeatureInfo(Def, SubtargetFeatures.size()));
- }
+ if (isa<InstructionPattern>(Pat.get())) {
+ if (hasAnyOpcodeMatchPattern()) {
+ PrintError("cannot use wip_match_opcode in combination with apply "
+ "instruction patterns!");
+ return false;
+ }
+
+ if (isa<PatFragPattern>(Pat.get())) {
+ PrintError("'" + Name + "': using " + PatFragClassName +
+ " is not supported in apply patterns");
+ return false;
+ }
+ }
+
+ if (auto *CXXPat = dyn_cast<CXXPattern>(Pat.get()))
+ CXXPat->setIsApply();
+
+ ApplyPats[Name] = std::move(Pat);
+ return true;
+}
+
+bool CombineRuleBuilder::addMatchPattern(std::unique_ptr<Pattern> Pat) {
+ StringRef Name = Pat->getName();
+ if (MatchPats.contains(Name)) {
+ PrintError("'" + Name + "' match pattern defined more than once!");
+ return false;
+ }
+
+ // TODO: Could move this to some "checkSemantics" method?
+ if (isa<AnyOpcodePattern>(Pat.get())) {
+ if (hasAnyOpcodeMatchPattern()) {
+ PrintError("wip_opcode_match can only be present once");
+ return false;
+ }
+ }
- M.addRequiredFeature(Def);
+ if (const auto *CXXPat = dyn_cast<CXXPattern>(Pat.get())) {
+ if (!CXXPat->getRawCode().contains("return ")) {
+ PrintWarning("'match' C++ code does not seem to return!");
}
}
+ MatchPats[Name] = std::move(Pat);
return true;
}
@@ -745,109 +1809,273 @@ void CombineRuleBuilder::declareAllMatchDatasExpansions(
CE.declare(MD.getPatternSymbol(), MD.getQualifiedVariableName());
}
-void CombineRuleBuilder::addCXXPredicate(InstructionMatcher &IM,
+void CombineRuleBuilder::addCXXPredicate(RuleMatcher &M,
const CodeExpansions &CE,
- const CXXPattern &P) {
- const auto &ExpandedCode = P.expandCode(CE, RuleDef.getLoc());
- IM.addPredicate<GenericInstructionPredicateMatcher>(
+ const CXXPattern &P,
+ const PatternAlternatives &Alts) {
+ // FIXME: Hack so C++ code is executed last. May not work for more complex
+ // patterns.
+ auto &IM = *std::prev(M.insnmatchers().end());
+ const auto &ExpandedCode =
+ P.expandCode(CE, RuleDef.getLoc(), [&](raw_ostream &OS) {
+ OS << "// Pattern Alternatives: ";
+ print(OS, Alts);
+ OS << "\n";
+ });
+ IM->addPredicate<GenericInstructionPredicateMatcher>(
ExpandedCode.getEnumNameWithPrefix(CXXPredPrefix));
}
-std::string CombineRuleBuilder::makeAnonPatName(StringRef Prefix) const {
- return to_string("__anon_pat_" + Prefix + "_" + to_string(RuleID) + "_" +
- to_string(AnonIDCnt++));
+void CombineRuleBuilder::addCXXAction(RuleMatcher &M, const CodeExpansions &CE,
+ const CXXPattern &P) {
+ const auto &ExpandedCode = P.expandCode(CE, RuleDef.getLoc());
+ M.addAction<CustomCXXAction>(
+ ExpandedCode.getEnumNameWithPrefix(CXXApplyPrefix));
+}
+
+bool CombineRuleBuilder::hasOnlyCXXApplyPatterns() const {
+ return all_of(ApplyPats, [&](auto &Entry) {
+ return isa<CXXPattern>(Entry.second.get());
+ });
+}
+
+bool CombineRuleBuilder::typecheckPatterns() {
+ OperandTypeChecker OTC(RuleDef.getLoc());
+
+ for (auto &Pat : values(MatchPats)) {
+ if (auto *IP = dyn_cast<InstructionPattern>(Pat.get())) {
+ if (!OTC.check(IP))
+ return false;
+ }
+ }
+
+ for (auto &Pat : values(ApplyPats)) {
+ if (auto *IP = dyn_cast<InstructionPattern>(Pat.get())) {
+ if (!OTC.check(IP))
+ return false;
+ }
+ }
+
+ OTC.setAllOperandTypes();
+ return true;
+}
+
+bool CombineRuleBuilder::buildPermutationsToEmit() {
+ PermutationsToEmit.clear();
+
+ // Start with one empty set of alternatives.
+ PermutationsToEmit.emplace_back();
+ for (const auto &Pat : values(MatchPats)) {
+ unsigned NumAlts = 0;
+ // Note: technically, AnyOpcodePattern also needs permutations, but:
+ // - We only allow a single one of them in the root.
+ // - They cannot be mixed with any other pattern other than C++ code.
+ // So we don't really need to take them into account here. We could, but
+ // that pattern is a hack anyway and the less it's involved, the better.
+ if (const auto *PFP = dyn_cast<PatFragPattern>(Pat.get()))
+ NumAlts = PFP->getPatFrag().num_alternatives();
+ else
+ continue;
+
+ // For each pattern that needs permutations, multiply the current set of
+ // alternatives.
+ auto CurPerms = PermutationsToEmit;
+ PermutationsToEmit.clear();
+
+ for (const auto &Perm : CurPerms) {
+ assert(!Perm.count(Pat.get()) && "Pattern already emitted?");
+ for (unsigned K = 0; K < NumAlts; ++K) {
+ PatternAlternatives NewPerm = Perm;
+ NewPerm[Pat.get()] = K;
+ PermutationsToEmit.emplace_back(std::move(NewPerm));
+ }
+ }
+ }
+
+ if (int64_t MaxPerms = RuleDef.getValueAsInt("MaxPermutations");
+ MaxPerms > 0) {
+ if ((int64_t)PermutationsToEmit.size() > MaxPerms) {
+ PrintError("cannot emit rule '" + RuleDef.getName() + "'; " +
+ Twine(PermutationsToEmit.size()) +
+ " permutations would be emitted, but the max is " +
+ Twine(MaxPerms));
+ return false;
+ }
+ }
+
+ // Ensure we always have a single empty entry, it simplifies the emission
+ // logic so it doesn't need to handle the case where there are no perms.
+ if (PermutationsToEmit.empty()) {
+ PermutationsToEmit.emplace_back();
+ return true;
+ }
+
+ return true;
}
-RuleMatcher &CombineRuleBuilder::addRuleMatcher(Twine AdditionalComment) {
+RuleMatcher &CombineRuleBuilder::addRuleMatcher(const PatternAlternatives &Alts,
+ Twine AdditionalComment) {
auto &RM = OutRMs.emplace_back(RuleDef.getLoc());
addFeaturePredicates(RM);
+ RM.setPermanentGISelFlags(GISF_IgnoreCopies);
RM.addRequiredSimplePredicate(getIsEnabledPredicateEnumName(RuleID));
- const std::string AdditionalCommentStr = AdditionalComment.str();
- RM.addAction<DebugCommentAction>(
- "Combiner Rule #" + to_string(RuleID) + ": " + RuleDef.getName().str() +
- (AdditionalCommentStr.empty() ? "" : "; " + AdditionalCommentStr));
+
+ std::string Comment;
+ raw_string_ostream CommentOS(Comment);
+ CommentOS << "Combiner Rule #" << RuleID << ": " << RuleDef.getName();
+ if (!Alts.empty()) {
+ CommentOS << " @ ";
+ print(CommentOS, Alts);
+ }
+ if (!AdditionalComment.isTriviallyEmpty())
+ CommentOS << "; " << AdditionalComment;
+ RM.addAction<DebugCommentAction>(Comment);
return RM;
}
+bool CombineRuleBuilder::addFeaturePredicates(RuleMatcher &M) {
+ if (!RuleDef.getValue("Predicates"))
+ return true;
+
+ ListInit *Preds = RuleDef.getValueAsListInit("Predicates");
+ for (Init *PI : Preds->getValues()) {
+ DefInit *Pred = dyn_cast<DefInit>(PI);
+ if (!Pred)
+ continue;
+
+ Record *Def = Pred->getDef();
+ if (!Def->isSubClassOf("Predicate")) {
+ ::PrintError(Def, "Unknown 'Predicate' Type");
+ return false;
+ }
+
+ if (Def->getValueAsString("CondString").empty())
+ continue;
+
+ if (SubtargetFeatures.count(Def) == 0) {
+ SubtargetFeatures.emplace(
+ Def, SubtargetFeatureInfo(Def, SubtargetFeatures.size()));
+ }
+
+ M.addRequiredFeature(Def);
+ }
+
+ return true;
+}
+
bool CombineRuleBuilder::findRoots() {
+ const auto Finish = [&]() {
+ assert(MatchRoot);
+
+ if (hasOnlyCXXApplyPatterns())
+ return true;
+
+ auto *IPRoot = dyn_cast<InstructionPattern>(MatchRoot);
+ if (!IPRoot)
+ return true;
+
+ if (IPRoot->getNumInstDefs() == 0) {
+ // No defs to work with -> find the root using the pattern name.
+ auto It = ApplyPats.find(RootName);
+ if (It == ApplyPats.end()) {
+ PrintError("Cannot find root '" + RootName + "' in apply patterns!");
+ return false;
+ }
+
+ auto *ApplyRoot = dyn_cast<InstructionPattern>(It->second.get());
+ if (!ApplyRoot) {
+ PrintError("apply pattern root '" + RootName +
+ "' must be an instruction pattern");
+ return false;
+ }
+
+ ApplyRoots.insert(ApplyRoot);
+ return true;
+ }
+
+ // Collect all redefinitions of the MatchRoot's defs and put them in
+ // ApplyRoots.
+ for (unsigned K = 0; K < IPRoot->getNumInstDefs(); ++K) {
+ auto &O = IPRoot->getOperand(K);
+ assert(O.isDef() && O.isNamedOperand());
+ StringRef Name = O.getOperandName();
+
+ auto *ApplyRedef = ApplyOpTable.getDef(Name);
+ if (!ApplyRedef) {
+ PrintError("def of pattern root '" + Name +
+ "' is not redefined in the apply pattern!");
+ PrintNote("match pattern root is '" + MatchRoot->getName() + "'");
+ return false;
+ }
+
+ ApplyRoots.insert((InstructionPattern *)ApplyRedef);
+ }
+
+ if (auto It = ApplyPats.find(RootName); It != ApplyPats.end()) {
+ if (find(ApplyRoots, It->second.get()) == ApplyRoots.end()) {
+ PrintError("apply pattern '" + RootName +
+ "' is supposed to be a root but it does not redefine any of "
+ "the defs of the match root");
+ return false;
+ }
+ }
+
+ return true;
+ };
+
// Look by pattern name, e.g.
// (G_FNEG $x, $y):$root
- if (auto It = MatchPats.find(RootName); It != MatchPats.end()) {
- MatchRoot = It->second.get();
- return true;
+ if (auto MatchPatIt = MatchPats.find(RootName);
+ MatchPatIt != MatchPats.end()) {
+ MatchRoot = MatchPatIt->second.get();
+ return Finish();
}
// Look by def:
// (G_FNEG $root, $y)
- auto It = OperandTable.find(RootName);
- if (It == OperandTable.end()) {
+ auto LookupRes = MatchOpTable.lookup(RootName);
+ if (!LookupRes.Found) {
PrintError("Cannot find root '" + RootName + "' in match patterns!");
return false;
}
- if (!It->second.MatchPat) {
+ MatchRoot = LookupRes.Def;
+ if (!MatchRoot) {
PrintError("Cannot use live-in operand '" + RootName +
"' as match pattern root!");
return false;
}
- MatchRoot = It->second.MatchPat;
- return true;
+ return Finish();
}
-bool CombineRuleBuilder::buildOperandsTable() {
- // Walk each instruction pattern
- for (auto &P : values(MatchPats)) {
- auto *IP = dyn_cast<InstructionPattern>(P.get());
- if (!IP)
- continue;
- for (const auto &Operand : IP->operands()) {
- // Create an entry, no matter if it's a use or a def.
- auto &Entry = OperandTable[Operand.Name];
-
- // We only need to do additional checking on defs, though.
- if (!Operand.IsDef)
- continue;
-
- if (Entry.MatchPat) {
- PrintError("Operand '" + Operand.Name +
- "' is defined multiple times in the 'match' patterns");
- return false;
- }
- Entry.MatchPat = IP;
- }
- }
+bool CombineRuleBuilder::buildRuleOperandsTable() {
+ const auto DiagnoseRedefMatch = [&](StringRef OpName) {
+ PrintError("Operand '" + OpName +
+ "' is defined multiple times in the 'match' patterns");
+ };
- for (auto &P : values(ApplyPats)) {
- auto *IP = dyn_cast<InstructionPattern>(P.get());
- if (!IP)
- continue;
- for (const auto &Operand : IP->operands()) {
- // Create an entry, no matter if it's a use or a def.
- auto &Entry = OperandTable[Operand.Name];
+ const auto DiagnoseRedefApply = [&](StringRef OpName) {
+ PrintError("Operand '" + OpName +
+ "' is defined multiple times in the 'apply' patterns");
+ };
- // We only need to do additional checking on defs, though.
- if (!Operand.IsDef)
- continue;
+ for (auto &Pat : values(MatchPats)) {
+ auto *IP = dyn_cast<InstructionPattern>(Pat.get());
+ if (IP && !MatchOpTable.addPattern(IP, DiagnoseRedefMatch))
+ return false;
+ }
- if (!Entry.MatchPat) {
- PrintError("Cannot define live-in operand '" + Operand.Name +
- "' in the 'apply' pattern");
- return false;
- }
- if (Entry.ApplyPat) {
- PrintError("Operand '" + Operand.Name +
- "' is defined multiple times in the 'apply' patterns");
- return false;
- }
- Entry.ApplyPat = IP;
- }
+ for (auto &Pat : values(ApplyPats)) {
+ auto *IP = dyn_cast<CodeGenInstructionPattern>(Pat.get());
+ if (IP && !ApplyOpTable.addPattern(IP, DiagnoseRedefApply))
+ return false;
}
return true;
}
-bool CombineRuleBuilder::parseDefs(DagInit &Def) {
+bool CombineRuleBuilder::parseDefs(const DagInit &Def) {
if (Def.getOperatorAsDef(RuleDef.getLoc())->getName() != "defs") {
PrintError("Expected defs operator");
return false;
@@ -893,98 +2121,84 @@ bool CombineRuleBuilder::parseDefs(DagInit &Def) {
return true;
}
-bool CombineRuleBuilder::parseMatch(DagInit &Match) {
- if (Match.getOperatorAsDef(RuleDef.getLoc())->getName() != "match") {
- PrintError("Expected match operator");
+bool CombineRuleBuilder::parsePatternList(
+ const DagInit &List,
+ function_ref<bool(std::unique_ptr<Pattern>)> ParseAction,
+ StringRef Operator, ArrayRef<SMLoc> DiagLoc,
+ StringRef AnonPatNamePrefix) const {
+ if (List.getOperatorAsDef(RuleDef.getLoc())->getName() != Operator) {
+ ::PrintError(DiagLoc, "Expected " + Operator + " operator");
return false;
}
- if (Match.getNumArgs() == 0) {
- PrintError("Matcher is empty");
+ if (List.getNumArgs() == 0) {
+ ::PrintError(DiagLoc, Operator + " pattern list is empty");
return false;
}
// The match section consists of a list of matchers and predicates. Parse each
// one and add the equivalent GIMatchDag nodes, predicates, and edges.
- bool HasOpcodeMatcher = false;
- for (unsigned I = 0; I < Match.getNumArgs(); ++I) {
- Init *Arg = Match.getArg(I);
- std::string Name = Match.getArgName(I)
- ? Match.getArgName(I)->getValue().str()
- : makeAnonPatName("match");
-
- if (MatchPats.contains(Name)) {
- PrintError("'" + Name + "' match pattern defined more than once!");
- return false;
- }
-
- if (auto Pat = parseInstructionMatcher(*Arg, Name)) {
- MatchPats[Pat->getName()] = std::move(Pat);
+ for (unsigned I = 0; I < List.getNumArgs(); ++I) {
+ Init *Arg = List.getArg(I);
+ std::string Name = List.getArgName(I)
+ ? List.getArgName(I)->getValue().str()
+ : makeAnonPatName(AnonPatNamePrefix, I);
+
+ if (auto Pat = parseInstructionPattern(*Arg, Name)) {
+ if (!ParseAction(std::move(Pat)))
+ return false;
continue;
}
if (auto Pat = parseWipMatchOpcodeMatcher(*Arg, Name)) {
- if (HasOpcodeMatcher) {
- PrintError("wip_opcode_match can only be present once");
+ if (!ParseAction(std::move(Pat)))
return false;
- }
- HasOpcodeMatcher = true;
- MatchPats[Pat->getName()] = std::move(Pat);
continue;
}
// Parse arbitrary C++ code
if (const auto *StringI = dyn_cast<StringInit>(Arg)) {
- auto CXXPat =
- std::make_unique<CXXPattern>(*StringI, Name, /*IsApply*/ false);
- if (!CXXPat->getRawCode().contains("return ")) {
- PrintWarning(RuleDef.getLoc(),
- "'match' C++ code does not seem to return!");
- }
- MatchPats[CXXPat->getName()] = std::move(CXXPat);
+ auto CXXPat = std::make_unique<CXXPattern>(*StringI, Name);
+ if (!ParseAction(std::move(CXXPat)))
+ return false;
continue;
}
- // TODO: don't print this on, e.g. bad operand count in inst pat
- PrintError("Expected a subclass of GIMatchKind or a sub-dag whose "
- "operator is either of a GIMatchKindWithArgs or Instruction");
- PrintNote("Pattern was `" + Arg->getAsString() + "'");
- return false;
- }
-
- return true;
-}
-
-bool CombineRuleBuilder::parseApply(DagInit &Apply) {
- // Currently we only support C++ :(
- if (Apply.getOperatorAsDef(RuleDef.getLoc())->getName() != "apply") {
- PrintError("Expected 'apply' operator in Apply DAG");
- return false;
- }
-
- if (Apply.getNumArgs() != 1) {
- PrintError("Expected exactly 1 argument in 'apply'");
+ ::PrintError(DiagLoc,
+ "Failed to parse pattern: '" + Arg->getAsString() + "'");
return false;
}
- const StringInit *Code = dyn_cast<StringInit>(Apply.getArg(0));
- auto Pat = std::make_unique<CXXPattern>(*Code, makeAnonPatName("apply"),
- /*IsApply*/ true);
- ApplyPats[Pat->getName()] = std::move(Pat);
return true;
}
std::unique_ptr<Pattern>
-CombineRuleBuilder::parseInstructionMatcher(const Init &Arg, StringRef Name) {
- const DagInit *Matcher = getDagWithOperatorOfSubClass(Arg, "Instruction");
- if (!Matcher)
+CombineRuleBuilder::parseInstructionPattern(const Init &Arg,
+ StringRef Name) const {
+ const DagInit *DagPat = dyn_cast<DagInit>(&Arg);
+ if (!DagPat)
return nullptr;
- auto &Instr = CGT.getInstruction(Matcher->getOperatorAsDef(RuleDef.getLoc()));
- auto Pat = std::make_unique<InstructionPattern>(Instr, Name);
+ std::unique_ptr<InstructionPattern> Pat;
+ if (const DagInit *IP = getDagWithOperatorOfSubClass(Arg, "Instruction")) {
+ auto &Instr = CGT.getInstruction(IP->getOperatorAsDef(RuleDef.getLoc()));
+ Pat = std::make_unique<CodeGenInstructionPattern>(Instr, Name);
+ } else if (const DagInit *PFP =
+ getDagWithOperatorOfSubClass(Arg, PatFragClassName)) {
+ const Record *Def = PFP->getOperatorAsDef(RuleDef.getLoc());
+ const PatFrag *PF = parsePatFrag(Def);
+ if (!PF)
+ return nullptr; // Already diagnosed by parsePatFrag
+ Pat = std::make_unique<PatFragPattern>(*PF, Name);
+ } else {
+ return nullptr;
+ }
- for (const auto &NameInit : Matcher->getArgNames())
- Pat->addOperand(NameInit->getAsUnquotedString());
+ for (unsigned K = 0; K < DagPat->getNumArgs(); ++K) {
+ if (!parseInstructionPatternOperand(*Pat, DagPat->getArg(K),
+ DagPat->getArgName(K)))
+ return nullptr;
+ }
if (!Pat->checkSemantics(RuleDef.getLoc()))
return nullptr;
@@ -994,7 +2208,7 @@ CombineRuleBuilder::parseInstructionMatcher(const Init &Arg, StringRef Name) {
std::unique_ptr<Pattern>
CombineRuleBuilder::parseWipMatchOpcodeMatcher(const Init &Arg,
- StringRef Name) {
+ StringRef Name) const {
const DagInit *Matcher = getDagWithSpecificOperator(Arg, "wip_match_opcode");
if (!Matcher)
return nullptr;
@@ -1020,15 +2234,230 @@ CombineRuleBuilder::parseWipMatchOpcodeMatcher(const Init &Arg,
return std::move(Result);
}
+bool CombineRuleBuilder::parseInstructionPatternOperand(
+ InstructionPattern &IP, const Init *OpInit,
+ const StringInit *OpName) const {
+ const auto ParseErr = [&]() {
+ PrintError("cannot parse operand '" + OpInit->getAsUnquotedString() + "' ");
+ if (OpName)
+ PrintNote("operand name is '" + OpName->getAsUnquotedString() + "'");
+ return false;
+ };
+
+ // untyped immediate, e.g. 0
+ if (const auto *IntImm = dyn_cast<IntInit>(OpInit)) {
+ std::string Name = OpName ? OpName->getAsUnquotedString() : "";
+ IP.addOperand(IntImm->getValue(), Name, /*Type=*/nullptr);
+ return true;
+ }
+
+ // typed immediate, e.g. (i32 0)
+ if (const auto *DagOp = dyn_cast<DagInit>(OpInit)) {
+ if (DagOp->getNumArgs() != 1)
+ return ParseErr();
+
+ Record *ImmTy = DagOp->getOperatorAsDef(RuleDef.getLoc());
+ if (!ImmTy->isSubClassOf("ValueType")) {
+ PrintError("cannot parse immediate '" + OpInit->getAsUnquotedString() +
+ "', '" + ImmTy->getName() + "' is not a ValueType!");
+ return false;
+ }
+
+ if (!IP.hasAllDefs()) {
+ PrintError("out operand of '" + IP.getInstName() +
+ "' cannot be an immediate");
+ return false;
+ }
+
+ const auto *Val = dyn_cast<IntInit>(DagOp->getArg(0));
+ if (!Val)
+ return ParseErr();
+
+ std::string Name = OpName ? OpName->getAsUnquotedString() : "";
+ IP.addOperand(Val->getValue(), Name, ImmTy);
+ return true;
+ }
+
+ // Typed operand e.g. $x/$z in (G_FNEG $x, $z)
+ if (auto *DefI = dyn_cast<DefInit>(OpInit)) {
+ if (!OpName) {
+ PrintError("expected an operand name after '" + OpInit->getAsString() +
+ "'");
+ return false;
+ }
+ const Record *Def = DefI->getDef();
+ if (!Def->isSubClassOf("ValueType")) {
+ PrintError("invalid operand type: '" + Def->getName() +
+ "' is not a ValueType");
+ return false;
+ }
+ IP.addOperand(OpName->getAsUnquotedString(), Def);
+ return true;
+ }
+
+ // Untyped operand e.g. $x/$z in (G_FNEG $x, $z)
+ if (isa<UnsetInit>(OpInit)) {
+ assert(OpName && "Unset w/ no OpName?");
+ IP.addOperand(OpName->getAsUnquotedString(), /*Type=*/nullptr);
+ return true;
+ }
+
+ return ParseErr();
+}
+
+std::unique_ptr<PatFrag>
+CombineRuleBuilder::parsePatFragImpl(const Record *Def) const {
+ auto StackTrace = PrettyStackTraceParse(*Def);
+ if (!Def->isSubClassOf(PatFragClassName))
+ return nullptr;
+
+ const DagInit *Ins = Def->getValueAsDag("InOperands");
+ if (Ins->getOperatorAsDef(Def->getLoc())->getName() != "ins") {
+ ::PrintError(Def, "expected 'ins' operator for " + PatFragClassName +
+ " in operands list");
+ return nullptr;
+ }
+
+ const DagInit *Outs = Def->getValueAsDag("OutOperands");
+ if (Outs->getOperatorAsDef(Def->getLoc())->getName() != "outs") {
+ ::PrintError(Def, "expected 'outs' operator for " + PatFragClassName +
+ " out operands list");
+ return nullptr;
+ }
+
+ auto Result = std::make_unique<PatFrag>(*Def);
+ if (!parsePatFragParamList(Def->getLoc(), *Outs,
+ [&](StringRef Name, PatFrag::ParamKind Kind) {
+ Result->addOutParam(Name, Kind);
+ return true;
+ }))
+ return nullptr;
+
+ if (!parsePatFragParamList(Def->getLoc(), *Ins,
+ [&](StringRef Name, PatFrag::ParamKind Kind) {
+ Result->addInParam(Name, Kind);
+ return true;
+ }))
+ return nullptr;
+
+ const ListInit *Alts = Def->getValueAsListInit("Alternatives");
+ unsigned AltIdx = 0;
+ for (const Init *Alt : *Alts) {
+ const auto *PatDag = dyn_cast<DagInit>(Alt);
+ if (!PatDag) {
+ ::PrintError(Def, "expected dag init for PatFrag pattern alternative");
+ return nullptr;
+ }
+
+ PatFrag::Alternative &A = Result->addAlternative();
+ const auto AddPat = [&](std::unique_ptr<Pattern> Pat) {
+ A.Pats.push_back(std::move(Pat));
+ return true;
+ };
+
+ if (!parsePatternList(
+ *PatDag, AddPat, "pattern", Def->getLoc(),
+ /*AnonPatPrefix*/
+ (Def->getName() + "_alt" + Twine(AltIdx++) + "_pattern").str()))
+ return nullptr;
+ }
+
+ if (!Result->buildOperandsTables() || !Result->checkSemantics())
+ return nullptr;
+
+ return Result;
+}
+
+bool CombineRuleBuilder::parsePatFragParamList(
+ ArrayRef<SMLoc> DiagLoc, const DagInit &OpsList,
+ function_ref<bool(StringRef, PatFrag::ParamKind)> ParseAction) const {
+ for (unsigned K = 0; K < OpsList.getNumArgs(); ++K) {
+ const StringInit *Name = OpsList.getArgName(K);
+ const Init *Ty = OpsList.getArg(K);
+
+ if (!Name) {
+ ::PrintError(DiagLoc, "all operands must be named'");
+ return false;
+ }
+ const std::string NameStr = Name->getAsUnquotedString();
+
+ PatFrag::ParamKind OpKind;
+ if (isSpecificDef(*Ty, "gi_imm"))
+ OpKind = PatFrag::PK_Imm;
+ else if (isSpecificDef(*Ty, "root"))
+ OpKind = PatFrag::PK_Root;
+ else if (isa<UnsetInit>(Ty) ||
+ isSpecificDef(*Ty, "gi_mo")) // no type = gi_mo.
+ OpKind = PatFrag::PK_MachineOperand;
+ else {
+ ::PrintError(
+ DiagLoc,
+ "'" + NameStr +
+ "' operand type was expected to be 'root', 'gi_imm' or 'gi_mo'");
+ return false;
+ }
+
+ if (!ParseAction(NameStr, OpKind))
+ return false;
+ }
+
+ return true;
+}
+
+const PatFrag *CombineRuleBuilder::parsePatFrag(const Record *Def) const {
+ // Cache already parsed PatFrags to avoid doing extra work.
+ static DenseMap<const Record *, std::unique_ptr<PatFrag>> ParsedPatFrags;
+
+ auto It = ParsedPatFrags.find(Def);
+ if (It != ParsedPatFrags.end()) {
+ SeenPatFrags.insert(It->second.get());
+ return It->second.get();
+ }
+
+ std::unique_ptr<PatFrag> NewPatFrag = parsePatFragImpl(Def);
+ if (!NewPatFrag) {
+ ::PrintError(Def, "Could not parse " + PatFragClassName + " '" +
+ Def->getName() + "'");
+ // Put a nullptr in the map so we don't attempt parsing this again.
+ ParsedPatFrags[Def] = nullptr;
+ return nullptr;
+ }
+
+ const auto *Res = NewPatFrag.get();
+ ParsedPatFrags[Def] = std::move(NewPatFrag);
+ SeenPatFrags.insert(Res);
+ return Res;
+}
+
bool CombineRuleBuilder::emitMatchPattern(CodeExpansions &CE,
+ const PatternAlternatives &Alts,
const InstructionPattern &IP) {
- auto &M = addRuleMatcher();
+ auto StackTrace = PrettyStackTraceEmit(RuleDef, &IP);
+
+ auto &M = addRuleMatcher(Alts);
InstructionMatcher &IM = M.addInstructionMatcher("root");
declareInstExpansion(CE, IM, IP.getName());
DenseSet<const Pattern *> SeenPats;
- if (!emitInstructionMatchPattern(CE, M, IM, IP, SeenPats))
- return false;
+
+ const auto FindOperandDef = [&](StringRef Op) -> InstructionPattern * {
+ return MatchOpTable.getDef(Op);
+ };
+
+ if (const auto *CGP = dyn_cast<CodeGenInstructionPattern>(&IP)) {
+ if (!emitCodeGenInstructionMatchPattern(CE, Alts, M, IM, *CGP, SeenPats,
+ FindOperandDef))
+ return false;
+ } else if (const auto *PFP = dyn_cast<PatFragPattern>(&IP)) {
+ if (!PFP->getPatFrag().canBeMatchRoot()) {
+ PrintError("cannot use '" + PFP->getInstName() + " as match root");
+ return false;
+ }
+
+ if (!emitPatFragMatchPattern(CE, Alts, M, &IM, *PFP, SeenPats))
+ return false;
+ } else
+ llvm_unreachable("Unknown kind of InstructionPattern!");
// Emit remaining patterns
for (auto &Pat : values(MatchPats)) {
@@ -1039,11 +2468,17 @@ bool CombineRuleBuilder::emitMatchPattern(CodeExpansions &CE,
case Pattern::K_AnyOpcode:
PrintError("wip_match_opcode can not be used with instruction patterns!");
return false;
- case Pattern::K_Inst:
+ case Pattern::K_PatFrag: {
+ if (!emitPatFragMatchPattern(CE, Alts, M, /*IM*/ nullptr,
+ *cast<PatFragPattern>(Pat.get()), SeenPats))
+ return false;
+ continue;
+ }
+ case Pattern::K_CodeGenInstruction:
cast<InstructionPattern>(Pat.get())->reportUnreachable(RuleDef.getLoc());
return false;
case Pattern::K_CXX: {
- addCXXPredicate(IM, CE, *cast<CXXPattern>(Pat.get()));
+ addCXXPredicate(M, CE, *cast<CXXPattern>(Pat.get()), Alts);
continue;
}
default:
@@ -1055,11 +2490,13 @@ bool CombineRuleBuilder::emitMatchPattern(CodeExpansions &CE,
}
bool CombineRuleBuilder::emitMatchPattern(CodeExpansions &CE,
+ const PatternAlternatives &Alts,
const AnyOpcodePattern &AOP) {
+ auto StackTrace = PrettyStackTraceEmit(RuleDef, &AOP);
for (const CodeGenInstruction *CGI : AOP.insts()) {
- auto &M = addRuleMatcher("wip_match_opcode alternative '" +
- CGI->TheDef->getName() + "'");
+ auto &M = addRuleMatcher(Alts, "wip_match_opcode '" +
+ CGI->TheDef->getName() + "'");
InstructionMatcher &IM = M.addInstructionMatcher(AOP.getName());
declareInstExpansion(CE, IM, AOP.getName());
@@ -1078,12 +2515,20 @@ bool CombineRuleBuilder::emitMatchPattern(CodeExpansions &CE,
case Pattern::K_AnyOpcode:
PrintError("wip_match_opcode can only be present once!");
return false;
- case Pattern::K_Inst:
+ case Pattern::K_PatFrag: {
+ DenseSet<const Pattern *> SeenPats;
+ if (!emitPatFragMatchPattern(CE, Alts, M, /*IM*/ nullptr,
+ *cast<PatFragPattern>(Pat.get()),
+ SeenPats))
+ return false;
+ continue;
+ }
+ case Pattern::K_CodeGenInstruction:
cast<InstructionPattern>(Pat.get())->reportUnreachable(
RuleDef.getLoc());
return false;
case Pattern::K_CXX: {
- addCXXPredicate(IM, CE, *cast<CXXPattern>(Pat.get()));
+ addCXXPredicate(M, CE, *cast<CXXPattern>(Pat.get()), Alts);
break;
}
default:
@@ -1098,30 +2543,374 @@ bool CombineRuleBuilder::emitMatchPattern(CodeExpansions &CE,
return true;
}
+bool CombineRuleBuilder::emitPatFragMatchPattern(
+ CodeExpansions &CE, const PatternAlternatives &Alts, RuleMatcher &RM,
+ InstructionMatcher *IM, const PatFragPattern &PFP,
+ DenseSet<const Pattern *> &SeenPats) {
+ auto StackTrace = PrettyStackTraceEmit(RuleDef, &PFP);
+
+ if (SeenPats.contains(&PFP))
+ return true;
+ SeenPats.insert(&PFP);
+
+ const auto &PF = PFP.getPatFrag();
+
+ if (!IM) {
+ // When we don't have an IM, this means this PatFrag isn't reachable from
+ // the root. This is only acceptable if it doesn't define anything (e.g. a
+ // pure C++ PatFrag).
+ if (PF.num_out_params() != 0) {
+ PFP.reportUnreachable(RuleDef.getLoc());
+ return false;
+ }
+ } else {
+ // When an IM is provided, this is reachable from the root, and we're
+ // expecting to have output operands.
+ // TODO: If we want to allow for multiple roots we'll need a map of IMs
+ // then, and emission becomes a bit more complicated.
+ assert(PF.num_roots() == 1);
+ }
+
+ CodeExpansions PatFragCEs;
+ if (!PFP.mapInputCodeExpansions(CE, PatFragCEs, RuleDef.getLoc()))
+ return false;
+
+ // List of {ParamName, ArgName}.
+ // When all patterns have been emitted, find expansions in PatFragCEs named
+ // ArgName and add their expansion to CE using ParamName as the key.
+ SmallVector<std::pair<std::string, std::string>, 4> CEsToImport;
+
+ // Map parameter names to the actual argument.
+ const auto OperandMapper =
+ [&](const InstructionPattern::Operand &O) -> InstructionPattern::Operand {
+ if (!O.isNamedOperand())
+ return O;
+
+ StringRef ParamName = O.getOperandName();
+
+ // Not sure what to do with those tbh. They should probably never be here.
+ assert(!O.isNamedImmediate() && "TODO: handle named imms");
+ unsigned PIdx = PF.getParamIdx(ParamName);
+
+ // Map parameters to the argument values.
+ if (PIdx == (unsigned)-1) {
+ // This is a temp of the PatFragPattern, prefix the name to avoid
+ // conflicts.
+ return O.withNewName((PFP.getName() + "." + ParamName).str());
+ }
+
+ // The operand will be added to PatFragCEs's code expansions using the
+ // parameter's name. If it's bound to some operand during emission of the
+ // patterns, we'll want to add it to CE.
+ auto ArgOp = PFP.getOperand(PIdx);
+ if (ArgOp.isNamedOperand())
+ CEsToImport.emplace_back(ArgOp.getOperandName().str(), ParamName);
+
+ if (ArgOp.getType() && O.getType() && ArgOp.getType() != O.getType()) {
+ StringRef PFName = PF.getName();
+ PrintWarning("impossible type constraints: operand " + Twine(PIdx) +
+ " of '" + PFP.getName() + "' has type '" +
+ ArgOp.getType()->getName() + "', but '" + PFName +
+ "' constrains it to '" + O.getType()->getName() + "'");
+ if (ArgOp.isNamedOperand())
+ PrintNote("operand " + Twine(PIdx) + " of '" + PFP.getName() +
+ "' is '" + ArgOp.getOperandName() + "'");
+ if (O.isNamedOperand())
+ PrintNote("argument " + Twine(PIdx) + " of '" + PFName + "' is '" +
+ ParamName + "'");
+ }
+
+ return ArgOp;
+ };
+
+ // PatFragPatterns are only made of InstructionPatterns or CXXPatterns.
+ // Emit instructions from the root.
+ const auto &FragAlt = PF.getAlternative(Alts.lookup(&PFP));
+ const auto &FragAltOT = FragAlt.OpTable;
+ const auto LookupOperandDef =
+ [&](StringRef Op) -> const InstructionPattern * {
+ return FragAltOT.getDef(Op);
+ };
+
+ DenseSet<const Pattern *> PatFragSeenPats;
+ for (const auto &[Idx, InOp] : enumerate(PF.out_params())) {
+ if (InOp.Kind != PatFrag::PK_Root)
+ continue;
+
+ StringRef ParamName = InOp.Name;
+ const auto *Def = FragAltOT.getDef(ParamName);
+ assert(Def && "PatFrag::checkSemantics should have emitted an error if "
+ "an out operand isn't defined!");
+ assert(isa<CodeGenInstructionPattern>(Def) &&
+ "Nested PatFrags not supported yet");
+
+ if (!emitCodeGenInstructionMatchPattern(
+ PatFragCEs, Alts, RM, *IM, *cast<CodeGenInstructionPattern>(Def),
+ PatFragSeenPats, LookupOperandDef, OperandMapper))
+ return false;
+ }
+
+ // Emit leftovers.
+ for (const auto &Pat : FragAlt.Pats) {
+ if (PatFragSeenPats.contains(Pat.get()))
+ continue;
+
+ if (const auto *CXXPat = dyn_cast<CXXPattern>(Pat.get())) {
+ addCXXPredicate(RM, PatFragCEs, *CXXPat, Alts);
+ continue;
+ }
+
+ if (const auto *IP = dyn_cast<InstructionPattern>(Pat.get())) {
+ IP->reportUnreachable(PF.getLoc());
+ return false;
+ }
+
+ llvm_unreachable("Unexpected pattern kind in PatFrag");
+ }
+
+ for (const auto &[ParamName, ArgName] : CEsToImport) {
+ // Note: we're find if ParamName already exists. It just means it's been
+ // bound before, so we prefer to keep the first binding.
+ CE.declare(ParamName, PatFragCEs.lookup(ArgName));
+ }
+
+ return true;
+}
+
bool CombineRuleBuilder::emitApplyPatterns(CodeExpansions &CE, RuleMatcher &M) {
+ if (hasOnlyCXXApplyPatterns()) {
+ for (auto &Pat : values(ApplyPats))
+ addCXXAction(M, CE, *cast<CXXPattern>(Pat.get()));
+ return true;
+ }
+
+ DenseSet<const Pattern *> SeenPats;
+ StringMap<unsigned> OperandToTempRegID;
+
+ for (auto *ApplyRoot : ApplyRoots) {
+ assert(isa<CodeGenInstructionPattern>(ApplyRoot) &&
+ "PatFragPatterns are not supported in apply patterns yet!");
+ if (!emitCodeGenInstructionApplyPattern(
+ CE, M, *cast<CodeGenInstructionPattern>(ApplyRoot), SeenPats,
+ OperandToTempRegID))
+ return false;
+ }
+
for (auto &Pat : values(ApplyPats)) {
+ if (SeenPats.contains(Pat.get()))
+ continue;
+
switch (Pat->getKind()) {
case Pattern::K_AnyOpcode:
- case Pattern::K_Inst:
- llvm_unreachable("Unsupported pattern kind in output pattern!");
+ llvm_unreachable("Unexpected pattern in apply!");
+ case Pattern::K_PatFrag:
+ // TODO: We could support pure C++ PatFrags as a temporary thing.
+ llvm_unreachable("Unexpected pattern in apply!");
+ case Pattern::K_CodeGenInstruction:
+ cast<CodeGenInstructionPattern>(Pat.get())->reportUnreachable(
+ RuleDef.getLoc());
+ return false;
case Pattern::K_CXX: {
- CXXPattern *CXXPat = cast<CXXPattern>(Pat.get());
- const auto &ExpandedCode = CXXPat->expandCode(CE, RuleDef.getLoc());
- M.addAction<CustomCXXAction>(
- ExpandedCode.getEnumNameWithPrefix(CXXApplyPrefix));
+ addCXXAction(M, CE, *cast<CXXPattern>(Pat.get()));
continue;
}
default:
- llvm_unreachable("Unknown pattern kind!");
+ llvm_unreachable("unknown pattern kind!");
+ }
+ }
+
+ return true;
+}
+
+bool CombineRuleBuilder::emitCodeGenInstructionApplyPattern(
+ CodeExpansions &CE, RuleMatcher &M, const CodeGenInstructionPattern &P,
+ DenseSet<const Pattern *> &SeenPats,
+ StringMap<unsigned> &OperandToTempRegID) {
+ auto StackTrace = PrettyStackTraceEmit(RuleDef, &P);
+
+ if (SeenPats.contains(&P))
+ return true;
+
+ SeenPats.insert(&P);
+
+ // First, render the uses.
+ for (auto &Op : P.named_operands()) {
+ if (Op.isDef())
+ continue;
+
+ StringRef OpName = Op.getOperandName();
+ if (const auto *DefPat = ApplyOpTable.getDef(OpName)) {
+ if (!emitCodeGenInstructionApplyPattern(CE, M, *DefPat, SeenPats,
+ OperandToTempRegID))
+ return false;
+ } else {
+ // If we have no def, check this exists in the MatchRoot.
+ if (!Op.isNamedImmediate() && !MatchOpTable.lookup(OpName).Found) {
+ PrintError("invalid output operand '" + OpName +
+ "': operand is not a live-in of the match pattern, and it "
+ "has no definition");
+ return false;
+ }
+ }
+ }
+
+ // Now render this inst.
+ auto &DstMI =
+ M.addAction<BuildMIAction>(M.allocateOutputInsnID(), &P.getInst());
+
+ for (auto &Op : P.operands()) {
+ if (Op.isNamedImmediate()) {
+ PrintError("invalid output operand '" + Op.getOperandName() +
+ "': output immediates cannot be named");
+ PrintNote("while emitting pattern '" + P.getName() + "' (" +
+ P.getInstName() + ")");
+ return false;
+ }
+
+ if (Op.hasImmValue()) {
+ if (!emitCodeGenInstructionApplyImmOperand(M, DstMI, P, Op))
+ return false;
+ continue;
+ }
+
+ StringRef OpName = Op.getOperandName();
+
+ // Uses of operand.
+ if (!Op.isDef()) {
+ if (auto It = OperandToTempRegID.find(OpName);
+ It != OperandToTempRegID.end()) {
+ assert(!MatchOpTable.lookup(OpName).Found &&
+ "Temp reg is also from match pattern?");
+ DstMI.addRenderer<TempRegRenderer>(It->second);
+ } else {
+ // This should be a match live in or a redef of a matched instr.
+ // If it's a use of a temporary register, then we messed up somewhere -
+ // the previous condition should have passed.
+ assert(MatchOpTable.lookup(OpName).Found &&
+ !ApplyOpTable.getDef(OpName) && "Temp reg not emitted yet!");
+ DstMI.addRenderer<CopyRenderer>(OpName);
+ }
+ continue;
+ }
+
+ // Determine what we're dealing with. Are we replace a matched instruction?
+ // Creating a new one?
+ auto OpLookupRes = MatchOpTable.lookup(OpName);
+ if (OpLookupRes.Found) {
+ if (OpLookupRes.isLiveIn()) {
+ // live-in of the match pattern.
+ PrintError("Cannot define live-in operand '" + OpName +
+ "' in the 'apply' pattern");
+ return false;
+ }
+ assert(OpLookupRes.Def);
+
+ // TODO: Handle this. We need to mutate the instr, or delete the old
+ // one.
+ // Likewise, we also need to ensure we redef everything, if the
+ // instr has more than one def, we need to redef all or nothing.
+ if (OpLookupRes.Def != MatchRoot) {
+ PrintError("redefining an instruction other than the root is not "
+ "supported (operand '" +
+ OpName + "')");
+ return false;
+ }
+ // redef of a match
+ DstMI.addRenderer<CopyRenderer>(OpName);
+ continue;
+ }
+
+ // Define a new register unique to the apply patterns (AKA a "temp"
+ // register).
+ unsigned TempRegID;
+ if (auto It = OperandToTempRegID.find(OpName);
+ It != OperandToTempRegID.end()) {
+ TempRegID = It->second;
+ } else {
+ // This is a brand new register.
+ TempRegID = M.allocateTempRegID();
+ OperandToTempRegID[OpName] = TempRegID;
+ const Record *Ty = Op.getType();
+ if (!Ty) {
+ PrintError("def of a new register '" + OpName +
+ "' in the apply patterns must have a type");
+ return false;
+ }
+ declareTempRegExpansion(CE, TempRegID, OpName);
+ // Always insert the action at the beginning, otherwise we may end up
+ // using the temp reg before it's available.
+ M.insertAction<MakeTempRegisterAction>(
+ M.actions_begin(), getLLTCodeGenFromRecord(Ty), TempRegID);
+ }
+
+ DstMI.addRenderer<TempRegRenderer>(TempRegID);
+ }
+
+ // TODO: works?
+ DstMI.chooseInsnToMutate(M);
+ declareInstExpansion(CE, DstMI, P.getName());
+
+ return true;
+}
+
+bool CombineRuleBuilder::emitCodeGenInstructionApplyImmOperand(
+ RuleMatcher &M, BuildMIAction &DstMI, const CodeGenInstructionPattern &P,
+ const InstructionPattern::Operand &O) {
+ // If we have a type, we implicitly emit a G_CONSTANT, except for G_CONSTANT
+ // itself where we emit a CImm.
+ //
+ // No type means we emit a simple imm.
+ // G_CONSTANT is a special case and needs a CImm though so this is likely a
+ // mistake.
+ const bool isGConstant = P.is("G_CONSTANT");
+ const Record *Ty = O.getType();
+ if (!Ty) {
+ if (isGConstant) {
+ PrintError("'G_CONSTANT' immediate must be typed!");
+ PrintNote("while emitting pattern '" + P.getName() + "' (" +
+ P.getInstName() + ")");
+ return false;
}
+
+ DstMI.addRenderer<ImmRenderer>(O.getImmValue());
+ return true;
}
+ LLTCodeGen LLT = getLLTCodeGenFromRecord(Ty);
+ if (isGConstant) {
+ DstMI.addRenderer<ImmRenderer>(O.getImmValue(), LLT);
+ return true;
+ }
+
+ unsigned TempRegID = M.allocateTempRegID();
+ auto ActIt = M.insertAction<BuildMIAction>(
+ M.actions_begin(), M.allocateOutputInsnID(), &getGConstant());
+ // Ensure MakeTempReg occurs before the BuildMI of th G_CONSTANT.
+ M.insertAction<MakeTempRegisterAction>(ActIt, LLT, TempRegID);
+ auto &ConstantMI = *static_cast<BuildMIAction *>(ActIt->get());
+ ConstantMI.addRenderer<TempRegRenderer>(TempRegID);
+ ConstantMI.addRenderer<ImmRenderer>(O.getImmValue(), LLT);
+ DstMI.addRenderer<TempRegRenderer>(TempRegID);
return true;
}
-bool CombineRuleBuilder::emitInstructionMatchPattern(
- CodeExpansions &CE, RuleMatcher &M, InstructionMatcher &IM,
- const InstructionPattern &P, DenseSet<const Pattern *> &SeenPats) {
+bool isLiteralImm(const InstructionPattern &P, unsigned OpIdx) {
+ if (const auto *CGP = dyn_cast<CodeGenInstructionPattern>(&P)) {
+ StringRef InstName = CGP->getInst().TheDef->getName();
+ return (InstName == "G_CONSTANT" || InstName == "G_FCONSTANT") &&
+ OpIdx == 1;
+ }
+
+ llvm_unreachable("TODO");
+}
+
+bool CombineRuleBuilder::emitCodeGenInstructionMatchPattern(
+ CodeExpansions &CE, const PatternAlternatives &Alts, RuleMatcher &M,
+ InstructionMatcher &IM, const CodeGenInstructionPattern &P,
+ DenseSet<const Pattern *> &SeenPats, OperandDefLookupFn LookupOperandDef,
+ OperandMapperFnRef OperandMapper) {
+ auto StackTrace = PrettyStackTraceEmit(RuleDef, &P);
+
if (SeenPats.contains(&P))
return true;
@@ -1130,31 +2919,97 @@ bool CombineRuleBuilder::emitInstructionMatchPattern(
IM.addPredicate<InstructionOpcodeMatcher>(&P.getInst());
declareInstExpansion(CE, IM, P.getName());
- unsigned OpIdx = 0;
- for (auto &O : P.operands()) {
- auto &OpTableEntry = OperandTable.find(O.Name)->second;
-
+ for (const auto &[Idx, OriginalO] : enumerate(P.operands())) {
+ // Remap the operand. This is used when emitting InstructionPatterns inside
+ // PatFrags, so it can remap them to the arguments passed to the pattern.
+ //
+ // We use the remapped operand to emit immediates, and for the symbolic
+ // operand names (in IM.addOperand). CodeExpansions and OperandTable lookups
+ // still use the original name.
+ //
+ // The "def" flag on the remapped operand is always ignored.
+ auto RemappedO = OperandMapper(OriginalO);
+ assert(RemappedO.isNamedOperand() == OriginalO.isNamedOperand() &&
+ "Cannot remap an unnamed operand to a named one!");
+
+ const auto OpName =
+ RemappedO.isNamedOperand() ? RemappedO.getOperandName().str() : "";
OperandMatcher &OM =
- IM.addOperand(OpIdx++, O.Name, AllocatedTemporariesBaseID++);
- declareOperandExpansion(CE, OM, O.Name);
+ IM.addOperand(Idx, OpName, AllocatedTemporariesBaseID++);
+ if (!OpName.empty())
+ declareOperandExpansion(CE, OM, OriginalO.getOperandName());
+
+ // Handle immediates.
+ if (RemappedO.hasImmValue()) {
+ if (isLiteralImm(P, Idx))
+ OM.addPredicate<LiteralIntOperandMatcher>(RemappedO.getImmValue());
+ else
+ OM.addPredicate<ConstantIntOperandMatcher>(RemappedO.getImmValue());
+ }
+
+ // Handle typed operands, but only bother to check if it hasn't been done
+ // before.
+ //
+ // getOperandMatcher will always return the first OM to have been created
+ // for that Operand. "OM" here is always a new OperandMatcher.
+ //
+ // Always emit a check for unnamed operands.
+ if (OpName.empty() ||
+ !M.getOperandMatcher(OpName).contains<LLTOperandMatcher>()) {
+ if (const Record *Ty = RemappedO.getType())
+ OM.addPredicate<LLTOperandMatcher>(getLLTCodeGenFromRecord(Ty));
+ }
+
+ // Stop here if the operand is a def, or if it had no name.
+ if (OriginalO.isDef() || !OriginalO.isNamedOperand())
+ continue;
- if (O.IsDef)
+ const auto *DefPat = LookupOperandDef(OriginalO.getOperandName());
+ if (!DefPat)
continue;
- if (InstructionPattern *DefPat = OpTableEntry.MatchPat) {
- auto InstOpM = OM.addPredicate<InstructionOperandMatcher>(M, O.Name);
- if (!InstOpM) {
- // TODO: copy-pasted from GlobalISelEmitter.cpp. Is it still relevant
- // here?
- PrintError("Nested instruction '" + DefPat->getName() +
- "' cannot be the same as another operand '" + O.Name + "'");
+ if (OriginalO.hasImmValue()) {
+ assert(!OpName.empty());
+ // This is a named immediate that also has a def, that's not okay.
+ // e.g.
+ // (G_SEXT $y, (i32 0))
+ // (COPY $x, 42:$y)
+ PrintError("'" + OpName +
+ "' is a named immediate, it cannot be defined by another "
+ "instruction");
+ PrintNote("'" + OpName + "' is defined by '" + DefPat->getName() + "'");
+ return false;
+ }
+
+ // From here we know that the operand defines an instruction, and we need to
+ // emit it.
+ auto InstOpM =
+ OM.addPredicate<InstructionOperandMatcher>(M, DefPat->getName());
+ if (!InstOpM) {
+ // TODO: copy-pasted from GlobalISelEmitter.cpp. Is it still relevant
+ // here?
+ PrintError("Nested instruction '" + DefPat->getName() +
+ "' cannot be the same as another operand '" +
+ OriginalO.getOperandName() + "'");
+ return false;
+ }
+
+ auto &IM = (*InstOpM)->getInsnMatcher();
+ if (const auto *CGIDef = dyn_cast<CodeGenInstructionPattern>(DefPat)) {
+ if (!emitCodeGenInstructionMatchPattern(CE, Alts, M, IM, *CGIDef,
+ SeenPats, LookupOperandDef,
+ OperandMapper))
return false;
- }
+ continue;
+ }
- if (!emitInstructionMatchPattern(CE, M, (*InstOpM)->getInsnMatcher(),
- *DefPat, SeenPats))
+ if (const auto *PFPDef = dyn_cast<PatFragPattern>(DefPat)) {
+ if (!emitPatFragMatchPattern(CE, Alts, M, &IM, *PFPDef, SeenPats))
return false;
+ continue;
}
+
+ llvm_unreachable("unknown type of InstructionPattern");
}
return true;
@@ -1162,10 +3017,12 @@ bool CombineRuleBuilder::emitInstructionMatchPattern(
//===- GICombinerEmitter --------------------------------------------------===//
-/// This class is essentially the driver. It fetches all TableGen records, calls
-/// CombineRuleBuilder to build the MatchTable's RuleMatchers, then creates the
-/// MatchTable & emits it. It also handles emitting all the supporting code such
-/// as the list of LLTs, the CXXPredicates, etc.
+/// Main implementation class. This emits the tablegenerated output.
+///
+/// It collects rules, uses `CombineRuleBuilder` to parse them and accumulate
+/// RuleMatchers, then takes all the necessary state/data from the various
+/// static storage pools and wires them together to emit the match table &
+/// associated function/data structures.
class GICombinerEmitter final : public GlobalISelMatchTableExecutorEmitter {
RecordKeeper &Records;
StringRef Name;
@@ -1338,7 +3195,7 @@ void GICombinerEmitter::emitAdditionalImpl(raw_ostream &OS) {
<< " if (executeMatchTable(*this, OutMIs, State, ExecInfo"
<< ", getMatchTable(), *ST.getInstrInfo(), MRI, "
"*MRI.getTargetRegisterInfo(), *ST.getRegBankInfo(), AvailableFeatures"
- << ", /*CoverageInfo*/ nullptr)) {\n"
+ << ", /*CoverageInfo*/ nullptr, &Observer)) {\n"
<< " return true;\n"
<< " }\n\n"
<< " return false;\n"
@@ -1346,7 +3203,7 @@ void GICombinerEmitter::emitAdditionalImpl(raw_ostream &OS) {
}
void GICombinerEmitter::emitMIPredicateFns(raw_ostream &OS) {
- auto MatchCode = getSorted(AllCXXMatchCode);
+ auto MatchCode = CXXPredicateCode::getAllMatchCode();
emitMIPredicateFnsImpl<const CXXPredicateCode *>(
OS, "", ArrayRef<const CXXPredicateCode *>(MatchCode),
[](const CXXPredicateCode *C) -> StringRef { return C->BaseEnumName; },
@@ -1400,7 +3257,7 @@ void GICombinerEmitter::emitTestSimplePredicate(raw_ostream &OS) {
}
void GICombinerEmitter::emitRunCustomAction(raw_ostream &OS) {
- const auto ApplyCode = getSorted(AllCXXApplyCode);
+ const auto ApplyCode = CXXPredicateCode::getAllApplyCode();
if (!ApplyCode.empty()) {
OS << "enum {\n";
@@ -1414,13 +3271,14 @@ void GICombinerEmitter::emitRunCustomAction(raw_ostream &OS) {
}
OS << "void " << getClassName()
- << "::runCustomAction(unsigned ApplyID, const MatcherState &State) const "
+ << "::runCustomAction(unsigned ApplyID, const MatcherState &State, "
+ "NewMIVector &OutMIs) const "
"{\n";
if (!ApplyCode.empty()) {
OS << " switch(ApplyID) {\n";
for (const auto &Apply : ApplyCode) {
OS << " case " << Apply->getEnumNameWithPrefix(CXXApplyPrefix) << ":{\n"
- << " " << Apply->Code << "\n"
+ << " " << join(split(Apply->Code, "\n"), "\n ") << "\n"
<< " return;\n";
OS << " }\n";
}
@@ -1490,24 +3348,28 @@ GICombinerEmitter::buildMatchTable(MutableArrayRef<RuleMatcher> Rules) {
void GICombinerEmitter::gatherRules(
std::vector<RuleMatcher> &ActiveRules,
const std::vector<Record *> &&RulesAndGroups) {
- for (Record *R : RulesAndGroups) {
- if (R->isValueUnset("Rules")) {
- AllCombineRules.emplace_back(NextRuleID, R->getName().str());
- CombineRuleBuilder CRB(Target, SubtargetFeatures, *R, NextRuleID++,
+ for (Record *Rec : RulesAndGroups) {
+ if (Rec->isValueUnset("Rules")) {
+ AllCombineRules.emplace_back(NextRuleID, Rec->getName().str());
+ CombineRuleBuilder CRB(Target, SubtargetFeatures, *Rec, NextRuleID++,
ActiveRules);
- if (!CRB.parseAll())
+ if (!CRB.parseAll()) {
+ assert(ErrorsPrinted && "Parsing failed without errors!");
continue;
+ }
if (StopAfterParse) {
CRB.print(outs());
continue;
}
- if (!CRB.emitRuleMatchers())
+ if (!CRB.emitRuleMatchers()) {
+ assert(ErrorsPrinted && "Emission failed without errors!");
continue;
+ }
} else
- gatherRules(ActiveRules, R->getValueAsListOfDefs("Rules"));
+ gatherRules(ActiveRules, Rec->getValueAsListOfDefs("Rules"));
}
}
@@ -1518,6 +3380,9 @@ void GICombinerEmitter::run(raw_ostream &OS) {
if (ErrorsPrinted)
PrintFatalError(Combiner->getLoc(), "Failed to parse one or more rules");
+ if (StopAfterParse)
+ return;
+
Records.startTimer("Creating Match Table");
unsigned MaxTemporaries = 0;
for (const auto &Rule : Rules)
@@ -1531,11 +3396,17 @@ void GICombinerEmitter::run(raw_ostream &OS) {
// Unused
std::vector<StringRef> CustomRendererFns;
- // Unused, but hack to avoid empty declarator
- std::vector<LLTCodeGen> TypeObjects = {LLTCodeGen(LLT::scalar(1))};
// Unused
std::vector<Record *> ComplexPredicates;
+ SmallVector<LLTCodeGen, 16> TypeObjects;
+ append_range(TypeObjects, KnownTypes);
+ llvm::sort(TypeObjects);
+
+ // Hack: Avoid empty declarator.
+ if (TypeObjects.empty())
+ TypeObjects.push_back(LLT::scalar(1));
+
// GET_GICOMBINER_DEPS, which pulls in extra dependencies.
OS << "#ifdef GET_GICOMBINER_DEPS\n"
<< "#include \"llvm/ADT/SparseBitVector.h\"\n"
@@ -1570,6 +3441,7 @@ void GICombinerEmitter::run(raw_ostream &OS) {
//===----------------------------------------------------------------------===//
static void EmitGICombiner(RecordKeeper &RK, raw_ostream &OS) {
+ EnablePrettyStackTrace();
CodeGenTarget Target(RK);
if (SelectedCombiners.empty())
diff --git a/llvm/utils/TableGen/GlobalISelEmitter.cpp b/llvm/utils/TableGen/GlobalISelEmitter.cpp
index b9a87368ca0e99..fca9fc59fab061 100644
--- a/llvm/utils/TableGen/GlobalISelEmitter.cpp
+++ b/llvm/utils/TableGen/GlobalISelEmitter.cpp
@@ -2366,7 +2366,8 @@ void GlobalISelEmitter::emitTestSimplePredicate(raw_ostream &OS) {
void GlobalISelEmitter::emitRunCustomAction(raw_ostream &OS) {
OS << "void " << getClassName()
- << "::runCustomAction(unsigned, const MatcherState&) const {\n"
+ << "::runCustomAction(unsigned, const MatcherState&, NewMIVector &) const "
+ "{\n"
<< " llvm_unreachable(\"" + getClassName() +
" does not support custom C++ actions!\");\n"
<< "}\n";
diff --git a/llvm/utils/TableGen/GlobalISelMatchTable.h b/llvm/utils/TableGen/GlobalISelMatchTable.h
index c773cbfbba6fc3..46490e8e3fb18c 100644
--- a/llvm/utils/TableGen/GlobalISelMatchTable.h
+++ b/llvm/utils/TableGen/GlobalISelMatchTable.h
@@ -526,6 +526,8 @@ class RuleMatcher : public Matcher {
std::make_unique<Kind>(std::forward<Args>(args)...));
}
+ void setPermanentGISelFlags(GISelFlags V) { Flags = V; }
+
// Update the active GISelFlags based on the GISelFlags Record R.
// A SaveAndRestore object is returned so the old GISelFlags are restored
// at the end of the scope.
@@ -648,6 +650,10 @@ template <class PredicateTy> class PredicateListMatcher {
}
bool predicates_empty() const { return Predicates.empty(); }
+ template <typename Ty> bool contains() const {
+ return any_of(Predicates, [&](auto &P) { return isa<Ty>(P.get()); });
+ }
+
std::unique_ptr<PredicateTy> predicates_pop_front() {
std::unique_ptr<PredicateTy> Front = std::move(Predicates.front());
Predicates.pop_front();
@@ -1930,23 +1936,41 @@ class TempRegRenderer : public OperandRenderer {
};
/// Adds a specific immediate to the instruction being built.
+/// If a LLT is passed, a ConstantInt immediate is created instead.
class ImmRenderer : public OperandRenderer {
protected:
unsigned InsnID;
int64_t Imm;
+ std::optional<LLTCodeGen> CImmLLT;
public:
ImmRenderer(unsigned InsnID, int64_t Imm)
: OperandRenderer(OR_Imm), InsnID(InsnID), Imm(Imm) {}
+ ImmRenderer(unsigned InsnID, int64_t Imm, const LLTCodeGen &CImmLLT)
+ : OperandRenderer(OR_Imm), InsnID(InsnID), Imm(Imm), CImmLLT(CImmLLT) {
+ KnownTypes.insert(CImmLLT);
+ }
+
static bool classof(const OperandRenderer *R) {
return R->getKind() == OR_Imm;
}
void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
- Table << MatchTable::Opcode("GIR_AddImm") << MatchTable::Comment("InsnID")
- << MatchTable::IntValue(InsnID) << MatchTable::Comment("Imm")
- << MatchTable::IntValue(Imm) << MatchTable::LineBreak;
+ if (CImmLLT) {
+ assert(Table.isCombiner() &&
+ "ConstantInt immediate are only for combiners!");
+ Table << MatchTable::Opcode("GIR_AddCImm")
+ << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
+ << MatchTable::Comment("Type")
+ << MatchTable::NamedValue(CImmLLT->getCxxEnumValue())
+ << MatchTable::Comment("Imm") << MatchTable::IntValue(Imm)
+ << MatchTable::LineBreak;
+ } else {
+ Table << MatchTable::Opcode("GIR_AddImm") << MatchTable::Comment("InsnID")
+ << MatchTable::IntValue(InsnID) << MatchTable::Comment("Imm")
+ << MatchTable::IntValue(Imm) << MatchTable::LineBreak;
+ }
}
};
diff --git a/llvm/utils/TableGen/GlobalISelMatchTableExecutorEmitter.cpp b/llvm/utils/TableGen/GlobalISelMatchTableExecutorEmitter.cpp
index 8dc422b140a5ee..ae3d85c717b1f8 100644
--- a/llvm/utils/TableGen/GlobalISelMatchTableExecutorEmitter.cpp
+++ b/llvm/utils/TableGen/GlobalISelMatchTableExecutorEmitter.cpp
@@ -222,7 +222,8 @@ void GlobalISelMatchTableExecutorEmitter::emitTemporariesDecl(
", const MatcherState &State) "
"const override;\n"
<< " bool testSimplePredicate(unsigned PredicateID) const override;\n"
- << " void runCustomAction(unsigned FnID, const MatcherState &State) "
+ << " void runCustomAction(unsigned FnID, const MatcherState &State, "
+ "NewMIVector &OutMIs) "
"const override;\n";
emitAdditionalTemporariesDecl(OS, " ");
OS << "#endif // ifdef " << IfDefName << "\n\n";
diff --git a/llvm/utils/TableGen/GlobalISelMatchTableExecutorEmitter.h b/llvm/utils/TableGen/GlobalISelMatchTableExecutorEmitter.h
index d526e08a96e310..0cde21f5f7d9b2 100644
--- a/llvm/utils/TableGen/GlobalISelMatchTableExecutorEmitter.h
+++ b/llvm/utils/TableGen/GlobalISelMatchTableExecutorEmitter.h
@@ -105,7 +105,8 @@ class GlobalISelMatchTableExecutorEmitter {
if (!Predicates.empty()) {
OS << " switch (PredicateID) {\n";
for (const auto &Pred : Predicates) {
- const auto Code = GetPredCode(Pred);
+ // Ensure all code is indented.
+ const auto Code = join(split(GetPredCode(Pred).str(), "\n"), "\n ");
OS << " case GICXXPred_" << TypeIdentifier << "_Predicate_"
<< GetPredEnumName(Pred) << ": {\n"
<< " " << Code << "\n";
More information about the llvm-commits
mailing list