[llvm-branch-commits] [clang] [llvm] [mlir] [CIR] Shard the dialect naturally along op partitions (PR #240479)
Henrich Lauko via llvm-branch-commits
llvm-branch-commits at lists.llvm.org
Mon Nov 9 09:47:39 PST 2026
https://github.com/xlauko created https://github.com/llvm/llvm-project/pull/240479
Draft stacked on #239254: it uses the op partitions that PR adds to ODS, so it only builds on top of that branch.
CIR currently splits its generated op definitions into 8 index shards, each of which still includes the whole `CIROps.h.inc`. This switches the dialect to natural sharding: `CIROps.td` assigns every op to one of eleven partitions (Core, Symbols, ControlFlow, Arith, Builtins, Math, Coroutine, EH, VectorMatrix, Complex, ClassMembers), each generated into its own `CIROps<Partition>.{h,cpp}.inc`, declared by a `CIROps<Partition>.h` on top of the new `CIRDialectBase.h` (dialect, attributes, types, interfaces, no ops), and compiled in its own `CIROps<Partition>.cpp` together with the hand-written op code moved out of `CIRDialect.cpp`. `CIRDialect.h` is now an umbrella over the partition headers, so includers are unchanged; files that use no op include `CIRDialectBase.h` instead. Generated `CIRLowering.inc` and the dialect doc are byte-identical to before.
Compiling the `MLIRCIR` objects (clang, Release+assertions, ccache off, one run each; sequential per-TU numbers summed, then one `-j16` build on a 16-thread machine):
| | base (8 shards) | partitions |
|---|---|---|
| CPU time | 109 s | 96 s (-12%) |
| `-j16` wall | 15.8 s | 14.5 s (-8%) |
| sum of max RSS | 9.6 GiB | 9.3 GiB (-4%) |
| largest TU max RSS | 0.83 GiB | 0.67 GiB (-19%) |
Assisted-by: Claude Code (Fable 5.1).
>From b5fc99bbaf7fc037e5cbb0f24567d23e37467ee1 Mon Sep 17 00:00:00 2001
From: Markus Boeck <mboeck at nvidia.com>
Date: Wed, 4 Nov 2026 15:17:01 +0000
Subject: [PATCH 1/5] [llvm][TableGen] Resolve intra-record field references to
`?`
An inconsistency existed in TableGen prior to this PR:
* Referencing a template argument whose value was `?` resolved to and propagated `?`
* Referencing a record whose field is `?` resolved and propagated `?`
However, referencing a field of the record being built remained unresolved, even in the final resolution pass.
This meant that code like:
```tablegen
class Self {
string str = ?;
assert !if(!initialized(str), !ne(str, ""), 1), "str must not be empty";
}
def : Self;
```
would fail to compile as `!initialized(str)` was left unfolded, and `!if` never received a bit value: https://godbolt.org/z/oGh1PzP88
Similar issues arose with other code such as with `!dump` etc.
This PR fixes that issue by resolving a record field to `?` if the record is complete and this is the final resolve pass. As a follow-on effect, some places that previously relied on being kept unresolved now resolve to `?` which required implementing semantics for `?.field` accesses (instead of the current crash using `llvm_unreachable`). This PR therefore also defines `?.field` as resolving to `?`, similar to `?{0}` or `!cast`.
---
llvm/docs/TableGen/BackEnds.md | 8 ++-
llvm/docs/TableGen/ProgRef.md | 3 +-
llvm/include/llvm/TableGen/Record.h | 3 +-
llvm/lib/TableGen/Record.cpp | 12 +++-
llvm/lib/TableGen/TGParser.cpp | 25 ++++++--
llvm/test/TableGen/JSON.td | 9 +--
llvm/test/TableGen/initialized.td | 34 ++++++++++
llvm/test/TableGen/unset-field-reference.td | 71 +++++++++++++++++++++
8 files changed, 147 insertions(+), 18 deletions(-)
create mode 100644 llvm/test/TableGen/unset-field-reference.td
diff --git a/llvm/docs/TableGen/BackEnds.md b/llvm/docs/TableGen/BackEnds.md
index 03df57b33cb27f..403b0c21ab176e 100644
--- a/llvm/docs/TableGen/BackEnds.md
+++ b/llvm/docs/TableGen/BackEnds.md
@@ -498,12 +498,16 @@ in `-print-records`.
- A reference to another variable in the same record has
`kind=="var"`, and has an extra field `var` giving the name of
- the variable referred to.
+ the variable referred to. Once the record is complete, a reference
+ to a variable that is still uninitialized (`?`) is replaced by `?`,
+ which is emitted as `null`.
- A reference to a specific bit of a `bits`-typed variable in the
same record has `kind=="varbit"`, and has two extra fields:
`var` gives the name of the variable referred to, and `index`
- gives the index of the bit.
+ gives the index of the bit. A reference to a bit that is still
+ uninitialized when the record is complete is kept, as such
+ references describe the operand fields of instruction encodings.
- A value of type `dag` has `kind=="dag"`, and has two extra
fields. `operator` gives the initial value after the opening
diff --git a/llvm/docs/TableGen/ProgRef.md b/llvm/docs/TableGen/ProgRef.md
index d70449eb71092a..a619733a3e19ee 100644
--- a/llvm/docs/TableGen/ProgRef.md
+++ b/llvm/docs/TableGen/ProgRef.md
@@ -596,7 +596,8 @@ primary value. Here are the possible suffixes for some primary *value*.
*value*`.`*field*
: The final value is the value of the specified *field* in the specified
- record *value*.
+ record *value*. If *value* is uninitialized (`?`), the result is `?` as
+ well.
### The paste operator
diff --git a/llvm/include/llvm/TableGen/Record.h b/llvm/include/llvm/TableGen/Record.h
index b3282145e878ef..e49464efe5b8f3 100644
--- a/llvm/include/llvm/TableGen/Record.h
+++ b/llvm/include/llvm/TableGen/Record.h
@@ -2391,7 +2391,8 @@ class MapResolver final : public Resolver {
void clearCachedResults() override { ResultCache.clear(); }
};
-/// Resolve all variables from a record except for unset variables.
+/// Resolve all variables from a record. In the final resolve, a reference to an
+/// unset field resolves to `?`; otherwise, it is kept.
class RecordResolver final : public Resolver {
DenseMap<const Init *, const Init *> Cache;
DenseMap<const Init *, std::pair<const Init *, bool>> ResultCache;
diff --git a/llvm/lib/TableGen/Record.cpp b/llvm/lib/TableGen/Record.cpp
index ec34fa71b0ea6d..d75b90b735aad5 100644
--- a/llvm/lib/TableGen/Record.cpp
+++ b/llvm/lib/TableGen/Record.cpp
@@ -2490,6 +2490,9 @@ const Init *FieldInit::getBit(unsigned Bit) const {
const Init *FieldInit::resolveReferences(Resolver &R) const {
const Init *NewRec = Rec->resolveReferences(R);
+ // A field of an unset record is unset.
+ if (isa<UnsetInit>(NewRec))
+ return NewRec;
if (NewRec != Rec)
return FieldInit::get(NewRec, FieldName)->Fold(R.getCurrentRecord());
return this;
@@ -3314,11 +3317,14 @@ const Init *RecordResolver::resolve(const Init *VarName) {
return nullptr; // prevent infinite recursion
if (const RecordVal *RV = getCurrentRecord()->getValue(VarName)) {
- if (!isa<UnsetInit>(RV->getValue())) {
- Val = RV->getValue();
+ const Init *Value = RV->getValue();
+ if (!isa<UnsetInit>(Value)) {
Stack.push_back(VarName);
- Val = Val->resolveReferences(*this);
+ Val = Value->resolveReferences(*this);
Stack.pop_back();
+ } else if (isFinal()) {
+ // The record is complete, so the field is unset.
+ Val = Value;
}
} else if (Name && VarName == getCurrentRecord()->getNameInit()) {
Stack.push_back(VarName);
diff --git a/llvm/lib/TableGen/TGParser.cpp b/llvm/lib/TableGen/TGParser.cpp
index d360fdf39e0ebc..337790dcc1cdd6 100644
--- a/llvm/lib/TableGen/TGParser.cpp
+++ b/llvm/lib/TableGen/TGParser.cpp
@@ -15,6 +15,7 @@
#include "llvm/ADT/STLExtras.h"
#include "llvm/ADT/Sequence.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/SmallVectorExtras.h"
#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringSwitch.h"
#include "llvm/ADT/Twine.h"
@@ -96,8 +97,13 @@ static bool checkBitsConcrete(Record &R, const RecordVal &RV) {
return true;
}
-static void checkConcrete(Record &R) {
- for (const RecordVal &RV : R.getValues()) {
+/// Reports the values of `R` that are not concrete after the final resolve.
+/// `UnresolvedValues` are the values of `R` before the final resolve, which
+/// replaced references to unset fields by `?`. The message shows the value
+/// with these references, so that unset field names are shown rather than `?`.
+static void checkConcrete(Record &R, ArrayRef<const Init *> UnresolvedValues) {
+ for (auto [RV, UnresolvedValue] :
+ zip_equal(R.getValues(), UnresolvedValues)) {
// HACK: Disable this check for variables declared with 'field'. This is
// done merely because existing targets have legitimate cases of
// non-concrete variables in helper defs. Ideally, we'd introduce a
@@ -108,10 +114,13 @@ static void checkConcrete(Record &R) {
const Init *V = RV.getValue();
bool Ok = isa<BitsInit>(V) ? checkBitsConcrete(R, RV) : V->isConcrete();
if (!Ok) {
- PrintError(R.getLoc(),
- Twine("Initializer of '") + RV.getNameInitAsString() +
- "' in '" + R.getNameInitAsString() +
- "' could not be fully resolved: " + V->getAsString());
+ // Before the final resolve, references to unset fields are kept.
+ RecordResolver Resolver(R);
+ PrintError(
+ R.getLoc(),
+ Twine("Initializer of '") + RV.getNameInitAsString() + "' in '" +
+ R.getNameInitAsString() + "' could not be fully resolved: " +
+ UnresolvedValue->resolveReferences(Resolver)->getAsString());
}
}
}
@@ -589,8 +598,10 @@ bool TGParser::addDefOne(std::unique_ptr<Record> Rec) {
NewName = Records.getNewAnonymousName();
}
+ SmallVector<const Init *> UnresolvedValues = map_to_vector(
+ Rec->getValues(), [](const RecordVal &RV) { return RV.getValue(); });
Rec->resolveReferences(NewName);
- checkConcrete(*Rec);
+ checkConcrete(*Rec, UnresolvedValues);
if (!isa<StringInit>(Rec->getNameInit())) {
PrintError(Rec->getLoc(), Twine("record name '") +
diff --git a/llvm/test/TableGen/JSON.td b/llvm/test/TableGen/JSON.td
index 4d8c3a466bfd07..be72065c3806b1 100644
--- a/llvm/test/TableGen/JSON.td
+++ b/llvm/test/TableGen/JSON.td
@@ -92,12 +92,13 @@ def VarObj : Variables {
// CHECK: data['VarObj']['d']['args'] == [[22, None], ["hello", "foo"]]
// CHECK: data['VarObj']['d']['printable'] == '(ExampleDagOp 22, "hello":$foo)'
+ // A reference to an unset variable is unset once the record is complete.
int undef_int;
field int ref_int = undef_int;
- // CHECK: data['VarObj']['ref_int']['kind'] == 'var'
- // CHECK: data['VarObj']['ref_int']['var'] == 'undef_int'
- // CHECK: data['VarObj']['ref_int']['printable'] == 'undef_int'
+ // CHECK: data['VarObj']['ref_int'] is None
+ // References to uninitialized bits are kept, as they describe the operand
+ // fields of instruction encodings.
bits<2> undef_bits;
bits<4> ref_bits;
let ref_bits{3...2} = 0b10;
@@ -115,7 +116,7 @@ def VarObj : Variables {
field int complex_ref_int = !add(undef_int, 2);
// CHECK: data['VarObj']['complex_ref_int']['kind'] == 'complex'
- // CHECK: data['VarObj']['complex_ref_int']['printable'] == '!add(undef_int, 2)'
+ // CHECK: data['VarObj']['complex_ref_int']['printable'] == '!add(?, 2)'
}
// Test the !anonymous member. This is tricky because when a def is
diff --git a/llvm/test/TableGen/initialized.td b/llvm/test/TableGen/initialized.td
index e1743909feb64f..585e5ea8ef822d 100644
--- a/llvm/test/TableGen/initialized.td
+++ b/llvm/test/TableGen/initialized.td
@@ -57,3 +57,37 @@ class Propagate<Thing t> {
// CHECK: Thing ret = aThing
def PropagateNothing : Propagate<?>;
def PropagateThing : Propagate<aThing>;
+
+// Once the record is complete, a reference to one of its unset fields is `?`:
+// `!initialized` folds, also in assertions, and a field of it is `?` as well.
+class Self {
+ string str = ?;
+ bit isInit = !initialized(str);
+ Y y = ?;
+ string yStr = y.str;
+ assert !if(!initialized(str), !ne(str, ""), 1), "str must not be empty";
+}
+class SelfSet : Self {
+ let str = "sub";
+}
+// CHECK-LABEL: def SelfLet {
+// CHECK: bit isInit = 1
+// CHECK-LABEL: def SelfLetIn {
+// CHECK: bit isInit = 1
+// CHECK-LABEL: def SelfSub {
+// CHECK: bit isInit = 1
+// CHECK-LABEL: def SelfUnset {
+// CHECK: bit isInit = 0
+// CHECK: string yStr = ?
+// CHECK-LABEL: def SelfY1 {
+// CHECK: string yStr = "foo"
+def SelfUnset : Self;
+def SelfLet : Self {
+ let str = "foo";
+}
+let str = "bar" in
+def SelfLetIn : Self;
+def SelfSub : SelfSet;
+def SelfY1 : Self {
+ let y = Y1;
+}
diff --git a/llvm/test/TableGen/unset-field-reference.td b/llvm/test/TableGen/unset-field-reference.td
new file mode 100644
index 00000000000000..35b446ef6ee1b0
--- /dev/null
+++ b/llvm/test/TableGen/unset-field-reference.td
@@ -0,0 +1,71 @@
+// RUN: llvm-tblgen %s | FileCheck %s
+// RUN: llvm-tblgen %s 2>&1 >/dev/null | FileCheck %s --check-prefix=DUMP
+// RUN: not llvm-tblgen -DERROR %s 2>&1 | FileCheck %s --check-prefix=ERROR
+
+// Once the record is complete, a reference to one of its unset fields is `?`,
+// like a template argument that is `?` or an unset field of another record.
+// (References to unset bits are kept, see JSON.td.)
+
+class Y {
+ string str = ?;
+}
+def Y1 : Y {
+ let str = "y1";
+}
+
+class IsSet<string s> {
+ bit ret = !initialized(s);
+}
+
+class Fields {
+ string str = ?;
+ Y y = ?;
+ // A field of `?` is `?`.
+ string yStr = y.str;
+ bit yStrIsSet = !initialized(y.str);
+ // A class is instantiated with `?` for an unset field.
+ bit strIsSet = IsSet<str>.ret;
+ // Dumps see `?`.
+ dump "str = " # !repr(str);
+}
+
+// CHECK-LABEL: def FieldsSet {
+// CHECK: string str = "s";
+// CHECK: Y y = Y1;
+// CHECK: string yStr = "y1";
+// CHECK: bit yStrIsSet = 1;
+// CHECK: bit strIsSet = 1;
+// CHECK-LABEL: def FieldsUnset {
+// CHECK: string str = ?;
+// CHECK: Y y = ?;
+// CHECK: string yStr = ?;
+// CHECK: bit yStrIsSet = 0;
+// CHECK: bit strIsSet = 0;
+// DUMP: note: str = ?
+// DUMP: note: str = "s"
+def FieldsUnset : Fields;
+def FieldsSet : Fields {
+ let str = "s";
+ let y = Y1;
+}
+
+// A field of a template argument that is `?` is `?` as well.
+class FromArg<Y y> {
+ string yStr = y.str;
+}
+// CHECK-LABEL: def FromSetArg {
+// CHECK: string yStr = "y1";
+// CHECK-LABEL: def FromUnsetArg {
+// CHECK: string yStr = ?;
+def FromUnsetArg : FromArg<?>;
+def FromSetArg : FromArg<Y1>;
+
+#ifdef ERROR
+// Assertions see `?`.
+class Asserted {
+ Y y = ?;
+ assert !initialized(y.str), "y.str is unset";
+}
+// ERROR: error: assertion failed: y.str is unset
+def AssertedUnset : Asserted;
+#endif
>From 2cc4f194d4c08c86b663fe3a6ed440716cca331b Mon Sep 17 00:00:00 2001
From: Markus Boeck <mboeck at nvidia.com>
Date: Thu, 5 Nov 2026 10:39:05 +0000
Subject: [PATCH 2/5] address review comments
---
llvm/docs/ReleaseNotes.md | 7 ++
llvm/test/TableGen/initialized.td | 34 ---------
llvm/test/TableGen/unset-field-reference.td | 79 ++++++++++++++++++++-
3 files changed, 85 insertions(+), 35 deletions(-)
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index 75318319d88b92..3978652e25970b 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -266,6 +266,13 @@ Makes programs 10x faster by doing Special New Thing.
* `!cond` operator short-circuits at the first `true` condition. Subsequent
`condition : value` pairs, along with their corresponding side effects,
are left unresolved.
+* Once a record is complete, a reference to one of its uninitialized fields
+ now resolves to `?`. As more values are resolved, more class instantiations
+ may occur: a class invoked with such a field (`Class<field>`) is now
+ instantiated with `?` as its template argument rather than left unresolved.
+ This creates additional anonymous records and runs the assertions and dumps
+ of the invoked class, which may fail for `?`. Use `!initialized` to guard
+ against such cases.
### Changes to Interprocedural Optimizations
diff --git a/llvm/test/TableGen/initialized.td b/llvm/test/TableGen/initialized.td
index 585e5ea8ef822d..e1743909feb64f 100644
--- a/llvm/test/TableGen/initialized.td
+++ b/llvm/test/TableGen/initialized.td
@@ -57,37 +57,3 @@ class Propagate<Thing t> {
// CHECK: Thing ret = aThing
def PropagateNothing : Propagate<?>;
def PropagateThing : Propagate<aThing>;
-
-// Once the record is complete, a reference to one of its unset fields is `?`:
-// `!initialized` folds, also in assertions, and a field of it is `?` as well.
-class Self {
- string str = ?;
- bit isInit = !initialized(str);
- Y y = ?;
- string yStr = y.str;
- assert !if(!initialized(str), !ne(str, ""), 1), "str must not be empty";
-}
-class SelfSet : Self {
- let str = "sub";
-}
-// CHECK-LABEL: def SelfLet {
-// CHECK: bit isInit = 1
-// CHECK-LABEL: def SelfLetIn {
-// CHECK: bit isInit = 1
-// CHECK-LABEL: def SelfSub {
-// CHECK: bit isInit = 1
-// CHECK-LABEL: def SelfUnset {
-// CHECK: bit isInit = 0
-// CHECK: string yStr = ?
-// CHECK-LABEL: def SelfY1 {
-// CHECK: string yStr = "foo"
-def SelfUnset : Self;
-def SelfLet : Self {
- let str = "foo";
-}
-let str = "bar" in
-def SelfLetIn : Self;
-def SelfSub : SelfSet;
-def SelfY1 : Self {
- let y = Y1;
-}
diff --git a/llvm/test/TableGen/unset-field-reference.td b/llvm/test/TableGen/unset-field-reference.td
index 35b446ef6ee1b0..248638d785284f 100644
--- a/llvm/test/TableGen/unset-field-reference.td
+++ b/llvm/test/TableGen/unset-field-reference.td
@@ -20,34 +20,55 @@ class IsSet<string s> {
class Fields {
string str = ?;
Y y = ?;
+ // `!initialized` folds.
+ bit strIsInit = !initialized(str);
// A field of `?` is `?`.
string yStr = y.str;
bit yStrIsSet = !initialized(y.str);
// A class is instantiated with `?` for an unset field.
bit strIsSet = IsSet<str>.ret;
- // Dumps see `?`.
+ // Dumps and assertions see `?`.
dump "str = " # !repr(str);
+ assert !if(!initialized(str), !ne(str, ""), 1), "str must not be empty";
+}
+class FieldsWithStr : Fields {
+ let str = "sub";
}
+// CHECK-LABEL: def FieldsLetIn {
+// CHECK: string str = "let-in";
+// CHECK: bit strIsInit = 1;
+// CHECK: bit strIsSet = 1;
// CHECK-LABEL: def FieldsSet {
// CHECK: string str = "s";
// CHECK: Y y = Y1;
+// CHECK: bit strIsInit = 1;
// CHECK: string yStr = "y1";
// CHECK: bit yStrIsSet = 1;
// CHECK: bit strIsSet = 1;
+// CHECK-LABEL: def FieldsSub {
+// CHECK: string str = "sub";
+// CHECK: bit strIsInit = 1;
+// CHECK: bit strIsSet = 1;
// CHECK-LABEL: def FieldsUnset {
// CHECK: string str = ?;
// CHECK: Y y = ?;
+// CHECK: bit strIsInit = 0;
// CHECK: string yStr = ?;
// CHECK: bit yStrIsSet = 0;
// CHECK: bit strIsSet = 0;
// DUMP: note: str = ?
// DUMP: note: str = "s"
+// DUMP: note: str = "let-in"
+// DUMP: note: str = "sub"
def FieldsUnset : Fields;
def FieldsSet : Fields {
let str = "s";
let y = Y1;
}
+let str = "let-in" in
+def FieldsLetIn : Fields;
+def FieldsSub : FieldsWithStr;
// A field of a template argument that is `?` is `?` as well.
class FromArg<Y y> {
@@ -60,6 +81,39 @@ class FromArg<Y y> {
def FromUnsetArg : FromArg<?>;
def FromSetArg : FromArg<Y1>;
+// Records defined by a multiclass are complete once instantiated by `defm`.
+multiclass MC<string s> {
+ def _Unset : Fields;
+ def _Set : Fields {
+ let str = s;
+ }
+}
+// CHECK-LABEL: def MC_Set {
+// CHECK: string str = "m";
+// CHECK: bit strIsInit = 1;
+// CHECK: bit strIsSet = 1;
+// CHECK-LABEL: def MC_Unset {
+// CHECK: string str = ?;
+// CHECK: bit strIsInit = 0;
+// CHECK: bit strIsSet = 0;
+defm MC : MC<"m">;
+
+// The anonymous record created by a class invocation is complete as well: its
+// unset fields are `?`, and the classes it invokes with them are instantiated
+// with `?`.
+class Optional<string fallback> {
+ string val = ?;
+ string ret = !if(!initialized(val), val, fallback);
+ bit valIsSet = IsSet<val>.ret;
+}
+// CHECK-LABEL: def UsesOptional {
+// CHECK: string ret = "fallback";
+// CHECK: bit valIsSet = 0;
+def UsesOptional {
+ string ret = Optional<"fallback">.ret;
+ bit valIsSet = Optional<"fallback">.valIsSet;
+}
+
#ifdef ERROR
// Assertions see `?`.
class Asserted {
@@ -68,4 +122,27 @@ class Asserted {
}
// ERROR: error: assertion failed: y.str is unset
def AssertedUnset : Asserted;
+
+// A class instantiated with an unset field sees `?`, also in its assertions,
+// even if its result is unused.
+class Required<int n> {
+ assert !initialized(n), "n is unset";
+ int ret = n;
+}
+class UsesRequired<int fallback> {
+ int val = ?;
+ int required = Required<val>.ret;
+ int ret = fallback;
+}
+// ERROR: error: assertion failed: n is unset
+def UsesRequiredUnset {
+ int ret = UsesRequired<3>.ret;
+}
+
+// The error shows the unset fields that a value depends on, not `?`.
+// ERROR: error: Initializer of 'b' in 'Unresolved' could not be fully resolved: !add(a, 1)
+def Unresolved {
+ int a;
+ int b = !add(a, 1);
+}
#endif
>From 215301bf0bf70c7ecf8d9d326bf017c889c7e595 Mon Sep 17 00:00:00 2001
From: Markus Boeck <mboeck at nvidia.com>
Date: Thu, 5 Nov 2026 10:43:15 +0000
Subject: [PATCH 3/5] address review comments
---
llvm/docs/TableGen/ProgRef.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/llvm/docs/TableGen/ProgRef.md b/llvm/docs/TableGen/ProgRef.md
index a619733a3e19ee..613775ed78acb0 100644
--- a/llvm/docs/TableGen/ProgRef.md
+++ b/llvm/docs/TableGen/ProgRef.md
@@ -596,8 +596,8 @@ primary value. Here are the possible suffixes for some primary *value*.
*value*`.`*field*
: The final value is the value of the specified *field* in the specified
- record *value*. If *value* is uninitialized (`?`), the result is `?` as
- well.
+ record *value*. If *value* resolves to uninitialized (`?`), for example a
+ template argument or field that is `?`, the result is `?` as well.
### The paste operator
>From be17f42296fb32d5b3fb1751a99d18453df824b8 Mon Sep 17 00:00:00 2001
From: Markus Boeck <mboeck at nvidia.com>
Date: Thu, 5 Nov 2026 12:33:43 +0000
Subject: [PATCH 4/5] [mlir][ODS] Add so-called "Natural Sharding" to ops
Prior to this PR, ODS had one mechanism for sharding which equally split a `<Dialect>Ops.cpp.inc` file into `n` many, as requested on the command line (using preprocessor defines for selective enablement).
The motivation for doing so is that this can speed up compile time by splitting class definition compilation to multiple translation units.
However, this approach has one large disadvantage: Since only the `<Dialect>Ops.cpp.inc` is split, the correpsonding `<Dialect>Ops.h.inc` is still included in each translation unit. For a large dialect like NVVM or LLVM this leads to **much higher** concurrent memory usage while compiling as every compilation process contains an in-memory copy of the header plus can lead to slower compilation time overall if too many shards are being used as overall work goes up (`n` header parses instead of 1).
This PR tries to address these issues by introducing a new kind of sharding called "natural sharding". The idea is that we split `<Dialect>Ops.cpp.inc` and `<Dialect>Ops.h.inc` along so called partitions. A new header can be created for every partition and every translation unit only contains the code from its partition. This means a TU no longer needs to include the entire dialect header but only the header from its partition.
One can think of these as effectively being sub-dialects from the perspective of the required boilerplate.
We call it "Natural Sharding", as the partitions are defined by the dialect authors and are meant to be categories that naturally categorizes a dialects operations (e.g., Memory ops, control flow ops, arithmetic ops, intrinsic ops, etc. in the LLVM Dialect). The new `-op-partition` then filters output generation to just a given partition.
This PR introduces the mechanism and for testing, already exercises it on the test dialect. Users of `TestOps.h` were further changed to just include a given partition if deemed useful.
Overall this yielded the following build gains vs the existing sharding, measured on a 16 thread machine (Release + assertions, 16 parallel jobs, median of 3 runs). "Users of `MLIRTestDialect`" are other TUs (unittests mainly) that include `MLIRTestDialect` headers. Total memory is the sum of the max RSS of every TU:
| TUs | Compiler | Wall time (s) | CPU time (s) | Total memory used during compilation (GiB) |
|----------------------------|----------|---------------------|--------------------|--------------------------------------------|
| `MLIRTestDialect` | clang | 49.9 -> 35.2 (-29%) | 710 -> 365 (-49%) | 23.3 -> 11.1 (-52%) |
| `MLIRTestDialect` | clang -g | 57.8 -> 43.1 (-25%) | 791 -> 427 (-46%) | 25.0 -> 12.5 (-50%) |
| `MLIRTestDialect` | gcc | 58.6 -> 43.8 (-25%) | 740 -> 395 (-47%) | 41.9 -> 19.4 (-54%) |
| `MLIRTestDialect` | gcc -g | 82.5 -> 61.5 (-25%) | 1059 -> 570 (-46%) | 52.9 -> 26.7 (-50%) |
| Users of `MLIRTestDialect` | clang | 45.4 -> 35.1 (-23%) | 626 -> 472 (-25%) | 27.6 -> 21.5 (-22%) |
| Users of `MLIRTestDialect` | clang -g | 46.6 -> 36.1 (-23%) | 644 -> 493 (-23%) | 28.4 -> 22.3 (-21%) |
| Users of `MLIRTestDialect` | gcc | 43.6 -> 33.9 (-22%) | 604 -> 449 (-26%) | 44.5 -> 32.8 (-26%) |
| Users of `MLIRTestDialect` | gcc -g | 60.9 -> 46.1 (-24%) | 859 -> 616 (-28%) | 57.1 -> 42.4 (-26%) |
Future plan is to introduce this for types and attributes as well
---
mlir/docs/DefiningDialects/Operations.md | 104 +++++
mlir/include/mlir/IR/DialectBase.td | 33 ++
mlir/include/mlir/IR/OpBase.td | 24 ++
mlir/include/mlir/TableGen/Dialect.h | 5 +
mlir/include/mlir/TableGen/Operator.h | 3 +
mlir/lib/TableGen/Dialect.cpp | 5 +
mlir/lib/TableGen/Operator.cpp | 5 +
.../TestDenseBackwardDataFlowAnalysis.cpp | 2 +-
.../TestDenseForwardDataFlowAnalysis.cpp | 3 +-
.../FuncToLLVM/TestConvertCallOp.cpp | 3 +-
.../Dialect/Affine/TestReifyValueBounds.cpp | 3 +-
.../TestOneShotModuleBufferize.cpp | 3 +-
.../lib/Dialect/DLTI/TestDataLayoutQuery.cpp | 2 +-
.../Func/TestDecomposeCallGraphTypes.cpp | 3 +-
mlir/test/lib/Dialect/Test/CMakeLists.txt | 21 +-
mlir/test/lib/Dialect/Test/TestDialect.cpp | 3 +-
mlir/test/lib/Dialect/Test/TestDialect.td | 20 +-
.../Test/TestFromLLVMIRTranslation.cpp | 3 +-
mlir/test/lib/Dialect/Test/TestOps.h | 250 +----------
mlir/test/lib/Dialect/Test/TestOps.td | 398 ++++++++++--------
.../{TestOps.cpp => TestOpsAttributes.cpp} | 9 +-
.../test/lib/Dialect/Test/TestOpsAttributes.h | 17 +
.../lib/Dialect/Test/TestOpsConversion.cpp | 15 +
.../test/lib/Dialect/Test/TestOpsConversion.h | 17 +
mlir/test/lib/Dialect/Test/TestOpsCore.cpp | 19 +
mlir/test/lib/Dialect/Test/TestOpsCore.h | 18 +
.../lib/Dialect/Test/TestOpsInterfaces.cpp | 16 +
.../test/lib/Dialect/Test/TestOpsInterfaces.h | 90 ++++
.../test/lib/Dialect/Test/TestOpsPatterns.cpp | 15 +
mlir/test/lib/Dialect/Test/TestOpsPatterns.h | 17 +
.../lib/Dialect/Test/TestOpsProperties.cpp | 16 +
.../test/lib/Dialect/Test/TestOpsProperties.h | 154 +++++++
mlir/test/lib/Dialect/Test/TestOpsSyntax.cpp | 10 +-
mlir/test/lib/Dialect/Test/TestOpsSyntax.td | 6 +-
mlir/test/lib/Dialect/Test/TestOpsTraits.cpp | 15 +
mlir/test/lib/Dialect/Test/TestOpsTraits.h | 17 +
.../Dialect/Test/TestToLLVMIRTranslation.cpp | 3 +-
mlir/test/lib/Dialect/Test/TestTraits.cpp | 2 +-
mlir/test/lib/IR/TestBytecodeRoundtrip.cpp | 3 +-
mlir/test/lib/IR/TestClone.cpp | 2 +-
mlir/test/lib/IR/TestLazyLoading.cpp | 1 -
mlir/test/lib/IR/TestRegions.cpp | 1 -
mlir/test/lib/IR/TestSideEffects.cpp | 2 +-
mlir/test/lib/IR/TestSymbolUses.cpp | 2 +-
mlir/test/lib/IR/TestTypes.cpp | 1 -
mlir/test/lib/IR/TestVisitorsGeneric.cpp | 2 +-
mlir/test/lib/Pass/TestPassManager.cpp | 3 +-
mlir/test/lib/Tools/PDLL/TestPDLL.cpp | 3 +-
mlir/test/lib/Transforms/TestInlining.cpp | 3 +-
.../lib/Transforms/TestInliningCallback.cpp | 4 +-
.../Transforms/TestMakeIsolatedFromAbove.cpp | 3 +-
mlir/test/mlir-tblgen/op-partitions-error.td | 74 ++++
mlir/test/mlir-tblgen/op-partitions.td | 110 +++++
mlir/tools/mlir-tblgen/DialectGen.cpp | 25 ++
mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp | 70 ++-
mlir/tools/mlir-tblgen/OpGenHelpers.cpp | 79 ++++
mlir/tools/mlir-tblgen/OpGenHelpers.h | 26 +-
mlir/unittests/IR/AdaptorTest.cpp | 3 +-
mlir/unittests/IR/IRMapping.cpp | 3 +-
mlir/unittests/IR/InterfaceAttachmentTest.cpp | 3 +-
mlir/unittests/IR/InterfaceTest.cpp | 3 +-
mlir/unittests/IR/OperationSupportTest.cpp | 3 +-
mlir/unittests/IR/PatternMatchTest.cpp | 3 +-
mlir/unittests/IR/ValueTest.cpp | 2 -
.../Interfaces/MemorySlotUtilsTest.cpp | 1 -
mlir/unittests/TableGen/OpBuildGen.cpp | 3 +-
66 files changed, 1293 insertions(+), 494 deletions(-)
rename mlir/test/lib/Dialect/Test/{TestOps.cpp => TestOpsAttributes.cpp} (65%)
create mode 100644 mlir/test/lib/Dialect/Test/TestOpsAttributes.h
create mode 100644 mlir/test/lib/Dialect/Test/TestOpsConversion.cpp
create mode 100644 mlir/test/lib/Dialect/Test/TestOpsConversion.h
create mode 100644 mlir/test/lib/Dialect/Test/TestOpsCore.cpp
create mode 100644 mlir/test/lib/Dialect/Test/TestOpsCore.h
create mode 100644 mlir/test/lib/Dialect/Test/TestOpsInterfaces.cpp
create mode 100644 mlir/test/lib/Dialect/Test/TestOpsInterfaces.h
create mode 100644 mlir/test/lib/Dialect/Test/TestOpsPatterns.cpp
create mode 100644 mlir/test/lib/Dialect/Test/TestOpsPatterns.h
create mode 100644 mlir/test/lib/Dialect/Test/TestOpsProperties.cpp
create mode 100644 mlir/test/lib/Dialect/Test/TestOpsProperties.h
create mode 100644 mlir/test/lib/Dialect/Test/TestOpsTraits.cpp
create mode 100644 mlir/test/lib/Dialect/Test/TestOpsTraits.h
create mode 100644 mlir/test/mlir-tblgen/op-partitions-error.td
create mode 100644 mlir/test/mlir-tblgen/op-partitions.td
diff --git a/mlir/docs/DefiningDialects/Operations.md b/mlir/docs/DefiningDialects/Operations.md
index f991f5308c5ae4..7c0228772d2488 100644
--- a/mlir/docs/DefiningDialects/Operations.md
+++ b/mlir/docs/DefiningDialects/Operations.md
@@ -1370,6 +1370,110 @@ cc_library(
)
```
+#### Natural Sharding with Op Partitions
+
+Sharding by index splits the ops arbitrarily: which shard an op ends up in
+depends on the number of shards and on the names of all other ops. As an
+alternative, a dialect can shard its ops "naturally" by grouping them into
+named *op partitions*, such as arithmetic, memory or control flow ops. Each
+partition is generated into its own declaration and definition files, almost as
+if it were a dialect of its own, but all ops are still registered in one
+dialect.
+
+Partitions are `Partition` records with a name, which must be a C++
+identifier. The dialect lists its partitions, and every op sets its `partition`
+to one of these records (a different record with the same name is rejected),
+typically with a `let` around a group of ops:
+
+```tablegen
+def MyDialect_ArithPartition : Partition<"Arith">;
+def MyDialect_MemoryPartition : Partition<"Memory">;
+
+def MyDialect : Dialect {
+ let name = "my_dialect";
+ let opPartitions = [MyDialect_ArithPartition, MyDialect_MemoryPartition];
+}
+
+let partition = MyDialect_ArithPartition in {
+def AddOp : MyDialect_Op<"add"> { ... }
+def MulOp : MyDialect_Op<"mul"> { ... }
+}
+
+def LoadOp : MyDialect_Op<"load"> {
+ let partition = MyDialect_MemoryPartition;
+ ...
+}
+```
+
+A dialect can also name one of its partitions as `defaultOpPartition`. Ops that
+do not set `partition` are then in this partition. This allows partitioning a
+dialect incrementally: initially all ops are in the default partition, and
+groups of ops can move to their own partitions one at a time:
+
+```tablegen
+def MyDialect : Dialect {
+ let name = "my_dialect";
+ let opPartitions = [MyDialect_CorePartition, MyDialect_MemoryPartition];
+ let defaultOpPartition = MyDialect_CorePartition;
+}
+
+// In partition `Core`.
+def AddOp : MyDialect_Op<"add"> { ... }
+```
+
+Passing `-op-partition=<partition>` with a partition name to `-gen-op-decls` and
+`-gen-op-defs` generates only the ops of that partition.
+
+Without `-op-partition`, `-gen-op-defs` generates all ops as before and
+additionally defines the registration hooks of all partitions in
+`GET_OP_CLASSES`. The hooks are private members of the dialect class, and the
+dialect registers the ops of all partitions by calling `registerOperations()`
+in its `initialize()`. A dialect can therefore declare its partitions and
+register its ops with `registerOperations()` first, and split its generated
+files by partition later.
+
+`mlir-tblgen` reports an error if a requested partition does not exist. A
+partition for which no definitions are compiled results in a link error for its
+hook.
+
+In CMake, generate and include the files of each partition:
+
+```cmake
+set(LLVM_TARGET_DEFINITIONS MyDialectOps.td)
+foreach(partition Arith Memory)
+ mlir_tablegen(MyDialect${partition}Ops.h.inc -gen-op-decls -op-partition=${partition})
+ mlir_tablegen(MyDialect${partition}Ops.cpp.inc -gen-op-defs -op-partition=${partition})
+endforeach()
+```
+
+Since the generated declarations of a partition forward declare the ops of all
+other partitions, each partition can get its own header, so that users only
+include the declarations of the ops they use:
+
+```c++
+// MyDialectArithOps.h
+#include "MyDialectBase.h" // The dialect, its attributes and types.
+
+#define GET_OP_CLASSES
+#include "MyDialectArithOps.h.inc"
+
+// MyDialect.h, for users that want all ops.
+#include "MyDialectArithOps.h"
+#include "MyDialectMemoryOps.h"
+
+// MyDialectArithOps.cpp, a natural home for the C++ implementations of the
+// arithmetic ops as well.
+#include "MyDialectArithOps.h"
+#include "OpUtils.h"
+
+#define GET_OP_CLASSES
+#include "MyDialectArithOps.cpp.inc"
+```
+
+As with index sharding, utilities shared by the generated op definitions must
+be declared in a header shared by the compilation units. A dialect with op
+partitions cannot use `-op-shard-count`.
+
## Constraints
Constraint is a core concept in table-driven operation definition: operation
diff --git a/mlir/include/mlir/IR/DialectBase.td b/mlir/include/mlir/IR/DialectBase.td
index da669c6ada639f..92b6601b7e2877 100644
--- a/mlir/include/mlir/IR/DialectBase.td
+++ b/mlir/include/mlir/IR/DialectBase.td
@@ -19,6 +19,17 @@ include "mlir/IR/Utils.td"
// Dialect definitions
//===----------------------------------------------------------------------===//
+// A partition of the ops of a dialect, for "natural sharding" of the generated
+// op declarations and definitions. See `Dialect::opPartitions`.
+class Partition<string name> {
+ // The name of the partition, a C++ identifier. `-gen-op-decls` and
+ // `-gen-op-defs` select a partition by this name with `-op-partition=...`.
+ string partitionName = name;
+
+ assert !match(partitionName, "^[A-Za-z_][A-Za-z0-9_]*$"),
+ "op partition name '" # partitionName # "' is not a C++ identifier";
+}
+
class Dialect {
// The name of the dialect.
string name = ?;
@@ -99,6 +110,28 @@ class Dialect {
// If this dialect can be extended at runtime with new operations or types.
bit isExtensible = 0;
+
+ // The op partitions of this dialect, enabling "natural sharding" of its op
+ // declarations and definitions (see `Op::partition`). When non-empty, every
+ // op of the dialect must be in one of these partitions.
+ list<Partition> opPartitions = [];
+
+ // Optional. The partition of the ops of this dialect that do not set their
+ // `partition`. Must be one of `opPartitions`. This allows partitioning the
+ // ops of a dialect incrementally: all ops start in the default partition and
+ // groups of ops can move to other partitions one at a time.
+ Partition defaultOpPartition = ?;
+
+ assert !if(!initialized(defaultOpPartition),
+ !not(!empty(!filter(p, opPartitions,
+ !eq(p, defaultOpPartition)))),
+ 1),
+ "default op partition '" #
+ !if(!initialized(defaultOpPartition),
+ defaultOpPartition.partitionName # "' (record '" #
+ !cast<string>(defaultOpPartition), "") #
+ "') of dialect '" # name #
+ "' is not one of its `opPartitions`";
}
#endif // DIALECTBASE_TD
diff --git a/mlir/include/mlir/IR/OpBase.td b/mlir/include/mlir/IR/OpBase.td
index a130d5847a1c72..3357f83e922bba 100644
--- a/mlir/include/mlir/IR/OpBase.td
+++ b/mlir/include/mlir/IR/OpBase.td
@@ -351,6 +351,30 @@ class Op<Dialect dialect, string mnemonic, list<Trait> props = []> {
// Optional. The group of ops this op is part of.
OpDocGroup opDocGroup = ?;
+ // Optional. The partition of this op. A dialect that lists its partitions in
+ // `opPartitions` is "naturally sharded": every op of the dialect must be in
+ // one of the listed partitions, `-gen-op-decls`/`-gen-op-defs` can generate
+ // the ops of one partition at a time with `-op-partition=...` into separate
+ // files.
+ // Defaults to the dialect's `defaultOpPartition`. The partition must be one
+ // of the `opPartitions` records of the dialect.
+ Partition partition = dialect.defaultOpPartition;
+
+ assert !if(!initialized(partition),
+ !not(!empty(!filter(p, opDialect.opPartitions,
+ !eq(p, partition)))),
+ 1),
+ "op '" # NAME # "' is in partition '" #
+ !if(!initialized(partition),
+ partition.partitionName # "' (record '" #
+ !cast<string>(partition), "") #
+ "'), which is not one of the `opPartitions` of dialect '" #
+ opDialect.name # "'";
+
+ assert !or(!initialized(partition), !empty(opDialect.opPartitions)),
+ "op '" # NAME # "' is in no partition, but dialect '" #
+ opDialect.name # "' has `opPartitions` and no `defaultOpPartition`";
+
// Dag containing the arguments of the op. Default to 0 arguments.
dag arguments = (ins);
diff --git a/mlir/include/mlir/TableGen/Dialect.h b/mlir/include/mlir/TableGen/Dialect.h
index 68230196b17cc0..7913360c294e48 100644
--- a/mlir/include/mlir/TableGen/Dialect.h
+++ b/mlir/include/mlir/TableGen/Dialect.h
@@ -94,6 +94,10 @@ class Dialect {
const llvm::DagInit *getDiscardableAttributes() const;
+ /// Returns the names of the op partitions of this dialect, see
+ /// `Op::partition`. A dialect with op partitions is "naturally sharded".
+ ArrayRef<StringRef> getOpPartitions() const;
+
const llvm::Record *getDef() const { return def; }
// Returns whether two dialects are equal by checking the equality of the
@@ -112,6 +116,7 @@ class Dialect {
private:
const llvm::Record *def;
std::vector<StringRef> dependentDialects;
+ std::vector<StringRef> opPartitions;
};
} // namespace tblgen
} // namespace mlir
diff --git a/mlir/include/mlir/TableGen/Operator.h b/mlir/include/mlir/TableGen/Operator.h
index bb36c077708e92..cd321dcd6ca6f3 100644
--- a/mlir/include/mlir/TableGen/Operator.h
+++ b/mlir/include/mlir/TableGen/Operator.h
@@ -330,6 +330,9 @@ class Operator {
/// Returns the dialect of the op.
const Dialect &getDialect() const { return dialect; }
+ /// Returns the name of the op's partition, or an empty string if it has none.
+ StringRef getPartitionName() const;
+
/// Prints the contents in this operator to the given `os`. This is used for
/// debugging purposes.
void print(llvm::raw_ostream &os) const;
diff --git a/mlir/lib/TableGen/Dialect.cpp b/mlir/lib/TableGen/Dialect.cpp
index 84530bfa4f6fd2..a9f9b49b663ead 100644
--- a/mlir/lib/TableGen/Dialect.cpp
+++ b/mlir/lib/TableGen/Dialect.cpp
@@ -21,6 +21,9 @@ Dialect::Dialect(const llvm::Record *def) : def(def) {
return;
for (StringRef dialect : def->getValueAsListOfStrings("dependentDialects"))
dependentDialects.push_back(dialect);
+ for (const llvm::Record *partition :
+ def->getValueAsListOfDefs("opPartitions"))
+ opPartitions.push_back(partition->getValueAsString("partitionName"));
}
StringRef Dialect::getName() const { return def->getValueAsString("name"); }
@@ -110,6 +113,8 @@ const llvm::DagInit *Dialect::getDiscardableAttributes() const {
return def->getValueAsDag("discardableAttrs");
}
+ArrayRef<StringRef> Dialect::getOpPartitions() const { return opPartitions; }
+
bool Dialect::operator==(const Dialect &other) const {
return def == other.def;
}
diff --git a/mlir/lib/TableGen/Operator.cpp b/mlir/lib/TableGen/Operator.cpp
index f01820bcf411c3..eb9bdb1b2b5e70 100644
--- a/mlir/lib/TableGen/Operator.cpp
+++ b/mlir/lib/TableGen/Operator.cpp
@@ -151,6 +151,11 @@ void Operator::assertInvariants() const {
StringRef Operator::getDialectName() const { return dialect.getName(); }
+StringRef Operator::getPartitionName() const {
+ const llvm::Record *partition = def.getValueAsOptionalDef("partition");
+ return partition ? partition->getValueAsString("partitionName") : "";
+}
+
StringRef Operator::getCppClassName() const { return cppClassName; }
std::string Operator::getQualCppClassName() const {
diff --git a/mlir/test/lib/Analysis/DataFlow/TestDenseBackwardDataFlowAnalysis.cpp b/mlir/test/lib/Analysis/DataFlow/TestDenseBackwardDataFlowAnalysis.cpp
index f969869b924824..5bb7fbaaac6815 100644
--- a/mlir/test/lib/Analysis/DataFlow/TestDenseBackwardDataFlowAnalysis.cpp
+++ b/mlir/test/lib/Analysis/DataFlow/TestDenseBackwardDataFlowAnalysis.cpp
@@ -11,7 +11,7 @@
//===----------------------------------------------------------------------===//
#include "TestDenseDataFlowAnalysis.h"
-#include "TestOps.h"
+#include "TestOpsInterfaces.h"
#include "mlir/Analysis/DataFlow/DenseAnalysis.h"
#include "mlir/Analysis/DataFlow/Utils.h"
#include "mlir/Analysis/DataFlowFramework.h"
diff --git a/mlir/test/lib/Analysis/DataFlow/TestDenseForwardDataFlowAnalysis.cpp b/mlir/test/lib/Analysis/DataFlow/TestDenseForwardDataFlowAnalysis.cpp
index c8b67b4ba61c00..725daef9d08e3e 100644
--- a/mlir/test/lib/Analysis/DataFlow/TestDenseForwardDataFlowAnalysis.cpp
+++ b/mlir/test/lib/Analysis/DataFlow/TestDenseForwardDataFlowAnalysis.cpp
@@ -11,8 +11,7 @@
//===----------------------------------------------------------------------===//
#include "TestDenseDataFlowAnalysis.h"
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsInterfaces.h"
#include "mlir/Analysis/DataFlow/DenseAnalysis.h"
#include "mlir/Analysis/DataFlow/Utils.h"
#include "mlir/Interfaces/SideEffectInterfaces.h"
diff --git a/mlir/test/lib/Conversion/FuncToLLVM/TestConvertCallOp.cpp b/mlir/test/lib/Conversion/FuncToLLVM/TestConvertCallOp.cpp
index 3b29867cc32631..89e02814950fd8 100644
--- a/mlir/test/lib/Conversion/FuncToLLVM/TestConvertCallOp.cpp
+++ b/mlir/test/lib/Conversion/FuncToLLVM/TestConvertCallOp.cpp
@@ -6,8 +6,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsConversion.h"
#include "TestTypes.h"
#include "mlir/Conversion/FuncToLLVM/ConvertFuncToLLVM.h"
#include "mlir/Conversion/LLVMCommon/Pattern.h"
diff --git a/mlir/test/lib/Dialect/Affine/TestReifyValueBounds.cpp b/mlir/test/lib/Dialect/Affine/TestReifyValueBounds.cpp
index f11c14bc09fb66..efd8ee7a8ebf3f 100644
--- a/mlir/test/lib/Dialect/Affine/TestReifyValueBounds.cpp
+++ b/mlir/test/lib/Dialect/Affine/TestReifyValueBounds.cpp
@@ -6,8 +6,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsInterfaces.h"
#include "mlir/Dialect/Affine/IR/AffineOps.h"
#include "mlir/Dialect/Affine/IR/ValueBoundsOpInterfaceImpl.h"
#include "mlir/Dialect/Affine/Transforms/Transforms.h"
diff --git a/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp b/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp
index d9f50ba0ee1ff4..634469278fa847 100644
--- a/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp
+++ b/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp
@@ -15,8 +15,7 @@
#include "mlir/Pass/Pass.h"
#include "TestAttributes.h" // TestTensorEncodingAttr, TestMemRefLayoutAttr
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsInterfaces.h"
#include "llvm/ADT/TypeSwitch.h"
diff --git a/mlir/test/lib/Dialect/DLTI/TestDataLayoutQuery.cpp b/mlir/test/lib/Dialect/DLTI/TestDataLayoutQuery.cpp
index 9fb89d3b4528b0..d0ca7fa5c2506f 100644
--- a/mlir/test/lib/Dialect/DLTI/TestDataLayoutQuery.cpp
+++ b/mlir/test/lib/Dialect/DLTI/TestDataLayoutQuery.cpp
@@ -6,7 +6,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestOps.h"
+#include "TestOpsInterfaces.h"
#include "mlir/Analysis/DataLayoutAnalysis.h"
#include "mlir/Dialect/DLTI/DLTI.h"
#include "mlir/IR/BuiltinAttributes.h"
diff --git a/mlir/test/lib/Dialect/Func/TestDecomposeCallGraphTypes.cpp b/mlir/test/lib/Dialect/Func/TestDecomposeCallGraphTypes.cpp
index c67bcd94ea0bd6..0f789a8c822e21 100644
--- a/mlir/test/lib/Dialect/Func/TestDecomposeCallGraphTypes.cpp
+++ b/mlir/test/lib/Dialect/Func/TestDecomposeCallGraphTypes.cpp
@@ -6,8 +6,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsInterfaces.h"
#include "mlir/Dialect/Func/IR/FuncOps.h"
#include "mlir/Dialect/Func/Transforms/FuncConversions.h"
#include "mlir/IR/Builders.h"
diff --git a/mlir/test/lib/Dialect/Test/CMakeLists.txt b/mlir/test/lib/Dialect/Test/CMakeLists.txt
index ca0024b8434e16..d2dab9e88304fd 100644
--- a/mlir/test/lib/Dialect/Test/CMakeLists.txt
+++ b/mlir/test/lib/Dialect/Test/CMakeLists.txt
@@ -34,15 +34,18 @@ set(LLVM_TARGET_DEFINITIONS TestOps.td)
mlir_tablegen(TestOpsDialect.h.inc -gen-dialect-decls -dialect=test)
mlir_tablegen(TestOpsDialect.cpp.inc -gen-dialect-defs -dialect=test)
mlir_tablegen(TestPatterns.inc -gen-rewriters)
+# The op partitions of TestOps.td, each with its own header and source file.
+foreach(partition Core Attributes Traits Patterns Conversion Interfaces Properties)
+ mlir_tablegen(TestOps${partition}.h.inc -gen-op-decls -op-partition=${partition})
+ mlir_tablegen(TestOps${partition}.cpp.inc -gen-op-defs -op-partition=${partition})
+endforeach()
add_public_tablegen_target(MLIRTestOpsIncGen)
set(LLVM_TARGET_DEFINITIONS TestOpsSyntax.td)
-mlir_tablegen(TestOpsSyntax.h.inc -gen-op-decls)
-mlir_tablegen(TestOpsSyntax.cpp.inc -gen-op-defs)
+mlir_tablegen(TestOpsSyntax.h.inc -gen-op-decls -op-partition=Syntax)
+mlir_tablegen(TestOpsSyntax.cpp.inc -gen-op-defs -op-partition=Syntax)
add_public_tablegen_target(MLIRTestOpsSyntaxIncGen)
-add_sharded_ops(TestOps 20)
-
# Exclude tests from libMLIR.so
add_mlir_library(MLIRTestDialect
TestAttributes.cpp
@@ -50,13 +53,18 @@ add_mlir_library(MLIRTestDialect
TestFormatUtils.cpp
TestInterfaces.cpp
TestOpDefs.cpp
- TestOps.cpp
+ TestOpsAttributes.cpp
+ TestOpsConversion.cpp
+ TestOpsCore.cpp
+ TestOpsInterfaces.cpp
+ TestOpsPatterns.cpp
+ TestOpsProperties.cpp
+ TestOpsTraits.cpp
TestPatterns.cpp
TestTraits.cpp
TestTypes.cpp
TestOpsSyntax.cpp
TestDialectInterfaces.cpp
- ${SHARDED_SRCS}
EXCLUDE_FROM_LIBMLIR
@@ -67,7 +75,6 @@ add_mlir_library(MLIRTestDialect
MLIRTestTypeDefIncGen
MLIRTestOpsIncGen
MLIRTestOpsSyntaxIncGen
- MLIRTestOpsShardGen
MLIRConvertToEmitCPatternInterfaceIncGen
)
mlir_target_link_libraries(MLIRTestDialect PUBLIC
diff --git a/mlir/test/lib/Dialect/Test/TestDialect.cpp b/mlir/test/lib/Dialect/Test/TestDialect.cpp
index 57bba2e5749a86..657f25771a45ee 100644
--- a/mlir/test/lib/Dialect/Test/TestDialect.cpp
+++ b/mlir/test/lib/Dialect/Test/TestDialect.cpp
@@ -346,9 +346,8 @@ struct TestOpEffectInterfaceFallback
void TestDialect::initialize() {
registerAttributes();
registerTypes();
- registerOpsSyntax();
addOperations<ManualCppOpWithFold>();
- registerTestDialectOperations(this);
+ registerOperations();
registerDynamicOp(getDynamicGenericOp(this));
registerDynamicOp(getDynamicTerminatorOp(this));
registerDynamicOp(getDynamicNoTerminatorOp(this));
diff --git a/mlir/test/lib/Dialect/Test/TestDialect.td b/mlir/test/lib/Dialect/Test/TestDialect.td
index 37a263f1d10b8b..debb239addc056 100644
--- a/mlir/test/lib/Dialect/Test/TestDialect.td
+++ b/mlir/test/lib/Dialect/Test/TestDialect.td
@@ -11,6 +11,19 @@
include "mlir/IR/OpBase.td"
+// Op partitions, each generated and compiled separately. Ops from TestOps.td
+// are in `Core` unless a `let` block or their base class picks another
+// partition. All ops from TestOpsSyntax.td are in `Syntax`, which is generated
+// from that file.
+def Test_CorePartition : Partition<"Core">;
+def Test_AttributesPartition : Partition<"Attributes">;
+def Test_TraitsPartition : Partition<"Traits">;
+def Test_PatternsPartition : Partition<"Patterns">;
+def Test_ConversionPartition : Partition<"Conversion">;
+def Test_InterfacesPartition : Partition<"Interfaces">;
+def Test_PropertiesPartition : Partition<"Properties">;
+def Test_SyntaxPartition : Partition<"Syntax">;
+
def Test_Dialect : Dialect {
let name = "test";
let cppNamespace = "::test";
@@ -24,6 +37,12 @@ def Test_Dialect : Dialect {
let useDefaultTypePrinterParser = 0;
let useDefaultAttributePrinterParser = 1;
let isExtensible = 1;
+ let opPartitions = [
+ Test_CorePartition, Test_AttributesPartition, Test_TraitsPartition,
+ Test_PatternsPartition, Test_ConversionPartition, Test_InterfacesPartition,
+ Test_PropertiesPartition, Test_SyntaxPartition
+ ];
+ let defaultOpPartition = Test_CorePartition;
let dependentDialects = [
"::mlir::DLTIDialect",
"::mlir::bufferization::BufferizationDialect"
@@ -37,7 +56,6 @@ def Test_Dialect : Dialect {
void registerAttributes();
void registerInterfaces();
void registerTypes();
- void registerOpsSyntax();
// Provides a custom printing/parsing for some operations.
::std::optional<ParseOpHook>
diff --git a/mlir/test/lib/Dialect/Test/TestFromLLVMIRTranslation.cpp b/mlir/test/lib/Dialect/Test/TestFromLLVMIRTranslation.cpp
index b98f6ce32cdceb..c1f8629c65df42 100644
--- a/mlir/test/lib/Dialect/Test/TestFromLLVMIRTranslation.cpp
+++ b/mlir/test/lib/Dialect/Test/TestFromLLVMIRTranslation.cpp
@@ -10,8 +10,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsTraits.h"
#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
#include "mlir/IR/Builders.h"
#include "mlir/IR/BuiltinAttributes.h"
diff --git a/mlir/test/lib/Dialect/Test/TestOps.h b/mlir/test/lib/Dialect/Test/TestOps.h
index 5fcee322e9e2e6..8862e81a03c79d 100644
--- a/mlir/test/lib/Dialect/Test/TestOps.h
+++ b/mlir/test/lib/Dialect/Test/TestOps.h
@@ -5,247 +5,21 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
+//
+// Declares all ops of TestOps.td. Each op partition has its own header, which
+// declares the ops of that partition and forward declares the others.
+//
+//===----------------------------------------------------------------------===//
#ifndef MLIR_TESTOPS_H
#define MLIR_TESTOPS_H
-#include "TestAttributes.h"
-#include "TestTypes.h"
-#include "mlir/Bytecode/BytecodeImplementation.h"
-#include "mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h"
-#include "mlir/Dialect/DLTI/DLTI.h"
-#include "mlir/Dialect/DLTI/Traits.h"
-#include "mlir/Dialect/Func/IR/FuncOps.h"
-#include "mlir/Dialect/LLVMIR/NVVMRequiresSMTraits.h"
-#include "mlir/Dialect/Linalg/IR/Linalg.h"
-#include "mlir/Dialect/Linalg/IR/LinalgInterfaces.h"
-#include "mlir/Dialect/Traits.h"
-#include "mlir/IR/AsmState.h"
-#include "mlir/IR/BuiltinOps.h"
-#include "mlir/IR/BuiltinTypes.h"
-#include "mlir/IR/Dialect.h"
-#include "mlir/IR/DialectResourceBlobManager.h"
-#include "mlir/IR/ExtensibleDialect.h"
-#include "mlir/IR/OpImplementation.h"
-#include "mlir/IR/RegionKindInterface.h"
-#include "mlir/IR/SymbolTable.h"
-#include "mlir/Interfaces/CallInterfaces.h"
-#include "mlir/Interfaces/ControlFlowInterfaces.h"
-#include "mlir/Interfaces/InferIntRangeInterface.h"
-#include "mlir/Interfaces/InferTypeOpInterface.h"
-#include "mlir/Interfaces/LoopLikeInterface.h"
-#include "mlir/Interfaces/MemorySlotInterfaces.h"
-#include "mlir/Interfaces/SideEffectInterfaces.h"
-#include "mlir/Interfaces/ValueBoundsOpInterface.h"
-#include "mlir/Interfaces/ViewLikeInterface.h"
-#include "llvm/ADT/SetVector.h"
-#include "llvm/ADT/SmallVector.h"
-#include <optional>
-
-namespace test {
-// Keep this type distinct from SmallVector<int32_t>, which another test parser
-// uses with the generic FieldParser specialization in a separate translation
-// unit.
-struct KeyValueSpecializedList : llvm::SmallVector<int32_t> {
- using llvm::SmallVector<int32_t>::SmallVector;
-};
-} // namespace test
-
-namespace mlir {
-// Self-delimiting full specializations remain usable in a keyed prop-dict even
-// when their storage type is optional or container-like.
-template <>
-struct FieldParser<test::KeyValueSpecializedList> {
- static FailureOr<test::KeyValueSpecializedList> parse(AsmParser &parser) {
- test::KeyValueSpecializedList values;
- if (parser.parseCommaSeparatedList(AsmParser::Delimiter::Square, [&]() {
- int32_t value;
- if (parser.parseInteger(value))
- return failure();
- values.push_back(value);
- return success();
- }))
- return failure();
- return values;
- }
-};
-
-template <>
-struct FieldParser<std::optional<int16_t>> {
- static FailureOr<std::optional<int16_t>> parse(AsmParser &parser) {
- if (succeeded(parser.parseOptionalKeyword("none")))
- return std::optional<int16_t>{};
- int16_t value;
- if (parser.parseKeyword("some") || parser.parseLess() ||
- parser.parseInteger(value) || parser.parseGreater())
- return failure();
- return std::optional<int16_t>{value};
- }
-};
-} // namespace mlir
-
-namespace test {
-class TestDialect;
-
-//===----------------------------------------------------------------------===//
-// TestResource
-//===----------------------------------------------------------------------===//
-
-/// A test resource for side effects (under DefaultResource).
-struct TestResource : public mlir::SideEffects::Resource::Base<
- TestResource, mlir::SideEffects::DefaultResource> {
- llvm::StringRef getName() const final { return "<Test>"; }
- mlir::SideEffects::Resource *getParent() const override {
- return mlir::SideEffects::DefaultResource::get();
- }
-};
-
-/// A test resource that is a root (disjoint from DefaultResource).
-struct TestNonAddressableResource
- : public mlir::SideEffects::Resource::Base<TestNonAddressableResource> {
- llvm::StringRef getName() const final { return "<TestNonAddressable>"; }
- bool isAddressable() const override { return false; }
-};
-
-/// Two disjoint sub-resources (roots) for testing sibling disjointness.
-struct TestNonAddressableSubResourceA
- : public mlir::SideEffects::Resource::Base<TestNonAddressableSubResourceA> {
- TestNonAddressableSubResourceA() = default;
- llvm::StringRef getName() const override {
- return "TestNonAddressableSubResourceA";
- }
- bool isAddressable() const override { return false; }
-
-protected:
- TestNonAddressableSubResourceA(mlir::TypeID id) : Base(id) {}
-};
-
-struct TestNonAddressableSubResourceB
- : public mlir::SideEffects::Resource::Base<TestNonAddressableSubResourceB> {
- TestNonAddressableSubResourceB() = default;
- llvm::StringRef getName() const override {
- return "TestNonAddressableSubResourceB";
- }
- bool isAddressable() const override { return false; }
-
-protected:
- TestNonAddressableSubResourceB(mlir::TypeID id) : Base(id) {}
-};
-
-struct TestNonAddressableResourceA
- : public mlir::SideEffects::Resource::Base<TestNonAddressableResourceA,
- TestNonAddressableSubResourceA> {
- llvm::StringRef getName() const final { return "<TestNonAddressableA>"; }
- bool isAddressable() const override { return false; }
- mlir::SideEffects::Resource *getParent() const override {
- return TestNonAddressableSubResourceA::get();
- }
-};
-
-struct TestNonAddressableResourceB
- : public mlir::SideEffects::Resource::Base<TestNonAddressableResourceB,
- TestNonAddressableSubResourceB> {
- llvm::StringRef getName() const final { return "<TestNonAddressableB>"; }
- bool isAddressable() const override { return false; }
- mlir::SideEffects::Resource *getParent() const override {
- return TestNonAddressableSubResourceB::get();
- }
-};
-
-//===----------------------------------------------------------------------===//
-// PropertiesWithCustomPrint
-//===----------------------------------------------------------------------===//
-
-struct PropertiesWithCustomPrint {
- /// A shared_ptr to a const object is safe: it is equivalent to a value-based
- /// member. Here the label will be deallocated when the last operation
- /// refering to it is destroyed. However there is no pool-allocation: this is
- /// offloaded to the client.
- std::shared_ptr<const std::string> label;
- int value;
- bool operator==(const PropertiesWithCustomPrint &rhs) const {
- return value == rhs.value && *label == *rhs.label;
- }
-};
-
-llvm::LogicalResult setPropertiesFromAttribute(
- PropertiesWithCustomPrint &prop, mlir::Attribute attr,
- llvm::function_ref<mlir::InFlightDiagnostic()> emitError);
-mlir::DictionaryAttr
-getPropertiesAsAttribute(mlir::MLIRContext *ctx,
- const PropertiesWithCustomPrint &prop);
-llvm::hash_code computeHash(const PropertiesWithCustomPrint &prop);
-void customPrintProperties(mlir::OpAsmPrinter &p,
- const PropertiesWithCustomPrint &prop);
-mlir::ParseResult customParseProperties(mlir::OpAsmParser &parser,
- PropertiesWithCustomPrint &prop);
-
-//===----------------------------------------------------------------------===//
-// MyPropStruct
-//===----------------------------------------------------------------------===//
-namespace test_properties {
-class MyPropStruct {
-public:
- std::string content;
- // These three methods are invoked through the `MyStructProperty` wrapper
- // defined in TestOps.td
- mlir::Attribute asAttribute(mlir::MLIRContext *ctx) const;
- static llvm::LogicalResult
- setFromAttr(MyPropStruct &prop, mlir::Attribute attr,
- llvm::function_ref<mlir::InFlightDiagnostic()> emitError);
- llvm::hash_code hash() const;
- bool operator==(const MyPropStruct &rhs) const {
- return content == rhs.content;
- }
-};
-inline llvm::hash_code hash_value(const MyPropStruct &S) { return S.hash(); }
-} // namespace test_properties
-using test_properties::MyPropStruct;
-
-llvm::LogicalResult readFromMlirBytecode(mlir::DialectBytecodeReader &reader,
- MyPropStruct &prop);
-void writeToMlirBytecode(mlir::DialectBytecodeWriter &writer,
- MyPropStruct &prop);
-
-//===----------------------------------------------------------------------===//
-// VersionedProperties
-//===----------------------------------------------------------------------===//
-
-struct VersionedProperties {
- // For the sake of testing, assume that this object was associated to version
- // 1.2 of the test dialect when having only one int value. In the current
- // version 2.0, the property has two values. We also assume that the class is
- // upgrade-able if value2 = 0.
- int value1;
- int value2;
- bool operator==(const VersionedProperties &rhs) const {
- return value1 == rhs.value1 && value2 == rhs.value2;
- }
-};
-
-llvm::LogicalResult setPropertiesFromAttribute(
- VersionedProperties &prop, mlir::Attribute attr,
- llvm::function_ref<mlir::InFlightDiagnostic()> emitError);
-mlir::DictionaryAttr getPropertiesAsAttribute(mlir::MLIRContext *ctx,
- const VersionedProperties &prop);
-llvm::hash_code computeHash(const VersionedProperties &prop);
-void customPrintProperties(mlir::OpAsmPrinter &p,
- const VersionedProperties &prop);
-mlir::ParseResult customParseProperties(mlir::OpAsmParser &parser,
- VersionedProperties &prop);
-
-//===----------------------------------------------------------------------===//
-// Bytecode Support
-//===----------------------------------------------------------------------===//
-
-llvm::LogicalResult readFromMlirBytecode(mlir::DialectBytecodeReader &reader,
- llvm::MutableArrayRef<int64_t> prop);
-void writeToMlirBytecode(mlir::DialectBytecodeWriter &writer,
- llvm::ArrayRef<int64_t> prop);
-
-} // namespace test
-
-#define GET_OP_CLASSES
-#include "TestOps.h.inc"
+#include "TestOpsAttributes.h"
+#include "TestOpsConversion.h"
+#include "TestOpsCore.h"
+#include "TestOpsInterfaces.h"
+#include "TestOpsPatterns.h"
+#include "TestOpsProperties.h"
+#include "TestOpsTraits.h"
#endif // MLIR_TESTOPS_H
diff --git a/mlir/test/lib/Dialect/Test/TestOps.td b/mlir/test/lib/Dialect/Test/TestOps.td
index 1790963e4f2edd..66b500ca0bed0d 100644
--- a/mlir/test/lib/Dialect/Test/TestOps.td
+++ b/mlir/test/lib/Dialect/Test/TestOps.td
@@ -45,6 +45,24 @@ include "TestTypeDefs.td"
class TEST_Op<string mnemonic, list<Trait> traits = []> :
Op<Test_Dialect, mnemonic, traits>;
+// Ops are in the dialect's default partition (`Core`), unless they are in a
+// `let partition = ... in { ... }` block. Since TableGen rejects `let` of a
+// field that a record does not have, such blocks cannot contain non-op records
+// like patterns. The sections that interleave ops and other records use these
+// base classes instead.
+class TEST_AttrOp<string mnemonic, list<Trait> traits = []> :
+ TEST_Op<mnemonic, traits> {
+ let partition = Test_AttributesPartition;
+}
+class TEST_PatternOp<string mnemonic, list<Trait> traits = []> :
+ TEST_Op<mnemonic, traits> {
+ let partition = Test_PatternsPartition;
+}
+class TEST_PropOp<string mnemonic, list<Trait> traits = []> :
+ TEST_Op<mnemonic, traits> {
+ let partition = Test_PropertiesPartition;
+}
+
//===----------------------------------------------------------------------===//
// Test Types
//===----------------------------------------------------------------------===//
@@ -256,45 +274,46 @@ def MixedNormalVariadicResults : TEST_Op<
// Test Attributes
//===----------------------------------------------------------------------===//
-def AnyAttrOfOp : TEST_Op<"any_attr_of_i32_str"> {
+def AnyAttrOfOp : TEST_AttrOp<"any_attr_of_i32_str"> {
let arguments = (ins AnyAttrOf<[I32Attr, StrAttr]>:$attr);
}
-def NonNegIntAttrOp : TEST_Op<"non_negative_int_attr"> {
+def NonNegIntAttrOp : TEST_AttrOp<"non_negative_int_attr"> {
let arguments = (ins
ConfinedAttr<I32Attr, [IntNonNegative]>:$i32attr,
ConfinedAttr<I64Attr, [IntNonNegative]>:$i64attr
);
}
-def PositiveIntAttrOp : TEST_Op<"positive_int_attr"> {
+def PositiveIntAttrOp : TEST_AttrOp<"positive_int_attr"> {
let arguments = (ins
ConfinedAttr<I32Attr, [IntPositive]>:$i32attr,
ConfinedAttr<I64Attr, [IntPositive]>:$i64attr
);
}
-def TypeArrayAttrOp : TEST_Op<"type_array_attr"> {
+def TypeArrayAttrOp : TEST_AttrOp<"type_array_attr"> {
let arguments = (ins TypeArrayAttr:$attr);
}
-def TypeArrayAttrWithDefaultOp : TEST_Op<"type_array_attr_with_default"> {
+def TypeArrayAttrWithDefaultOp : TEST_AttrOp<"type_array_attr_with_default"> {
let arguments = (ins DefaultValuedAttr<TypeArrayAttr, "{}">:$attr);
}
-def TypedStringAttrWithTypeOp : TEST_Op<"string_attr_with_type"> {
+def TypedStringAttrWithTypeOp : TEST_AttrOp<"string_attr_with_type"> {
let arguments = (ins TypedStrAttr<AnyInteger>:$attr);
let assemblyFormat = "$attr attr-dict";
}
-def TypedStringAttrWithMixedTypeOp : TEST_Op<"string_attr_with_mixed_type"> {
+def TypedStringAttrWithMixedTypeOp :
+ TEST_AttrOp<"string_attr_with_mixed_type"> {
let arguments = (ins
AnyAttrOf<[TypedStrAttr<AnyInteger>, I64Attr]>:$attr
);
let assemblyFormat = "$attr attr-dict";
}
-def FloatAttrOp : TEST_Op<"float_attrs"> {
+def FloatAttrOp : TEST_AttrOp<"float_attrs"> {
// TODO: Clean up the OpBase float type and attribute selectors so they
// can express all of the types.
let arguments = (ins
@@ -302,17 +321,17 @@ def FloatAttrOp : TEST_Op<"float_attrs"> {
);
}
-def I32EnumAttrOp : TEST_Op<"i32_enum_attr"> {
+def I32EnumAttrOp : TEST_AttrOp<"i32_enum_attr"> {
let arguments = (ins SomeI32Enum:$attr);
let results = (outs I32:$val);
}
-def I64EnumAttrOp : TEST_Op<"i64_enum_attr"> {
+def I64EnumAttrOp : TEST_AttrOp<"i64_enum_attr"> {
let arguments = (ins SomeI64Enum:$attr);
let results = (outs I32:$val);
}
-def IntAttrOp : TEST_Op<"int_attrs"> {
+def IntAttrOp : TEST_AttrOp<"int_attrs"> {
let arguments = (ins
AnyI32Attr:$any_i32_attr,
IndexAttr:$index_attr,
@@ -321,7 +340,7 @@ def IntAttrOp : TEST_Op<"int_attrs"> {
);
}
-def FloatElementsAttrOp : TEST_Op<"float_elements_attr"> {
+def FloatElementsAttrOp : TEST_AttrOp<"float_elements_attr"> {
let arguments = (ins
RankedF32ElementsAttr<[2]>:$scalar_f32_attr,
RankedF64ElementsAttr<[4, 8]>:$tensor_f64_attr
@@ -338,21 +357,21 @@ def UpdateFloatElementsAttr : Pat<
ConstantAttr<RankedF32ElementsAttr<[2]>, "{5.0f, 6.0f}">:$f32attr,
$f64attr)>;
-def IntElementsAttrOp : TEST_Op<"int_elements_attr"> {
+def IntElementsAttrOp : TEST_AttrOp<"int_elements_attr"> {
let arguments = (ins
AnyI32ElementsAttr:$any_i32_attr,
I32ElementsAttr:$i32_attr
);
}
-def RankedIntElementsAttrOp : TEST_Op<"ranked_int_elements_attr"> {
+def RankedIntElementsAttrOp : TEST_AttrOp<"ranked_int_elements_attr"> {
let arguments = (ins
RankedI32ElementsAttr<[2]>:$vector_i32_attr,
RankedI64ElementsAttr<[4, 8]>:$matrix_i64_attr
);
}
-def DerivedTypeAttrOp : TEST_Op<"derived_type_attr", []> {
+def DerivedTypeAttrOp : TEST_AttrOp<"derived_type_attr", []> {
let results = (outs AnyTensor:$output);
DerivedTypeAttr element_dtype =
DerivedTypeAttr<"return getElementTypeOrSelf(getOutput().getType());">;
@@ -361,7 +380,7 @@ def DerivedTypeAttrOp : TEST_Op<"derived_type_attr", []> {
"$_builder.getI32IntegerAttr($_self)">;
}
-def TestPropOp : TEST_Op<"prop">,
+def TestPropOp : TEST_AttrOp<"prop">,
Arguments<(ins Variadic<Index>:$upperInits,
I32ElementsAttr:$transforms)>,
Results<(outs Variadic<AnyType>:$results)> {
@@ -371,27 +390,27 @@ def TestPropOp : TEST_Op<"prop">,
}
-def StringElementsAttrOp : TEST_Op<"string_elements_attr"> {
+def StringElementsAttrOp : TEST_AttrOp<"string_elements_attr"> {
let arguments = (ins
StringElementsAttr:$scalar_string_attr
);
}
-def TypedAttrOp : TEST_Op<"typed_attr"> {
+def TypedAttrOp : TEST_AttrOp<"typed_attr"> {
let arguments = (ins TypeAttr:$type, AnyAttr:$attr);
let assemblyFormat = [{
attr-dict $type `=` custom<AttrElideType>(ref($type), $attr)
}];
}
-def TypeAttrOfOp : TEST_Op<"type_attr_of"> {
+def TypeAttrOfOp : TEST_AttrOp<"type_attr_of"> {
let arguments = (ins TypeAttrOf<I64>:$type);
let assemblyFormat = [{
attr-dict $type
}];
}
-def DenseArrayAttrOp : TEST_Op<"dense_array_attr"> {
+def DenseArrayAttrOp : TEST_AttrOp<"dense_array_attr"> {
let arguments = (ins
DenseBoolArrayAttr:$i1attr,
DenseI8ArrayAttr:$i8attr,
@@ -409,7 +428,7 @@ def DenseArrayAttrOp : TEST_Op<"dense_array_attr"> {
}];
}
-def SlashAttrOp : TEST_Op<"slash_attr"> {
+def SlashAttrOp : TEST_AttrOp<"slash_attr"> {
let arguments = (ins SlashAttr:$attr);
}
@@ -417,7 +436,7 @@ def SlashAttrOp : TEST_Op<"slash_attr"> {
// Test Attributes Constraints
//===----------------------------------------------------------------------===//
-def ConfinedDenseArrayAttrOp : TEST_Op<"confined_dense_array_attr"> {
+def ConfinedDenseArrayAttrOp : TEST_AttrOp<"confined_dense_array_attr"> {
let arguments = (ins
ConfinedAttr<DenseI16ArrayAttr,
[DenseArrayStrictlySorted<DenseI16ArrayAttr>]>:$emptyattr,
@@ -429,7 +448,7 @@ def ConfinedDenseArrayAttrOp : TEST_Op<"confined_dense_array_attr"> {
}
// It does not make sense to have this constraint on a DenseBoolArrayAttr.
-def DenseArrayStrictlyPositiveAttrOp : TEST_Op<"confined_strictly_positive_attr"> {
+def DenseArrayStrictlyPositiveAttrOp : TEST_AttrOp<"confined_strictly_positive_attr"> {
let arguments = (ins
ConfinedAttr<DenseI8ArrayAttr,
[DenseArrayStrictlyPositive<DenseI8ArrayAttr>]>:$i8attr,
@@ -450,7 +469,7 @@ def DenseArrayStrictlyPositiveAttrOp : TEST_Op<"confined_strictly_positive_attr"
// It does not make sense to have this constraint on a DenseBoolArrayAttr.
// It is always true.
-def DenseArrayNonNegativeOp : TEST_Op<"confined_non_negative_attr"> {
+def DenseArrayNonNegativeOp : TEST_AttrOp<"confined_non_negative_attr"> {
let arguments = (ins
ConfinedAttr<DenseI8ArrayAttr,
[DenseArrayNonNegative<DenseI8ArrayAttr>]>:$i8attr,
@@ -473,7 +492,7 @@ def DenseArrayNonNegativeOp : TEST_Op<"confined_non_negative_attr"> {
// Test Promised Interfaces Constraints
//===----------------------------------------------------------------------===//
-def PromisedInterfacesOp : TEST_Op<"promised_interfaces"> {
+def PromisedInterfacesOp : TEST_AttrOp<"promised_interfaces"> {
let arguments = (ins
ConfinedAttr<AnyAttr,
[PromisedAttrInterface<TestExternalAttrInterface>]>:$promisedAttr,
@@ -494,14 +513,14 @@ def TestPrettyEnumAttr
: EnumAttr<Test_Dialect, TestPrettyEnum, "pretty_enum">;
// Define an op that contains the enum attribute.
-def OpWithEnum : TEST_Op<"op_with_enum"> {
+def OpWithEnum : TEST_AttrOp<"op_with_enum"> {
let arguments = (ins TestEnumAttr:$value, OptionalAttr<AnyAttr>:$tag);
let assemblyFormat = "enum($value) (`tag` $tag^)? attr-dict";
}
// Define an op that prints just the symbolic value of an enum attribute whose
// own assembly format includes angle brackets.
-def OpWithEnumDirective : TEST_Op<"op_with_enum_directive"> {
+def OpWithEnumDirective : TEST_AttrOp<"op_with_enum_directive"> {
let arguments = (ins TestPrettyEnumAttr:$value);
let assemblyFormat = "enum($value) attr-dict";
}
@@ -519,13 +538,13 @@ def : Pat<(OpWithEnum ConstantEnumCase<TestEnumAttr, "first">:$value,
// Define the enum property.
def TestEnumProp : EnumProp<TestEnum>;
// Define an op that contains the enum property.
-def OpWithEnumProp : TEST_Op<"op_with_enum_prop"> {
+def OpWithEnumProp : TEST_AttrOp<"op_with_enum_prop"> {
let arguments = (ins TestEnumProp:$value);
let assemblyFormat = "$value attr-dict";
}
def TestEnumPropAttrForm : EnumPropWithAttrForm<TestEnum, TestEnumAttr>;
-def OpWithEnumPropAttrForm : TEST_Op<"op_with_enum_prop_attr_form"> {
+def OpWithEnumPropAttrForm : TEST_AttrOp<"op_with_enum_prop_attr_form"> {
let arguments = (ins TestEnumPropAttrForm:$value);
let assemblyFormat = "prop-dict attr-dict";
}
@@ -533,7 +552,7 @@ def OpWithEnumPropAttrForm : TEST_Op<"op_with_enum_prop_attr_form"> {
def TestEnumPropAttrFormAlways : EnumPropWithAttrForm<TestEnum, TestEnumAttr> {
let storeInCustomAttribute = 1;
}
-def OpWithEnumPropAttrFormAlways : TEST_Op<"op_with_enum_prop_attr_form_always"> {
+def OpWithEnumPropAttrFormAlways : TEST_AttrOp<"op_with_enum_prop_attr_form_always"> {
let arguments = (ins TestEnumPropAttrFormAlways:$value);
let assemblyFormat = "prop-dict attr-dict";
}
@@ -541,7 +560,7 @@ def OpWithEnumPropAttrFormAlways : TEST_Op<"op_with_enum_prop_attr_form_always">
def TestBitEnumProp : EnumProp<TestBitEnum> {
let defaultValue = TestBitEnum.cppType # "::Read";
}
-def OpWithTestBitEnum : TEST_Op<"op_with_bit_enum_prop"> {
+def OpWithTestBitEnum : TEST_AttrOp<"op_with_bit_enum_prop"> {
let arguments = (ins
TestBitEnumProp:$value1,
TestBitEnumProp:$value2);
@@ -549,7 +568,7 @@ def OpWithTestBitEnum : TEST_Op<"op_with_bit_enum_prop"> {
}
def OpWithTestBitEnumPropDict
- : TEST_Op<"op_with_bit_enum_prop_dict"> {
+ : TEST_AttrOp<"op_with_bit_enum_prop_dict"> {
let arguments = (ins TestBitEnumProp:$flags, I64Prop:$next);
let assemblyFormat = "prop-dict attr-dict";
}
@@ -557,7 +576,7 @@ def OpWithTestBitEnumPropDict
def TestBitEnumPropNamed : NamedEnumProp<TestBitEnum, "bit_enum"> {
let defaultValue = TestBitEnum.cppType # "::Read";
}
-def OpWithBitEnumPropNamed : TEST_Op<"op_with_bit_enum_prop_named"> {
+def OpWithBitEnumPropNamed : TEST_AttrOp<"op_with_bit_enum_prop_named"> {
let arguments = (ins
TestBitEnumPropNamed:$value1,
TestBitEnumPropNamed:$value2);
@@ -566,14 +585,14 @@ def OpWithBitEnumPropNamed : TEST_Op<"op_with_bit_enum_prop_named"> {
def TestSimpleEnum8Prop : EnumProp<TestSimpleEnum8>;
-def OpWithSimpleEnum8Prop : TEST_Op<"op_with_simple_enum_8_prop"> {
+def OpWithSimpleEnum8Prop : TEST_AttrOp<"op_with_simple_enum_8_prop"> {
let arguments = (ins TestSimpleEnum8Prop:$value);
let assemblyFormat = "$value attr-dict";
}
def TestSimpleEnum16Prop : EnumProp<TestSimpleEnum16>;
-def OpWithSimpleEnum16Prop : TEST_Op<"op_with_simple_enum_16_prop"> {
+def OpWithSimpleEnum16Prop : TEST_AttrOp<"op_with_simple_enum_16_prop"> {
let arguments = (ins TestSimpleEnum16Prop:$value);
let assemblyFormat = "$value attr-dict";
}
@@ -586,7 +605,7 @@ def OpWithSimpleEnum16Prop : TEST_Op<"op_with_simple_enum_16_prop"> {
def TestBitEnumAttr : EnumAttr<Test_Dialect, TestBitEnum, "bit_enum">;
// Define an op that contains the enum attribute.
-def OpWithBitEnum : TEST_Op<"op_with_bit_enum"> {
+def OpWithBitEnum : TEST_AttrOp<"op_with_bit_enum"> {
let arguments = (ins TestBitEnumAttr:$value, OptionalAttr<AnyAttr>:$tag);
let assemblyFormat = "$value (`tag` $tag^)? attr-dict";
}
@@ -595,19 +614,19 @@ def TestBitEnumVerticalBarAttr
: EnumAttr<Test_Dialect, TestBitEnumVerticalBar, "bit_enum_vbar">;
// Define an op that contains the enum attribute.
-def OpWithBitEnumVerticalBar : TEST_Op<"op_with_bit_enum_vbar"> {
+def OpWithBitEnumVerticalBar : TEST_AttrOp<"op_with_bit_enum_vbar"> {
let arguments = (ins TestBitEnumVerticalBarAttr:$value,
OptionalAttr<AnyAttr>:$tag);
let assemblyFormat = "$value (`tag` $tag^)? attr-dict";
}
-def OpWithBitEnumDirective : TEST_Op<"op_with_bit_enum_directive"> {
+def OpWithBitEnumDirective : TEST_AttrOp<"op_with_bit_enum_directive"> {
let arguments = (ins TestBitEnumAttr:$value);
let assemblyFormat = "enum($value) attr-dict";
}
def OpWithBitEnumVerticalBarDirective
- : TEST_Op<"op_with_bit_enum_vbar_directive"> {
+ : TEST_AttrOp<"op_with_bit_enum_vbar_directive"> {
let arguments = (ins TestBitEnumVerticalBarAttr:$value);
let assemblyFormat = "enum($value) attr-dict";
}
@@ -819,6 +838,7 @@ def FoldToCallOp : TEST_Op<"fold_to_call_op"> {
// Test Traits
//===----------------------------------------------------------------------===//
+let partition = Test_TraitsPartition in {
def SameOperandElementTypeOp : TEST_Op<"same_operand_element_type",
[SameOperandsElementType]> {
let arguments = (ins AnyType, AnyType);
@@ -1126,6 +1146,7 @@ def UnreifiableDimOfResultShapeOp : TEST_Op<"unreifiable_dim_of_result_shape",
let arguments = (ins 2DTensorOf<[AnyType]>:$operand);
let results = (outs 2DTensorOf<[AnyType]>:$result);
}
+} // let partition = Test_TraitsPartition
def IsNotScalar : Constraint<CPred<"$0.getType().getRank() != 0">>;
@@ -1133,6 +1154,7 @@ def UpdateAttr : Pat<(I32ElementsAttrOp $attr),
(I32ElementsAttrOp ConstantAttr<I32ElementsAttr, "0">),
[(IsNotScalar $attr)]>;
+let partition = Test_TraitsPartition in {
def TestGreedyCfgRewriteOp : TEST_Op<"greedy_cfg_rewrite"> {
let description = [{
Exercises reachability changes while the greedy driver is processing its
@@ -1203,6 +1225,7 @@ def TestInternalBranchOp : TEST_Op<"internal_br",
let successors = (successor AnySuccessor:$successPath, AnySuccessor:$errorPath);
}
+} // let partition = Test_TraitsPartition
// A successor that must be the block immediately following the op's block.
def FallthroughSuccessor : Successor<
@@ -1222,6 +1245,7 @@ def ForwardSuccessor : Successor<
}]>,
"a forward block (a block listed after the op's block)">;
+let partition = Test_TraitsPartition in {
def TestFallthroughBranchOp : TEST_Op<"fallthrough_br", [Terminator]> {
let summary = "branch whose successors are constrained by predicates";
let successors = (successor FallthroughSuccessor:$target,
@@ -1360,6 +1384,7 @@ def AttrWithTraitOp : TEST_Op<"attr_with_trait", []> {
let arguments = (ins AnyAttr:$attr);
let hasVerifier = 1;
}
+} // let partition = Test_TraitsPartition
//===----------------------------------------------------------------------===//
// Test Locations
@@ -1395,12 +1420,12 @@ def TestLocationAttrOp : TEST_Op<"op_with_loc_attr"> {
// Test Patterns
//===----------------------------------------------------------------------===//
-def OpA : TEST_Op<"op_a"> {
+def OpA : TEST_PatternOp<"op_a"> {
let arguments = (ins I32, I32Attr:$attr);
let results = (outs I32);
}
-def OpB : TEST_Op<"op_b"> {
+def OpB : TEST_PatternOp<"op_b"> {
let arguments = (ins I32, I32Attr:$attr);
let results = (outs I32);
}
@@ -1412,10 +1437,10 @@ def TestNamedPatternRule : Pat<(OpA $input, $attr), (OpB $input, $attr)>;
def : Pat<(OpA (OpA $input, $attr), $bttr), (OpB $input, $bttr)>;
// Test added benefit.
-def OpD : TEST_Op<"op_d">, Arguments<(ins I32)>, Results<(outs I32)>;
-def OpE : TEST_Op<"op_e">, Arguments<(ins I32)>, Results<(outs I32)>;
-def OpF : TEST_Op<"op_f">, Arguments<(ins I32)>, Results<(outs I32)>;
-def OpG : TEST_Op<"op_g">, Arguments<(ins I32)>, Results<(outs I32)>;
+def OpD : TEST_PatternOp<"op_d">, Arguments<(ins I32)>, Results<(outs I32)>;
+def OpE : TEST_PatternOp<"op_e">, Arguments<(ins I32)>, Results<(outs I32)>;
+def OpF : TEST_PatternOp<"op_f">, Arguments<(ins I32)>, Results<(outs I32)>;
+def OpG : TEST_PatternOp<"op_g">, Arguments<(ins I32)>, Results<(outs I32)>;
// Verify that bumping benefit results in selecting different op.
def : Pat<(OpD $input), (OpE $input)>;
def : Pat<(OpD $input), (OpF $input), [], [], (addBenefit 10)>;
@@ -1424,43 +1449,43 @@ def : Pat<(OpG $input), (OpB $input, ConstantAttr<I32Attr, "20">:$attr)>;
def : Pat<(OpG (OpG $input)), (OpB $input, ConstantAttr<I32Attr, "34">:$attr)>;
// Test patterns for zero-result op.
-def OpH : TEST_Op<"op_h">, Arguments<(ins I32)>, Results<(outs)>;
-def OpI : TEST_Op<"op_i">, Arguments<(ins I32)>, Results<(outs)>;
+def OpH : TEST_PatternOp<"op_h">, Arguments<(ins I32)>, Results<(outs)>;
+def OpI : TEST_PatternOp<"op_i">, Arguments<(ins I32)>, Results<(outs)>;
def : Pat<(OpH $input), (OpI $input)>;
// Test patterns for zero-input op.
-def OpJ : TEST_Op<"op_j">, Arguments<(ins)>, Results<(outs I32)>;
-def OpK : TEST_Op<"op_k">, Arguments<(ins)>, Results<(outs I32)>;
+def OpJ : TEST_PatternOp<"op_j">, Arguments<(ins)>, Results<(outs I32)>;
+def OpK : TEST_PatternOp<"op_k">, Arguments<(ins)>, Results<(outs I32)>;
def : Pat<(OpJ), (OpK)>;
// Test that natives calls are only called once during rewrites.
-def OpM : TEST_Op<"op_m"> {
+def OpM : TEST_PatternOp<"op_m"> {
let arguments = (ins I32, OptionalAttr<I32Attr>:$optional_attr);
let results = (outs I32);
}
-def OpN : TEST_Op<"op_n"> {
+def OpN : TEST_PatternOp<"op_n"> {
let arguments = (ins I32, I32);
let results = (outs I32);
}
-def OpO : TEST_Op<"op_o"> {
+def OpO : TEST_PatternOp<"op_o"> {
let arguments = (ins I32);
let results = (outs I32);
}
-def OpP : TEST_Op<"op_p"> {
+def OpP : TEST_PatternOp<"op_p"> {
let arguments = (ins I32, I32, I32, I32, I32, I32);
let results = (outs I32);
}
-def OpQ : TEST_Op<"op_q"> {
+def OpQ : TEST_PatternOp<"op_q"> {
let arguments = (ins AnyType, AnyType);
let results = (outs AnyType);
}
// Test constant-folding a pattern that maps `(F32) -> I32`.
-def SignOp : TEST_Op<"sign", [SameOperandsAndResultShape]> {
+def SignOp : TEST_PatternOp<"sign", [SameOperandsAndResultShape]> {
let arguments = (ins RankedTensorOf<[F32]>:$operand);
let results = (outs RankedTensorOf<[I32]>:$result);
@@ -1470,7 +1495,7 @@ def SignOp : TEST_Op<"sign", [SameOperandsAndResultShape]> {
}
// Test constant-folding a pattern that maps `(F32, F32) -> I1`.
-def LessThanOp : TEST_Op<"less_than", [SameOperandsAndResultShape]> {
+def LessThanOp : TEST_PatternOp<"less_than", [SameOperandsAndResultShape]> {
let arguments = (ins RankedTensorOf<[F32]>:$lhs, RankedTensorOf<[F32]>:$rhs);
let results = (outs RankedTensorOf<[I1]>:$result);
@@ -1506,22 +1531,22 @@ def TestEqualArgsCheckBeforeUserConstraintsPattern :
[(AssertBinOpEqualArgsAndReturnTrue $op)]>;
// Test for memrefs normalization of an op with normalizable memrefs.
-def OpNorm : TEST_Op<"op_norm", [MemRefsNormalizable]> {
+def OpNorm : TEST_PatternOp<"op_norm", [MemRefsNormalizable]> {
let arguments = (ins AnyMemRef:$X, AnyMemRef:$Y);
}
// Test for memrefs normalization of an op without normalizable memrefs.
-def OpNonNorm : TEST_Op<"op_nonnorm"> {
+def OpNonNorm : TEST_PatternOp<"op_nonnorm"> {
let arguments = (ins AnyMemRef:$X, AnyMemRef:$Y);
}
// Test for memrefs normalization of an op that has normalizable memref results.
-def OpNormRet : TEST_Op<"op_norm_ret", [MemRefsNormalizable]> {
+def OpNormRet : TEST_PatternOp<"op_norm_ret", [MemRefsNormalizable]> {
let arguments = (ins AnyMemRef:$X);
let results = (outs AnyMemRef:$Y, AnyMemRef:$Z);
}
// Test for memrefs normalization of an op with a reference to a function
// symbol.
-def OpFuncRef : TEST_Op<"op_funcref"> {
+def OpFuncRef : TEST_PatternOp<"op_funcref"> {
let summary = "Test op with a reference to a function symbol";
let description = [{
The "test.op_funcref" is a test op with a reference to a function symbol.
@@ -1538,18 +1563,19 @@ def : Pat<(OpM $attr, $optAttr), (OpM $attr, (OpMVal $attr) ),
[(OpMAttributeIsNull:$optAttr)]>;
// Test `$_` for ignoring op argument match.
-def TestIgnoreArgMatchSrcOp : TEST_Op<"ignore_arg_match_src"> {
+def TestIgnoreArgMatchSrcOp : TEST_PatternOp<"ignore_arg_match_src"> {
let arguments = (ins
AnyType:$a, AnyType:$b, AnyType:$c,
AnyAttr:$d, AnyAttr:$e, AnyAttr:$f);
}
-def TestIgnoreArgMatchDstOp : TEST_Op<"ignore_arg_match_dst"> {
+def TestIgnoreArgMatchDstOp : TEST_PatternOp<"ignore_arg_match_dst"> {
let arguments = (ins AnyType:$b, AnyAttr:$f);
}
def : Pat<(TestIgnoreArgMatchSrcOp $_, $b, I32, I64Attr:$_, $_, $f),
(TestIgnoreArgMatchDstOp $b, $f)>;
-def OpInterleavedOperandAttribute1 : TEST_Op<"interleaved_operand_attr1"> {
+def OpInterleavedOperandAttribute1 :
+ TEST_PatternOp<"interleaved_operand_attr1"> {
let arguments = (ins
I32:$input1,
I64Attr:$attr1,
@@ -1558,7 +1584,8 @@ def OpInterleavedOperandAttribute1 : TEST_Op<"interleaved_operand_attr1"> {
);
}
-def OpInterleavedOperandAttribute2 : TEST_Op<"interleaved_operand_attr2"> {
+def OpInterleavedOperandAttribute2 :
+ TEST_PatternOp<"interleaved_operand_attr2"> {
let arguments = (ins
I32:$input1,
I64Attr:$attr1,
@@ -1567,7 +1594,7 @@ def OpInterleavedOperandAttribute2 : TEST_Op<"interleaved_operand_attr2"> {
);
}
-def ManyArgsOp : TEST_Op<"many_arguments"> {
+def ManyArgsOp : TEST_PatternOp<"many_arguments"> {
let arguments = (ins
I32:$input1, I32:$input2, I32:$input3, I32:$input4, I32:$input5,
I32:$input6, I32:$input7, I32:$input8, I32:$input9,
@@ -1596,7 +1623,7 @@ def : Pat<(OpInterleavedOperandAttribute1 $input1, $attr1, $input2, $attr2),
(OpInterleavedOperandAttribute2 $input1, $attr1, $input2, $attr2)>;
// Test NativeCodeCall.
-def OpNativeCodeCall1 : TEST_Op<"native_code_call1"> {
+def OpNativeCodeCall1 : TEST_PatternOp<"native_code_call1"> {
let arguments = (ins
I32:$input1, I32:$input2,
BoolAttr:$choice,
@@ -1604,7 +1631,7 @@ def OpNativeCodeCall1 : TEST_Op<"native_code_call1"> {
);
let results = (outs I32);
}
-def OpNativeCodeCall2 : TEST_Op<"native_code_call2"> {
+def OpNativeCodeCall2 : TEST_PatternOp<"native_code_call2"> {
let arguments = (ins I32:$input, I64ArrayAttr:$attr);
let results = (outs I32);
}
@@ -1627,7 +1654,7 @@ def : Pat<(OpNativeCodeCall1 $input1, $input2,
ConstBoolAttrFalse, $attr1, $attr2),
(UseOpResult $input2)>;
-def OpNativeCodeCall3 : TEST_Op<"native_code_call3"> {
+def OpNativeCodeCall3 : TEST_PatternOp<"native_code_call3"> {
let arguments = (ins I32:$input);
let results = (outs I32);
}
@@ -1637,11 +1664,11 @@ def : Pattern<(OpNativeCodeCall3 $input),
[(NativeCodeCallVoid<"createOpI($_builder, $_loc, $0)"> $input),
(OpK)]>;
-def OpNativeCodeCall4 : TEST_Op<"native_code_call4"> {
+def OpNativeCodeCall4 : TEST_PatternOp<"native_code_call4"> {
let arguments = (ins AnyType:$input1);
let results = (outs I32:$output1, I32:$output2);
}
-def OpNativeCodeCall5 : TEST_Op<"native_code_call5"> {
+def OpNativeCodeCall5 : TEST_PatternOp<"native_code_call5"> {
let arguments = (ins I32:$input1, I32:$input2);
let results = (outs I32:$output1, I32:$output2);
}
@@ -1651,11 +1678,11 @@ def BindNativeCodeCallResult : NativeCodeCall<"bindNativeCodeCallResult($0)">;
def : Pat<(OpNativeCodeCall4 (GetFirstI32Result $ret)),
(OpNativeCodeCall5 (BindNativeCodeCallResult:$native $ret), $native)>;
-def OpNativeCodeCall6 : TEST_Op<"native_code_call6"> {
+def OpNativeCodeCall6 : TEST_PatternOp<"native_code_call6"> {
let arguments = (ins I32:$input1, I32:$input2);
let results = (outs I32:$output1, I32:$output2);
}
-def OpNativeCodeCall7 : TEST_Op<"native_code_call7"> {
+def OpNativeCodeCall7 : TEST_PatternOp<"native_code_call7"> {
let arguments = (ins I32:$input);
let results = (outs I32);
}
@@ -1665,11 +1692,11 @@ def : Pattern<(OpNativeCodeCall6 $arg1, $arg2),
(OpNativeCodeCall7 $native__1)]>;
// Test AllAttrOf.
-def OpAllAttrConstraint1 : TEST_Op<"all_attr_constraint_of1"> {
+def OpAllAttrConstraint1 : TEST_PatternOp<"all_attr_constraint_of1"> {
let arguments = (ins I64ArrayAttr:$attr);
let results = (outs I32);
}
-def OpAllAttrConstraint2 : TEST_Op<"all_attr_constraint_of2"> {
+def OpAllAttrConstraint2 : TEST_PatternOp<"all_attr_constraint_of2"> {
let arguments = (ins I64ArrayAttr:$attr);
let results = (outs I32);
}
@@ -1684,114 +1711,116 @@ def : Pat<(OpAllAttrConstraint1
(OpAllAttrConstraint2 $attr)>;
// Op for testing RewritePattern removing op with inner ops.
-def TestOpWithRegionPattern : TEST_Op<"op_with_region_pattern"> {
+def TestOpWithRegionPattern : TEST_PatternOp<"op_with_region_pattern"> {
let regions = (region SizedRegion<1>:$region);
let hasCanonicalizer = 1;
}
-def TestOpConstant : TEST_Op<"constant", [ConstantLike, NoMemoryEffect]> {
+def TestOpConstant :
+ TEST_PatternOp<"constant", [ConstantLike, NoMemoryEffect]> {
let arguments = (ins AnyAttr:$value);
let results = (outs AnyType);
let hasFolder = 1;
}
-def OpR : TEST_Op<"op_r">, Arguments<(ins AnyInteger, AnyInteger)>, Results<(outs AnyInteger)>;
-def OpS : TEST_Op<"op_s">, Arguments<(ins AnyInteger, AnyAttr:$value)>, Results<(outs AnyInteger)>;
+def OpR : TEST_PatternOp<"op_r">, Arguments<(ins AnyInteger, AnyInteger)>, Results<(outs AnyInteger)>;
+def OpS : TEST_PatternOp<"op_s">, Arguments<(ins AnyInteger, AnyAttr:$value)>, Results<(outs AnyInteger)>;
def : Pat<(OpR $input1, (ConstantLikeMatcher I32Attr:$input2)),
(OpS:$unused $input1, $input2)>;
// Op for testing trivial removal via folding of op with inner ops and no uses.
-def TestOpWithRegionFoldNoMemoryEffect : TEST_Op<
+def TestOpWithRegionFoldNoMemoryEffect : TEST_PatternOp<
"op_with_region_fold_no_side_effect", [NoMemoryEffect]> {
let regions = (region SizedRegion<1>:$region);
}
// Op for testing folding of outer op with inner ops.
-def TestOpWithRegionFold : TEST_Op<"op_with_region_fold"> {
+def TestOpWithRegionFold : TEST_PatternOp<"op_with_region_fold"> {
let arguments = (ins AnyType:$operand);
let results = (outs AnyType);
let regions = (region SizedRegion<1>:$region);
let hasFolder = 1;
}
-def TestOpWithVariadicResultsAndFolder: TEST_Op<"op_with_variadic_results_and_folder"> {
+def TestOpWithVariadicResultsAndFolder: TEST_PatternOp<"op_with_variadic_results_and_folder"> {
let arguments = (ins Variadic<I32>);
let results = (outs Variadic<I32>);
let hasFolder = 1;
}
-def TestAddIOp : TEST_Op<"addi"> {
+def TestAddIOp : TEST_PatternOp<"addi"> {
let arguments = (ins AnyTypeOf<[I32, TestI32]>:$op1,
AnyTypeOf<[I32, TestI32]>:$op2);
let results = (outs AnyTypeOf<[I32, TestI32]>);
}
-def TestCommutativeOp : TEST_Op<"op_commutative", [Commutative]> {
+def TestCommutativeOp : TEST_PatternOp<"op_commutative", [Commutative]> {
let arguments = (ins I32:$op1, I32:$op2, I32:$op3, I32:$op4);
let results = (outs I32);
}
-def TestLargeCommutativeOp : TEST_Op<"op_large_commutative", [Commutative]> {
+def TestLargeCommutativeOp :
+ TEST_PatternOp<"op_large_commutative", [Commutative]> {
let arguments = (ins I32:$op1, I32:$op2, I32:$op3, I32:$op4, I32:$op5, I32:$op6, I32:$op7);
let results = (outs I32);
}
-def TestCommutative2Op : TEST_Op<"op_commutative2", [Commutative]> {
+def TestCommutative2Op : TEST_PatternOp<"op_commutative2", [Commutative]> {
let arguments = (ins I32:$op1, I32:$op2);
let results = (outs I32);
}
def TestIdempotentTraitOp
- : TEST_Op<"op_idempotent_trait",
- [SameOperandsAndResultType, NoMemoryEffect, Idempotent]> {
+ : TEST_PatternOp<"op_idempotent_trait",
+ [SameOperandsAndResultType, NoMemoryEffect, Idempotent]> {
let arguments = (ins I32:$op1);
let results = (outs I32);
}
def TestIdempotentTraitBinaryOp
- : TEST_Op<"op_idempotent_trait_binary",
- [SameOperandsAndResultType, NoMemoryEffect, Idempotent]> {
+ : TEST_PatternOp<"op_idempotent_trait_binary",
+ [SameOperandsAndResultType, NoMemoryEffect, Idempotent]> {
let arguments = (ins I32:$op1, I32:$op2);
let results = (outs I32);
}
def TestInvolutionTraitNoOperationFolderOp
- : TEST_Op<"op_involution_trait_no_operation_fold",
- [SameOperandsAndResultType, NoMemoryEffect, Involution]> {
+ : TEST_PatternOp<"op_involution_trait_no_operation_fold",
+ [SameOperandsAndResultType, NoMemoryEffect, Involution]> {
let arguments = (ins I32:$op1);
let results = (outs I32);
}
def TestInvolutionTraitFailingOperationFolderOp
- : TEST_Op<"op_involution_trait_failing_operation_fold",
- [SameOperandsAndResultType, NoMemoryEffect, Involution]> {
+ : TEST_PatternOp<"op_involution_trait_failing_operation_fold",
+ [SameOperandsAndResultType, NoMemoryEffect, Involution]> {
let arguments = (ins I32:$op1);
let results = (outs I32);
let hasFolder = 1;
}
def TestInvolutionTraitSuccesfulOperationFolderOp
- : TEST_Op<"op_involution_trait_succesful_operation_fold",
- [SameOperandsAndResultType, NoMemoryEffect, Involution]> {
+ : TEST_PatternOp<"op_involution_trait_succesful_operation_fold",
+ [SameOperandsAndResultType, NoMemoryEffect, Involution]> {
let arguments = (ins I32:$op1);
let results = (outs I32);
let hasFolder = 1;
}
-def TestOpInPlaceFoldAnchor : TEST_Op<"op_in_place_fold_anchor"> {
+def TestOpInPlaceFoldAnchor : TEST_PatternOp<"op_in_place_fold_anchor"> {
let arguments = (ins I32);
let results = (outs I32);
}
-def TestOpInPlaceFold : TEST_Op<"op_in_place_fold"> {
+def TestOpInPlaceFold : TEST_PatternOp<"op_in_place_fold"> {
let arguments = (ins I32:$op, OptionalAttr<I32Attr>:$attr);
let results = (outs I32);
let hasFolder = 1;
}
-def TestOpInPlaceSelfFold : TEST_Op<"op_in_place_self_fold"> {
+def TestOpInPlaceSelfFold : TEST_PatternOp<"op_in_place_self_fold"> {
let arguments = (ins UnitAttr:$folded);
let results = (outs I32);
let hasFolder = 1;
@@ -1800,7 +1829,7 @@ def : Pat<(TestOpInPlaceSelfFold:$op $_),
(TestOpConstant ConstantAttr<I32Attr, "97">)>;
// Test op that simply returns success.
-def TestOpInPlaceFoldSuccess : TEST_Op<"op_in_place_fold_success"> {
+def TestOpInPlaceFoldSuccess : TEST_PatternOp<"op_in_place_fold_success"> {
let results = (outs Variadic<I1>);
let hasFolder = 1;
let extraClassDefinition = [{
@@ -1812,7 +1841,7 @@ def TestOpInPlaceFoldSuccess : TEST_Op<"op_in_place_fold_success"> {
}
def TestOpFoldWithFoldAdaptor
- : TEST_Op<"fold_with_fold_adaptor",
+ : TEST_PatternOp<"fold_with_fold_adaptor",
[AttrSizedOperandSegments, NoTerminator]> {
let arguments = (ins
I32:$op,
@@ -1832,7 +1861,7 @@ def TestOpFoldWithFoldAdaptor
let hasFolder = 1;
}
-def TestDialectCanonicalizerOp : TEST_Op<"dialect_canonicalizable"> {
+def TestDialectCanonicalizerOp : TEST_PatternOp<"dialect_canonicalizable"> {
let arguments = (ins);
let results = (outs I32);
}
@@ -1842,20 +1871,20 @@ def TestDialectCanonicalizerOp : TEST_Op<"dialect_canonicalizable"> {
//===----------------------------------------------------------------------===//
// Test symbol binding.
-def OpSymbolBindingA : TEST_Op<"symbol_binding_a", []> {
+def OpSymbolBindingA : TEST_PatternOp<"symbol_binding_a", []> {
let arguments = (ins I32:$operand, I64Attr:$attr);
let results = (outs I32);
}
-def OpSymbolBindingB : TEST_Op<"symbol_binding_b", []> {
+def OpSymbolBindingB : TEST_PatternOp<"symbol_binding_b", []> {
let arguments = (ins I32:$operand);
let results = (outs I32);
}
-def OpSymbolBindingC : TEST_Op<"symbol_binding_c", []> {
+def OpSymbolBindingC : TEST_PatternOp<"symbol_binding_c", []> {
let arguments = (ins I32:$operand);
let results = (outs I32);
let builders = OpSymbolBindingB.builders;
}
-def OpSymbolBindingD : TEST_Op<"symbol_binding_d", []> {
+def OpSymbolBindingD : TEST_PatternOp<"symbol_binding_d", []> {
let arguments = (ins I32:$input1, I32:$input2, I64Attr:$attr);
let results = (outs I32);
}
@@ -1871,7 +1900,7 @@ def : Pattern<
// Use bound symbols in additional constraints
[(HasOneUse $res_a)]>;
-def OpSymbolBindingNoResult : TEST_Op<"symbol_binding_no_result", []> {
+def OpSymbolBindingNoResult : TEST_PatternOp<"symbol_binding_no_result", []> {
let arguments = (ins I32:$operand);
}
@@ -1884,7 +1913,7 @@ def : Pat<(OpSymbolBindingNoResult:$op $operand),
//===----------------------------------------------------------------------===//
// Test matching against op attributes.
-def OpAttrMatch1 : TEST_Op<"match_op_attribute1"> {
+def OpAttrMatch1 : TEST_PatternOp<"match_op_attribute1"> {
let arguments = (ins
I32Attr:$required_attr,
OptionalAttr<I32Attr>:$optional_attr,
@@ -1893,7 +1922,7 @@ def OpAttrMatch1 : TEST_Op<"match_op_attribute1"> {
);
let results = (outs I32);
}
-def OpAttrMatch2 : TEST_Op<"match_op_attribute2"> {
+def OpAttrMatch2 : TEST_PatternOp<"match_op_attribute2"> {
let arguments = OpAttrMatch1.arguments;
let results = (outs I32);
}
@@ -1904,57 +1933,57 @@ def : Pat<(OpAttrMatch1 $required, $optional, $default_valued,
(OpAttrMatch2 $required, $optional, $default_valued, $more)>;
// Test unit attrs.
-def OpAttrMatch3 : TEST_Op<"match_op_attribute3"> {
+def OpAttrMatch3 : TEST_PatternOp<"match_op_attribute3"> {
let arguments = (ins UnitAttr:$attr);
let results = (outs I32);
}
-def OpAttrMatch4 : TEST_Op<"match_op_attribute4"> {
+def OpAttrMatch4 : TEST_PatternOp<"match_op_attribute4"> {
let arguments = (ins UnitAttr:$attr1, UnitAttr:$attr2);
let results = (outs I32);
}
def : Pat<(OpAttrMatch3 $attr), (OpAttrMatch4 ConstUnitAttr, $attr)>;
// Test with constant attr.
-def OpC : TEST_Op<"op_c">, Arguments<(ins I32)>, Results<(outs I32)>;
+def OpC : TEST_PatternOp<"op_c">, Arguments<(ins I32)>, Results<(outs I32)>;
def : Pat<(OpC $input), (OpB $input, ConstantAttr<I32Attr, "17">:$attr)>;
// Test integer enum attribute in rewrites.
def : Pat<(I32EnumAttrOp I32Case5), (I32EnumAttrOp I32Case10)>;
def : Pat<(I64EnumAttrOp I64Case5), (I64EnumAttrOp I64Case10)>;
-def ThreeResultOp : TEST_Op<"three_result"> {
+def ThreeResultOp : TEST_PatternOp<"three_result"> {
let arguments = (ins MultiResultOpEnum:$kind);
let results = (outs I32:$result1, F32:$result2, F32:$result3);
}
def AnotherThreeResultOp
- : TEST_Op<"another_three_result",
- [DeclareOpInterfaceMethods<InferTypeOpInterface>]> {
+ : TEST_PatternOp<"another_three_result",
+ [DeclareOpInterfaceMethods<InferTypeOpInterface>]> {
let arguments = (ins MultiResultOpEnum:$kind);
let results = (outs I32:$result1, F32:$result2, F32:$result3);
}
-def TwoResultOp : TEST_Op<"two_result"> {
+def TwoResultOp : TEST_PatternOp<"two_result"> {
let arguments = (ins MultiResultOpEnum:$kind);
let results = (outs I32:$result1, F32:$result2);
}
-def AnotherTwoResultOp : TEST_Op<"another_two_result"> {
+def AnotherTwoResultOp : TEST_PatternOp<"another_two_result"> {
let arguments = (ins MultiResultOpEnum:$kind);
let results = (outs F32:$result1, F32:$result2);
}
-def OneResultOp1 : TEST_Op<"one_result1"> {
+def OneResultOp1 : TEST_PatternOp<"one_result1"> {
let arguments = (ins MultiResultOpEnum:$kind);
let results = (outs F32:$result1);
}
-def OneResultOp2 : TEST_Op<"one_result2"> {
+def OneResultOp2 : TEST_PatternOp<"one_result2"> {
let arguments = (ins MultiResultOpEnum:$kind);
let results = (outs I32:$result1);
}
-def OneResultOp3 : TEST_Op<"one_result3"> {
+def OneResultOp3 : TEST_PatternOp<"one_result3"> {
let arguments = (ins F32);
let results = (outs I32:$result1);
}
@@ -1999,11 +2028,13 @@ def : Pattern<
// Test Patterns (Variadic Ops)
//===----------------------------------------------------------------------===//
-def OneVResOneVOperandOp1 : TEST_Op<"one_variadic_out_one_variadic_in1"> {
+def OneVResOneVOperandOp1 :
+ TEST_PatternOp<"one_variadic_out_one_variadic_in1"> {
let arguments = (ins Variadic<I32>);
let results = (outs Variadic<I32>);
}
-def OneVResOneVOperandOp2 : TEST_Op<"one_variadic_out_one_variadic_in2"> {
+def OneVResOneVOperandOp2 :
+ TEST_PatternOp<"one_variadic_out_one_variadic_in2"> {
let arguments = (ins Variadic<I32>);
let results = (outs Variadic<I32>);
}
@@ -2012,8 +2043,8 @@ def OneVResOneVOperandOp2 : TEST_Op<"one_variadic_out_one_variadic_in2"> {
// another similar op.
def : Pat<(OneVResOneVOperandOp1 $inputs), (OneVResOneVOperandOp2 $inputs)>;
-def MixedVOperandOp1 : TEST_Op<"mixed_variadic_in1",
- [SameVariadicOperandSize]> {
+def MixedVOperandOp1 : TEST_PatternOp<"mixed_variadic_in1",
+ [SameVariadicOperandSize]> {
let arguments = (ins
Variadic<I32>:$input1,
F32:$input2,
@@ -2021,8 +2052,8 @@ def MixedVOperandOp1 : TEST_Op<"mixed_variadic_in1",
);
}
-def MixedVOperandOp2 : TEST_Op<"mixed_variadic_in2",
- [SameVariadicOperandSize]> {
+def MixedVOperandOp2 : TEST_PatternOp<"mixed_variadic_in2",
+ [SameVariadicOperandSize]> {
let arguments = (ins
Variadic<I32>:$input1,
F32:$input2,
@@ -2034,7 +2065,8 @@ def MixedVOperandOp2 : TEST_Op<"mixed_variadic_in2",
def : Pat<(MixedVOperandOp1 $input1, $input2, $input3),
(MixedVOperandOp2 $input1, $input2, $input3)>;
-def MixedVResultOp1 : TEST_Op<"mixed_variadic_out1", [SameVariadicResultSize]> {
+def MixedVResultOp1 :
+ TEST_PatternOp<"mixed_variadic_out1", [SameVariadicResultSize]> {
let results = (outs
Variadic<I32>:$output1,
F32:$output2,
@@ -2042,7 +2074,8 @@ def MixedVResultOp1 : TEST_Op<"mixed_variadic_out1", [SameVariadicResultSize]> {
);
}
-def MixedVResultOp2 : TEST_Op<"mixed_variadic_out2", [SameVariadicResultSize]> {
+def MixedVResultOp2 :
+ TEST_PatternOp<"mixed_variadic_out2", [SameVariadicResultSize]> {
let results = (outs
Variadic<I32>:$output1,
F32:$output2,
@@ -2056,12 +2089,12 @@ def MixedVResultOp2 : TEST_Op<"mixed_variadic_out2", [SameVariadicResultSize]> {
// the information from the matched root op.
def : Pat<(MixedVResultOp1), (MixedVResultOp2)>;
-def OneI32ResultOp : TEST_Op<"one_i32_out"> {
+def OneI32ResultOp : TEST_PatternOp<"one_i32_out"> {
let results = (outs I32);
}
-def MixedVOperandOp3 : TEST_Op<"mixed_variadic_in3",
- [SameVariadicOperandSize]> {
+def MixedVOperandOp3 : TEST_PatternOp<"mixed_variadic_in3",
+ [SameVariadicOperandSize]> {
let arguments = (ins
I32:$input1,
Variadic<I32>:$input2,
@@ -2072,8 +2105,8 @@ def MixedVOperandOp3 : TEST_Op<"mixed_variadic_in3",
let results = (outs I32);
}
-def MixedVResultOp3 : TEST_Op<"mixed_variadic_out3",
- [SameVariadicResultSize]> {
+def MixedVResultOp3 : TEST_PatternOp<"mixed_variadic_out3",
+ [SameVariadicResultSize]> {
let arguments = (ins I32Attr:$count);
let results = (outs
@@ -2106,7 +2139,7 @@ def : Pat<(OneI32ResultOp),
ConstantAttr<I32Attr, "2">)>;
// Variadic structured matching
-def MixedVOperandOp4 : TEST_Op<"mixed_variadic_in4"> {
+def MixedVOperandOp4 : TEST_PatternOp<"mixed_variadic_in4"> {
let arguments = (ins
Variadic<I32>:$input1,
I32:$input2,
@@ -2114,7 +2147,7 @@ def MixedVOperandOp4 : TEST_Op<"mixed_variadic_in4"> {
);
}
-def MixedVOperandOp5 : TEST_Op<"mixed_variadic_in5"> {
+def MixedVOperandOp5 : TEST_PatternOp<"mixed_variadic_in5"> {
let arguments = (ins
I32:$input1,
I32:$input2,
@@ -2125,7 +2158,7 @@ def MixedVOperandOp5 : TEST_Op<"mixed_variadic_in5"> {
}
// Helper op to test variadic recursive pattern matching
-def MixedVOperandInOutI32Op : TEST_Op<"mixed_variadic_in_out_i32"> {
+def MixedVOperandInOutI32Op : TEST_PatternOp<"mixed_variadic_in_out_i32"> {
let arguments = (ins
I32:$input
);
@@ -2159,8 +2192,8 @@ def : Pat<
(MixedVOperandOp5 $input1, $input1, $input2, $attr1,
ConstantStrAttr<StrAttr, "MatchVariadicSameSymbol">)>;
-def MixedVOperandOp6 : TEST_Op<"mixed_variadic_in6",
- [SameVariadicOperandSize]> {
+def MixedVOperandOp6 : TEST_PatternOp<"mixed_variadic_in6",
+ [SameVariadicOperandSize]> {
let arguments = (ins
Variadic<I32>:$input1,
Variadic<I32>:$input2,
@@ -2181,8 +2214,8 @@ def : Pat<
(MixedVOperandOp5 $input2a, $input2b, $input1b, $attr1,
ConstantStrAttr<StrAttr, "MatchMultiVariadicSubSymbol">)>;
-def MixedVOperandOp7 : TEST_Op<"mixed_variadic_optional_in7",
- [AttrSizedOperandSegments]> {
+def MixedVOperandOp7 : TEST_PatternOp<"mixed_variadic_optional_in7",
+ [AttrSizedOperandSegments]> {
let arguments = (ins
Variadic<I32>:$input1,
Optional<I32>:$input2,
@@ -2199,17 +2232,17 @@ def : Pat<
// Test Patterns (either)
//===----------------------------------------------------------------------===//
-def TestEitherOpA : TEST_Op<"either_op_a"> {
+def TestEitherOpA : TEST_PatternOp<"either_op_a"> {
let arguments = (ins AnyInteger:$arg0, AnyInteger:$arg1, AnyInteger:$arg2);
let results = (outs I32:$output);
}
-def TestEitherOpB : TEST_Op<"either_op_b"> {
+def TestEitherOpB : TEST_PatternOp<"either_op_b"> {
let arguments = (ins AnyInteger:$arg0, AnyInteger:$arg1);
let results = (outs I32:$output);
}
-def TestEitherOpC : TEST_Op<"either_op_c"> {
+def TestEitherOpC : TEST_PatternOp<"either_op_c"> {
let arguments = (ins AnyI32Attr:$attr, AnyInteger:$arg0, AnyInteger:$arg1);
let results = (outs I32:$output);
}
@@ -2228,12 +2261,12 @@ def : Pat<(TestEitherOpA (either (TestEitherOpB I32:$arg1, $_),
def : Pat<(TestEitherOpC ConstantAttr<I32Attr, "0">, (either $arg1, I32:$arg2)),
(TestEitherOpB $arg1, $arg2)>;
-def TestEitherHelperOpA : TEST_Op<"either_helper_op_a"> {
+def TestEitherHelperOpA : TEST_PatternOp<"either_helper_op_a"> {
let arguments = (ins I32:$arg0);
let results = (outs I32:$output);
}
-def TestEitherHelperOpB : TEST_Op<"either_helper_op_b"> {
+def TestEitherHelperOpB : TEST_PatternOp<"either_helper_op_b"> {
let arguments = (ins I32:$arg0);
let results = (outs I32:$output);
}
@@ -2269,13 +2302,13 @@ def : Pat<(TestLocationSrcNoResOp:$loc
// Test Patterns (Type Builders)
//===----------------------------------------------------------------------===//
-def SourceOp : TEST_Op<"source_op"> {
+def SourceOp : TEST_PatternOp<"source_op"> {
let arguments = (ins AnyInteger:$arg, AnyI32Attr:$tag);
let results = (outs AnyInteger);
}
// An op without return type deduction.
-def OpX : TEST_Op<"op_x"> {
+def OpX : TEST_PatternOp<"op_x"> {
let arguments = (ins AnyInteger:$input);
let results = (outs AnyInteger);
}
@@ -2294,12 +2327,12 @@ def : Pat<(SourceOp $val, ConstantAttr<I32Attr, "22">:$attr),
def MakeI64Type : NativeCodeCall<"$_builder.getI64Type()">;
def MakeI32Type : NativeCodeCall<"$_builder.getI32Type()">;
-def OneToTwo : TEST_Op<"one_to_two"> {
+def OneToTwo : TEST_PatternOp<"one_to_two"> {
let arguments = (ins AnyInteger);
let results = (outs AnyInteger, AnyInteger);
}
-def TwoToOne : TEST_Op<"two_to_one"> {
+def TwoToOne : TEST_PatternOp<"two_to_one"> {
let arguments = (ins AnyInteger, AnyInteger);
let results = (outs AnyInteger);
}
@@ -2332,6 +2365,7 @@ def : Pat<(SourceOp $val, ConstantAttr<I32Attr, "66">:$attr),
def Test_LegalizerEnum_Success : ConstantStrAttr<StrAttr, "Success">;
def Test_LegalizerEnum_Failure : ConstantStrAttr<StrAttr, "Failure">;
+let partition = Test_ConversionPartition in {
def ILLegalOpA : TEST_Op<"illegal_op_a">, Results<(outs I32)>;
def ILLegalOpB : TEST_Op<"illegal_op_b">, Results<(outs I32)>;
def ILLegalOpC : TEST_Op<"illegal_op_c">, Results<(outs I32)>;
@@ -2375,6 +2409,7 @@ def IllegalOpWithRegion : TEST_Op<"illegal_op_with_region"> {
}]>];
}
def IllegalOpWithRegionAnchor : TEST_Op<"illegal_op_with_region_anchor">;
+} // let partition = Test_ConversionPartition
// Check that smaller pattern depths are chosen, i.e. prioritize more direct
// mappings.
@@ -2396,6 +2431,7 @@ def TestRewriteOp : TEST_Op<"rewrite">,
Arguments<(ins AnyType)>, Results<(outs AnyType)>;
def : Pat<(TestRewriteOp $input), (replaceWithValue $input)>;
+let partition = Test_ConversionPartition in {
// Check that patterns can specify bounded recursion when rewriting.
def TestRecursiveRewriteOp : TEST_Op<"recursive_rewrite"> {
let arguments = (ins I64Attr:$depth);
@@ -2497,6 +2533,7 @@ def TestReturnWithIgnoredValueOp : TEST_Op<"return_with_ignored_value", [
]> {
let arguments = (ins Variadic<AnyType>:$values, AnyType:$unwanted_value);
}
+} // let partition = Test_ConversionPartition
//===----------------------------------------------------------------------===//
// Test parser.
@@ -2650,6 +2687,7 @@ def ArrayOfAttrOp : TEST_Op<"array_of_attr_op"> {
// Test SideEffects
//===----------------------------------------------------------------------===//
+let partition = Test_InterfacesPartition in {
def SideEffectOp : TEST_Op<"side_effect_op",
[DeclareOpInterfaceMethods<MemoryEffectsOpInterface>,
DeclareOpInterfaceMethods<TestEffectOpInterface>]> {
@@ -3067,6 +3105,7 @@ def TestRegionTypeChangerOp : TEST_Op<"region_type_changer",
let regions = (region SizedRegion<1>:$body);
let assemblyFormat = "`(` $entries `)` $body `:` functional-type($entries, $results) attr-dict";
}
+} // let partition = Test_InterfacesPartition
//===----------------------------------------------------------------------===//
// Test TableGen generated build() methods
@@ -3202,6 +3241,7 @@ def TableGenBuildOp10 : TEST_Op<"tblgen_build_10",
// Test BufferPlacement
//===----------------------------------------------------------------------===//
+let partition = Test_InterfacesPartition in {
def GetTupleElementOp: TEST_Op<"get_tuple_element"> {
let description = [{
Test op that returns a specified element of the tuple.
@@ -3262,6 +3302,7 @@ def OpCrashLong : TEST_Op<"op_crash_long"> {
def OpCrashShort : TEST_Op<"op_crash_short"> {
let results = (outs I32);
}
+} // let partition = Test_InterfacesPartition
def : Pat<(OpCrashLong $_, $_, $_), (OpCrashShort)>;
@@ -3269,6 +3310,7 @@ def : Pat<(OpCrashLong $_, $_, $_), (OpCrashShort)>;
// Test DestinationStyleOpInterface.
//===----------------------------------------------------------------------===//
+let partition = Test_InterfacesPartition in {
def TestDestinationStyleOp :
TEST_Op<"destination_style_op", [
DestinationStyleOpInterface,
@@ -3481,6 +3523,7 @@ def TestNVVMRequiresSMArchOrFamilyCondOp :
let arguments = (ins );
let assemblyFormat = "attr-dict";
}
+} // let partition = Test_InterfacesPartition
//===----------------------------------------------------------------------===//
// Test Ops with Default-Valued String Attributes
@@ -3540,6 +3583,7 @@ def TestResource : Resource<"TestResource">;
def ParameterizedTestResource : Resource<"TestResource",
[{::mlir::StringAttr::get(getContext(), "test parameter")}]>;
+let partition = Test_InterfacesPartition in {
def TestEffectsOpA : TEST_Op<"op_with_effects_a"> {
let arguments = (ins
Arg<Variadic<AnyMemRef>, "", [MemRead]>,
@@ -3570,6 +3614,7 @@ def TestEffectsWrite : TEST_Op<"op_with_memwrite",
def TestEffectsResult : TEST_Op<"test_effects_result"> {
let results = (outs Res<I32, "", [MemAlloc, MemWrite]>);
}
+} // let partition = Test_InterfacesPartition
//===----------------------------------------------------------------------===//
// Test Ops with verifiers
@@ -3610,6 +3655,7 @@ def TestGraphLoopOp : TEST_Op<"graph_loop",
//===----------------------------------------------------------------------===//
def InferIntRangeType : AnyTypeOf<[AnyInteger, Index, VectorOfNonZeroRankOf<[AnyInteger, Index]>]>;
+let partition = Test_InterfacesPartition in {
def TestWithBoundsOp : TEST_Op<"with_bounds",
[DeclareOpInterfaceMethods<InferIntRangeInterface, ["inferResultRanges"]>,
NoMemoryEffect]> {
@@ -3755,6 +3801,7 @@ def RecursivelySpeculatableOp : TEST_Op<"recursively_speculatable_op", [
let results = (outs I32:$result);
let regions = (region SizedRegion<1>:$body);
}
+} // let partition = Test_InterfacesPartition
//===---------------------------------------------------------------------===//
// Test CSE
@@ -3826,7 +3873,7 @@ def TestVersionedOpC : TEST_Op<"versionedC"> {
// Op with a properties struct defined inline.
-def TestOpWithProperties : TEST_Op<"with_properties"> {
+def TestOpWithProperties : TEST_PropOp<"with_properties"> {
let assemblyFormat = [{
`a` `=` $a `,`
`b` `=` $b `,`
@@ -3845,13 +3892,13 @@ def TestOpWithProperties : TEST_Op<"with_properties"> {
}
def TestOpWithPropertiesAndAttr
- : TEST_Op<"with_properties_and_attr"> {
+ : TEST_PropOp<"with_properties_and_attr"> {
let assemblyFormat = "$lhs prop-dict attr-dict";
let arguments = (ins I32Attr:$lhs, IntProp<"int64_t">:$rhs);
}
-def TestOpWithCustomPropDict : TEST_Op<"with_custom_prop_dict"> {
+def TestOpWithCustomPropDict : TEST_PropOp<"with_custom_prop_dict"> {
let assemblyFormat = "prop-dict attr-dict";
let arguments = (ins
@@ -3865,7 +3912,7 @@ def TestOpWithCustomPropDict : TEST_Op<"with_custom_prop_dict"> {
}
def TestOpWithCustomAttrPropDict
- : TEST_Op<"with_custom_attr_prop_dict"> {
+ : TEST_PropOp<"with_custom_attr_prop_dict"> {
let assemblyFormat = "prop-dict attr-dict";
let arguments = (ins
TestEnumAttr:$attr,
@@ -3875,13 +3922,13 @@ def TestOpWithCustomAttrPropDict
}
def TestOpWithCustomBitEnumAttrPropDict
- : TEST_Op<"with_custom_bit_enum_attr_prop_dict"> {
+ : TEST_PropOp<"with_custom_bit_enum_attr_prop_dict"> {
let assemblyFormat = "prop-dict attr-dict";
let arguments = (ins TestBitEnumAttr:$flags, I64Attr:$next);
}
def TestOpWithPropertiesAndInferredType
- : TEST_Op<"with_properties_and_inferred_type", [
+ : TEST_PropOp<"with_properties_and_inferred_type", [
DeclareOpInterfaceMethods<InferTypeOpInterface>
]> {
let assemblyFormat = "$lhs prop-dict attr-dict";
@@ -3900,7 +3947,7 @@ def MyStructProperty : Property<"MyPropStruct"> {
let optionalParser = "return std::nullopt;";
}
-def TestOpWithWrappedProperties : TEST_Op<"with_wrapped_properties"> {
+def TestOpWithWrappedProperties : TEST_PropOp<"with_wrapped_properties"> {
let assemblyFormat = "prop-dict attr-dict";
let arguments = (ins
MyStructProperty:$prop
@@ -3908,7 +3955,7 @@ def TestOpWithWrappedProperties : TEST_Op<"with_wrapped_properties"> {
}
def TestOpWithDefaultWrappedProperties
- : TEST_Op<"with_default_wrapped_properties"> {
+ : TEST_PropOp<"with_default_wrapped_properties"> {
let assemblyFormat = "prop-dict attr-dict";
let arguments = (ins
DefaultValuedProp<MyStructProperty, "MyPropStruct{}">:$prop
@@ -3958,7 +4005,7 @@ def MyStructArrayProperty
}
def TestOpWithWrappedArrayProperties
- : TEST_Op<"with_wrapped_array_properties"> {
+ : TEST_PropOp<"with_wrapped_array_properties"> {
let assemblyFormat = "prop-dict attr-dict";
let arguments = (ins MyStructArrayProperty:$prop);
}
@@ -4154,7 +4201,7 @@ def KeyValueSpecializedOptionalProperty
}
def TestOpWithKeyValueParserBoundaries
- : TEST_Op<"with_key_value_parser_boundaries"> {
+ : TEST_PropOp<"with_key_value_parser_boundaries"> {
let assemblyFormat = "prop-dict attr-dict";
let arguments = (ins
KeyValueListProperty:$values,
@@ -4173,25 +4220,25 @@ def MyStructProperty2 : Property<"MyPropStruct"> {
let convertFromAttribute = "return MyPropStruct::setFromAttr($_storage, $_attr, $_diag);";
}
-def TestOpWithWrappedProperties2 : TEST_Op<"with_wrapped_properties2"> {
+def TestOpWithWrappedProperties2 : TEST_PropOp<"with_wrapped_properties2"> {
let assemblyFormat = "prop-dict attr-dict";
let arguments = (ins
MyStructProperty2:$prop
);
}
-def TestOpWithEmptyProperties : TEST_Op<"empty_properties"> {
+def TestOpWithEmptyProperties : TEST_PropOp<"empty_properties"> {
let assemblyFormat = "prop-dict attr-dict";
let arguments = (ins);
}
-def TestOpUsingPropertyInCustom : TEST_Op<"using_property_in_custom"> {
+def TestOpUsingPropertyInCustom : TEST_PropOp<"using_property_in_custom"> {
let assemblyFormat = "custom<UsingPropertyInCustom>($prop) attr-dict";
let arguments = (ins IntArrayProp<I64Prop>:$prop);
}
def TestOpUsingPropertyInCustomAndOther
- : TEST_Op<"using_property_in_custom_and_other"> {
+ : TEST_PropOp<"using_property_in_custom_and_other"> {
let assemblyFormat = "custom<UsingPropertyInCustom>($prop) prop-dict attr-dict";
let arguments = (ins
IntArrayProp<I64Prop>:$prop,
@@ -4199,7 +4246,7 @@ def TestOpUsingPropertyInCustomAndOther
);
}
-def TestOpWithVariadicSegmentProperties : TEST_Op<"variadic_segment_prop",
+def TestOpWithVariadicSegmentProperties : TEST_PropOp<"variadic_segment_prop",
[AttrSizedOperandSegments, AttrSizedResultSegments]> {
let arguments = (ins Variadic<I64>:$a1, Variadic<I64>:$a2);
let results = (outs Variadic<I64>:$b1, Variadic<I64>:$b2);
@@ -4215,7 +4262,7 @@ def TestOpWithVariadicSegmentProperties : TEST_Op<"variadic_segment_prop",
// `<{...}>` properties dictionary (since there is no other way to recover the
// segment sizes on re-parse), so this op exercises the parser's ability to
// read those trait-injected properties back out of `prop-dict`.
-def TestOpWithBulkTypesAndSegmentProperties : TEST_Op<"variadic_segment_prop_bulk_type",
+def TestOpWithBulkTypesAndSegmentProperties : TEST_PropOp<"variadic_segment_prop_bulk_type",
[AttrSizedOperandSegments, AttrSizedResultSegments]> {
let arguments = (ins Variadic<I64>:$a1, Variadic<I64>:$a2);
let results = (outs Variadic<I64>:$b1, Variadic<I64>:$b2);
@@ -4224,7 +4271,8 @@ def TestOpWithBulkTypesAndSegmentProperties : TEST_Op<"variadic_segment_prop_bul
}];
}
-def TestOpUsingPropertyRefInCustom : TEST_Op<"using_property_ref_in_custom"> {
+def TestOpUsingPropertyRefInCustom :
+ TEST_PropOp<"using_property_ref_in_custom"> {
let assemblyFormat = "custom<IntProperty>($first) `+` custom<SumProperty>($second, ref($first)) attr-dict";
let arguments = (ins IntProp<"int64_t">:$first, IntProp<"int64_t">:$second);
}
@@ -4236,7 +4284,7 @@ def IntPropertyWithWorseBytecode : Property<"int64_t"> {
}
def TestOpUsingIntPropertyWithWorseBytecode
- : TEST_Op<"using_int_property_with_worse_bytecode"> {
+ : TEST_PropOp<"using_int_property_with_worse_bytecode"> {
let arguments = (ins IntPropertyWithWorseBytecode:$value);
}
@@ -4255,7 +4303,7 @@ def PropertiesWithCustomPrint : Property<"PropertiesWithCustomPrint"> {
}];
}
-def TestOpWithNiceProperties : TEST_Op<"with_nice_properties"> {
+def TestOpWithNiceProperties : TEST_PropOp<"with_nice_properties"> {
let hasCustomPropertiesPrinter = 1;
let assemblyFormat = "prop-dict attr-dict";
let arguments = (ins
@@ -4320,7 +4368,7 @@ def VersionedProperties : Property<"VersionedProperties"> {
}];
}
-def TestOpWithVersionedProperties : TEST_Op<"with_versioned_properties"> {
+def TestOpWithVersionedProperties : TEST_PropOp<"with_versioned_properties"> {
let hasCustomPropertiesPrinter = 1;
let assemblyFormat = "prop-dict attr-dict";
let arguments = (ins
@@ -4356,7 +4404,7 @@ def TestOpWithVersionedProperties : TEST_Op<"with_versioned_properties"> {
}];
}
-def TestOpWithDefaultValuedProperties : TEST_Op<"with_default_valued_properties"> {
+def TestOpWithDefaultValuedProperties : TEST_PropOp<"with_default_valued_properties"> {
let assemblyFormat = [{
($a^) : (`na`)?
($b^)?
@@ -4370,7 +4418,7 @@ def TestOpWithDefaultValuedProperties : TEST_Op<"with_default_valued_properties"
UnitProp:$unit);
}
-def TestOpWithOptionalProperties : TEST_Op<"with_optional_properties"> {
+def TestOpWithOptionalProperties : TEST_PropOp<"with_optional_properties"> {
let assemblyFormat = [{
(`anAttr` `=` $anAttr^)?
(`simple` `=` $simple^)?
@@ -4399,7 +4447,7 @@ def TestOpWithOptionalProperties : TEST_Op<"with_optional_properties"> {
OptionalProp<UnitProp>:$maybeUnit);
}
-def TestOpWithArrayProperties : TEST_Op<"with_array_properties"> {
+def TestOpWithArrayProperties : TEST_PropOp<"with_array_properties"> {
let assemblyFormat = [{
`ints` `=` $ints
`strings` `=` $strings
@@ -4429,7 +4477,7 @@ class NonEmptyArray<Property p> : ConfinedProp
<ArrayProp<p>, Neg<CPred<"$_self.empty()">>,
"non-empty array of " # p.summary>;
-def OpWithPropertyPredicates : TEST_Op<"op_with_property_predicates"> {
+def OpWithPropertyPredicates : TEST_PropOp<"op_with_property_predicates"> {
let arguments = (ins
NonNegativeI64Prop:$scalar,
OptionalProp<NonNegativeI64Prop>:$optional,
@@ -4446,7 +4494,7 @@ def OpWithPropertyPredicates : TEST_Op<"op_with_property_predicates"> {
let assemblyFormat = "attr-dict prop-dict";
}
-def TestPropPatternOp1 : TEST_Op<"prop_pattern_op_1"> {
+def TestPropPatternOp1 : TEST_PropOp<"prop_pattern_op_1"> {
let arguments = (ins
StringProp:$tag,
I64Prop:$val,
@@ -4456,7 +4504,7 @@ def TestPropPatternOp1 : TEST_Op<"prop_pattern_op_1"> {
let assemblyFormat = "$tag $val $cond attr-dict";
}
-def TestPropPatternOp2 : TEST_Op<"prop_pattern_op_2"> {
+def TestPropPatternOp2 : TEST_PropOp<"prop_pattern_op_2"> {
let arguments = (ins
I32:$input,
StringProp:$tag
@@ -4482,6 +4530,7 @@ def : Pat<
// Test Dataflow
//===----------------------------------------------------------------------===//
+let partition = Test_InterfacesPartition in {
def TestCallAndStoreOp : TEST_Op<"call_and_store",
[DeclareOpInterfaceMethods<CallOpInterface>]> {
let arguments = (ins
@@ -4892,6 +4941,7 @@ def TestDummyCastOp : TEST_Op<"dummy_cast_op", []> {
);
let assemblyFormat = "$source attr-dict `:` type($source) `to` type($dest)";
}
+} // let partition = Test_InterfacesPartition
//===----------------------------------------------------------------------===//
// Test assembly format references
diff --git a/mlir/test/lib/Dialect/Test/TestOps.cpp b/mlir/test/lib/Dialect/Test/TestOpsAttributes.cpp
similarity index 65%
rename from mlir/test/lib/Dialect/Test/TestOps.cpp
rename to mlir/test/lib/Dialect/Test/TestOpsAttributes.cpp
index 47d5b1b19121ef..b0bbf1b053ac0a 100644
--- a/mlir/test/lib/Dialect/Test/TestOps.cpp
+++ b/mlir/test/lib/Dialect/Test/TestOpsAttributes.cpp
@@ -1,4 +1,4 @@
-//===- TestOps.cpp - MLIR Test Dialect Operations ------------------------===//
+//===- TestOpsAttributes.cpp - Test dialect ops in partition `Attributes` -===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
@@ -6,12 +6,11 @@
//
//===----------------------------------------------------------------------===//
-#include "TestOps.h"
-#include "TestDialect.h"
+#include "TestOpsAttributes.h"
#include "TestFormatUtils.h"
-#include "mlir/Dialect/Arith/IR/Arith.h"
using namespace mlir;
using namespace test;
-#include "TestOps.cpp.inc"
+#define GET_OP_CLASSES
+#include "TestOpsAttributes.cpp.inc"
diff --git a/mlir/test/lib/Dialect/Test/TestOpsAttributes.h b/mlir/test/lib/Dialect/Test/TestOpsAttributes.h
new file mode 100644
index 00000000000000..039313b24931ab
--- /dev/null
+++ b/mlir/test/lib/Dialect/Test/TestOpsAttributes.h
@@ -0,0 +1,17 @@
+//===- TestOpsAttributes.h - Test dialect ops in `Attributes` ---*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_TEST_DIALECT_TEST_TESTOPSATTRIBUTES_H
+#define MLIR_TEST_DIALECT_TEST_TESTOPSATTRIBUTES_H
+
+#include "TestDialect.h"
+
+#define GET_OP_CLASSES
+#include "TestOpsAttributes.h.inc"
+
+#endif // MLIR_TEST_DIALECT_TEST_TESTOPSATTRIBUTES_H
diff --git a/mlir/test/lib/Dialect/Test/TestOpsConversion.cpp b/mlir/test/lib/Dialect/Test/TestOpsConversion.cpp
new file mode 100644
index 00000000000000..0547b852821fec
--- /dev/null
+++ b/mlir/test/lib/Dialect/Test/TestOpsConversion.cpp
@@ -0,0 +1,15 @@
+//===- TestOpsConversion.cpp - Test dialect ops in partition `Conversion` -===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "TestOpsConversion.h"
+
+using namespace mlir;
+using namespace test;
+
+#define GET_OP_CLASSES
+#include "TestOpsConversion.cpp.inc"
diff --git a/mlir/test/lib/Dialect/Test/TestOpsConversion.h b/mlir/test/lib/Dialect/Test/TestOpsConversion.h
new file mode 100644
index 00000000000000..a46e4dee6a94bf
--- /dev/null
+++ b/mlir/test/lib/Dialect/Test/TestOpsConversion.h
@@ -0,0 +1,17 @@
+//===- TestOpsConversion.h - Test dialect ops in `Conversion` ---*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_TEST_DIALECT_TEST_TESTOPSCONVERSION_H
+#define MLIR_TEST_DIALECT_TEST_TESTOPSCONVERSION_H
+
+#include "TestDialect.h"
+
+#define GET_OP_CLASSES
+#include "TestOpsConversion.h.inc"
+
+#endif // MLIR_TEST_DIALECT_TEST_TESTOPSCONVERSION_H
diff --git a/mlir/test/lib/Dialect/Test/TestOpsCore.cpp b/mlir/test/lib/Dialect/Test/TestOpsCore.cpp
new file mode 100644
index 00000000000000..2f6da308774291
--- /dev/null
+++ b/mlir/test/lib/Dialect/Test/TestOpsCore.cpp
@@ -0,0 +1,19 @@
+//===- TestOpsCore.cpp - Test dialect ops in partition `Core` -------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "TestOpsCore.h"
+#include "TestFormatUtils.h"
+// Some implicit terminators of `Core` ops are in other partitions.
+#include "TestOpsInterfaces.h"
+#include "TestOpsTraits.h"
+
+using namespace mlir;
+using namespace test;
+
+#define GET_OP_CLASSES
+#include "TestOpsCore.cpp.inc"
diff --git a/mlir/test/lib/Dialect/Test/TestOpsCore.h b/mlir/test/lib/Dialect/Test/TestOpsCore.h
new file mode 100644
index 00000000000000..fbcf4fced9d35c
--- /dev/null
+++ b/mlir/test/lib/Dialect/Test/TestOpsCore.h
@@ -0,0 +1,18 @@
+//===- TestOpsCore.h - Test dialect ops in partition `Core` -----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_TEST_DIALECT_TEST_TESTOPSCORE_H
+#define MLIR_TEST_DIALECT_TEST_TESTOPSCORE_H
+
+#include "TestDialect.h"
+#include "mlir/Interfaces/FunctionInterfaces.h"
+
+#define GET_OP_CLASSES
+#include "TestOpsCore.h.inc"
+
+#endif // MLIR_TEST_DIALECT_TEST_TESTOPSCORE_H
diff --git a/mlir/test/lib/Dialect/Test/TestOpsInterfaces.cpp b/mlir/test/lib/Dialect/Test/TestOpsInterfaces.cpp
new file mode 100644
index 00000000000000..31a53af4bf383f
--- /dev/null
+++ b/mlir/test/lib/Dialect/Test/TestOpsInterfaces.cpp
@@ -0,0 +1,16 @@
+//===- TestOpsInterfaces.cpp - Test dialect ops in partition `Interfaces` -===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "TestOpsInterfaces.h"
+#include "TestFormatUtils.h"
+
+using namespace mlir;
+using namespace test;
+
+#define GET_OP_CLASSES
+#include "TestOpsInterfaces.cpp.inc"
diff --git a/mlir/test/lib/Dialect/Test/TestOpsInterfaces.h b/mlir/test/lib/Dialect/Test/TestOpsInterfaces.h
new file mode 100644
index 00000000000000..b9e614232f9a03
--- /dev/null
+++ b/mlir/test/lib/Dialect/Test/TestOpsInterfaces.h
@@ -0,0 +1,90 @@
+//===- TestOpsInterfaces.h - Test dialect ops in `Interfaces` ---*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_TEST_DIALECT_TEST_TESTOPSINTERFACES_H
+#define MLIR_TEST_DIALECT_TEST_TESTOPSINTERFACES_H
+
+#include "TestDialect.h"
+#include "mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h"
+#include "mlir/Dialect/LLVMIR/NVVMRequiresSMTraits.h"
+#include "mlir/Interfaces/DataLayoutInterfaces.h"
+#include "mlir/Interfaces/DestinationStyleOpInterface.h"
+#include "mlir/Interfaces/MemorySlotInterfaces.h"
+#include "mlir/Interfaces/TilingInterface.h"
+
+namespace test {
+//===----------------------------------------------------------------------===//
+// TestResource
+//===----------------------------------------------------------------------===//
+
+/// A test resource for side effects (under DefaultResource).
+struct TestResource : public mlir::SideEffects::Resource::Base<
+ TestResource, mlir::SideEffects::DefaultResource> {
+ llvm::StringRef getName() const final { return "<Test>"; }
+ mlir::SideEffects::Resource *getParent() const override {
+ return mlir::SideEffects::DefaultResource::get();
+ }
+};
+
+/// A test resource that is a root (disjoint from DefaultResource).
+struct TestNonAddressableResource
+ : public mlir::SideEffects::Resource::Base<TestNonAddressableResource> {
+ llvm::StringRef getName() const final { return "<TestNonAddressable>"; }
+ bool isAddressable() const override { return false; }
+};
+
+/// Two disjoint sub-resources (roots) for testing sibling disjointness.
+struct TestNonAddressableSubResourceA
+ : public mlir::SideEffects::Resource::Base<TestNonAddressableSubResourceA> {
+ TestNonAddressableSubResourceA() = default;
+ llvm::StringRef getName() const override {
+ return "TestNonAddressableSubResourceA";
+ }
+ bool isAddressable() const override { return false; }
+
+protected:
+ TestNonAddressableSubResourceA(mlir::TypeID id) : Base(id) {}
+};
+
+struct TestNonAddressableSubResourceB
+ : public mlir::SideEffects::Resource::Base<TestNonAddressableSubResourceB> {
+ TestNonAddressableSubResourceB() = default;
+ llvm::StringRef getName() const override {
+ return "TestNonAddressableSubResourceB";
+ }
+ bool isAddressable() const override { return false; }
+
+protected:
+ TestNonAddressableSubResourceB(mlir::TypeID id) : Base(id) {}
+};
+
+struct TestNonAddressableResourceA
+ : public mlir::SideEffects::Resource::Base<TestNonAddressableResourceA,
+ TestNonAddressableSubResourceA> {
+ llvm::StringRef getName() const final { return "<TestNonAddressableA>"; }
+ bool isAddressable() const override { return false; }
+ mlir::SideEffects::Resource *getParent() const override {
+ return TestNonAddressableSubResourceA::get();
+ }
+};
+
+struct TestNonAddressableResourceB
+ : public mlir::SideEffects::Resource::Base<TestNonAddressableResourceB,
+ TestNonAddressableSubResourceB> {
+ llvm::StringRef getName() const final { return "<TestNonAddressableB>"; }
+ bool isAddressable() const override { return false; }
+ mlir::SideEffects::Resource *getParent() const override {
+ return TestNonAddressableSubResourceB::get();
+ }
+};
+} // namespace test
+
+#define GET_OP_CLASSES
+#include "TestOpsInterfaces.h.inc"
+
+#endif // MLIR_TEST_DIALECT_TEST_TESTOPSINTERFACES_H
diff --git a/mlir/test/lib/Dialect/Test/TestOpsPatterns.cpp b/mlir/test/lib/Dialect/Test/TestOpsPatterns.cpp
new file mode 100644
index 00000000000000..a3423bda38a02d
--- /dev/null
+++ b/mlir/test/lib/Dialect/Test/TestOpsPatterns.cpp
@@ -0,0 +1,15 @@
+//===- TestOpsPatterns.cpp - Test dialect ops in partition `Patterns` -----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "TestOpsPatterns.h"
+
+using namespace mlir;
+using namespace test;
+
+#define GET_OP_CLASSES
+#include "TestOpsPatterns.cpp.inc"
diff --git a/mlir/test/lib/Dialect/Test/TestOpsPatterns.h b/mlir/test/lib/Dialect/Test/TestOpsPatterns.h
new file mode 100644
index 00000000000000..3df3f7fb6af166
--- /dev/null
+++ b/mlir/test/lib/Dialect/Test/TestOpsPatterns.h
@@ -0,0 +1,17 @@
+//===- TestOpsPatterns.h - Test dialect ops in `Patterns` -------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_TEST_DIALECT_TEST_TESTOPSPATTERNS_H
+#define MLIR_TEST_DIALECT_TEST_TESTOPSPATTERNS_H
+
+#include "TestDialect.h"
+
+#define GET_OP_CLASSES
+#include "TestOpsPatterns.h.inc"
+
+#endif // MLIR_TEST_DIALECT_TEST_TESTOPSPATTERNS_H
diff --git a/mlir/test/lib/Dialect/Test/TestOpsProperties.cpp b/mlir/test/lib/Dialect/Test/TestOpsProperties.cpp
new file mode 100644
index 00000000000000..81becab1e83857
--- /dev/null
+++ b/mlir/test/lib/Dialect/Test/TestOpsProperties.cpp
@@ -0,0 +1,16 @@
+//===- TestOpsProperties.cpp - Test dialect ops in partition `Properties` -===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "TestOpsProperties.h"
+#include "TestFormatUtils.h"
+
+using namespace mlir;
+using namespace test;
+
+#define GET_OP_CLASSES
+#include "TestOpsProperties.cpp.inc"
diff --git a/mlir/test/lib/Dialect/Test/TestOpsProperties.h b/mlir/test/lib/Dialect/Test/TestOpsProperties.h
new file mode 100644
index 00000000000000..d597a3f725f875
--- /dev/null
+++ b/mlir/test/lib/Dialect/Test/TestOpsProperties.h
@@ -0,0 +1,154 @@
+//===- TestOpsProperties.h - Test dialect ops in `Properties` ---*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_TEST_DIALECT_TEST_TESTOPSPROPERTIES_H
+#define MLIR_TEST_DIALECT_TEST_TESTOPSPROPERTIES_H
+
+#include "TestDialect.h"
+#include "llvm/ADT/SmallVector.h"
+#include <optional>
+
+namespace test {
+// Keep this type distinct from SmallVector<int32_t>, which another test parser
+// uses with the generic FieldParser specialization in a separate translation
+// unit.
+struct KeyValueSpecializedList : llvm::SmallVector<int32_t> {
+ using llvm::SmallVector<int32_t>::SmallVector;
+};
+} // namespace test
+
+namespace mlir {
+// Self-delimiting full specializations remain usable in a keyed prop-dict even
+// when their storage type is optional or container-like.
+template <>
+struct FieldParser<test::KeyValueSpecializedList> {
+ static FailureOr<test::KeyValueSpecializedList> parse(AsmParser &parser) {
+ test::KeyValueSpecializedList values;
+ if (parser.parseCommaSeparatedList(AsmParser::Delimiter::Square, [&]() {
+ int32_t value;
+ if (parser.parseInteger(value))
+ return failure();
+ values.push_back(value);
+ return success();
+ }))
+ return failure();
+ return values;
+ }
+};
+
+template <>
+struct FieldParser<std::optional<int16_t>> {
+ static FailureOr<std::optional<int16_t>> parse(AsmParser &parser) {
+ if (succeeded(parser.parseOptionalKeyword("none")))
+ return std::optional<int16_t>{};
+ int16_t value;
+ if (parser.parseKeyword("some") || parser.parseLess() ||
+ parser.parseInteger(value) || parser.parseGreater())
+ return failure();
+ return std::optional<int16_t>{value};
+ }
+};
+} // namespace mlir
+
+namespace test {
+//===----------------------------------------------------------------------===//
+// PropertiesWithCustomPrint
+//===----------------------------------------------------------------------===//
+
+struct PropertiesWithCustomPrint {
+ /// A shared_ptr to a const object is safe: it is equivalent to a value-based
+ /// member. Here the label will be deallocated when the last operation
+ /// refering to it is destroyed. However there is no pool-allocation: this is
+ /// offloaded to the client.
+ std::shared_ptr<const std::string> label;
+ int value;
+ bool operator==(const PropertiesWithCustomPrint &rhs) const {
+ return value == rhs.value && *label == *rhs.label;
+ }
+};
+
+llvm::LogicalResult setPropertiesFromAttribute(
+ PropertiesWithCustomPrint &prop, mlir::Attribute attr,
+ llvm::function_ref<mlir::InFlightDiagnostic()> emitError);
+mlir::DictionaryAttr
+getPropertiesAsAttribute(mlir::MLIRContext *ctx,
+ const PropertiesWithCustomPrint &prop);
+llvm::hash_code computeHash(const PropertiesWithCustomPrint &prop);
+void customPrintProperties(mlir::OpAsmPrinter &p,
+ const PropertiesWithCustomPrint &prop);
+mlir::ParseResult customParseProperties(mlir::OpAsmParser &parser,
+ PropertiesWithCustomPrint &prop);
+
+//===----------------------------------------------------------------------===//
+// MyPropStruct
+//===----------------------------------------------------------------------===//
+namespace test_properties {
+class MyPropStruct {
+public:
+ std::string content;
+ // These three methods are invoked through the `MyStructProperty` wrapper
+ // defined in TestOps.td
+ mlir::Attribute asAttribute(mlir::MLIRContext *ctx) const;
+ static llvm::LogicalResult
+ setFromAttr(MyPropStruct &prop, mlir::Attribute attr,
+ llvm::function_ref<mlir::InFlightDiagnostic()> emitError);
+ llvm::hash_code hash() const;
+ bool operator==(const MyPropStruct &rhs) const {
+ return content == rhs.content;
+ }
+};
+inline llvm::hash_code hash_value(const MyPropStruct &S) { return S.hash(); }
+} // namespace test_properties
+using test_properties::MyPropStruct;
+
+llvm::LogicalResult readFromMlirBytecode(mlir::DialectBytecodeReader &reader,
+ MyPropStruct &prop);
+void writeToMlirBytecode(mlir::DialectBytecodeWriter &writer,
+ MyPropStruct &prop);
+
+//===----------------------------------------------------------------------===//
+// VersionedProperties
+//===----------------------------------------------------------------------===//
+
+struct VersionedProperties {
+ // For the sake of testing, assume that this object was associated to version
+ // 1.2 of the test dialect when having only one int value. In the current
+ // version 2.0, the property has two values. We also assume that the class is
+ // upgrade-able if value2 = 0.
+ int value1;
+ int value2;
+ bool operator==(const VersionedProperties &rhs) const {
+ return value1 == rhs.value1 && value2 == rhs.value2;
+ }
+};
+
+llvm::LogicalResult setPropertiesFromAttribute(
+ VersionedProperties &prop, mlir::Attribute attr,
+ llvm::function_ref<mlir::InFlightDiagnostic()> emitError);
+mlir::DictionaryAttr getPropertiesAsAttribute(mlir::MLIRContext *ctx,
+ const VersionedProperties &prop);
+llvm::hash_code computeHash(const VersionedProperties &prop);
+void customPrintProperties(mlir::OpAsmPrinter &p,
+ const VersionedProperties &prop);
+mlir::ParseResult customParseProperties(mlir::OpAsmParser &parser,
+ VersionedProperties &prop);
+
+//===----------------------------------------------------------------------===//
+// Bytecode Support
+//===----------------------------------------------------------------------===//
+
+llvm::LogicalResult readFromMlirBytecode(mlir::DialectBytecodeReader &reader,
+ llvm::MutableArrayRef<int64_t> prop);
+void writeToMlirBytecode(mlir::DialectBytecodeWriter &writer,
+ llvm::ArrayRef<int64_t> prop);
+} // namespace test
+
+#define GET_OP_CLASSES
+#include "TestOpsProperties.h.inc"
+
+#endif // MLIR_TEST_DIALECT_TEST_TESTOPSPROPERTIES_H
diff --git a/mlir/test/lib/Dialect/Test/TestOpsSyntax.cpp b/mlir/test/lib/Dialect/Test/TestOpsSyntax.cpp
index 3a971e462e2b8a..4c122c7dc34dc7 100644
--- a/mlir/test/lib/Dialect/Test/TestOpsSyntax.cpp
+++ b/mlir/test/lib/Dialect/Test/TestOpsSyntax.cpp
@@ -8,7 +8,8 @@
#include "TestOpsSyntax.h"
#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsConversion.h"
+#include "TestOpsCore.h"
#include "mlir/IR/OpImplementation.h"
#include "llvm/Support/Base64.h"
@@ -509,10 +510,3 @@ static void printCustomOperationNameEntry(OpAsmPrinter &p, Operation *op,
#define GET_OP_CLASSES
#include "TestOpsSyntax.cpp.inc"
-
-void TestDialect::registerOpsSyntax() {
- addOperations<
-#define GET_OP_LIST
-#include "TestOpsSyntax.cpp.inc"
- >();
-}
diff --git a/mlir/test/lib/Dialect/Test/TestOpsSyntax.td b/mlir/test/lib/Dialect/Test/TestOpsSyntax.td
index f2d6ba3c5855e5..a81766aaa2b9c1 100644
--- a/mlir/test/lib/Dialect/Test/TestOpsSyntax.td
+++ b/mlir/test/lib/Dialect/Test/TestOpsSyntax.td
@@ -16,8 +16,12 @@ include "TestTypeDefs.td"
include "mlir/Interfaces/InferTypeOpInterface.td"
include "mlir/IR/OpBase.td"
+// The ops of this file are generated separately from TestOps.td, so they are
+// all in `Syntax`, which TestOps.td does not use.
class TEST_Op<string mnemonic, list<Trait> traits = []> :
- Op<Test_Dialect, mnemonic, traits>;
+ Op<Test_Dialect, mnemonic, traits> {
+ let partition = Test_SyntaxPartition;
+}
def WrappingRegionOp : TEST_Op<"wrapping_region",
[SingleBlockImplicitTerminator<"TestReturnOp">]> {
diff --git a/mlir/test/lib/Dialect/Test/TestOpsTraits.cpp b/mlir/test/lib/Dialect/Test/TestOpsTraits.cpp
new file mode 100644
index 00000000000000..41ce5043d261f3
--- /dev/null
+++ b/mlir/test/lib/Dialect/Test/TestOpsTraits.cpp
@@ -0,0 +1,15 @@
+//===- TestOpsTraits.cpp - Test dialect ops in partition `Traits` ---------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "TestOpsTraits.h"
+
+using namespace mlir;
+using namespace test;
+
+#define GET_OP_CLASSES
+#include "TestOpsTraits.cpp.inc"
diff --git a/mlir/test/lib/Dialect/Test/TestOpsTraits.h b/mlir/test/lib/Dialect/Test/TestOpsTraits.h
new file mode 100644
index 00000000000000..3b7606fb1ecb55
--- /dev/null
+++ b/mlir/test/lib/Dialect/Test/TestOpsTraits.h
@@ -0,0 +1,17 @@
+//===- TestOpsTraits.h - Test dialect ops in partition `Traits` -*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_TEST_DIALECT_TEST_TESTOPSTRAITS_H
+#define MLIR_TEST_DIALECT_TEST_TESTOPSTRAITS_H
+
+#include "TestDialect.h"
+
+#define GET_OP_CLASSES
+#include "TestOpsTraits.h.inc"
+
+#endif // MLIR_TEST_DIALECT_TEST_TESTOPSTRAITS_H
diff --git a/mlir/test/lib/Dialect/Test/TestToLLVMIRTranslation.cpp b/mlir/test/lib/Dialect/Test/TestToLLVMIRTranslation.cpp
index 19624c71f841e1..6f3a610a4c2bb1 100644
--- a/mlir/test/lib/Dialect/Test/TestToLLVMIRTranslation.cpp
+++ b/mlir/test/lib/Dialect/Test/TestToLLVMIRTranslation.cpp
@@ -10,8 +10,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsCore.h"
#include "mlir/IR/Builders.h"
#include "mlir/IR/BuiltinAttributes.h"
#include "mlir/IR/BuiltinOps.h"
diff --git a/mlir/test/lib/Dialect/Test/TestTraits.cpp b/mlir/test/lib/Dialect/Test/TestTraits.cpp
index d8763f562cbefc..ca111c7648fb2a 100644
--- a/mlir/test/lib/Dialect/Test/TestTraits.cpp
+++ b/mlir/test/lib/Dialect/Test/TestTraits.cpp
@@ -6,7 +6,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestOps.h"
+#include "TestOpsPatterns.h"
#include "mlir/Pass/Pass.h"
#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
diff --git a/mlir/test/lib/IR/TestBytecodeRoundtrip.cpp b/mlir/test/lib/IR/TestBytecodeRoundtrip.cpp
index d7e224f8b04ea9..e5db4c94452e2e 100644
--- a/mlir/test/lib/IR/TestBytecodeRoundtrip.cpp
+++ b/mlir/test/lib/IR/TestBytecodeRoundtrip.cpp
@@ -6,8 +6,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsCore.h"
#include "mlir/Bytecode/BytecodeReader.h"
#include "mlir/Bytecode/BytecodeWriter.h"
#include "mlir/IR/BuiltinOps.h"
diff --git a/mlir/test/lib/IR/TestClone.cpp b/mlir/test/lib/IR/TestClone.cpp
index b742b316c77126..7b18f219b915f4 100644
--- a/mlir/test/lib/IR/TestClone.cpp
+++ b/mlir/test/lib/IR/TestClone.cpp
@@ -6,7 +6,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestOps.h"
+#include "TestDialect.h"
#include "mlir/IR/BuiltinOps.h"
#include "mlir/Pass/Pass.h"
diff --git a/mlir/test/lib/IR/TestLazyLoading.cpp b/mlir/test/lib/IR/TestLazyLoading.cpp
index 187b977c7daaac..d6e284f4b8fcf5 100644
--- a/mlir/test/lib/IR/TestLazyLoading.cpp
+++ b/mlir/test/lib/IR/TestLazyLoading.cpp
@@ -6,7 +6,6 @@
//
//===----------------------------------------------------------------------===//
-#include "TestDialect.h"
#include "mlir/Bytecode/BytecodeReader.h"
#include "mlir/Bytecode/BytecodeWriter.h"
#include "mlir/IR/BuiltinOps.h"
diff --git a/mlir/test/lib/IR/TestRegions.cpp b/mlir/test/lib/IR/TestRegions.cpp
index e850691631be2f..7a4a842847d9d9 100644
--- a/mlir/test/lib/IR/TestRegions.cpp
+++ b/mlir/test/lib/IR/TestRegions.cpp
@@ -6,7 +6,6 @@
//
//===----------------------------------------------------------------------===//
-#include "TestDialect.h"
#include "mlir/Dialect/Func/IR/FuncOps.h"
#include "mlir/IR/BuiltinOps.h"
#include "mlir/Pass/Pass.h"
diff --git a/mlir/test/lib/IR/TestSideEffects.cpp b/mlir/test/lib/IR/TestSideEffects.cpp
index db27b323c3fd25..c51f406bd0c000 100644
--- a/mlir/test/lib/IR/TestSideEffects.cpp
+++ b/mlir/test/lib/IR/TestSideEffects.cpp
@@ -6,7 +6,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestOps.h"
+#include "TestDialect.h"
#include "mlir/Pass/Pass.h"
using namespace mlir;
diff --git a/mlir/test/lib/IR/TestSymbolUses.cpp b/mlir/test/lib/IR/TestSymbolUses.cpp
index e8170b8531d1d8..4d4c8e769c25bd 100644
--- a/mlir/test/lib/IR/TestSymbolUses.cpp
+++ b/mlir/test/lib/IR/TestSymbolUses.cpp
@@ -6,7 +6,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestOps.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
#include "mlir/IR/BuiltinOps.h"
#include "mlir/Pass/Pass.h"
diff --git a/mlir/test/lib/IR/TestTypes.cpp b/mlir/test/lib/IR/TestTypes.cpp
index c6bce111d3ea7f..ce0f799d517a68 100644
--- a/mlir/test/lib/IR/TestTypes.cpp
+++ b/mlir/test/lib/IR/TestTypes.cpp
@@ -7,7 +7,6 @@
//===----------------------------------------------------------------------===//
#include "TestTypes.h"
-#include "TestDialect.h"
#include "mlir/Dialect/Func/IR/FuncOps.h"
#include "mlir/Pass/Pass.h"
diff --git a/mlir/test/lib/IR/TestVisitorsGeneric.cpp b/mlir/test/lib/IR/TestVisitorsGeneric.cpp
index 20193bd71a6f5d..04f7363a079275 100644
--- a/mlir/test/lib/IR/TestVisitorsGeneric.cpp
+++ b/mlir/test/lib/IR/TestVisitorsGeneric.cpp
@@ -6,7 +6,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestOps.h"
+#include "TestOpsCore.h"
#include "mlir/Pass/Pass.h"
using namespace mlir;
diff --git a/mlir/test/lib/Pass/TestPassManager.cpp b/mlir/test/lib/Pass/TestPassManager.cpp
index df2736bd666ca2..56d8c9ae6d1594 100644
--- a/mlir/test/lib/Pass/TestPassManager.cpp
+++ b/mlir/test/lib/Pass/TestPassManager.cpp
@@ -6,8 +6,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsCore.h"
#include "mlir/Dialect/Func/IR/FuncOps.h"
#include "mlir/IR/BuiltinOps.h"
#include "mlir/Pass/Pass.h"
diff --git a/mlir/test/lib/Tools/PDLL/TestPDLL.cpp b/mlir/test/lib/Tools/PDLL/TestPDLL.cpp
index d2c17fc63f1788..3e60f23deac7ea 100644
--- a/mlir/test/lib/Tools/PDLL/TestPDLL.cpp
+++ b/mlir/test/lib/Tools/PDLL/TestPDLL.cpp
@@ -6,8 +6,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsPatterns.h"
#include "mlir/Dialect/PDL/IR/PDL.h"
#include "mlir/Dialect/PDLInterp/IR/PDLInterp.h"
#include "mlir/Interfaces/CastInterfaces.h"
diff --git a/mlir/test/lib/Transforms/TestInlining.cpp b/mlir/test/lib/Transforms/TestInlining.cpp
index ae904a92a5d68d..9c941afc916522 100644
--- a/mlir/test/lib/Transforms/TestInlining.cpp
+++ b/mlir/test/lib/Transforms/TestInlining.cpp
@@ -12,8 +12,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsCore.h"
#include "mlir/Dialect/Func/IR/FuncOps.h"
#include "mlir/IR/BuiltinOps.h"
#include "mlir/IR/IRMapping.h"
diff --git a/mlir/test/lib/Transforms/TestInliningCallback.cpp b/mlir/test/lib/Transforms/TestInliningCallback.cpp
index 2888c3cf3dda84..d55b53f0652ffe 100644
--- a/mlir/test/lib/Transforms/TestInliningCallback.cpp
+++ b/mlir/test/lib/Transforms/TestInliningCallback.cpp
@@ -10,8 +10,8 @@
// canHandleMultipleBlocks and doClone.
//===----------------------------------------------------------------------===//
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsConversion.h"
+#include "TestOpsCore.h"
#include "mlir/Analysis/CallGraph.h"
#include "mlir/Dialect/Func/IR/FuncOps.h"
#include "mlir/Dialect/SCF/IR/SCF.h"
diff --git a/mlir/test/lib/Transforms/TestMakeIsolatedFromAbove.cpp b/mlir/test/lib/Transforms/TestMakeIsolatedFromAbove.cpp
index 85d4e7498c918c..fc1b31f773ba05 100644
--- a/mlir/test/lib/Transforms/TestMakeIsolatedFromAbove.cpp
+++ b/mlir/test/lib/Transforms/TestMakeIsolatedFromAbove.cpp
@@ -6,8 +6,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsCore.h"
#include "mlir/Dialect/Func/IR/FuncOps.h"
#include "mlir/IR/PatternMatch.h"
#include "mlir/Pass/Pass.h"
diff --git a/mlir/test/mlir-tblgen/op-partitions-error.td b/mlir/test/mlir-tblgen/op-partitions-error.td
new file mode 100644
index 00000000000000..2aaf4b7ed03d72
--- /dev/null
+++ b/mlir/test/mlir-tblgen/op-partitions-error.td
@@ -0,0 +1,74 @@
+// RUN: not mlir-tblgen -gen-op-decls -I %S/../../include -DERROR1 %s 2>&1 | FileCheck --check-prefix=ERROR1 %s
+// RUN: not mlir-tblgen -gen-op-defs -I %S/../../include -DERROR2 %s 2>&1 | FileCheck --check-prefix=ERROR2 %s
+// RUN: not mlir-tblgen -gen-op-defs -I %S/../../include -DERROR3 %s 2>&1 | FileCheck --check-prefix=ERROR3 %s
+// RUN: not mlir-tblgen -gen-op-defs -op-partition=Memroy -I %S/../../include %s 2>&1 | FileCheck --check-prefix=ERROR4 %s
+// RUN: not mlir-tblgen -gen-op-defs -op-partition=Memory -op-shard-count=2 -I %S/../../include %s 2>&1 | FileCheck --check-prefix=ERROR5 %s
+// RUN: not mlir-tblgen -gen-op-defs -op-shard-count=2 -I %S/../../include %s 2>&1 | FileCheck --check-prefix=ERROR5 %s
+// RUN: not mlir-tblgen -gen-dialect-decls -I %S/../../include -DERROR6 %s 2>&1 | FileCheck --check-prefix=ERROR6 %s
+// RUN: not mlir-tblgen -gen-dialect-decls -I %S/../../include -DERROR7 %s 2>&1 | FileCheck --check-prefix=ERROR7 %s
+// RUN: not mlir-tblgen -gen-op-doc -op-partition=Memroy -I %S/../../include %s 2>&1 | FileCheck --check-prefix=ERROR4 %s
+// RUN: not mlir-tblgen -gen-op-defs -I %S/../../include -DERROR8 %s 2>&1 | FileCheck --check-prefix=ERROR8 %s
+// RUN: not mlir-tblgen -gen-op-defs -I %S/../../include -DERROR9 %s 2>&1 | FileCheck --check-prefix=ERROR9 %s
+
+include "mlir/IR/OpBase.td"
+
+def MemoryPartition : Partition<"Memory">;
+def CorePartition : Partition<"Core">;
+#ifdef ERROR6
+// ERROR6: error: assertion failed: op partition name 'my-partition' is not a C++ identifier
+def InvalidPartition : Partition<"my-partition">;
+#endif
+def SnakeCasePartition : Partition<"control_flow">;
+def CamelCasePartition : Partition<"ControlFlow">;
+
+def Test_Dialect : Dialect {
+ let name = "test";
+ let opPartitions = [MemoryPartition];
+#ifdef ERROR7
+ // ERROR7: error: op partitions 'control_flow' and 'ControlFlow' of dialect 'test' have the same registration hook 'registerOperationsControlFlow'
+ let opPartitions = [MemoryPartition, SnakeCasePartition, CamelCasePartition];
+#endif
+#ifdef ERROR9
+ // ERROR9: error: assertion failed: default op partition 'Core' (record 'CorePartition') of dialect 'test' is not one of its `opPartitions`
+ let defaultOpPartition = CorePartition;
+#endif
+}
+
+def LoadOp : Op<Test_Dialect, "load"> {
+ let partition = MemoryPartition;
+}
+
+#ifdef ERROR1
+// ERROR1: error: assertion failed: op 'NoPartitionOp' is in no partition, but dialect 'test' has `opPartitions` and no `defaultOpPartition`
+def NoPartitionOp : Op<Test_Dialect, "no_partition">;
+#endif
+
+#ifdef ERROR2
+// ERROR2: error: assertion failed: op 'UnknownPartitionOp' is in partition 'Core' (record 'CorePartition'), which is not one of the `opPartitions` of dialect 'test'
+def UnknownPartitionOp : Op<Test_Dialect, "unknown_partition"> {
+ let partition = CorePartition;
+}
+#endif
+
+#ifdef ERROR3
+def Other_Dialect : Dialect {
+ let name = "other";
+}
+// ERROR3: error: assertion failed: op 'UnpartitionedDialectOp' is in partition 'Memory' (record 'MemoryPartition'), which is not one of the `opPartitions` of dialect 'other'
+def UnpartitionedDialectOp : Op<Other_Dialect, "op"> {
+ let partition = MemoryPartition;
+}
+#endif
+
+#ifdef ERROR8
+// Partitions are matched by record, not by name.
+def OtherMemoryPartition : Partition<"Memory">;
+// ERROR8: error: assertion failed: op 'OtherMemoryOp' is in partition 'Memory' (record 'OtherMemoryPartition'), which is not one of the `opPartitions` of dialect 'test'
+def OtherMemoryOp : Op<Test_Dialect, "other_memory"> {
+ let partition = OtherMemoryPartition;
+}
+#endif
+
+// ERROR4: error: requested op partition 'Memroy' is not declared in the `opPartitions` of any dialect with ops
+
+// ERROR5: error: dialect 'test' has `opPartitions`, which cannot be combined with -op-shard-count
diff --git a/mlir/test/mlir-tblgen/op-partitions.td b/mlir/test/mlir-tblgen/op-partitions.td
new file mode 100644
index 00000000000000..5b660687ea5dc2
--- /dev/null
+++ b/mlir/test/mlir-tblgen/op-partitions.td
@@ -0,0 +1,110 @@
+// RUN: mlir-tblgen -gen-op-decls -op-partition=memory -I %S/../../include %s | FileCheck %s --check-prefix=DECLS
+// RUN: mlir-tblgen -gen-op-defs -op-partition=memory -I %S/../../include %s | FileCheck %s --check-prefix=DEFS
+// RUN: mlir-tblgen -gen-op-defs -op-partition=empty -I %S/../../include %s | FileCheck %s --check-prefix=EMPTY
+// RUN: mlir-tblgen -gen-op-defs -I %S/../../include %s | FileCheck %s --check-prefix=ALL
+// RUN: mlir-tblgen -gen-dialect-decls -I %S/../../include %s | FileCheck %s --check-prefix=DIALECT-DECLS
+
+include "mlir/IR/OpBase.td"
+
+def Test_ArithPartition : Partition<"Arith">;
+def Test_MemoryPartition : Partition<"memory">;
+def Test_EmptyPartition : Partition<"empty">;
+
+def Test_Dialect : Dialect {
+ let name = "test";
+ let cppNamespace = "test";
+ let opPartitions = [Test_ArithPartition, Test_MemoryPartition,
+ Test_EmptyPartition];
+ let defaultOpPartition = Test_ArithPartition;
+}
+
+class Test_Op<string mnemonic, list<Trait> traits = []>
+ : Op<Test_Dialect, mnemonic, traits>;
+
+// These ops are in the default partition, Arith.
+def AddOp : Test_Op<"add">, Arguments<(ins I32:$lhs, I32:$rhs)>;
+def MulOp : Test_Op<"mul">, Arguments<(ins I32:$lhs, I32:$rhs)>;
+
+let partition = Test_MemoryPartition in {
+def LoadOp : Test_Op<"load">, Arguments<(ins I64:$addr)>;
+def StoreOp : Test_Op<"store">, Arguments<(ins I32:$value, I64:$addr)>;
+def FenceOp : Test_Op<"fence">;
+}
+
+// Only the ops of the requested partition are declared. The ops of the other
+// partitions are forward declared, so that declarations can refer to them.
+// DECLS: namespace test {
+// DECLS-EMPTY:
+// DECLS-NEXT: class AddOp;
+// DECLS: class MulOp;
+// DECLS: class FenceOp;
+// DECLS: class LoadOp;
+// DECLS: class StoreOp;
+// DECLS-LABEL: #ifdef GET_OP_CLASSES
+// DECLS-NOT: AddOp
+// DECLS-NOT: MulOp
+// DECLS: class LoadOp : public
+// DECLS-NOT: AddOp
+// DECLS-NOT: MulOp
+// DECLS-NOT: register
+
+// The registration hooks replace the op list.
+// DEFS-NOT: GET_OP_LIST
+// DEFS-LABEL: #ifdef GET_OP_CLASSES
+// DEFS: static ::llvm::LogicalResult __mlir_ods_local_type_constraint_6memory_op2Dpartitions{{[0-9]+}}(
+// DEFS-NOT: AddOp
+// DEFS: LoadOpAdaptor
+// DEFS: StoreOpAdaptor
+// DEFS-NOT: AddOp
+// DEFS: void test::TestDialect::registerOperationsMemory() {
+// DEFS-NEXT: ::mlir::RegisteredOperationName::insert<test::FenceOp>(*this);
+// DEFS-NEXT: ::mlir::RegisteredOperationName::insert<test::LoadOp>(*this);
+// DEFS-NEXT: ::mlir::RegisteredOperationName::insert<test::StoreOp>(*this);
+// DEFS-NEXT: }
+// DEFS-EMPTY:
+// DEFS-NEXT: #endif // GET_OP_CLASSES
+
+// A partition without ops still gets its hook, which the dialect calls.
+// EMPTY-LABEL: #ifdef GET_OP_CLASSES
+// EMPTY: void test::TestDialect::registerOperationsEmpty() {
+// EMPTY-NEXT: }
+// EMPTY-EMPTY:
+// EMPTY-NEXT: #endif // GET_OP_CLASSES
+
+// Without a requested partition, all ops are generated with both the op list
+// and the hooks of all partitions, so that a dialect can adopt partitions
+// before it generates its ops by partition. Ops without a partition are in the
+// default partition.
+// ALL-LABEL: #ifdef GET_OP_LIST
+// ALL: test::AddOp,
+// ALL: test::FenceOp,
+// ALL: test::LoadOp,
+// ALL: test::MulOp,
+// ALL: test::StoreOp
+// ALL-LABEL: #ifdef GET_OP_CLASSES
+// ALL: static ::llvm::LogicalResult __mlir_ods_local_type_constraint_op2Dpartitions{{[0-9]+}}(
+// ALL: void test::TestDialect::registerOperationsArith() {
+// ALL-NEXT: ::mlir::RegisteredOperationName::insert<test::AddOp>(*this);
+// ALL-NEXT: ::mlir::RegisteredOperationName::insert<test::MulOp>(*this);
+// ALL-NEXT: }
+// ALL: void test::TestDialect::registerOperationsMemory() {
+// ALL: void test::TestDialect::registerOperationsEmpty() {
+
+// The hooks are private members of the dialect class; the dialect's own
+// declarations that follow stay public.
+// DIALECT-DECLS: class TestDialect
+// DIALECT-DECLS: private:
+// DIALECT-DECLS-NEXT: /// Registers the operations of all op partitions of the dialect.
+// DIALECT-DECLS-NEXT: void registerOperations() {
+// DIALECT-DECLS-NEXT: registerOperationsArith();
+// DIALECT-DECLS-NEXT: registerOperationsMemory();
+// DIALECT-DECLS-NEXT: registerOperationsEmpty();
+// DIALECT-DECLS-NEXT: }
+// DIALECT-DECLS-NEXT: /// Registers the operations in op partition `Arith`.
+// DIALECT-DECLS-NEXT: void registerOperationsArith();
+// DIALECT-DECLS-NEXT: /// Registers the operations in op partition `memory`.
+// DIALECT-DECLS-NEXT: void registerOperationsMemory();
+// DIALECT-DECLS-NEXT: /// Registers the operations in op partition `empty`.
+// DIALECT-DECLS-NEXT: void registerOperationsEmpty();
+// DIALECT-DECLS-NEXT: public:
+// DIALECT-DECLS-NEXT: };
diff --git a/mlir/tools/mlir-tblgen/DialectGen.cpp b/mlir/tools/mlir-tblgen/DialectGen.cpp
index d6977b1f225794..101d5f81d5d106 100644
--- a/mlir/tools/mlir-tblgen/DialectGen.cpp
+++ b/mlir/tools/mlir-tblgen/DialectGen.cpp
@@ -12,6 +12,7 @@
#include "CppGenUtilities.h"
#include "DialectGenUtilities.h"
+#include "OpGenHelpers.h"
#include "mlir/TableGen/Class.h"
#include "mlir/TableGen/CodeGenHelpers.h"
#include "mlir/TableGen/Format.h"
@@ -216,8 +217,29 @@ static const char *const discardableAttrHelperDecl = R"(
public:
)";
+/// Emit the op registration hooks of a dialect with op partitions as private
+/// members of the dialect class: one per partition, defined by `-gen-op-defs
+/// -op-partition=...`, and one that registers the ops of all partitions.
+static void emitOpPartitionRegistrationHooks(const Dialect &dialect,
+ raw_ostream &os) {
+ os << "\nprivate:\n"
+ << " /// Registers the operations of all op partitions of the dialect.\n"
+ << llvm::formatv(" void {0}() {{\n", getOpRegistrationHookName());
+ for (StringRef partition : dialect.getOpPartitions())
+ os << llvm::formatv(" {0}();\n", getOpRegistrationHookName(partition));
+ os << " }\n";
+ for (StringRef partition : dialect.getOpPartitions())
+ os << llvm::formatv(
+ " /// Registers the operations in op partition `{0}`.\n"
+ " void {1}();\n",
+ partition, getOpRegistrationHookName(partition));
+ os << "public:\n";
+}
+
/// Generate the declaration for the given dialect class.
static void emitDialectDecl(Dialect &dialect, raw_ostream &os) {
+ verifyDialectOpPartitions(dialect);
+
// Emit all nested namespaces.
{
DialectNamespaceEmitter nsEmitter(os, dialect);
@@ -272,6 +294,9 @@ static void emitDialectDecl(Dialect &dialect, raw_ostream &os) {
dialect.getName());
}
+ if (!dialect.getOpPartitions().empty())
+ emitOpPartitionRegistrationHooks(dialect, os);
+
if (std::optional<StringRef> extraDecl = dialect.getExtraClassDeclaration())
os << *extraDecl;
diff --git a/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp b/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp
index 5eda6ce65c82f2..472c163dff5836 100644
--- a/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp
+++ b/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp
@@ -4742,11 +4742,28 @@ static void emitOpClassDefs(const RecordKeeper &records,
os << definitions;
}
+/// When generating the ops of an op partition, forward declare the ops of
+/// all other partitions, so that the declarations can refer to ops of any
+/// partition of their dialect, like they could without partitions.
+static void emitOtherOpPartitionForwardDecls(const RecordKeeper &records,
+ raw_ostream &os) {
+ for (const Record *def : records.getAllDerivedDefinitions("Op")) {
+ Operator op(def);
+ StringRef partition = op.getPartitionName();
+ if (partition.empty() || isRequestedOpPartition(partition))
+ continue;
+
+ NamespaceEmitter emitter(os, op.getCppNamespace());
+ os << "class " << op.getCppClassName() << ";\n";
+ }
+}
+
/// Emit op declarations for all op records.
static bool emitOpDecls(const RecordKeeper &records, raw_ostream &os) {
emitSourceFileHeader("Op Declarations", os, records);
std::vector<const Record *> defs = getRequestedOpDefinitions(records);
+ emitOtherOpPartitionForwardDecls(records, os);
emitOpClassDecls(records, defs, os);
// If we are generating sharded op definitions, emit the sharded op
@@ -4810,6 +4827,50 @@ static void emitOpDefShard(const RecordKeeper &records,
emitOpClassDefs(records, defs, os, indexStr);
}
+/// Emit the registration hooks of the op partitions of `dialects` that are
+/// requested by "op-partition", or of all their partitions if none is. A hook
+/// is emitted even if its partition has no ops, because the dialect calls the
+/// hooks of all its partitions.
+static void emitOpPartitionRegistrationHooks(ArrayRef<Dialect> dialects,
+ ArrayRef<const Record *> defs,
+ raw_ostream &os) {
+ std::vector<Operator> ops(defs.begin(), defs.end());
+ for (const Dialect &dialect : dialects) {
+ for (StringRef partition : dialect.getOpPartitions()) {
+ if (!isRequestedOpPartition(partition))
+ continue;
+
+ os << formatv(opCommentHeader, dialect.getCppClassName(),
+ (partition + " Op Registration Hook").str())
+ << formatv("void {0}::{1}::{2}() {{\n", dialect.getCppNamespace(),
+ dialect.getCppClassName(),
+ getOpRegistrationHookName(partition));
+ // The hooks register each op with a separate call instead of
+ // `addOperations<...>()`, whose single instantiation over all ops of a
+ // partition takes noticeably longer to compile.
+ for (const Operator &op : ops)
+ if (op.getDialect() == dialect && op.getPartitionName() == partition)
+ os << formatv(
+ " ::mlir::RegisteredOperationName::insert<{0}>(*this);\n",
+ op.getQualCppClassName());
+ os << "}\n";
+ }
+ }
+}
+
+/// Returns the tag that makes the static verifier functions generated for the
+/// requested op partition unique, so that the definitions of several
+/// partitions can be included in one compilation unit. The partition name is
+/// prefixed with its length and the tag ends with `_`, so that the partition
+/// name and the file name that follows the tag cannot run together.
+static std::string getOpPartitionConstraintTag() {
+ StringRef partition = getRequestedOpPartition();
+ if (partition.empty())
+ return "";
+
+ return std::to_string(partition.size()) + partition.str() + "_";
+}
+
/// Emit op definitions for all op records.
static bool emitOpDefs(const RecordKeeper &records, raw_ostream &os) {
emitSourceFileHeader("Op Definitions", os, records);
@@ -4818,9 +4879,11 @@ static bool emitOpDefs(const RecordKeeper &records, raw_ostream &os) {
SmallVector<ArrayRef<const Record *>, 4> shardedDefs;
shardOpDefinitions(defs, shardedDefs);
- // If no shard was requested, emit the regular op list and class definitions.
+ // If no shard was requested, emit the class definitions and the registration
+ // hooks of the op partitions. Without a requested op partition, also emit the
+ // op list, for dialects that register their ops without hooks.
if (shardedDefs.size() == 1) {
- {
+ if (getRequestedOpPartition().empty()) {
IfDefEmitter scope(os, "GET_OP_LIST");
interleave(
defs, os,
@@ -4829,7 +4892,8 @@ static bool emitOpDefs(const RecordKeeper &records, raw_ostream &os) {
}
{
IfDefEmitter scope(os, "GET_OP_CLASSES");
- emitOpClassDefs(records, defs, os);
+ emitOpClassDefs(records, defs, os, getOpPartitionConstraintTag());
+ emitOpPartitionRegistrationHooks(getOpDialects(records), defs, os);
}
return false;
}
diff --git a/mlir/tools/mlir-tblgen/OpGenHelpers.cpp b/mlir/tools/mlir-tblgen/OpGenHelpers.cpp
index afc8272bc2a630..4f30ee333e1d61 100644
--- a/mlir/tools/mlir-tblgen/OpGenHelpers.cpp
+++ b/mlir/tools/mlir-tblgen/OpGenHelpers.cpp
@@ -11,6 +11,8 @@
//===----------------------------------------------------------------------===//
#include "OpGenHelpers.h"
+#include "llvm/ADT/SetVector.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringSet.h"
#include "llvm/Support/CommandLine.h"
#include "llvm/Support/FormatVariadic.h"
@@ -35,6 +37,11 @@ static cl::opt<unsigned> opShardCount(
"op-shard-count",
cl::desc("The number of shards into which the op classes will be divided"),
cl::cat(opDefGenCat), cl::init(1));
+static cl::opt<std::string> opPartition(
+ "op-partition",
+ cl::desc("Only consider the ops in this op partition (see `Op::partition`) "
+ "and, for -gen-op-defs, emit its registration hook"),
+ cl::cat(opDefGenCat));
static std::string getOperationName(const Record &def) {
auto prefix = def.getValueAsDef("opDialect")->getValueAsString("name");
@@ -44,12 +51,54 @@ static std::string getOperationName(const Record &def) {
return std::string(formatv("{0}.{1}", prefix, opName));
}
+/// Like `Operator::getPartitionName`, for filtering without constructing an
+/// `Operator`.
+static StringRef getOpPartitionName(const Record &def) {
+ const Record *partition = def.getValueAsOptionalDef("partition");
+ return partition ? partition->getValueAsString("partitionName") : "";
+}
+
+std::string mlir::tblgen::getOpRegistrationHookName(StringRef partition) {
+ return "registerOperations" +
+ convertToCamelFromSnakeCase(partition, /*capitalizeFirst=*/true);
+}
+
+/// Verifies that the partition requested by "op-partition" exists and that
+/// "op-shard-count" is not used with partitions.
+static void verifyOpPartitions(ArrayRef<Dialect> dialects) {
+ for (const Dialect &dialect : dialects)
+ verifyDialectOpPartitions(dialect);
+
+ // The requested partition must belong to a dialect with ops.
+ if (!opPartition.empty() &&
+ llvm::none_of(dialects, [&](const Dialect &dialect) {
+ return llvm::is_contained(dialect.getOpPartitions(), opPartition);
+ }))
+ PrintFatalError("requested op partition '" + opPartition +
+ "' is not declared in the `opPartitions` of any dialect "
+ "with ops");
+
+ // Index sharding does not emit the registration hooks of partitions, which
+ // `registerOperations()` of a dialect with partitions calls. This also
+ // rejects -op-partition, as the requested partition belongs to a dialect.
+ if (opShardCount > 1) {
+ for (const Dialect &dialect : dialects)
+ if (!dialect.getOpPartitions().empty())
+ PrintFatalError(dialect.getDef(),
+ "dialect '" + dialect.getName() +
+ "' has `opPartitions`, which cannot be combined "
+ "with -op-shard-count");
+ }
+}
+
std::vector<const Record *>
mlir::tblgen::getRequestedOpDefinitions(const RecordKeeper &records) {
const Record *classDef = records.getClass("Op");
if (!classDef)
PrintFatalError("ERROR: Couldn't find the 'Op' class!\n");
+ verifyOpPartitions(getOpDialects(records));
+
Regex includeRegex(opIncFilter), excludeRegex(opExcFilter);
std::vector<const Record *> defs;
for (const Record *def : records.getDefs()) {
@@ -61,6 +110,9 @@ mlir::tblgen::getRequestedOpDefinitions(const RecordKeeper &records) {
// Unless there is an exclude filter and it matches.
if (!opExcFilter.empty() && excludeRegex.match(getOperationName(*def)))
continue;
+ // Only include ops of the requested partition, if any.
+ if (!isRequestedOpPartition(getOpPartitionName(*def)))
+ continue;
defs.push_back(def);
}
@@ -101,3 +153,30 @@ void mlir::tblgen::shardOpDefinitions(
start += size;
}
}
+
+StringRef mlir::tblgen::getRequestedOpPartition() { return opPartition; }
+
+bool mlir::tblgen::isRequestedOpPartition(StringRef partition) {
+ return opPartition.empty() || partition == opPartition;
+}
+
+void mlir::tblgen::verifyDialectOpPartitions(const Dialect &dialect) {
+ StringMap<StringRef> hookNames;
+ for (StringRef partition : dialect.getOpPartitions()) {
+ auto [it, inserted] =
+ hookNames.try_emplace(getOpRegistrationHookName(partition), partition);
+ if (!inserted)
+ PrintFatalError(dialect.getDef(),
+ "op partitions '" + it->second + "' and '" + partition +
+ "' of dialect '" + dialect.getName() +
+ "' have the same registration hook '" + it->first() +
+ "'");
+ }
+}
+
+std::vector<Dialect> mlir::tblgen::getOpDialects(const RecordKeeper &records) {
+ SetVector<const Record *> dialectDefs;
+ for (const Record *def : records.getAllDerivedDefinitions("Op"))
+ dialectDefs.insert(def->getValueAsDef("opDialect"));
+ return std::vector<Dialect>(dialectDefs.begin(), dialectDefs.end());
+}
diff --git a/mlir/tools/mlir-tblgen/OpGenHelpers.h b/mlir/tools/mlir-tblgen/OpGenHelpers.h
index 3cd171d10c08ab..4ffe046cbc09de 100644
--- a/mlir/tools/mlir-tblgen/OpGenHelpers.h
+++ b/mlir/tools/mlir-tblgen/OpGenHelpers.h
@@ -14,17 +14,25 @@
#define MLIR_TOOLS_MLIRTBLGEN_OPGENHELPERS_H_
#include "mlir/Support/LLVM.h"
+#include "mlir/TableGen/Dialect.h"
#include "llvm/TableGen/Record.h"
+#include <string>
#include <vector>
namespace mlir {
namespace tblgen {
-
/// Returns all the op definitions filtered by the user. The filtering is via
-/// command-line option "op-include-regex" and "op-exclude-regex".
+/// command-line options "op-include-regex", "op-exclude-regex" and
+/// "op-partition". Verifies "op-partition" and the op partitions of the
+/// dialects with ops.
std::vector<const llvm::Record *>
getRequestedOpDefinitions(const llvm::RecordKeeper &records);
+/// Returns the name of the private member function of a dialect class that
+/// registers the ops of the dialect in op partition `partition`, or all its ops
+/// if `partition` is empty.
+std::string getOpRegistrationHookName(StringRef partition = "");
+
/// Checks whether `str` is a Python keyword or would shadow builtin function.
/// Regenerate using python -c"print(set(sorted(__import__('keyword').kwlist)))"
bool isPythonReserved(llvm::StringRef str);
@@ -34,6 +42,20 @@ void shardOpDefinitions(
ArrayRef<const llvm::Record *> defs,
SmallVectorImpl<ArrayRef<const llvm::Record *>> &shardedDefs);
+/// Returns the op partition set by "op-partition", or an empty string if all
+/// ops are requested.
+StringRef getRequestedOpPartition();
+
+/// Returns whether "op-partition" requests `partition`.
+/// Returns true if "op-partition" is not set.
+bool isRequestedOpPartition(StringRef partition);
+
+/// Verifies that the op partitions of `dialect` have unique registration hooks.
+void verifyDialectOpPartitions(const Dialect &dialect);
+
+/// Returns the dialects that have ops in `records`, in order of first
+/// appearance.
+std::vector<Dialect> getOpDialects(const llvm::RecordKeeper &records);
} // namespace tblgen
} // namespace mlir
diff --git a/mlir/unittests/IR/AdaptorTest.cpp b/mlir/unittests/IR/AdaptorTest.cpp
index 39e7eec5a956d6..f19264b985100a 100644
--- a/mlir/unittests/IR/AdaptorTest.cpp
+++ b/mlir/unittests/IR/AdaptorTest.cpp
@@ -6,8 +6,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsCore.h"
#include "TestOpsSyntax.h"
#include "gmock/gmock.h"
#include "gtest/gtest.h"
diff --git a/mlir/unittests/IR/IRMapping.cpp b/mlir/unittests/IR/IRMapping.cpp
index 62814f7ba39266..cd7797069d7553 100644
--- a/mlir/unittests/IR/IRMapping.cpp
+++ b/mlir/unittests/IR/IRMapping.cpp
@@ -10,8 +10,7 @@
#include "mlir/IR/Builders.h"
#include "gtest/gtest.h"
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsPatterns.h"
using namespace mlir;
diff --git a/mlir/unittests/IR/InterfaceAttachmentTest.cpp b/mlir/unittests/IR/InterfaceAttachmentTest.cpp
index 02cc9312e9f493..a107c40774a578 100644
--- a/mlir/unittests/IR/InterfaceAttachmentTest.cpp
+++ b/mlir/unittests/IR/InterfaceAttachmentTest.cpp
@@ -18,8 +18,7 @@
#include "gtest/gtest.h"
#include "TestAttributes.h"
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsPatterns.h"
#include "TestTypes.h"
#include "mlir/IR/OwningOpRef.h"
diff --git a/mlir/unittests/IR/InterfaceTest.cpp b/mlir/unittests/IR/InterfaceTest.cpp
index cd4e76a63e257d..d7554e7bceaa86 100644
--- a/mlir/unittests/IR/InterfaceTest.cpp
+++ b/mlir/unittests/IR/InterfaceTest.cpp
@@ -14,8 +14,7 @@
#include "gtest/gtest.h"
#include "TestAttributes.h"
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsInterfaces.h"
#include "TestTypes.h"
using namespace mlir;
diff --git a/mlir/unittests/IR/OperationSupportTest.cpp b/mlir/unittests/IR/OperationSupportTest.cpp
index fdeeb3d2cb3cc1..d44f17870ffc01 100644
--- a/mlir/unittests/IR/OperationSupportTest.cpp
+++ b/mlir/unittests/IR/OperationSupportTest.cpp
@@ -7,8 +7,7 @@
//===----------------------------------------------------------------------===//
#include "mlir/IR/OperationSupport.h"
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsPatterns.h"
#include "mlir/IR/Builders.h"
#include "mlir/IR/BuiltinTypes.h"
#include "llvm/ADT/BitVector.h"
diff --git a/mlir/unittests/IR/PatternMatchTest.cpp b/mlir/unittests/IR/PatternMatchTest.cpp
index f4eb3b027a2228..92b4623431f640 100644
--- a/mlir/unittests/IR/PatternMatchTest.cpp
+++ b/mlir/unittests/IR/PatternMatchTest.cpp
@@ -12,8 +12,7 @@
#include <string>
#include <vector>
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsPatterns.h"
using namespace mlir;
diff --git a/mlir/unittests/IR/ValueTest.cpp b/mlir/unittests/IR/ValueTest.cpp
index 8f794d172844cd..dc40a1b505d29c 100644
--- a/mlir/unittests/IR/ValueTest.cpp
+++ b/mlir/unittests/IR/ValueTest.cpp
@@ -7,8 +7,6 @@
//===----------------------------------------------------------------------===//
#include "mlir/IR/Value.h"
-#include "TestDialect.h"
-#include "TestOps.h"
#include "mlir/IR/Builders.h"
#include "mlir/IR/BuiltinTypes.h"
#include "mlir/IR/OperationSupport.h"
diff --git a/mlir/unittests/Interfaces/MemorySlotUtilsTest.cpp b/mlir/unittests/Interfaces/MemorySlotUtilsTest.cpp
index ffc1709201f26d..baad58d29f8cb2 100644
--- a/mlir/unittests/Interfaces/MemorySlotUtilsTest.cpp
+++ b/mlir/unittests/Interfaces/MemorySlotUtilsTest.cpp
@@ -8,7 +8,6 @@
#include "mlir/Interfaces/Utils/MemorySlotUtils.h"
#include "TestDialect.h"
-#include "TestOps.h"
#include "mlir/Parser/Parser.h"
#include "gtest/gtest.h"
diff --git a/mlir/unittests/TableGen/OpBuildGen.cpp b/mlir/unittests/TableGen/OpBuildGen.cpp
index e0e1ac99884bad..b3e620f3ce65ea 100644
--- a/mlir/unittests/TableGen/OpBuildGen.cpp
+++ b/mlir/unittests/TableGen/OpBuildGen.cpp
@@ -10,8 +10,7 @@
//
//===----------------------------------------------------------------------===//
-#include "TestDialect.h"
-#include "TestOps.h"
+#include "TestOpsCore.h"
#include "mlir/IR/Attributes.h"
#include "mlir/IR/Builders.h"
#include "mlir/IR/BuiltinTypes.h"
>From 83d1c6450e0113db95b40327987f59c3a5861166 Mon Sep 17 00:00:00 2001
From: Henrich Lauko <hlauko at nvidia.com>
Date: Mon, 9 Nov 2026 16:19:03 +0000
Subject: [PATCH 5/5] [CIR] Shard the dialect naturally along op partitions
Replace the count-based `-op-shard-count=8` sharding of CIROps.td with the
op partitions introduced by the ODS natural sharding (llvm/llvm-project
PR 239254). Each of the eleven partitions (Core, Symbols, ControlFlow,
Arith, Builtins, Math, Coroutine, EH, VectorMatrix, Complex, ClassMembers)
is generated into its own CIROps<Partition>.{h,cpp}.inc, declared by its own
CIROps<Partition>.h on top of the new CIRDialectBase.h (the dialect,
attributes, types and interfaces without any op), and compiled in its own
CIROps<Partition>.cpp together with the hand-written op code that used to
live in CIRDialect.cpp. CIRDialect.h becomes an umbrella over the
partition headers, so existing includers are unchanged; files that use no
op switch to CIRDialectBase.h.
Assisted-by: Claude Code (Fable 5.1).
---
clang/CMakeLists.txt | 4 -
.../include/clang/CIR/Dialect/IR/CIRDialect.h | 121 +-
.../clang/CIR/Dialect/IR/CIRDialect.td | 25 +
.../clang/CIR/Dialect/IR/CIRDialectBase.h | 107 +
clang/include/clang/CIR/Dialect/IR/CIROps.td | 110 +
.../clang/CIR/Dialect/IR/CIROpsArith.h | 24 +
.../clang/CIR/Dialect/IR/CIROpsBuiltins.h | 25 +
.../clang/CIR/Dialect/IR/CIROpsClassMembers.h | 25 +
.../clang/CIR/Dialect/IR/CIROpsComplex.h | 23 +
.../clang/CIR/Dialect/IR/CIROpsControlFlow.h | 24 +
.../include/clang/CIR/Dialect/IR/CIROpsCore.h | 33 +
.../clang/CIR/Dialect/IR/CIROpsCoroutine.h | 23 +
clang/include/clang/CIR/Dialect/IR/CIROpsEH.h | 23 +
.../include/clang/CIR/Dialect/IR/CIROpsMath.h | 23 +
.../clang/CIR/Dialect/IR/CIROpsSymbols.h | 24 +
.../clang/CIR/Dialect/IR/CIROpsVectorMatrix.h | 23 +
.../include/clang/CIR/Dialect/IR/CIRStdOps.td | 1 +
.../clang/CIR/Dialect/IR/CMakeLists.txt | 11 +-
clang/lib/CIR/CodeGen/CIRGenAMDGPU.cpp | 2 +-
clang/lib/CIR/Dialect/IR/CIRAttrs.cpp | 33 +-
clang/lib/CIR/Dialect/IR/CIRDialect.cpp | 5562 +----------------
.../lib/CIR/Dialect/IR/CIRDialectBytecode.cpp | 2 +-
clang/lib/CIR/Dialect/IR/CIROpDefinitions.h | 30 +-
clang/lib/CIR/Dialect/IR/CIROpsArith.cpp | 315 +
clang/lib/CIR/Dialect/IR/CIROpsBuiltins.cpp | 483 ++
.../lib/CIR/Dialect/IR/CIROpsClassMembers.cpp | 296 +
clang/lib/CIR/Dialect/IR/CIROpsComplex.cpp | 147 +
.../lib/CIR/Dialect/IR/CIROpsControlFlow.cpp | 905 +++
clang/lib/CIR/Dialect/IR/CIROpsCore.cpp | 660 ++
clang/lib/CIR/Dialect/IR/CIROpsCoroutine.cpp | 340 +
clang/lib/CIR/Dialect/IR/CIROpsEH.cpp | 706 +++
.../{CIROpDefinitions.cpp => CIROpsMath.cpp} | 16 +-
clang/lib/CIR/Dialect/IR/CIROpsSymbols.cpp | 1536 +++++
.../lib/CIR/Dialect/IR/CIROpsVectorMatrix.cpp | 426 ++
clang/lib/CIR/Dialect/IR/CIRTypes.cpp | 3 +-
clang/lib/CIR/Dialect/IR/CMakeLists.txt | 30 +-
.../OpenACC/RegisterOpenACCExtensions.cpp | 2 +-
clang/lib/CIR/Dialect/Transforms/LibOpt.cpp | 2 +-
clang/lib/CIR/FrontendAction/CIRGenAction.cpp | 2 +-
.../lib/CIR/Interfaces/CIRLoopOpInterface.cpp | 3 +-
clang/lib/CIR/Lowering/CIRPasses.cpp | 2 +-
clang/lib/CIR/RegisterAllDialects.cpp | 2 +-
clang/tools/cir-translate/cir-translate.cpp | 2 +-
.../CIR/GetFloatingPointTypeTest.cpp | 2 +-
clang/unittests/CIR/IntTypeABIAlignTest.cpp | 2 +-
clang/unittests/CIR/RecordMemberKindTest.cpp | 2 +-
.../unittests/CIR/RecordTypeMetadataTest.cpp | 2 +-
clang/unittests/CIR/RecordTypeVerifyTest.cpp | 2 +-
clang/unittests/CIR/UnionTypeSizeTest.cpp | 2 +-
.../unittests/CIR/VectorTypeABIAlignTest.cpp | 2 +-
50 files changed, 6457 insertions(+), 5713 deletions(-)
create mode 100644 clang/include/clang/CIR/Dialect/IR/CIRDialectBase.h
create mode 100644 clang/include/clang/CIR/Dialect/IR/CIROpsArith.h
create mode 100644 clang/include/clang/CIR/Dialect/IR/CIROpsBuiltins.h
create mode 100644 clang/include/clang/CIR/Dialect/IR/CIROpsClassMembers.h
create mode 100644 clang/include/clang/CIR/Dialect/IR/CIROpsComplex.h
create mode 100644 clang/include/clang/CIR/Dialect/IR/CIROpsControlFlow.h
create mode 100644 clang/include/clang/CIR/Dialect/IR/CIROpsCore.h
create mode 100644 clang/include/clang/CIR/Dialect/IR/CIROpsCoroutine.h
create mode 100644 clang/include/clang/CIR/Dialect/IR/CIROpsEH.h
create mode 100644 clang/include/clang/CIR/Dialect/IR/CIROpsMath.h
create mode 100644 clang/include/clang/CIR/Dialect/IR/CIROpsSymbols.h
create mode 100644 clang/include/clang/CIR/Dialect/IR/CIROpsVectorMatrix.h
create mode 100644 clang/lib/CIR/Dialect/IR/CIROpsArith.cpp
create mode 100644 clang/lib/CIR/Dialect/IR/CIROpsBuiltins.cpp
create mode 100644 clang/lib/CIR/Dialect/IR/CIROpsClassMembers.cpp
create mode 100644 clang/lib/CIR/Dialect/IR/CIROpsComplex.cpp
create mode 100644 clang/lib/CIR/Dialect/IR/CIROpsControlFlow.cpp
create mode 100644 clang/lib/CIR/Dialect/IR/CIROpsCore.cpp
create mode 100644 clang/lib/CIR/Dialect/IR/CIROpsCoroutine.cpp
create mode 100644 clang/lib/CIR/Dialect/IR/CIROpsEH.cpp
rename clang/lib/CIR/Dialect/IR/{CIROpDefinitions.cpp => CIROpsMath.cpp} (51%)
create mode 100644 clang/lib/CIR/Dialect/IR/CIROpsSymbols.cpp
create mode 100644 clang/lib/CIR/Dialect/IR/CIROpsVectorMatrix.cpp
diff --git a/clang/CMakeLists.txt b/clang/CMakeLists.txt
index fbb4cfaf090059..580ec8ddcc8a74 100644
--- a/clang/CMakeLists.txt
+++ b/clang/CMakeLists.txt
@@ -205,10 +205,6 @@ if(CLANG_ENABLE_CIR)
message(FATAL_ERROR
"Cannot build ClangIR without MLIR enabled")
endif()
- # Number of translation units the TableGen-generated CIR operation
- # definitions are split into. Shared by the CIROps.td TableGen invocations
- # and the MLIRCIR library sources.
- set(CLANG_CIR_OP_SHARD_COUNT 8)
endif()
include(CheckIncludeFile)
diff --git a/clang/include/clang/CIR/Dialect/IR/CIRDialect.h b/clang/include/clang/CIR/Dialect/IR/CIRDialect.h
index d9018e93628b69..0985b7a8767ad9 100644
--- a/clang/include/clang/CIR/Dialect/IR/CIRDialect.h
+++ b/clang/include/clang/CIR/Dialect/IR/CIRDialect.h
@@ -6,115 +6,28 @@
//
//===----------------------------------------------------------------------===//
//
-// This file declares the CIR dialect.
+// This file declares the CIR dialect with all of its ops. The ops are split
+// into partitions (see CIRDialect.td), each with its own CIROps<Partition>.h
+// that declares the ops of that partition and forward declares the others;
+// code that uses only a few ops can include just the partition headers it
+// needs, together with CIRDialectBase.h for the dialect, attributes and types.
//
//===----------------------------------------------------------------------===//
#ifndef CLANG_CIR_DIALECT_IR_CIRDIALECT_H
#define CLANG_CIR_DIALECT_IR_CIRDIALECT_H
-#include "mlir/IR/Builders.h"
-#include "mlir/IR/BuiltinOps.h"
-#include "mlir/IR/BuiltinTypes.h"
-#include "mlir/IR/Dialect.h"
-#include "mlir/IR/OpDefinition.h"
-#include "mlir/Interfaces/CallInterfaces.h"
-#include "mlir/Interfaces/ControlFlowInterfaces.h"
-#include "mlir/Interfaces/FunctionInterfaces.h"
-#include "mlir/Interfaces/InferTypeOpInterface.h"
-#include "mlir/Interfaces/LoopLikeInterface.h"
-#include "mlir/Interfaces/MemorySlotInterfaces.h"
-#include "mlir/Interfaces/SideEffectInterfaces.h"
-
-#include "clang/CIR/Dialect/IR/CIRAttrs.h"
-#include "clang/CIR/Dialect/IR/CIROpsDialect.h.inc"
-#include "clang/CIR/Dialect/IR/CIROpsEnums.h"
-#include "clang/CIR/Dialect/IR/CIRTypes.h"
-#include "clang/CIR/Interfaces/CIRLoopOpInterface.h"
-#include "clang/CIR/Interfaces/CIROpInterfaces.h"
-#include "clang/CIR/MissingFeatures.h"
-
-using BuilderCallbackRef =
- llvm::function_ref<void(mlir::OpBuilder &, mlir::Location)>;
-using BuilderOpStateCallbackRef = llvm::function_ref<void(
- mlir::OpBuilder &, mlir::Location, mlir::OperationState &)>;
-
-namespace cir {
-void buildTerminatedBody(mlir::OpBuilder &builder, mlir::Location loc);
-
-/// The process floating-point environment, including its rounding mode and
-/// exception state.
-struct FloatingPointEnvironmentResource
- : public mlir::SideEffects::Resource::Base<
- FloatingPointEnvironmentResource> {
- mlir::StringRef getName() const final { return "FloatingPointEnvironment"; }
- bool isAddressable() const final { return false; }
-};
-
-template <typename ConcreteType>
-class FenvOpTrait : public mlir::OpTrait::TraitBase<ConcreteType, FenvOpTrait> {
-public:
- mlir::Speculation::Speculatability getSpeculatability() {
- // Masked exceptions cannot trap. When strict_except is false, exception
- // side effects are non-deterministic, so speculation is still safe.
- FPEnvConstrainedOpInterface fenvOp = getFenvOp();
- if (fenvOp.getFenvExceptionMode() == cir::FPExceptionMode::Masked &&
- !fenvOp.getFenvStrictExcept())
- return mlir::Speculation::Speculatable;
-
- return mlir::Speculation::NotSpeculatable;
- }
-
- void getEffects(
- llvm::SmallVectorImpl<mlir::MemoryEffects::EffectInstance> &effects) {
- if (!getFenvOp().getFenvAttr())
- return;
- effects.emplace_back(mlir::MemoryEffects::Read::get(),
- FloatingPointEnvironmentResource::get());
- effects.emplace_back(mlir::MemoryEffects::Write::get(),
- FloatingPointEnvironmentResource::get());
- }
-
-private:
- FPEnvConstrainedOpInterface getFenvOp() {
- return mlir::cast<FPEnvConstrainedOpInterface>(this->getOperation());
- }
-};
-
-/// The properties of \p Op with every ODS-declared default applied. The
-/// aggregate op builders that take a `Properties` struct do not apply those
-/// defaults themselves, so callers of those builders have to do it here.
-template <typename Op>
-typename Op::Properties getDefaultProperties(mlir::MLIRContext *context) {
- typename Op::Properties properties{};
- Op::populateDefaultProperties(
- mlir::OperationName(Op::getOperationName(), context), properties);
- return properties;
-}
-
-/// Look up the RecordLayoutAttr for a named record in the module's
-/// cir.record_layouts dictionary. Asserts if the entry is missing.
-RecordLayoutAttr getRecordLayout(mlir::ModuleOp mod, mlir::StringAttr name);
-
-/// Same lookup as getRecordLayout, but returns a null attribute instead of
-/// asserting when the record has no layout entry.
-RecordLayoutAttr tryGetRecordLayout(mlir::ModuleOp mod, mlir::StringAttr name);
-} // namespace cir
-
-// TableGen'erated files for MLIR dialects require that a macro be defined when
-// they are included. GET_OP_CLASSES tells the file to define the classes for
-// the operations of that dialect.
-#define GET_OP_CLASSES
-#include "clang/CIR/Dialect/IR/CIROps.h.inc"
-
-namespace cir {
-/// The alloca that defines \p addr, looking through casts that preserve the
-/// underlying storage. Null when the chain does not end at an alloca.
-///
-/// Use the result only to inspect the allocation, such as its alignment or
-/// its allocated type. Addressing has to keep using \p addr, whose type and
-/// address space may differ from the alloca's own result.
-AllocaOp getUnderlyingAlloca(mlir::Value addr);
-} // namespace cir
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
+#include "clang/CIR/Dialect/IR/CIROpsArith.h"
+#include "clang/CIR/Dialect/IR/CIROpsBuiltins.h"
+#include "clang/CIR/Dialect/IR/CIROpsClassMembers.h"
+#include "clang/CIR/Dialect/IR/CIROpsComplex.h"
+#include "clang/CIR/Dialect/IR/CIROpsControlFlow.h"
+#include "clang/CIR/Dialect/IR/CIROpsCore.h"
+#include "clang/CIR/Dialect/IR/CIROpsCoroutine.h"
+#include "clang/CIR/Dialect/IR/CIROpsEH.h"
+#include "clang/CIR/Dialect/IR/CIROpsMath.h"
+#include "clang/CIR/Dialect/IR/CIROpsSymbols.h"
+#include "clang/CIR/Dialect/IR/CIROpsVectorMatrix.h"
#endif // CLANG_CIR_DIALECT_IR_CIRDIALECT_H
diff --git a/clang/include/clang/CIR/Dialect/IR/CIRDialect.td b/clang/include/clang/CIR/Dialect/IR/CIRDialect.td
index 54d7370a7990fa..bdc88627ca35d9 100644
--- a/clang/include/clang/CIR/Dialect/IR/CIRDialect.td
+++ b/clang/include/clang/CIR/Dialect/IR/CIRDialect.td
@@ -15,10 +15,35 @@
include "mlir/IR/OpBase.td"
+// The op partitions of the dialect. Each one is generated into its own
+// CIROps<Partition>.h.inc / .cpp.inc and compiled in its own translation unit
+// (see "Natural Sharding" in mlir/docs/DefiningDialects/Operations.md). Ops in
+// CIROps.td pick a partition with `let partition = ...`, on the op, on its
+// base class or around a group of ops; ops that pick none are in `Core`.
+def CIR_CorePartition : Partition<"Core">;
+def CIR_SymbolsPartition : Partition<"Symbols">;
+def CIR_ControlFlowPartition : Partition<"ControlFlow">;
+def CIR_ArithPartition : Partition<"Arith">;
+def CIR_BuiltinsPartition : Partition<"Builtins">;
+def CIR_MathPartition : Partition<"Math">;
+def CIR_CoroutinePartition : Partition<"Coroutine">;
+def CIR_EHPartition : Partition<"EH">;
+def CIR_VectorMatrixPartition : Partition<"VectorMatrix">;
+def CIR_ComplexPartition : Partition<"Complex">;
+def CIR_ClassMembersPartition : Partition<"ClassMembers">;
+
def CIR_Dialect : Dialect {
let name = "cir";
let useStrictPropertiesInAssemblyFormat = 1;
+ let opPartitions = [
+ CIR_CorePartition, CIR_SymbolsPartition, CIR_ControlFlowPartition,
+ CIR_ArithPartition, CIR_BuiltinsPartition, CIR_MathPartition,
+ CIR_CoroutinePartition, CIR_EHPartition, CIR_VectorMatrixPartition,
+ CIR_ComplexPartition, CIR_ClassMembersPartition
+ ];
+ let defaultOpPartition = CIR_CorePartition;
+
// A short one-line summary of our dialect.
let summary = "A high-level dialect for analyzing and optimizing Clang "
"supported languages";
diff --git a/clang/include/clang/CIR/Dialect/IR/CIRDialectBase.h b/clang/include/clang/CIR/Dialect/IR/CIRDialectBase.h
new file mode 100644
index 00000000000000..d5ce2162e7c9c7
--- /dev/null
+++ b/clang/include/clang/CIR/Dialect/IR/CIRDialectBase.h
@@ -0,0 +1,107 @@
+//===- CIRDialectBase.h - CIR dialect without its ops -----------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the CIR dialect class, its attributes, types, enums and
+// interfaces, and the helpers shared by the op partition headers
+// (CIROps<Partition>.h). Include CIRDialect.h for the dialect with all of its
+// ops.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef CLANG_CIR_DIALECT_IR_CIRDIALECTBASE_H
+#define CLANG_CIR_DIALECT_IR_CIRDIALECTBASE_H
+
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Dialect.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/Interfaces/CallInterfaces.h"
+#include "mlir/Interfaces/ControlFlowInterfaces.h"
+#include "mlir/Interfaces/FunctionInterfaces.h"
+#include "mlir/Interfaces/InferTypeOpInterface.h"
+#include "mlir/Interfaces/LoopLikeInterface.h"
+#include "mlir/Interfaces/MemorySlotInterfaces.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+
+#include "clang/CIR/Dialect/IR/CIRAttrs.h"
+#include "clang/CIR/Dialect/IR/CIROpsDialect.h.inc"
+#include "clang/CIR/Dialect/IR/CIROpsEnums.h"
+#include "clang/CIR/Dialect/IR/CIRTypes.h"
+#include "clang/CIR/Interfaces/CIRLoopOpInterface.h"
+#include "clang/CIR/Interfaces/CIROpInterfaces.h"
+#include "clang/CIR/MissingFeatures.h"
+
+using BuilderCallbackRef =
+ llvm::function_ref<void(mlir::OpBuilder &, mlir::Location)>;
+using BuilderOpStateCallbackRef = llvm::function_ref<void(
+ mlir::OpBuilder &, mlir::Location, mlir::OperationState &)>;
+
+namespace cir {
+void buildTerminatedBody(mlir::OpBuilder &builder, mlir::Location loc);
+
+/// The process floating-point environment, including its rounding mode and
+/// exception state.
+struct FloatingPointEnvironmentResource
+ : public mlir::SideEffects::Resource::Base<
+ FloatingPointEnvironmentResource> {
+ mlir::StringRef getName() const final { return "FloatingPointEnvironment"; }
+ bool isAddressable() const final { return false; }
+};
+
+template <typename ConcreteType>
+class FenvOpTrait : public mlir::OpTrait::TraitBase<ConcreteType, FenvOpTrait> {
+public:
+ mlir::Speculation::Speculatability getSpeculatability() {
+ // Masked exceptions cannot trap. When strict_except is false, exception
+ // side effects are non-deterministic, so speculation is still safe.
+ FPEnvConstrainedOpInterface fenvOp = getFenvOp();
+ if (fenvOp.getFenvExceptionMode() == cir::FPExceptionMode::Masked &&
+ !fenvOp.getFenvStrictExcept())
+ return mlir::Speculation::Speculatable;
+
+ return mlir::Speculation::NotSpeculatable;
+ }
+
+ void getEffects(
+ llvm::SmallVectorImpl<mlir::MemoryEffects::EffectInstance> &effects) {
+ if (!getFenvOp().getFenvAttr())
+ return;
+ effects.emplace_back(mlir::MemoryEffects::Read::get(),
+ FloatingPointEnvironmentResource::get());
+ effects.emplace_back(mlir::MemoryEffects::Write::get(),
+ FloatingPointEnvironmentResource::get());
+ }
+
+private:
+ FPEnvConstrainedOpInterface getFenvOp() {
+ return mlir::cast<FPEnvConstrainedOpInterface>(this->getOperation());
+ }
+};
+
+/// The properties of \p Op with every ODS-declared default applied. The
+/// aggregate op builders that take a `Properties` struct do not apply those
+/// defaults themselves, so callers of those builders have to do it here.
+template <typename Op>
+typename Op::Properties getDefaultProperties(mlir::MLIRContext *context) {
+ typename Op::Properties properties{};
+ Op::populateDefaultProperties(
+ mlir::OperationName(Op::getOperationName(), context), properties);
+ return properties;
+}
+
+/// Look up the RecordLayoutAttr for a named record in the module's
+/// cir.record_layouts dictionary. Asserts if the entry is missing.
+RecordLayoutAttr getRecordLayout(mlir::ModuleOp mod, mlir::StringAttr name);
+
+/// Same lookup as getRecordLayout, but returns a null attribute instead of
+/// asserting when the record has no layout entry.
+RecordLayoutAttr tryGetRecordLayout(mlir::ModuleOp mod, mlir::StringAttr name);
+} // namespace cir
+
+#endif // CLANG_CIR_DIALECT_IR_CIRDIALECTBASE_H
diff --git a/clang/include/clang/CIR/Dialect/IR/CIROps.td b/clang/include/clang/CIR/Dialect/IR/CIROps.td
index 4ffde906be3937..e037382b25702c 100644
--- a/clang/include/clang/CIR/Dialect/IR/CIROps.td
+++ b/clang/include/clang/CIR/Dialect/IR/CIROps.td
@@ -447,6 +447,7 @@ def CIR_DynamicCastKind : CIR_I32Enum<
def CIR_DynamicCastKindAttr
: CIR_EnumAttr<CIR_DynamicCastKind, "dynamic_cast">;
+let partition = CIR_ClassMembersPartition in {
def CIR_DynamicCastOp : CIR_Op<"dyn_cast"> {
let summary = "Perform dynamic cast on record pointers";
let description = [{
@@ -532,6 +533,7 @@ def CIR_DynamicCastOp : CIR_Op<"dyn_cast"> {
let hasLLVMLowering = false;
let hasCXXABILowering = true;
}
+} // let partition = CIR_ClassMembersPartition
//===----------------------------------------------------------------------===//
// PtrStrideOp
@@ -1046,6 +1048,7 @@ defvar CIR_ReturnableScopes = [
"CoroutineOp", "DoWhileOp", "WhileOp", "ForOp", "TryOp"
];
+let partition = CIR_ControlFlowPartition in {
def CIR_ReturnOp : CIR_Op<"return", [
ParentOneOf<CIR_ReturnableScopes>, Terminator
]> {
@@ -1185,6 +1188,7 @@ def CIR_ConditionOp : CIR_Op<"condition", [
let hasVerifier = 1;
let hasLLVMLowering = false;
}
+} // let partition = CIR_ControlFlowPartition
//===----------------------------------------------------------------------===//
// YieldOp
@@ -1196,6 +1200,7 @@ defvar CIR_YieldableScopes = [
"ScopeOp", "SwitchOp", "TernaryOp", "TryOp", "WhileOp"
];
+let partition = CIR_ControlFlowPartition in {
def CIR_YieldOp : CIR_Op<"yield", [
ReturnLike, Terminator, ParentOneOf<CIR_YieldableScopes>, NoMemoryEffect
]> {
@@ -1255,6 +1260,7 @@ def CIR_YieldOp : CIR_Op<"yield", [
let hasLLVMLowering = false;
}
+} // let partition = CIR_ControlFlowPartition
//===----------------------------------------------------------------------===//
// BreakOp
@@ -1263,6 +1269,7 @@ def CIR_YieldOp : CIR_Op<"yield", [
defvar CIR_LoopScopes = ["WhileOp", "DoWhileOp", "ForOp"];
defvar CIR_BreakableScopes = !listconcat(CIR_LoopScopes, ["SwitchOp"]);
+let partition = CIR_ControlFlowPartition in {
def CIR_BreakOp : CIR_Op<"break", [
Terminator, AncestorOneOf<CIR_BreakableScopes>
]> {
@@ -1437,6 +1444,7 @@ def CIR_ScopeOp : CIR_EnterAnyRegionBranchOpBase<"scope", [
let hasLLVMLowering = false;
}
+} // let partition = CIR_ControlFlowPartition
//===----------------------------------------------------------------------===//
// CleanupScopeOp
@@ -1461,6 +1469,7 @@ def CIR_CleanupKindAttr : CIR_EnumAttr<CIR_CleanupKind, "cleanup"> {
}];
}
+let partition = CIR_ControlFlowPartition in {
def CIR_CleanupScopeOp : CIR_RegionBranchOpBase<"cleanup.scope", [
RecursivelySpeculatable, AutomaticAllocationScope, NoRegionArguments,
RecursiveMemoryEffects
@@ -1522,7 +1531,9 @@ def CIR_CleanupScopeOp : CIR_RegionBranchOpBase<"cleanup.scope", [
let hasVerifier = 1;
let hasLLVMLowering = false;
}
+} // let partition = CIR_ControlFlowPartition
+let partition = CIR_ArithPartition in {
//===----------------------------------------------------------------------===//
// CmpThreeWayOp
//===----------------------------------------------------------------------===//
@@ -1590,6 +1601,7 @@ def CIR_CmpThreeWayOp : CIR_Op<"cmp3way", [Pure, SameTypeOperands]> {
let hasLLVMLowering = false;
}
+} // let partition = CIR_ArithPartition
//===----------------------------------------------------------------------===//
// SwitchOp
@@ -1604,6 +1616,7 @@ def CIR_CaseOpKind : CIR_I32Enum<"CaseOpKind", "case kind", [
def CIR_CaseOpKindAttr : CIR_EnumAttr<CIR_CaseOpKind, "case">;
+let partition = CIR_ControlFlowPartition in {
def CIR_CaseOp : CIR_EnterAnyRegionBranchOpBase<"case", [
RecursivelySpeculatable, AutomaticAllocationScope
]> {
@@ -1816,7 +1829,9 @@ def CIR_SwitchOp : CIR_EnterAnyRegionBranchOpBase<"switch", [
let hasLLVMLowering = false;
}
+} // let partition = CIR_ControlFlowPartition
+let partition = CIR_BuiltinsPartition in {
//===----------------------------------------------------------------------===//
// IsConstantOp
//===----------------------------------------------------------------------===//
@@ -1847,7 +1862,9 @@ def CIR_IsConstantOp : CIR_Op<"is_constant", [Pure]> {
let llvmOp = "IsConstantOp";
}
+} // let partition = CIR_BuiltinsPartition
+let partition = CIR_ControlFlowPartition in {
//===----------------------------------------------------------------------===//
// SwitchFlatOp
//===----------------------------------------------------------------------===//
@@ -2051,6 +2068,7 @@ def CIR_LabelOp : CIR_Op<"label", [AlwaysSpeculatable]> {
let customLLVMLoweringConstructorDecl =
LoweringBuilders<(ins "LLVMBlockAddressInfo &":$blockInfoAddr)>;
}
+} // let partition = CIR_ControlFlowPartition
//===----------------------------------------------------------------------===//
// Unary Operations
@@ -2064,6 +2082,7 @@ class CIR_UnaryOp<string mnemonic, Type type, list<Trait> traits = []>
DeclareOpInterfaceMethods<CIR_UnaryOpInterface>
], traits)>
{
+ let partition = CIR_ArithPartition;
dag commonArgs = (ins type:$input);
let arguments = commonArgs;
@@ -2210,6 +2229,7 @@ def CIR_FNegOp : CIR_UnaryOp<"fneg", CIR_AnyFloatOrVecOfFloatType> {
let llvmOp = "FNegOp";
}
+let partition = CIR_ArithPartition in {
//===----------------------------------------------------------------------===//
// FreezeOp
//===----------------------------------------------------------------------===//
@@ -2238,7 +2258,9 @@ def CIR_FreezeOp : CIR_Op<"freeze", [Pure, SameOperandsAndResultType]> {
let llvmOp = "FreezeOp";
}
+} // let partition = CIR_ArithPartition
+let partition = CIR_ControlFlowPartition in {
//===----------------------------------------------------------------------===//
// BrCondOp
//===----------------------------------------------------------------------===//
@@ -2346,6 +2368,7 @@ def CIR_IndirectBrOp : CIR_Op<"indirect_br", [
attr-dict
}];
}
+} // let partition = CIR_ControlFlowPartition
//===----------------------------------------------------------------------===//
// Common loop op definitions
@@ -2354,6 +2377,7 @@ def CIR_IndirectBrOp : CIR_Op<"indirect_br", [
class CIR_LoopOpBase<string mnemonic> : CIR_Op<mnemonic, [
LoopOpInterface, NoRegionArguments, LoopHintsRequireLoopId
]> {
+ let partition = CIR_ControlFlowPartition;
let extraClassDefinition = [{
void $cppClass::getSuccessorRegions(
mlir::RegionBranchPoint point,
@@ -2645,6 +2669,7 @@ def CIR_CmpOpKind : CIR_I32Enum<"CmpOpKind", "compare operation kind", [
def CIR_CmpOpKindAttr : CIR_EnumAttr<CIR_CmpOpKind, "cmp">;
+let partition = CIR_ArithPartition in {
def CIR_CmpOp : CIR_Op<"cmp",
!listconcat([SameTypeOperands], CIR_FenvOpTraits)> {
let summary = "Compare two values and produce a boolean result";
@@ -2710,6 +2735,7 @@ def CIR_CmpOp : CIR_Op<"cmp",
let isLLVMLoweringRecursive = true;
let hasCXXABILowering = true;
}
+} // let partition = CIR_ArithPartition
//===----------------------------------------------------------------------===//
// Checked Binary Arithmetic Operations (with overflow)
@@ -2734,6 +2760,7 @@ def CIR_CmpOp : CIR_Op<"cmp",
// have different widths or signedness.
class CIR_BinOpOverflow<string mnemonic, list<Trait> traits = []>
: CIR_Op<mnemonic, !listconcat([Pure, SameTypeOperands], traits)> {
+ let partition = CIR_ArithPartition;
let arguments = (ins
CIR_IntType:$lhs,
@@ -2817,6 +2844,7 @@ class CIR_BinaryOp<string mnemonic, Type type, list<Trait> traits = [],
DeclareOpInterfaceMethods<CIR_BinaryOpInterface>
], traits)>
{
+ let partition = CIR_ArithPartition;
dag commonArgs = (ins
type:$lhs,
type:$rhs
@@ -3013,6 +3041,7 @@ class CIR_FPBinaryOp<string mnemonic, list<Trait> traits = [],
: CIR_BinaryOp<mnemonic, type,
!listconcat(CIR_FenvOpTraits, traits),
CIR_DynamicMemoryEffects> {
+ let partition = CIR_ArithPartition;
let arguments = !con(commonArgs, (ins
OptionalAttr<CIR_FenvAttr>:$fenv,
OptionalAttr<CIR_FastMathFlagsAttr>:$fastmath_flags));
@@ -3264,6 +3293,7 @@ def CIR_MinOp : CIR_BinaryOp<"min", CIR_AnyIntOrVecOfIntType, [
}];
}
+let partition = CIR_ArithPartition in {
//===----------------------------------------------------------------------===//
// ShiftOp
//===----------------------------------------------------------------------===//
@@ -3304,7 +3334,9 @@ def CIR_ShiftOp : CIR_Op<"shift", [Pure]> {
let hasVerifier = 1;
}
+} // let partition = CIR_ArithPartition
+let partition = CIR_ControlFlowPartition in {
//===----------------------------------------------------------------------===//
// SelectOp
//===----------------------------------------------------------------------===//
@@ -3412,6 +3444,7 @@ def CIR_TernaryOp : CIR_EnterAnyRegionBranchOpBase<"ternary", [
let hasLLVMLowering = false;
}
+} // let partition = CIR_ControlFlowPartition
//===----------------------------------------------------------------------===//
// GlobalOp
@@ -3470,6 +3503,7 @@ def CIR_TLSModelAttr: CIR_EnumAttr<CIR_TLSModel, "tls_model"> {
}];
}
+let partition = CIR_SymbolsPartition in {
def CIR_GlobalOp : CIR_RegionBranchOpBase<"global", [
SymbolName, SymbolVisibility,
DeclareOpInterfaceMethods<CIRGlobalValueInterface>,
@@ -3657,7 +3691,9 @@ def CIR_GetGlobalOp : CIR_Op<"get_global", [
$name `:` qualified(type($addr)) attr-dict
}];
}
+} // let partition = CIR_SymbolsPartition
+let partition = CIR_ClassMembersPartition in {
//===----------------------------------------------------------------------===//
// VTableAddrPointOp
//===----------------------------------------------------------------------===//
@@ -4067,6 +4103,7 @@ def CIR_GetBitfieldOp : CIR_Op<"get_bitfield"> {
}]>
];
}
+} // let partition = CIR_ClassMembersPartition
//===----------------------------------------------------------------------===//
// GetMemberOp
@@ -4317,6 +4354,7 @@ def CIR_CallingConv : CIR_I32Enum<"CallingConv", "calling convention", [
def CIR_CallingConvAttr : CIR_EnumAttr<CIR_CallingConv, "calling_conv">;
+let partition = CIR_SymbolsPartition in {
def CIR_FuncOp : CIR_Op<"func", [
AutomaticAllocationScope, CallableOpInterface, SymbolName, SymbolVisibility,
FunctionOpInterface,
@@ -4680,6 +4718,7 @@ def CIR_LLVMIntrinsicCallOp : CIR_Op<"call_llvm_intrinsic"> {
}]>,
];
}
+} // let partition = CIR_SymbolsPartition
//===----------------------------------------------------------------------===//
// DeleteArrayOp
@@ -4742,6 +4781,7 @@ def CIR_DeleteArrayOp : CIR_Op<"delete_array"> {
let hasCXXABILowering = true;
}
+let partition = CIR_SymbolsPartition in {
//===----------------------------------------------------------------------===//
// CallOp and TryCallOp
//===----------------------------------------------------------------------===//
@@ -4951,7 +4991,9 @@ def CIR_TryCallOp : CIR_CallOpBase<"try_call",[
}]>
];
}
+} // let partition = CIR_SymbolsPartition
+let partition = CIR_CoroutinePartition in {
//===----------------------------------------------------------------------===//
// CoroutineOp
//===----------------------------------------------------------------------===//
@@ -5037,6 +5079,7 @@ def CIR_CoroutineOp : CIR_Op<"coroutine", [
let hasLLVMLowering = false;
let hasVerifier = 1;
}
+} // let partition = CIR_CoroutinePartition
//===----------------------------------------------------------------------===//
// AwaitOp
@@ -5051,6 +5094,7 @@ def CIR_AwaitKind : CIR_I32Enum<"AwaitKind", "await kind", [
def CIR_AwaitKindAttr : CIR_EnumAttr<CIR_AwaitKind, "await">;
+let partition = CIR_CoroutinePartition in {
def CIR_AwaitOp : CIR_RegionBranchOpBase<"await", [
RecursivelySpeculatable, NoRegionArguments, HasAncestor<"CoroutineOp">
]> {
@@ -5155,6 +5199,7 @@ def CIR_CoReturnOp : CIR_Op<"co_return", [
let hasVerifier = 1;
let hasLLVMLowering = false;
}
+} // let partition = CIR_CoroutinePartition
//===----------------------------------------------------------------------===//
// Coroutine intrinsics
@@ -5163,6 +5208,7 @@ def CIR_CoReturnOp : CIR_Op<"co_return", [
class CIR_CoroIntrinsicOp<string mnem, dag ins, dag outs,
list<Trait> traits = []>
: CIR_Op<"coro.intrinsic." # mnem, traits> {
+ let partition = CIR_CoroutinePartition;
let hasLLVMLowering = 1;
let arguments = ins;
let results = outs;
@@ -5336,6 +5382,7 @@ def CIR_CoroSaveOp : CIR_CoroIntrinsicOp<"save",
let llvmOp = "CoroSaveOp";
}
+let partition = CIR_CoroutinePartition in {
//===----------------------------------------------------------------------===//
// Coroutine CoroExitOP
//===----------------------------------------------------------------------===//
@@ -5372,6 +5419,7 @@ def CIR_CoroCleanupOp : CIR_Op<"coro.cleanup", [
let assemblyFormat = "attr-dict";
let hasLLVMLowering = false;
}
+} // let partition = CIR_CoroutinePartition
//===----------------------------------------------------------------------===//
// Coroutine intrinsic PromiseOp
@@ -5439,6 +5487,7 @@ def CIR_CoroDoneOp : CIR_CoroIntrinsicOp<"done", (ins CIR_VoidPtrType:$handle),
let llvmOp = "CoroDoneOp";
}
+let partition = CIR_CoroutinePartition in {
//===----------------------------------------------------------------------===//
// CoroSuspendPoint
//===----------------------------------------------------------------------===//
@@ -5486,6 +5535,7 @@ def CIR_CoroSuspendPoint : CIR_Op<"coro.suspend_point", [
let hasLLVMLowering = false;
}
+} // let partition = CIR_CoroutinePartition
//===----------------------------------------------------------------------===//
// Coroutine intrinsic NoopOp
@@ -5500,6 +5550,7 @@ def CIR_CoroNoopOp : CIR_CoroIntrinsicOp<"noop", (ins),
let llvmOp = "CoroNoopOp";
}
+let partition = CIR_BuiltinsPartition in {
//===----------------------------------------------------------------------===//
// CopyOp
//===----------------------------------------------------------------------===//
@@ -5572,6 +5623,7 @@ def CIR_CopyOp : CIR_Op<"copy",[
}
}];
}
+} // let partition = CIR_BuiltinsPartition
//===----------------------------------------------------------------------===//
// MemCpyOp && MemMoveOp
@@ -5580,6 +5632,7 @@ def CIR_CopyOp : CIR_Op<"copy",[
class CIR_MemOp<string mnemonic> : CIR_Op<mnemonic, [
AllTypesMatch<["dst", "src"]>
]> {
+ let partition = CIR_BuiltinsPartition;
dag commonArgs = (ins
Arg<CIR_VoidPtrType, "", [MemWrite]>:$dst,
Arg<CIR_VoidPtrType, "", [MemRead]>:$src
@@ -5663,6 +5716,7 @@ def CIR_MemMoveOp : CIR_MemOp<"libc.memmove"> {
}];
}
+let partition = CIR_BuiltinsPartition in {
//===----------------------------------------------------------------------===//
// MemSetOp
//===----------------------------------------------------------------------===//
@@ -5907,6 +5961,7 @@ def CIR_LifetimeEndOp : CIR_Op<"lifetime.end"> {
let hasVerifier = 1;
let llvmOp = "LifetimeEndOp";
}
+} // let partition = CIR_BuiltinsPartition
//===----------------------------------------------------------------------===//
// InlineAsmOp
@@ -5918,6 +5973,7 @@ def CIR_AsmFlavor : CIR_I32Enum<"AsmFlavor", "ATT or Intel",
def CIR_AsmFlavorAttr : CIR_EnumAttr<CIR_AsmFlavor, "asm_flavor">;
+let partition = CIR_BuiltinsPartition in {
def CIR_InlineAsmOp : CIR_Op<"asm",
[DeclareOpInterfaceMethods<MemoryEffectsOpInterface>]> {
let description = [{
@@ -5991,6 +6047,7 @@ def CIR_InlineAsmOp : CIR_Op<"asm",
let hasCustomAssemblyFormat = 1;
}
+} // let partition = CIR_BuiltinsPartition
//===----------------------------------------------------------------------===//
// UnreachableOp
@@ -6027,6 +6084,7 @@ def CIR_TrapOp : CIR_Op<"trap", [Terminator]> {
let assemblyFormat = "attr-dict";
}
+let partition = CIR_ClassMembersPartition in {
//===----------------------------------------------------------------------===//
// ArrayCtor & ArrayDtor
//===----------------------------------------------------------------------===//
@@ -6332,7 +6390,9 @@ def CIR_GetMethodOp : CIR_Op<"get_method"> {
let hasLLVMLowering = false;
let hasCXXABILowering = true;
}
+} // let partition = CIR_ClassMembersPartition
+let partition = CIR_VectorMatrixPartition in {
//===----------------------------------------------------------------------===//
// VecCreate
//===----------------------------------------------------------------------===//
@@ -6424,6 +6484,7 @@ def CIR_VecExtractOp : CIR_Op<"vec.extract", [
let hasFolder = 1;
}
+} // let partition = CIR_VectorMatrixPartition
//===----------------------------------------------------------------------===//
// Vector reduction operations
@@ -6436,6 +6497,7 @@ class CIR_VecReduceOp<string mnemonic, Type inputType, string llvmOpName>
"input", "result",
"mlir::cast<cir::VectorType>($_self).getElementType()">
]> {
+ let partition = CIR_VectorMatrixPartition;
let summary = "Reduce a vector using " # mnemonic;
let description = [{
Reduces the elements of a vector to a scalar of the vector element type.
@@ -6490,6 +6552,7 @@ class CIR_VecReduceFPOp<string mnemonic>
"input", "result",
"mlir::cast<cir::VectorType>($_self).getElementType()">
]> {
+ let partition = CIR_VectorMatrixPartition;
let summary = "Reduce a floating-point vector using " # mnemonic;
let arguments = (ins
CIR_VectorOfFloatType:$input,
@@ -6516,6 +6579,7 @@ class CIR_VecReduceAccFPOp<string mnemonic>
"input", "result",
"mlir::cast<cir::VectorType>($_self).getElementType()">
]> {
+ let partition = CIR_VectorMatrixPartition;
let summary = "Reduce a floating-point vector using " # mnemonic;
let arguments = (ins
CIR_AnyFloatType:$accumulator,
@@ -6533,6 +6597,7 @@ class CIR_VecReduceAccFPOp<string mnemonic>
def CIR_VecReduceFAddOp : CIR_VecReduceAccFPOp<"fadd">;
def CIR_VecReduceFMulOp : CIR_VecReduceAccFPOp<"fmul">;
+let partition = CIR_VectorMatrixPartition in {
//===----------------------------------------------------------------------===//
// VecCmpOp
//===----------------------------------------------------------------------===//
@@ -7033,12 +7098,14 @@ def CIR_MatrixTransposeOp : CIR_Op<"matrix.transpose", [
let hasVerifier = 1;
}
+} // let partition = CIR_VectorMatrixPartition
//===----------------------------------------------------------------------===//
// BaseClassAddrOp & DerivedClassAddrOp
//===----------------------------------------------------------------------===//
class CIR_ClassCastOp<string mnemonic> : CIR_Op<mnemonic, [Pure]> {
+ let partition = CIR_ClassMembersPartition;
let arguments = (ins
CIR_PointerType:$src_addr,
IndexAttr:$offset,
@@ -7120,6 +7187,7 @@ def CIR_DerivedClassAddrOp : CIR_ClassCastOp<"derived_class_addr"> {
}];
}
+let partition = CIR_ClassMembersPartition in {
//===----------------------------------------------------------------------===//
// BaseDataMemberOp & DerivedDataMemberOp
//===----------------------------------------------------------------------===//
@@ -7245,7 +7313,9 @@ def CIR_DerivedMethodOp : CIR_Op<"derived_method", [Pure]> {
let hasLLVMLowering = false;
let hasCXXABILowering = true;
}
+} // let partition = CIR_ClassMembersPartition
+let partition = CIR_ComplexPartition in {
//===----------------------------------------------------------------------===//
// ComplexCreateOp
//===----------------------------------------------------------------------===//
@@ -7277,12 +7347,14 @@ def CIR_ComplexCreateOp : CIR_Op<"complex.create", [Pure, SameTypeOperands]> {
let hasVerifier = 1;
let hasFolder = 1;
}
+} // let partition = CIR_ComplexPartition
//===----------------------------------------------------------------------===//
// ComplexRealOp and ComplexImagOp
//===----------------------------------------------------------------------===//
class CIR_ComplexPartOp<string mnemonic> : CIR_Op<mnemonic, [Pure]> {
+ let partition = CIR_ComplexPartition;
let results = (outs CIR_AnyIntOrBoolOrFloatType:$result);
let arguments = (ins CIR_AnyComplexOrIntOrBoolOrFloatType:$operand);
@@ -7332,6 +7404,7 @@ def CIR_ComplexImagOp : CIR_ComplexPartOp<"complex.imag"> {
//===----------------------------------------------------------------------===//
class CIR_ComplexPartPtrOp<string mnemonic> : CIR_Op<mnemonic, [Pure]> {
+ let partition = CIR_ComplexPartition;
let arguments = (ins CIR_PtrToComplexType:$operand);
let results = (outs CIR_PtrToIntOrFloatType:$result);
@@ -7381,6 +7454,7 @@ def CIR_ComplexImagPtrOp : CIR_ComplexPartPtrOp<"complex.imag_ptr"> {
class CIR_ComplexBinOp<string mnemonic>
: CIR_Op<mnemonic, [Pure, SameOperandsAndResultType]> {
+ let partition = CIR_ComplexPartition;
let arguments = (ins CIR_ComplexOfIntType:$lhs, CIR_ComplexOfIntType:$rhs);
let results = (outs CIR_ComplexOfIntType:$result);
@@ -7451,6 +7525,7 @@ def CIR_ComplexDivOp : CIR_ComplexBinOp<"complex.div"> {
class CIR_ComplexFPBinOp<string mnemonic>
: CIR_Op<mnemonic, [Pure, SameOperandsAndResultType]> {
+ let partition = CIR_ComplexPartition;
let arguments = (ins
CIR_ComplexOfFloatType:$lhs,
CIR_ComplexOfFloatType:$rhs,
@@ -7509,6 +7584,7 @@ def CIR_ComplexRangeKindAttr
class CIR_ComplexRangeBinOp<string mnemonic>
: CIR_Op<mnemonic, [Pure, SameOperandsAndResultType]> {
+ let partition = CIR_ComplexPartition;
let arguments = (ins
CIR_ComplexOfFloatType:$lhs,
CIR_ComplexOfFloatType:$rhs,
@@ -7574,6 +7650,7 @@ def CIR_ComplexFDivOp : CIR_ComplexRangeBinOp<"complex.fdiv"> {
}];
}
+let partition = CIR_ComplexPartition in {
//===----------------------------------------------------------------------===//
// ComplexConjOp
//===----------------------------------------------------------------------===//
@@ -7601,6 +7678,7 @@ def CIR_ComplexConjOp
let hasLLVMLowering = false;
}
+} // let partition = CIR_ComplexPartition
//===----------------------------------------------------------------------===//
// Bit Manipulation Operations
@@ -7608,6 +7686,7 @@ def CIR_ComplexConjOp
class CIR_BitOpBase<string mnemonic, TypeConstraint operandTy>
: CIR_Op<mnemonic, [Pure, SameOperandsAndResultType]> {
+ let partition = CIR_ArithPartition;
let arguments = (ins operandTy:$input);
let results = (outs operandTy:$result);
@@ -7814,6 +7893,7 @@ def CIR_ByteSwapOp : CIR_BitOpBase<"byte_swap",
let llvmOp = "ByteSwapOp";
}
+let partition = CIR_ArithPartition in {
//===----------------------------------------------------------------------===//
// RotateOp
//===----------------------------------------------------------------------===//
@@ -7854,6 +7934,7 @@ def CIR_RotateOp : CIR_Op<"rotate", [Pure, SameOperandsAndResultType]> {
let hasFolder = 1;
}
+} // let partition = CIR_ArithPartition
//===----------------------------------------------------------------------===//
// Funnel Shift Operations
@@ -7864,6 +7945,7 @@ def CIR_RotateOp : CIR_Op<"rotate", [Pure, SameOperandsAndResultType]> {
// type.
class CIR_FunnelShiftOp<string mnemonic, string llvmOpName>
: CIR_Op<mnemonic, [Pure, SameOperandsAndResultType]> {
+ let partition = CIR_ArithPartition;
let arguments = (ins
CIR_AnyIntOrVecOfIntType:$a,
CIR_AnyIntOrVecOfIntType:$b,
@@ -7974,6 +8056,7 @@ def FPClassTestEnum
def CIR_FPClassTestAttr : CIR_EnumAttr<FPClassTestEnum, "fp_class">;
+let partition = CIR_ArithPartition in {
def CIR_IsFPClassOp : CIR_Op<"is_fp_class", [Pure]> {
let summary = "Corresponding to the `__builtin_fpclassify` builtin function in clang";
@@ -8006,6 +8089,7 @@ def CIR_IsFPClassOp : CIR_Op<"is_fp_class", [Pure]> {
$src `,` enum($flags) `:` functional-type($src, $result) attr-dict
}];
}
+} // let partition = CIR_ArithPartition
//===----------------------------------------------------------------------===//
// Assume Operations
@@ -8022,6 +8106,7 @@ def CIR_AssumeBundleKind : CIR_I32Enum<
def CIR_AssumeBundleKindAttr
: CIR_EnumAttr<CIR_AssumeBundleKind, "assume_bundle">;
+let partition = CIR_BuiltinsPartition in {
def CIR_AssumeOp : CIR_Op<"assume"> {
let summary = "Tell the optimizer that a boolean value is true";
let description = [{
@@ -8225,7 +8310,9 @@ def CIR_ObjSizeOp : CIR_Op<"objsize", [Pure]> {
$ptr `:` qualified(type($ptr)) `->` qualified(type($result)) attr-dict
}];
}
+} // let partition = CIR_BuiltinsPartition
+let partition = CIR_ArithPartition in {
//===----------------------------------------------------------------------===//
// PtrDiffOp
//===----------------------------------------------------------------------===//
@@ -8262,6 +8349,7 @@ def CIR_PtrDiffOp : CIR_Op<"ptr_diff", [Pure, SameTypeOperands]> {
attr-dict
}];
}
+} // let partition = CIR_ArithPartition
//===----------------------------------------------------------------------===//
// Floating Point Ops
@@ -8273,6 +8361,7 @@ class CIR_UnaryFPToFPBuiltinOp<string mnemonic, string llvmOpName>
: CIR_Op<mnemonic,
!listconcat([SameOperandsAndResultType], CIR_FenvOpTraits)>
{
+ let partition = CIR_MathPartition;
let arguments = (ins CIR_AnyFloatOrVecOfFloatType:$src,
OptionalAttr<CIR_FenvAttr>:$fenv);
let results = (outs CIR_AnyFloatOrVecOfFloatType:$result);
@@ -8544,6 +8633,7 @@ def CIR_FAbsOp : CIR_UnaryFPToFPBuiltinOp<"fabs", "FAbsOp"> {
let constrainedLLVMIntrinsic = "";
}
+let partition = CIR_MathPartition in {
def CIR_AbsOp : CIR_Op<"abs", [Pure, SameOperandsAndResultType]> {
let summary = "Computes the absolute value of a signed integer";
let description = [{
@@ -8572,6 +8662,7 @@ def CIR_AbsOp : CIR_Op<"abs", [Pure, SameOperandsAndResultType]> {
let assemblyFormat = "$src ( `min_is_poison` $min_is_poison^ )? `:` type($src) attr-dict";
}
+} // let partition = CIR_MathPartition
def CIR_FloorOp : CIR_UnaryFPToFPBuiltinOp<"floor", "FFloorOp"> {
let summary = "Computes the floating-point floor value";
@@ -8595,6 +8686,7 @@ def CIR_FloorOp : CIR_UnaryFPToFPBuiltinOp<"floor", "FFloorOp"> {
class CIR_UnaryFPToIntBuiltinOp<string mnemonic, string llvmOpName>
: CIR_Op<mnemonic, CIR_FenvOpTraits>
{
+ let partition = CIR_MathPartition;
let arguments = (ins CIR_AnyFloatType:$src,
OptionalAttr<CIR_FenvAttr>:$fenv);
let results = (outs CIR_IntType:$result);
@@ -8659,6 +8751,7 @@ def CIR_LlrintOp : CIR_UnaryFPToIntBuiltinOp<"llrint", "LlrintOp"> {
class CIR_BinaryFPToFPBuiltinOp<string mnemonic, string llvmOpName>
: CIR_Op<mnemonic,
!listconcat([SameOperandsAndResultType], CIR_FenvOpTraits)> {
+ let partition = CIR_MathPartition;
let summary = [{
libc builtin equivalent ignoring floating-point exceptions and errno.
}];
@@ -8769,6 +8862,7 @@ def CIR_ATan2Op : CIR_BinaryFPToFPBuiltinOp<"atan2", "ATan2Op"> {
let constrainedLLVMIntrinsic = "atan2";
}
+let partition = CIR_MathPartition in {
def CIR_FrexpOp : CIR_Op<"frexp", [Pure]> {
let summary = "Decomposes a floating-point value into significand and exponent";
let description = [{
@@ -8809,11 +8903,13 @@ def CIR_ModfOp : CIR_Op<"modf", [Pure]> {
$src `:` type($src) `->` type($fractional) `,` type($integral) attr-dict
}];
}
+} // let partition = CIR_MathPartition
class CIR_TernaryFPToFPBuiltinOp<string mnemonic, string llvmOpName>
: CIR_Op<mnemonic,
!listconcat([SameOperandsAndResultType], CIR_FenvOpTraits)>
{
+ let partition = CIR_MathPartition;
let arguments = (ins
CIR_AnyFloatOrVecOfFloatType:$a,
CIR_AnyFloatOrVecOfFloatType:$b,
@@ -8893,6 +8989,7 @@ def CIR_FMulAddOp : CIR_TernaryFPToFPBuiltinOp<"fmuladd", "FMulAddOp"> {
//===----------------------------------------------------------------------===//
class CIR_VAOp<string mnemonic> : CIR_Op<mnemonic> {
+ let partition = CIR_BuiltinsPartition;
let arguments = (ins CIR_PointerType:$arg_list);
let assemblyFormat = [{
@@ -8961,6 +9058,7 @@ def CIR_VAEndOp : CIR_VAOp<"va_end"> {
}];
}
+let partition = CIR_BuiltinsPartition in {
def CIR_VACopyOp : CIR_Op<"va_copy"> {
let summary = "Copied a variable argument list";
let description = [{
@@ -9032,6 +9130,7 @@ def CIR_VAArgOp : CIR_Op<"va_arg"> {
$arg_list attr-dict `:` functional-type(operands, $result)
}];
}
+} // let partition = CIR_BuiltinsPartition
//===----------------------------------------------------------------------===//
// ThrowOpBase
@@ -9044,6 +9143,7 @@ def CIR_VAArgOp : CIR_Op<"va_arg"> {
// any additional traits, successors and assembly format pieces.
class CIR_ThrowOpBase<string mnemonic, list<Trait> traits = []>
: CIR_Op<mnemonic, traits> {
+ let partition = CIR_EHPartition;
let arguments = (ins
Optional<CIR_PointerType>:$exception_ptr,
OptionalAttr<FlatSymbolRefAttr>:$type_info,
@@ -9158,6 +9258,7 @@ def CIR_TryThrowOp : CIR_ThrowOpBase<"try_throw", [Terminator]> {
let hasLLVMLowering = false;
}
+let partition = CIR_EHPartition in {
//===----------------------------------------------------------------------===//
// AllocExceptionOp
//===----------------------------------------------------------------------===//
@@ -9884,6 +9985,7 @@ def CIR_EndCatchOp : CIR_Op<"end_catch"> {
let hasLLVMLowering = false;
}
+} // let partition = CIR_EHPartition
//===----------------------------------------------------------------------===//
// EH Operations: InitCatchParamOp
@@ -9903,6 +10005,7 @@ def CIR_InitCatchKind : CIR_I32Enum<
def CIR_InitCatchKindAttr : CIR_EnumAttr<CIR_InitCatchKind, "init_catch">;
+let partition = CIR_EHPartition in {
def CIR_InitCatchParamOp : CIR_Op<"init_catch_param"> {
let summary = "Initialize a catch parameter from the exception pointer";
let description = [{
@@ -10006,6 +10109,7 @@ def CIR_ConstructCatchParamOp : CIR_Op<"construct_catch_param", [
let hasVerifier = 1;
let hasLLVMLowering = false;
}
+} // let partition = CIR_EHPartition
//===----------------------------------------------------------------------===//
// TokenNoneOp
@@ -10052,6 +10156,7 @@ def CIR_AtomicFetchKind : CIR_I32Enum<
def CIR_AtomicFetchKindAttr
: CIR_EnumAttr<CIR_AtomicFetchKind, "atomic_fetch">;
+let partition = CIR_BuiltinsPartition in {
def CIR_AtomicFetchOp : CIR_Op<"atomic.fetch", [
AllTypesMatch<["result", "val"]>,
TypesMatchWith<"type of 'val' must match the pointee type of 'ptr'",
@@ -10332,7 +10437,9 @@ def CIR_AtomicFenceOp : CIR_Op<"atomic.fence"> {
(`syncscope` `(` enum($syncscope)^ `)`)? enum($ordering) attr-dict
}];
}
+} // let partition = CIR_BuiltinsPartition
+let partition = CIR_SymbolsPartition in {
//===----------------------------------------------------------------------===//
// BlockAddressOp
//===----------------------------------------------------------------------===//
@@ -10368,7 +10475,9 @@ def CIR_BlockAddressOp : CIR_Op<"block_address", [Pure]> {
let customLLVMLoweringConstructorDecl =
LoweringBuilders<(ins "LLVMBlockAddressInfo &":$blockInfoAddr)>;
}
+} // let partition = CIR_SymbolsPartition
+let partition = CIR_BuiltinsPartition in {
//===----------------------------------------------------------------------===//
// CpuIdOp
//===----------------------------------------------------------------------===//
@@ -10565,6 +10674,7 @@ def CIR_ClearPaddingOp : CIR_Op<"clear_padding", []> {
$padding `)` `:` type($arg) `->` `(` `)` attr-dict
}];
}
+} // let partition = CIR_BuiltinsPartition
include "clang/CIR/Dialect/IR/CIRStdOps.td"
diff --git a/clang/include/clang/CIR/Dialect/IR/CIROpsArith.h b/clang/include/clang/CIR/Dialect/IR/CIROpsArith.h
new file mode 100644
index 00000000000000..380edb31376567
--- /dev/null
+++ b/clang/include/clang/CIR/Dialect/IR/CIROpsArith.h
@@ -0,0 +1,24 @@
+//===- CIROpsArith.h - CIR ops in partition `Arith` -------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the CIR arithmetic ops: unary and binary arithmetic,
+// comparisons, shifts and bit manipulation. The ops of the other partitions
+// are forward declared; include their headers, or CIRDialect.h for all ops,
+// to use them.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef CLANG_CIR_DIALECT_IR_CIROPSARITH_H
+#define CLANG_CIR_DIALECT_IR_CIROPSARITH_H
+
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsArith.h.inc"
+
+#endif // CLANG_CIR_DIALECT_IR_CIROPSARITH_H
diff --git a/clang/include/clang/CIR/Dialect/IR/CIROpsBuiltins.h b/clang/include/clang/CIR/Dialect/IR/CIROpsBuiltins.h
new file mode 100644
index 00000000000000..31b2da7b564445
--- /dev/null
+++ b/clang/include/clang/CIR/Dialect/IR/CIROpsBuiltins.h
@@ -0,0 +1,25 @@
+//===- CIROpsBuiltins.h - CIR ops in partition `Builtins` -------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the CIR ops for compiler builtins: memory and stack
+// intrinsics, atomics, variadic argument handling, inline assembly and the
+// raised standard library calls (the math builtins are in CIROpsMath.h). The
+// ops of the other partitions are forward declared; include their headers, or
+// CIRDialect.h for all ops, to use them.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef CLANG_CIR_DIALECT_IR_CIROPSBUILTINS_H
+#define CLANG_CIR_DIALECT_IR_CIROPSBUILTINS_H
+
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsBuiltins.h.inc"
+
+#endif // CLANG_CIR_DIALECT_IR_CIROPSBUILTINS_H
diff --git a/clang/include/clang/CIR/Dialect/IR/CIROpsClassMembers.h b/clang/include/clang/CIR/Dialect/IR/CIROpsClassMembers.h
new file mode 100644
index 00000000000000..0335c80659a1da
--- /dev/null
+++ b/clang/include/clang/CIR/Dialect/IR/CIROpsClassMembers.h
@@ -0,0 +1,25 @@
+//===- CIROpsClassMembers.h - CIR ops, `ClassMembers` partition -*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the CIR ops for C++ classes: vtables, bit-fields,
+// base and derived class addresses, member and member function pointers,
+// dynamic casts and array constructors and destructors. The ops of the other
+// partitions are forward declared; include their headers, or CIRDialect.h
+// for all ops, to use them.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef CLANG_CIR_DIALECT_IR_CIROPSCLASSMEMBERS_H
+#define CLANG_CIR_DIALECT_IR_CIROPSCLASSMEMBERS_H
+
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsClassMembers.h.inc"
+
+#endif // CLANG_CIR_DIALECT_IR_CIROPSCLASSMEMBERS_H
diff --git a/clang/include/clang/CIR/Dialect/IR/CIROpsComplex.h b/clang/include/clang/CIR/Dialect/IR/CIROpsComplex.h
new file mode 100644
index 00000000000000..6d23d0d2b54df6
--- /dev/null
+++ b/clang/include/clang/CIR/Dialect/IR/CIROpsComplex.h
@@ -0,0 +1,23 @@
+//===- CIROpsComplex.h - CIR ops in partition `Complex` ---------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the CIR complex number ops. The ops of the other
+// partitions are forward declared; include their headers, or CIRDialect.h
+// for all ops, to use them.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef CLANG_CIR_DIALECT_IR_CIROPSCOMPLEX_H
+#define CLANG_CIR_DIALECT_IR_CIROPSCOMPLEX_H
+
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsComplex.h.inc"
+
+#endif // CLANG_CIR_DIALECT_IR_CIROPSCOMPLEX_H
diff --git a/clang/include/clang/CIR/Dialect/IR/CIROpsControlFlow.h b/clang/include/clang/CIR/Dialect/IR/CIROpsControlFlow.h
new file mode 100644
index 00000000000000..469120b6fcae35
--- /dev/null
+++ b/clang/include/clang/CIR/Dialect/IR/CIROpsControlFlow.h
@@ -0,0 +1,24 @@
+//===- CIROpsControlFlow.h - CIR ops in partition `ControlFlow` -*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the CIR control flow ops: structured scopes, loops and
+// switches, their terminators, branches, gotos and labels. The ops of the
+// other partitions are forward declared; include their headers, or
+// CIRDialect.h for all ops, to use them.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef CLANG_CIR_DIALECT_IR_CIROPSCONTROLFLOW_H
+#define CLANG_CIR_DIALECT_IR_CIROPSCONTROLFLOW_H
+
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsControlFlow.h.inc"
+
+#endif // CLANG_CIR_DIALECT_IR_CIROPSCONTROLFLOW_H
diff --git a/clang/include/clang/CIR/Dialect/IR/CIROpsCore.h b/clang/include/clang/CIR/Dialect/IR/CIROpsCore.h
new file mode 100644
index 00000000000000..5981aecd21c4bf
--- /dev/null
+++ b/clang/include/clang/CIR/Dialect/IR/CIROpsCore.h
@@ -0,0 +1,33 @@
+//===- CIROpsCore.h - CIR ops in partition `Core` ---------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the core CIR ops: casts, constants, memory accesses and
+// record member accesses. The ops of the other partitions are forward
+// declared; include their headers, or CIRDialect.h for all ops, to use them.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef CLANG_CIR_DIALECT_IR_CIROPSCORE_H
+#define CLANG_CIR_DIALECT_IR_CIROPSCORE_H
+
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsCore.h.inc"
+
+namespace cir {
+/// The alloca that defines \p addr, looking through casts that preserve the
+/// underlying storage. Null when the chain does not end at an alloca.
+///
+/// Use the result only to inspect the allocation, such as its alignment or
+/// its allocated type. Addressing has to keep using \p addr, whose type and
+/// address space may differ from the alloca's own result.
+AllocaOp getUnderlyingAlloca(mlir::Value addr);
+} // namespace cir
+
+#endif // CLANG_CIR_DIALECT_IR_CIROPSCORE_H
diff --git a/clang/include/clang/CIR/Dialect/IR/CIROpsCoroutine.h b/clang/include/clang/CIR/Dialect/IR/CIROpsCoroutine.h
new file mode 100644
index 00000000000000..769521e958fc45
--- /dev/null
+++ b/clang/include/clang/CIR/Dialect/IR/CIROpsCoroutine.h
@@ -0,0 +1,23 @@
+//===- CIROpsCoroutine.h - CIR ops in partition `Coroutine` -----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the CIR coroutine ops and intrinsics. The ops of the
+// other partitions are forward declared; include their headers, or
+// CIRDialect.h for all ops, to use them.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef CLANG_CIR_DIALECT_IR_CIROPSCOROUTINE_H
+#define CLANG_CIR_DIALECT_IR_CIROPSCOROUTINE_H
+
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsCoroutine.h.inc"
+
+#endif // CLANG_CIR_DIALECT_IR_CIROPSCOROUTINE_H
diff --git a/clang/include/clang/CIR/Dialect/IR/CIROpsEH.h b/clang/include/clang/CIR/Dialect/IR/CIROpsEH.h
new file mode 100644
index 00000000000000..c31c4216ad5fab
--- /dev/null
+++ b/clang/include/clang/CIR/Dialect/IR/CIROpsEH.h
@@ -0,0 +1,23 @@
+//===- CIROpsEH.h - CIR ops in partition `EH` -------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the CIR exception handling ops. The ops of the other
+// partitions are forward declared; include their headers, or CIRDialect.h
+// for all ops, to use them.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef CLANG_CIR_DIALECT_IR_CIROPSEH_H
+#define CLANG_CIR_DIALECT_IR_CIROPSEH_H
+
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsEH.h.inc"
+
+#endif // CLANG_CIR_DIALECT_IR_CIROPSEH_H
diff --git a/clang/include/clang/CIR/Dialect/IR/CIROpsMath.h b/clang/include/clang/CIR/Dialect/IR/CIROpsMath.h
new file mode 100644
index 00000000000000..a00b8658d0dc74
--- /dev/null
+++ b/clang/include/clang/CIR/Dialect/IR/CIROpsMath.h
@@ -0,0 +1,23 @@
+//===- CIROpsMath.h - CIR ops in partition `Math` ---------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the CIR ops for the math builtins. The ops of the other
+// partitions are forward declared; include their headers, or CIRDialect.h
+// for all ops, to use them.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef CLANG_CIR_DIALECT_IR_CIROPSMATH_H
+#define CLANG_CIR_DIALECT_IR_CIROPSMATH_H
+
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsMath.h.inc"
+
+#endif // CLANG_CIR_DIALECT_IR_CIROPSMATH_H
diff --git a/clang/include/clang/CIR/Dialect/IR/CIROpsSymbols.h b/clang/include/clang/CIR/Dialect/IR/CIROpsSymbols.h
new file mode 100644
index 00000000000000..46c4efb6b2b32f
--- /dev/null
+++ b/clang/include/clang/CIR/Dialect/IR/CIROpsSymbols.h
@@ -0,0 +1,24 @@
+//===- CIROpsSymbols.h - CIR ops in partition `Symbols` ---------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the CIR ops that define or refer to symbols: functions,
+// globals, calls and the initialization of globals. The ops of the other
+// partitions are forward declared; include their headers, or CIRDialect.h for
+// all ops, to use them.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef CLANG_CIR_DIALECT_IR_CIROPSSYMBOLS_H
+#define CLANG_CIR_DIALECT_IR_CIROPSSYMBOLS_H
+
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsSymbols.h.inc"
+
+#endif // CLANG_CIR_DIALECT_IR_CIROPSSYMBOLS_H
diff --git a/clang/include/clang/CIR/Dialect/IR/CIROpsVectorMatrix.h b/clang/include/clang/CIR/Dialect/IR/CIROpsVectorMatrix.h
new file mode 100644
index 00000000000000..62cd5259c1b502
--- /dev/null
+++ b/clang/include/clang/CIR/Dialect/IR/CIROpsVectorMatrix.h
@@ -0,0 +1,23 @@
+//===- CIROpsVectorMatrix.h - CIR ops, `VectorMatrix` partition -*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the CIR vector and matrix ops. The ops of the other
+// partitions are forward declared; include their headers, or CIRDialect.h
+// for all ops, to use them.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef CLANG_CIR_DIALECT_IR_CIROPSVECTORMATRIX_H
+#define CLANG_CIR_DIALECT_IR_CIROPSVECTORMATRIX_H
+
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsVectorMatrix.h.inc"
+
+#endif // CLANG_CIR_DIALECT_IR_CIROPSVECTORMATRIX_H
diff --git a/clang/include/clang/CIR/Dialect/IR/CIRStdOps.td b/clang/include/clang/CIR/Dialect/IR/CIRStdOps.td
index d95f109ffb812e..120bd1ba6f5a18 100644
--- a/clang/include/clang/CIR/Dialect/IR/CIRStdOps.td
+++ b/clang/include/clang/CIR/Dialect/IR/CIRStdOps.td
@@ -19,6 +19,7 @@
class CIR_StdOp<string functionName, dag args, dag res,
list<Trait> traits = [], string knownKind = "">
: CIR_Op<"std." # functionName, traits # [CIR_StdOpInterface]> {
+ let partition = CIR_BuiltinsPartition;
let arguments = !con(args, (ins FlatSymbolRefAttr:$original_fn));
let results = res;
let hasLLVMLowering = false;
diff --git a/clang/include/clang/CIR/Dialect/IR/CMakeLists.txt b/clang/include/clang/CIR/Dialect/IR/CMakeLists.txt
index 7cadc1424a82ba..a597353884c79e 100644
--- a/clang/include/clang/CIR/Dialect/IR/CMakeLists.txt
+++ b/clang/include/clang/CIR/Dialect/IR/CMakeLists.txt
@@ -5,8 +5,15 @@
# Equivalent to add_mlir_dialect(CIROps cir)
set(LLVM_TARGET_DEFINITIONS CIROps.td)
-mlir_tablegen(CIROps.h.inc -gen-op-decls -op-shard-count=${CLANG_CIR_OP_SHARD_COUNT})
-mlir_tablegen(CIROps.cpp.inc -gen-op-defs -op-shard-count=${CLANG_CIR_OP_SHARD_COUNT})
+# The op partitions of CIROps.td (see CIRDialect.td), each generated into its
+# own header and source and compiled in its own translation unit.
+set(CLANG_CIR_OP_PARTITIONS
+ Core Symbols ControlFlow Arith Builtins Math Coroutine EH VectorMatrix
+ Complex ClassMembers)
+foreach(partition ${CLANG_CIR_OP_PARTITIONS})
+ mlir_tablegen(CIROps${partition}.h.inc -gen-op-decls -op-partition=${partition})
+ mlir_tablegen(CIROps${partition}.cpp.inc -gen-op-defs -op-partition=${partition})
+endforeach()
mlir_tablegen(CIROpsTypes.h.inc -gen-typedef-decls)
mlir_tablegen(CIROpsTypes.cpp.inc -gen-typedef-defs)
mlir_tablegen(CIROpsDialect.h.inc -gen-dialect-decls)
diff --git a/clang/lib/CIR/CodeGen/CIRGenAMDGPU.cpp b/clang/lib/CIR/CodeGen/CIRGenAMDGPU.cpp
index a28449d0ce7abc..231b2e05ad7b6b 100644
--- a/clang/lib/CIR/CodeGen/CIRGenAMDGPU.cpp
+++ b/clang/lib/CIR/CodeGen/CIRGenAMDGPU.cpp
@@ -13,7 +13,7 @@
#include "CIRGenModule.h"
#include "clang/Basic/TargetOptions.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "llvm/TargetParser/Triple.h"
using namespace clang;
diff --git a/clang/lib/CIR/Dialect/IR/CIRAttrs.cpp b/clang/lib/CIR/Dialect/IR/CIRAttrs.cpp
index 5d2f9f6def5446..7f9d8b27e8fda1 100644
--- a/clang/lib/CIR/Dialect/IR/CIRAttrs.cpp
+++ b/clang/lib/CIR/Dialect/IR/CIRAttrs.cpp
@@ -11,7 +11,7 @@
//===----------------------------------------------------------------------===//
#include "mlir/Dialect/Ptr/IR/MemorySpaceInterfaces.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "mlir/IR/Attributes.h"
#include "mlir/IR/DialectImplementation.h"
@@ -1073,3 +1073,34 @@ void CIRDialect::registerAttributes() {
#include "clang/CIR/Dialect/IR/CIROpsAttributes.cpp.inc"
>();
}
+
+//===----------------------------------------------------------------------===//
+// AnnotationAttr
+//===----------------------------------------------------------------------===//
+
+LogicalResult
+cir::AnnotationAttr::verify(function_ref<InFlightDiagnostic()> emitError,
+ mlir::StringAttr name, mlir::ArrayAttr args) {
+ if (!args)
+ return success();
+ for (mlir::Attribute arg : args) {
+ if (!isa<mlir::StringAttr, mlir::IntegerAttr>(arg))
+ return emitError() << "annotation args must be StringAttr or IntegerAttr,"
+ << " got " << arg;
+ }
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// TypeInfoAttr
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::TypeInfoAttr::verify(
+ ::llvm::function_ref<::mlir::InFlightDiagnostic()> emitError,
+ ::mlir::Type type, ::mlir::ArrayAttr typeInfoData) {
+
+ if (cir::ConstRecordAttr::verify(emitError, type, typeInfoData).failed())
+ return failure();
+
+ return success();
+}
diff --git a/clang/lib/CIR/Dialect/IR/CIRDialect.cpp b/clang/lib/CIR/Dialect/IR/CIRDialect.cpp
index f38161648b3fcd..a316937e758bce 100644
--- a/clang/lib/CIR/Dialect/IR/CIRDialect.cpp
+++ b/clang/lib/CIR/Dialect/IR/CIRDialect.cpp
@@ -6,39 +6,27 @@
//
//===----------------------------------------------------------------------===//
//
-// This file implements the CIR dialect and its operations.
+// This file implements the CIR dialect. Its operations are implemented in the
+// CIROps<Partition>.cpp files, one per op partition.
//
//===----------------------------------------------------------------------===//
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
-
-#include "clang/CIR/Dialect/IR/CIRAttrs.h"
-#include "clang/CIR/Dialect/IR/CIRDataLayout.h"
-#include "clang/CIR/Dialect/IR/CIROpsEnums.h"
-#include "clang/CIR/Dialect/IR/CIRTypes.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
+#include "clang/CIR/Dialect/IR/CIROpsCore.h"
#include "mlir/IR/Attributes.h"
#include "mlir/IR/BuiltinOps.h"
#include "mlir/IR/BuiltinTypes.h"
#include "mlir/IR/DialectImplementation.h"
-#include "mlir/IR/PatternMatch.h"
-#include "mlir/IR/Value.h"
-#include "mlir/Interfaces/ControlFlowInterfaces.h"
-#include "mlir/Interfaces/FunctionImplementation.h"
#include "mlir/Support/LLVM.h"
#include "clang/Basic/TargetCXXABI.h"
#include "clang/CIR/Dialect/IR/CIROpsDialect.cpp.inc"
#include "clang/CIR/Dialect/IR/CIROpsEnums.cpp.inc"
-#include "clang/CIR/MissingFeatures.h"
-#include "llvm/ADT/SetOperations.h"
-#include "llvm/ADT/SmallSet.h"
-#include "llvm/ADT/TypeSwitch.h"
#include "llvm/Support/Casting.h"
#include "llvm/Support/LogicalResult.h"
#include "CIRDialectBytecode.h"
-#include "CIROpDefinitions.h"
using namespace mlir;
using namespace cir;
@@ -102,7 +90,7 @@ struct CIROpAsmDialectInterface : public OpAsmDialectInterface {
void cir::CIRDialect::initialize() {
registerTypes();
registerAttributes();
- registerCIRDialectOperations(this);
+ registerOperations();
addInterfaces<CIROpAsmDialectInterface>();
detail::addBytecodeInterface(this);
}
@@ -332,5543 +320,3 @@ cir::CIRDialect::verifyOperationAttribute(mlir::Operation *op,
return success();
}
-
-//===----------------------------------------------------------------------===//
-// Helpers
-//===----------------------------------------------------------------------===//
-
-// Parses one of the keywords provided in the list `keywords` and returns the
-// position of the parsed keyword in the list. If none of the keywords from the
-// list is parsed, returns -1.
-static int parseOptionalKeywordAlternative(AsmParser &parser,
- ArrayRef<llvm::StringRef> keywords) {
- for (auto en : llvm::enumerate(keywords)) {
- if (succeeded(parser.parseOptionalKeyword(en.value())))
- return en.index();
- }
- return -1;
-}
-
-namespace {
-template <typename Ty> struct EnumTraits {};
-
-#define REGISTER_ENUM_TYPE(Ty) \
- template <> struct EnumTraits<cir::Ty> { \
- static llvm::StringRef stringify(cir::Ty value) { \
- return stringify##Ty(value); \
- } \
- static unsigned getMaxEnumVal() { return cir::getMaxEnumValFor##Ty(); } \
- }
-
-REGISTER_ENUM_TYPE(GlobalLinkageKind);
-REGISTER_ENUM_TYPE(VisibilityKind);
-REGISTER_ENUM_TYPE(CallingConv);
-} // namespace
-
-/// Parse an enum from the keyword, or default to the provided default value.
-/// The return type is the enum type by default, unless overriden with the
-/// second template argument.
-template <typename EnumTy, typename RetTy = EnumTy>
-static RetTy parseOptionalCIRKeyword(AsmParser &parser, EnumTy defaultValue) {
- llvm::SmallVector<llvm::StringRef, 10> names;
- for (unsigned i = 0, e = EnumTraits<EnumTy>::getMaxEnumVal(); i <= e; ++i)
- names.push_back(EnumTraits<EnumTy>::stringify(static_cast<EnumTy>(i)));
-
- int index = parseOptionalKeywordAlternative(parser, names);
- if (index == -1)
- return static_cast<RetTy>(defaultValue);
- return static_cast<RetTy>(index);
-}
-
-/// Parse an enum from the keyword, return failure if the keyword is not found.
-template <typename EnumTy, typename RetTy = EnumTy>
-static ParseResult parseCIRKeyword(AsmParser &parser, RetTy &result) {
- llvm::SmallVector<llvm::StringRef, 10> names;
- for (unsigned i = 0, e = EnumTraits<EnumTy>::getMaxEnumVal(); i <= e; ++i)
- names.push_back(EnumTraits<EnumTy>::stringify(static_cast<EnumTy>(i)));
-
- int index = parseOptionalKeywordAlternative(parser, names);
- if (index == -1)
- return failure();
- result = static_cast<RetTy>(index);
- return success();
-}
-
-// Check if a region's termination omission is valid and, if so, creates and
-// inserts the omitted terminator into the region.
-static LogicalResult ensureRegionTerm(OpAsmParser &parser, Region ®ion,
- SMLoc errLoc) {
- Location eLoc = parser.getEncodedSourceLoc(parser.getCurrentLocation());
- OpBuilder builder(parser.getBuilder().getContext());
-
- // Insert empty block in case the region is empty to ensure the terminator
- // will be inserted
- if (region.empty())
- builder.createBlock(®ion);
-
- Block &block = region.back();
- // Region is properly terminated: nothing to do.
- if (!block.empty() && block.back().hasTrait<OpTrait::IsTerminator>())
- return success();
-
- // Check for invalid terminator omissions.
- if (!region.hasOneBlock())
- return parser.emitError(errLoc,
- "multi-block region must not omit terminator");
-
- // Terminator was omitted correctly: recreate it.
- builder.setInsertionPointToEnd(&block);
- cir::YieldOp::create(builder, eLoc);
- return success();
-}
-
-// True if the region's terminator should be omitted.
-static bool omitRegionTerm(mlir::Region &r) {
- const auto singleNonEmptyBlock = r.hasOneBlock() && !r.back().empty();
- const auto yieldsNothing = [&r]() {
- auto y = dyn_cast<cir::YieldOp>(r.back().getTerminator());
- return y && y.getArgs().empty();
- };
- return singleNonEmptyBlock && yieldsNothing();
-}
-
-// Verifies that the given operand is produced by an operation of type
-// ExpectedProducerOp.
-template <typename ExpectedProducerOp>
-static LogicalResult verifyProducedBy(Operation *op, Value operand,
- StringRef operandName) {
- Operation *producer = operand.getDefiningOp();
- if (!producer || !isa<ExpectedProducerOp>(producer))
- return op->emitOpError()
- << "operand '" << operandName << "' must be produced by '"
- << ExpectedProducerOp::getOperationName() << "'";
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// InlineKindAttr (FIXME: remove once FuncOp uses assembly format)
-//===----------------------------------------------------------------------===//
-
-ParseResult parseInlineKindAttr(OpAsmParser &parser,
- cir::InlineKindAttr &inlineKindAttr) {
- // Static list of possible inline kind keywords
- static constexpr llvm::StringRef keywords[] = {"no_inline", "always_inline",
- "inline_hint"};
-
- // Parse the inline kind keyword (optional)
- llvm::StringRef keyword;
- if (parser.parseOptionalKeyword(&keyword, keywords).failed()) {
- // Not an inline kind keyword, leave inlineKindAttr empty
- return success();
- }
-
- // Parse the enum value from the keyword
- auto inlineKindResult = ::cir::symbolizeEnum<::cir::InlineKind>(keyword);
- if (!inlineKindResult) {
- return parser.emitError(parser.getCurrentLocation(), "expected one of [")
- << llvm::join(llvm::ArrayRef(keywords), ", ")
- << "] for inlineKind, got: " << keyword;
- }
-
- inlineKindAttr =
- ::cir::InlineKindAttr::get(parser.getContext(), *inlineKindResult);
- return success();
-}
-
-void printInlineKindAttr(OpAsmPrinter &p, cir::InlineKindAttr inlineKindAttr) {
- if (inlineKindAttr) {
- p << " " << stringifyInlineKind(inlineKindAttr.getValue());
- }
-}
-
-//===----------------------------------------------------------------------===//
-// CIR Custom Parsers/Printers
-//===----------------------------------------------------------------------===//
-
-mlir::ParseResult cir::parseOmittedTerminatorRegion(mlir::OpAsmParser &parser,
- mlir::Region ®ion) {
- auto regionLoc = parser.getCurrentLocation();
- if (parser.parseRegion(region))
- return failure();
- if (ensureRegionTerm(parser, region, regionLoc).failed())
- return failure();
- return success();
-}
-
-void cir::printOmittedTerminatorRegion(mlir::OpAsmPrinter &printer,
- cir::ScopeOp &op, mlir::Region ®ion) {
- printer.printRegion(region,
- /*printEntryBlockArgs=*/false,
- /*printBlockTerminators=*/!omitRegionTerm(region));
-}
-
-//===----------------------------------------------------------------------===//
-// AllocaOp
-//===----------------------------------------------------------------------===//
-
-void cir::AllocaOp::build(mlir::OpBuilder &odsBuilder,
- mlir::OperationState &odsState, mlir::Type addr,
- llvm::StringRef name, mlir::IntegerAttr alignment) {
- odsState.addAttribute(getNameAttrName(odsState.name),
- odsBuilder.getStringAttr(name));
- if (alignment) {
- odsState.addAttribute(getAlignmentAttrName(odsState.name), alignment);
- }
- odsState.addTypes(addr);
-}
-
-cir::AllocaOp cir::getUnderlyingAlloca(mlir::Value addr) {
- mlir::Value ptr = addr;
- while (cir::CastOp castOp = ptr.getDefiningOp<cir::CastOp>()) {
- if (!castOp.isAllocaPreservingCast())
- break;
- ptr = castOp.getSrc();
- }
- return ptr.getDefiningOp<cir::AllocaOp>();
-}
-
-//===----------------------------------------------------------------------===//
-// ArrayCtor & ArrayDtor
-//===----------------------------------------------------------------------===//
-
-template <typename Op> static LogicalResult verifyArrayCtorDtor(Op op) {
- auto ptrTy = mlir::cast<cir::PointerType>(op.getAddr().getType());
- mlir::Type pointeeTy = ptrTy.getPointee();
-
- mlir::Block &body = op.getBody().front();
- if (body.getNumArguments() != 1)
- return op.emitOpError("body must have exactly one block argument");
-
- auto expectedEltPtrTy =
- mlir::dyn_cast<cir::PointerType>(body.getArgument(0).getType());
- if (!expectedEltPtrTy)
- return op.emitOpError("block argument must be a !cir.ptr type");
-
- if (op.getNumElements()) {
- auto recTy = mlir::dyn_cast<cir::RecordType>(pointeeTy);
- if (!recTy)
- return op.emitOpError(
- "when 'num_elements' is present, 'addr' must be a pointer to a "
- "!cir.struct or !cir.union type");
-
- if (expectedEltPtrTy != ptrTy)
- return op.emitOpError("when 'num_elements' is present, 'addr' type must "
- "match the block argument type");
- } else {
- auto arrayTy = mlir::dyn_cast<cir::ArrayType>(pointeeTy);
- if (!arrayTy)
- return op.emitOpError(
- "when 'num_elements' is absent, 'addr' must be a pointer to a "
- "!cir.array type");
-
- mlir::Type innerEltTy = arrayTy.getElementType();
- while (auto nested = mlir::dyn_cast<cir::ArrayType>(innerEltTy))
- innerEltTy = nested.getElementType();
-
- auto recTy = mlir::dyn_cast<cir::RecordType>(innerEltTy);
- if (!recTy)
- return op.emitOpError("the block argument type must be a pointer to a "
- "!cir.struct or !cir.union type");
-
- if (expectedEltPtrTy.getPointee() != innerEltTy)
- return op.emitOpError(
- "block argument pointee type must match the innermost array "
- "element type");
- }
-
- return success();
-}
-
-LogicalResult cir::ArrayCtor::verify() {
- if (failed(verifyArrayCtorDtor(*this)))
- return failure();
-
- mlir::Region &partialDtor = getPartialDtor();
- if (!partialDtor.empty()) {
- mlir::Block &dtorBlock = partialDtor.front();
- if (dtorBlock.getNumArguments() != 1)
- return emitOpError("partial_dtor must have exactly one block argument");
-
- auto bodyArgTy = getBody().front().getArgument(0).getType();
- if (dtorBlock.getArgument(0).getType() != bodyArgTy)
- return emitOpError("partial_dtor block argument type must match "
- "the body block argument type");
- }
- return success();
-}
-LogicalResult cir::ArrayDtor::verify() { return verifyArrayCtorDtor(*this); }
-
-//===----------------------------------------------------------------------===//
-// DeleteArrayOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::DeleteArrayOp::verify() {
- if (getDtorMayThrow() && !getElementDtorAttr())
- return emitOpError(
- "'dtor_may_throw' requires an 'element_dtor' to be present");
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// AssumeOp
-//===----------------------------------------------------------------------===//
-
-void cir::printAssumeBundle(OpAsmPrinter &p, cir::AssumeOp op,
- cir::AssumeBundleKindAttr kindAttr,
- OperandRange bundleArgs, TypeRange bundleArgTypes) {
- cir::AssumeBundleKind kind = kindAttr.getValue();
- if (kind == cir::AssumeBundleKind::None)
- return;
-
- p << " " << cir::stringifyAssumeBundleKind(kind);
- if (bundleArgs.empty())
- return;
-
- p << "(";
- p.printOperands(bundleArgs);
- p << " : ";
- llvm::interleaveComma(bundleArgTypes, p);
- p << ")";
-}
-
-ParseResult cir::parseAssumeBundle(
- OpAsmParser &p, cir::AssumeBundleKindAttr &bundleKindAttr,
- llvm::SmallVector<mlir::OpAsmParser::UnresolvedOperand, 4> &bundleArgs,
- llvm::SmallVector<mlir::Type, 1> &bundleArgTypes) {
- StringRef keyword;
- auto loc = p.getCurrentLocation();
- if (failed(p.parseOptionalKeyword(&keyword))) {
- bundleKindAttr = cir::AssumeBundleKindAttr::get(
- p.getContext(), cir::AssumeBundleKind::None);
- return success();
- }
-
- std::optional<cir::AssumeBundleKind> parsedKind =
- cir::symbolizeAssumeBundleKind(keyword);
- if (!parsedKind)
- return p.emitError(loc, "unknown assume bundle kind '") << keyword << "'";
-
- bundleKindAttr = cir::AssumeBundleKindAttr::get(p.getContext(), *parsedKind);
-
- if (p.parseOptionalLParen())
- return success();
-
- if (p.parseOperandList(bundleArgs) || p.parseColon() ||
- p.parseTypeList(bundleArgTypes) || p.parseRParen())
- return failure();
-
- return success();
-}
-
-LogicalResult cir::AssumeOp::verify() {
- cir::AssumeBundleKind kind = getBundleKind();
- size_t numArgs = getBundleArgs().size();
-
- if (kind == cir::AssumeBundleKind::None) {
- if (numArgs != 0)
- return emitOpError("unexpected bundle operands for kind 'none'");
- return success();
- }
-
- if (numArgs == 0)
- return emitOpError("expected bundle operands for kind '")
- << cir::stringifyAssumeBundleKind(kind) << "'";
-
- switch (kind) {
- case cir::AssumeBundleKind::Align:
- if (numArgs != 2 && numArgs != 3)
- return emitOpError("align bundle expects 2 or 3 operands");
- break;
- case cir::AssumeBundleKind::SeparateStorage:
- if (numArgs != 2)
- return emitOpError("separate_storage bundle expects 2 operands");
- break;
- case cir::AssumeBundleKind::Dereferenceable:
- if (numArgs != 2)
- return emitOpError("dereferenceable bundle expects 2 operands");
- break;
- default:
- break;
- }
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// LocalInitOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult
-cir::LocalInitOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
- cir::GlobalOp global = getReferencedGlobal(symbolTable);
- if (!global)
- return emitOpError("'")
- << getGlobalName() << "' does not reference a valid cir.global";
-
- if (getTls() && !global.getTlsModel())
- return emitOpError("access to global not marked thread local");
-
- if (!global.getDynamicInitGuard().has_value())
- return emitOpError("static_local attribute mismatch");
-
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// RegisterExitDtorOp
-//===----------------------------------------------------------------------===//
-
-/// Returns true if \p op is nested in the ctor region of a cir.global or
-/// cir.local_init without crossing a cir.func or a cir.register_exit_dtor.
-static bool isInVarCtorRegion(mlir::Operation *op) {
- for (mlir::Operation *parent = op->getParentOp(); parent;
- parent = parent->getParentOp()) {
- mlir::Region *ctorRegion = nullptr;
- if (auto global = mlir::dyn_cast<cir::GlobalOp>(parent))
- ctorRegion = &global.getCtorRegion();
- else if (auto localInit = mlir::dyn_cast<cir::LocalInitOp>(parent))
- ctorRegion = &localInit.getCtorRegion();
- else if (mlir::isa<cir::FuncOp, cir::RegisterExitDtorOp>(parent))
- return false;
- if (ctorRegion)
- return ctorRegion->isAncestor(op->getParentRegion());
- }
- return false;
-}
-
-LogicalResult cir::RegisterExitDtorOp::verify() {
- mlir::Block &body = getBody().front();
- if (body.without_terminator().empty())
- return emitOpError("body must destroy the object");
- if (body.getTerminator()->getNumOperands())
- return emitOpError("body must not yield a value");
- if (!isInVarCtorRegion(*this))
- return emitOpError("must be in the ctor region of a cir.global or "
- "cir.local_init");
- return success();
-}
-
-LogicalResult
-cir::RegisterExitDtorOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
- auto global = symbolTable.lookupNearestSymbolFrom<cir::GlobalOp>(
- *this, getObjectAttr());
- if (!global)
- return emitOpError("'")
- << getObject() << "' does not reference a valid cir.global";
- if (!global.getCtorRegion().empty() || !global.getDtorRegion().empty())
- return emitOpError("object must not have ctor or dtor regions");
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// ConditionOp
-//===----------------------------------------------------------------------===//
-
-//===----------------------------------
-// BranchOpTerminatorInterface Methods
-//===----------------------------------
-
-void cir::ConditionOp::getSuccessorRegions(
- ArrayRef<Attribute> operands, SmallVectorImpl<RegionSuccessor> ®ions) {
- // TODO(cir): The condition value may be folded to a constant, narrowing
- // down its list of possible successors.
-
- // Parent is a loop: condition may branch to the body, or on the false edge
- // to the per-iteration cleanup region if present, otherwise to the parent op.
- if (auto loopOp = dyn_cast<LoopOpInterface>(getOperation()->getParentOp())) {
- regions.emplace_back(&loopOp.getBody());
- if (mlir::Region *cleanup = loopOp.maybeGetCleanup())
- regions.emplace_back(cleanup);
- else
- regions.emplace_back(getOperation());
- return;
- }
-
- // Parent is an await: condition may branch to resume or suspend regions.
- auto await = cast<AwaitOp>(getOperation()->getParentOp());
- regions.emplace_back(&await.getResume());
- regions.emplace_back(&await.getSuspend());
-}
-
-MutableOperandRange
-cir::ConditionOp::getMutableSuccessorOperands(RegionSuccessor point) {
- // No values are yielded to the successor region.
- return MutableOperandRange(getOperation(), 0, 0);
-}
-
-MutableOperandRange
-cir::ResumeOp::getMutableSuccessorOperands(RegionSuccessor point) {
- // The eh_token operand is not modeled as forwarded to any successor.
- return MutableOperandRange(getOperation(), 0, 0);
-}
-
-MutableOperandRange
-cir::CoroSuspendPoint::getMutableSuccessorOperands(RegionSuccessor point) {
- // The condition operand is not forwarded to successor regions.
- return MutableOperandRange(getOperation(), 0, 0);
-}
-
-LogicalResult cir::ConditionOp::verify() {
- if (!isa<LoopOpInterface, AwaitOp>(getOperation()->getParentOp()))
- return emitOpError("condition must be within a conditional region");
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// WhileOp & ForOp
-//===----------------------------------------------------------------------===//
-
-template <typename LoopOpTy>
-static LogicalResult verifyLoopCleanup(LoopOpTy op) {
- std::optional<cir::CleanupKind> cleanupKind = op.getCleanupKind();
-
- // The cleanup kind attribute must be present exactly when a (non-empty)
- // cleanup region is present.
- if (cleanupKind.has_value() == op.getCleanup().empty())
- return op.emitOpError("cleanup kind must be present if and only if the "
- "cleanup region is non-empty");
-
- // A loop's per-iteration cleanup runs on every normal exit edge (loop exit,
- // end of iteration, break/continue), so an EH-only cleanup is meaningless.
- // Only 'normal' (exceptions disabled) and 'all' (normal + EH unwind) apply.
- if (cleanupKind == cir::CleanupKind::EH)
- return op.emitOpError("loop cleanup kind must be 'normal' or 'all', "
- "not 'eh'");
-
- return success();
-}
-
-LogicalResult cir::WhileOp::verify() { return verifyLoopCleanup(*this); }
-
-LogicalResult cir::ForOp::verify() { return verifyLoopCleanup(*this); }
-
-//===----------------------------------------------------------------------===//
-// ConstantOp
-//===----------------------------------------------------------------------===//
-
-static LogicalResult checkConstantTypes(mlir::Operation *op, mlir::Type opType,
- mlir::Attribute attrType) {
- if (isa<cir::ConstPtrAttr>(attrType)) {
- if (!mlir::isa<cir::PointerType>(opType))
- return op->emitOpError(
- "pointer constant initializing a non-pointer type");
- return success();
- }
-
- if (isa<cir::DataMemberAttr, cir::DataMemberOffsetAttr, cir::MethodAttr>(
- attrType)) {
- // More detailed type verifications are already done in
- // DataMemberAttr::verify or MethodAttr::verify. Don't need to repeat here.
- return success();
- }
-
- if (isa<cir::ZeroAttr>(attrType)) {
- if (isa<cir::RecordType, cir::ArrayType, cir::MatrixType, cir::VectorType,
- cir::ComplexType>(opType))
- return success();
- return op->emitOpError(
- "zero expects struct, array, vector, or complex type");
- }
-
- if (mlir::isa<cir::UndefAttr>(attrType)) {
- if (!mlir::isa<cir::VoidType>(opType))
- return success();
- return op->emitOpError("undef expects non-void type");
- }
-
- if (mlir::isa<cir::BoolAttr>(attrType)) {
- if (!mlir::isa<cir::BoolType>(opType))
- return op->emitOpError("result type (")
- << opType << ") must be '!cir.bool' for '" << attrType << "'";
- return success();
- }
-
- if (mlir::isa<cir::IntAttr, cir::FPAttr>(attrType)) {
- auto at = cast<TypedAttr>(attrType);
- if (at.getType() != opType) {
- return op->emitOpError("result type (")
- << opType << ") does not match value type (" << at.getType()
- << ")";
- }
- return success();
- }
-
- if (mlir::isa<cir::BlockAddrDiffAttr, cir::BlockAddrInfoAttr,
- cir::ConstArrayAttr, cir::ConstVectorAttr,
- cir::ConstComplexAttr, cir::ConstRecordAttr,
- cir::GlobalOffsetAttr, cir::GlobalViewAttr, cir::PoisonAttr,
- cir::TypeInfoAttr, cir::VTableAttr>(attrType))
- return success();
-
- assert(isa<TypedAttr>(attrType) && "What else could we be looking at here?");
- return op->emitOpError("global with type ")
- << cast<TypedAttr>(attrType).getType() << " not yet supported";
-}
-
-LogicalResult cir::ConstantOp::verify() {
- // ODS already generates checks to make sure the result type is valid. We just
- // need to additionally check that the value's attribute type is consistent
- // with the result type.
- return checkConstantTypes(getOperation(), getType(), getValue());
-}
-
-OpFoldResult cir::ConstantOp::fold(FoldAdaptor /*adaptor*/) {
- return getValue();
-}
-
-//===----------------------------------------------------------------------===//
-// CastOp
-//===----------------------------------------------------------------------===//
-
-static bool isFloatingPointCastKind(cir::CastKind kind) {
- switch (kind) {
- case cir::CastKind::floating:
- case cir::CastKind::int_to_float:
- case cir::CastKind::float_to_int:
- case cir::CastKind::float_to_bool:
- case cir::CastKind::bool_to_float:
- case cir::CastKind::float_to_complex:
- case cir::CastKind::float_complex_to_real:
- case cir::CastKind::float_complex_to_bool:
- case cir::CastKind::float_complex:
- case cir::CastKind::float_complex_to_int_complex:
- case cir::CastKind::int_complex_to_float_complex:
- return true;
- default:
- return false;
- }
-}
-
-LogicalResult cir::CastOp::verify() {
- mlir::Type resType = getType();
- mlir::Type srcType = getSrc().getType();
- cir::CastKind kind = getKind();
-
- if (getFenvAttr() && !isFloatingPointCastKind(kind))
- return emitOpError()
- << "'fenv' is only valid for floating-point cast kinds";
-
- // Verify address space casts for pointer types. given that
- // casts for within a different address space are illegal.
- auto srcPtrTy = mlir::dyn_cast<cir::PointerType>(srcType);
- auto resPtrTy = mlir::dyn_cast<cir::PointerType>(resType);
- if (srcPtrTy && resPtrTy && (kind != cir::CastKind::address_space))
- if (srcPtrTy.getAddrSpace() != resPtrTy.getAddrSpace()) {
- return emitOpError() << "result type address space does not match the "
- "address space of the operand";
- }
-
- auto srcVTy = mlir::dyn_cast<cir::VectorType>(srcType);
- auto resVTy = mlir::dyn_cast<cir::VectorType>(resType);
- if (srcVTy && resVTy) {
- if ((kind == cir::CastKind::int_to_float ||
- kind == cir::CastKind::float_to_int) &&
- srcVTy.getSize() != resVTy.getSize()) {
- return emitOpError()
- << "vector float-to-int and int-to-float casts require "
- "source and destination vectors to have the same number of "
- "elements";
- }
- // Use the element type of the vector to verify the cast kind. (Except for
- // bitcast, see below.)
- srcType = srcVTy.getElementType();
- resType = resVTy.getElementType();
- }
-
- switch (kind) {
- case cir::CastKind::int_to_bool: {
- if (!mlir::isa<cir::BoolType>(resType))
- return emitOpError() << "requires !cir.bool type for result";
- if (!mlir::isa<cir::IntType>(srcType))
- return emitOpError() << "requires !cir.int type for source";
- return success();
- }
- case cir::CastKind::ptr_to_bool: {
- if (!mlir::isa<cir::BoolType>(resType))
- return emitOpError() << "requires !cir.bool type for result";
- if (!mlir::isa<cir::PointerType>(srcType))
- return emitOpError() << "requires !cir.ptr type for source";
- return success();
- }
- case cir::CastKind::integral: {
- if (!mlir::isa<cir::IntType>(resType))
- return emitOpError() << "requires !cir.int type for result";
- if (!mlir::isa<cir::IntType>(srcType))
- return emitOpError() << "requires !cir.int type for source";
- return success();
- }
- case cir::CastKind::array_to_ptrdecay: {
- const auto arrayPtrTy = mlir::dyn_cast<cir::PointerType>(srcType);
- const auto flatPtrTy = mlir::dyn_cast<cir::PointerType>(resType);
- if (!arrayPtrTy || !flatPtrTy)
- return emitOpError() << "requires !cir.ptr type for source and result";
-
- // TODO(CIR): Make sure the AddrSpace of both types are equals
- return success();
- }
- case cir::CastKind::bitcast: {
- // Handle the pointer types first.
- auto srcPtrTy = mlir::dyn_cast<cir::PointerType>(srcType);
- auto resPtrTy = mlir::dyn_cast<cir::PointerType>(resType);
-
- if (srcPtrTy && resPtrTy) {
- return success();
- }
-
- return success();
- }
- case cir::CastKind::floating: {
- if (!mlir::isa<cir::FPTypeInterface>(srcType) ||
- !mlir::isa<cir::FPTypeInterface>(resType))
- return emitOpError() << "requires !cir.float type for source and result";
- return success();
- }
- case cir::CastKind::float_to_int: {
- if (!mlir::isa<cir::FPTypeInterface>(srcType))
- return emitOpError() << "requires !cir.float type for source";
- if (!mlir::dyn_cast<cir::IntType>(resType))
- return emitOpError() << "requires !cir.int type for result";
- return success();
- }
- case cir::CastKind::int_to_ptr: {
- if (!mlir::dyn_cast<cir::IntType>(srcType))
- return emitOpError() << "requires !cir.int type for source";
- if (!mlir::dyn_cast<cir::PointerType>(resType))
- return emitOpError() << "requires !cir.ptr type for result";
- return success();
- }
- case cir::CastKind::ptr_to_int: {
- if (!mlir::dyn_cast<cir::PointerType>(srcType))
- return emitOpError() << "requires !cir.ptr type for source";
- if (!mlir::dyn_cast<cir::IntType>(resType))
- return emitOpError() << "requires !cir.int type for result";
- return success();
- }
- case cir::CastKind::float_to_bool: {
- if (!mlir::isa<cir::FPTypeInterface>(srcType))
- return emitOpError() << "requires !cir.float type for source";
- if (!mlir::isa<cir::BoolType>(resType))
- return emitOpError() << "requires !cir.bool type for result";
- return success();
- }
- case cir::CastKind::bool_to_int: {
- if (!mlir::isa<cir::BoolType>(srcType))
- return emitOpError() << "requires !cir.bool type for source";
- if (!mlir::isa<cir::IntType>(resType))
- return emitOpError() << "requires !cir.int type for result";
- return success();
- }
- case cir::CastKind::int_to_float: {
- if (!mlir::isa<cir::IntType>(srcType))
- return emitOpError() << "requires !cir.int type for source";
- if (!mlir::isa<cir::FPTypeInterface>(resType))
- return emitOpError() << "requires !cir.float type for result";
- return success();
- }
- case cir::CastKind::bool_to_float: {
- if (!mlir::isa<cir::BoolType>(srcType))
- return emitOpError() << "requires !cir.bool type for source";
- if (!mlir::isa<cir::FPTypeInterface>(resType))
- return emitOpError() << "requires !cir.float type for result";
- return success();
- }
- case cir::CastKind::address_space: {
- auto srcPtrTy = mlir::dyn_cast<cir::PointerType>(srcType);
- auto resPtrTy = mlir::dyn_cast<cir::PointerType>(resType);
- if (!srcPtrTy || !resPtrTy)
- return emitOpError() << "requires !cir.ptr type for source and result";
- if (srcPtrTy.getPointee() != resPtrTy.getPointee())
- return emitOpError() << "requires two types differ in addrspace only";
- return success();
- }
- case cir::CastKind::float_to_complex: {
- if (!mlir::isa<cir::FPTypeInterface>(srcType))
- return emitOpError() << "requires !cir.float type for source";
- auto resComplexTy = mlir::dyn_cast<cir::ComplexType>(resType);
- if (!resComplexTy)
- return emitOpError() << "requires !cir.complex type for result";
- if (srcType != resComplexTy.getElementType())
- return emitOpError() << "requires source type match result element type";
- return success();
- }
- case cir::CastKind::int_to_complex: {
- if (!mlir::isa<cir::IntType>(srcType))
- return emitOpError() << "requires !cir.int type for source";
- auto resComplexTy = mlir::dyn_cast<cir::ComplexType>(resType);
- if (!resComplexTy)
- return emitOpError() << "requires !cir.complex type for result";
- if (srcType != resComplexTy.getElementType())
- return emitOpError() << "requires source type match result element type";
- return success();
- }
- case cir::CastKind::float_complex_to_real: {
- auto srcComplexTy = mlir::dyn_cast<cir::ComplexType>(srcType);
- if (!srcComplexTy)
- return emitOpError() << "requires !cir.complex type for source";
- if (!mlir::isa<cir::FPTypeInterface>(resType))
- return emitOpError() << "requires !cir.float type for result";
- if (srcComplexTy.getElementType() != resType)
- return emitOpError() << "requires source element type match result type";
- return success();
- }
- case cir::CastKind::int_complex_to_real: {
- auto srcComplexTy = mlir::dyn_cast<cir::ComplexType>(srcType);
- if (!srcComplexTy)
- return emitOpError() << "requires !cir.complex type for source";
- if (!mlir::isa<cir::IntType>(resType))
- return emitOpError() << "requires !cir.int type for result";
- if (srcComplexTy.getElementType() != resType)
- return emitOpError() << "requires source element type match result type";
- return success();
- }
- case cir::CastKind::float_complex_to_bool: {
- auto srcComplexTy = mlir::dyn_cast<cir::ComplexType>(srcType);
- if (!srcComplexTy || !srcComplexTy.isFloatingPointComplex())
- return emitOpError()
- << "requires floating point !cir.complex type for source";
- if (!mlir::isa<cir::BoolType>(resType))
- return emitOpError() << "requires !cir.bool type for result";
- return success();
- }
- case cir::CastKind::int_complex_to_bool: {
- auto srcComplexTy = mlir::dyn_cast<cir::ComplexType>(srcType);
- if (!srcComplexTy || !srcComplexTy.isIntegerComplex())
- return emitOpError()
- << "requires floating point !cir.complex type for source";
- if (!mlir::isa<cir::BoolType>(resType))
- return emitOpError() << "requires !cir.bool type for result";
- return success();
- }
- case cir::CastKind::float_complex: {
- auto srcComplexTy = mlir::dyn_cast<cir::ComplexType>(srcType);
- if (!srcComplexTy || !srcComplexTy.isFloatingPointComplex())
- return emitOpError()
- << "requires floating point !cir.complex type for source";
- auto resComplexTy = mlir::dyn_cast<cir::ComplexType>(resType);
- if (!resComplexTy || !resComplexTy.isFloatingPointComplex())
- return emitOpError()
- << "requires floating point !cir.complex type for result";
- return success();
- }
- case cir::CastKind::float_complex_to_int_complex: {
- auto srcComplexTy = mlir::dyn_cast<cir::ComplexType>(srcType);
- if (!srcComplexTy || !srcComplexTy.isFloatingPointComplex())
- return emitOpError()
- << "requires floating point !cir.complex type for source";
- auto resComplexTy = mlir::dyn_cast<cir::ComplexType>(resType);
- if (!resComplexTy || !resComplexTy.isIntegerComplex())
- return emitOpError() << "requires integer !cir.complex type for result";
- return success();
- }
- case cir::CastKind::int_complex: {
- auto srcComplexTy = mlir::dyn_cast<cir::ComplexType>(srcType);
- if (!srcComplexTy || !srcComplexTy.isIntegerComplex())
- return emitOpError() << "requires integer !cir.complex type for source";
- auto resComplexTy = mlir::dyn_cast<cir::ComplexType>(resType);
- if (!resComplexTy || !resComplexTy.isIntegerComplex())
- return emitOpError() << "requires integer !cir.complex type for result";
- return success();
- }
- case cir::CastKind::int_complex_to_float_complex: {
- auto srcComplexTy = mlir::dyn_cast<cir::ComplexType>(srcType);
- if (!srcComplexTy || !srcComplexTy.isIntegerComplex())
- return emitOpError() << "requires integer !cir.complex type for source";
- auto resComplexTy = mlir::dyn_cast<cir::ComplexType>(resType);
- if (!resComplexTy || !resComplexTy.isFloatingPointComplex())
- return emitOpError()
- << "requires floating point !cir.complex type for result";
- return success();
- }
- case cir::CastKind::member_ptr_to_bool: {
- if (!mlir::isa<cir::DataMemberType, cir::MethodType>(srcType))
- return emitOpError()
- << "requires !cir.data_member or !cir.method type for source";
- if (!mlir::isa<cir::BoolType>(resType))
- return emitOpError() << "requires !cir.bool type for result";
- return success();
- }
- }
- llvm_unreachable("Unknown CastOp kind?");
-}
-
-static bool isIntOrBoolCast(cir::CastOp op) {
- auto kind = op.getKind();
- return kind == cir::CastKind::bool_to_int ||
- kind == cir::CastKind::int_to_bool || kind == cir::CastKind::integral;
-}
-
-static bool isCirFunctionPointerType(mlir::Type ty) {
- const auto ptrTy = mlir::dyn_cast<cir::PointerType>(ty);
- return ptrTy && mlir::isa<cir::FuncType>(ptrTy.getPointee());
-}
-
-static Value tryFoldCastChain(cir::CastOp op) {
- cir::CastOp head = op, tail = op;
-
- while (op) {
- if (!isIntOrBoolCast(op))
- break;
- head = op;
- op = head.getSrc().getDefiningOp<cir::CastOp>();
- }
-
- if (head != tail) {
- // if bool_to_int -> ... -> int_to_bool: take the bool
- // as we had it was before all casts
- if (head.getKind() == cir::CastKind::bool_to_int &&
- tail.getKind() == cir::CastKind::int_to_bool)
- return head.getSrc();
-
- // if int_to_bool -> ... -> int_to_bool: take the result
- // of the first one, as no other casts (and ext casts as well)
- // don't change the first result
- if (head.getKind() == cir::CastKind::int_to_bool &&
- tail.getKind() == cir::CastKind::int_to_bool)
- return head.getResult();
-
- return {};
- }
-
- // Bitcast round-trip on function pointers: T0 -> T1 -> T0 (e.g. no-proto
- // redeclaration vs. actual prototype). Restrict to function pointers so
- // other pointer bitcast chains are unchanged.
- if (tail.getKind() == cir::CastKind::bitcast) {
- auto *inner = tail.getSrc().getDefiningOp();
- if (inner && isCirFunctionPointerType(tail.getType())) {
- auto innerCast = mlir::dyn_cast<cir::CastOp>(inner);
- if (innerCast && innerCast.getKind() == cir::CastKind::bitcast &&
- innerCast.getSrc().getType() == tail.getType() &&
- innerCast.getType() == tail.getSrc().getType()) {
- return innerCast.getSrc();
- }
- }
- }
-
- return {};
-}
-
-OpFoldResult cir::CastOp::fold(FoldAdaptor adaptor) {
- if (mlir::isa_and_present<cir::PoisonAttr>(adaptor.getSrc())) {
- // Propagate poison value
- return cir::PoisonAttr::get(getContext(), getType());
- }
-
- // Propagate Undef value
- if (mlir::isa_and_present<cir::UndefAttr>(adaptor.getSrc()))
- return cir::UndefAttr::get(getType());
-
- if (getSrc().getType() == getType()) {
- switch (getKind()) {
- case cir::CastKind::integral:
- case cir::CastKind::bitcast:
- case cir::CastKind::address_space:
- case cir::CastKind::float_complex:
- case cir::CastKind::int_complex: {
- return getSrc();
- }
- default:
- return {};
- }
- }
-
- // Handle cases where a chain of casts cancel out.
- Value result = tryFoldCastChain(*this);
- if (result)
- return result;
-
- // Handle simple constant casts.
- if (auto srcConst = getSrc().getDefiningOp<cir::ConstantOp>()) {
- switch (getKind()) {
- case cir::CastKind::integral: {
- mlir::Type srcTy = getSrc().getType();
- // Don't try to fold vector casts for now.
- assert(mlir::isa<cir::VectorType>(srcTy) ==
- mlir::isa<cir::VectorType>(getType()));
- if (mlir::isa<cir::VectorType>(srcTy))
- break;
-
- auto srcIntTy = mlir::cast<cir::IntType>(srcTy);
- auto dstIntTy = mlir::cast<cir::IntType>(getType());
- auto constIntAttr = srcConst.getValueAttr<cir::IntAttr>();
- if (!constIntAttr)
- return {};
-
- APInt srcValue = constIntAttr.getValue();
- APInt newVal = srcIntTy.isSigned()
- ? srcValue.sextOrTrunc(dstIntTy.getWidth())
- : srcValue.zextOrTrunc(dstIntTy.getWidth());
- return cir::IntAttr::get(dstIntTy, newVal);
- }
- default:
- break;
- }
- }
- return {};
-}
-
-//===----------------------------------------------------------------------===//
-// BuiltinIntCastOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::BuiltinIntCastOp::verify() {
- mlir::Type srcType = getSrc().getType();
- mlir::Type resType = getType();
-
- auto srcCirInt = mlir::dyn_cast<cir::IntType>(srcType);
- auto resCirInt = mlir::dyn_cast<cir::IntType>(resType);
-
- // One side must be a CIR integer the other must be a builtin
- // integer or index type.
- if (static_cast<bool>(srcCirInt) == static_cast<bool>(resCirInt))
- return emitOpError()
- << "requires exactly one '!cir.int' operand or result; the other "
- "must be a builtin integer or 'index' type";
-
- mlir::Type builtinType = srcCirInt ? resType : srcType;
- if (!mlir::isa<mlir::IntegerType, mlir::IndexType>(builtinType))
- return emitOpError() << "requires a builtin integer or 'index' type on the "
- "non-CIR side";
-
- // The cast preserves bit width. 'index' has no fixed width, so only check
- // when the builtin side is a fixed-width integer.
- if (auto builtinInt = mlir::dyn_cast<mlir::IntegerType>(builtinType)) {
- cir::IntType cirInt = srcCirInt ? srcCirInt : resCirInt;
- if (cirInt.getWidth() != builtinInt.getWidth())
- return emitOpError()
- << "requires the CIR and builtin integer types to have the same "
- "width; use 'cir.cast' for width conversions";
- }
-
- return success();
-}
-
-OpFoldResult cir::BuiltinIntCastOp::fold(FoldAdaptor adaptor) {
- // Fold: builtin_int_cast(builtin_int_cast(x)) -> x
- // Inner source type must match the cast's result type.
- if (auto inner = getSrc().getDefiningOp<cir::BuiltinIntCastOp>())
- if (inner.getSrc().getType() == getType())
- return inner.getSrc();
- return {};
-}
-
-//===----------------------------------------------------------------------===//
-// CallOp
-//===----------------------------------------------------------------------===//
-
-mlir::OperandRange cir::CallOp::getArgOperands() {
- if (isIndirect())
- return getArgs().drop_front(1);
- return getArgs();
-}
-
-mlir::MutableOperandRange cir::CallOp::getArgOperandsMutable() {
- mlir::MutableOperandRange args = getArgsMutable();
- if (isIndirect())
- return args.slice(1, args.size() - 1);
- return args;
-}
-
-mlir::Value cir::CallOp::getIndirectCall() {
- assert(isIndirect());
- return getOperand(0);
-}
-
-/// Return the operand at index 'i'.
-Value cir::CallOp::getArgOperand(unsigned i) {
- if (isIndirect())
- ++i;
- return getOperand(i);
-}
-
-/// Return the number of operands.
-unsigned cir::CallOp::getNumArgOperands() {
- if (isIndirect())
- return this->getOperation()->getNumOperands() - 1;
- return this->getOperation()->getNumOperands();
-}
-
-static mlir::ParseResult
-parseTryCallDestinations(mlir::OpAsmParser &parser,
- mlir::OperationState &result) {
- mlir::Block *normalDestSuccessor;
- if (parser.parseSuccessor(normalDestSuccessor))
- return mlir::failure();
-
- if (parser.parseComma())
- return mlir::failure();
-
- mlir::Block *unwindDestSuccessor;
- if (parser.parseSuccessor(unwindDestSuccessor))
- return mlir::failure();
-
- result.addSuccessors(normalDestSuccessor);
- result.addSuccessors(unwindDestSuccessor);
- return mlir::success();
-}
-
-/// Reject an attribute in an explicit attribute dictionary whose value is not
-/// the attribute class expected for its name. Where an op declares the
-/// attribute it is stored as a property, so a value of the wrong class would
-/// be dropped without a diagnostic.
-static ParseResult checkEffectAttrKinds(mlir::OpAsmParser &parser,
- llvm::SMLoc loc,
- const mlir::NamedAttrList &attrs) {
- if (mlir::Attribute effects =
- attrs.get(CIRDialect::getMemoryEffectsAttrName()))
- if (!mlir::isa<cir::MemoryEffectsAttr>(effects))
- return parser.emitError(loc, "attribute '")
- << CIRDialect::getMemoryEffectsAttrName()
- << "' must be a #cir.memory_effects attribute";
-
- if (mlir::Attribute uwtable = attrs.get(CIRDialect::getUwtableAttrName()))
- if (!mlir::isa<cir::UnwindTableKindAttr>(uwtable))
- return parser.emitError(loc, "attribute '")
- << CIRDialect::getUwtableAttrName()
- << "' must be a #cir.uwtable attribute";
-
- if (mlir::Attribute framePointer =
- attrs.get(CIRDialect::getFramePointerAttrName()))
- if (!mlir::isa<cir::FramePointerKindAttr>(framePointer))
- return parser.emitError(loc, "attribute '")
- << CIRDialect::getFramePointerAttrName()
- << "' must be a #cir.frame_pointer attribute";
-
- for (llvm::StringRef name :
- {CIRDialect::getNoUnwindAttrName(), CIRDialect::getWillReturnAttrName(),
- CIRDialect::getMustProgressAttrName()})
- if (mlir::Attribute flag = attrs.get(name))
- if (!mlir::isa<mlir::UnitAttr>(flag))
- return parser.emitError(loc, "attribute '")
- << name << "' must be a unit attribute";
-
- return mlir::success();
-}
-
-static mlir::ParseResult parseCallCommon(mlir::OpAsmParser &parser,
- mlir::OperationState &result,
- bool hasDestinationBlocks = false) {
- llvm::SmallVector<mlir::OpAsmParser::UnresolvedOperand, 4> ops;
- llvm::SMLoc opsLoc;
- mlir::FlatSymbolRefAttr calleeAttr;
-
- // If we cannot parse a string callee, it means this is an indirect call.
- if (!parser
- .parseOptionalAttribute(calleeAttr, CIRDialect::getCalleeAttrName(),
- result.attributes)
- .has_value()) {
- OpAsmParser::UnresolvedOperand indirectVal;
- // Do not resolve right now, since we need to figure out the type
- if (parser.parseOperand(indirectVal).failed())
- return failure();
- ops.push_back(indirectVal);
- }
-
- if (parser.parseLParen())
- return mlir::failure();
-
- opsLoc = parser.getCurrentLocation();
- if (parser.parseOperandList(ops))
- return mlir::failure();
- if (parser.parseRParen())
- return mlir::failure();
-
- if (hasDestinationBlocks &&
- parseTryCallDestinations(parser, result).failed()) {
- return ::mlir::failure();
- }
-
- if (parser.parseOptionalKeyword("cc").succeeded()) {
- cir::CallingConv callingConv;
- if (parser.parseLParen().failed())
- return failure();
- if (parseCIRKeyword<cir::CallingConv>(parser, callingConv).failed())
- return parser.emitError(parser.getCurrentLocation(),
- "unknown calling convention");
- if (parser.parseRParen().failed())
- return failure();
- result.addAttribute(
- CIRDialect::getCallingConvAttrName(),
- cir::CallingConvAttr::get(parser.getContext(), callingConv));
- }
-
- if (parser.parseOptionalKeyword("musttail").succeeded())
- result.addAttribute(CIRDialect::getMustTailAttrName(),
- mlir::UnitAttr::get(parser.getContext()));
-
- if (parser.parseOptionalKeyword("nothrow").succeeded())
- result.addAttribute(CIRDialect::getNoThrowAttrName(),
- mlir::UnitAttr::get(parser.getContext()));
-
- if (parser.parseOptionalKeyword("nounwind").succeeded())
- result.addAttribute(CIRDialect::getNoUnwindAttrName(),
- mlir::UnitAttr::get(parser.getContext()));
-
- if (parser.parseOptionalKeyword("willreturn").succeeded())
- result.addAttribute(CIRDialect::getWillReturnAttrName(),
- mlir::UnitAttr::get(parser.getContext()));
-
- llvm::SMLoc attrsLoc = parser.getCurrentLocation();
- if (parser.parseOptionalAttrDict(result.attributes))
- return ::mlir::failure();
-
- if (checkEffectAttrKinds(parser, attrsLoc, result.attributes).failed())
- return ::mlir::failure();
-
- if (parser.parseColon())
- return ::mlir::failure();
-
- SmallVector<Type> argTypes;
- SmallVector<DictionaryAttr> argAttrs;
- SmallVector<Type> resultTypes;
- SmallVector<DictionaryAttr> resultAttrs;
- if (call_interface_impl::parseFunctionSignature(parser, argTypes, argAttrs,
- resultTypes, resultAttrs))
- return mlir::failure();
-
- if (resultTypes.size() > 1 || resultAttrs.size() > 1)
- return parser.emitError(
- parser.getCurrentLocation(),
- "functions with multiple return types are not supported");
-
- result.addTypes(resultTypes);
-
- if (parser.resolveOperands(ops, argTypes, opsLoc, result.operands))
- return mlir::failure();
-
- if (!resultAttrs.empty() && resultAttrs[0])
- result.addAttribute(
- CIRDialect::getResAttrsAttrName(),
- mlir::ArrayAttr::get(parser.getContext(), {resultAttrs[0]}));
-
- // ArrayAttr requires a vector of 'Attribute', so we have to do the conversion
- // here into a separate collection.
- llvm::SmallVector<Attribute> convertedArgAttrs;
- bool argAttrsEmpty = true;
-
- llvm::transform(argAttrs, std::back_inserter(convertedArgAttrs),
- [&](DictionaryAttr da) -> mlir::Attribute {
- if (da)
- argAttrsEmpty = false;
- return da;
- });
-
- if (!argAttrsEmpty) {
- llvm::ArrayRef argAttrsRef = convertedArgAttrs;
- if (!calleeAttr) {
- // Fixup for indirect calls, which get an extra entry in the 'args' for
- // the indirect type, which doesn't get attributes.
- argAttrsRef = argAttrsRef.drop_front();
- }
- result.addAttribute(CIRDialect::getArgAttrsAttrName(),
- mlir::ArrayAttr::get(parser.getContext(), argAttrsRef));
- }
-
- return mlir::success();
-}
-
-static void printCallCommon(mlir::Operation *op,
- mlir::FlatSymbolRefAttr calleeSym,
- mlir::Value indirectCallee,
- mlir::OpAsmPrinter &printer, bool isNothrow,
- ArrayAttr argAttrs, ArrayAttr resAttrs,
- mlir::Block *normalDest = nullptr,
- mlir::Block *unwindDest = nullptr) {
- printer << ' ';
-
- auto callLikeOp = mlir::cast<cir::CIRCallOpInterface>(op);
- auto ops = callLikeOp.getArgOperands();
-
- if (calleeSym) {
- // Direct calls
- printer.printAttributeWithoutType(calleeSym);
- } else {
- // Indirect calls
- assert(indirectCallee);
- printer << indirectCallee;
- }
-
- printer << "(" << ops << ")";
-
- if (normalDest) {
- assert(unwindDest && "expected two successors");
- auto tryCall = cast<cir::TryCallOp>(op);
- printer << ' ' << tryCall.getNormalDest();
- printer << ",";
- printer << ' ';
- printer << tryCall.getUnwindDest();
- }
-
- cir::CallingConv callingConv = callLikeOp.getCallingConv();
- if (callingConv != cir::CallingConv::C)
- printer << " cc(" << stringifyCallingConv(callingConv) << ")";
-
- if (op->hasAttr(CIRDialect::getMustTailAttrName()))
- printer << " musttail";
-
- if (isNothrow)
- printer << " nothrow";
-
- if (op->hasAttr(CIRDialect::getNoUnwindAttrName()))
- printer << " nounwind";
-
- if (op->hasAttr(CIRDialect::getWillReturnAttrName()))
- printer << " willreturn";
-
- llvm::StringRef elidedAttrs[] = {
- CIRDialect::getCalleeAttrName(),
- CIRDialect::getCallingConvAttrName(),
- CIRDialect::getMustTailAttrName(),
- CIRDialect::getNoThrowAttrName(),
- CIRDialect::getNoUnwindAttrName(),
- CIRDialect::getWillReturnAttrName(),
- CIRDialect::getOperandSegmentSizesAttrName(),
- "res_attrs",
- "arg_attrs",
- };
- // TODO: Split inherent and discardable attribute printing instead of
- // materializing a single dictionary that mixes the two storage classes.
- llvm::SmallVector<mlir::NamedAttribute> attrs;
- for (mlir::NamedAttribute attr : op->getDiscardableAttrs())
- if (!llvm::is_contained(elidedAttrs, attr.getName()))
- attrs.push_back(attr);
- op->getName().walkInherentAttrs(op, [&](llvm::StringRef name,
- mlir::Attribute &attr) {
- if (!llvm::is_contained(elidedAttrs, name))
- attrs.emplace_back(mlir::StringAttr::get(op->getContext(), name), attr);
- });
- llvm::sort(attrs, [](mlir::NamedAttribute lhs, mlir::NamedAttribute rhs) {
- return lhs.getName().strref() < rhs.getName().strref();
- });
- printer.printOptionalAttrDict(attrs);
- printer << " : ";
- if (calleeSym || !argAttrs) {
- call_interface_impl::printFunctionSignature(
- printer, op->getOperands().getTypes(), argAttrs,
- /*isVariadic=*/false, op->getResultTypes(), resAttrs);
- } else {
- // indirect function calls use an 'arg' type for the type of its indirect
- // argument. However, we don't store a similar attribute collection. In
- // order to make `printFunctionSignature` have the attributes line up, we
- // have to make a 'shimmed' copy of the attributes that have a blank set of
- // attributes for the indirect argument.
- llvm::SmallVector<Attribute> shimmedArgAttrs;
- shimmedArgAttrs.push_back(mlir::DictionaryAttr::get(op->getContext(), {}));
- shimmedArgAttrs.append(argAttrs.begin(), argAttrs.end());
- call_interface_impl::printFunctionSignature(
- printer, op->getOperands().getTypes(),
- mlir::ArrayAttr::get(op->getContext(), shimmedArgAttrs),
- /*isVariadic=*/false, op->getResultTypes(), resAttrs);
- }
-}
-
-mlir::ParseResult cir::CallOp::parse(mlir::OpAsmParser &parser,
- mlir::OperationState &result) {
- return parseCallCommon(parser, result);
-}
-
-void cir::CallOp::print(mlir::OpAsmPrinter &p) {
- mlir::Value indirectCallee = isIndirect() ? getIndirectCall() : nullptr;
- printCallCommon(*this, getCalleeAttr(), indirectCallee, p, getNothrow(),
- getArgAttrsAttr(), getResAttrsAttr());
-}
-
-static LogicalResult
-verifyCallCommInSymbolUses(mlir::Operation *op,
- SymbolTableCollection &symbolTable) {
- auto callIf = cast<cir::CIRCallOpInterface>(op);
-
- // An indirect call is checked against the function type its callee pointer
- // points to, and a direct call against its callee's declaration.
- cir::FuncType fnType;
- bool hasPrototype = true;
- auto fnAttr =
- op->getAttrOfType<FlatSymbolRefAttr>(CIRDialect::getCalleeAttrName());
- if (!fnAttr) {
- if (op->getNumOperands() == 0)
- return op->emitOpError("indirect call requires a callee operand");
- mlir::Type calleeTy = callIf.getIndirectCall().getType();
- if (auto ptrTy = mlir::dyn_cast<cir::PointerType>(calleeTy))
- fnType = mlir::dyn_cast<cir::FuncType>(ptrTy.getPointee());
- if (!fnType)
- return op->emitOpError()
- << "indirect callee must be a pointer to a function, but has type "
- << calleeTy;
- } else {
- auto fn = symbolTable.lookupNearestSymbolFrom<cir::FuncOp>(op, fnAttr);
- if (!fn)
- return op->emitOpError() << "'" << fnAttr.getValue()
- << "' does not reference a valid function";
- fnType = fn.getFunctionType();
- hasPrototype = !fn.getNoProto();
- }
-
- // Verify that the operand and result types match the callee. Operands are
- // not checked for a direct call to a function declared no_proto.
- if (hasPrototype) {
- unsigned numCallOperands = callIf.getNumArgOperands();
- unsigned numFnOpOperands = fnType.getNumInputs();
-
- if (!fnType.isVarArg() && numCallOperands != numFnOpOperands)
- return op->emitOpError("incorrect number of operands for callee");
- if (fnType.isVarArg() && numCallOperands < numFnOpOperands)
- return op->emitOpError("too few operands for callee");
-
- for (unsigned i = 0, e = numFnOpOperands; i != e; ++i)
- if (callIf.getArgOperand(i).getType() != fnType.getInput(i))
- return op->emitOpError("operand type mismatch: expected operand type ")
- << fnType.getInput(i) << ", but provided "
- << callIf.getArgOperand(i).getType() << " for operand number "
- << i;
- }
-
- assert(!cir::MissingFeatures::opCallCallConv());
-
- // Void function must not return any results.
- if (fnType.hasVoidReturn() && op->getNumResults() != 0)
- return op->emitOpError("callee returns void but call has results");
-
- // Non-void function calls must return exactly one result.
- if (!fnType.hasVoidReturn() && op->getNumResults() != 1)
- return op->emitOpError("incorrect number of results for callee");
-
- // Parent function and return value types must match.
- if (!fnType.hasVoidReturn() &&
- op->getResultTypes().front() != fnType.getReturnType()) {
- return op->emitOpError("result type mismatch: expected ")
- << fnType.getReturnType() << ", but provided "
- << op->getResult(0).getType();
- }
-
- return mlir::success();
-}
-
-LogicalResult
-cir::CallOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
- return verifyCallCommInSymbolUses(*this, symbolTable);
-}
-
-//===----------------------------------------------------------------------===//
-// TryCallOp
-//===----------------------------------------------------------------------===//
-
-mlir::OperandRange cir::TryCallOp::getArgOperands() {
- if (isIndirect())
- return getArgs().drop_front(1);
- return getArgs();
-}
-
-mlir::MutableOperandRange cir::TryCallOp::getArgOperandsMutable() {
- mlir::MutableOperandRange args = getArgsMutable();
- if (isIndirect())
- return args.slice(1, args.size() - 1);
- return args;
-}
-
-mlir::Value cir::TryCallOp::getIndirectCall() {
- assert(isIndirect());
- return getOperand(0);
-}
-
-/// Return the operand at index 'i'.
-Value cir::TryCallOp::getArgOperand(unsigned i) {
- if (isIndirect())
- ++i;
- return getOperand(i);
-}
-
-/// Return the number of operands.
-unsigned cir::TryCallOp::getNumArgOperands() {
- if (isIndirect())
- return this->getOperation()->getNumOperands() - 1;
- return this->getOperation()->getNumOperands();
-}
-
-LogicalResult
-cir::TryCallOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
- return verifyCallCommInSymbolUses(*this, symbolTable);
-}
-
-mlir::ParseResult cir::TryCallOp::parse(mlir::OpAsmParser &parser,
- mlir::OperationState &result) {
- return parseCallCommon(parser, result, /*hasDestinationBlocks=*/true);
-}
-
-void cir::TryCallOp::print(::mlir::OpAsmPrinter &p) {
- mlir::Value indirectCallee = isIndirect() ? getIndirectCall() : nullptr;
- printCallCommon(*this, getCalleeAttr(), indirectCallee, p, getNothrow(),
- getArgAttrsAttr(), getResAttrsAttr(), getNormalDest(),
- getUnwindDest());
-}
-
-//===----------------------------------------------------------------------===//
-// ReturnOp
-//===----------------------------------------------------------------------===//
-
-static mlir::LogicalResult checkReturnAndFunction(cir::ReturnOp op,
- cir::FuncOp function) {
- // ReturnOps currently only have a single optional operand.
- if (op.getNumOperands() > 1)
- return op.emitOpError() << "expects at most 1 return operand";
-
- // Ensure returned type matches the function signature.
- auto expectedTy = function.getFunctionType().getReturnType();
- auto actualTy =
- (op.getNumOperands() == 0 ? cir::VoidType::get(op.getContext())
- : op.getOperand(0).getType());
- if (actualTy != expectedTy)
- return op.emitOpError() << "returns " << actualTy
- << " but enclosing function returns " << expectedTy;
-
- return mlir::success();
-}
-
-mlir::LogicalResult cir::ReturnOp::verify() {
- // Returns can be present in multiple different scopes, get the
- // wrapping function and start from there.
- auto *fnOp = getOperation()->getParentOp();
- while (!isa<cir::FuncOp>(fnOp))
- fnOp = fnOp->getParentOp();
-
- // Make sure return types match function return type.
- if (checkReturnAndFunction(*this, cast<cir::FuncOp>(fnOp)).failed())
- return failure();
-
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// IfOp
-//===----------------------------------------------------------------------===//
-
-ParseResult cir::IfOp::parse(OpAsmParser &parser, OperationState &result) {
- // create the regions for 'then'.
- result.regions.reserve(2);
- Region *thenRegion = result.addRegion();
- Region *elseRegion = result.addRegion();
-
- mlir::Builder &builder = parser.getBuilder();
- OpAsmParser::UnresolvedOperand cond;
- Type boolType = cir::BoolType::get(builder.getContext());
-
- if (parser.parseOperand(cond) ||
- parser.resolveOperand(cond, boolType, result.operands))
- return failure();
-
- // Parse 'then' region.
- mlir::SMLoc parseThenLoc = parser.getCurrentLocation();
- if (parser.parseRegion(*thenRegion, /*arguments=*/{}, /*argTypes=*/{}))
- return failure();
-
- if (ensureRegionTerm(parser, *thenRegion, parseThenLoc).failed())
- return failure();
-
- // If we find an 'else' keyword, parse the 'else' region.
- if (!parser.parseOptionalKeyword("else")) {
- mlir::SMLoc parseElseLoc = parser.getCurrentLocation();
- if (parser.parseRegion(*elseRegion, /*arguments=*/{}, /*argTypes=*/{}))
- return failure();
- if (ensureRegionTerm(parser, *elseRegion, parseElseLoc).failed())
- return failure();
- }
-
- // Parse the optional attribute list.
- if (parser.parseOptionalAttrDict(result.attributes))
- return failure();
- return success();
-}
-
-void cir::IfOp::print(OpAsmPrinter &p) {
- p << " " << getCondition() << " ";
- mlir::Region &thenRegion = this->getThenRegion();
- p.printRegion(thenRegion,
- /*printEntryBlockArgs=*/false,
- /*printBlockTerminators=*/!omitRegionTerm(thenRegion));
-
- // Print the 'else' regions if it exists and has a block.
- mlir::Region &elseRegion = this->getElseRegion();
- if (!elseRegion.empty()) {
- p << " else ";
- p.printRegion(elseRegion,
- /*printEntryBlockArgs=*/false,
- /*printBlockTerminators=*/!omitRegionTerm(elseRegion));
- }
-
- p.printOptionalAttrDict(
- getOperation()->getDiscardableAttrDictionary().getValue());
-}
-
-/// Default callback for IfOp builders.
-void cir::buildTerminatedBody(OpBuilder &builder, Location loc) {
- // add cir.yield to end of the block
- cir::YieldOp::create(builder, loc);
-}
-
-/// Given the region at `index`, or the parent operation if `index` is None,
-/// return the successor regions. These are the regions that may be selected
-/// during the flow of control. `operands` is a set of optional attributes that
-/// correspond to a constant value for each operand, or null if that operand is
-/// not a constant.
-void cir::IfOp::getSuccessorRegions(mlir::RegionBranchPoint point,
- SmallVectorImpl<RegionSuccessor> ®ions) {
- // The `then` and the `else` region branch back to the parent operation.
- if (!point.isParent()) {
- regions.emplace_back(getOperation());
- return;
- }
-
- // Don't consider the else region if it is empty.
- Region *elseRegion = &this->getElseRegion();
- if (elseRegion->empty())
- elseRegion = nullptr;
-
- // If the condition isn't constant, both regions may be executed.
- regions.push_back(RegionSuccessor(&getThenRegion()));
- if (elseRegion)
- regions.push_back(RegionSuccessor(elseRegion));
- else
- regions.emplace_back(getOperation());
-}
-
-void cir::IfOp::build(OpBuilder &builder, OperationState &result, Value cond,
- bool withElseRegion, BuilderCallbackRef thenBuilder,
- BuilderCallbackRef elseBuilder) {
- assert(thenBuilder && "the builder callback for 'then' must be present");
- result.addOperands(cond);
-
- OpBuilder::InsertionGuard guard(builder);
- Region *thenRegion = result.addRegion();
- builder.createBlock(thenRegion);
- thenBuilder(builder, result.location);
-
- Region *elseRegion = result.addRegion();
- if (!withElseRegion)
- return;
-
- builder.createBlock(elseRegion);
- elseBuilder(builder, result.location);
-}
-
-//===----------------------------------------------------------------------===//
-// ScopeOp
-//===----------------------------------------------------------------------===//
-
-void cir::ScopeOp::build(
- OpBuilder &builder, OperationState &result,
- function_ref<void(OpBuilder &, Type &, Location)> scopeBuilder) {
- assert(scopeBuilder && "the builder callback for 'then' must be present");
-
- OpBuilder::InsertionGuard guard(builder);
- Region *scopeRegion = result.addRegion();
- builder.createBlock(scopeRegion);
- assert(!cir::MissingFeatures::opScopeCleanupRegion());
-
- mlir::Type yieldTy;
- scopeBuilder(builder, yieldTy, result.location);
-
- if (yieldTy)
- result.addTypes(TypeRange{yieldTy});
-}
-
-void cir::ScopeOp::build(
- OpBuilder &builder, OperationState &result,
- function_ref<void(OpBuilder &, Location)> scopeBuilder) {
- assert(scopeBuilder && "the builder callback for 'then' must be present");
- OpBuilder::InsertionGuard guard(builder);
- Region *scopeRegion = result.addRegion();
- builder.createBlock(scopeRegion);
- assert(!cir::MissingFeatures::opScopeCleanupRegion());
- scopeBuilder(builder, result.location);
-}
-
-LogicalResult cir::ScopeOp::verify() {
- if (getRegion().empty()) {
- return emitOpError() << "cir.scope must not be empty since it should "
- "include at least an implicit cir.yield ";
- }
-
- mlir::Block &lastBlock = getRegion().back();
- if (lastBlock.empty() || !lastBlock.mightHaveTerminator() ||
- !lastBlock.getTerminator()->hasTrait<OpTrait::IsTerminator>())
- return emitOpError() << "last block of cir.scope must be terminated";
- return success();
-}
-
-LogicalResult cir::ScopeOp::fold(FoldAdaptor /*adaptor*/,
- SmallVectorImpl<OpFoldResult> &results) {
- // Only fold "trivial" scopes: a single block containing only a `cir.yield`.
- if (!getRegion().hasOneBlock())
- return failure();
- Block &block = getRegion().front();
- if (block.getOperations().size() != 1)
- return failure();
-
- auto yield = dyn_cast<cir::YieldOp>(block.front());
- if (!yield)
- return failure();
-
- // Only fold when the scope produces a value.
- if (getNumResults() != 1 || yield.getNumOperands() != 1)
- return failure();
-
- results.push_back(yield.getOperand(0));
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// CleanupScopeOp
-//===----------------------------------------------------------------------===//
-
-static bool isRedundantBeforeReturn(mlir::Region &cleanupRegion) {
- for (mlir::Block &block : cleanupRegion) {
- for (mlir::Operation &op : block) {
- if (isa<cir::YieldOp, cir::LifetimeEndOp, cir::StackRestoreOp>(op))
- continue;
- // The stack restore reloads the pointer saved before the VLA.
- auto loadOp = dyn_cast<cir::LoadOp>(op);
- if (loadOp && loadOp.getResult().hasOneUse() &&
- isa<cir::StackRestoreOp>(*loadOp.getResult().getUsers().begin()))
- continue;
- return false;
- }
- }
- return true;
-}
-
-LogicalResult cir::CleanupScopeOp::verify() {
- // If the cleanup contains a musttail call, it must be a cleanup that can be
- // skipped on return (such as a lifetime end or a stack restore). Other
- // cleanups must never contain musttail calls.
- cir::CallOp mustTailCall;
- getBodyRegion().walk([&](cir::CallOp callOp) {
- if (!callOp.getMusttail())
- return WalkResult::advance();
- mustTailCall = callOp;
- return WalkResult::interrupt();
- });
- if (!mustTailCall)
- return success();
-
- if (isRedundantBeforeReturn(getCleanupRegion()))
- return success();
-
- InFlightDiagnostic diag =
- emitOpError("cleanup is not redundant before a return, so it cannot be "
- "skipped by a musttail call");
- diag.attachNote(mustTailCall.getLoc()) << "musttail call is here";
- return diag;
-}
-
-void cir::CleanupScopeOp::getSuccessorRegions(
- mlir::RegionBranchPoint point, SmallVectorImpl<RegionSuccessor> ®ions) {
- // The cleanup can also run from the middle of the body, on an unwind out of
- // a call or on a cir.return, cir.break, cir.continue or cir.goto. None of
- // these is a branch point, so the cleanup region is a successor of the
- // parent and of every body exit to stand in for them.
- if (point.isParent()) {
- regions.reserve(regions.size() + 2);
- regions.emplace_back(&getBodyRegion());
- regions.emplace_back(&getCleanupRegion());
- return;
- }
-
- mlir::Operation *term = point.getTerminatorPredecessorOrNull();
- if (term->getParentRegion() == &getCleanupRegion()) {
- regions.emplace_back(getOperation());
- return;
- }
-
- regions.emplace_back(&getCleanupRegion());
-
- // cir.resume unwinds out of the body, and every other terminator is a
- // normal exit. An exit that the cleanup kind does not cover can also leave
- // the operation directly.
- cir::CleanupKindAttr kind = getCleanupKindAttr();
- if (isa<cir::ResumeOp>(term) ? !kind.isEH() : !kind.isNormal())
- regions.emplace_back(getOperation());
-}
-
-LogicalResult cir::CleanupScopeOp::canonicalize(CleanupScopeOp op,
- PatternRewriter &rewriter) {
- auto isRegionTrivial = [](Region ®ion) {
- assert(!region.empty() && "CleanupScopeOp regions must not be empty");
- if (!region.hasOneBlock())
- return false;
- Block &block = llvm::getSingleElement(region);
- return llvm::hasSingleElement(block) &&
- isa<cir::YieldOp>(llvm::getSingleElement(block));
- };
-
- Region &body = op.getBodyRegion();
- Region &cleanup = op.getCleanupRegion();
-
- // An EH-only cleanup scope with an empty body can never trigger its cleanup
- // region — there are no operations in the body that could throw. Erase it.
- if (op.getCleanupKind() == CleanupKind::EH && isRegionTrivial(body)) {
- rewriter.eraseOp(op);
- return success();
- }
-
- // A cleanup scope with a trivial cleanup region has no cleanup to perform.
- // Inline the body into the parent block and erase the scope.
- if (!isRegionTrivial(cleanup) || !body.hasOneBlock())
- return failure();
-
- Block &bodyBlock = body.front();
- if (!isa<cir::YieldOp>(bodyBlock.getTerminator()))
- return failure();
-
- Operation *yield = bodyBlock.getTerminator();
- rewriter.inlineBlockBefore(&bodyBlock, op);
- rewriter.eraseOp(yield);
- rewriter.eraseOp(op);
- return success();
-}
-
-void cir::CleanupScopeOp::build(
- OpBuilder &builder, OperationState &result, CleanupKind cleanupKind,
- function_ref<void(OpBuilder &, Location)> bodyBuilder,
- function_ref<void(OpBuilder &, Location)> cleanupBuilder) {
- result.addAttribute(getCleanupKindAttrName(result.name),
- CleanupKindAttr::get(builder.getContext(), cleanupKind));
-
- OpBuilder::InsertionGuard guard(builder);
-
- // Build body region.
- Region *bodyRegion = result.addRegion();
- builder.createBlock(bodyRegion);
- if (bodyBuilder)
- bodyBuilder(builder, result.location);
-
- // Build cleanup region.
- Region *cleanupRegion = result.addRegion();
- builder.createBlock(cleanupRegion);
- if (cleanupBuilder)
- cleanupBuilder(builder, result.location);
-}
-
-//===----------------------------------------------------------------------===//
-// BrOp
-//===----------------------------------------------------------------------===//
-
-/// Merges blocks connected by a unique unconditional branch.
-///
-/// ^bb0: ^bb0:
-/// ... ...
-/// cir.br ^bb1 => ...
-/// ^bb1: cir.return
-/// ...
-/// cir.return
-LogicalResult cir::BrOp::canonicalize(BrOp op, PatternRewriter &rewriter) {
- Block *src = op->getBlock();
- Block *dst = op.getDest();
-
- // Do not fold self-loops.
- if (src == dst)
- return failure();
-
- // Only merge when this is the unique edge between the blocks.
- if (src->getNumSuccessors() != 1 || dst->getSinglePredecessor() != src)
- return failure();
-
- // Don't merge blocks that start with LabelOp or IndirectBrOp.
- // This is to avoid merging blocks that have an indirect predecessor.
- if (isa<cir::LabelOp, cir::IndirectBrOp>(dst->front()))
- return failure();
-
- // An operand that is an argument of the destination itself (possible in an
- // unreachable cycle) would be replaced by itself, leaving its other uses
- // dangling once the destination is erased.
- if (llvm::any_of(op.getDestOperands(), [&](Value operand) {
- auto arg = dyn_cast<BlockArgument>(operand);
- return arg && arg.getOwner() == dst;
- }))
- return failure();
-
- // Copy the operands out: erasing the branch frees its operand storage.
- SmallVector<Value> operands(op.getDestOperands());
- rewriter.eraseOp(op);
- rewriter.mergeBlocks(dst, src, operands);
- return success();
-}
-
-mlir::SuccessorOperands cir::BrOp::getSuccessorOperands(unsigned index) {
- assert(index == 0 && "invalid successor index");
- return mlir::SuccessorOperands(getDestOperandsMutable());
-}
-
-Block *cir::BrOp::getSuccessorForOperands(ArrayRef<Attribute>) {
- return getDest();
-}
-
-//===----------------------------------------------------------------------===//
-// IndirectBrCondOp
-//===----------------------------------------------------------------------===//
-
-mlir::SuccessorOperands
-cir::IndirectBrOp::getSuccessorOperands(unsigned index) {
- assert(index < getNumSuccessors() && "invalid successor index");
- return mlir::SuccessorOperands(getSuccOperandsMutable()[index]);
-}
-
-ParseResult cir::parseIndirectBrOpSucessors(
- OpAsmParser &parser, Type &flagType,
- SmallVectorImpl<Block *> &succOperandBlocks,
- SmallVectorImpl<SmallVector<OpAsmParser::UnresolvedOperand>> &succOperands,
- SmallVectorImpl<SmallVector<Type>> &succOperandsTypes) {
- if (failed(parser.parseCommaSeparatedList(
- OpAsmParser::Delimiter::Square,
- [&]() {
- Block *destination = nullptr;
- SmallVector<OpAsmParser::UnresolvedOperand> operands;
- SmallVector<Type> operandTypes;
-
- if (parser.parseSuccessor(destination).failed())
- return failure();
-
- if (succeeded(parser.parseOptionalLParen())) {
- if (failed(parser.parseOperandList(
- operands, OpAsmParser::Delimiter::None)) ||
- failed(parser.parseColonTypeList(operandTypes)) ||
- failed(parser.parseRParen()))
- return failure();
- }
- succOperandBlocks.push_back(destination);
- succOperands.emplace_back(operands);
- succOperandsTypes.emplace_back(operandTypes);
- return success();
- },
- "successor blocks")))
- return failure();
- return success();
-}
-
-void cir::printIndirectBrOpSucessors(OpAsmPrinter &p, cir::IndirectBrOp op,
- Type flagType, SuccessorRange succs,
- OperandRangeRange succOperands,
- const TypeRangeRange &succOperandsTypes) {
- p << "[";
- llvm::interleave(
- llvm::zip(succs, succOperands),
- [&](auto i) {
- p.printNewline();
- p.printSuccessorAndUseList(std::get<0>(i), std::get<1>(i));
- },
- [&] { p << ','; });
- if (!succOperands.empty())
- p.printNewline();
- p << "]";
-}
-
-//===----------------------------------------------------------------------===//
-// BrCondOp
-//===----------------------------------------------------------------------===//
-
-mlir::SuccessorOperands cir::BrCondOp::getSuccessorOperands(unsigned index) {
- assert(index < getNumSuccessors() && "invalid successor index");
- return SuccessorOperands(index == 0 ? getDestOperandsTrueMutable()
- : getDestOperandsFalseMutable());
-}
-
-Block *cir::BrCondOp::getSuccessorForOperands(ArrayRef<Attribute> operands) {
- if (IntegerAttr condAttr = dyn_cast_if_present<IntegerAttr>(operands.front()))
- return condAttr.getValue().isOne() ? getDestTrue() : getDestFalse();
- return nullptr;
-}
-
-//===----------------------------------------------------------------------===//
-// CaseOp
-//===----------------------------------------------------------------------===//
-
-void cir::CaseOp::build(OpBuilder &builder, OperationState &result,
- ArrayAttr value, CaseOpKind kind,
- OpBuilder::InsertPoint &insertPoint) {
- OpBuilder::InsertionGuard guardSwitch(builder);
- result.addAttribute("value", value);
- result.getOrAddProperties<Properties>().kind =
- cir::CaseOpKindAttr::get(builder.getContext(), kind);
- Region *caseRegion = result.addRegion();
- builder.createBlock(caseRegion);
-
- insertPoint = builder.saveInsertionPoint();
-}
-
-//===----------------------------------------------------------------------===//
-// SwitchOp
-//===----------------------------------------------------------------------===//
-
-void cir::SwitchOp::build(OpBuilder &builder, OperationState &result,
- Value cond, BuilderOpStateCallbackRef switchBuilder) {
- assert(switchBuilder && "the builder callback for regions must be present");
- OpBuilder::InsertionGuard guardSwitch(builder);
- Region *switchRegion = result.addRegion();
- builder.createBlock(switchRegion);
- result.addOperands({cond});
- switchBuilder(builder, result.location, result);
-}
-
-void cir::SwitchOp::collectCases(llvm::SmallVectorImpl<CaseOp> &cases) {
- walk<mlir::WalkOrder::PreOrder>([&](mlir::Operation *op) {
- // Don't walk in nested switch op.
- if (isa<cir::SwitchOp>(op) && op != *this)
- return WalkResult::skip();
-
- if (auto caseOp = dyn_cast<cir::CaseOp>(op))
- cases.push_back(caseOp);
-
- return WalkResult::advance();
- });
-}
-
-bool cir::SwitchOp::isSimpleForm(llvm::SmallVectorImpl<CaseOp> &cases) {
- collectCases(cases);
-
- if (getBody().empty())
- return false;
-
- if (!isa<YieldOp>(getBody().front().back()))
- return false;
-
- if (!llvm::all_of(getBody().front(),
- [](Operation &op) { return isa<CaseOp, YieldOp>(op); }))
- return false;
-
- return llvm::all_of(cases, [this](CaseOp op) {
- return op->getParentOfType<SwitchOp>() == *this;
- });
-}
-
-//===----------------------------------------------------------------------===//
-// SwitchFlatOp
-//===----------------------------------------------------------------------===//
-
-void cir::SwitchFlatOp::build(OpBuilder &builder, OperationState &result,
- Value value, Block *defaultDestination,
- ValueRange defaultOperands,
- ArrayRef<APInt> caseValues,
- BlockRange caseDestinations,
- ArrayRef<ValueRange> caseOperands) {
-
- std::vector<mlir::Attribute> caseValuesAttrs;
- for (const APInt &val : caseValues)
- caseValuesAttrs.push_back(cir::IntAttr::get(value.getType(), val));
- mlir::ArrayAttr attrs = ArrayAttr::get(builder.getContext(), caseValuesAttrs);
-
- build(builder, result, value, defaultOperands, caseOperands, attrs,
- defaultDestination, caseDestinations);
-}
-
-SuccessorOperands cir::SwitchFlatOp::getSuccessorOperands(unsigned index) {
- assert(index < getNumSuccessors() && "invalid successor index");
- if (index == 0)
- return SuccessorOperands(getDefaultOperandsMutable());
- return SuccessorOperands(getCaseOperandsMutable()[index - 1]);
-}
-
-Block *
-cir::SwitchFlatOp::getSuccessorForOperands(ArrayRef<Attribute> operands) {
- auto cond = dyn_cast_if_present<cir::IntAttr>(operands.front());
- if (!cond)
- return nullptr;
- for (auto [value, dest] : llvm::zip(getCaseValues(), getCaseDestinations())) {
- const APInt &caseValue = cast<cir::IntAttr>(value).getValue();
- if (caseValue == cond.getValue())
- return dest;
- }
- return getDefaultDestination();
-}
-
-/// <cases> ::= `[` (case (`,` case )* )? `]`
-/// <case> ::= integer `:` bb-id (`(` ssa-use-and-type-list `)`)?
-ParseResult cir::parseSwitchFlatOpCases(
- OpAsmParser &parser, Type flagType, mlir::ArrayAttr &caseValues,
- SmallVectorImpl<Block *> &caseDestinations,
- SmallVectorImpl<llvm::SmallVector<OpAsmParser::UnresolvedOperand>>
- &caseOperands,
- SmallVectorImpl<llvm::SmallVector<Type>> &caseOperandTypes) {
- if (failed(parser.parseLSquare()))
- return failure();
- if (succeeded(parser.parseOptionalRSquare()))
- return success();
- llvm::SmallVector<mlir::Attribute> values;
-
- auto parseCase = [&]() {
- int64_t value = 0;
- if (failed(parser.parseInteger(value)))
- return failure();
-
- values.push_back(cir::IntAttr::get(flagType, value));
-
- Block *destination;
- llvm::SmallVector<OpAsmParser::UnresolvedOperand> operands;
- llvm::SmallVector<Type> operandTypes;
- if (parser.parseColon() || parser.parseSuccessor(destination))
- return failure();
- if (!parser.parseOptionalLParen()) {
- if (parser.parseOperandList(operands, OpAsmParser::Delimiter::None,
- /*allowResultNumber=*/false) ||
- parser.parseColonTypeList(operandTypes) || parser.parseRParen())
- return failure();
- }
- caseDestinations.push_back(destination);
- caseOperands.emplace_back(operands);
- caseOperandTypes.emplace_back(operandTypes);
- return success();
- };
- if (failed(parser.parseCommaSeparatedList(parseCase)))
- return failure();
-
- caseValues = ArrayAttr::get(flagType.getContext(), values);
-
- return parser.parseRSquare();
-}
-
-void cir::printSwitchFlatOpCases(OpAsmPrinter &p, cir::SwitchFlatOp op,
- Type flagType, mlir::ArrayAttr caseValues,
- SuccessorRange caseDestinations,
- OperandRangeRange caseOperands,
- const TypeRangeRange &caseOperandTypes) {
- p << '[';
- p.printNewline();
- if (!caseValues) {
- p << ']';
- return;
- }
-
- size_t index = 0;
- llvm::interleave(
- llvm::zip(caseValues, caseDestinations),
- [&](auto i) {
- p << " ";
- mlir::Attribute a = std::get<0>(i);
- p << mlir::cast<cir::IntAttr>(a).getValue();
- p << ": ";
- p.printSuccessorAndUseList(std::get<1>(i), caseOperands[index++]);
- },
- [&] {
- p << ',';
- p.printNewline();
- });
- p.printNewline();
- p << ']';
-}
-
-//===----------------------------------------------------------------------===//
-// GlobalOp
-//===----------------------------------------------------------------------===//
-
-static ParseResult parseConstantValue(OpAsmParser &parser,
- mlir::Attribute &valueAttr) {
- NamedAttrList attr;
- return parser.parseAttribute(valueAttr, "value", attr);
-}
-
-static void printConstant(OpAsmPrinter &p, Attribute value) {
- p.printAttribute(value);
-}
-
-mlir::LogicalResult cir::GlobalOp::verify() {
- // A function is not an object, so it cannot be the type of a global. A
- // global that holds a function's address carries a pointer type instead.
- if (mlir::isa<cir::FuncType>(getSymType()))
- return emitOpError("global type cannot be a function type");
-
- // Verify that the initial value, if present, is either a unit attribute or
- // an attribute CIR supports.
- if (getInitialValue().has_value()) {
- if (checkConstantTypes(getOperation(), getSymType(), *getInitialValue())
- .failed())
- return failure();
- }
-
- // The initial value can't be recomputed from the type once the ctor is
- // lowered (e.g. a null member pointer becomes -1 after CXXABILowering).
- if (!getCtorRegion().empty() && !getInitialValue().has_value())
- return emitOpError(
- "cannot have a constructor region without an initial value");
-
- std::optional<DynamicInitInfoAttr> info = getDynamicInitInfo();
-
- if ((getDynamicInitGuard().has_value()) && (!info || info->getLocal()) &&
- (!getCtorRegion().empty() || !getDtorRegion().empty()))
- return emitOpError(
- "Cannot have a static-local global-op with a constructor or "
- "destructor, they require in-function initialization via LocalInitOp");
-
- // CIRGen emits 'dynamic_init_guard' and 'dynamic_init_info' together and
- // they are only meaningful together: the guard drives lowering, which reads
- // the info. Require both or neither so malformed .cir can carry neither a
- // guard without info nor a dangling info nothing will read.
- if (getDynamicInitGuard().has_value() != getDynamicInitInfo().has_value())
- return emitOpError("'dynamic_init_guard' and 'dynamic_init_info' must be "
- "present together");
-
- if (getTlsRefs()) {
- // 'Unordered' TLS globals (variable template instantiations) legitimately
- // carry both: they can't use the shared __tls_init guard, so CIRGen also
- // gives them their own dynamic-init guard, mangled the same as
- // 'tls_refs's own guard name. 'Ordered' TLS globals share __tls_init and
- // have no per-variable guard name, so a dynamic-init guard there would be
- // a mismatch.
- if (getDynamicInitGuard().has_value() && !getTlsRefs()->getGuardName())
- return emitOpError("cannot have a dynamic-init guard combined with "
- "ordered tls references");
- if (!getTlsModel())
- return emitOpError("'tls_refs' only valid for tls");
- }
-
- if (getAliasee().has_value()) {
- if (getInitialValue().has_value() || !getCtorRegion().empty() ||
- !getDtorRegion().empty())
- return emitOpError("global alias shall not have an initializer or "
- "constructor/destructor regions");
- }
-
- // TODO(CIR): Many other checks for properties that haven't been upstreamed
- // yet.
-
- return success();
-}
-
-void cir::GlobalOp::build(
- OpBuilder &odsBuilder, OperationState &odsState, llvm::StringRef sym_name,
- mlir::Type sym_type, bool isConstant,
- mlir::ptr::MemorySpaceAttrInterface addrSpace,
- cir::GlobalLinkageKind linkage,
- function_ref<void(OpBuilder &, Location)> ctorBuilder,
- function_ref<void(OpBuilder &, Location)> dtorBuilder) {
- odsState.addAttribute(getSymNameAttrName(odsState.name),
- odsBuilder.getStringAttr(sym_name));
- odsState.addAttribute(getSymTypeAttrName(odsState.name),
- mlir::TypeAttr::get(sym_type));
- auto &properties = odsState.getOrAddProperties<cir::GlobalOp::Properties>();
- properties.setConstant(isConstant);
-
- addrSpace = normalizeDefaultAddressSpace(addrSpace);
- if (addrSpace)
- odsState.addAttribute(getAddrSpaceAttrName(odsState.name), addrSpace);
-
- cir::GlobalLinkageKindAttr linkageAttr =
- cir::GlobalLinkageKindAttr::get(odsBuilder.getContext(), linkage);
- odsState.addAttribute(getLinkageAttrName(odsState.name), linkageAttr);
-
- Region *ctorRegion = odsState.addRegion();
- if (ctorBuilder) {
- odsBuilder.createBlock(ctorRegion);
- ctorBuilder(odsBuilder, odsState.location);
- }
-
- Region *dtorRegion = odsState.addRegion();
- if (dtorBuilder) {
- odsBuilder.createBlock(dtorRegion);
- dtorBuilder(odsBuilder, odsState.location);
- }
-}
-
-/// Given the region at `index`, or the parent operation if `index` is None,
-/// return the successor regions. These are the regions that may be selected
-/// during the flow of control. `operands` is a set of optional attributes that
-/// correspond to a constant value for each operand, or null if that operand is
-/// not a constant.
-void cir::GlobalOp::getSuccessorRegions(
- mlir::RegionBranchPoint point, SmallVectorImpl<RegionSuccessor> ®ions) {
- // The `ctor` and `dtor` regions always branch back to the parent operation.
- if (!point.isParent()) {
- regions.emplace_back(getOperation());
- return;
- }
-
- // Don't consider the ctor region if it is empty.
- Region *ctorRegion = &this->getCtorRegion();
- if (ctorRegion->empty())
- ctorRegion = nullptr;
-
- // Don't consider the dtor region if it is empty.
- Region *dtorRegion = &this->getDtorRegion();
- if (dtorRegion->empty())
- dtorRegion = nullptr;
-
- // If the condition isn't constant, both regions may be executed.
- if (ctorRegion)
- regions.push_back(RegionSuccessor(ctorRegion));
- if (dtorRegion)
- regions.push_back(RegionSuccessor(dtorRegion));
-}
-
-static void printComdatName(OpAsmPrinter &p, StringAttr comdat) {
- if (!comdat)
- return;
- p << "comdat";
- if (!comdat.getValue().empty())
- p << "(\"" << comdat.getValue() << "\")";
-}
-
-void cir::printComdatName(OpAsmPrinter &p, cir::GlobalOp op,
- StringAttr comdat) {
- ::printComdatName(p, comdat);
-}
-
-ParseResult cir::parseComdatName(OpAsmParser &parser, StringAttr &comdatAttr) {
- if (parser.parseOptionalKeyword("comdat").failed())
- return success();
- std::string comdatKey;
- if (succeeded(parser.parseOptionalLParen())) {
- if (parser.parseString(&comdatKey).failed() ||
- parser.parseRParen().failed())
- return failure();
- }
- comdatAttr = parser.getBuilder().getStringAttr(comdatKey);
- return success();
-}
-
-void cir::printGlobalOpTypeAndInitialValue(OpAsmPrinter &p, cir::GlobalOp op,
- TypeAttr type, Attribute initAttr,
- mlir::Region &ctorRegion,
- mlir::Region &dtorRegion) {
- auto printType = [&]() { p << ": " << type; };
- // Aliases are definitions but they have no initial value or ctor/dtor; the
- // assembly prints them like declarations (`: type`).
- if (op.isDeclaration() || op.getAliasee()) {
- printType();
- return;
- }
-
- p << "= ";
- if (initAttr)
- printConstant(p, initAttr);
-
- if (!ctorRegion.empty()) {
- p << " ctor ";
- p.printRegion(ctorRegion,
- /*printEntryBlockArgs=*/false,
- /*printBlockTerminators=*/false);
- }
-
- if (!dtorRegion.empty()) {
- p << " dtor ";
- p.printRegion(dtorRegion,
- /*printEntryBlockArgs=*/false,
- /*printBlockTerminators=*/false);
- }
-}
-
-ParseResult cir::parseGlobalOpTypeAndInitialValue(OpAsmParser &parser,
- TypeAttr &typeAttr,
- Attribute &initialValueAttr,
- mlir::Region &ctorRegion,
- mlir::Region &dtorRegion) {
- mlir::Type opTy;
- if (parser.parseOptionalEqual().failed()) {
- // Absence of equal means a declaration, so we need to parse the type.
- // cir.global @a : !cir.int<s, 32>
- if (parser.parseColonType(opTy))
- return failure();
- } else {
- // Parse constant with initializer, examples:
- // cir.global @y = 3.400000e+00 : f32
- // cir.global @rgb = #cir.const_array<[...] : !cir.array<!s8i x 3>>
- if (parseConstantValue(parser, initialValueAttr).failed())
- return failure();
-
- assert(mlir::isa<mlir::TypedAttr>(initialValueAttr) &&
- "Non-typed attrs shouldn't appear here.");
- opTy = mlir::cast<mlir::TypedAttr>(initialValueAttr).getType();
-
- // Parse constructor, example:
- // cir.global @rgb = #cir.zero : type ctor { ... }
- if (!parser.parseOptionalKeyword("ctor")) {
- auto parseLoc = parser.getCurrentLocation();
- if (parser.parseRegion(ctorRegion, /*arguments=*/{}, /*argTypes=*/{}))
- return failure();
- if (ensureRegionTerm(parser, ctorRegion, parseLoc).failed())
- return failure();
- }
-
- // Parse destructor, example:
- // dtor { ... }
- if (!parser.parseOptionalKeyword("dtor")) {
- auto parseLoc = parser.getCurrentLocation();
- if (parser.parseRegion(dtorRegion, /*arguments=*/{}, /*argTypes=*/{}))
- return failure();
- if (ensureRegionTerm(parser, dtorRegion, parseLoc).failed())
- return failure();
- }
- }
-
- typeAttr = TypeAttr::get(opTy);
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// GetGlobalOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult
-cir::GetGlobalOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
- // Verify that the result type underlying pointer type matches the type of
- // the referenced cir.global or cir.func op.
- mlir::Operation *op =
- symbolTable.lookupNearestSymbolFrom(*this, getNameAttr());
- if (op == nullptr || !(isa<GlobalOp>(op) || isa<FuncOp>(op)))
- return emitOpError("'")
- << getName()
- << "' does not reference a valid cir.global or cir.func";
-
- mlir::Type symTy;
- mlir::ptr::MemorySpaceAttrInterface symAddrSpaceAttr{};
- if (auto g = dyn_cast<GlobalOp>(op)) {
- symTy = g.getSymType();
- symAddrSpaceAttr = g.getAddrSpaceAttr();
- // Verify that for thread local global access, the global needs to
- // be marked with tls bits.
- if (getTls() && !g.getTlsModel())
- return emitOpError("access to global not marked thread local");
-
- // Verify that the static_local attribute on GetGlobalOp matches the
- // dynamic_init_guard attribute on GlobalOp. GetGlobalOp uses a UnitAttr,
- // GlobalOp uses DynamicInitGuardAttr. Both should be present, or neither.
- bool getGlobalIsStaticLocal = getStaticLocal();
- bool globalIsStaticLocal =
- g.getDynamicInitGuard().has_value() &&
- (!g.getDynamicInitInfo() || g.getDynamicInitInfo()->getLocal());
-
- if (getGlobalIsStaticLocal != globalIsStaticLocal &&
- !getOperation()->getParentOfType<cir::GlobalOp>())
- return emitOpError("static_local attribute mismatch");
- } else if (auto f = dyn_cast<FuncOp>(op)) {
- symTy = f.getFunctionType();
- } else {
- llvm_unreachable("Unexpected operation for GetGlobalOp");
- }
-
- auto resultType = dyn_cast<PointerType>(getAddr().getType());
- if (!resultType || symTy != resultType.getPointee())
- return emitOpError("result type pointee type '")
- << resultType.getPointee() << "' does not match type " << symTy
- << " of the global @" << getName();
-
- // A function address may also be taken in the program address space.
- auto targetAS =
- dyn_cast_if_present<TargetAddressSpaceAttr>(resultType.getAddrSpace());
- bool isProgramAS =
- isa<FuncOp>(op) && targetAS &&
- targetAS.getValue() ==
- CIRDataLayout(getOperation()->getParentOfType<ModuleOp>())
- .getProgramAddressSpace();
- if (!isProgramAS && symAddrSpaceAttr != resultType.getAddrSpace()) {
- return emitOpError()
- << "result type address space does not match the address "
- "space of the global @"
- << getName();
- }
-
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// VTableAddrPointOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult
-cir::VTableAddrPointOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
- StringRef name = getName();
-
- // Verify that the result type underlying pointer type matches the type of
- // the referenced cir.global.
- auto op =
- symbolTable.lookupNearestSymbolFrom<cir::GlobalOp>(*this, getNameAttr());
- if (!op)
- return emitOpError("'")
- << name << "' does not reference a valid cir.global";
- std::optional<mlir::Attribute> init = op.getInitialValue();
- if (!init)
- return success();
- if (!isa<cir::VTableAttr>(*init))
- return emitOpError("Expected #cir.vtable in initializer for global '")
- << name << "'";
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// VTTAddrPointOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult
-cir::VTTAddrPointOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
- // VTT ptr is not coming from a symbol.
- if (!getName())
- return success();
- StringRef name = *getName();
-
- // Verify that the result type underlying pointer type matches the type of
- // the referenced cir.global op.
- auto op =
- symbolTable.lookupNearestSymbolFrom<cir::GlobalOp>(*this, getNameAttr());
- if (!op)
- return emitOpError("'")
- << name << "' does not reference a valid cir.global";
- std::optional<mlir::Attribute> init = op.getInitialValue();
- if (!init)
- return success();
- if (!isa<cir::ConstArrayAttr>(*init))
- return emitOpError(
- "Expected constant array in initializer for global VTT '")
- << name << "'";
- return success();
-}
-
-LogicalResult cir::VTTAddrPointOp::verify() {
- // The operation uses either a symbol or a value to operate, but not both
- if (getName() && getSymAddr())
- return emitOpError("should use either a symbol or value, but not both");
-
- // If not a symbol, stick with the concrete type used for getSymAddr.
- if (getSymAddr())
- return success();
-
- mlir::Type resultType = getAddr().getType();
- mlir::Type resTy = cir::PointerType::get(
- cir::PointerType::get(cir::VoidType::get(getContext())));
-
- if (resultType != resTy)
- return emitOpError("result type must be ")
- << resTy << ", but provided result type is " << resultType;
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// FuncOp
-//===----------------------------------------------------------------------===//
-
-/// Returns the name used for the linkage attribute. This *must* correspond to
-/// the name of the attribute in ODS.
-static llvm::StringRef getLinkageAttrNameString() { return "linkage"; }
-
-/// Declared cir.func attributes with no dedicated syntax, which print in and
-/// parse from the attribute dictionary instead.
-static bool isFuncAttrInDict(llvm::StringRef name) {
- return name == CIRDialect::getMemoryEffectsAttrName() ||
- name == CIRDialect::getUwtableAttrName() ||
- name == CIRDialect::getFramePointerAttrName();
-}
-
-void cir::FuncOp::build(OpBuilder &builder, OperationState &result,
- StringRef name, FuncType type,
- GlobalLinkageKind linkage, CallingConv callingConv) {
- result.addRegion();
- result.addAttribute(getSymNameAttrName(result.name),
- builder.getStringAttr(name));
- result.addAttribute(getFunctionTypeAttrName(result.name),
- TypeAttr::get(type));
- result.addAttribute(
- getLinkageAttrNameString(),
- GlobalLinkageKindAttr::get(builder.getContext(), linkage));
- result.addAttribute(getCallingConvAttrName(result.name),
- CallingConvAttr::get(builder.getContext(), callingConv));
- // A function starts out written in source-level types, and CallConvLowering
- // rewrites it to the target ABI later.
- result.addAttribute(getRequiresCallConvLoweringAttrName(result.name),
- builder.getUnitAttr());
-}
-
-//===----------------------------------------------------------------------===//
-// AnnotationAttr
-//===----------------------------------------------------------------------===//
-
-LogicalResult
-cir::AnnotationAttr::verify(function_ref<InFlightDiagnostic()> emitError,
- mlir::StringAttr name, mlir::ArrayAttr args) {
- if (!args)
- return success();
- for (mlir::Attribute arg : args) {
- if (!isa<mlir::StringAttr, mlir::IntegerAttr>(arg))
- return emitError() << "annotation args must be StringAttr or IntegerAttr,"
- << " got " << arg;
- }
- return success();
-}
-
-ParseResult cir::FuncOp::parse(OpAsmParser &parser, OperationState &state) {
- llvm::SMLoc loc = parser.getCurrentLocation();
- mlir::Builder &builder = parser.getBuilder();
-
- mlir::StringAttr builtinNameAttr = getBuiltinAttrName(state.name);
- mlir::StringAttr coroutineKindNameAttr = getCoroutineKindAttrName(state.name);
- mlir::StringAttr inlineKindNameAttr = getInlineKindAttrName(state.name);
- mlir::StringAttr lambdaNameAttr = getLambdaAttrName(state.name);
- mlir::StringAttr noProtoNameAttr = getNoProtoAttrName(state.name);
- mlir::StringAttr requiresCallConvLoweringNameAttr =
- getRequiresCallConvLoweringAttrName(state.name);
- mlir::StringAttr comdatNameAttr = getComdatAttrName(state.name);
- mlir::StringAttr alignmentNameAttr = getAlignmentAttrName(state.name);
- mlir::StringAttr preferredAlignmentNameAttr =
- getPreferredAlignmentAttrName(state.name);
- mlir::StringAttr visNameAttr = getSymVisibilityAttrName(state.name);
- mlir::StringAttr dsoLocalNameAttr = getDsoLocalAttrName(state.name);
- mlir::StringAttr funcInfoNameAttr = getFuncInfoAttrName(state.name);
-
- if (::mlir::succeeded(parser.parseOptionalKeyword(builtinNameAttr.strref())))
- state.addAttribute(builtinNameAttr, parser.getBuilder().getUnitAttr());
-
- // Parse optional coroutine kind attribute
- {
- static constexpr llvm::StringRef keywords[] = {"coroutine",
- "flattened_coroutine"};
- llvm::StringRef keyword;
- if (parser.parseOptionalKeyword(&keyword, keywords).succeeded())
- state.addAttribute(
- coroutineKindNameAttr,
- cir::CoroutineKindAttr::get(builder.getContext(),
- *cir::symbolizeCoroutineKind(keyword)));
- }
-
- // Parse optional inline kind attribute
- cir::InlineKindAttr inlineKindAttr;
- if (failed(parseInlineKindAttr(parser, inlineKindAttr)))
- return failure();
- if (inlineKindAttr)
- state.addAttribute(inlineKindNameAttr, inlineKindAttr);
-
- if (::mlir::succeeded(parser.parseOptionalKeyword(lambdaNameAttr.strref())))
- state.addAttribute(lambdaNameAttr, parser.getBuilder().getUnitAttr());
- if (parser.parseOptionalKeyword(noProtoNameAttr).succeeded())
- state.addAttribute(noProtoNameAttr, parser.getBuilder().getUnitAttr());
-
- {
- StringAttr comdatAttr;
- if (parseComdatName(parser, comdatAttr).failed())
- return failure();
- if (comdatAttr)
- state.addAttribute(comdatNameAttr, comdatAttr);
- }
-
- auto parseAlignmentBody = [&](int64_t &value) {
- if (parser.parseLParen().failed() || parser.parseInteger(value).failed() ||
- parser.parseRParen().failed())
- return failure();
-
- if (value <= 0)
- return static_cast<LogicalResult>(parser.emitError(
- loc, "function alignment must be a positive integer"));
-
- return success();
- };
-
- if (parser.parseOptionalKeyword(alignmentNameAttr).succeeded()) {
- int64_t value;
- if (parseAlignmentBody(value).failed())
- return failure();
- state.addAttribute(alignmentNameAttr, builder.getI64IntegerAttr(value));
- }
-
- if (parser.parseOptionalKeyword(preferredAlignmentNameAttr).succeeded()) {
- int64_t value;
- if (parseAlignmentBody(value).failed())
- return failure();
- state.addAttribute(preferredAlignmentNameAttr,
- builder.getI64IntegerAttr(value));
- }
-
- // Default to external linkage if no keyword is provided.
- state.addAttribute(getLinkageAttrNameString(),
- GlobalLinkageKindAttr::get(
- parser.getContext(),
- parseOptionalCIRKeyword<GlobalLinkageKind>(
- parser, GlobalLinkageKind::ExternalLinkage)));
-
- ::llvm::StringRef visAttrStr;
- if (parser.parseOptionalKeyword(&visAttrStr, {"private", "public", "nested"})
- .succeeded()) {
- state.addAttribute(visNameAttr,
- parser.getBuilder().getStringAttr(visAttrStr));
- }
-
- state.getOrAddProperties<cir::FuncOp::Properties>().global_visibility =
- parseOptionalCIRKeyword(parser, cir::VisibilityKind::Default);
-
- if (parser.parseOptionalKeyword(dsoLocalNameAttr).succeeded())
- state.addAttribute(dsoLocalNameAttr, parser.getBuilder().getUnitAttr());
-
- StringAttr nameAttr;
- if (parser.parseSymbolName(nameAttr, getSymNameAttrName(state.name),
- state.attributes))
- return failure();
- llvm::SmallVector<OpAsmParser::Argument, 8> arguments;
- llvm::SmallVector<mlir::Type> resultTypes;
- llvm::SmallVector<DictionaryAttr> resultAttrs;
- bool isVariadic = false;
- if (function_interface_impl::parseFunctionSignatureWithArguments(
- parser, /*allowVariadic=*/true, arguments, isVariadic, resultTypes,
- resultAttrs))
- return failure();
- llvm::SmallVector<mlir::Type> argTypes;
- llvm::SmallVector<mlir::Attribute> argAttrs;
- bool argAttrsEmpty = true;
- for (OpAsmParser::Argument &arg : arguments) {
- argTypes.push_back(arg.type);
- // Add the 'empty' attribute anyway to make sure the arity matches, but we
- // only want to 'set' the attribute at the top level if there is SOME data
- // along the way.
- argAttrs.push_back(arg.attrs);
- if (arg.attrs)
- argAttrsEmpty = false;
- }
-
- // These should be in sync anyway, but test both of them anyway.
- if (resultTypes.size() > 1 || resultAttrs.size() > 1)
- return parser.emitError(
- loc, "functions with multiple return types are not supported");
-
- mlir::Type returnType =
- (resultTypes.empty() ? cir::VoidType::get(builder.getContext())
- : resultTypes.front());
-
- cir::FuncType fnType =
- cir::FuncType::getChecked([&]() { return parser.emitError(loc); },
- argTypes, returnType, isVariadic);
- if (!fnType)
- return failure();
-
- state.addAttribute(getFunctionTypeAttrName(state.name),
- TypeAttr::get(fnType));
-
- if (!resultAttrs.empty() && resultAttrs[0])
- state.addAttribute(
- getResAttrsAttrName(state.name),
- mlir::ArrayAttr::get(parser.getContext(), {resultAttrs[0]}));
-
- if (!argAttrsEmpty)
- state.addAttribute(getArgAttrsAttrName(state.name),
- mlir::ArrayAttr::get(parser.getContext(), argAttrs));
-
- bool hasAlias = false;
- mlir::StringAttr aliaseeNameAttr = getAliaseeAttrName(state.name);
- if (parser.parseOptionalKeyword("alias").succeeded()) {
- if (parser.parseLParen().failed())
- return failure();
- mlir::StringAttr aliaseeAttr;
- if (parser.parseOptionalSymbolName(aliaseeAttr).failed())
- return failure();
- state.addAttribute(aliaseeNameAttr, FlatSymbolRefAttr::get(aliaseeAttr));
- if (parser.parseRParen().failed())
- return failure();
- hasAlias = true;
- }
-
- mlir::StringAttr personalityNameAttr = getPersonalityAttrName(state.name);
- if (parser.parseOptionalKeyword("personality").succeeded()) {
- if (parser.parseLParen().failed())
- return failure();
- mlir::StringAttr personalityAttr;
- if (parser.parseOptionalSymbolName(personalityAttr).failed())
- return failure();
- state.addAttribute(personalityNameAttr,
- FlatSymbolRefAttr::get(personalityAttr));
- if (parser.parseRParen().failed())
- return failure();
- }
-
- // Default to C calling convention if no keyword is provided.
- mlir::StringAttr callConvNameAttr = getCallingConvAttrName(state.name);
- cir::CallingConv callConv = cir::CallingConv::C;
- if (parser.parseOptionalKeyword("cc").succeeded()) {
- if (parser.parseLParen().failed())
- return failure();
- if (parseCIRKeyword<cir::CallingConv>(parser, callConv).failed())
- return parser.emitError(loc) << "unknown calling convention";
- if (parser.parseRParen().failed())
- return failure();
- }
- state.addAttribute(callConvNameAttr,
- cir::CallingConvAttr::get(parser.getContext(), callConv));
-
- auto parseGlobalDtorCtor =
- [&](StringRef keyword,
- llvm::function_ref<void(std::optional<int> prio)> createAttr)
- -> mlir::LogicalResult {
- if (mlir::succeeded(parser.parseOptionalKeyword(keyword))) {
- std::optional<int> priority;
- if (mlir::succeeded(parser.parseOptionalLParen())) {
- auto parsedPriority = mlir::FieldParser<int>::parse(parser);
- if (mlir::failed(parsedPriority))
- return parser.emitError(parser.getCurrentLocation(),
- "failed to parse 'priority', of type 'int'");
- priority = parsedPriority.value_or(int());
- // Parse literal ')'
- if (parser.parseRParen())
- return failure();
- }
- createAttr(priority);
- }
- return success();
- };
-
- // Parse the func_info attribute
- if (parser.parseOptionalKeyword("func_info").succeeded()) {
- if (parser.parseLess().failed())
- return failure();
-
- llvm::SMLoc attrLoc = parser.getCurrentLocation();
- mlir::Attribute attr;
- if (parser.parseAttribute(attr).failed())
- return failure();
- if (!mlir::isa<cir::CXXCtorAttr, cir::CXXDtorAttr, cir::CXXAssignAttr,
- cir::FuncIdentityAttr>(attr))
- return parser.emitError(attrLoc,
- "expected a function info attribute, got ")
- << attr;
- state.addAttribute(funcInfoNameAttr, attr);
-
- if (parser.parseGreater().failed())
- return failure();
- }
-
- if (parseGlobalDtorCtor("global_ctor", [&](std::optional<int> priority) {
- mlir::IntegerAttr globalCtorPriorityAttr =
- builder.getI32IntegerAttr(priority.value_or(65535));
- state.addAttribute(getGlobalCtorPriorityAttrName(state.name),
- globalCtorPriorityAttr);
- }).failed())
- return failure();
-
- if (parseGlobalDtorCtor("global_dtor", [&](std::optional<int> priority) {
- mlir::IntegerAttr globalDtorPriorityAttr =
- builder.getI32IntegerAttr(priority.value_or(65535));
- state.addAttribute(getGlobalDtorPriorityAttrName(state.name),
- globalDtorPriorityAttr);
- }).failed())
- return failure();
-
- // Parse optional annotations attribute (an ArrayAttr of AnnotationAttr).
- mlir::StringAttr annotationsNameAttr = getAnnotationsAttrName(state.name);
- mlir::ArrayAttr annotationsAttr;
- if (parser.parseOptionalAttribute(annotationsAttr).has_value() &&
- annotationsAttr)
- state.addAttribute(annotationsNameAttr, annotationsAttr);
-
- if (parser.parseOptionalKeyword(requiresCallConvLoweringNameAttr).succeeded())
- state.addAttribute(requiresCallConvLoweringNameAttr,
- parser.getBuilder().getUnitAttr());
-
- // Parse the rest of the attributes.
- NamedAttrList parsedAttrs;
- llvm::SMLoc attrsLoc = parser.getCurrentLocation();
- if (parser.parseOptionalAttrDictWithKeyword(parsedAttrs))
- return failure();
-
- // Every other declared attribute has dedicated syntax above.
- for (StringRef disallowed : cir::FuncOp::getAttributeNames()) {
- if (isFuncAttrInDict(disallowed))
- continue;
- if (parsedAttrs.get(disallowed))
- return parser.emitError(loc, "attribute '")
- << disallowed
- << "' should not be specified in the explicit attribute list";
- }
-
- if (checkEffectAttrKinds(parser, attrsLoc, parsedAttrs).failed())
- return failure();
-
- state.attributes.append(parsedAttrs);
-
- // Parse the optional function body.
- auto *body = state.addRegion();
- OptionalParseResult parseResult = parser.parseOptionalRegion(
- *body, arguments, /*enableNameShadowing=*/false);
- if (parseResult.has_value()) {
- if (hasAlias)
- return parser.emitError(loc, "function alias shall not have a body");
- if (failed(*parseResult))
- return failure();
- // Function body was parsed, make sure its not empty.
- if (body->empty())
- return parser.emitError(loc, "expected non-empty function body");
- }
-
- return success();
-}
-
-// This function corresponds to `llvm::GlobalValue::isDeclaration` and should
-// have a similar implementation. We don't currently ifuncs or materializable
-// functions, but those should be handled here as they are implemented.
-bool cir::FuncOp::isDeclaration() {
- assert(!cir::MissingFeatures::supportIFuncAttr());
-
- std::optional<StringRef> aliasee = getAliasee();
- if (!aliasee)
- return getFunctionBody().empty();
-
- // Aliases are always definitions.
- return false;
-}
-
-bool cir::FuncOp::isCXXSpecialMemberFunction() {
- // The func_info union can grow forms that are not special members, so the
- // check names the concrete forms rather than testing for presence.
- mlir::Attribute attr = getFuncInfoAttr();
- return attr && mlir::isa<CXXCtorAttr, CXXDtorAttr, CXXAssignAttr>(attr);
-}
-
-bool cir::FuncOp::isCxxConstructor() {
- auto attr = getFuncInfoAttr();
- return attr && dyn_cast<CXXCtorAttr>(attr);
-}
-
-bool cir::FuncOp::isCxxDestructor() {
- auto attr = getFuncInfoAttr();
- return attr && dyn_cast<CXXDtorAttr>(attr);
-}
-
-bool cir::FuncOp::isCxxSpecialAssignment() {
- auto attr = getFuncInfoAttr();
- return attr && dyn_cast<CXXAssignAttr>(attr);
-}
-
-std::optional<CtorKind> cir::FuncOp::getCxxConstructorKind() {
- mlir::Attribute attr = getFuncInfoAttr();
- if (attr) {
- if (auto ctor = dyn_cast<CXXCtorAttr>(attr))
- return ctor.getCtorKind();
- }
- return std::nullopt;
-}
-
-std::optional<AssignKind> cir::FuncOp::getCxxSpecialAssignKind() {
- mlir::Attribute attr = getFuncInfoAttr();
- if (attr) {
- if (auto assign = dyn_cast<CXXAssignAttr>(attr))
- return assign.getAssignKind();
- }
- return std::nullopt;
-}
-
-bool cir::FuncOp::isCxxTrivialMemberFunction() {
- mlir::Attribute attr = getFuncInfoAttr();
- if (attr) {
- if (auto ctor = dyn_cast<CXXCtorAttr>(attr))
- return ctor.getIsTrivial();
- if (auto dtor = dyn_cast<CXXDtorAttr>(attr))
- return dtor.getIsTrivial();
- if (auto assign = dyn_cast<CXXAssignAttr>(attr))
- return assign.getIsTrivial();
- }
- return false;
-}
-
-mlir::Region *cir::FuncOp::getCallableRegion() {
- // Declarations and aliases have no body to analyze or inline. Returning the
- // empty region would make interprocedural analyses (e.g. SCCP) treat calls
- // to them as having no returns, leaving their results uninitialized.
- if (getBody().empty())
- return nullptr;
- return &getBody();
-}
-
-void cir::FuncOp::print(OpAsmPrinter &p) {
- if (getBuiltin())
- p << " builtin";
-
- if (std::optional<cir::CoroutineKind> coroKind = getCoroutineKind())
- p << ' ' << cir::stringifyCoroutineKind(*coroKind);
-
- printInlineKindAttr(p, getInlineKindAttr());
-
- if (getLambda())
- p << " lambda";
-
- if (getNoProto())
- p << " no_proto";
-
- if (getComdatAttr()) {
- p << ' ';
- ::printComdatName(p, getComdatAttr());
- }
-
- if (getAlignment())
- p << " alignment(" << *getAlignment() << ')';
-
- if (getPreferredAlignment())
- p << " preferred_alignment(" << *getPreferredAlignment() << ')';
-
- if (getLinkage() != GlobalLinkageKind::ExternalLinkage)
- p << ' ' << stringifyGlobalLinkageKind(getLinkage());
-
- mlir::SymbolTable::Visibility vis = getVisibility();
- if (vis != mlir::SymbolTable::Visibility::Public)
- p << ' ' << vis;
-
- if (getGlobalVisibility() != cir::VisibilityKind::Default)
- p << ' ' << stringifyVisibilityKind(getGlobalVisibility());
-
- if (getDsoLocal())
- p << " dso_local";
-
- p << ' ';
- p.printSymbolName(getSymName());
- cir::FuncType fnType = getFunctionType();
- function_interface_impl::printFunctionSignature(
- p, *this, fnType.getInputs(), fnType.isVarArg(), fnType.getReturnTypes());
-
- if (std::optional<StringRef> aliaseeName = getAliasee()) {
- p << " alias(";
- p.printSymbolName(*aliaseeName);
- p << ")";
- }
-
- if (getCallingConv() != cir::CallingConv::C) {
- p << " cc(";
- p << stringifyCallingConv(getCallingConv());
- p << ")";
- }
-
- if (std::optional<StringRef> personalityName = getPersonality()) {
- p << " personality(";
- p.printSymbolName(*personalityName);
- p << ")";
- }
-
- if (mlir::Attribute funcInfo = getFuncInfoAttr()) {
- p << " func_info<";
- p.printAttribute(funcInfo);
- p << '>';
- }
-
- if (auto globalCtorPriority = getGlobalCtorPriority()) {
- p << " global_ctor";
- if (globalCtorPriority.value() != 65535)
- p << "(" << globalCtorPriority.value() << ")";
- }
-
- if (auto globalDtorPriority = getGlobalDtorPriority()) {
- p << " global_dtor";
- if (globalDtorPriority.value() != 65535)
- p << "(" << globalDtorPriority.value() << ")";
- }
-
- if (mlir::ArrayAttr annotations = getAnnotationsAttr()) {
- p << ' ';
- p.printAttribute(annotations);
- }
-
- if (getRequiresCallConvLowering())
- p << ' ' << getRequiresCallConvLoweringAttrName().getValue();
-
- // Every other declared attribute is printed by the syntax above.
- llvm::SmallVector<llvm::StringRef> elidedAttrs;
- for (llvm::StringRef name : cir::FuncOp::getAttributeNames())
- if (!isFuncAttrInDict(name))
- elidedAttrs.push_back(name);
- function_interface_impl::printFunctionAttributes(p, *this, elidedAttrs);
-
- // Print the body if this is not an external function.
- Region &body = getOperation()->getRegion(0);
- if (!body.empty()) {
- p << ' ';
- p.printRegion(body, /*printEntryBlockArgs=*/false,
- /*printBlockTerminators=*/true);
- }
-}
-
-mlir::LogicalResult cir::FuncOp::verify() {
-
- if (!isDeclaration() && getCoroutineKind() == cir::CoroutineKind::Coroutine) {
- int coroutineOpCount = 0;
- this->walk([&](Operation *op) {
- if (isa<CoroutineOp>(op)) {
- coroutineOpCount++;
- if (coroutineOpCount > 1) {
- return mlir::WalkResult::interrupt();
- }
- }
- return mlir::WalkResult::advance();
- });
- if (coroutineOpCount != 1)
- return emitOpError()
- << "coroutine function must have exactly one cir.coroutine op";
- }
-
- llvm::SmallSet<llvm::StringRef, 16> labels;
- llvm::SmallSet<llvm::StringRef, 16> gotos;
- llvm::SmallSet<llvm::StringRef, 16> blockAddresses;
- bool invalidBlockAddress = false;
- getOperation()->walk([&](mlir::Operation *op) {
- if (auto lab = dyn_cast<cir::LabelOp>(op)) {
- labels.insert(lab.getLabel());
- } else if (auto goTo = dyn_cast<cir::GotoOp>(op)) {
- gotos.insert(goTo.getLabel());
- } else if (auto blkAdd = dyn_cast<cir::BlockAddressOp>(op)) {
- if (blkAdd.getBlockAddrInfoAttr().getFunc().getAttr() != getSymName()) {
- // Stop the walk early, no need to continue
- invalidBlockAddress = true;
- return mlir::WalkResult::interrupt();
- }
- blockAddresses.insert(blkAdd.getBlockAddrInfoAttr().getLabel());
- }
- return mlir::WalkResult::advance();
- });
-
- if (invalidBlockAddress)
- return emitOpError() << "blockaddress references a different function";
-
- llvm::SmallSet<llvm::StringRef, 16> mismatched;
- if (!labels.empty() || !gotos.empty()) {
- mismatched = llvm::set_difference(gotos, labels);
-
- if (!mismatched.empty())
- return emitOpError() << "goto/label mismatch";
- }
-
- mismatched.clear();
-
- if (!labels.empty() || !blockAddresses.empty()) {
- mismatched = llvm::set_difference(blockAddresses, labels);
-
- if (!mismatched.empty())
- return emitOpError()
- << "expects an existing label target in the referenced function";
- }
-
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// AddOp / SubOp
-//===----------------------------------------------------------------------===//
-
-// The integer-only type constraint on these ops makes the nsw/nuw/sat flag
-// type checks unnecessary. Only the mutual-exclusivity between nsw/nuw and
-// sat needs to be verified.
-
-LogicalResult cir::AddOp::verify() {
- if (getSaturated() && (getNoSignedWrap() || getNoUnsignedWrap()))
- return emitOpError()
- << "the nsw/nuw flags and the saturated flag are mutually exclusive";
- return mlir::success();
-}
-
-LogicalResult cir::SubOp::verify() {
- if (getSaturated() && (getNoSignedWrap() || getNoUnsignedWrap()))
- return emitOpError()
- << "the nsw/nuw flags and the saturated flag are mutually exclusive";
- return mlir::success();
-}
-
-//===----------------------------------------------------------------------===//
-// TernaryOp
-//===----------------------------------------------------------------------===//
-
-void cir::TernaryOp::build(
- OpBuilder &builder, OperationState &result, Value cond,
- function_ref<void(OpBuilder &, Location)> trueBuilder,
- function_ref<void(OpBuilder &, Location)> falseBuilder) {
- result.addOperands(cond);
- OpBuilder::InsertionGuard guard(builder);
- Region *trueRegion = result.addRegion();
- builder.createBlock(trueRegion);
- trueBuilder(builder, result.location);
- Region *falseRegion = result.addRegion();
- builder.createBlock(falseRegion);
- falseBuilder(builder, result.location);
-
- // Get result type from whichever branch has a yield (the other may have
- // unreachable from a throw expression)
- cir::YieldOp yield;
- if (trueRegion->back().mightHaveTerminator())
- yield = dyn_cast_or_null<cir::YieldOp>(trueRegion->back().getTerminator());
- if (!yield && falseRegion->back().mightHaveTerminator())
- yield = dyn_cast_or_null<cir::YieldOp>(falseRegion->back().getTerminator());
-
- assert((!yield || yield.getNumOperands() <= 1) &&
- "expected zero or one result type");
- if (yield && yield.getNumOperands() == 1)
- result.addTypes(TypeRange{yield.getOperandTypes().front()});
-}
-
-//===----------------------------------------------------------------------===//
-// SelectOp
-//===----------------------------------------------------------------------===//
-
-OpFoldResult cir::SelectOp::fold(FoldAdaptor adaptor) {
- if (auto condition =
- mlir::dyn_cast_if_present<cir::BoolAttr>(adaptor.getCondition()))
- return condition.getValue() ? getTrueValue() : getFalseValue();
-
- // cir.select if %0 then x else x -> x
- mlir::Attribute trueValue = adaptor.getTrueValue();
- mlir::Attribute falseValue = adaptor.getFalseValue();
- if (trueValue == falseValue)
- return trueValue;
- if (getTrueValue() == getFalseValue())
- return getTrueValue();
-
- return {};
-}
-
-LogicalResult cir::SelectOp::verify() {
- // AllTypesMatch already guarantees trueVal and falseVal have matching types.
- auto condTy = dyn_cast<cir::VectorType>(getCondition().getType());
-
- // If condition is not a vector, no further checks are needed.
- if (!condTy)
- return success();
-
- // When condition is a vector, both other operands must also be vectors.
- if (!isa<cir::VectorType>(getTrueValue().getType()) ||
- !isa<cir::VectorType>(getFalseValue().getType())) {
- return emitOpError()
- << "expected both true and false operands to be vector types "
- "when the condition is a vector boolean type";
- }
-
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// ShiftOp
-//===----------------------------------------------------------------------===//
-LogicalResult cir::ShiftOp::verify() {
- mlir::Operation *op = getOperation();
- auto op0VecTy = mlir::dyn_cast<cir::VectorType>(op->getOperand(0).getType());
- auto op1VecTy = mlir::dyn_cast<cir::VectorType>(op->getOperand(1).getType());
- if (!op0VecTy ^ !op1VecTy)
- return emitOpError() << "input types cannot be one vector and one scalar";
-
- if (op0VecTy) {
- if (op0VecTy.getSize() != op1VecTy.getSize())
- return emitOpError() << "input vector types must have the same size";
-
- auto opResultTy = mlir::dyn_cast<cir::VectorType>(getType());
- if (!opResultTy)
- return emitOpError() << "the type of the result must be a vector "
- << "if it is vector shift";
-
- auto op0VecEleTy = mlir::cast<cir::IntType>(op0VecTy.getElementType());
- auto op1VecEleTy = mlir::cast<cir::IntType>(op1VecTy.getElementType());
- if (op0VecEleTy.getWidth() != op1VecEleTy.getWidth())
- return emitOpError()
- << "vector operands do not have the same elements sizes";
-
- auto resVecEleTy = mlir::cast<cir::IntType>(opResultTy.getElementType());
- if (op0VecEleTy.getWidth() != resVecEleTy.getWidth())
- return emitOpError() << "vector operands and result type do not have the "
- "same elements sizes";
- }
-
- return mlir::success();
-}
-
-//===----------------------------------------------------------------------===//
-// LabelOp Definitions
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::LabelOp::verify() {
- mlir::Operation *op = getOperation();
- mlir::Block *blk = op->getBlock();
- if (&blk->front() != op)
- return emitError() << "must be the first operation in a block";
-
- return mlir::success();
-}
-
-//===----------------------------------------------------------------------===//
-// IncOp
-//===----------------------------------------------------------------------===//
-
-OpFoldResult cir::IncOp::fold(FoldAdaptor adaptor) {
- if (mlir::isa_and_present<cir::PoisonAttr>(adaptor.getInput()))
- return adaptor.getInput();
- return {};
-}
-
-//===----------------------------------------------------------------------===//
-// DecOp
-//===----------------------------------------------------------------------===//
-
-OpFoldResult cir::DecOp::fold(FoldAdaptor adaptor) {
- if (mlir::isa_and_present<cir::PoisonAttr>(adaptor.getInput()))
- return adaptor.getInput();
- return {};
-}
-
-//===----------------------------------------------------------------------===//
-// MinusOp
-//===----------------------------------------------------------------------===//
-
-OpFoldResult cir::MinusOp::fold(FoldAdaptor adaptor) {
- if (mlir::isa_and_present<cir::PoisonAttr>(adaptor.getInput()))
- return adaptor.getInput();
-
- // Fold with constant inputs. Floating-point negation is handled by
- // cir::FNegOp.
- if (auto intAttr =
- mlir::dyn_cast_if_present<cir::IntAttr>(adaptor.getInput())) {
- APInt val = intAttr.getValue();
- val.negate();
- return cir::IntAttr::get(getType(), val);
- }
-
- return {};
-}
-
-//===----------------------------------------------------------------------===//
-// FNegOp
-//===----------------------------------------------------------------------===//
-
-OpFoldResult cir::FNegOp::fold(FoldAdaptor adaptor) {
- if (mlir::isa_and_present<cir::PoisonAttr>(adaptor.getInput()))
- return adaptor.getInput();
-
- // Fold with constant inputs.
- if (auto fpAttr =
- mlir::dyn_cast_if_present<cir::FPAttr>(adaptor.getInput())) {
- APFloat val = fpAttr.getValue();
- val.changeSign();
- return cir::FPAttr::get(getType(), val);
- }
-
- return {};
-}
-
-//===----------------------------------------------------------------------===//
-// NotOp
-//===----------------------------------------------------------------------===//
-
-OpFoldResult cir::NotOp::fold(FoldAdaptor adaptor) {
- if (mlir::isa_and_present<cir::PoisonAttr>(adaptor.getInput()))
- return adaptor.getInput();
-
- // not(not(x)) -> x is handled by the Involution trait.
-
- // Fold with constant inputs.
- if (mlir::Attribute attr = adaptor.getInput()) {
- if (auto intAttr = mlir::dyn_cast<cir::IntAttr>(attr)) {
- APInt val = intAttr.getValue();
- val.flipAllBits();
- return cir::IntAttr::get(getType(), val);
- }
- if (auto boolAttr = mlir::dyn_cast<cir::BoolAttr>(attr))
- return cir::BoolAttr::get(getContext(), !boolAttr.getValue());
- }
-
- return {};
-}
-
-//===----------------------------------------------------------------------===//
-// BaseClassAddrOp & DerivedClassAddrOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::BaseClassAddrOp::verify() {
- if (getSrcAddr().getType().getAddrSpace() != getType().getAddrSpace())
- return emitOpError() << "address space mismatch";
- return mlir::success();
-}
-
-LogicalResult cir::DerivedClassAddrOp::verify() {
- if (getSrcAddr().getType().getAddrSpace() != getType().getAddrSpace())
- return emitOpError() << "address space mismatch";
- return mlir::success();
-}
-
-//===----------------------------------------------------------------------===//
-// BaseDataMemberOp & DerivedDataMemberOp
-//===----------------------------------------------------------------------===//
-
-static LogicalResult verifyMemberPtrCast(Operation *op, mlir::Value src,
- mlir::Type resultTy) {
- // Let the operand type be T1 C1::*, let the result type be T2 C2::*.
- // Verify that T1 and T2 are the same type.
- mlir::Type inputMemberTy;
- mlir::Type resultMemberTy;
- if (mlir::isa<cir::DataMemberType>(src.getType())) {
- inputMemberTy =
- mlir::cast<cir::DataMemberType>(src.getType()).getMemberTy();
- resultMemberTy = mlir::cast<cir::DataMemberType>(resultTy).getMemberTy();
- }
- assert(!cir::MissingFeatures::memberFuncPtrCast());
- if (inputMemberTy != resultMemberTy)
- return op->emitOpError()
- << "member types of the operand and the result do not match";
-
- return mlir::success();
-}
-
-LogicalResult cir::BaseDataMemberOp::verify() {
- return verifyMemberPtrCast(getOperation(), getSrc(), getType());
-}
-
-LogicalResult cir::DerivedDataMemberOp::verify() {
- return verifyMemberPtrCast(getOperation(), getSrc(), getType());
-}
-
-//===----------------------------------------------------------------------===//
-// BaseMethodOp & DerivedMethodOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::BaseMethodOp::verify() {
- return verifyMemberPtrCast(getOperation(), getSrc(), getType());
-}
-
-LogicalResult cir::DerivedMethodOp::verify() {
- return verifyMemberPtrCast(getOperation(), getSrc(), getType());
-}
-
-//===----------------------------------------------------------------------===//
-// AwaitOp
-//===----------------------------------------------------------------------===//
-
-void cir::AwaitOp::build(OpBuilder &builder, OperationState &result,
- cir::AwaitKind kind, BuilderCallbackRef readyBuilder,
- BuilderCallbackRef suspendBuilder,
- BuilderCallbackRef resumeBuilder) {
- result.addAttribute(getKindAttrName(result.name),
- cir::AwaitKindAttr::get(builder.getContext(), kind));
- {
- OpBuilder::InsertionGuard guard(builder);
- Region *readyRegion = result.addRegion();
- builder.createBlock(readyRegion);
- readyBuilder(builder, result.location);
- }
-
- {
- OpBuilder::InsertionGuard guard(builder);
- Region *suspendRegion = result.addRegion();
- builder.createBlock(suspendRegion);
- suspendBuilder(builder, result.location);
- }
-
- {
- OpBuilder::InsertionGuard guard(builder);
- Region *resumeRegion = result.addRegion();
- builder.createBlock(resumeRegion);
- resumeBuilder(builder, result.location);
- }
-}
-
-void cir::AwaitOp::getSuccessorRegions(
- mlir::RegionBranchPoint point, SmallVectorImpl<RegionSuccessor> ®ions) {
- assert(point.isParent() || point.getTerminatorPredecessorOrNull());
-
- // Execution always starts in the ready region.
- if (point.isParent()) {
- regions.emplace_back(&getReady());
- return;
- }
-
- mlir::Region *parentRegion =
- point.getTerminatorPredecessorOrNull()->getParentRegion();
-
- // Branching from ready: the cir.condition terminating it selects between
- // suspending and resuming. Keep in sync with
- // ConditionOp::getSuccessorRegions.
- //
- // TODO: retrieve information from the promise and only push the
- // necessary ones. Example: `std::suspend_never` on initial or final
- // await's might allow suspend region to be skipped.
- if (&getReady() == parentRegion) {
- regions.emplace_back(&getResume());
- regions.emplace_back(&getSuspend());
- return;
- }
-
- // Branching from suspend: continue in resume once the coroutine is resumed
- // (or immediately if suspension is vetoed), or exit to the parent operation
- // if it stays suspended or is destroyed.
- if (&getSuspend() == parentRegion) {
- regions.emplace_back(&getResume());
- regions.emplace_back(getOperation());
- return;
- }
-
- // Branching from resume: exit to the parent operation.
- regions.emplace_back(getOperation());
-}
-
-LogicalResult cir::AwaitOp::verify() {
- if (!isa<ConditionOp>(this->getReady().back().getTerminator()))
- return emitOpError("ready region must end with cir.condition");
- if (this->getSuspend().empty())
- return emitOpError("suspend region must not be empty");
- if (!isa<CoroSuspendPoint>(this->getSuspend().back().getTerminator()))
- return emitOpError("suspend region must end with cir.coro.suspend_point");
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// CoReturnOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::CoReturnOp::verify() {
- mlir::Operation *coRet = getOperation();
- auto coroutine = coRet->getParentOfType<CoroutineOp>();
- if (!coroutine.getBody().isAncestor(getOperation()->getParentRegion()))
- return emitOpError("must be inside the cir.coroutine body region");
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// CoroutineOp
-//===----------------------------------------------------------------------===//
-
-void cir::CoroutineOp::build(OpBuilder &builder, OperationState &result,
- BuilderCallbackRef initialSuspendBuilder,
- BuilderCallbackRef bodyBuilder,
- BuilderCallbackRef finalSuspendBuilder,
- BuilderCallbackRef destroyBuilder,
- BuilderCallbackRef exitBuilder) {
- {
- OpBuilder::InsertionGuard guard(builder);
- Region *initialRegion = result.addRegion();
- builder.createBlock(initialRegion);
- initialSuspendBuilder(builder, result.location);
- }
-
- {
- OpBuilder::InsertionGuard guard(builder);
- Region *bodyRegion = result.addRegion();
- builder.createBlock(bodyRegion);
- bodyBuilder(builder, result.location);
- }
-
- {
- OpBuilder::InsertionGuard guard(builder);
- Region *finalRegion = result.addRegion();
- builder.createBlock(finalRegion);
- finalSuspendBuilder(builder, result.location);
- }
-
- {
- OpBuilder::InsertionGuard guard(builder);
- Region *destroyRegion = result.addRegion();
- builder.createBlock(destroyRegion);
- destroyBuilder(builder, result.location);
- }
-
- {
- OpBuilder::InsertionGuard guard(builder);
- Region *exitRegion = result.addRegion();
- builder.createBlock(exitRegion);
- exitBuilder(builder, result.location);
- }
-}
-
-void cir::CoroutineOp::getSuccessorRegions(
- mlir::RegionBranchPoint point, SmallVectorImpl<RegionSuccessor> ®ions) {
- // The parent op always enters through initial_suspend get_return_object()
- // (if present) and the init await always run first.
- if (point.isParent()) {
- regions.emplace_back(&getInitialSuspend());
- return;
- }
-
- mlir::Region *parent =
- point.getTerminatorPredecessorOrNull()->getParentRegion();
-
- // initial_suspend either falls into body (resumed, or never actually
- // suspended because await_ready() was true), exits directly, or reaches
- // destroy when the coroutine is destroyed while suspended at its initial
- // suspend point.
- if (parent == &getInitialSuspend()) {
- regions.emplace_back(&getBody());
- regions.emplace_back(&getExit());
- regions.emplace_back(&getDestroy());
- return;
- }
-
- // body can fall through into final_suspend (co_return), exit directly
- // from any suspend_point inside it that just suspends, or reach destroy,
- // either from a suspend_point being destroyed, or from an exception
- // escaping body's own catch-all.
- if (parent == &getBody()) {
- regions.emplace_back(&getFinalSuspend());
- regions.emplace_back(&getExit()); // any suspend_point inside body
- regions.emplace_back(&getDestroy()); // destroy() on any suspend inside body
- return;
- }
-
- // final_suspend's only live edge in valid programs is destroy. Resuming
- // past the final suspend is UB. The ready-immediately edge to exit is
- // kept for structural symmetry with the other two suspend regions even
- // though it's effectively dead.
- if (parent == &getFinalSuspend()) {
- regions.emplace_back(
- &getDestroy()); // the only real exit from final_suspend
- regions.emplace_back(&getExit()); // ready==true edge, rarely taken
- return;
- }
-
- // destroy has two possible outcomes depending on why it was entered:
- // ordinary destroy dispatch falls through to exit (normal return); an
- // exception that reached destroy needs to keep propagating instead, i.e.
- // leave the whole op rather than go through exit's cir.return.
- if (parent == &getDestroy()) {
- regions.emplace_back(&getExit());
- regions.push_back(RegionSuccessor(getOperation())); // unwind out of the op
- return;
- }
- // exit always terminates the op there's no region it loops back into.
- if (parent == &getExit()) {
- regions.emplace_back(getOperation());
- return;
- }
-}
-
-mlir::ValueRange
-cir::CoroutineOp::getSuccessorInputs(RegionSuccessor successor) {
- return ValueRange();
-}
-
-LogicalResult cir::CoroutineOp::verify() {
- mlir::Region &initialSuspend = getInitialSuspend();
- mlir::Region &body = getBody();
- mlir::Region &finalSuspend = getFinalSuspend();
- mlir::Region &destroy = getDestroy();
- mlir::Region &exit = getExit();
-
- // initial_suspend must contain exactly one 'init'-kind cir.await an
- // optional prelude of ordinary statements (get_return_object() and
- // similar) is allowed before it, but no *other* await may appear.
- int initAwaitCount = 0;
- bool hasInvalidAwait = false;
-
- initialSuspend.walk([&](Operation *op) {
- auto awaitOp = mlir::dyn_cast<AwaitOp>(op);
- if (!awaitOp)
- return mlir::WalkResult::advance();
-
- if (awaitOp.getKind() == cir::AwaitKind::Init)
- ++initAwaitCount;
- else
- hasInvalidAwait = true;
-
- if (initAwaitCount > 1 || hasInvalidAwait)
- return mlir::WalkResult::interrupt();
-
- return mlir::WalkResult::advance();
- });
-
- if (hasInvalidAwait)
- return emitOpError("'initial_suspend' must not contain any cir.await other "
- "than the 'init' one");
- if (initAwaitCount != 1)
- return emitOpError(
- "must have exactly one 'init' cir.await in 'initial_suspend'");
-
- // final_suspend must contain only one or zero 'final'-kind cir.await
- int finalAwaitCount = 0;
- hasInvalidAwait = false;
-
- finalSuspend.walk([&](Operation *op) {
- auto awaitOp = mlir::dyn_cast<AwaitOp>(op);
- if (!awaitOp)
- return mlir::WalkResult::advance();
-
- if (awaitOp.getKind() == cir::AwaitKind::Final)
- ++finalAwaitCount;
- else
- hasInvalidAwait = true;
-
- if (finalAwaitCount > 1 || hasInvalidAwait)
- return mlir::WalkResult::interrupt();
-
- return mlir::WalkResult::advance();
- });
-
- if (hasInvalidAwait)
- return emitOpError("'final_suspend' must not contain any cir.await other "
- "than the 'final' one");
- if (finalAwaitCount > 1)
- return emitOpError(
- "must have one or zero 'final' cir.await in 'final_suspend'");
-
- // Each region must end with the terminator its role requires:
- // initial_suspend/final_suspend/destroy yield back into cir.coroutine's
- // own control flow, body ends in a cir.co_return (or a plain yield if
- // some path never reaches one), and exit actually returns from the
- // function.
- if (initialSuspend.empty() || initialSuspend.back().empty())
- return emitOpError("initial_suspend region must not be empty");
- if (!isa<YieldOp>(initialSuspend.back().back()))
- return emitOpError("'initial_suspend' must end with cir.yield");
-
- if (body.empty() || body.back().empty())
- return emitOpError("'body' region must not be empty");
- if (!isa<YieldOp, CoReturnOp>(body.back().back()))
- return emitOpError("'body' must end with cir.yield or cir.co_return");
-
- if (finalSuspend.empty() || finalSuspend.back().empty())
- return emitOpError("'final_suspend' region must not be empty");
- if (!isa<YieldOp>(finalSuspend.back().back()))
- return emitOpError("'final_suspend' must end with cir.yield");
-
- if (destroy.empty() || destroy.back().empty())
- return emitOpError("'destroy' region must not be empty");
- if (!isa<YieldOp>(destroy.back().back()))
- return emitOpError("'destroy' must end with cir.yield");
-
- if (exit.empty() || exit.back().empty())
- return emitOpError("'exit' region must not be empty");
- if (!isa<ReturnOp>(exit.back().back()))
- return emitOpError("'exit' must end with cir.return");
-
- return mlir::success();
-}
-
-//===----------------------------------------------------------------------===//
-// CopyOp Definitions
-//===----------------------------------------------------------------------===//
-
-// Prints the pointer type(s) for a `cir.copy`. `src` and `dst` share the same
-// pointee type but may differ in address space; a single type is printed when
-// they match, otherwise both are printed (`src` first).
-void cir::printCopyTypes(mlir::OpAsmPrinter &printer, mlir::Operation *,
- mlir::Type srcType, mlir::Type dstType) {
- printer.printType(srcType);
- if (srcType != dstType) {
- printer << ", ";
- printer.printType(dstType);
- }
-}
-
-mlir::ParseResult cir::parseCopyTypes(mlir::OpAsmParser &parser,
- mlir::Type &srcType,
- mlir::Type &dstType) {
- if (parser.parseType(srcType))
- return mlir::failure();
- if (parser.parseOptionalComma().succeeded()) {
- if (parser.parseType(dstType))
- return mlir::failure();
- } else {
- dstType = srcType;
- }
- return mlir::success();
-}
-
-LogicalResult cir::CopyOp::verify() {
- // The pointee types of `src` and `dst` are guaranteed to match by the
- // SameOperandsPointeeType trait; they may still differ in address space.
-
- // A data layout is required for us to know the number of bytes to be copied.
- if (!getType().getPointee().hasTrait<DataLayoutTypeInterface::Trait>())
- return emitError() << "missing data layout for pointee type";
-
- if (getSkipTailPadding() &&
- !mlir::isa<cir::RecordType>(getType().getPointee()))
- return emitError()
- << "skip_tail_padding is only valid for record pointee types";
-
- return mlir::success();
-}
-
-//===----------------------------------------------------------------------===//
-// PtrMaskOp Definitions
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::PtrMaskOp::verify() {
- mlir::DataLayout layout = mlir::DataLayout::closest(*this);
- std::optional<uint64_t> indexWidth =
- layout.getTypeIndexBitwidth(getPtr().getType());
- if (!indexWidth)
- return emitOpError() << "pointer has no index width";
-
- uint64_t maskWidth = getMask().getType().getWidth();
- if (maskWidth != *indexWidth)
- return emitOpError() << "mask width " << maskWidth
- << " must equal the pointer index width "
- << *indexWidth;
-
- return mlir::success();
-}
-
-//===----------------------------------------------------------------------===//
-// GetRuntimeMemberOp Definitions
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::GetRuntimeMemberOp::verify() {
- cir::DataMemberType memberPtrTy = getMember().getType();
-
- if (getAddr().getType().getPointee() != memberPtrTy.getClassTy())
- return emitError() << "record type does not match the member pointer type";
- if (getType().getPointee() != memberPtrTy.getMemberTy())
- return emitError() << "result type does not match the member pointer type";
- return mlir::success();
-}
-
-//===----------------------------------------------------------------------===//
-// GetMethodOp Definitions
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::GetMethodOp::verify() {
- cir::MethodType methodTy = getMethod().getType();
-
- // Assume objectTy is !cir.ptr<!T>
- cir::PointerType objectPtrTy = getObject().getType();
- mlir::Type objectTy = objectPtrTy.getPointee();
-
- if (methodTy.getClassTy() != objectTy)
- return emitError() << "method class type and object type do not match";
-
- // Assume methodFuncTy is !cir.func<!Ret (!Args)>
- auto calleeTy = mlir::cast<cir::FuncType>(getCallee().getType().getPointee());
- cir::FuncType methodFuncTy = methodTy.getMemberFuncTy();
-
- // We verify at here that calleeTy is !cir.func<!Ret (!cir.ptr<!void>, !Args)>
- // Note that the first parameter type of the callee is !cir.ptr<!void> instead
- // of !cir.ptr<!T> because the "this" pointer may be adjusted before calling
- // the callee.
-
- if (methodFuncTy.getReturnType() != calleeTy.getReturnType())
- return emitError()
- << "method return type and callee return type do not match";
-
- llvm::ArrayRef<mlir::Type> calleeArgsTy = calleeTy.getInputs();
- llvm::ArrayRef<mlir::Type> methodFuncArgsTy = methodFuncTy.getInputs();
-
- if (calleeArgsTy.empty())
- return emitError() << "callee parameter list lacks receiver object ptr";
-
- auto calleeThisArgPtrTy = mlir::dyn_cast<cir::PointerType>(calleeArgsTy[0]);
- if (!calleeThisArgPtrTy ||
- !mlir::isa<cir::VoidType>(calleeThisArgPtrTy.getPointee())) {
- return emitError()
- << "the first parameter of callee must be a void pointer";
- }
-
- if (calleeArgsTy.size() != methodFuncArgsTy.size())
- return emitError() << "callee and method parameter counts do not match";
-
- if (calleeArgsTy.size() > 1 &&
- calleeArgsTy.slice(1) != methodFuncArgsTy.slice(1))
- return emitError()
- << "callee parameters and method parameters do not match";
-
- return mlir::success();
-}
-
-//===----------------------------------------------------------------------===//
-// GetMemberOp Definitions
-//===----------------------------------------------------------------------===//
-
-static mlir::Type memberPointeeType(cir::RecordType recordTy, unsigned idx) {
- return cir::memberStorageType(recordTy.getMembers()[idx]);
-}
-
-LogicalResult cir::GetMemberOp::verify() {
- const auto recordTy = dyn_cast<RecordType>(getAddrTy().getPointee());
- if (!recordTy)
- return emitError() << "expected pointer to a record type";
-
- if (recordTy.getMembers().size() <= getIndex())
- return emitError() << "member index out of bounds";
-
- mlir::Type pointeeTy = memberPointeeType(recordTy, getIndex());
- if (!pointeeTy)
- return emitError() << "member owns no storage to point at";
-
- if (pointeeTy != getType().getPointee())
- return emitError() << "member type mismatch";
-
- if (getAddrTy().getAddrSpace() != getType().getAddrSpace())
- return emitError() << "address space mismatch";
-
- return mlir::success();
-}
-
-//===----------------------------------------------------------------------===//
-// ExtractMemberOp Definitions
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::ExtractMemberOp::verify() {
- if (mlir::isa<cir::UnionType>(getRecord().getType()))
- return emitError()
- << "cir.extract_member currently does not support unions";
- auto structTy = mlir::cast<cir::StructType>(getRecord().getType());
- if (structTy.getMembers().size() <= getIndex())
- return emitError() << "member index out of bounds";
- mlir::Type memberTy = structTy.getMembers()[getIndex()];
- if (mlir::isa<cir::BitFieldType>(memberTy))
- return emitError() << "cir.extract_member does not support bit-fields";
- if (memberTy != getType())
- return emitError() << "member type mismatch";
- return mlir::success();
-}
-
-//===----------------------------------------------------------------------===//
-// InsertMemberOp Definitions
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::InsertMemberOp::verify() {
- if (mlir::isa<cir::UnionType>(getRecord().getType()))
- return emitError() << "cir.insert_member currently does not support unions";
- auto structTy = mlir::cast<cir::StructType>(getRecord().getType());
- if (structTy.getMembers().size() <= getIndex())
- return emitError() << "member index out of bounds";
- mlir::Type memberTy = structTy.getMembers()[getIndex()];
- if (mlir::isa<cir::BitFieldType>(memberTy))
- return emitError() << "cir.insert_member does not support bit-fields";
- if (memberTy != getValue().getType())
- return emitError() << "member type mismatch";
- // The op trait already checks that the types of $result and $record match.
- return mlir::success();
-}
-
-//===----------------------------------------------------------------------===//
-// VecCreateOp
-//===----------------------------------------------------------------------===//
-
-OpFoldResult cir::VecCreateOp::fold(FoldAdaptor adaptor) {
- if (llvm::any_of(getElements(), [](mlir::Value value) {
- return !value.getDefiningOp<cir::ConstantOp>();
- }))
- return {};
-
- return cir::ConstVectorAttr::get(
- getType(), mlir::ArrayAttr::get(getContext(), adaptor.getElements()));
-}
-
-LogicalResult cir::VecCreateOp::verify() {
- // Verify that the number of arguments matches the number of elements in the
- // vector, and that the type of all the arguments matches the type of the
- // elements in the vector.
- const cir::VectorType vecTy = getType();
- if (vecTy.isScalable())
- return emitOpError() << "scalable vectors are not supported";
-
- if (getElements().size() != vecTy.getFixedSize()) {
- return emitOpError() << "operand count of " << getElements().size()
- << " doesn't match vector type " << vecTy
- << " element count of " << vecTy.getFixedSize();
- }
-
- const mlir::Type elementType = vecTy.getElementType();
- for (const mlir::Value element : getElements()) {
- if (element.getType() != elementType) {
- return emitOpError() << "operand type " << element.getType()
- << " doesn't match vector element type "
- << elementType;
- }
- }
-
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// VecExtractOp
-//===----------------------------------------------------------------------===//
-
-OpFoldResult cir::VecExtractOp::fold(FoldAdaptor adaptor) {
- const auto vectorAttr =
- llvm::dyn_cast_if_present<cir::ConstVectorAttr>(adaptor.getVec());
- if (!vectorAttr)
- return {};
-
- const auto indexAttr =
- llvm::dyn_cast_if_present<cir::IntAttr>(adaptor.getIndex());
- if (!indexAttr)
- return {};
-
- const mlir::ArrayAttr elements = vectorAttr.getElts();
- const uint64_t index = indexAttr.getUInt();
- if (index >= elements.size())
- return {};
-
- return elements[index];
-}
-
-//===----------------------------------------------------------------------===//
-// CmpOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::CmpOp::verify() {
- if (getFenvAttr() && !cir::isAnyFloatingPointType(getLhs().getType()))
- return emitOpError()
- << "'fenv' is only valid for floating-point comparisons";
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// VecCmpOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::VecCmpOp::verify() {
- if (getFenvAttr() && !cir::isFPOrVectorOfFPType(getLhs().getType()))
- return emitOpError()
- << "'fenv' is only valid for floating-point comparisons";
- return success();
-}
-
-OpFoldResult cir::VecCmpOp::fold(FoldAdaptor adaptor) {
- // Do not fold when fenv is present.
- // TODO(cir): This is overly conservative. We could fold comparisons as long
- // as we can prove that the operation wouldn't raise exceptions or
- // when the fenv attribute does not require strict exception
- // semantics.
- if (getFenvAttr())
- return {};
-
- auto lhsVecAttr =
- mlir::dyn_cast_if_present<cir::ConstVectorAttr>(adaptor.getLhs());
- auto rhsVecAttr =
- mlir::dyn_cast_if_present<cir::ConstVectorAttr>(adaptor.getRhs());
- if (!lhsVecAttr || !rhsVecAttr)
- return {};
-
- mlir::Type inputElemTy =
- mlir::cast<cir::VectorType>(lhsVecAttr.getType()).getElementType();
- if (!isAnyIntegerOrFloatingPointType(inputElemTy))
- return {};
-
- cir::CmpOpKind opKind = adaptor.getKind();
- mlir::ArrayAttr lhsVecElhs = lhsVecAttr.getElts();
- mlir::ArrayAttr rhsVecElhs = rhsVecAttr.getElts();
- uint64_t vecSize = lhsVecElhs.size();
-
- SmallVector<mlir::Attribute, 16> elements(vecSize);
- bool isIntAttr = vecSize && mlir::isa<cir::IntAttr>(lhsVecElhs[0]);
- bool isUnsignedInt =
- isIntAttr && mlir::cast<cir::IntType>(inputElemTy).isUnsigned();
- for (uint64_t i = 0; i < vecSize; i++) {
- mlir::Attribute lhsAttr = lhsVecElhs[i];
- mlir::Attribute rhsAttr = rhsVecElhs[i];
- bool cmpResult = false;
- switch (opKind) {
- case cir::CmpOpKind::lt: {
- if (isIntAttr) {
- if (isUnsignedInt)
- cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getUInt() <
- mlir::cast<cir::IntAttr>(rhsAttr).getUInt();
- else
- cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getSInt() <
- mlir::cast<cir::IntAttr>(rhsAttr).getSInt();
- } else {
- cmpResult = mlir::cast<cir::FPAttr>(lhsAttr).getValue() <
- mlir::cast<cir::FPAttr>(rhsAttr).getValue();
- }
- break;
- }
- case cir::CmpOpKind::le: {
- if (isIntAttr) {
- if (isUnsignedInt)
- cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getUInt() <=
- mlir::cast<cir::IntAttr>(rhsAttr).getUInt();
- else
- cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getSInt() <=
- mlir::cast<cir::IntAttr>(rhsAttr).getSInt();
- } else {
- cmpResult = mlir::cast<cir::FPAttr>(lhsAttr).getValue() <=
- mlir::cast<cir::FPAttr>(rhsAttr).getValue();
- }
- break;
- }
- case cir::CmpOpKind::gt: {
- if (isIntAttr) {
- if (isUnsignedInt)
- cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getUInt() >
- mlir::cast<cir::IntAttr>(rhsAttr).getUInt();
- else
- cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getSInt() >
- mlir::cast<cir::IntAttr>(rhsAttr).getSInt();
- } else {
- cmpResult = mlir::cast<cir::FPAttr>(lhsAttr).getValue() >
- mlir::cast<cir::FPAttr>(rhsAttr).getValue();
- }
- break;
- }
- case cir::CmpOpKind::ge: {
- if (isIntAttr) {
- if (isUnsignedInt)
- cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getUInt() >=
- mlir::cast<cir::IntAttr>(rhsAttr).getUInt();
- else
- cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getSInt() >=
- mlir::cast<cir::IntAttr>(rhsAttr).getSInt();
- } else {
- cmpResult = mlir::cast<cir::FPAttr>(lhsAttr).getValue() >=
- mlir::cast<cir::FPAttr>(rhsAttr).getValue();
- }
- break;
- }
- case cir::CmpOpKind::eq: {
- if (isIntAttr) {
- cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getSInt() ==
- mlir::cast<cir::IntAttr>(rhsAttr).getSInt();
- } else {
- cmpResult = mlir::cast<cir::FPAttr>(lhsAttr).getValue() ==
- mlir::cast<cir::FPAttr>(rhsAttr).getValue();
- }
- break;
- }
- case cir::CmpOpKind::ne: {
- if (isIntAttr) {
- cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getSInt() !=
- mlir::cast<cir::IntAttr>(rhsAttr).getSInt();
- } else {
- cmpResult = mlir::cast<cir::FPAttr>(lhsAttr).getValue() !=
- mlir::cast<cir::FPAttr>(rhsAttr).getValue();
- }
- break;
- }
- case cir::CmpOpKind::one: {
- llvm::APFloat::cmpResult cr =
- mlir::cast<cir::FPAttr>(lhsAttr).getValue().compare(
- mlir::cast<cir::FPAttr>(rhsAttr).getValue());
- cmpResult =
- cr != llvm::APFloat::cmpUnordered && cr != llvm::APFloat::cmpEqual;
- break;
- }
- case cir::CmpOpKind::uno: {
- cmpResult = mlir::cast<cir::FPAttr>(lhsAttr).getValue().compare(
- mlir::cast<cir::FPAttr>(rhsAttr).getValue()) ==
- llvm::APFloat::cmpUnordered;
- break;
- }
- }
-
- // A true result is all bits set (-1 in two's complement), and a false
- // result is all bits clear. For a 1-bit element type these are the same
- // bit pattern as 1 and 0, respectively.
- elements[i] =
- cir::IntAttr::get(getType().getElementType(), cmpResult ? -1LL : 0LL);
- }
-
- return cir::ConstVectorAttr::get(
- getType(), mlir::ArrayAttr::get(getContext(), elements));
-}
-
-//===----------------------------------------------------------------------===//
-// VecShuffleOp
-//===----------------------------------------------------------------------===//
-
-OpFoldResult cir::VecShuffleOp::fold(FoldAdaptor adaptor) {
- auto vec1Attr =
- mlir::dyn_cast_if_present<cir::ConstVectorAttr>(adaptor.getVec1());
- auto vec2Attr =
- mlir::dyn_cast_if_present<cir::ConstVectorAttr>(adaptor.getVec2());
- if (!vec1Attr || !vec2Attr)
- return {};
-
- mlir::Type vec1ElemTy =
- mlir::cast<cir::VectorType>(vec1Attr.getType()).getElementType();
-
- mlir::ArrayAttr vec1Elts = vec1Attr.getElts();
- mlir::ArrayAttr vec2Elts = vec2Attr.getElts();
- mlir::ArrayAttr indicesElts = adaptor.getIndices();
-
- SmallVector<mlir::Attribute, 16> elements;
- elements.reserve(indicesElts.size());
-
- uint64_t vec1Size = vec1Elts.size();
- for (const auto &idxAttr : indicesElts.getAsRange<cir::IntAttr>()) {
- if (idxAttr.getSInt() == -1) {
- elements.push_back(cir::UndefAttr::get(vec1ElemTy));
- continue;
- }
-
- uint64_t idxValue = idxAttr.getUInt();
- elements.push_back(idxValue < vec1Size ? vec1Elts[idxValue]
- : vec2Elts[idxValue - vec1Size]);
- }
-
- return cir::ConstVectorAttr::get(
- getType(), mlir::ArrayAttr::get(getContext(), elements));
-}
-
-LogicalResult cir::VecShuffleOp::verify() {
- if (getVec1().getType().isScalable() || getVec2().getType().isScalable() ||
- getResult().getType().isScalable())
- return emitOpError() << "scalable vectors are not supported";
-
- // The number of elements in the indices array must match the number of
- // elements in the result type.
- if (getIndices().size() != getResult().getType().getFixedSize()) {
- return emitOpError() << ": the number of elements in " << getIndices()
- << " and " << getResult().getType() << " don't match";
- }
-
- // The element types of the two input vectors and of the result type must
- // match.
- if (getVec1().getType().getElementType() !=
- getResult().getType().getElementType()) {
- return emitOpError() << ": element types of " << getVec1().getType()
- << " and " << getResult().getType() << " don't match";
- }
-
- const uint64_t maxValidIndex = getVec1().getType().getFixedSize() +
- getVec2().getType().getFixedSize() - 1;
- if (llvm::any_of(
- getIndices().getAsRange<cir::IntAttr>(), [&](cir::IntAttr idxAttr) {
- return idxAttr.getSInt() != -1 && idxAttr.getUInt() > maxValidIndex;
- })) {
- return emitOpError() << ": index for __builtin_shufflevector must be "
- "less than the total number of vector elements";
- }
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// VecShuffleDynamicOp
-//===----------------------------------------------------------------------===//
-
-OpFoldResult cir::VecShuffleDynamicOp::fold(FoldAdaptor adaptor) {
- mlir::Attribute vec = adaptor.getVec();
- mlir::Attribute indices = adaptor.getIndices();
- if (mlir::isa_and_nonnull<cir::ConstVectorAttr>(vec) &&
- mlir::isa_and_nonnull<cir::ConstVectorAttr>(indices)) {
- auto vecAttr = mlir::cast<cir::ConstVectorAttr>(vec);
- auto indicesAttr = mlir::cast<cir::ConstVectorAttr>(indices);
-
- mlir::ArrayAttr vecElts = vecAttr.getElts();
- mlir::ArrayAttr indicesElts = indicesAttr.getElts();
-
- const uint64_t numElements = vecElts.size();
-
- SmallVector<mlir::Attribute, 16> elements;
- elements.reserve(numElements);
-
- const uint64_t maskBits = llvm::NextPowerOf2(numElements - 1) - 1;
- for (const auto &idxAttr : indicesElts.getAsRange<cir::IntAttr>()) {
- uint64_t idxValue = idxAttr.getUInt();
- uint64_t newIdx = idxValue & maskBits;
- elements.push_back(vecElts[newIdx]);
- }
-
- return cir::ConstVectorAttr::get(
- getType(), mlir::ArrayAttr::get(getContext(), elements));
- }
-
- return {};
-}
-
-LogicalResult cir::VecShuffleDynamicOp::verify() {
- // The number of elements in the two input vectors must match.
- if (getVec().getType().getSize() !=
- mlir::cast<cir::VectorType>(getIndices().getType()).getSize()) {
- return emitOpError() << ": the number of elements in " << getVec().getType()
- << " and " << getIndices().getType() << " don't match";
- }
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// VecTernaryOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::VecTernaryOp::verify() {
- // Verify that the condition operand has the same number of elements as the
- // other operands. (The automatic verification already checked that all
- // operands are vector types and that the second and third operands are the
- // same type.)
- if (getCond().getType().getSize() != getLhs().getType().getSize()) {
- return emitOpError() << ": the number of elements in "
- << getCond().getType() << " and " << getLhs().getType()
- << " don't match";
- }
- return success();
-}
-
-OpFoldResult cir::VecTernaryOp::fold(FoldAdaptor adaptor) {
- mlir::Attribute cond = adaptor.getCond();
- mlir::Attribute lhs = adaptor.getLhs();
- mlir::Attribute rhs = adaptor.getRhs();
-
- if (!mlir::isa_and_nonnull<cir::ConstVectorAttr>(cond) ||
- !mlir::isa_and_nonnull<cir::ConstVectorAttr>(lhs) ||
- !mlir::isa_and_nonnull<cir::ConstVectorAttr>(rhs))
- return {};
- auto condVec = mlir::cast<cir::ConstVectorAttr>(cond);
- auto lhsVec = mlir::cast<cir::ConstVectorAttr>(lhs);
- auto rhsVec = mlir::cast<cir::ConstVectorAttr>(rhs);
-
- mlir::ArrayAttr condElts = condVec.getElts();
-
- SmallVector<mlir::Attribute, 16> elements;
- elements.reserve(condElts.size());
-
- for (const auto &[idx, condAttr] :
- llvm::enumerate(condElts.getAsRange<cir::IntAttr>())) {
- if (condAttr.getSInt()) {
- elements.push_back(lhsVec.getElts()[idx]);
- } else {
- elements.push_back(rhsVec.getElts()[idx]);
- }
- }
-
- cir::VectorType vecTy = getLhs().getType();
- return cir::ConstVectorAttr::get(
- vecTy, mlir::ArrayAttr::get(getContext(), elements));
-}
-
-//===----------------------------------------------------------------------===//
-// MatrixTransposeOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::MatrixTransposeOp::verify() {
- cir::MatrixType valueTy = getValue().getType();
- cir::MatrixType resultTy = getResult().getType();
-
- if ((valueTy.getElementType() != resultTy.getElementType()) ||
- (valueTy.getNumRows() != resultTy.getNumColumns()) ||
- (valueTy.getNumColumns() != resultTy.getNumRows())) {
- auto expectedTy =
- cir::MatrixType::get(valueTy.getElementType(), valueTy.getNumColumns(),
- valueTy.getNumRows());
- emitOpError() << "operand type " << valueTy << " expects result type of "
- << expectedTy << " but got " << resultTy;
- return failure();
- }
-
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// ComplexCreateOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::ComplexCreateOp::verify() {
- if (getType().getElementType() != getReal().getType()) {
- emitOpError()
- << "operand type of cir.complex.create does not match its result type";
- return failure();
- }
-
- return success();
-}
-
-OpFoldResult cir::ComplexCreateOp::fold(FoldAdaptor adaptor) {
- mlir::Attribute real = adaptor.getReal();
- mlir::Attribute imag = adaptor.getImag();
- if (!real || !imag)
- return {};
-
- // When both of real and imag are constants, we can fold the operation into an
- // `#cir.const_complex` operation.
- auto realAttr = mlir::cast<mlir::TypedAttr>(real);
- auto imagAttr = mlir::cast<mlir::TypedAttr>(imag);
- return cir::ConstComplexAttr::get(realAttr, imagAttr);
-}
-
-//===----------------------------------------------------------------------===//
-// ComplexRealOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::ComplexRealOp::verify() {
- mlir::Type operandTy = getOperand().getType();
- if (auto complexOperandTy = mlir::dyn_cast<cir::ComplexType>(operandTy))
- operandTy = complexOperandTy.getElementType();
-
- if (getType() != operandTy) {
- emitOpError() << ": result type does not match operand type";
- return failure();
- }
-
- return success();
-}
-
-OpFoldResult cir::ComplexRealOp::fold(FoldAdaptor adaptor) {
- if (!mlir::isa<cir::ComplexType>(getOperand().getType()))
- return nullptr;
-
- if (auto complexCreateOp = getOperand().getDefiningOp<cir::ComplexCreateOp>())
- return complexCreateOp.getOperand(0);
-
- auto complex =
- mlir::cast_if_present<cir::ConstComplexAttr>(adaptor.getOperand());
- return complex ? complex.getReal() : nullptr;
-}
-
-//===----------------------------------------------------------------------===//
-// ComplexImagOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::ComplexImagOp::verify() {
- mlir::Type operandTy = getOperand().getType();
- if (auto complexOperandTy = mlir::dyn_cast<cir::ComplexType>(operandTy))
- operandTy = complexOperandTy.getElementType();
-
- if (getType() != operandTy) {
- emitOpError() << ": result type does not match operand type";
- return failure();
- }
-
- return success();
-}
-
-OpFoldResult cir::ComplexImagOp::fold(FoldAdaptor adaptor) {
- if (!mlir::isa<cir::ComplexType>(getOperand().getType()))
- return nullptr;
-
- if (auto complexCreateOp = getOperand().getDefiningOp<cir::ComplexCreateOp>())
- return complexCreateOp.getOperand(1);
-
- auto complex =
- mlir::cast_if_present<cir::ConstComplexAttr>(adaptor.getOperand());
- return complex ? complex.getImag() : nullptr;
-}
-
-//===----------------------------------------------------------------------===//
-// ComplexRealPtrOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::ComplexRealPtrOp::verify() {
- mlir::Type resultPointeeTy = getType().getPointee();
- cir::PointerType operandPtrTy = getOperand().getType();
- auto operandPointeeTy =
- mlir::cast<cir::ComplexType>(operandPtrTy.getPointee());
-
- if (resultPointeeTy != operandPointeeTy.getElementType()) {
- return emitOpError() << ": result type does not match operand type";
- }
-
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// ComplexImagPtrOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::ComplexImagPtrOp::verify() {
- mlir::Type resultPointeeTy = getType().getPointee();
- cir::PointerType operandPtrTy = getOperand().getType();
- auto operandPointeeTy =
- mlir::cast<cir::ComplexType>(operandPtrTy.getPointee());
-
- if (resultPointeeTy != operandPointeeTy.getElementType()) {
- return emitOpError()
- << "cir.complex.imag_ptr result type does not match operand type";
- }
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// Bit manipulation operations
-//===----------------------------------------------------------------------===//
-
-static OpFoldResult
-foldUnaryBitOp(mlir::Attribute inputAttr,
- llvm::function_ref<llvm::APInt(const llvm::APInt &)> func,
- bool poisonZero = false) {
- if (mlir::isa_and_present<cir::PoisonAttr>(inputAttr)) {
- // Propagate poison value
- return inputAttr;
- }
-
- auto input = mlir::dyn_cast_if_present<IntAttr>(inputAttr);
- if (!input)
- return nullptr;
-
- llvm::APInt inputValue = input.getValue();
- if (poisonZero && inputValue.isZero())
- return cir::PoisonAttr::get(input.getType());
-
- llvm::APInt resultValue = func(inputValue);
- return IntAttr::get(input.getType(), resultValue);
-}
-
-OpFoldResult BitClrsbOp::fold(FoldAdaptor adaptor) {
- return foldUnaryBitOp(adaptor.getInput(), [](const llvm::APInt &inputValue) {
- unsigned resultValue =
- inputValue.getBitWidth() - inputValue.getSignificantBits();
- return llvm::APInt(inputValue.getBitWidth(), resultValue);
- });
-}
-
-OpFoldResult BitClzOp::fold(FoldAdaptor adaptor) {
- return foldUnaryBitOp(
- adaptor.getInput(),
- [](const llvm::APInt &inputValue) {
- unsigned resultValue = inputValue.countLeadingZeros();
- return llvm::APInt(inputValue.getBitWidth(), resultValue);
- },
- getPoisonZero());
-}
-
-OpFoldResult BitCtzOp::fold(FoldAdaptor adaptor) {
- return foldUnaryBitOp(
- adaptor.getInput(),
- [](const llvm::APInt &inputValue) {
- return llvm::APInt(inputValue.getBitWidth(),
- inputValue.countTrailingZeros());
- },
- getPoisonZero());
-}
-
-OpFoldResult BitFfsOp::fold(FoldAdaptor adaptor) {
- return foldUnaryBitOp(adaptor.getInput(), [](const llvm::APInt &inputValue) {
- unsigned trailingZeros = inputValue.countTrailingZeros();
- unsigned result =
- trailingZeros == inputValue.getBitWidth() ? 0 : trailingZeros + 1;
- return llvm::APInt(inputValue.getBitWidth(), result);
- });
-}
-
-OpFoldResult BitParityOp::fold(FoldAdaptor adaptor) {
- return foldUnaryBitOp(adaptor.getInput(), [](const llvm::APInt &inputValue) {
- return llvm::APInt(inputValue.getBitWidth(), inputValue.popcount() % 2);
- });
-}
-
-OpFoldResult BitPopcountOp::fold(FoldAdaptor adaptor) {
- return foldUnaryBitOp(adaptor.getInput(), [](const llvm::APInt &inputValue) {
- return llvm::APInt(inputValue.getBitWidth(), inputValue.popcount());
- });
-}
-
-OpFoldResult BitReverseOp::fold(FoldAdaptor adaptor) {
- return foldUnaryBitOp(adaptor.getInput(), [](const llvm::APInt &inputValue) {
- return inputValue.reverseBits();
- });
-}
-
-OpFoldResult ByteSwapOp::fold(FoldAdaptor adaptor) {
- return foldUnaryBitOp(adaptor.getInput(), [](const llvm::APInt &inputValue) {
- return inputValue.byteSwap();
- });
-}
-
-OpFoldResult RotateOp::fold(FoldAdaptor adaptor) {
- if (mlir::isa_and_present<cir::PoisonAttr>(adaptor.getInput()) ||
- mlir::isa_and_present<cir::PoisonAttr>(adaptor.getAmount())) {
- // Propagate poison values
- return cir::PoisonAttr::get(getType());
- }
-
- auto input = mlir::dyn_cast_if_present<IntAttr>(adaptor.getInput());
- auto amount = mlir::dyn_cast_if_present<IntAttr>(adaptor.getAmount());
- if (!input && !amount)
- return nullptr;
-
- // We could fold cir.rotate even if one of its two operands is not a constant:
- // - `cir.rotate left/right %0, 0` could be folded into just %0 even if %0
- // is not a constant.
- // - `cir.rotate left/right 0/0b111...111, %0` could be folded into 0 or
- // 0b111...111 even if %0 is not a constant.
-
- llvm::APInt inputValue;
- if (input) {
- inputValue = input.getValue();
- if (inputValue.isZero() || inputValue.isAllOnes()) {
- // An input value of all 0s or all 1s will not change after rotation
- return input;
- }
- }
-
- uint64_t amountValue;
- if (amount) {
- amountValue = amount.getValue().urem(getInput().getType().getWidth());
- if (amountValue == 0) {
- // A shift amount of 0 will not change the input value
- return getInput();
- }
- }
-
- if (!input || !amount)
- return nullptr;
-
- assert(inputValue.getBitWidth() == getInput().getType().getWidth() &&
- "input value must have the same bit width as the input type");
-
- llvm::APInt resultValue;
- if (isRotateLeft())
- resultValue = inputValue.rotl(amountValue);
- else
- resultValue = inputValue.rotr(amountValue);
-
- return IntAttr::get(input.getContext(), input.getType(), resultValue);
-}
-
-//===----------------------------------------------------------------------===//
-// InlineAsmOp
-//===----------------------------------------------------------------------===//
-
-void cir::InlineAsmOp::print(OpAsmPrinter &p) {
- p << '(' << getAsmFlavor() << ", ";
- p.increaseIndent();
- p.printNewline();
-
- llvm::SmallVector<std::string, 3> names{"out", "in", "in_out"};
- auto *nameIt = names.begin();
- auto *attrIt = getOperandAttrs().begin();
-
- for (mlir::OperandRange ops : getAsmOperands()) {
- p << *nameIt << " = ";
-
- p << '[';
- llvm::interleaveComma(llvm::make_range(ops.begin(), ops.end()), p,
- [&](Value value) {
- p.printOperand(value);
- p << " : " << value.getType();
- if (mlir::isa<mlir::UnitAttr>(*attrIt))
- p << " (maybe_memory)";
- attrIt++;
- });
- p << "],";
- p.printNewline();
- ++nameIt;
- }
-
- p << "{";
- p.printString(getAsmString());
- p << " ";
- p.printString(getConstraints());
- p << "}";
- p.decreaseIndent();
- p << ')';
- if (getSideEffects())
- p << " side_effects";
-
- p.printOptionalAttrDict(
- getOperation()->getDiscardableAttrDictionary().getValue());
-
- if (auto v = getRes())
- p << " -> " << v.getType();
-}
-
-void cir::InlineAsmOp::build(OpBuilder &odsBuilder, OperationState &odsState,
- ArrayRef<ValueRange> asmOperands,
- StringRef asmString, StringRef constraints,
- bool sideEffects, cir::AsmFlavor asmFlavor,
- ArrayRef<Attribute> operandAttrs) {
- // Set up the operands_segments for VariadicOfVariadic
- SmallVector<int32_t> segments;
- for (auto operandRange : asmOperands) {
- segments.push_back(operandRange.size());
- odsState.addOperands(operandRange);
- }
-
- odsState.addAttribute(
- "operands_segments",
- DenseI32ArrayAttr::get(odsBuilder.getContext(), segments));
- odsState.addAttribute("asm_string", odsBuilder.getStringAttr(asmString));
- odsState.addAttribute("constraints", odsBuilder.getStringAttr(constraints));
- odsState.addAttribute("asm_flavor",
- AsmFlavorAttr::get(odsBuilder.getContext(), asmFlavor));
-
- if (sideEffects)
- odsState.addAttribute("side_effects", odsBuilder.getUnitAttr());
-
- odsState.addAttribute("operand_attrs", odsBuilder.getArrayAttr(operandAttrs));
-}
-
-ParseResult cir::InlineAsmOp::parse(OpAsmParser &parser,
- OperationState &result) {
- llvm::SmallVector<mlir::Attribute> operandAttrs;
- llvm::SmallVector<int32_t> operandsGroupSizes;
- std::string asmString, constraints;
- Type resType;
- MLIRContext *ctxt = parser.getBuilder().getContext();
-
- auto error = [&](const Twine &msg) -> LogicalResult {
- return parser.emitError(parser.getCurrentLocation(), msg);
- };
-
- auto expected = [&](const std::string &c) {
- return error("expected '" + c + "'");
- };
-
- if (parser.parseLParen().failed())
- return expected("(");
-
- auto flavor = FieldParser<AsmFlavor, AsmFlavor>::parse(parser);
- if (failed(flavor))
- return error("Unknown AsmFlavor");
-
- if (parser.parseComma().failed())
- return expected(",");
-
- auto parseValue = [&](Value &v) {
- OpAsmParser::UnresolvedOperand op;
-
- if (parser.parseOperand(op) || parser.parseColon())
- return error("can't parse operand");
-
- Type typ;
- if (parser.parseType(typ).failed())
- return error("can't parse operand type");
- llvm::SmallVector<mlir::Value> tmp;
- if (parser.resolveOperand(op, typ, tmp))
- return error("can't resolve operand");
- v = tmp[0];
- return mlir::success();
- };
-
- auto parseOperands = [&](llvm::StringRef name) {
- if (parser.parseKeyword(name).failed())
- return error("expected " + name + " operands here");
- if (parser.parseEqual().failed())
- return expected("=");
- if (parser.parseLSquare().failed())
- return expected("[");
-
- int size = 0;
- if (parser.parseOptionalRSquare().succeeded()) {
- operandsGroupSizes.push_back(size);
- if (parser.parseComma())
- return expected(",");
- return mlir::success();
- }
-
- auto parseOperand = [&]() {
- Value val;
- if (parseValue(val).succeeded()) {
- result.operands.push_back(val);
- size++;
-
- if (parser.parseOptionalLParen().failed()) {
- operandAttrs.push_back(mlir::DictionaryAttr::get(ctxt));
- return mlir::success();
- }
-
- if (parser.parseKeyword("maybe_memory").succeeded()) {
- operandAttrs.push_back(mlir::UnitAttr::get(ctxt));
- if (parser.parseRParen())
- return expected(")");
- return mlir::success();
- } else {
- return expected("maybe_memory");
- }
- }
- return mlir::failure();
- };
-
- if (parser.parseCommaSeparatedList(parseOperand).failed())
- return mlir::failure();
-
- if (parser.parseRSquare().failed() || parser.parseComma().failed())
- return expected("]");
- operandsGroupSizes.push_back(size);
- return mlir::success();
- };
-
- if (parseOperands("out").failed() || parseOperands("in").failed() ||
- parseOperands("in_out").failed())
- return error("failed to parse operands");
-
- if (parser.parseLBrace())
- return expected("{");
- if (parser.parseString(&asmString))
- return error("asm string parsing failed");
- if (parser.parseString(&constraints))
- return error("constraints string parsing failed");
- if (parser.parseRBrace())
- return expected("}");
- if (parser.parseRParen())
- return expected(")");
-
- if (parser.parseOptionalKeyword("side_effects").succeeded())
- result.attributes.set("side_effects", UnitAttr::get(ctxt));
-
- if (parser.parseOptionalAttrDict(result.attributes).failed())
- return mlir::failure();
-
- if (parser.parseOptionalArrow().succeeded() &&
- parser.parseType(resType).failed())
- return mlir::failure();
-
- result.attributes.set("asm_flavor", AsmFlavorAttr::get(ctxt, *flavor));
- result.attributes.set("asm_string", StringAttr::get(ctxt, asmString));
- result.attributes.set("constraints", StringAttr::get(ctxt, constraints));
- result.attributes.set("operand_attrs", ArrayAttr::get(ctxt, operandAttrs));
- result.getOrAddProperties<InlineAsmOp::Properties>().operands_segments =
- parser.getBuilder().getDenseI32ArrayAttr(operandsGroupSizes);
- if (resType)
- result.addTypes(TypeRange{resType});
-
- return mlir::success();
-}
-
-void InlineAsmOp::getEffects(
- llvm::SmallVectorImpl<mlir::MemoryEffects::EffectInstance> &effects) {
- // If we have any side effects (that is, we're volatile asm), add a read and
- // write memory effect. We do this the same as the llvm dialect InlineAsmOp.
- if (getSideEffects()) {
- effects.emplace_back(mlir::MemoryEffects::Read::get());
- effects.emplace_back(mlir::MemoryEffects::Write::get());
- }
-}
-
-//===----------------------------------------------------------------------===//
-// ThrowOp / TryThrowOp
-//===----------------------------------------------------------------------===//
-
-template <typename ThrowOpTy>
-static mlir::LogicalResult verifyThrowOpImpl(ThrowOpTy op) {
- if (op.rethrows())
- return mlir::success();
-
- if (op.getNumOperands() != 0) {
- if (op.getTypeInfo())
- return mlir::success();
- return op.emitOpError() << "'type_info' symbol attribute missing";
- }
-
- return mlir::failure();
-}
-
-mlir::LogicalResult cir::ThrowOp::verify() { return verifyThrowOpImpl(*this); }
-
-mlir::LogicalResult cir::TryThrowOp::verify() {
- return verifyThrowOpImpl(*this);
-}
-
-//===----------------------------------------------------------------------===//
-// AtomicFetchOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::AtomicFetchOp::verify() {
- if (getBinop() != cir::AtomicFetchKind::Add &&
- getBinop() != cir::AtomicFetchKind::Sub &&
- getBinop() != cir::AtomicFetchKind::Max &&
- getBinop() != cir::AtomicFetchKind::Min &&
- getBinop() != cir::AtomicFetchKind::Maximum &&
- getBinop() != cir::AtomicFetchKind::Minimum &&
- getBinop() != cir::AtomicFetchKind::MaximumNum &&
- getBinop() != cir::AtomicFetchKind::MinimumNum &&
- !mlir::isa<cir::IntType>(getVal().getType()))
- return emitError("only atomic add, sub, max, min, maximum, minimum, "
- "maximum_num, and minimum_num operation could operate on "
- "floating-point values");
-
- if ((getBinop() == cir::AtomicFetchKind::Maximum ||
- getBinop() == cir::AtomicFetchKind::Minimum ||
- getBinop() == cir::AtomicFetchKind::MaximumNum ||
- getBinop() == cir::AtomicFetchKind::MinimumNum) &&
- !mlir::isa<cir::FPTypeInterface>(getVal().getType()))
- return emitError("atomic maximum, minimum, maximum_num, and minimum_num "
- "operation could only operate on floating-point values");
-
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// TypeInfoAttr
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::TypeInfoAttr::verify(
- ::llvm::function_ref<::mlir::InFlightDiagnostic()> emitError,
- ::mlir::Type type, ::mlir::ArrayAttr typeInfoData) {
-
- if (cir::ConstRecordAttr::verify(emitError, type, typeInfoData).failed())
- return failure();
-
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// TryOp
-//===----------------------------------------------------------------------===//
-
-void cir::TryOp::getSuccessorRegions(
- mlir::RegionBranchPoint point,
- llvm::SmallVectorImpl<mlir::RegionSuccessor> ®ions) {
- if (point.isParent()) {
- regions.emplace_back(&getTryRegion());
- } else {
- // A handler exit leaves the operation.
- mlir::Operation *term = point.getTerminatorPredecessorOrNull();
- if (term->getParentRegion() != &getTryRegion()) {
- regions.emplace_back(getOperation());
- return;
- }
- // A cir.resume ending the try region unwinds into a handler, so it leaves
- // the operation only when there is none. Every other exit can leave it.
- if (!isa<cir::ResumeOp>(term) || getHandlerRegions().empty())
- regions.emplace_back(getOperation());
- }
-
- // Most handlers are entered by an unwind out of a call in the try region,
- // which is not a branch point, so the handlers are successors of the parent
- // and of every exit from the try region to stand in for it.
- //
- // TODO(CIR): If we know a target function never throws a specific type, we
- // can remove the catch handler.
- regions.reserve(regions.size() + getHandlerRegions().size());
- for (mlir::Region &handlerRegion : getHandlerRegions())
- regions.emplace_back(&handlerRegion);
-}
-
-/// Verify that each cir.init_catch_param in \p handlerRegion, outside any
-/// nested cir.try, agrees with the handler's leading cir.construct_catch_param
-/// \p constructOp, which may be null. A cir.init_catch_param of kind
-/// reference_to_pointer or non_trivial_copy needs \p constructOp on its
-/// parameter, and a cir.init_catch_param on the parameter of \p constructOp
-/// must have the kind of \p constructOp.
-static LogicalResult
-verifyCatchParamPairing(mlir::Region &handlerRegion,
- cir::ConstructCatchParamOp constructOp) {
- mlir::WalkResult result =
- handlerRegion.walk<mlir::WalkOrder::PreOrder>([&](mlir::Operation *op) {
- if (mlir::isa<cir::TryOp>(op))
- return mlir::WalkResult::skip();
- auto initOp = mlir::dyn_cast<cir::InitCatchParamOp>(op);
- if (!initOp)
- return mlir::WalkResult::advance();
-
- cir::InitCatchKind kind = initOp.getKind();
- bool sameParam =
- constructOp && constructOp.getParamAddr() == initOp.getParamAddr();
- if (sameParam && constructOp.getKind() != kind) {
- initOp.emitOpError("kind '")
- << cir::stringifyInitCatchKind(kind)
- << "' does not match the kind '"
- << cir::stringifyInitCatchKind(constructOp.getKind())
- << "' of the preceding 'cir.construct_catch_param'";
- return mlir::WalkResult::interrupt();
- }
- if ((kind == cir::InitCatchKind::ReferenceToPointer ||
- kind == cir::InitCatchKind::NonTrivialCopy) &&
- !sameParam) {
- initOp.emitOpError("'")
- << cir::stringifyInitCatchKind(kind)
- << "' requires a preceding 'cir.construct_catch_param' of the "
- "same kind on the same parameter";
- return mlir::WalkResult::interrupt();
- }
- return mlir::WalkResult::advance();
- });
- return failure(result.wasInterrupted());
-}
-
-LogicalResult cir::TryOp::verifyRegions() {
- mlir::ArrayAttr handlerTypes = getHandlerTypes();
- if (!handlerTypes) {
- if (!getHandlerRegions().empty())
- return emitOpError(
- "handler regions must be empty when no handler types are present");
- return success();
- }
-
- mlir::MutableArrayRef<mlir::Region> handlerRegions = getHandlerRegions();
-
- // The parser and builder won't allow this to happen, but the loop below
- // relies on the sizes being the same, so we check it here.
- if (handlerRegions.size() != handlerTypes.size())
- return emitOpError(
- "number of handler regions and handler types must match");
-
- // A filter handler and an unexpected handler together implement a dynamic
- // exception specification. The filter try operation wraps the whole function
- // body and exists only to check the specification, so the two must appear
- // together and must stand alone. A catch-all in the same list would make the
- // filter unreachable, and the filter handler already provides the
- // continue-unwinding path that an unwind handler would supply.
- if (llvm::any_of(handlerTypes, [](mlir::Attribute typeAttr) {
- return mlir::isa<cir::EhFilterAttr, cir::EhUnexpectedAttr>(typeAttr);
- })) {
- if (handlerTypes.size() != 2 ||
- !mlir::isa<cir::EhFilterAttr>(handlerTypes[0]) ||
- !mlir::isa<cir::EhUnexpectedAttr>(handlerTypes[1]))
- return emitOpError("a filter handler must be followed by an unexpected "
- "handler, and the two must be the only handlers");
- }
-
- for (const auto &[typeAttr, handlerRegion] :
- llvm::zip(handlerTypes, handlerRegions)) {
- // Verify that handler regions have a !cir.eh_token block argument.
- mlir::Block &entryBlock = handlerRegion.front();
- if (entryBlock.getNumArguments() != 1 ||
- !mlir::isa<cir::EhTokenType>(entryBlock.getArgument(0).getType()))
- return emitOpError(
- "handler region must have a single '!cir.eh_token' argument");
-
- // The unwind, filter and unexpected regions do not require a
- // cir.begin_catch. None of them catches the exception.
- if (mlir::isa<cir::UnwindAttr, cir::EhFilterAttr, cir::EhUnexpectedAttr>(
- typeAttr))
- continue;
-
- if (entryBlock.empty())
- return emitOpError("catch handler region must not be empty");
-
- // A terminate scope, which wraps the body of a function that cannot throw,
- // is a catch-all handler that only terminates the program. The exception is
- // caught by the runtime helper that cir.eh.terminate lowers to, so the
- // handler region has no cir.begin_catch of its own.
- if (mlir::isa<cir::EhTerminateOp>(entryBlock.front())) {
- if (!mlir::isa<cir::CatchAllAttr>(typeAttr))
- return emitOpError("'cir.eh.terminate' is only allowed in a catch-all "
- "handler region");
- continue;
- }
-
- // Nothing may run in a catch handler before cir.begin_catch, so it has to
- // be the handler region's first operation, with two exceptions.
- //
- // When lifetime markers are enabled, the catch parameter's storage can be
- // marked by a cir.lifetime.start. That parameter is the only variable
- // whose lifetime begins before the catch is entered, so there is at most
- // one such marker. Its lifetime-end cleanup has to run after the catch
- // handler is finished (or exited by an exception unwind), so if there is a
- // lifetime begin marker, it is followed by a cir.cleanup.scope that
- // encloses the the rest of the handler with a cir.lifetime.end in its
- // cleanup region.
- //
- // A cir.construct_catch_param may also precede cir.begin_catch, to
- // perform any pre-begin_catch initialization of the catch parameter.
- mlir::Operation *firstOp = &entryBlock.front();
- if (mlir::isa<cir::LifetimeStartOp>(firstOp)) {
- mlir::Operation *next = firstOp->getNextNode();
- auto lifetimeScope = mlir::dyn_cast_if_present<cir::CleanupScopeOp>(next);
- if (!lifetimeScope)
- return emitOpError("'cir.lifetime.start' in a catch handler region "
- "must be followed by the 'cir.cleanup.scope' of "
- "its lifetime-end cleanup");
- if (lifetimeScope.getBodyRegion().empty())
- return emitOpError(
- "'cir.lifetime.start' in a catch handler region must be "
- "followed by the 'cir.cleanup.scope' of its lifetime-end "
- "cleanup");
- mlir::Block &scopeBody = lifetimeScope.getBodyRegion().front();
- firstOp = scopeBody.empty() ? nullptr : &scopeBody.front();
- }
-
- auto constructOp =
- mlir::dyn_cast_if_present<cir::ConstructCatchParamOp>(firstOp);
- if (constructOp)
- firstOp = firstOp->getNextNode();
- if (!mlir::isa_and_present<cir::BeginCatchOp>(firstOp))
- return emitOpError(
- "catch handler region must start with 'cir.begin_catch'");
-
- if (failed(verifyCatchParamPairing(handlerRegion, constructOp)))
- return failure();
- }
-
- return success();
-}
-
-void cir::printTryHandlerRegions(
- mlir::OpAsmPrinter &printer, cir::TryOp op,
- mlir::MutableArrayRef<mlir::Region> handlerRegions,
- mlir::ArrayAttr handlerTypes) {
- if (!handlerTypes)
- return;
-
- for (const auto [typeIdx, typeAttr] : llvm::enumerate(handlerTypes)) {
- if (typeIdx)
- printer << " ";
-
- if (mlir::isa<cir::CatchAllAttr>(typeAttr)) {
- printer << "catch all ";
- } else if (mlir::isa<cir::UnwindAttr>(typeAttr)) {
- printer << "unwind ";
- } else if (auto filterAttr = mlir::dyn_cast<cir::EhFilterAttr>(typeAttr)) {
- printer << "filter [";
- llvm::interleaveComma(
- filterAttr.getPermittedTypes(), printer,
- [&](mlir::Attribute sym) { printer.printAttribute(sym); });
- printer << "] ";
- } else if (mlir::isa<cir::EhUnexpectedAttr>(typeAttr)) {
- printer << "unexpected ";
- } else {
- printer << "catch [type ";
- printer.printAttribute(typeAttr);
- printer << "] ";
- }
-
- // Print the handler region's !cir.eh_token block argument.
- mlir::Region ®ion = handlerRegions[typeIdx];
- if (!region.empty() && region.front().getNumArguments() > 0) {
- printer << "(";
- printer.printRegionArgument(region.front().getArgument(0));
- printer << ") ";
- }
-
- printer.printRegion(region,
- /*printEntryBLockArgs=*/false,
- /*printBlockTerminators=*/true);
- }
-}
-
-mlir::ParseResult cir::parseTryHandlerRegions(
- mlir::OpAsmParser &parser,
- llvm::SmallVectorImpl<std::unique_ptr<mlir::Region>> &handlerRegions,
- mlir::ArrayAttr &handlerTypes) {
-
- auto parseCheckedCatcherRegion = [&]() -> mlir::ParseResult {
- handlerRegions.emplace_back(new mlir::Region);
-
- mlir::Region &currRegion = *handlerRegions.back();
-
- // Parse the required region argument: (%eh_token : !cir.eh_token)
- llvm::SmallVector<mlir::OpAsmParser::Argument> regionArgs;
- if (parser.parseLParen())
- return failure();
- mlir::OpAsmParser::Argument arg;
- if (parser.parseArgument(arg, /*allowType=*/true))
- return failure();
- regionArgs.push_back(arg);
- if (parser.parseRParen())
- return failure();
-
- mlir::SMLoc regionLoc = parser.getCurrentLocation();
- if (parser.parseRegion(currRegion, regionArgs)) {
- handlerRegions.clear();
- return failure();
- }
-
- if (currRegion.empty())
- return parser.emitError(regionLoc, "handler region shall not be empty");
-
- if (!(currRegion.back().mightHaveTerminator() &&
- currRegion.back().getTerminator()))
- return parser.emitError(
- regionLoc, "blocks are expected to be explicitly terminated");
-
- return success();
- };
-
- bool hasCatchAll = false;
- llvm::SmallVector<mlir::Attribute, 4> catcherAttrs;
- while (parser.parseOptionalKeyword("catch").succeeded()) {
- bool hasLSquare = parser.parseOptionalLSquare().succeeded();
-
- llvm::StringRef attrStr;
- if (parser.parseOptionalKeyword(&attrStr, {"all", "type"}).failed())
- return parser.emitError(parser.getCurrentLocation(),
- "expected 'all' or 'type' keyword");
-
- bool isCatchAll = attrStr == "all";
- if (isCatchAll) {
- if (hasCatchAll)
- return parser.emitError(parser.getCurrentLocation(),
- "can't have more than one catch all");
- hasCatchAll = true;
- }
-
- mlir::Attribute exceptionRTTIAttr;
- if (!isCatchAll && parser.parseAttribute(exceptionRTTIAttr).failed())
- return parser.emitError(parser.getCurrentLocation(),
- "expected valid RTTI info attribute");
-
- catcherAttrs.push_back(isCatchAll
- ? cir::CatchAllAttr::get(parser.getContext())
- : exceptionRTTIAttr);
-
- if (hasLSquare && isCatchAll)
- return parser.emitError(parser.getCurrentLocation(),
- "catch all dosen't need RTTI info attribute");
-
- if (hasLSquare && parser.parseRSquare().failed())
- return parser.emitError(parser.getCurrentLocation(),
- "expected `]` after RTTI info attribute");
-
- if (parseCheckedCatcherRegion().failed())
- return mlir::failure();
- }
-
- // A filter handler carries the type info symbols permitted by the enclosing
- // function's dynamic exception specification. TryOp::verifyRegions enforces
- // that it is paired with an unexpected handler and that the two stand alone.
- if (parser.parseOptionalKeyword("filter").succeeded()) {
- mlir::SMLoc filterLoc = parser.getCurrentLocation();
- llvm::SmallVector<mlir::Attribute, 4> permittedTypes;
- auto parsePermittedType = [&]() -> mlir::ParseResult {
- mlir::SMLoc typeLoc = parser.getCurrentLocation();
- mlir::Attribute rtti;
- if (parser.parseAttribute(rtti).failed())
- return mlir::failure();
- if (!mlir::isa<cir::GlobalViewAttr>(rtti))
- return parser.emitError(typeLoc, "expected a type info symbol naming a "
- "permitted exception type");
- permittedTypes.push_back(rtti);
- return mlir::success();
- };
- if (parser
- .parseCommaSeparatedList(mlir::OpAsmParser::Delimiter::Square,
- parsePermittedType)
- .failed())
- return mlir::failure();
-
- auto filterAttr = cir::EhFilterAttr::getChecked(
- [&]() { return parser.emitError(filterLoc); }, parser.getContext(),
- parser.getBuilder().getArrayAttr(permittedTypes));
- if (!filterAttr)
- return mlir::failure();
- catcherAttrs.push_back(filterAttr);
- if (parseCheckedCatcherRegion().failed())
- return mlir::failure();
- }
-
- if (parser.parseOptionalKeyword("unexpected").succeeded()) {
- catcherAttrs.push_back(cir::EhUnexpectedAttr::get(parser.getContext()));
- if (parseCheckedCatcherRegion().failed())
- return mlir::failure();
- }
-
- if (parser.parseOptionalKeyword("unwind").succeeded()) {
- if (hasCatchAll)
- return parser.emitError(parser.getCurrentLocation(),
- "unwind can't be used with catch all");
-
- catcherAttrs.push_back(cir::UnwindAttr::get(parser.getContext()));
- if (parseCheckedCatcherRegion().failed())
- return mlir::failure();
- }
-
- handlerTypes = parser.getBuilder().getArrayAttr(catcherAttrs);
- return mlir::success();
-}
-
-//===----------------------------------------------------------------------===//
-// EhTypeIdOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult
-cir::EhTypeIdOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
- Operation *op = symbolTable.lookupNearestSymbolFrom(*this, getTypeSymAttr());
- if (!isa_and_nonnull<GlobalOp>(op))
- return emitOpError("'")
- << getTypeSym() << "' does not reference a valid cir.global";
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// LifetimeStartOp & LifetimeEndOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::LifetimeStartOp::verify() {
- return verifyProducedBy<cir::AllocaOp>(*this, getPtr(), "ptr");
-}
-
-LogicalResult cir::LifetimeEndOp::verify() {
- return verifyProducedBy<cir::AllocaOp>(*this, getPtr(), "ptr");
-}
-
-//===----------------------------------------------------------------------===//
-// MemChrOp
-//===----------------------------------------------------------------------===//
-
-/// Reads a fundamental integer width from a signless i32 attribute.
-static std::optional<unsigned> getRecordedIntegerWidth(mlir::Attribute attr) {
- auto intAttr = mlir::dyn_cast<mlir::IntegerAttr>(attr);
- if (!intAttr || !intAttr.getType().isSignlessInteger(32))
- return std::nullopt;
- int64_t width = intAttr.getInt();
- if (width < 0 ||
- !cir::isValidFundamentalIntWidth(static_cast<unsigned>(width)))
- return std::nullopt;
- return static_cast<unsigned>(width);
-}
-
-LogicalResult cir::MemChrOp::verify() {
- auto moduleOp = (*this)->getParentOfType<mlir::ModuleOp>();
- if (!moduleOp)
- return emitOpError("expects an enclosing module");
-
- // libc memchr uses pointers in the target's default address space.
- if (mlir::cast<cir::PointerType>(getSrc().getType()).getAddrSpace())
- return emitOpError("src must be in the default address space");
-
- auto checkWidth = [&](cir::IntType type, llvm::StringRef operandName,
- llvm::StringRef attrName) -> LogicalResult {
- mlir::Attribute attr = moduleOp->getAttr(attrName);
- if (!attr)
- return emitOpError("expects the module to record ") << attrName;
- std::optional<unsigned> width = getRecordedIntegerWidth(attr);
- if (!width)
- return emitOpError("requires ")
- << attrName
- << " to be a signless i32 holding a fundamental integer width";
- if (type.getWidth() != *width)
- return emitOpError() << operandName << " must have the width recorded in "
- << attrName;
- return success();
- };
-
- if (failed(checkWidth(getPattern().getType(), "pattern",
- cir::CIRDialect::getIntTypeWidthAttrName())))
- return failure();
- return checkWidth(getLen().getType(), "len",
- cir::CIRDialect::getSizeTypeWidthAttrName());
-}
-
-//===----------------------------------------------------------------------===//
-// InitCatchParamOp
-//===----------------------------------------------------------------------===//
-
-/// For the reference_to_pointer and reference_to_record_pointer kinds, emit an
-/// error on \p op unless \p paramAddrType is the type of the address of a
-/// reference to a pointer. Any other \p kind is accepted.
-static LogicalResult verifyCatchParamAddr(mlir::Operation *op,
- cir::InitCatchKind kind,
- cir::PointerType paramAddrType) {
- if (kind != cir::InitCatchKind::ReferenceToPointer &&
- kind != cir::InitCatchKind::ReferenceToRecordPointer)
- return success();
-
- auto refType = mlir::dyn_cast<cir::PointerType>(paramAddrType.getPointee());
- if (!refType || !mlir::isa<cir::PointerType>(refType.getPointee()))
- return op->emitOpError("'")
- << cir::stringifyInitCatchKind(kind)
- << "' requires 'param_addr' to be the address of a reference to a "
- "pointer";
- return success();
-}
-
-LogicalResult cir::InitCatchParamOp::verify() {
- return verifyCatchParamAddr(*this, getKind(), getParamAddr().getType());
-}
-
-//===----------------------------------------------------------------------===//
-// ConstructCatchParamOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::ConstructCatchParamOp::verify() {
- return verifyCatchParamAddr(*this, getKind(), getParamAddr().getType());
-}
-
-LogicalResult cir::ConstructCatchParamOp::verifySymbolUses(
- SymbolTableCollection &symbolTable) {
- auto copyFnAttr = getCopyFnAttr();
- if (!copyFnAttr)
- return success();
- auto fn =
- symbolTable.lookupNearestSymbolFrom<cir::FuncOp>(*this, getCopyFnAttr());
- if (!fn)
- return emitOpError("'")
- << *getCopyFn() << "' does not reference a valid cir.func";
-
- if (!fn->hasAttr(cir::CIRDialect::getCatchCopyThunkAttrName()))
- return emitOpError("catch-init copy_fn must be tagged with the ")
- << cir::CIRDialect::getCatchCopyThunkAttrName() << " attribute";
-
- cir::FuncType fnType = fn.getFunctionType();
- if (fnType.getNumInputs() != 2 || !fnType.hasVoidReturn())
- return emitOpError("catch-init copy_fn must take two pointer arguments and "
- "return void");
-
- if (fnType.getInput(0) != getParamAddr().getType())
- return emitOpError("first argument of catch-init copy_fn must match the "
- "type of 'param_addr'");
-
- if (fnType.getInput(1) != getParamAddr().getType())
- return emitOpError(
- "second argument of catch-init copy_fn must be a pointer "
- "to the catch type");
-
- return success();
-}
-
-//===----------------------------------------------------------------------===//
-// EhDispatchOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult cir::EhDispatchOp::verify() {
- mlir::ArrayAttr handlerTypes = getCatchTypesAttr();
- if (!handlerTypes)
- return success();
-
- bool hasFilter = false;
- for (mlir::Attribute typeAttr : handlerTypes) {
- if (!mlir::isa<cir::EhFilterAttr>(typeAttr))
- continue;
- if (hasFilter)
- return emitOpError("can't have more than one 'filter' handler");
- hasFilter = true;
- }
-
- // Unlike the other handlers, a filter does not take the place of the default
- // destination. Its destination is taken when the exception is not permitted
- // by the specification, and the default 'unwind' destination is taken when
- // it is. A catch-all default would leave the filter unreachable.
- if (hasFilter && getDefaultIsCatchAll())
- return emitOpError(
- "'filter' handler requires an 'unwind' default destination");
-
- return success();
-}
-
-ParseResult cir::parseEhDispatchDestinations(
- OpAsmParser &parser, mlir::ArrayAttr &catchTypes,
- SmallVectorImpl<Block *> &catchDestinations, Block *&defaultDestination,
- mlir::UnitAttr &defaultIsCatchAll) {
- // Parse: [ ... ]
- if (parser.parseLSquare())
- return failure();
-
- SmallVector<Attribute> handlerTypes;
- bool hasCatchAll = false;
- bool hasUnwind = false;
-
- // Parse handler list.
- auto parseHandler = [&]() -> ParseResult {
- // Check for 'catch_all' or 'unwind' keywords.
- if (succeeded(parser.parseOptionalKeyword("catch_all"))) {
- if (hasCatchAll)
- return parser.emitError(parser.getCurrentLocation(),
- "duplicate 'catch_all' handler");
- if (hasUnwind)
- return parser.emitError(parser.getCurrentLocation(),
- "cannot have both 'catch_all' and 'unwind'");
- hasCatchAll = true;
-
- if (parser.parseColon().failed())
- return failure();
-
- if (parser.parseSuccessor(defaultDestination).failed())
- return failure();
-
- return success();
- }
-
- if (succeeded(parser.parseOptionalKeyword("unwind"))) {
- if (hasUnwind)
- return parser.emitError(parser.getCurrentLocation(),
- "duplicate 'unwind' handler");
- if (hasCatchAll)
- return parser.emitError(parser.getCurrentLocation(),
- "cannot have both 'catch_all' and 'unwind'");
- hasUnwind = true;
-
- if (parser.parseColon().failed())
- return failure();
-
- if (parser.parseSuccessor(defaultDestination).failed())
- return failure();
- return success();
- }
-
- // A filter handler carries the type info symbols permitted by the
- // enclosing function's dynamic exception specification. Its destination is
- // taken when the exception is *not* one of those types.
- if (succeeded(parser.parseOptionalKeyword("filter"))) {
- SMLoc filterLoc = parser.getCurrentLocation();
- SmallVector<Attribute, 4> permittedTypes;
- auto parsePermittedType = [&]() -> ParseResult {
- mlir::SMLoc typeLoc = parser.getCurrentLocation();
- mlir::Attribute rtti;
- if (parser.parseAttribute(rtti).failed())
- return failure();
- if (!mlir::isa<cir::GlobalViewAttr>(rtti))
- return parser.emitError(typeLoc,
- "expected a type info symbol naming a "
- "permitted exception type");
- permittedTypes.push_back(rtti);
- return success();
- };
- if (parser
- .parseCommaSeparatedList(OpAsmParser::Delimiter::Paren,
- parsePermittedType)
- .failed())
- return failure();
-
- auto filterAttr = cir::EhFilterAttr::getChecked(
- [&]() { return parser.emitError(filterLoc); }, parser.getContext(),
- parser.getBuilder().getArrayAttr(permittedTypes));
- if (!filterAttr)
- return failure();
- handlerTypes.push_back(filterAttr);
-
- if (parser.parseColon().failed())
- return failure();
-
- Block *dest;
- if (parser.parseSuccessor(dest).failed())
- return failure();
- catchDestinations.push_back(dest);
- return success();
- }
-
- // Otherwise, expect 'catch(<attr> : <type>) : ^block'.
- // The 'catch(...)' wrapper allows the attribute to include its type
- // without conflicting with the ':' used for the block destination.
- if (parser.parseKeyword("catch").failed())
- return failure();
-
- if (parser.parseLParen().failed())
- return failure();
-
- mlir::Attribute catchTypeAttr;
- if (parser.parseAttribute(catchTypeAttr).failed())
- return failure();
- handlerTypes.push_back(catchTypeAttr);
-
- if (parser.parseRParen().failed())
- return failure();
-
- if (parser.parseColon().failed())
- return failure();
-
- Block *dest;
- if (parser.parseSuccessor(dest).failed())
- return failure();
- catchDestinations.push_back(dest);
- return success();
- };
-
- if (parser.parseCommaSeparatedList(parseHandler).failed())
- return failure();
-
- if (parser.parseRSquare().failed())
- return failure();
-
- // Verify we have catch_all or unwind.
- if (!hasCatchAll && !hasUnwind)
- return parser.emitError(parser.getCurrentLocation(),
- "must have either 'catch_all' or 'unwind' handler");
-
- // Add attributes and successors.
- if (!handlerTypes.empty())
- catchTypes = parser.getBuilder().getArrayAttr(handlerTypes);
-
- if (hasCatchAll)
- defaultIsCatchAll = parser.getBuilder().getUnitAttr();
-
- return success();
-}
-
-void cir::printEhDispatchDestinations(OpAsmPrinter &p, cir::EhDispatchOp op,
- mlir::ArrayAttr catchTypes,
- SuccessorRange catchDestinations,
- Block *defaultDestination,
- mlir::UnitAttr defaultIsCatchAll) {
- p << " [";
- p.printNewline();
-
- // If we have at least one catch type, print them.
- if (catchTypes) {
- // Print type handlers using 'catch(<attr>) : ^block' syntax.
- llvm::interleave(
- llvm::zip(catchTypes, catchDestinations),
- [&](auto i) {
- mlir::Attribute typeAttr = std::get<0>(i);
- if (auto filterAttr = mlir::dyn_cast<cir::EhFilterAttr>(typeAttr)) {
- p << " filter(";
- llvm::interleaveComma(
- filterAttr.getPermittedTypes(), p,
- [&](mlir::Attribute sym) { p.printAttribute(sym); });
- } else {
- p << " catch(";
- p.printAttribute(typeAttr);
- }
- p << ") : ";
- p.printSuccessor(std::get<1>(i));
- },
- [&] {
- p << ',';
- p.printNewline();
- });
-
- p << ", ";
- p.printNewline();
- }
-
- // Print catch_all or unwind handler.
- if (defaultIsCatchAll)
- p << " catch_all : ";
- else
- p << " unwind : ";
- p.printSuccessor(defaultDestination);
- p.printNewline();
-
- p << "]";
-}
-
-//===----------------------------------------------------------------------===//
-// Standard library op signature matching
-//===----------------------------------------------------------------------===//
-
-bool cir::StdFindOp::signatureMatches(mlir::TypeRange operands,
- mlir::TypeRange results) {
- if (operands.size() != getNumArgs() || results.size() != 1)
- return false;
- mlir::Type iterTy = operands[0];
- return iterTy == operands[1] && iterTy == operands[2] && iterTy == results[0];
-}
diff --git a/clang/lib/CIR/Dialect/IR/CIRDialectBytecode.cpp b/clang/lib/CIR/Dialect/IR/CIRDialectBytecode.cpp
index 877ab487bba448..89ddebcd4ce224 100644
--- a/clang/lib/CIR/Dialect/IR/CIRDialectBytecode.cpp
+++ b/clang/lib/CIR/Dialect/IR/CIRDialectBytecode.cpp
@@ -10,7 +10,7 @@
#include "mlir/Bytecode/BytecodeImplementation.h"
#include "mlir/IR/Diagnostics.h"
#include "clang/CIR/Dialect/IR/CIRAttrs.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "clang/CIR/Dialect/IR/CIRTypes.h"
#include "llvm/ADT/SmallVector.h"
#include "llvm/ADT/StringRef.h"
diff --git a/clang/lib/CIR/Dialect/IR/CIROpDefinitions.h b/clang/lib/CIR/Dialect/IR/CIROpDefinitions.h
index 9268f540f3ce37..a9d35554eca807 100644
--- a/clang/lib/CIR/Dialect/IR/CIROpDefinitions.h
+++ b/clang/lib/CIR/Dialect/IR/CIROpDefinitions.h
@@ -6,20 +6,42 @@
//
//===----------------------------------------------------------------------===//
//
-// This header declares the custom directive parsers and printers implemented
-// in CIRDialect.cpp and referenced by the TableGen-generated CIR operation
-// definitions, which are compiled in separate shards.
+// This header declares the helpers shared by the CIROps<Partition>.cpp files:
+// the custom directive parsers and printers referenced by the
+// TableGen-generated op definitions, and the region terminator helpers. Each
+// is implemented in the partition file of the op it belongs to.
//
//===----------------------------------------------------------------------===//
#ifndef CLANG_LIB_CIR_DIALECT_IR_CIROPDEFINITIONS_H
#define CLANG_LIB_CIR_DIALECT_IR_CIROPDEFINITIONS_H
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "mlir/IR/OpImplementation.h"
namespace cir {
+class AssumeOp;
+class EhDispatchOp;
+class GlobalOp;
+class IndirectBrOp;
+class ScopeOp;
+class SwitchFlatOp;
+class TryOp;
+
+// Checks that omitting the terminator of `region` is valid and, if so, creates
+// and inserts the omitted `cir.yield` into the region. Implemented in
+// CIROpsControlFlow.cpp.
+mlir::LogicalResult ensureRegionTerm(mlir::OpAsmParser &parser,
+ mlir::Region ®ion, llvm::SMLoc errLoc);
+
+// True if the terminator of `r` should be omitted when printing.
+bool omitRegionTerm(mlir::Region &r);
+
+// Checks that the constant `attrType` is a valid initializer for `opType` on
+// `op` (a cir.const or cir.global). Implemented in CIROpsCore.cpp.
+mlir::LogicalResult checkConstantTypes(mlir::Operation *op, mlir::Type opType,
+ mlir::Attribute attrType);
mlir::ParseResult parseOmittedTerminatorRegion(mlir::OpAsmParser &parser,
mlir::Region ®ion);
diff --git a/clang/lib/CIR/Dialect/IR/CIROpsArith.cpp b/clang/lib/CIR/Dialect/IR/CIROpsArith.cpp
new file mode 100644
index 00000000000000..ec974968387cf2
--- /dev/null
+++ b/clang/lib/CIR/Dialect/IR/CIROpsArith.cpp
@@ -0,0 +1,315 @@
+//===- CIROpsArith.cpp - CIR ops in partition `Arith` ---------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the CIR ops in partition `Arith`: the hand-written
+// parts first, then the TableGen-generated definitions.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/CIR/Dialect/IR/CIROpsArith.h"
+
+#include "CIROpDefinitions.h"
+
+#include "mlir/IR/DialectImplementation.h"
+
+using namespace mlir;
+using namespace cir;
+
+//===----------------------------------------------------------------------===//
+// AddOp / SubOp
+//===----------------------------------------------------------------------===//
+
+// The integer-only type constraint on these ops makes the nsw/nuw/sat flag
+// type checks unnecessary. Only the mutual-exclusivity between nsw/nuw and
+// sat needs to be verified.
+
+LogicalResult cir::AddOp::verify() {
+ if (getSaturated() && (getNoSignedWrap() || getNoUnsignedWrap()))
+ return emitOpError()
+ << "the nsw/nuw flags and the saturated flag are mutually exclusive";
+ return mlir::success();
+}
+
+LogicalResult cir::SubOp::verify() {
+ if (getSaturated() && (getNoSignedWrap() || getNoUnsignedWrap()))
+ return emitOpError()
+ << "the nsw/nuw flags and the saturated flag are mutually exclusive";
+ return mlir::success();
+}
+
+//===----------------------------------------------------------------------===//
+// ShiftOp
+//===----------------------------------------------------------------------===//
+LogicalResult cir::ShiftOp::verify() {
+ mlir::Operation *op = getOperation();
+ auto op0VecTy = mlir::dyn_cast<cir::VectorType>(op->getOperand(0).getType());
+ auto op1VecTy = mlir::dyn_cast<cir::VectorType>(op->getOperand(1).getType());
+ if (!op0VecTy ^ !op1VecTy)
+ return emitOpError() << "input types cannot be one vector and one scalar";
+
+ if (op0VecTy) {
+ if (op0VecTy.getSize() != op1VecTy.getSize())
+ return emitOpError() << "input vector types must have the same size";
+
+ auto opResultTy = mlir::dyn_cast<cir::VectorType>(getType());
+ if (!opResultTy)
+ return emitOpError() << "the type of the result must be a vector "
+ << "if it is vector shift";
+
+ auto op0VecEleTy = mlir::cast<cir::IntType>(op0VecTy.getElementType());
+ auto op1VecEleTy = mlir::cast<cir::IntType>(op1VecTy.getElementType());
+ if (op0VecEleTy.getWidth() != op1VecEleTy.getWidth())
+ return emitOpError()
+ << "vector operands do not have the same elements sizes";
+
+ auto resVecEleTy = mlir::cast<cir::IntType>(opResultTy.getElementType());
+ if (op0VecEleTy.getWidth() != resVecEleTy.getWidth())
+ return emitOpError() << "vector operands and result type do not have the "
+ "same elements sizes";
+ }
+
+ return mlir::success();
+}
+
+//===----------------------------------------------------------------------===//
+// IncOp
+//===----------------------------------------------------------------------===//
+
+OpFoldResult cir::IncOp::fold(FoldAdaptor adaptor) {
+ if (mlir::isa_and_present<cir::PoisonAttr>(adaptor.getInput()))
+ return adaptor.getInput();
+ return {};
+}
+
+//===----------------------------------------------------------------------===//
+// DecOp
+//===----------------------------------------------------------------------===//
+
+OpFoldResult cir::DecOp::fold(FoldAdaptor adaptor) {
+ if (mlir::isa_and_present<cir::PoisonAttr>(adaptor.getInput()))
+ return adaptor.getInput();
+ return {};
+}
+
+//===----------------------------------------------------------------------===//
+// MinusOp
+//===----------------------------------------------------------------------===//
+
+OpFoldResult cir::MinusOp::fold(FoldAdaptor adaptor) {
+ if (mlir::isa_and_present<cir::PoisonAttr>(adaptor.getInput()))
+ return adaptor.getInput();
+
+ // Fold with constant inputs. Floating-point negation is handled by
+ // cir::FNegOp.
+ if (auto intAttr =
+ mlir::dyn_cast_if_present<cir::IntAttr>(adaptor.getInput())) {
+ APInt val = intAttr.getValue();
+ val.negate();
+ return cir::IntAttr::get(getType(), val);
+ }
+
+ return {};
+}
+
+//===----------------------------------------------------------------------===//
+// FNegOp
+//===----------------------------------------------------------------------===//
+
+OpFoldResult cir::FNegOp::fold(FoldAdaptor adaptor) {
+ if (mlir::isa_and_present<cir::PoisonAttr>(adaptor.getInput()))
+ return adaptor.getInput();
+
+ // Fold with constant inputs.
+ if (auto fpAttr =
+ mlir::dyn_cast_if_present<cir::FPAttr>(adaptor.getInput())) {
+ APFloat val = fpAttr.getValue();
+ val.changeSign();
+ return cir::FPAttr::get(getType(), val);
+ }
+
+ return {};
+}
+
+//===----------------------------------------------------------------------===//
+// NotOp
+//===----------------------------------------------------------------------===//
+
+OpFoldResult cir::NotOp::fold(FoldAdaptor adaptor) {
+ if (mlir::isa_and_present<cir::PoisonAttr>(adaptor.getInput()))
+ return adaptor.getInput();
+
+ // not(not(x)) -> x is handled by the Involution trait.
+
+ // Fold with constant inputs.
+ if (mlir::Attribute attr = adaptor.getInput()) {
+ if (auto intAttr = mlir::dyn_cast<cir::IntAttr>(attr)) {
+ APInt val = intAttr.getValue();
+ val.flipAllBits();
+ return cir::IntAttr::get(getType(), val);
+ }
+ if (auto boolAttr = mlir::dyn_cast<cir::BoolAttr>(attr))
+ return cir::BoolAttr::get(getContext(), !boolAttr.getValue());
+ }
+
+ return {};
+}
+
+//===----------------------------------------------------------------------===//
+// CmpOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::CmpOp::verify() {
+ if (getFenvAttr() && !cir::isAnyFloatingPointType(getLhs().getType()))
+ return emitOpError()
+ << "'fenv' is only valid for floating-point comparisons";
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// Bit manipulation operations
+//===----------------------------------------------------------------------===//
+
+static OpFoldResult
+foldUnaryBitOp(mlir::Attribute inputAttr,
+ llvm::function_ref<llvm::APInt(const llvm::APInt &)> func,
+ bool poisonZero = false) {
+ if (mlir::isa_and_present<cir::PoisonAttr>(inputAttr)) {
+ // Propagate poison value
+ return inputAttr;
+ }
+
+ auto input = mlir::dyn_cast_if_present<IntAttr>(inputAttr);
+ if (!input)
+ return nullptr;
+
+ llvm::APInt inputValue = input.getValue();
+ if (poisonZero && inputValue.isZero())
+ return cir::PoisonAttr::get(input.getType());
+
+ llvm::APInt resultValue = func(inputValue);
+ return IntAttr::get(input.getType(), resultValue);
+}
+
+OpFoldResult BitClrsbOp::fold(FoldAdaptor adaptor) {
+ return foldUnaryBitOp(adaptor.getInput(), [](const llvm::APInt &inputValue) {
+ unsigned resultValue =
+ inputValue.getBitWidth() - inputValue.getSignificantBits();
+ return llvm::APInt(inputValue.getBitWidth(), resultValue);
+ });
+}
+
+OpFoldResult BitClzOp::fold(FoldAdaptor adaptor) {
+ return foldUnaryBitOp(
+ adaptor.getInput(),
+ [](const llvm::APInt &inputValue) {
+ unsigned resultValue = inputValue.countLeadingZeros();
+ return llvm::APInt(inputValue.getBitWidth(), resultValue);
+ },
+ getPoisonZero());
+}
+
+OpFoldResult BitCtzOp::fold(FoldAdaptor adaptor) {
+ return foldUnaryBitOp(
+ adaptor.getInput(),
+ [](const llvm::APInt &inputValue) {
+ return llvm::APInt(inputValue.getBitWidth(),
+ inputValue.countTrailingZeros());
+ },
+ getPoisonZero());
+}
+
+OpFoldResult BitFfsOp::fold(FoldAdaptor adaptor) {
+ return foldUnaryBitOp(adaptor.getInput(), [](const llvm::APInt &inputValue) {
+ unsigned trailingZeros = inputValue.countTrailingZeros();
+ unsigned result =
+ trailingZeros == inputValue.getBitWidth() ? 0 : trailingZeros + 1;
+ return llvm::APInt(inputValue.getBitWidth(), result);
+ });
+}
+
+OpFoldResult BitParityOp::fold(FoldAdaptor adaptor) {
+ return foldUnaryBitOp(adaptor.getInput(), [](const llvm::APInt &inputValue) {
+ return llvm::APInt(inputValue.getBitWidth(), inputValue.popcount() % 2);
+ });
+}
+
+OpFoldResult BitPopcountOp::fold(FoldAdaptor adaptor) {
+ return foldUnaryBitOp(adaptor.getInput(), [](const llvm::APInt &inputValue) {
+ return llvm::APInt(inputValue.getBitWidth(), inputValue.popcount());
+ });
+}
+
+OpFoldResult BitReverseOp::fold(FoldAdaptor adaptor) {
+ return foldUnaryBitOp(adaptor.getInput(), [](const llvm::APInt &inputValue) {
+ return inputValue.reverseBits();
+ });
+}
+
+OpFoldResult ByteSwapOp::fold(FoldAdaptor adaptor) {
+ return foldUnaryBitOp(adaptor.getInput(), [](const llvm::APInt &inputValue) {
+ return inputValue.byteSwap();
+ });
+}
+
+OpFoldResult RotateOp::fold(FoldAdaptor adaptor) {
+ if (mlir::isa_and_present<cir::PoisonAttr>(adaptor.getInput()) ||
+ mlir::isa_and_present<cir::PoisonAttr>(adaptor.getAmount())) {
+ // Propagate poison values
+ return cir::PoisonAttr::get(getType());
+ }
+
+ auto input = mlir::dyn_cast_if_present<IntAttr>(adaptor.getInput());
+ auto amount = mlir::dyn_cast_if_present<IntAttr>(adaptor.getAmount());
+ if (!input && !amount)
+ return nullptr;
+
+ // We could fold cir.rotate even if one of its two operands is not a constant:
+ // - `cir.rotate left/right %0, 0` could be folded into just %0 even if %0
+ // is not a constant.
+ // - `cir.rotate left/right 0/0b111...111, %0` could be folded into 0 or
+ // 0b111...111 even if %0 is not a constant.
+
+ llvm::APInt inputValue;
+ if (input) {
+ inputValue = input.getValue();
+ if (inputValue.isZero() || inputValue.isAllOnes()) {
+ // An input value of all 0s or all 1s will not change after rotation
+ return input;
+ }
+ }
+
+ uint64_t amountValue;
+ if (amount) {
+ amountValue = amount.getValue().urem(getInput().getType().getWidth());
+ if (amountValue == 0) {
+ // A shift amount of 0 will not change the input value
+ return getInput();
+ }
+ }
+
+ if (!input || !amount)
+ return nullptr;
+
+ assert(inputValue.getBitWidth() == getInput().getType().getWidth() &&
+ "input value must have the same bit width as the input type");
+
+ llvm::APInt resultValue;
+ if (isRotateLeft())
+ resultValue = inputValue.rotl(amountValue);
+ else
+ resultValue = inputValue.rotr(amountValue);
+
+ return IntAttr::get(input.getContext(), input.getType(), resultValue);
+}
+
+//===----------------------------------------------------------------------===//
+// TableGen-generated op definitions
+//===----------------------------------------------------------------------===//
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsArith.cpp.inc"
diff --git a/clang/lib/CIR/Dialect/IR/CIROpsBuiltins.cpp b/clang/lib/CIR/Dialect/IR/CIROpsBuiltins.cpp
new file mode 100644
index 00000000000000..f57d717df964b4
--- /dev/null
+++ b/clang/lib/CIR/Dialect/IR/CIROpsBuiltins.cpp
@@ -0,0 +1,483 @@
+//===- CIROpsBuiltins.cpp - CIR ops in partition `Builtins` ---------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the CIR ops in partition `Builtins`: the hand-written
+// parts first, then the TableGen-generated definitions.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/CIR/Dialect/IR/CIROpsBuiltins.h"
+#include "clang/CIR/Dialect/IR/CIROpsCore.h"
+
+#include "CIROpDefinitions.h"
+
+#include "mlir/IR/DialectImplementation.h"
+
+using namespace mlir;
+using namespace cir;
+
+// Verifies that the given operand is produced by an operation of type
+// ExpectedProducerOp.
+template <typename ExpectedProducerOp>
+static LogicalResult verifyProducedBy(Operation *op, Value operand,
+ StringRef operandName) {
+ Operation *producer = operand.getDefiningOp();
+ if (!producer || !isa<ExpectedProducerOp>(producer))
+ return op->emitOpError()
+ << "operand '" << operandName << "' must be produced by '"
+ << ExpectedProducerOp::getOperationName() << "'";
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// AssumeOp
+//===----------------------------------------------------------------------===//
+
+void cir::printAssumeBundle(OpAsmPrinter &p, cir::AssumeOp op,
+ cir::AssumeBundleKindAttr kindAttr,
+ OperandRange bundleArgs, TypeRange bundleArgTypes) {
+ cir::AssumeBundleKind kind = kindAttr.getValue();
+ if (kind == cir::AssumeBundleKind::None)
+ return;
+
+ p << " " << cir::stringifyAssumeBundleKind(kind);
+ if (bundleArgs.empty())
+ return;
+
+ p << "(";
+ p.printOperands(bundleArgs);
+ p << " : ";
+ llvm::interleaveComma(bundleArgTypes, p);
+ p << ")";
+}
+
+ParseResult cir::parseAssumeBundle(
+ OpAsmParser &p, cir::AssumeBundleKindAttr &bundleKindAttr,
+ llvm::SmallVector<mlir::OpAsmParser::UnresolvedOperand, 4> &bundleArgs,
+ llvm::SmallVector<mlir::Type, 1> &bundleArgTypes) {
+ StringRef keyword;
+ auto loc = p.getCurrentLocation();
+ if (failed(p.parseOptionalKeyword(&keyword))) {
+ bundleKindAttr = cir::AssumeBundleKindAttr::get(
+ p.getContext(), cir::AssumeBundleKind::None);
+ return success();
+ }
+
+ std::optional<cir::AssumeBundleKind> parsedKind =
+ cir::symbolizeAssumeBundleKind(keyword);
+ if (!parsedKind)
+ return p.emitError(loc, "unknown assume bundle kind '") << keyword << "'";
+
+ bundleKindAttr = cir::AssumeBundleKindAttr::get(p.getContext(), *parsedKind);
+
+ if (p.parseOptionalLParen())
+ return success();
+
+ if (p.parseOperandList(bundleArgs) || p.parseColon() ||
+ p.parseTypeList(bundleArgTypes) || p.parseRParen())
+ return failure();
+
+ return success();
+}
+
+LogicalResult cir::AssumeOp::verify() {
+ cir::AssumeBundleKind kind = getBundleKind();
+ size_t numArgs = getBundleArgs().size();
+
+ if (kind == cir::AssumeBundleKind::None) {
+ if (numArgs != 0)
+ return emitOpError("unexpected bundle operands for kind 'none'");
+ return success();
+ }
+
+ if (numArgs == 0)
+ return emitOpError("expected bundle operands for kind '")
+ << cir::stringifyAssumeBundleKind(kind) << "'";
+
+ switch (kind) {
+ case cir::AssumeBundleKind::Align:
+ if (numArgs != 2 && numArgs != 3)
+ return emitOpError("align bundle expects 2 or 3 operands");
+ break;
+ case cir::AssumeBundleKind::SeparateStorage:
+ if (numArgs != 2)
+ return emitOpError("separate_storage bundle expects 2 operands");
+ break;
+ case cir::AssumeBundleKind::Dereferenceable:
+ if (numArgs != 2)
+ return emitOpError("dereferenceable bundle expects 2 operands");
+ break;
+ default:
+ break;
+ }
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// CopyOp Definitions
+//===----------------------------------------------------------------------===//
+
+// Prints the pointer type(s) for a `cir.copy`. `src` and `dst` share the same
+// pointee type but may differ in address space; a single type is printed when
+// they match, otherwise both are printed (`src` first).
+void cir::printCopyTypes(mlir::OpAsmPrinter &printer, mlir::Operation *,
+ mlir::Type srcType, mlir::Type dstType) {
+ printer.printType(srcType);
+ if (srcType != dstType) {
+ printer << ", ";
+ printer.printType(dstType);
+ }
+}
+
+mlir::ParseResult cir::parseCopyTypes(mlir::OpAsmParser &parser,
+ mlir::Type &srcType,
+ mlir::Type &dstType) {
+ if (parser.parseType(srcType))
+ return mlir::failure();
+ if (parser.parseOptionalComma().succeeded()) {
+ if (parser.parseType(dstType))
+ return mlir::failure();
+ } else {
+ dstType = srcType;
+ }
+ return mlir::success();
+}
+
+LogicalResult cir::CopyOp::verify() {
+ // The pointee types of `src` and `dst` are guaranteed to match by the
+ // SameOperandsPointeeType trait; they may still differ in address space.
+
+ // A data layout is required for us to know the number of bytes to be copied.
+ if (!getType().getPointee().hasTrait<DataLayoutTypeInterface::Trait>())
+ return emitError() << "missing data layout for pointee type";
+
+ if (getSkipTailPadding() &&
+ !mlir::isa<cir::RecordType>(getType().getPointee()))
+ return emitError()
+ << "skip_tail_padding is only valid for record pointee types";
+
+ return mlir::success();
+}
+
+//===----------------------------------------------------------------------===//
+// InlineAsmOp
+//===----------------------------------------------------------------------===//
+
+void cir::InlineAsmOp::print(OpAsmPrinter &p) {
+ p << '(' << getAsmFlavor() << ", ";
+ p.increaseIndent();
+ p.printNewline();
+
+ llvm::SmallVector<std::string, 3> names{"out", "in", "in_out"};
+ auto *nameIt = names.begin();
+ auto *attrIt = getOperandAttrs().begin();
+
+ for (mlir::OperandRange ops : getAsmOperands()) {
+ p << *nameIt << " = ";
+
+ p << '[';
+ llvm::interleaveComma(llvm::make_range(ops.begin(), ops.end()), p,
+ [&](Value value) {
+ p.printOperand(value);
+ p << " : " << value.getType();
+ if (mlir::isa<mlir::UnitAttr>(*attrIt))
+ p << " (maybe_memory)";
+ attrIt++;
+ });
+ p << "],";
+ p.printNewline();
+ ++nameIt;
+ }
+
+ p << "{";
+ p.printString(getAsmString());
+ p << " ";
+ p.printString(getConstraints());
+ p << "}";
+ p.decreaseIndent();
+ p << ')';
+ if (getSideEffects())
+ p << " side_effects";
+
+ p.printOptionalAttrDict(
+ getOperation()->getDiscardableAttrDictionary().getValue());
+
+ if (auto v = getRes())
+ p << " -> " << v.getType();
+}
+
+void cir::InlineAsmOp::build(OpBuilder &odsBuilder, OperationState &odsState,
+ ArrayRef<ValueRange> asmOperands,
+ StringRef asmString, StringRef constraints,
+ bool sideEffects, cir::AsmFlavor asmFlavor,
+ ArrayRef<Attribute> operandAttrs) {
+ // Set up the operands_segments for VariadicOfVariadic
+ SmallVector<int32_t> segments;
+ for (auto operandRange : asmOperands) {
+ segments.push_back(operandRange.size());
+ odsState.addOperands(operandRange);
+ }
+
+ odsState.addAttribute(
+ "operands_segments",
+ DenseI32ArrayAttr::get(odsBuilder.getContext(), segments));
+ odsState.addAttribute("asm_string", odsBuilder.getStringAttr(asmString));
+ odsState.addAttribute("constraints", odsBuilder.getStringAttr(constraints));
+ odsState.addAttribute("asm_flavor",
+ AsmFlavorAttr::get(odsBuilder.getContext(), asmFlavor));
+
+ if (sideEffects)
+ odsState.addAttribute("side_effects", odsBuilder.getUnitAttr());
+
+ odsState.addAttribute("operand_attrs", odsBuilder.getArrayAttr(operandAttrs));
+}
+
+ParseResult cir::InlineAsmOp::parse(OpAsmParser &parser,
+ OperationState &result) {
+ llvm::SmallVector<mlir::Attribute> operandAttrs;
+ llvm::SmallVector<int32_t> operandsGroupSizes;
+ std::string asmString, constraints;
+ Type resType;
+ MLIRContext *ctxt = parser.getBuilder().getContext();
+
+ auto error = [&](const Twine &msg) -> LogicalResult {
+ return parser.emitError(parser.getCurrentLocation(), msg);
+ };
+
+ auto expected = [&](const std::string &c) {
+ return error("expected '" + c + "'");
+ };
+
+ if (parser.parseLParen().failed())
+ return expected("(");
+
+ auto flavor = FieldParser<AsmFlavor, AsmFlavor>::parse(parser);
+ if (failed(flavor))
+ return error("Unknown AsmFlavor");
+
+ if (parser.parseComma().failed())
+ return expected(",");
+
+ auto parseValue = [&](Value &v) {
+ OpAsmParser::UnresolvedOperand op;
+
+ if (parser.parseOperand(op) || parser.parseColon())
+ return error("can't parse operand");
+
+ Type typ;
+ if (parser.parseType(typ).failed())
+ return error("can't parse operand type");
+ llvm::SmallVector<mlir::Value> tmp;
+ if (parser.resolveOperand(op, typ, tmp))
+ return error("can't resolve operand");
+ v = tmp[0];
+ return mlir::success();
+ };
+
+ auto parseOperands = [&](llvm::StringRef name) {
+ if (parser.parseKeyword(name).failed())
+ return error("expected " + name + " operands here");
+ if (parser.parseEqual().failed())
+ return expected("=");
+ if (parser.parseLSquare().failed())
+ return expected("[");
+
+ int size = 0;
+ if (parser.parseOptionalRSquare().succeeded()) {
+ operandsGroupSizes.push_back(size);
+ if (parser.parseComma())
+ return expected(",");
+ return mlir::success();
+ }
+
+ auto parseOperand = [&]() {
+ Value val;
+ if (parseValue(val).succeeded()) {
+ result.operands.push_back(val);
+ size++;
+
+ if (parser.parseOptionalLParen().failed()) {
+ operandAttrs.push_back(mlir::DictionaryAttr::get(ctxt));
+ return mlir::success();
+ }
+
+ if (parser.parseKeyword("maybe_memory").succeeded()) {
+ operandAttrs.push_back(mlir::UnitAttr::get(ctxt));
+ if (parser.parseRParen())
+ return expected(")");
+ return mlir::success();
+ } else {
+ return expected("maybe_memory");
+ }
+ }
+ return mlir::failure();
+ };
+
+ if (parser.parseCommaSeparatedList(parseOperand).failed())
+ return mlir::failure();
+
+ if (parser.parseRSquare().failed() || parser.parseComma().failed())
+ return expected("]");
+ operandsGroupSizes.push_back(size);
+ return mlir::success();
+ };
+
+ if (parseOperands("out").failed() || parseOperands("in").failed() ||
+ parseOperands("in_out").failed())
+ return error("failed to parse operands");
+
+ if (parser.parseLBrace())
+ return expected("{");
+ if (parser.parseString(&asmString))
+ return error("asm string parsing failed");
+ if (parser.parseString(&constraints))
+ return error("constraints string parsing failed");
+ if (parser.parseRBrace())
+ return expected("}");
+ if (parser.parseRParen())
+ return expected(")");
+
+ if (parser.parseOptionalKeyword("side_effects").succeeded())
+ result.attributes.set("side_effects", UnitAttr::get(ctxt));
+
+ if (parser.parseOptionalAttrDict(result.attributes).failed())
+ return mlir::failure();
+
+ if (parser.parseOptionalArrow().succeeded() &&
+ parser.parseType(resType).failed())
+ return mlir::failure();
+
+ result.attributes.set("asm_flavor", AsmFlavorAttr::get(ctxt, *flavor));
+ result.attributes.set("asm_string", StringAttr::get(ctxt, asmString));
+ result.attributes.set("constraints", StringAttr::get(ctxt, constraints));
+ result.attributes.set("operand_attrs", ArrayAttr::get(ctxt, operandAttrs));
+ result.getOrAddProperties<InlineAsmOp::Properties>().operands_segments =
+ parser.getBuilder().getDenseI32ArrayAttr(operandsGroupSizes);
+ if (resType)
+ result.addTypes(TypeRange{resType});
+
+ return mlir::success();
+}
+
+void InlineAsmOp::getEffects(
+ llvm::SmallVectorImpl<mlir::MemoryEffects::EffectInstance> &effects) {
+ // If we have any side effects (that is, we're volatile asm), add a read and
+ // write memory effect. We do this the same as the llvm dialect InlineAsmOp.
+ if (getSideEffects()) {
+ effects.emplace_back(mlir::MemoryEffects::Read::get());
+ effects.emplace_back(mlir::MemoryEffects::Write::get());
+ }
+}
+
+//===----------------------------------------------------------------------===//
+// AtomicFetchOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::AtomicFetchOp::verify() {
+ if (getBinop() != cir::AtomicFetchKind::Add &&
+ getBinop() != cir::AtomicFetchKind::Sub &&
+ getBinop() != cir::AtomicFetchKind::Max &&
+ getBinop() != cir::AtomicFetchKind::Min &&
+ getBinop() != cir::AtomicFetchKind::Maximum &&
+ getBinop() != cir::AtomicFetchKind::Minimum &&
+ getBinop() != cir::AtomicFetchKind::MaximumNum &&
+ getBinop() != cir::AtomicFetchKind::MinimumNum &&
+ !mlir::isa<cir::IntType>(getVal().getType()))
+ return emitError("only atomic add, sub, max, min, maximum, minimum, "
+ "maximum_num, and minimum_num operation could operate on "
+ "floating-point values");
+
+ if ((getBinop() == cir::AtomicFetchKind::Maximum ||
+ getBinop() == cir::AtomicFetchKind::Minimum ||
+ getBinop() == cir::AtomicFetchKind::MaximumNum ||
+ getBinop() == cir::AtomicFetchKind::MinimumNum) &&
+ !mlir::isa<cir::FPTypeInterface>(getVal().getType()))
+ return emitError("atomic maximum, minimum, maximum_num, and minimum_num "
+ "operation could only operate on floating-point values");
+
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// LifetimeStartOp & LifetimeEndOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::LifetimeStartOp::verify() {
+ return verifyProducedBy<cir::AllocaOp>(*this, getPtr(), "ptr");
+}
+
+LogicalResult cir::LifetimeEndOp::verify() {
+ return verifyProducedBy<cir::AllocaOp>(*this, getPtr(), "ptr");
+}
+
+//===----------------------------------------------------------------------===//
+// MemChrOp
+//===----------------------------------------------------------------------===//
+
+/// Reads a fundamental integer width from a signless i32 attribute.
+static std::optional<unsigned> getRecordedIntegerWidth(mlir::Attribute attr) {
+ auto intAttr = mlir::dyn_cast<mlir::IntegerAttr>(attr);
+ if (!intAttr || !intAttr.getType().isSignlessInteger(32))
+ return std::nullopt;
+ int64_t width = intAttr.getInt();
+ if (width < 0 ||
+ !cir::isValidFundamentalIntWidth(static_cast<unsigned>(width)))
+ return std::nullopt;
+ return static_cast<unsigned>(width);
+}
+
+LogicalResult cir::MemChrOp::verify() {
+ auto moduleOp = (*this)->getParentOfType<mlir::ModuleOp>();
+ if (!moduleOp)
+ return emitOpError("expects an enclosing module");
+
+ // libc memchr uses pointers in the target's default address space.
+ if (mlir::cast<cir::PointerType>(getSrc().getType()).getAddrSpace())
+ return emitOpError("src must be in the default address space");
+
+ auto checkWidth = [&](cir::IntType type, llvm::StringRef operandName,
+ llvm::StringRef attrName) -> LogicalResult {
+ mlir::Attribute attr = moduleOp->getAttr(attrName);
+ if (!attr)
+ return emitOpError("expects the module to record ") << attrName;
+ std::optional<unsigned> width = getRecordedIntegerWidth(attr);
+ if (!width)
+ return emitOpError("requires ")
+ << attrName
+ << " to be a signless i32 holding a fundamental integer width";
+ if (type.getWidth() != *width)
+ return emitOpError() << operandName << " must have the width recorded in "
+ << attrName;
+ return success();
+ };
+
+ if (failed(checkWidth(getPattern().getType(), "pattern",
+ cir::CIRDialect::getIntTypeWidthAttrName())))
+ return failure();
+ return checkWidth(getLen().getType(), "len",
+ cir::CIRDialect::getSizeTypeWidthAttrName());
+}
+
+//===----------------------------------------------------------------------===//
+// Standard library op signature matching
+//===----------------------------------------------------------------------===//
+
+bool cir::StdFindOp::signatureMatches(mlir::TypeRange operands,
+ mlir::TypeRange results) {
+ if (operands.size() != getNumArgs() || results.size() != 1)
+ return false;
+ mlir::Type iterTy = operands[0];
+ return iterTy == operands[1] && iterTy == operands[2] && iterTy == results[0];
+}
+
+//===----------------------------------------------------------------------===//
+// TableGen-generated op definitions
+//===----------------------------------------------------------------------===//
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsBuiltins.cpp.inc"
diff --git a/clang/lib/CIR/Dialect/IR/CIROpsClassMembers.cpp b/clang/lib/CIR/Dialect/IR/CIROpsClassMembers.cpp
new file mode 100644
index 00000000000000..b67957fd026dc4
--- /dev/null
+++ b/clang/lib/CIR/Dialect/IR/CIROpsClassMembers.cpp
@@ -0,0 +1,296 @@
+//===- CIROpsClassMembers.cpp - CIR ops in partition `ClassMembers` -------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the CIR ops in partition `ClassMembers`: the
+// hand-written parts first, then the TableGen-generated definitions.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/CIR/Dialect/IR/CIROpsClassMembers.h"
+#include "clang/CIR/Dialect/IR/CIROpsControlFlow.h"
+#include "clang/CIR/Dialect/IR/CIROpsSymbols.h"
+
+#include "CIROpDefinitions.h"
+
+#include "mlir/IR/DialectImplementation.h"
+
+using namespace mlir;
+using namespace cir;
+
+//===----------------------------------------------------------------------===//
+// ArrayCtor & ArrayDtor
+//===----------------------------------------------------------------------===//
+
+template <typename Op> static LogicalResult verifyArrayCtorDtor(Op op) {
+ auto ptrTy = mlir::cast<cir::PointerType>(op.getAddr().getType());
+ mlir::Type pointeeTy = ptrTy.getPointee();
+
+ mlir::Block &body = op.getBody().front();
+ if (body.getNumArguments() != 1)
+ return op.emitOpError("body must have exactly one block argument");
+
+ auto expectedEltPtrTy =
+ mlir::dyn_cast<cir::PointerType>(body.getArgument(0).getType());
+ if (!expectedEltPtrTy)
+ return op.emitOpError("block argument must be a !cir.ptr type");
+
+ if (op.getNumElements()) {
+ auto recTy = mlir::dyn_cast<cir::RecordType>(pointeeTy);
+ if (!recTy)
+ return op.emitOpError(
+ "when 'num_elements' is present, 'addr' must be a pointer to a "
+ "!cir.struct or !cir.union type");
+
+ if (expectedEltPtrTy != ptrTy)
+ return op.emitOpError("when 'num_elements' is present, 'addr' type must "
+ "match the block argument type");
+ } else {
+ auto arrayTy = mlir::dyn_cast<cir::ArrayType>(pointeeTy);
+ if (!arrayTy)
+ return op.emitOpError(
+ "when 'num_elements' is absent, 'addr' must be a pointer to a "
+ "!cir.array type");
+
+ mlir::Type innerEltTy = arrayTy.getElementType();
+ while (auto nested = mlir::dyn_cast<cir::ArrayType>(innerEltTy))
+ innerEltTy = nested.getElementType();
+
+ auto recTy = mlir::dyn_cast<cir::RecordType>(innerEltTy);
+ if (!recTy)
+ return op.emitOpError("the block argument type must be a pointer to a "
+ "!cir.struct or !cir.union type");
+
+ if (expectedEltPtrTy.getPointee() != innerEltTy)
+ return op.emitOpError(
+ "block argument pointee type must match the innermost array "
+ "element type");
+ }
+
+ return success();
+}
+
+LogicalResult cir::ArrayCtor::verify() {
+ if (failed(verifyArrayCtorDtor(*this)))
+ return failure();
+
+ mlir::Region &partialDtor = getPartialDtor();
+ if (!partialDtor.empty()) {
+ mlir::Block &dtorBlock = partialDtor.front();
+ if (dtorBlock.getNumArguments() != 1)
+ return emitOpError("partial_dtor must have exactly one block argument");
+
+ auto bodyArgTy = getBody().front().getArgument(0).getType();
+ if (dtorBlock.getArgument(0).getType() != bodyArgTy)
+ return emitOpError("partial_dtor block argument type must match "
+ "the body block argument type");
+ }
+ return success();
+}
+LogicalResult cir::ArrayDtor::verify() { return verifyArrayCtorDtor(*this); }
+
+//===----------------------------------------------------------------------===//
+// VTableAddrPointOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult
+cir::VTableAddrPointOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
+ StringRef name = getName();
+
+ // Verify that the result type underlying pointer type matches the type of
+ // the referenced cir.global.
+ auto op =
+ symbolTable.lookupNearestSymbolFrom<cir::GlobalOp>(*this, getNameAttr());
+ if (!op)
+ return emitOpError("'")
+ << name << "' does not reference a valid cir.global";
+ std::optional<mlir::Attribute> init = op.getInitialValue();
+ if (!init)
+ return success();
+ if (!isa<cir::VTableAttr>(*init))
+ return emitOpError("Expected #cir.vtable in initializer for global '")
+ << name << "'";
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// VTTAddrPointOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult
+cir::VTTAddrPointOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
+ // VTT ptr is not coming from a symbol.
+ if (!getName())
+ return success();
+ StringRef name = *getName();
+
+ // Verify that the result type underlying pointer type matches the type of
+ // the referenced cir.global op.
+ auto op =
+ symbolTable.lookupNearestSymbolFrom<cir::GlobalOp>(*this, getNameAttr());
+ if (!op)
+ return emitOpError("'")
+ << name << "' does not reference a valid cir.global";
+ std::optional<mlir::Attribute> init = op.getInitialValue();
+ if (!init)
+ return success();
+ if (!isa<cir::ConstArrayAttr>(*init))
+ return emitOpError(
+ "Expected constant array in initializer for global VTT '")
+ << name << "'";
+ return success();
+}
+
+LogicalResult cir::VTTAddrPointOp::verify() {
+ // The operation uses either a symbol or a value to operate, but not both
+ if (getName() && getSymAddr())
+ return emitOpError("should use either a symbol or value, but not both");
+
+ // If not a symbol, stick with the concrete type used for getSymAddr.
+ if (getSymAddr())
+ return success();
+
+ mlir::Type resultType = getAddr().getType();
+ mlir::Type resTy = cir::PointerType::get(
+ cir::PointerType::get(cir::VoidType::get(getContext())));
+
+ if (resultType != resTy)
+ return emitOpError("result type must be ")
+ << resTy << ", but provided result type is " << resultType;
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// BaseClassAddrOp & DerivedClassAddrOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::BaseClassAddrOp::verify() {
+ if (getSrcAddr().getType().getAddrSpace() != getType().getAddrSpace())
+ return emitOpError() << "address space mismatch";
+ return mlir::success();
+}
+
+LogicalResult cir::DerivedClassAddrOp::verify() {
+ if (getSrcAddr().getType().getAddrSpace() != getType().getAddrSpace())
+ return emitOpError() << "address space mismatch";
+ return mlir::success();
+}
+
+//===----------------------------------------------------------------------===//
+// BaseDataMemberOp & DerivedDataMemberOp
+//===----------------------------------------------------------------------===//
+
+static LogicalResult verifyMemberPtrCast(Operation *op, mlir::Value src,
+ mlir::Type resultTy) {
+ // Let the operand type be T1 C1::*, let the result type be T2 C2::*.
+ // Verify that T1 and T2 are the same type.
+ mlir::Type inputMemberTy;
+ mlir::Type resultMemberTy;
+ if (mlir::isa<cir::DataMemberType>(src.getType())) {
+ inputMemberTy =
+ mlir::cast<cir::DataMemberType>(src.getType()).getMemberTy();
+ resultMemberTy = mlir::cast<cir::DataMemberType>(resultTy).getMemberTy();
+ }
+ assert(!cir::MissingFeatures::memberFuncPtrCast());
+ if (inputMemberTy != resultMemberTy)
+ return op->emitOpError()
+ << "member types of the operand and the result do not match";
+
+ return mlir::success();
+}
+
+LogicalResult cir::BaseDataMemberOp::verify() {
+ return verifyMemberPtrCast(getOperation(), getSrc(), getType());
+}
+
+LogicalResult cir::DerivedDataMemberOp::verify() {
+ return verifyMemberPtrCast(getOperation(), getSrc(), getType());
+}
+
+//===----------------------------------------------------------------------===//
+// BaseMethodOp & DerivedMethodOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::BaseMethodOp::verify() {
+ return verifyMemberPtrCast(getOperation(), getSrc(), getType());
+}
+
+LogicalResult cir::DerivedMethodOp::verify() {
+ return verifyMemberPtrCast(getOperation(), getSrc(), getType());
+}
+
+//===----------------------------------------------------------------------===//
+// GetRuntimeMemberOp Definitions
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::GetRuntimeMemberOp::verify() {
+ cir::DataMemberType memberPtrTy = getMember().getType();
+
+ if (getAddr().getType().getPointee() != memberPtrTy.getClassTy())
+ return emitError() << "record type does not match the member pointer type";
+ if (getType().getPointee() != memberPtrTy.getMemberTy())
+ return emitError() << "result type does not match the member pointer type";
+ return mlir::success();
+}
+
+//===----------------------------------------------------------------------===//
+// GetMethodOp Definitions
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::GetMethodOp::verify() {
+ cir::MethodType methodTy = getMethod().getType();
+
+ // Assume objectTy is !cir.ptr<!T>
+ cir::PointerType objectPtrTy = getObject().getType();
+ mlir::Type objectTy = objectPtrTy.getPointee();
+
+ if (methodTy.getClassTy() != objectTy)
+ return emitError() << "method class type and object type do not match";
+
+ // Assume methodFuncTy is !cir.func<!Ret (!Args)>
+ auto calleeTy = mlir::cast<cir::FuncType>(getCallee().getType().getPointee());
+ cir::FuncType methodFuncTy = methodTy.getMemberFuncTy();
+
+ // We verify at here that calleeTy is !cir.func<!Ret (!cir.ptr<!void>, !Args)>
+ // Note that the first parameter type of the callee is !cir.ptr<!void> instead
+ // of !cir.ptr<!T> because the "this" pointer may be adjusted before calling
+ // the callee.
+
+ if (methodFuncTy.getReturnType() != calleeTy.getReturnType())
+ return emitError()
+ << "method return type and callee return type do not match";
+
+ llvm::ArrayRef<mlir::Type> calleeArgsTy = calleeTy.getInputs();
+ llvm::ArrayRef<mlir::Type> methodFuncArgsTy = methodFuncTy.getInputs();
+
+ if (calleeArgsTy.empty())
+ return emitError() << "callee parameter list lacks receiver object ptr";
+
+ auto calleeThisArgPtrTy = mlir::dyn_cast<cir::PointerType>(calleeArgsTy[0]);
+ if (!calleeThisArgPtrTy ||
+ !mlir::isa<cir::VoidType>(calleeThisArgPtrTy.getPointee())) {
+ return emitError()
+ << "the first parameter of callee must be a void pointer";
+ }
+
+ if (calleeArgsTy.size() != methodFuncArgsTy.size())
+ return emitError() << "callee and method parameter counts do not match";
+
+ if (calleeArgsTy.size() > 1 &&
+ calleeArgsTy.slice(1) != methodFuncArgsTy.slice(1))
+ return emitError()
+ << "callee parameters and method parameters do not match";
+
+ return mlir::success();
+}
+
+//===----------------------------------------------------------------------===//
+// TableGen-generated op definitions
+//===----------------------------------------------------------------------===//
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsClassMembers.cpp.inc"
diff --git a/clang/lib/CIR/Dialect/IR/CIROpsComplex.cpp b/clang/lib/CIR/Dialect/IR/CIROpsComplex.cpp
new file mode 100644
index 00000000000000..396665117d4e06
--- /dev/null
+++ b/clang/lib/CIR/Dialect/IR/CIROpsComplex.cpp
@@ -0,0 +1,147 @@
+//===- CIROpsComplex.cpp - CIR ops in partition `Complex` -----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the CIR ops in partition `Complex`: the hand-written
+// parts first, then the TableGen-generated definitions.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/CIR/Dialect/IR/CIROpsComplex.h"
+
+#include "CIROpDefinitions.h"
+
+#include "mlir/IR/DialectImplementation.h"
+
+using namespace mlir;
+using namespace cir;
+
+//===----------------------------------------------------------------------===//
+// ComplexCreateOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::ComplexCreateOp::verify() {
+ if (getType().getElementType() != getReal().getType()) {
+ emitOpError()
+ << "operand type of cir.complex.create does not match its result type";
+ return failure();
+ }
+
+ return success();
+}
+
+OpFoldResult cir::ComplexCreateOp::fold(FoldAdaptor adaptor) {
+ mlir::Attribute real = adaptor.getReal();
+ mlir::Attribute imag = adaptor.getImag();
+ if (!real || !imag)
+ return {};
+
+ // When both of real and imag are constants, we can fold the operation into an
+ // `#cir.const_complex` operation.
+ auto realAttr = mlir::cast<mlir::TypedAttr>(real);
+ auto imagAttr = mlir::cast<mlir::TypedAttr>(imag);
+ return cir::ConstComplexAttr::get(realAttr, imagAttr);
+}
+
+//===----------------------------------------------------------------------===//
+// ComplexRealOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::ComplexRealOp::verify() {
+ mlir::Type operandTy = getOperand().getType();
+ if (auto complexOperandTy = mlir::dyn_cast<cir::ComplexType>(operandTy))
+ operandTy = complexOperandTy.getElementType();
+
+ if (getType() != operandTy) {
+ emitOpError() << ": result type does not match operand type";
+ return failure();
+ }
+
+ return success();
+}
+
+OpFoldResult cir::ComplexRealOp::fold(FoldAdaptor adaptor) {
+ if (!mlir::isa<cir::ComplexType>(getOperand().getType()))
+ return nullptr;
+
+ if (auto complexCreateOp = getOperand().getDefiningOp<cir::ComplexCreateOp>())
+ return complexCreateOp.getOperand(0);
+
+ auto complex =
+ mlir::cast_if_present<cir::ConstComplexAttr>(adaptor.getOperand());
+ return complex ? complex.getReal() : nullptr;
+}
+
+//===----------------------------------------------------------------------===//
+// ComplexImagOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::ComplexImagOp::verify() {
+ mlir::Type operandTy = getOperand().getType();
+ if (auto complexOperandTy = mlir::dyn_cast<cir::ComplexType>(operandTy))
+ operandTy = complexOperandTy.getElementType();
+
+ if (getType() != operandTy) {
+ emitOpError() << ": result type does not match operand type";
+ return failure();
+ }
+
+ return success();
+}
+
+OpFoldResult cir::ComplexImagOp::fold(FoldAdaptor adaptor) {
+ if (!mlir::isa<cir::ComplexType>(getOperand().getType()))
+ return nullptr;
+
+ if (auto complexCreateOp = getOperand().getDefiningOp<cir::ComplexCreateOp>())
+ return complexCreateOp.getOperand(1);
+
+ auto complex =
+ mlir::cast_if_present<cir::ConstComplexAttr>(adaptor.getOperand());
+ return complex ? complex.getImag() : nullptr;
+}
+
+//===----------------------------------------------------------------------===//
+// ComplexRealPtrOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::ComplexRealPtrOp::verify() {
+ mlir::Type resultPointeeTy = getType().getPointee();
+ cir::PointerType operandPtrTy = getOperand().getType();
+ auto operandPointeeTy =
+ mlir::cast<cir::ComplexType>(operandPtrTy.getPointee());
+
+ if (resultPointeeTy != operandPointeeTy.getElementType()) {
+ return emitOpError() << ": result type does not match operand type";
+ }
+
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// ComplexImagPtrOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::ComplexImagPtrOp::verify() {
+ mlir::Type resultPointeeTy = getType().getPointee();
+ cir::PointerType operandPtrTy = getOperand().getType();
+ auto operandPointeeTy =
+ mlir::cast<cir::ComplexType>(operandPtrTy.getPointee());
+
+ if (resultPointeeTy != operandPointeeTy.getElementType()) {
+ return emitOpError()
+ << "cir.complex.imag_ptr result type does not match operand type";
+ }
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// TableGen-generated op definitions
+//===----------------------------------------------------------------------===//
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsComplex.cpp.inc"
diff --git a/clang/lib/CIR/Dialect/IR/CIROpsControlFlow.cpp b/clang/lib/CIR/Dialect/IR/CIROpsControlFlow.cpp
new file mode 100644
index 00000000000000..46bd2209cd5c2a
--- /dev/null
+++ b/clang/lib/CIR/Dialect/IR/CIROpsControlFlow.cpp
@@ -0,0 +1,905 @@
+//===- CIROpsControlFlow.cpp - CIR ops in partition `ControlFlow` ---------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the CIR ops in partition `ControlFlow`: the hand-written
+// parts first, then the TableGen-generated definitions.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/CIR/Dialect/IR/CIROpsControlFlow.h"
+#include "clang/CIR/Dialect/IR/CIROpsBuiltins.h"
+#include "clang/CIR/Dialect/IR/CIROpsClassMembers.h"
+#include "clang/CIR/Dialect/IR/CIROpsCore.h"
+#include "clang/CIR/Dialect/IR/CIROpsCoroutine.h"
+#include "clang/CIR/Dialect/IR/CIROpsEH.h"
+#include "clang/CIR/Dialect/IR/CIROpsSymbols.h"
+
+#include "CIROpDefinitions.h"
+
+#include "mlir/IR/DialectImplementation.h"
+#include "mlir/IR/PatternMatch.h"
+
+using namespace mlir;
+using namespace cir;
+
+// Check if a region's termination omission is valid and, if so, creates and
+// inserts the omitted terminator into the region.
+LogicalResult cir::ensureRegionTerm(OpAsmParser &parser, Region ®ion,
+ SMLoc errLoc) {
+ Location eLoc = parser.getEncodedSourceLoc(parser.getCurrentLocation());
+ OpBuilder builder(parser.getBuilder().getContext());
+
+ // Insert empty block in case the region is empty to ensure the terminator
+ // will be inserted
+ if (region.empty())
+ builder.createBlock(®ion);
+
+ Block &block = region.back();
+ // Region is properly terminated: nothing to do.
+ if (!block.empty() && block.back().hasTrait<OpTrait::IsTerminator>())
+ return success();
+
+ // Check for invalid terminator omissions.
+ if (!region.hasOneBlock())
+ return parser.emitError(errLoc,
+ "multi-block region must not omit terminator");
+
+ // Terminator was omitted correctly: recreate it.
+ builder.setInsertionPointToEnd(&block);
+ cir::YieldOp::create(builder, eLoc);
+ return success();
+}
+
+// True if the region's terminator should be omitted.
+bool cir::omitRegionTerm(mlir::Region &r) {
+ const auto singleNonEmptyBlock = r.hasOneBlock() && !r.back().empty();
+ const auto yieldsNothing = [&r]() {
+ auto y = dyn_cast<cir::YieldOp>(r.back().getTerminator());
+ return y && y.getArgs().empty();
+ };
+ return singleNonEmptyBlock && yieldsNothing();
+}
+
+//===----------------------------------------------------------------------===//
+// CIR Custom Parsers/Printers
+//===----------------------------------------------------------------------===//
+
+mlir::ParseResult cir::parseOmittedTerminatorRegion(mlir::OpAsmParser &parser,
+ mlir::Region ®ion) {
+ auto regionLoc = parser.getCurrentLocation();
+ if (parser.parseRegion(region))
+ return failure();
+ if (ensureRegionTerm(parser, region, regionLoc).failed())
+ return failure();
+ return success();
+}
+
+void cir::printOmittedTerminatorRegion(mlir::OpAsmPrinter &printer,
+ cir::ScopeOp &op, mlir::Region ®ion) {
+ printer.printRegion(region,
+ /*printEntryBlockArgs=*/false,
+ /*printBlockTerminators=*/!omitRegionTerm(region));
+}
+
+//===----------------------------------------------------------------------===//
+// ConditionOp
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------
+// BranchOpTerminatorInterface Methods
+//===----------------------------------
+
+void cir::ConditionOp::getSuccessorRegions(
+ ArrayRef<Attribute> operands, SmallVectorImpl<RegionSuccessor> ®ions) {
+ // TODO(cir): The condition value may be folded to a constant, narrowing
+ // down its list of possible successors.
+
+ // Parent is a loop: condition may branch to the body, or on the false edge
+ // to the per-iteration cleanup region if present, otherwise to the parent op.
+ if (auto loopOp = dyn_cast<LoopOpInterface>(getOperation()->getParentOp())) {
+ regions.emplace_back(&loopOp.getBody());
+ if (mlir::Region *cleanup = loopOp.maybeGetCleanup())
+ regions.emplace_back(cleanup);
+ else
+ regions.emplace_back(getOperation());
+ return;
+ }
+
+ // Parent is an await: condition may branch to resume or suspend regions.
+ auto await = cast<AwaitOp>(getOperation()->getParentOp());
+ regions.emplace_back(&await.getResume());
+ regions.emplace_back(&await.getSuspend());
+}
+
+MutableOperandRange
+cir::ConditionOp::getMutableSuccessorOperands(RegionSuccessor point) {
+ // No values are yielded to the successor region.
+ return MutableOperandRange(getOperation(), 0, 0);
+}
+
+MutableOperandRange
+cir::ResumeOp::getMutableSuccessorOperands(RegionSuccessor point) {
+ // The eh_token operand is not modeled as forwarded to any successor.
+ return MutableOperandRange(getOperation(), 0, 0);
+}
+
+LogicalResult cir::ConditionOp::verify() {
+ if (!isa<LoopOpInterface, AwaitOp>(getOperation()->getParentOp()))
+ return emitOpError("condition must be within a conditional region");
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// WhileOp & ForOp
+//===----------------------------------------------------------------------===//
+
+template <typename LoopOpTy>
+static LogicalResult verifyLoopCleanup(LoopOpTy op) {
+ std::optional<cir::CleanupKind> cleanupKind = op.getCleanupKind();
+
+ // The cleanup kind attribute must be present exactly when a (non-empty)
+ // cleanup region is present.
+ if (cleanupKind.has_value() == op.getCleanup().empty())
+ return op.emitOpError("cleanup kind must be present if and only if the "
+ "cleanup region is non-empty");
+
+ // A loop's per-iteration cleanup runs on every normal exit edge (loop exit,
+ // end of iteration, break/continue), so an EH-only cleanup is meaningless.
+ // Only 'normal' (exceptions disabled) and 'all' (normal + EH unwind) apply.
+ if (cleanupKind == cir::CleanupKind::EH)
+ return op.emitOpError("loop cleanup kind must be 'normal' or 'all', "
+ "not 'eh'");
+
+ return success();
+}
+
+LogicalResult cir::WhileOp::verify() { return verifyLoopCleanup(*this); }
+
+LogicalResult cir::ForOp::verify() { return verifyLoopCleanup(*this); }
+
+//===----------------------------------------------------------------------===//
+// ReturnOp
+//===----------------------------------------------------------------------===//
+
+static mlir::LogicalResult checkReturnAndFunction(cir::ReturnOp op,
+ cir::FuncOp function) {
+ // ReturnOps currently only have a single optional operand.
+ if (op.getNumOperands() > 1)
+ return op.emitOpError() << "expects at most 1 return operand";
+
+ // Ensure returned type matches the function signature.
+ auto expectedTy = function.getFunctionType().getReturnType();
+ auto actualTy =
+ (op.getNumOperands() == 0 ? cir::VoidType::get(op.getContext())
+ : op.getOperand(0).getType());
+ if (actualTy != expectedTy)
+ return op.emitOpError() << "returns " << actualTy
+ << " but enclosing function returns " << expectedTy;
+
+ return mlir::success();
+}
+
+mlir::LogicalResult cir::ReturnOp::verify() {
+ // Returns can be present in multiple different scopes, get the
+ // wrapping function and start from there.
+ auto *fnOp = getOperation()->getParentOp();
+ while (!isa<cir::FuncOp>(fnOp))
+ fnOp = fnOp->getParentOp();
+
+ // Make sure return types match function return type.
+ if (checkReturnAndFunction(*this, cast<cir::FuncOp>(fnOp)).failed())
+ return failure();
+
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// IfOp
+//===----------------------------------------------------------------------===//
+
+ParseResult cir::IfOp::parse(OpAsmParser &parser, OperationState &result) {
+ // create the regions for 'then'.
+ result.regions.reserve(2);
+ Region *thenRegion = result.addRegion();
+ Region *elseRegion = result.addRegion();
+
+ mlir::Builder &builder = parser.getBuilder();
+ OpAsmParser::UnresolvedOperand cond;
+ Type boolType = cir::BoolType::get(builder.getContext());
+
+ if (parser.parseOperand(cond) ||
+ parser.resolveOperand(cond, boolType, result.operands))
+ return failure();
+
+ // Parse 'then' region.
+ mlir::SMLoc parseThenLoc = parser.getCurrentLocation();
+ if (parser.parseRegion(*thenRegion, /*arguments=*/{}, /*argTypes=*/{}))
+ return failure();
+
+ if (ensureRegionTerm(parser, *thenRegion, parseThenLoc).failed())
+ return failure();
+
+ // If we find an 'else' keyword, parse the 'else' region.
+ if (!parser.parseOptionalKeyword("else")) {
+ mlir::SMLoc parseElseLoc = parser.getCurrentLocation();
+ if (parser.parseRegion(*elseRegion, /*arguments=*/{}, /*argTypes=*/{}))
+ return failure();
+ if (ensureRegionTerm(parser, *elseRegion, parseElseLoc).failed())
+ return failure();
+ }
+
+ // Parse the optional attribute list.
+ if (parser.parseOptionalAttrDict(result.attributes))
+ return failure();
+ return success();
+}
+
+void cir::IfOp::print(OpAsmPrinter &p) {
+ p << " " << getCondition() << " ";
+ mlir::Region &thenRegion = this->getThenRegion();
+ p.printRegion(thenRegion,
+ /*printEntryBlockArgs=*/false,
+ /*printBlockTerminators=*/!omitRegionTerm(thenRegion));
+
+ // Print the 'else' regions if it exists and has a block.
+ mlir::Region &elseRegion = this->getElseRegion();
+ if (!elseRegion.empty()) {
+ p << " else ";
+ p.printRegion(elseRegion,
+ /*printEntryBlockArgs=*/false,
+ /*printBlockTerminators=*/!omitRegionTerm(elseRegion));
+ }
+
+ p.printOptionalAttrDict(
+ getOperation()->getDiscardableAttrDictionary().getValue());
+}
+
+/// Default callback for IfOp builders.
+void cir::buildTerminatedBody(OpBuilder &builder, Location loc) {
+ // add cir.yield to end of the block
+ cir::YieldOp::create(builder, loc);
+}
+
+/// Given the region at `index`, or the parent operation if `index` is None,
+/// return the successor regions. These are the regions that may be selected
+/// during the flow of control. `operands` is a set of optional attributes that
+/// correspond to a constant value for each operand, or null if that operand is
+/// not a constant.
+void cir::IfOp::getSuccessorRegions(mlir::RegionBranchPoint point,
+ SmallVectorImpl<RegionSuccessor> ®ions) {
+ // The `then` and the `else` region branch back to the parent operation.
+ if (!point.isParent()) {
+ regions.emplace_back(getOperation());
+ return;
+ }
+
+ // Don't consider the else region if it is empty.
+ Region *elseRegion = &this->getElseRegion();
+ if (elseRegion->empty())
+ elseRegion = nullptr;
+
+ // If the condition isn't constant, both regions may be executed.
+ regions.push_back(RegionSuccessor(&getThenRegion()));
+ if (elseRegion)
+ regions.push_back(RegionSuccessor(elseRegion));
+ else
+ regions.emplace_back(getOperation());
+}
+
+void cir::IfOp::build(OpBuilder &builder, OperationState &result, Value cond,
+ bool withElseRegion, BuilderCallbackRef thenBuilder,
+ BuilderCallbackRef elseBuilder) {
+ assert(thenBuilder && "the builder callback for 'then' must be present");
+ result.addOperands(cond);
+
+ OpBuilder::InsertionGuard guard(builder);
+ Region *thenRegion = result.addRegion();
+ builder.createBlock(thenRegion);
+ thenBuilder(builder, result.location);
+
+ Region *elseRegion = result.addRegion();
+ if (!withElseRegion)
+ return;
+
+ builder.createBlock(elseRegion);
+ elseBuilder(builder, result.location);
+}
+
+//===----------------------------------------------------------------------===//
+// ScopeOp
+//===----------------------------------------------------------------------===//
+
+void cir::ScopeOp::build(
+ OpBuilder &builder, OperationState &result,
+ function_ref<void(OpBuilder &, Type &, Location)> scopeBuilder) {
+ assert(scopeBuilder && "the builder callback for 'then' must be present");
+
+ OpBuilder::InsertionGuard guard(builder);
+ Region *scopeRegion = result.addRegion();
+ builder.createBlock(scopeRegion);
+ assert(!cir::MissingFeatures::opScopeCleanupRegion());
+
+ mlir::Type yieldTy;
+ scopeBuilder(builder, yieldTy, result.location);
+
+ if (yieldTy)
+ result.addTypes(TypeRange{yieldTy});
+}
+
+void cir::ScopeOp::build(
+ OpBuilder &builder, OperationState &result,
+ function_ref<void(OpBuilder &, Location)> scopeBuilder) {
+ assert(scopeBuilder && "the builder callback for 'then' must be present");
+ OpBuilder::InsertionGuard guard(builder);
+ Region *scopeRegion = result.addRegion();
+ builder.createBlock(scopeRegion);
+ assert(!cir::MissingFeatures::opScopeCleanupRegion());
+ scopeBuilder(builder, result.location);
+}
+
+LogicalResult cir::ScopeOp::verify() {
+ if (getRegion().empty()) {
+ return emitOpError() << "cir.scope must not be empty since it should "
+ "include at least an implicit cir.yield ";
+ }
+
+ mlir::Block &lastBlock = getRegion().back();
+ if (lastBlock.empty() || !lastBlock.mightHaveTerminator() ||
+ !lastBlock.getTerminator()->hasTrait<OpTrait::IsTerminator>())
+ return emitOpError() << "last block of cir.scope must be terminated";
+ return success();
+}
+
+LogicalResult cir::ScopeOp::fold(FoldAdaptor /*adaptor*/,
+ SmallVectorImpl<OpFoldResult> &results) {
+ // Only fold "trivial" scopes: a single block containing only a `cir.yield`.
+ if (!getRegion().hasOneBlock())
+ return failure();
+ Block &block = getRegion().front();
+ if (block.getOperations().size() != 1)
+ return failure();
+
+ auto yield = dyn_cast<cir::YieldOp>(block.front());
+ if (!yield)
+ return failure();
+
+ // Only fold when the scope produces a value.
+ if (getNumResults() != 1 || yield.getNumOperands() != 1)
+ return failure();
+
+ results.push_back(yield.getOperand(0));
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// CleanupScopeOp
+//===----------------------------------------------------------------------===//
+
+static bool isRedundantBeforeReturn(mlir::Region &cleanupRegion) {
+ for (mlir::Block &block : cleanupRegion) {
+ for (mlir::Operation &op : block) {
+ if (isa<cir::YieldOp, cir::LifetimeEndOp, cir::StackRestoreOp>(op))
+ continue;
+ // The stack restore reloads the pointer saved before the VLA.
+ auto loadOp = dyn_cast<cir::LoadOp>(op);
+ if (loadOp && loadOp.getResult().hasOneUse() &&
+ isa<cir::StackRestoreOp>(*loadOp.getResult().getUsers().begin()))
+ continue;
+ return false;
+ }
+ }
+ return true;
+}
+
+LogicalResult cir::CleanupScopeOp::verify() {
+ // If the cleanup contains a musttail call, it must be a cleanup that can be
+ // skipped on return (such as a lifetime end or a stack restore). Other
+ // cleanups must never contain musttail calls.
+ cir::CallOp mustTailCall;
+ getBodyRegion().walk([&](cir::CallOp callOp) {
+ if (!callOp.getMusttail())
+ return WalkResult::advance();
+ mustTailCall = callOp;
+ return WalkResult::interrupt();
+ });
+ if (!mustTailCall)
+ return success();
+
+ if (isRedundantBeforeReturn(getCleanupRegion()))
+ return success();
+
+ InFlightDiagnostic diag =
+ emitOpError("cleanup is not redundant before a return, so it cannot be "
+ "skipped by a musttail call");
+ diag.attachNote(mustTailCall.getLoc()) << "musttail call is here";
+ return diag;
+}
+
+void cir::CleanupScopeOp::getSuccessorRegions(
+ mlir::RegionBranchPoint point, SmallVectorImpl<RegionSuccessor> ®ions) {
+ // The cleanup can also run from the middle of the body, on an unwind out of
+ // a call or on a cir.return, cir.break, cir.continue or cir.goto. None of
+ // these is a branch point, so the cleanup region is a successor of the
+ // parent and of every body exit to stand in for them.
+ if (point.isParent()) {
+ regions.reserve(regions.size() + 2);
+ regions.emplace_back(&getBodyRegion());
+ regions.emplace_back(&getCleanupRegion());
+ return;
+ }
+
+ mlir::Operation *term = point.getTerminatorPredecessorOrNull();
+ if (term->getParentRegion() == &getCleanupRegion()) {
+ regions.emplace_back(getOperation());
+ return;
+ }
+
+ regions.emplace_back(&getCleanupRegion());
+
+ // cir.resume unwinds out of the body, and every other terminator is a
+ // normal exit. An exit that the cleanup kind does not cover can also leave
+ // the operation directly.
+ cir::CleanupKindAttr kind = getCleanupKindAttr();
+ if (isa<cir::ResumeOp>(term) ? !kind.isEH() : !kind.isNormal())
+ regions.emplace_back(getOperation());
+}
+
+LogicalResult cir::CleanupScopeOp::canonicalize(CleanupScopeOp op,
+ PatternRewriter &rewriter) {
+ auto isRegionTrivial = [](Region ®ion) {
+ assert(!region.empty() && "CleanupScopeOp regions must not be empty");
+ if (!region.hasOneBlock())
+ return false;
+ Block &block = llvm::getSingleElement(region);
+ return llvm::hasSingleElement(block) &&
+ isa<cir::YieldOp>(llvm::getSingleElement(block));
+ };
+
+ Region &body = op.getBodyRegion();
+ Region &cleanup = op.getCleanupRegion();
+
+ // An EH-only cleanup scope with an empty body can never trigger its cleanup
+ // region — there are no operations in the body that could throw. Erase it.
+ if (op.getCleanupKind() == CleanupKind::EH && isRegionTrivial(body)) {
+ rewriter.eraseOp(op);
+ return success();
+ }
+
+ // A cleanup scope with a trivial cleanup region has no cleanup to perform.
+ // Inline the body into the parent block and erase the scope.
+ if (!isRegionTrivial(cleanup) || !body.hasOneBlock())
+ return failure();
+
+ Block &bodyBlock = body.front();
+ if (!isa<cir::YieldOp>(bodyBlock.getTerminator()))
+ return failure();
+
+ Operation *yield = bodyBlock.getTerminator();
+ rewriter.inlineBlockBefore(&bodyBlock, op);
+ rewriter.eraseOp(yield);
+ rewriter.eraseOp(op);
+ return success();
+}
+
+void cir::CleanupScopeOp::build(
+ OpBuilder &builder, OperationState &result, CleanupKind cleanupKind,
+ function_ref<void(OpBuilder &, Location)> bodyBuilder,
+ function_ref<void(OpBuilder &, Location)> cleanupBuilder) {
+ result.addAttribute(getCleanupKindAttrName(result.name),
+ CleanupKindAttr::get(builder.getContext(), cleanupKind));
+
+ OpBuilder::InsertionGuard guard(builder);
+
+ // Build body region.
+ Region *bodyRegion = result.addRegion();
+ builder.createBlock(bodyRegion);
+ if (bodyBuilder)
+ bodyBuilder(builder, result.location);
+
+ // Build cleanup region.
+ Region *cleanupRegion = result.addRegion();
+ builder.createBlock(cleanupRegion);
+ if (cleanupBuilder)
+ cleanupBuilder(builder, result.location);
+}
+
+//===----------------------------------------------------------------------===//
+// BrOp
+//===----------------------------------------------------------------------===//
+
+/// Merges blocks connected by a unique unconditional branch.
+///
+/// ^bb0: ^bb0:
+/// ... ...
+/// cir.br ^bb1 => ...
+/// ^bb1: cir.return
+/// ...
+/// cir.return
+LogicalResult cir::BrOp::canonicalize(BrOp op, PatternRewriter &rewriter) {
+ Block *src = op->getBlock();
+ Block *dst = op.getDest();
+
+ // Do not fold self-loops.
+ if (src == dst)
+ return failure();
+
+ // Only merge when this is the unique edge between the blocks.
+ if (src->getNumSuccessors() != 1 || dst->getSinglePredecessor() != src)
+ return failure();
+
+ // Don't merge blocks that start with LabelOp or IndirectBrOp.
+ // This is to avoid merging blocks that have an indirect predecessor.
+ if (isa<cir::LabelOp, cir::IndirectBrOp>(dst->front()))
+ return failure();
+
+ // An operand that is an argument of the destination itself (possible in an
+ // unreachable cycle) would be replaced by itself, leaving its other uses
+ // dangling once the destination is erased.
+ if (llvm::any_of(op.getDestOperands(), [&](Value operand) {
+ auto arg = dyn_cast<BlockArgument>(operand);
+ return arg && arg.getOwner() == dst;
+ }))
+ return failure();
+
+ // Copy the operands out: erasing the branch frees its operand storage.
+ SmallVector<Value> operands(op.getDestOperands());
+ rewriter.eraseOp(op);
+ rewriter.mergeBlocks(dst, src, operands);
+ return success();
+}
+
+mlir::SuccessorOperands cir::BrOp::getSuccessorOperands(unsigned index) {
+ assert(index == 0 && "invalid successor index");
+ return mlir::SuccessorOperands(getDestOperandsMutable());
+}
+
+Block *cir::BrOp::getSuccessorForOperands(ArrayRef<Attribute>) {
+ return getDest();
+}
+
+//===----------------------------------------------------------------------===//
+// IndirectBrCondOp
+//===----------------------------------------------------------------------===//
+
+mlir::SuccessorOperands
+cir::IndirectBrOp::getSuccessorOperands(unsigned index) {
+ assert(index < getNumSuccessors() && "invalid successor index");
+ return mlir::SuccessorOperands(getSuccOperandsMutable()[index]);
+}
+
+ParseResult cir::parseIndirectBrOpSucessors(
+ OpAsmParser &parser, Type &flagType,
+ SmallVectorImpl<Block *> &succOperandBlocks,
+ SmallVectorImpl<SmallVector<OpAsmParser::UnresolvedOperand>> &succOperands,
+ SmallVectorImpl<SmallVector<Type>> &succOperandsTypes) {
+ if (failed(parser.parseCommaSeparatedList(
+ OpAsmParser::Delimiter::Square,
+ [&]() {
+ Block *destination = nullptr;
+ SmallVector<OpAsmParser::UnresolvedOperand> operands;
+ SmallVector<Type> operandTypes;
+
+ if (parser.parseSuccessor(destination).failed())
+ return failure();
+
+ if (succeeded(parser.parseOptionalLParen())) {
+ if (failed(parser.parseOperandList(
+ operands, OpAsmParser::Delimiter::None)) ||
+ failed(parser.parseColonTypeList(operandTypes)) ||
+ failed(parser.parseRParen()))
+ return failure();
+ }
+ succOperandBlocks.push_back(destination);
+ succOperands.emplace_back(operands);
+ succOperandsTypes.emplace_back(operandTypes);
+ return success();
+ },
+ "successor blocks")))
+ return failure();
+ return success();
+}
+
+void cir::printIndirectBrOpSucessors(OpAsmPrinter &p, cir::IndirectBrOp op,
+ Type flagType, SuccessorRange succs,
+ OperandRangeRange succOperands,
+ const TypeRangeRange &succOperandsTypes) {
+ p << "[";
+ llvm::interleave(
+ llvm::zip(succs, succOperands),
+ [&](auto i) {
+ p.printNewline();
+ p.printSuccessorAndUseList(std::get<0>(i), std::get<1>(i));
+ },
+ [&] { p << ','; });
+ if (!succOperands.empty())
+ p.printNewline();
+ p << "]";
+}
+
+//===----------------------------------------------------------------------===//
+// BrCondOp
+//===----------------------------------------------------------------------===//
+
+mlir::SuccessorOperands cir::BrCondOp::getSuccessorOperands(unsigned index) {
+ assert(index < getNumSuccessors() && "invalid successor index");
+ return SuccessorOperands(index == 0 ? getDestOperandsTrueMutable()
+ : getDestOperandsFalseMutable());
+}
+
+Block *cir::BrCondOp::getSuccessorForOperands(ArrayRef<Attribute> operands) {
+ if (IntegerAttr condAttr = dyn_cast_if_present<IntegerAttr>(operands.front()))
+ return condAttr.getValue().isOne() ? getDestTrue() : getDestFalse();
+ return nullptr;
+}
+
+//===----------------------------------------------------------------------===//
+// CaseOp
+//===----------------------------------------------------------------------===//
+
+void cir::CaseOp::build(OpBuilder &builder, OperationState &result,
+ ArrayAttr value, CaseOpKind kind,
+ OpBuilder::InsertPoint &insertPoint) {
+ OpBuilder::InsertionGuard guardSwitch(builder);
+ result.addAttribute("value", value);
+ result.getOrAddProperties<Properties>().kind =
+ cir::CaseOpKindAttr::get(builder.getContext(), kind);
+ Region *caseRegion = result.addRegion();
+ builder.createBlock(caseRegion);
+
+ insertPoint = builder.saveInsertionPoint();
+}
+
+//===----------------------------------------------------------------------===//
+// SwitchOp
+//===----------------------------------------------------------------------===//
+
+void cir::SwitchOp::build(OpBuilder &builder, OperationState &result,
+ Value cond, BuilderOpStateCallbackRef switchBuilder) {
+ assert(switchBuilder && "the builder callback for regions must be present");
+ OpBuilder::InsertionGuard guardSwitch(builder);
+ Region *switchRegion = result.addRegion();
+ builder.createBlock(switchRegion);
+ result.addOperands({cond});
+ switchBuilder(builder, result.location, result);
+}
+
+void cir::SwitchOp::collectCases(llvm::SmallVectorImpl<CaseOp> &cases) {
+ walk<mlir::WalkOrder::PreOrder>([&](mlir::Operation *op) {
+ // Don't walk in nested switch op.
+ if (isa<cir::SwitchOp>(op) && op != *this)
+ return WalkResult::skip();
+
+ if (auto caseOp = dyn_cast<cir::CaseOp>(op))
+ cases.push_back(caseOp);
+
+ return WalkResult::advance();
+ });
+}
+
+bool cir::SwitchOp::isSimpleForm(llvm::SmallVectorImpl<CaseOp> &cases) {
+ collectCases(cases);
+
+ if (getBody().empty())
+ return false;
+
+ if (!isa<YieldOp>(getBody().front().back()))
+ return false;
+
+ if (!llvm::all_of(getBody().front(),
+ [](Operation &op) { return isa<CaseOp, YieldOp>(op); }))
+ return false;
+
+ return llvm::all_of(cases, [this](CaseOp op) {
+ return op->getParentOfType<SwitchOp>() == *this;
+ });
+}
+
+//===----------------------------------------------------------------------===//
+// SwitchFlatOp
+//===----------------------------------------------------------------------===//
+
+void cir::SwitchFlatOp::build(OpBuilder &builder, OperationState &result,
+ Value value, Block *defaultDestination,
+ ValueRange defaultOperands,
+ ArrayRef<APInt> caseValues,
+ BlockRange caseDestinations,
+ ArrayRef<ValueRange> caseOperands) {
+
+ std::vector<mlir::Attribute> caseValuesAttrs;
+ for (const APInt &val : caseValues)
+ caseValuesAttrs.push_back(cir::IntAttr::get(value.getType(), val));
+ mlir::ArrayAttr attrs = ArrayAttr::get(builder.getContext(), caseValuesAttrs);
+
+ build(builder, result, value, defaultOperands, caseOperands, attrs,
+ defaultDestination, caseDestinations);
+}
+
+SuccessorOperands cir::SwitchFlatOp::getSuccessorOperands(unsigned index) {
+ assert(index < getNumSuccessors() && "invalid successor index");
+ if (index == 0)
+ return SuccessorOperands(getDefaultOperandsMutable());
+ return SuccessorOperands(getCaseOperandsMutable()[index - 1]);
+}
+
+Block *
+cir::SwitchFlatOp::getSuccessorForOperands(ArrayRef<Attribute> operands) {
+ auto cond = dyn_cast_if_present<cir::IntAttr>(operands.front());
+ if (!cond)
+ return nullptr;
+ for (auto [value, dest] : llvm::zip(getCaseValues(), getCaseDestinations())) {
+ const APInt &caseValue = cast<cir::IntAttr>(value).getValue();
+ if (caseValue == cond.getValue())
+ return dest;
+ }
+ return getDefaultDestination();
+}
+
+/// <cases> ::= `[` (case (`,` case )* )? `]`
+/// <case> ::= integer `:` bb-id (`(` ssa-use-and-type-list `)`)?
+ParseResult cir::parseSwitchFlatOpCases(
+ OpAsmParser &parser, Type flagType, mlir::ArrayAttr &caseValues,
+ SmallVectorImpl<Block *> &caseDestinations,
+ SmallVectorImpl<llvm::SmallVector<OpAsmParser::UnresolvedOperand>>
+ &caseOperands,
+ SmallVectorImpl<llvm::SmallVector<Type>> &caseOperandTypes) {
+ if (failed(parser.parseLSquare()))
+ return failure();
+ if (succeeded(parser.parseOptionalRSquare()))
+ return success();
+ llvm::SmallVector<mlir::Attribute> values;
+
+ auto parseCase = [&]() {
+ int64_t value = 0;
+ if (failed(parser.parseInteger(value)))
+ return failure();
+
+ values.push_back(cir::IntAttr::get(flagType, value));
+
+ Block *destination;
+ llvm::SmallVector<OpAsmParser::UnresolvedOperand> operands;
+ llvm::SmallVector<Type> operandTypes;
+ if (parser.parseColon() || parser.parseSuccessor(destination))
+ return failure();
+ if (!parser.parseOptionalLParen()) {
+ if (parser.parseOperandList(operands, OpAsmParser::Delimiter::None,
+ /*allowResultNumber=*/false) ||
+ parser.parseColonTypeList(operandTypes) || parser.parseRParen())
+ return failure();
+ }
+ caseDestinations.push_back(destination);
+ caseOperands.emplace_back(operands);
+ caseOperandTypes.emplace_back(operandTypes);
+ return success();
+ };
+ if (failed(parser.parseCommaSeparatedList(parseCase)))
+ return failure();
+
+ caseValues = ArrayAttr::get(flagType.getContext(), values);
+
+ return parser.parseRSquare();
+}
+
+void cir::printSwitchFlatOpCases(OpAsmPrinter &p, cir::SwitchFlatOp op,
+ Type flagType, mlir::ArrayAttr caseValues,
+ SuccessorRange caseDestinations,
+ OperandRangeRange caseOperands,
+ const TypeRangeRange &caseOperandTypes) {
+ p << '[';
+ p.printNewline();
+ if (!caseValues) {
+ p << ']';
+ return;
+ }
+
+ size_t index = 0;
+ llvm::interleave(
+ llvm::zip(caseValues, caseDestinations),
+ [&](auto i) {
+ p << " ";
+ mlir::Attribute a = std::get<0>(i);
+ p << mlir::cast<cir::IntAttr>(a).getValue();
+ p << ": ";
+ p.printSuccessorAndUseList(std::get<1>(i), caseOperands[index++]);
+ },
+ [&] {
+ p << ',';
+ p.printNewline();
+ });
+ p.printNewline();
+ p << ']';
+}
+
+//===----------------------------------------------------------------------===//
+// TernaryOp
+//===----------------------------------------------------------------------===//
+
+void cir::TernaryOp::build(
+ OpBuilder &builder, OperationState &result, Value cond,
+ function_ref<void(OpBuilder &, Location)> trueBuilder,
+ function_ref<void(OpBuilder &, Location)> falseBuilder) {
+ result.addOperands(cond);
+ OpBuilder::InsertionGuard guard(builder);
+ Region *trueRegion = result.addRegion();
+ builder.createBlock(trueRegion);
+ trueBuilder(builder, result.location);
+ Region *falseRegion = result.addRegion();
+ builder.createBlock(falseRegion);
+ falseBuilder(builder, result.location);
+
+ // Get result type from whichever branch has a yield (the other may have
+ // unreachable from a throw expression)
+ cir::YieldOp yield;
+ if (trueRegion->back().mightHaveTerminator())
+ yield = dyn_cast_or_null<cir::YieldOp>(trueRegion->back().getTerminator());
+ if (!yield && falseRegion->back().mightHaveTerminator())
+ yield = dyn_cast_or_null<cir::YieldOp>(falseRegion->back().getTerminator());
+
+ assert((!yield || yield.getNumOperands() <= 1) &&
+ "expected zero or one result type");
+ if (yield && yield.getNumOperands() == 1)
+ result.addTypes(TypeRange{yield.getOperandTypes().front()});
+}
+
+//===----------------------------------------------------------------------===//
+// SelectOp
+//===----------------------------------------------------------------------===//
+
+OpFoldResult cir::SelectOp::fold(FoldAdaptor adaptor) {
+ if (auto condition =
+ mlir::dyn_cast_if_present<cir::BoolAttr>(adaptor.getCondition()))
+ return condition.getValue() ? getTrueValue() : getFalseValue();
+
+ // cir.select if %0 then x else x -> x
+ mlir::Attribute trueValue = adaptor.getTrueValue();
+ mlir::Attribute falseValue = adaptor.getFalseValue();
+ if (trueValue == falseValue)
+ return trueValue;
+ if (getTrueValue() == getFalseValue())
+ return getTrueValue();
+
+ return {};
+}
+
+LogicalResult cir::SelectOp::verify() {
+ // AllTypesMatch already guarantees trueVal and falseVal have matching types.
+ auto condTy = dyn_cast<cir::VectorType>(getCondition().getType());
+
+ // If condition is not a vector, no further checks are needed.
+ if (!condTy)
+ return success();
+
+ // When condition is a vector, both other operands must also be vectors.
+ if (!isa<cir::VectorType>(getTrueValue().getType()) ||
+ !isa<cir::VectorType>(getFalseValue().getType())) {
+ return emitOpError()
+ << "expected both true and false operands to be vector types "
+ "when the condition is a vector boolean type";
+ }
+
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// LabelOp Definitions
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::LabelOp::verify() {
+ mlir::Operation *op = getOperation();
+ mlir::Block *blk = op->getBlock();
+ if (&blk->front() != op)
+ return emitError() << "must be the first operation in a block";
+
+ return mlir::success();
+}
+
+//===----------------------------------------------------------------------===//
+// TableGen-generated op definitions
+//===----------------------------------------------------------------------===//
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsControlFlow.cpp.inc"
diff --git a/clang/lib/CIR/Dialect/IR/CIROpsCore.cpp b/clang/lib/CIR/Dialect/IR/CIROpsCore.cpp
new file mode 100644
index 00000000000000..249a7d831b9ece
--- /dev/null
+++ b/clang/lib/CIR/Dialect/IR/CIROpsCore.cpp
@@ -0,0 +1,660 @@
+//===- CIROpsCore.cpp - CIR ops in partition `Core` -----------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the CIR ops in partition `Core`: the hand-written
+// parts first, then the TableGen-generated definitions.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/CIR/Dialect/IR/CIROpsCore.h"
+
+#include "CIROpDefinitions.h"
+
+#include "mlir/IR/DialectImplementation.h"
+
+using namespace mlir;
+using namespace cir;
+
+//===----------------------------------------------------------------------===//
+// AllocaOp
+//===----------------------------------------------------------------------===//
+
+void cir::AllocaOp::build(mlir::OpBuilder &odsBuilder,
+ mlir::OperationState &odsState, mlir::Type addr,
+ llvm::StringRef name, mlir::IntegerAttr alignment) {
+ odsState.addAttribute(getNameAttrName(odsState.name),
+ odsBuilder.getStringAttr(name));
+ if (alignment) {
+ odsState.addAttribute(getAlignmentAttrName(odsState.name), alignment);
+ }
+ odsState.addTypes(addr);
+}
+
+cir::AllocaOp cir::getUnderlyingAlloca(mlir::Value addr) {
+ mlir::Value ptr = addr;
+ while (cir::CastOp castOp = ptr.getDefiningOp<cir::CastOp>()) {
+ if (!castOp.isAllocaPreservingCast())
+ break;
+ ptr = castOp.getSrc();
+ }
+ return ptr.getDefiningOp<cir::AllocaOp>();
+}
+
+//===----------------------------------------------------------------------===//
+// DeleteArrayOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::DeleteArrayOp::verify() {
+ if (getDtorMayThrow() && !getElementDtorAttr())
+ return emitOpError(
+ "'dtor_may_throw' requires an 'element_dtor' to be present");
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// ConstantOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::checkConstantTypes(mlir::Operation *op, mlir::Type opType,
+ mlir::Attribute attrType) {
+ if (isa<cir::ConstPtrAttr>(attrType)) {
+ if (!mlir::isa<cir::PointerType>(opType))
+ return op->emitOpError(
+ "pointer constant initializing a non-pointer type");
+ return success();
+ }
+
+ if (isa<cir::DataMemberAttr, cir::DataMemberOffsetAttr, cir::MethodAttr>(
+ attrType)) {
+ // More detailed type verifications are already done in
+ // DataMemberAttr::verify or MethodAttr::verify. Don't need to repeat here.
+ return success();
+ }
+
+ if (isa<cir::ZeroAttr>(attrType)) {
+ if (isa<cir::RecordType, cir::ArrayType, cir::MatrixType, cir::VectorType,
+ cir::ComplexType>(opType))
+ return success();
+ return op->emitOpError(
+ "zero expects struct, array, vector, or complex type");
+ }
+
+ if (mlir::isa<cir::UndefAttr>(attrType)) {
+ if (!mlir::isa<cir::VoidType>(opType))
+ return success();
+ return op->emitOpError("undef expects non-void type");
+ }
+
+ if (mlir::isa<cir::BoolAttr>(attrType)) {
+ if (!mlir::isa<cir::BoolType>(opType))
+ return op->emitOpError("result type (")
+ << opType << ") must be '!cir.bool' for '" << attrType << "'";
+ return success();
+ }
+
+ if (mlir::isa<cir::IntAttr, cir::FPAttr>(attrType)) {
+ auto at = cast<TypedAttr>(attrType);
+ if (at.getType() != opType) {
+ return op->emitOpError("result type (")
+ << opType << ") does not match value type (" << at.getType()
+ << ")";
+ }
+ return success();
+ }
+
+ if (mlir::isa<cir::BlockAddrDiffAttr, cir::BlockAddrInfoAttr,
+ cir::ConstArrayAttr, cir::ConstVectorAttr,
+ cir::ConstComplexAttr, cir::ConstRecordAttr,
+ cir::GlobalOffsetAttr, cir::GlobalViewAttr, cir::PoisonAttr,
+ cir::TypeInfoAttr, cir::VTableAttr>(attrType))
+ return success();
+
+ assert(isa<TypedAttr>(attrType) && "What else could we be looking at here?");
+ return op->emitOpError("global with type ")
+ << cast<TypedAttr>(attrType).getType() << " not yet supported";
+}
+
+LogicalResult cir::ConstantOp::verify() {
+ // ODS already generates checks to make sure the result type is valid. We just
+ // need to additionally check that the value's attribute type is consistent
+ // with the result type.
+ return checkConstantTypes(getOperation(), getType(), getValue());
+}
+
+OpFoldResult cir::ConstantOp::fold(FoldAdaptor /*adaptor*/) {
+ return getValue();
+}
+
+//===----------------------------------------------------------------------===//
+// CastOp
+//===----------------------------------------------------------------------===//
+
+static bool isFloatingPointCastKind(cir::CastKind kind) {
+ switch (kind) {
+ case cir::CastKind::floating:
+ case cir::CastKind::int_to_float:
+ case cir::CastKind::float_to_int:
+ case cir::CastKind::float_to_bool:
+ case cir::CastKind::bool_to_float:
+ case cir::CastKind::float_to_complex:
+ case cir::CastKind::float_complex_to_real:
+ case cir::CastKind::float_complex_to_bool:
+ case cir::CastKind::float_complex:
+ case cir::CastKind::float_complex_to_int_complex:
+ case cir::CastKind::int_complex_to_float_complex:
+ return true;
+ default:
+ return false;
+ }
+}
+
+LogicalResult cir::CastOp::verify() {
+ mlir::Type resType = getType();
+ mlir::Type srcType = getSrc().getType();
+ cir::CastKind kind = getKind();
+
+ if (getFenvAttr() && !isFloatingPointCastKind(kind))
+ return emitOpError()
+ << "'fenv' is only valid for floating-point cast kinds";
+
+ // Verify address space casts for pointer types. given that
+ // casts for within a different address space are illegal.
+ auto srcPtrTy = mlir::dyn_cast<cir::PointerType>(srcType);
+ auto resPtrTy = mlir::dyn_cast<cir::PointerType>(resType);
+ if (srcPtrTy && resPtrTy && (kind != cir::CastKind::address_space))
+ if (srcPtrTy.getAddrSpace() != resPtrTy.getAddrSpace()) {
+ return emitOpError() << "result type address space does not match the "
+ "address space of the operand";
+ }
+
+ auto srcVTy = mlir::dyn_cast<cir::VectorType>(srcType);
+ auto resVTy = mlir::dyn_cast<cir::VectorType>(resType);
+ if (srcVTy && resVTy) {
+ if ((kind == cir::CastKind::int_to_float ||
+ kind == cir::CastKind::float_to_int) &&
+ srcVTy.getSize() != resVTy.getSize()) {
+ return emitOpError()
+ << "vector float-to-int and int-to-float casts require "
+ "source and destination vectors to have the same number of "
+ "elements";
+ }
+ // Use the element type of the vector to verify the cast kind. (Except for
+ // bitcast, see below.)
+ srcType = srcVTy.getElementType();
+ resType = resVTy.getElementType();
+ }
+
+ switch (kind) {
+ case cir::CastKind::int_to_bool: {
+ if (!mlir::isa<cir::BoolType>(resType))
+ return emitOpError() << "requires !cir.bool type for result";
+ if (!mlir::isa<cir::IntType>(srcType))
+ return emitOpError() << "requires !cir.int type for source";
+ return success();
+ }
+ case cir::CastKind::ptr_to_bool: {
+ if (!mlir::isa<cir::BoolType>(resType))
+ return emitOpError() << "requires !cir.bool type for result";
+ if (!mlir::isa<cir::PointerType>(srcType))
+ return emitOpError() << "requires !cir.ptr type for source";
+ return success();
+ }
+ case cir::CastKind::integral: {
+ if (!mlir::isa<cir::IntType>(resType))
+ return emitOpError() << "requires !cir.int type for result";
+ if (!mlir::isa<cir::IntType>(srcType))
+ return emitOpError() << "requires !cir.int type for source";
+ return success();
+ }
+ case cir::CastKind::array_to_ptrdecay: {
+ const auto arrayPtrTy = mlir::dyn_cast<cir::PointerType>(srcType);
+ const auto flatPtrTy = mlir::dyn_cast<cir::PointerType>(resType);
+ if (!arrayPtrTy || !flatPtrTy)
+ return emitOpError() << "requires !cir.ptr type for source and result";
+
+ // TODO(CIR): Make sure the AddrSpace of both types are equals
+ return success();
+ }
+ case cir::CastKind::bitcast: {
+ // Handle the pointer types first.
+ auto srcPtrTy = mlir::dyn_cast<cir::PointerType>(srcType);
+ auto resPtrTy = mlir::dyn_cast<cir::PointerType>(resType);
+
+ if (srcPtrTy && resPtrTy) {
+ return success();
+ }
+
+ return success();
+ }
+ case cir::CastKind::floating: {
+ if (!mlir::isa<cir::FPTypeInterface>(srcType) ||
+ !mlir::isa<cir::FPTypeInterface>(resType))
+ return emitOpError() << "requires !cir.float type for source and result";
+ return success();
+ }
+ case cir::CastKind::float_to_int: {
+ if (!mlir::isa<cir::FPTypeInterface>(srcType))
+ return emitOpError() << "requires !cir.float type for source";
+ if (!mlir::dyn_cast<cir::IntType>(resType))
+ return emitOpError() << "requires !cir.int type for result";
+ return success();
+ }
+ case cir::CastKind::int_to_ptr: {
+ if (!mlir::dyn_cast<cir::IntType>(srcType))
+ return emitOpError() << "requires !cir.int type for source";
+ if (!mlir::dyn_cast<cir::PointerType>(resType))
+ return emitOpError() << "requires !cir.ptr type for result";
+ return success();
+ }
+ case cir::CastKind::ptr_to_int: {
+ if (!mlir::dyn_cast<cir::PointerType>(srcType))
+ return emitOpError() << "requires !cir.ptr type for source";
+ if (!mlir::dyn_cast<cir::IntType>(resType))
+ return emitOpError() << "requires !cir.int type for result";
+ return success();
+ }
+ case cir::CastKind::float_to_bool: {
+ if (!mlir::isa<cir::FPTypeInterface>(srcType))
+ return emitOpError() << "requires !cir.float type for source";
+ if (!mlir::isa<cir::BoolType>(resType))
+ return emitOpError() << "requires !cir.bool type for result";
+ return success();
+ }
+ case cir::CastKind::bool_to_int: {
+ if (!mlir::isa<cir::BoolType>(srcType))
+ return emitOpError() << "requires !cir.bool type for source";
+ if (!mlir::isa<cir::IntType>(resType))
+ return emitOpError() << "requires !cir.int type for result";
+ return success();
+ }
+ case cir::CastKind::int_to_float: {
+ if (!mlir::isa<cir::IntType>(srcType))
+ return emitOpError() << "requires !cir.int type for source";
+ if (!mlir::isa<cir::FPTypeInterface>(resType))
+ return emitOpError() << "requires !cir.float type for result";
+ return success();
+ }
+ case cir::CastKind::bool_to_float: {
+ if (!mlir::isa<cir::BoolType>(srcType))
+ return emitOpError() << "requires !cir.bool type for source";
+ if (!mlir::isa<cir::FPTypeInterface>(resType))
+ return emitOpError() << "requires !cir.float type for result";
+ return success();
+ }
+ case cir::CastKind::address_space: {
+ auto srcPtrTy = mlir::dyn_cast<cir::PointerType>(srcType);
+ auto resPtrTy = mlir::dyn_cast<cir::PointerType>(resType);
+ if (!srcPtrTy || !resPtrTy)
+ return emitOpError() << "requires !cir.ptr type for source and result";
+ if (srcPtrTy.getPointee() != resPtrTy.getPointee())
+ return emitOpError() << "requires two types differ in addrspace only";
+ return success();
+ }
+ case cir::CastKind::float_to_complex: {
+ if (!mlir::isa<cir::FPTypeInterface>(srcType))
+ return emitOpError() << "requires !cir.float type for source";
+ auto resComplexTy = mlir::dyn_cast<cir::ComplexType>(resType);
+ if (!resComplexTy)
+ return emitOpError() << "requires !cir.complex type for result";
+ if (srcType != resComplexTy.getElementType())
+ return emitOpError() << "requires source type match result element type";
+ return success();
+ }
+ case cir::CastKind::int_to_complex: {
+ if (!mlir::isa<cir::IntType>(srcType))
+ return emitOpError() << "requires !cir.int type for source";
+ auto resComplexTy = mlir::dyn_cast<cir::ComplexType>(resType);
+ if (!resComplexTy)
+ return emitOpError() << "requires !cir.complex type for result";
+ if (srcType != resComplexTy.getElementType())
+ return emitOpError() << "requires source type match result element type";
+ return success();
+ }
+ case cir::CastKind::float_complex_to_real: {
+ auto srcComplexTy = mlir::dyn_cast<cir::ComplexType>(srcType);
+ if (!srcComplexTy)
+ return emitOpError() << "requires !cir.complex type for source";
+ if (!mlir::isa<cir::FPTypeInterface>(resType))
+ return emitOpError() << "requires !cir.float type for result";
+ if (srcComplexTy.getElementType() != resType)
+ return emitOpError() << "requires source element type match result type";
+ return success();
+ }
+ case cir::CastKind::int_complex_to_real: {
+ auto srcComplexTy = mlir::dyn_cast<cir::ComplexType>(srcType);
+ if (!srcComplexTy)
+ return emitOpError() << "requires !cir.complex type for source";
+ if (!mlir::isa<cir::IntType>(resType))
+ return emitOpError() << "requires !cir.int type for result";
+ if (srcComplexTy.getElementType() != resType)
+ return emitOpError() << "requires source element type match result type";
+ return success();
+ }
+ case cir::CastKind::float_complex_to_bool: {
+ auto srcComplexTy = mlir::dyn_cast<cir::ComplexType>(srcType);
+ if (!srcComplexTy || !srcComplexTy.isFloatingPointComplex())
+ return emitOpError()
+ << "requires floating point !cir.complex type for source";
+ if (!mlir::isa<cir::BoolType>(resType))
+ return emitOpError() << "requires !cir.bool type for result";
+ return success();
+ }
+ case cir::CastKind::int_complex_to_bool: {
+ auto srcComplexTy = mlir::dyn_cast<cir::ComplexType>(srcType);
+ if (!srcComplexTy || !srcComplexTy.isIntegerComplex())
+ return emitOpError()
+ << "requires floating point !cir.complex type for source";
+ if (!mlir::isa<cir::BoolType>(resType))
+ return emitOpError() << "requires !cir.bool type for result";
+ return success();
+ }
+ case cir::CastKind::float_complex: {
+ auto srcComplexTy = mlir::dyn_cast<cir::ComplexType>(srcType);
+ if (!srcComplexTy || !srcComplexTy.isFloatingPointComplex())
+ return emitOpError()
+ << "requires floating point !cir.complex type for source";
+ auto resComplexTy = mlir::dyn_cast<cir::ComplexType>(resType);
+ if (!resComplexTy || !resComplexTy.isFloatingPointComplex())
+ return emitOpError()
+ << "requires floating point !cir.complex type for result";
+ return success();
+ }
+ case cir::CastKind::float_complex_to_int_complex: {
+ auto srcComplexTy = mlir::dyn_cast<cir::ComplexType>(srcType);
+ if (!srcComplexTy || !srcComplexTy.isFloatingPointComplex())
+ return emitOpError()
+ << "requires floating point !cir.complex type for source";
+ auto resComplexTy = mlir::dyn_cast<cir::ComplexType>(resType);
+ if (!resComplexTy || !resComplexTy.isIntegerComplex())
+ return emitOpError() << "requires integer !cir.complex type for result";
+ return success();
+ }
+ case cir::CastKind::int_complex: {
+ auto srcComplexTy = mlir::dyn_cast<cir::ComplexType>(srcType);
+ if (!srcComplexTy || !srcComplexTy.isIntegerComplex())
+ return emitOpError() << "requires integer !cir.complex type for source";
+ auto resComplexTy = mlir::dyn_cast<cir::ComplexType>(resType);
+ if (!resComplexTy || !resComplexTy.isIntegerComplex())
+ return emitOpError() << "requires integer !cir.complex type for result";
+ return success();
+ }
+ case cir::CastKind::int_complex_to_float_complex: {
+ auto srcComplexTy = mlir::dyn_cast<cir::ComplexType>(srcType);
+ if (!srcComplexTy || !srcComplexTy.isIntegerComplex())
+ return emitOpError() << "requires integer !cir.complex type for source";
+ auto resComplexTy = mlir::dyn_cast<cir::ComplexType>(resType);
+ if (!resComplexTy || !resComplexTy.isFloatingPointComplex())
+ return emitOpError()
+ << "requires floating point !cir.complex type for result";
+ return success();
+ }
+ case cir::CastKind::member_ptr_to_bool: {
+ if (!mlir::isa<cir::DataMemberType, cir::MethodType>(srcType))
+ return emitOpError()
+ << "requires !cir.data_member or !cir.method type for source";
+ if (!mlir::isa<cir::BoolType>(resType))
+ return emitOpError() << "requires !cir.bool type for result";
+ return success();
+ }
+ }
+ llvm_unreachable("Unknown CastOp kind?");
+}
+
+static bool isIntOrBoolCast(cir::CastOp op) {
+ auto kind = op.getKind();
+ return kind == cir::CastKind::bool_to_int ||
+ kind == cir::CastKind::int_to_bool || kind == cir::CastKind::integral;
+}
+
+static bool isCirFunctionPointerType(mlir::Type ty) {
+ const auto ptrTy = mlir::dyn_cast<cir::PointerType>(ty);
+ return ptrTy && mlir::isa<cir::FuncType>(ptrTy.getPointee());
+}
+
+static Value tryFoldCastChain(cir::CastOp op) {
+ cir::CastOp head = op, tail = op;
+
+ while (op) {
+ if (!isIntOrBoolCast(op))
+ break;
+ head = op;
+ op = head.getSrc().getDefiningOp<cir::CastOp>();
+ }
+
+ if (head != tail) {
+ // if bool_to_int -> ... -> int_to_bool: take the bool
+ // as we had it was before all casts
+ if (head.getKind() == cir::CastKind::bool_to_int &&
+ tail.getKind() == cir::CastKind::int_to_bool)
+ return head.getSrc();
+
+ // if int_to_bool -> ... -> int_to_bool: take the result
+ // of the first one, as no other casts (and ext casts as well)
+ // don't change the first result
+ if (head.getKind() == cir::CastKind::int_to_bool &&
+ tail.getKind() == cir::CastKind::int_to_bool)
+ return head.getResult();
+
+ return {};
+ }
+
+ // Bitcast round-trip on function pointers: T0 -> T1 -> T0 (e.g. no-proto
+ // redeclaration vs. actual prototype). Restrict to function pointers so
+ // other pointer bitcast chains are unchanged.
+ if (tail.getKind() == cir::CastKind::bitcast) {
+ auto *inner = tail.getSrc().getDefiningOp();
+ if (inner && isCirFunctionPointerType(tail.getType())) {
+ auto innerCast = mlir::dyn_cast<cir::CastOp>(inner);
+ if (innerCast && innerCast.getKind() == cir::CastKind::bitcast &&
+ innerCast.getSrc().getType() == tail.getType() &&
+ innerCast.getType() == tail.getSrc().getType()) {
+ return innerCast.getSrc();
+ }
+ }
+ }
+
+ return {};
+}
+
+OpFoldResult cir::CastOp::fold(FoldAdaptor adaptor) {
+ if (mlir::isa_and_present<cir::PoisonAttr>(adaptor.getSrc())) {
+ // Propagate poison value
+ return cir::PoisonAttr::get(getContext(), getType());
+ }
+
+ // Propagate Undef value
+ if (mlir::isa_and_present<cir::UndefAttr>(adaptor.getSrc()))
+ return cir::UndefAttr::get(getType());
+
+ if (getSrc().getType() == getType()) {
+ switch (getKind()) {
+ case cir::CastKind::integral:
+ case cir::CastKind::bitcast:
+ case cir::CastKind::address_space:
+ case cir::CastKind::float_complex:
+ case cir::CastKind::int_complex: {
+ return getSrc();
+ }
+ default:
+ return {};
+ }
+ }
+
+ // Handle cases where a chain of casts cancel out.
+ Value result = tryFoldCastChain(*this);
+ if (result)
+ return result;
+
+ // Handle simple constant casts.
+ if (auto srcConst = getSrc().getDefiningOp<cir::ConstantOp>()) {
+ switch (getKind()) {
+ case cir::CastKind::integral: {
+ mlir::Type srcTy = getSrc().getType();
+ // Don't try to fold vector casts for now.
+ assert(mlir::isa<cir::VectorType>(srcTy) ==
+ mlir::isa<cir::VectorType>(getType()));
+ if (mlir::isa<cir::VectorType>(srcTy))
+ break;
+
+ auto srcIntTy = mlir::cast<cir::IntType>(srcTy);
+ auto dstIntTy = mlir::cast<cir::IntType>(getType());
+ auto constIntAttr = srcConst.getValueAttr<cir::IntAttr>();
+ if (!constIntAttr)
+ return {};
+
+ APInt srcValue = constIntAttr.getValue();
+ APInt newVal = srcIntTy.isSigned()
+ ? srcValue.sextOrTrunc(dstIntTy.getWidth())
+ : srcValue.zextOrTrunc(dstIntTy.getWidth());
+ return cir::IntAttr::get(dstIntTy, newVal);
+ }
+ default:
+ break;
+ }
+ }
+ return {};
+}
+
+//===----------------------------------------------------------------------===//
+// BuiltinIntCastOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::BuiltinIntCastOp::verify() {
+ mlir::Type srcType = getSrc().getType();
+ mlir::Type resType = getType();
+
+ auto srcCirInt = mlir::dyn_cast<cir::IntType>(srcType);
+ auto resCirInt = mlir::dyn_cast<cir::IntType>(resType);
+
+ // One side must be a CIR integer the other must be a builtin
+ // integer or index type.
+ if (static_cast<bool>(srcCirInt) == static_cast<bool>(resCirInt))
+ return emitOpError()
+ << "requires exactly one '!cir.int' operand or result; the other "
+ "must be a builtin integer or 'index' type";
+
+ mlir::Type builtinType = srcCirInt ? resType : srcType;
+ if (!mlir::isa<mlir::IntegerType, mlir::IndexType>(builtinType))
+ return emitOpError() << "requires a builtin integer or 'index' type on the "
+ "non-CIR side";
+
+ // The cast preserves bit width. 'index' has no fixed width, so only check
+ // when the builtin side is a fixed-width integer.
+ if (auto builtinInt = mlir::dyn_cast<mlir::IntegerType>(builtinType)) {
+ cir::IntType cirInt = srcCirInt ? srcCirInt : resCirInt;
+ if (cirInt.getWidth() != builtinInt.getWidth())
+ return emitOpError()
+ << "requires the CIR and builtin integer types to have the same "
+ "width; use 'cir.cast' for width conversions";
+ }
+
+ return success();
+}
+
+OpFoldResult cir::BuiltinIntCastOp::fold(FoldAdaptor adaptor) {
+ // Fold: builtin_int_cast(builtin_int_cast(x)) -> x
+ // Inner source type must match the cast's result type.
+ if (auto inner = getSrc().getDefiningOp<cir::BuiltinIntCastOp>())
+ if (inner.getSrc().getType() == getType())
+ return inner.getSrc();
+ return {};
+}
+
+//===----------------------------------------------------------------------===//
+// PtrMaskOp Definitions
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::PtrMaskOp::verify() {
+ mlir::DataLayout layout = mlir::DataLayout::closest(*this);
+ std::optional<uint64_t> indexWidth =
+ layout.getTypeIndexBitwidth(getPtr().getType());
+ if (!indexWidth)
+ return emitOpError() << "pointer has no index width";
+
+ uint64_t maskWidth = getMask().getType().getWidth();
+ if (maskWidth != *indexWidth)
+ return emitOpError() << "mask width " << maskWidth
+ << " must equal the pointer index width "
+ << *indexWidth;
+
+ return mlir::success();
+}
+
+//===----------------------------------------------------------------------===//
+// GetMemberOp Definitions
+//===----------------------------------------------------------------------===//
+
+static mlir::Type memberPointeeType(cir::RecordType recordTy, unsigned idx) {
+ return cir::memberStorageType(recordTy.getMembers()[idx]);
+}
+
+LogicalResult cir::GetMemberOp::verify() {
+ const auto recordTy = dyn_cast<RecordType>(getAddrTy().getPointee());
+ if (!recordTy)
+ return emitError() << "expected pointer to a record type";
+
+ if (recordTy.getMembers().size() <= getIndex())
+ return emitError() << "member index out of bounds";
+
+ mlir::Type pointeeTy = memberPointeeType(recordTy, getIndex());
+ if (!pointeeTy)
+ return emitError() << "member owns no storage to point at";
+
+ if (pointeeTy != getType().getPointee())
+ return emitError() << "member type mismatch";
+
+ if (getAddrTy().getAddrSpace() != getType().getAddrSpace())
+ return emitError() << "address space mismatch";
+
+ return mlir::success();
+}
+
+//===----------------------------------------------------------------------===//
+// ExtractMemberOp Definitions
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::ExtractMemberOp::verify() {
+ if (mlir::isa<cir::UnionType>(getRecord().getType()))
+ return emitError()
+ << "cir.extract_member currently does not support unions";
+ auto structTy = mlir::cast<cir::StructType>(getRecord().getType());
+ if (structTy.getMembers().size() <= getIndex())
+ return emitError() << "member index out of bounds";
+ mlir::Type memberTy = structTy.getMembers()[getIndex()];
+ if (mlir::isa<cir::BitFieldType>(memberTy))
+ return emitError() << "cir.extract_member does not support bit-fields";
+ if (memberTy != getType())
+ return emitError() << "member type mismatch";
+ return mlir::success();
+}
+
+//===----------------------------------------------------------------------===//
+// InsertMemberOp Definitions
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::InsertMemberOp::verify() {
+ if (mlir::isa<cir::UnionType>(getRecord().getType()))
+ return emitError() << "cir.insert_member currently does not support unions";
+ auto structTy = mlir::cast<cir::StructType>(getRecord().getType());
+ if (structTy.getMembers().size() <= getIndex())
+ return emitError() << "member index out of bounds";
+ mlir::Type memberTy = structTy.getMembers()[getIndex()];
+ if (mlir::isa<cir::BitFieldType>(memberTy))
+ return emitError() << "cir.insert_member does not support bit-fields";
+ if (memberTy != getValue().getType())
+ return emitError() << "member type mismatch";
+ // The op trait already checks that the types of $result and $record match.
+ return mlir::success();
+}
+
+//===----------------------------------------------------------------------===//
+// TableGen-generated op definitions
+//===----------------------------------------------------------------------===//
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsCore.cpp.inc"
diff --git a/clang/lib/CIR/Dialect/IR/CIROpsCoroutine.cpp b/clang/lib/CIR/Dialect/IR/CIROpsCoroutine.cpp
new file mode 100644
index 00000000000000..aa152549ee8a23
--- /dev/null
+++ b/clang/lib/CIR/Dialect/IR/CIROpsCoroutine.cpp
@@ -0,0 +1,340 @@
+//===- CIROpsCoroutine.cpp - CIR ops in partition `Coroutine` -------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the CIR ops in partition `Coroutine`: the hand-written
+// parts first, then the TableGen-generated definitions.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/CIR/Dialect/IR/CIROpsCoroutine.h"
+#include "clang/CIR/Dialect/IR/CIROpsControlFlow.h"
+
+#include "CIROpDefinitions.h"
+
+#include "mlir/IR/DialectImplementation.h"
+
+using namespace mlir;
+using namespace cir;
+
+//===----------------------------------------------------------------------===//
+// CoroSuspendPoint
+//===----------------------------------------------------------------------===//
+
+MutableOperandRange
+cir::CoroSuspendPoint::getMutableSuccessorOperands(RegionSuccessor point) {
+ // The condition operand is not forwarded to successor regions.
+ return MutableOperandRange(getOperation(), 0, 0);
+}
+
+//===----------------------------------------------------------------------===//
+// AwaitOp
+//===----------------------------------------------------------------------===//
+
+void cir::AwaitOp::build(OpBuilder &builder, OperationState &result,
+ cir::AwaitKind kind, BuilderCallbackRef readyBuilder,
+ BuilderCallbackRef suspendBuilder,
+ BuilderCallbackRef resumeBuilder) {
+ result.addAttribute(getKindAttrName(result.name),
+ cir::AwaitKindAttr::get(builder.getContext(), kind));
+ {
+ OpBuilder::InsertionGuard guard(builder);
+ Region *readyRegion = result.addRegion();
+ builder.createBlock(readyRegion);
+ readyBuilder(builder, result.location);
+ }
+
+ {
+ OpBuilder::InsertionGuard guard(builder);
+ Region *suspendRegion = result.addRegion();
+ builder.createBlock(suspendRegion);
+ suspendBuilder(builder, result.location);
+ }
+
+ {
+ OpBuilder::InsertionGuard guard(builder);
+ Region *resumeRegion = result.addRegion();
+ builder.createBlock(resumeRegion);
+ resumeBuilder(builder, result.location);
+ }
+}
+
+void cir::AwaitOp::getSuccessorRegions(
+ mlir::RegionBranchPoint point, SmallVectorImpl<RegionSuccessor> ®ions) {
+ assert(point.isParent() || point.getTerminatorPredecessorOrNull());
+
+ // Execution always starts in the ready region.
+ if (point.isParent()) {
+ regions.emplace_back(&getReady());
+ return;
+ }
+
+ mlir::Region *parentRegion =
+ point.getTerminatorPredecessorOrNull()->getParentRegion();
+
+ // Branching from ready: the cir.condition terminating it selects between
+ // suspending and resuming. Keep in sync with
+ // ConditionOp::getSuccessorRegions.
+ //
+ // TODO: retrieve information from the promise and only push the
+ // necessary ones. Example: `std::suspend_never` on initial or final
+ // await's might allow suspend region to be skipped.
+ if (&getReady() == parentRegion) {
+ regions.emplace_back(&getResume());
+ regions.emplace_back(&getSuspend());
+ return;
+ }
+
+ // Branching from suspend: continue in resume once the coroutine is resumed
+ // (or immediately if suspension is vetoed), or exit to the parent operation
+ // if it stays suspended or is destroyed.
+ if (&getSuspend() == parentRegion) {
+ regions.emplace_back(&getResume());
+ regions.emplace_back(getOperation());
+ return;
+ }
+
+ // Branching from resume: exit to the parent operation.
+ regions.emplace_back(getOperation());
+}
+
+LogicalResult cir::AwaitOp::verify() {
+ if (!isa<ConditionOp>(this->getReady().back().getTerminator()))
+ return emitOpError("ready region must end with cir.condition");
+ if (this->getSuspend().empty())
+ return emitOpError("suspend region must not be empty");
+ if (!isa<CoroSuspendPoint>(this->getSuspend().back().getTerminator()))
+ return emitOpError("suspend region must end with cir.coro.suspend_point");
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// CoReturnOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::CoReturnOp::verify() {
+ mlir::Operation *coRet = getOperation();
+ auto coroutine = coRet->getParentOfType<CoroutineOp>();
+ if (!coroutine.getBody().isAncestor(getOperation()->getParentRegion()))
+ return emitOpError("must be inside the cir.coroutine body region");
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// CoroutineOp
+//===----------------------------------------------------------------------===//
+
+void cir::CoroutineOp::build(OpBuilder &builder, OperationState &result,
+ BuilderCallbackRef initialSuspendBuilder,
+ BuilderCallbackRef bodyBuilder,
+ BuilderCallbackRef finalSuspendBuilder,
+ BuilderCallbackRef destroyBuilder,
+ BuilderCallbackRef exitBuilder) {
+ {
+ OpBuilder::InsertionGuard guard(builder);
+ Region *initialRegion = result.addRegion();
+ builder.createBlock(initialRegion);
+ initialSuspendBuilder(builder, result.location);
+ }
+
+ {
+ OpBuilder::InsertionGuard guard(builder);
+ Region *bodyRegion = result.addRegion();
+ builder.createBlock(bodyRegion);
+ bodyBuilder(builder, result.location);
+ }
+
+ {
+ OpBuilder::InsertionGuard guard(builder);
+ Region *finalRegion = result.addRegion();
+ builder.createBlock(finalRegion);
+ finalSuspendBuilder(builder, result.location);
+ }
+
+ {
+ OpBuilder::InsertionGuard guard(builder);
+ Region *destroyRegion = result.addRegion();
+ builder.createBlock(destroyRegion);
+ destroyBuilder(builder, result.location);
+ }
+
+ {
+ OpBuilder::InsertionGuard guard(builder);
+ Region *exitRegion = result.addRegion();
+ builder.createBlock(exitRegion);
+ exitBuilder(builder, result.location);
+ }
+}
+
+void cir::CoroutineOp::getSuccessorRegions(
+ mlir::RegionBranchPoint point, SmallVectorImpl<RegionSuccessor> ®ions) {
+ // The parent op always enters through initial_suspend get_return_object()
+ // (if present) and the init await always run first.
+ if (point.isParent()) {
+ regions.emplace_back(&getInitialSuspend());
+ return;
+ }
+
+ mlir::Region *parent =
+ point.getTerminatorPredecessorOrNull()->getParentRegion();
+
+ // initial_suspend either falls into body (resumed, or never actually
+ // suspended because await_ready() was true), exits directly, or reaches
+ // destroy when the coroutine is destroyed while suspended at its initial
+ // suspend point.
+ if (parent == &getInitialSuspend()) {
+ regions.emplace_back(&getBody());
+ regions.emplace_back(&getExit());
+ regions.emplace_back(&getDestroy());
+ return;
+ }
+
+ // body can fall through into final_suspend (co_return), exit directly
+ // from any suspend_point inside it that just suspends, or reach destroy,
+ // either from a suspend_point being destroyed, or from an exception
+ // escaping body's own catch-all.
+ if (parent == &getBody()) {
+ regions.emplace_back(&getFinalSuspend());
+ regions.emplace_back(&getExit()); // any suspend_point inside body
+ regions.emplace_back(&getDestroy()); // destroy() on any suspend inside body
+ return;
+ }
+
+ // final_suspend's only live edge in valid programs is destroy. Resuming
+ // past the final suspend is UB. The ready-immediately edge to exit is
+ // kept for structural symmetry with the other two suspend regions even
+ // though it's effectively dead.
+ if (parent == &getFinalSuspend()) {
+ regions.emplace_back(
+ &getDestroy()); // the only real exit from final_suspend
+ regions.emplace_back(&getExit()); // ready==true edge, rarely taken
+ return;
+ }
+
+ // destroy has two possible outcomes depending on why it was entered:
+ // ordinary destroy dispatch falls through to exit (normal return); an
+ // exception that reached destroy needs to keep propagating instead, i.e.
+ // leave the whole op rather than go through exit's cir.return.
+ if (parent == &getDestroy()) {
+ regions.emplace_back(&getExit());
+ regions.push_back(RegionSuccessor(getOperation())); // unwind out of the op
+ return;
+ }
+ // exit always terminates the op there's no region it loops back into.
+ if (parent == &getExit()) {
+ regions.emplace_back(getOperation());
+ return;
+ }
+}
+
+mlir::ValueRange
+cir::CoroutineOp::getSuccessorInputs(RegionSuccessor successor) {
+ return ValueRange();
+}
+
+LogicalResult cir::CoroutineOp::verify() {
+ mlir::Region &initialSuspend = getInitialSuspend();
+ mlir::Region &body = getBody();
+ mlir::Region &finalSuspend = getFinalSuspend();
+ mlir::Region &destroy = getDestroy();
+ mlir::Region &exit = getExit();
+
+ // initial_suspend must contain exactly one 'init'-kind cir.await an
+ // optional prelude of ordinary statements (get_return_object() and
+ // similar) is allowed before it, but no *other* await may appear.
+ int initAwaitCount = 0;
+ bool hasInvalidAwait = false;
+
+ initialSuspend.walk([&](Operation *op) {
+ auto awaitOp = mlir::dyn_cast<AwaitOp>(op);
+ if (!awaitOp)
+ return mlir::WalkResult::advance();
+
+ if (awaitOp.getKind() == cir::AwaitKind::Init)
+ ++initAwaitCount;
+ else
+ hasInvalidAwait = true;
+
+ if (initAwaitCount > 1 || hasInvalidAwait)
+ return mlir::WalkResult::interrupt();
+
+ return mlir::WalkResult::advance();
+ });
+
+ if (hasInvalidAwait)
+ return emitOpError("'initial_suspend' must not contain any cir.await other "
+ "than the 'init' one");
+ if (initAwaitCount != 1)
+ return emitOpError(
+ "must have exactly one 'init' cir.await in 'initial_suspend'");
+
+ // final_suspend must contain only one or zero 'final'-kind cir.await
+ int finalAwaitCount = 0;
+ hasInvalidAwait = false;
+
+ finalSuspend.walk([&](Operation *op) {
+ auto awaitOp = mlir::dyn_cast<AwaitOp>(op);
+ if (!awaitOp)
+ return mlir::WalkResult::advance();
+
+ if (awaitOp.getKind() == cir::AwaitKind::Final)
+ ++finalAwaitCount;
+ else
+ hasInvalidAwait = true;
+
+ if (finalAwaitCount > 1 || hasInvalidAwait)
+ return mlir::WalkResult::interrupt();
+
+ return mlir::WalkResult::advance();
+ });
+
+ if (hasInvalidAwait)
+ return emitOpError("'final_suspend' must not contain any cir.await other "
+ "than the 'final' one");
+ if (finalAwaitCount > 1)
+ return emitOpError(
+ "must have one or zero 'final' cir.await in 'final_suspend'");
+
+ // Each region must end with the terminator its role requires:
+ // initial_suspend/final_suspend/destroy yield back into cir.coroutine's
+ // own control flow, body ends in a cir.co_return (or a plain yield if
+ // some path never reaches one), and exit actually returns from the
+ // function.
+ if (initialSuspend.empty() || initialSuspend.back().empty())
+ return emitOpError("initial_suspend region must not be empty");
+ if (!isa<YieldOp>(initialSuspend.back().back()))
+ return emitOpError("'initial_suspend' must end with cir.yield");
+
+ if (body.empty() || body.back().empty())
+ return emitOpError("'body' region must not be empty");
+ if (!isa<YieldOp, CoReturnOp>(body.back().back()))
+ return emitOpError("'body' must end with cir.yield or cir.co_return");
+
+ if (finalSuspend.empty() || finalSuspend.back().empty())
+ return emitOpError("'final_suspend' region must not be empty");
+ if (!isa<YieldOp>(finalSuspend.back().back()))
+ return emitOpError("'final_suspend' must end with cir.yield");
+
+ if (destroy.empty() || destroy.back().empty())
+ return emitOpError("'destroy' region must not be empty");
+ if (!isa<YieldOp>(destroy.back().back()))
+ return emitOpError("'destroy' must end with cir.yield");
+
+ if (exit.empty() || exit.back().empty())
+ return emitOpError("'exit' region must not be empty");
+ if (!isa<ReturnOp>(exit.back().back()))
+ return emitOpError("'exit' must end with cir.return");
+
+ return mlir::success();
+}
+
+//===----------------------------------------------------------------------===//
+// TableGen-generated op definitions
+//===----------------------------------------------------------------------===//
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsCoroutine.cpp.inc"
diff --git a/clang/lib/CIR/Dialect/IR/CIROpsEH.cpp b/clang/lib/CIR/Dialect/IR/CIROpsEH.cpp
new file mode 100644
index 00000000000000..a398ab600d5885
--- /dev/null
+++ b/clang/lib/CIR/Dialect/IR/CIROpsEH.cpp
@@ -0,0 +1,706 @@
+//===- CIROpsEH.cpp - CIR ops in partition `EH` ---------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the CIR ops in partition `EH`: the hand-written
+// parts first, then the TableGen-generated definitions.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/CIR/Dialect/IR/CIROpsEH.h"
+#include "clang/CIR/Dialect/IR/CIROpsBuiltins.h"
+#include "clang/CIR/Dialect/IR/CIROpsControlFlow.h"
+#include "clang/CIR/Dialect/IR/CIROpsSymbols.h"
+
+#include "CIROpDefinitions.h"
+
+#include "mlir/IR/DialectImplementation.h"
+
+using namespace mlir;
+using namespace cir;
+
+//===----------------------------------------------------------------------===//
+// ThrowOp / TryThrowOp
+//===----------------------------------------------------------------------===//
+
+template <typename ThrowOpTy>
+static mlir::LogicalResult verifyThrowOpImpl(ThrowOpTy op) {
+ if (op.rethrows())
+ return mlir::success();
+
+ if (op.getNumOperands() != 0) {
+ if (op.getTypeInfo())
+ return mlir::success();
+ return op.emitOpError() << "'type_info' symbol attribute missing";
+ }
+
+ return mlir::failure();
+}
+
+mlir::LogicalResult cir::ThrowOp::verify() { return verifyThrowOpImpl(*this); }
+
+mlir::LogicalResult cir::TryThrowOp::verify() {
+ return verifyThrowOpImpl(*this);
+}
+
+//===----------------------------------------------------------------------===//
+// TryOp
+//===----------------------------------------------------------------------===//
+
+void cir::TryOp::getSuccessorRegions(
+ mlir::RegionBranchPoint point,
+ llvm::SmallVectorImpl<mlir::RegionSuccessor> ®ions) {
+ if (point.isParent()) {
+ regions.emplace_back(&getTryRegion());
+ } else {
+ // A handler exit leaves the operation.
+ mlir::Operation *term = point.getTerminatorPredecessorOrNull();
+ if (term->getParentRegion() != &getTryRegion()) {
+ regions.emplace_back(getOperation());
+ return;
+ }
+ // A cir.resume ending the try region unwinds into a handler, so it leaves
+ // the operation only when there is none. Every other exit can leave it.
+ if (!isa<cir::ResumeOp>(term) || getHandlerRegions().empty())
+ regions.emplace_back(getOperation());
+ }
+
+ // Most handlers are entered by an unwind out of a call in the try region,
+ // which is not a branch point, so the handlers are successors of the parent
+ // and of every exit from the try region to stand in for it.
+ //
+ // TODO(CIR): If we know a target function never throws a specific type, we
+ // can remove the catch handler.
+ regions.reserve(regions.size() + getHandlerRegions().size());
+ for (mlir::Region &handlerRegion : getHandlerRegions())
+ regions.emplace_back(&handlerRegion);
+}
+
+/// Verify that each cir.init_catch_param in \p handlerRegion, outside any
+/// nested cir.try, agrees with the handler's leading cir.construct_catch_param
+/// \p constructOp, which may be null. A cir.init_catch_param of kind
+/// reference_to_pointer or non_trivial_copy needs \p constructOp on its
+/// parameter, and a cir.init_catch_param on the parameter of \p constructOp
+/// must have the kind of \p constructOp.
+static LogicalResult
+verifyCatchParamPairing(mlir::Region &handlerRegion,
+ cir::ConstructCatchParamOp constructOp) {
+ mlir::WalkResult result =
+ handlerRegion.walk<mlir::WalkOrder::PreOrder>([&](mlir::Operation *op) {
+ if (mlir::isa<cir::TryOp>(op))
+ return mlir::WalkResult::skip();
+ auto initOp = mlir::dyn_cast<cir::InitCatchParamOp>(op);
+ if (!initOp)
+ return mlir::WalkResult::advance();
+
+ cir::InitCatchKind kind = initOp.getKind();
+ bool sameParam =
+ constructOp && constructOp.getParamAddr() == initOp.getParamAddr();
+ if (sameParam && constructOp.getKind() != kind) {
+ initOp.emitOpError("kind '")
+ << cir::stringifyInitCatchKind(kind)
+ << "' does not match the kind '"
+ << cir::stringifyInitCatchKind(constructOp.getKind())
+ << "' of the preceding 'cir.construct_catch_param'";
+ return mlir::WalkResult::interrupt();
+ }
+ if ((kind == cir::InitCatchKind::ReferenceToPointer ||
+ kind == cir::InitCatchKind::NonTrivialCopy) &&
+ !sameParam) {
+ initOp.emitOpError("'")
+ << cir::stringifyInitCatchKind(kind)
+ << "' requires a preceding 'cir.construct_catch_param' of the "
+ "same kind on the same parameter";
+ return mlir::WalkResult::interrupt();
+ }
+ return mlir::WalkResult::advance();
+ });
+ return failure(result.wasInterrupted());
+}
+
+LogicalResult cir::TryOp::verifyRegions() {
+ mlir::ArrayAttr handlerTypes = getHandlerTypes();
+ if (!handlerTypes) {
+ if (!getHandlerRegions().empty())
+ return emitOpError(
+ "handler regions must be empty when no handler types are present");
+ return success();
+ }
+
+ mlir::MutableArrayRef<mlir::Region> handlerRegions = getHandlerRegions();
+
+ // The parser and builder won't allow this to happen, but the loop below
+ // relies on the sizes being the same, so we check it here.
+ if (handlerRegions.size() != handlerTypes.size())
+ return emitOpError(
+ "number of handler regions and handler types must match");
+
+ // A filter handler and an unexpected handler together implement a dynamic
+ // exception specification. The filter try operation wraps the whole function
+ // body and exists only to check the specification, so the two must appear
+ // together and must stand alone. A catch-all in the same list would make the
+ // filter unreachable, and the filter handler already provides the
+ // continue-unwinding path that an unwind handler would supply.
+ if (llvm::any_of(handlerTypes, [](mlir::Attribute typeAttr) {
+ return mlir::isa<cir::EhFilterAttr, cir::EhUnexpectedAttr>(typeAttr);
+ })) {
+ if (handlerTypes.size() != 2 ||
+ !mlir::isa<cir::EhFilterAttr>(handlerTypes[0]) ||
+ !mlir::isa<cir::EhUnexpectedAttr>(handlerTypes[1]))
+ return emitOpError("a filter handler must be followed by an unexpected "
+ "handler, and the two must be the only handlers");
+ }
+
+ for (const auto &[typeAttr, handlerRegion] :
+ llvm::zip(handlerTypes, handlerRegions)) {
+ // Verify that handler regions have a !cir.eh_token block argument.
+ mlir::Block &entryBlock = handlerRegion.front();
+ if (entryBlock.getNumArguments() != 1 ||
+ !mlir::isa<cir::EhTokenType>(entryBlock.getArgument(0).getType()))
+ return emitOpError(
+ "handler region must have a single '!cir.eh_token' argument");
+
+ // The unwind, filter and unexpected regions do not require a
+ // cir.begin_catch. None of them catches the exception.
+ if (mlir::isa<cir::UnwindAttr, cir::EhFilterAttr, cir::EhUnexpectedAttr>(
+ typeAttr))
+ continue;
+
+ if (entryBlock.empty())
+ return emitOpError("catch handler region must not be empty");
+
+ // A terminate scope, which wraps the body of a function that cannot throw,
+ // is a catch-all handler that only terminates the program. The exception is
+ // caught by the runtime helper that cir.eh.terminate lowers to, so the
+ // handler region has no cir.begin_catch of its own.
+ if (mlir::isa<cir::EhTerminateOp>(entryBlock.front())) {
+ if (!mlir::isa<cir::CatchAllAttr>(typeAttr))
+ return emitOpError("'cir.eh.terminate' is only allowed in a catch-all "
+ "handler region");
+ continue;
+ }
+
+ // Nothing may run in a catch handler before cir.begin_catch, so it has to
+ // be the handler region's first operation, with two exceptions.
+ //
+ // When lifetime markers are enabled, the catch parameter's storage can be
+ // marked by a cir.lifetime.start. That parameter is the only variable
+ // whose lifetime begins before the catch is entered, so there is at most
+ // one such marker. Its lifetime-end cleanup has to run after the catch
+ // handler is finished (or exited by an exception unwind), so if there is a
+ // lifetime begin marker, it is followed by a cir.cleanup.scope that
+ // encloses the the rest of the handler with a cir.lifetime.end in its
+ // cleanup region.
+ //
+ // A cir.construct_catch_param may also precede cir.begin_catch, to
+ // perform any pre-begin_catch initialization of the catch parameter.
+ mlir::Operation *firstOp = &entryBlock.front();
+ if (mlir::isa<cir::LifetimeStartOp>(firstOp)) {
+ mlir::Operation *next = firstOp->getNextNode();
+ auto lifetimeScope = mlir::dyn_cast_if_present<cir::CleanupScopeOp>(next);
+ if (!lifetimeScope)
+ return emitOpError("'cir.lifetime.start' in a catch handler region "
+ "must be followed by the 'cir.cleanup.scope' of "
+ "its lifetime-end cleanup");
+ if (lifetimeScope.getBodyRegion().empty())
+ return emitOpError(
+ "'cir.lifetime.start' in a catch handler region must be "
+ "followed by the 'cir.cleanup.scope' of its lifetime-end "
+ "cleanup");
+ mlir::Block &scopeBody = lifetimeScope.getBodyRegion().front();
+ firstOp = scopeBody.empty() ? nullptr : &scopeBody.front();
+ }
+
+ auto constructOp =
+ mlir::dyn_cast_if_present<cir::ConstructCatchParamOp>(firstOp);
+ if (constructOp)
+ firstOp = firstOp->getNextNode();
+ if (!mlir::isa_and_present<cir::BeginCatchOp>(firstOp))
+ return emitOpError(
+ "catch handler region must start with 'cir.begin_catch'");
+
+ if (failed(verifyCatchParamPairing(handlerRegion, constructOp)))
+ return failure();
+ }
+
+ return success();
+}
+
+void cir::printTryHandlerRegions(
+ mlir::OpAsmPrinter &printer, cir::TryOp op,
+ mlir::MutableArrayRef<mlir::Region> handlerRegions,
+ mlir::ArrayAttr handlerTypes) {
+ if (!handlerTypes)
+ return;
+
+ for (const auto [typeIdx, typeAttr] : llvm::enumerate(handlerTypes)) {
+ if (typeIdx)
+ printer << " ";
+
+ if (mlir::isa<cir::CatchAllAttr>(typeAttr)) {
+ printer << "catch all ";
+ } else if (mlir::isa<cir::UnwindAttr>(typeAttr)) {
+ printer << "unwind ";
+ } else if (auto filterAttr = mlir::dyn_cast<cir::EhFilterAttr>(typeAttr)) {
+ printer << "filter [";
+ llvm::interleaveComma(
+ filterAttr.getPermittedTypes(), printer,
+ [&](mlir::Attribute sym) { printer.printAttribute(sym); });
+ printer << "] ";
+ } else if (mlir::isa<cir::EhUnexpectedAttr>(typeAttr)) {
+ printer << "unexpected ";
+ } else {
+ printer << "catch [type ";
+ printer.printAttribute(typeAttr);
+ printer << "] ";
+ }
+
+ // Print the handler region's !cir.eh_token block argument.
+ mlir::Region ®ion = handlerRegions[typeIdx];
+ if (!region.empty() && region.front().getNumArguments() > 0) {
+ printer << "(";
+ printer.printRegionArgument(region.front().getArgument(0));
+ printer << ") ";
+ }
+
+ printer.printRegion(region,
+ /*printEntryBLockArgs=*/false,
+ /*printBlockTerminators=*/true);
+ }
+}
+
+mlir::ParseResult cir::parseTryHandlerRegions(
+ mlir::OpAsmParser &parser,
+ llvm::SmallVectorImpl<std::unique_ptr<mlir::Region>> &handlerRegions,
+ mlir::ArrayAttr &handlerTypes) {
+
+ auto parseCheckedCatcherRegion = [&]() -> mlir::ParseResult {
+ handlerRegions.emplace_back(new mlir::Region);
+
+ mlir::Region &currRegion = *handlerRegions.back();
+
+ // Parse the required region argument: (%eh_token : !cir.eh_token)
+ llvm::SmallVector<mlir::OpAsmParser::Argument> regionArgs;
+ if (parser.parseLParen())
+ return failure();
+ mlir::OpAsmParser::Argument arg;
+ if (parser.parseArgument(arg, /*allowType=*/true))
+ return failure();
+ regionArgs.push_back(arg);
+ if (parser.parseRParen())
+ return failure();
+
+ mlir::SMLoc regionLoc = parser.getCurrentLocation();
+ if (parser.parseRegion(currRegion, regionArgs)) {
+ handlerRegions.clear();
+ return failure();
+ }
+
+ if (currRegion.empty())
+ return parser.emitError(regionLoc, "handler region shall not be empty");
+
+ if (!(currRegion.back().mightHaveTerminator() &&
+ currRegion.back().getTerminator()))
+ return parser.emitError(
+ regionLoc, "blocks are expected to be explicitly terminated");
+
+ return success();
+ };
+
+ bool hasCatchAll = false;
+ llvm::SmallVector<mlir::Attribute, 4> catcherAttrs;
+ while (parser.parseOptionalKeyword("catch").succeeded()) {
+ bool hasLSquare = parser.parseOptionalLSquare().succeeded();
+
+ llvm::StringRef attrStr;
+ if (parser.parseOptionalKeyword(&attrStr, {"all", "type"}).failed())
+ return parser.emitError(parser.getCurrentLocation(),
+ "expected 'all' or 'type' keyword");
+
+ bool isCatchAll = attrStr == "all";
+ if (isCatchAll) {
+ if (hasCatchAll)
+ return parser.emitError(parser.getCurrentLocation(),
+ "can't have more than one catch all");
+ hasCatchAll = true;
+ }
+
+ mlir::Attribute exceptionRTTIAttr;
+ if (!isCatchAll && parser.parseAttribute(exceptionRTTIAttr).failed())
+ return parser.emitError(parser.getCurrentLocation(),
+ "expected valid RTTI info attribute");
+
+ catcherAttrs.push_back(isCatchAll
+ ? cir::CatchAllAttr::get(parser.getContext())
+ : exceptionRTTIAttr);
+
+ if (hasLSquare && isCatchAll)
+ return parser.emitError(parser.getCurrentLocation(),
+ "catch all dosen't need RTTI info attribute");
+
+ if (hasLSquare && parser.parseRSquare().failed())
+ return parser.emitError(parser.getCurrentLocation(),
+ "expected `]` after RTTI info attribute");
+
+ if (parseCheckedCatcherRegion().failed())
+ return mlir::failure();
+ }
+
+ // A filter handler carries the type info symbols permitted by the enclosing
+ // function's dynamic exception specification. TryOp::verifyRegions enforces
+ // that it is paired with an unexpected handler and that the two stand alone.
+ if (parser.parseOptionalKeyword("filter").succeeded()) {
+ mlir::SMLoc filterLoc = parser.getCurrentLocation();
+ llvm::SmallVector<mlir::Attribute, 4> permittedTypes;
+ auto parsePermittedType = [&]() -> mlir::ParseResult {
+ mlir::SMLoc typeLoc = parser.getCurrentLocation();
+ mlir::Attribute rtti;
+ if (parser.parseAttribute(rtti).failed())
+ return mlir::failure();
+ if (!mlir::isa<cir::GlobalViewAttr>(rtti))
+ return parser.emitError(typeLoc, "expected a type info symbol naming a "
+ "permitted exception type");
+ permittedTypes.push_back(rtti);
+ return mlir::success();
+ };
+ if (parser
+ .parseCommaSeparatedList(mlir::OpAsmParser::Delimiter::Square,
+ parsePermittedType)
+ .failed())
+ return mlir::failure();
+
+ auto filterAttr = cir::EhFilterAttr::getChecked(
+ [&]() { return parser.emitError(filterLoc); }, parser.getContext(),
+ parser.getBuilder().getArrayAttr(permittedTypes));
+ if (!filterAttr)
+ return mlir::failure();
+ catcherAttrs.push_back(filterAttr);
+ if (parseCheckedCatcherRegion().failed())
+ return mlir::failure();
+ }
+
+ if (parser.parseOptionalKeyword("unexpected").succeeded()) {
+ catcherAttrs.push_back(cir::EhUnexpectedAttr::get(parser.getContext()));
+ if (parseCheckedCatcherRegion().failed())
+ return mlir::failure();
+ }
+
+ if (parser.parseOptionalKeyword("unwind").succeeded()) {
+ if (hasCatchAll)
+ return parser.emitError(parser.getCurrentLocation(),
+ "unwind can't be used with catch all");
+
+ catcherAttrs.push_back(cir::UnwindAttr::get(parser.getContext()));
+ if (parseCheckedCatcherRegion().failed())
+ return mlir::failure();
+ }
+
+ handlerTypes = parser.getBuilder().getArrayAttr(catcherAttrs);
+ return mlir::success();
+}
+
+//===----------------------------------------------------------------------===//
+// EhTypeIdOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult
+cir::EhTypeIdOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
+ Operation *op = symbolTable.lookupNearestSymbolFrom(*this, getTypeSymAttr());
+ if (!isa_and_nonnull<GlobalOp>(op))
+ return emitOpError("'")
+ << getTypeSym() << "' does not reference a valid cir.global";
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// InitCatchParamOp
+//===----------------------------------------------------------------------===//
+
+/// For the reference_to_pointer and reference_to_record_pointer kinds, emit an
+/// error on \p op unless \p paramAddrType is the type of the address of a
+/// reference to a pointer. Any other \p kind is accepted.
+static LogicalResult verifyCatchParamAddr(mlir::Operation *op,
+ cir::InitCatchKind kind,
+ cir::PointerType paramAddrType) {
+ if (kind != cir::InitCatchKind::ReferenceToPointer &&
+ kind != cir::InitCatchKind::ReferenceToRecordPointer)
+ return success();
+
+ auto refType = mlir::dyn_cast<cir::PointerType>(paramAddrType.getPointee());
+ if (!refType || !mlir::isa<cir::PointerType>(refType.getPointee()))
+ return op->emitOpError("'")
+ << cir::stringifyInitCatchKind(kind)
+ << "' requires 'param_addr' to be the address of a reference to a "
+ "pointer";
+ return success();
+}
+
+LogicalResult cir::InitCatchParamOp::verify() {
+ return verifyCatchParamAddr(*this, getKind(), getParamAddr().getType());
+}
+
+//===----------------------------------------------------------------------===//
+// ConstructCatchParamOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::ConstructCatchParamOp::verify() {
+ return verifyCatchParamAddr(*this, getKind(), getParamAddr().getType());
+}
+
+LogicalResult cir::ConstructCatchParamOp::verifySymbolUses(
+ SymbolTableCollection &symbolTable) {
+ auto copyFnAttr = getCopyFnAttr();
+ if (!copyFnAttr)
+ return success();
+ auto fn =
+ symbolTable.lookupNearestSymbolFrom<cir::FuncOp>(*this, getCopyFnAttr());
+ if (!fn)
+ return emitOpError("'")
+ << *getCopyFn() << "' does not reference a valid cir.func";
+
+ if (!fn->hasAttr(cir::CIRDialect::getCatchCopyThunkAttrName()))
+ return emitOpError("catch-init copy_fn must be tagged with the ")
+ << cir::CIRDialect::getCatchCopyThunkAttrName() << " attribute";
+
+ cir::FuncType fnType = fn.getFunctionType();
+ if (fnType.getNumInputs() != 2 || !fnType.hasVoidReturn())
+ return emitOpError("catch-init copy_fn must take two pointer arguments and "
+ "return void");
+
+ if (fnType.getInput(0) != getParamAddr().getType())
+ return emitOpError("first argument of catch-init copy_fn must match the "
+ "type of 'param_addr'");
+
+ if (fnType.getInput(1) != getParamAddr().getType())
+ return emitOpError(
+ "second argument of catch-init copy_fn must be a pointer "
+ "to the catch type");
+
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// EhDispatchOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::EhDispatchOp::verify() {
+ mlir::ArrayAttr handlerTypes = getCatchTypesAttr();
+ if (!handlerTypes)
+ return success();
+
+ bool hasFilter = false;
+ for (mlir::Attribute typeAttr : handlerTypes) {
+ if (!mlir::isa<cir::EhFilterAttr>(typeAttr))
+ continue;
+ if (hasFilter)
+ return emitOpError("can't have more than one 'filter' handler");
+ hasFilter = true;
+ }
+
+ // Unlike the other handlers, a filter does not take the place of the default
+ // destination. Its destination is taken when the exception is not permitted
+ // by the specification, and the default 'unwind' destination is taken when
+ // it is. A catch-all default would leave the filter unreachable.
+ if (hasFilter && getDefaultIsCatchAll())
+ return emitOpError(
+ "'filter' handler requires an 'unwind' default destination");
+
+ return success();
+}
+
+ParseResult cir::parseEhDispatchDestinations(
+ OpAsmParser &parser, mlir::ArrayAttr &catchTypes,
+ SmallVectorImpl<Block *> &catchDestinations, Block *&defaultDestination,
+ mlir::UnitAttr &defaultIsCatchAll) {
+ // Parse: [ ... ]
+ if (parser.parseLSquare())
+ return failure();
+
+ SmallVector<Attribute> handlerTypes;
+ bool hasCatchAll = false;
+ bool hasUnwind = false;
+
+ // Parse handler list.
+ auto parseHandler = [&]() -> ParseResult {
+ // Check for 'catch_all' or 'unwind' keywords.
+ if (succeeded(parser.parseOptionalKeyword("catch_all"))) {
+ if (hasCatchAll)
+ return parser.emitError(parser.getCurrentLocation(),
+ "duplicate 'catch_all' handler");
+ if (hasUnwind)
+ return parser.emitError(parser.getCurrentLocation(),
+ "cannot have both 'catch_all' and 'unwind'");
+ hasCatchAll = true;
+
+ if (parser.parseColon().failed())
+ return failure();
+
+ if (parser.parseSuccessor(defaultDestination).failed())
+ return failure();
+
+ return success();
+ }
+
+ if (succeeded(parser.parseOptionalKeyword("unwind"))) {
+ if (hasUnwind)
+ return parser.emitError(parser.getCurrentLocation(),
+ "duplicate 'unwind' handler");
+ if (hasCatchAll)
+ return parser.emitError(parser.getCurrentLocation(),
+ "cannot have both 'catch_all' and 'unwind'");
+ hasUnwind = true;
+
+ if (parser.parseColon().failed())
+ return failure();
+
+ if (parser.parseSuccessor(defaultDestination).failed())
+ return failure();
+ return success();
+ }
+
+ // A filter handler carries the type info symbols permitted by the
+ // enclosing function's dynamic exception specification. Its destination is
+ // taken when the exception is *not* one of those types.
+ if (succeeded(parser.parseOptionalKeyword("filter"))) {
+ SMLoc filterLoc = parser.getCurrentLocation();
+ SmallVector<Attribute, 4> permittedTypes;
+ auto parsePermittedType = [&]() -> ParseResult {
+ mlir::SMLoc typeLoc = parser.getCurrentLocation();
+ mlir::Attribute rtti;
+ if (parser.parseAttribute(rtti).failed())
+ return failure();
+ if (!mlir::isa<cir::GlobalViewAttr>(rtti))
+ return parser.emitError(typeLoc,
+ "expected a type info symbol naming a "
+ "permitted exception type");
+ permittedTypes.push_back(rtti);
+ return success();
+ };
+ if (parser
+ .parseCommaSeparatedList(OpAsmParser::Delimiter::Paren,
+ parsePermittedType)
+ .failed())
+ return failure();
+
+ auto filterAttr = cir::EhFilterAttr::getChecked(
+ [&]() { return parser.emitError(filterLoc); }, parser.getContext(),
+ parser.getBuilder().getArrayAttr(permittedTypes));
+ if (!filterAttr)
+ return failure();
+ handlerTypes.push_back(filterAttr);
+
+ if (parser.parseColon().failed())
+ return failure();
+
+ Block *dest;
+ if (parser.parseSuccessor(dest).failed())
+ return failure();
+ catchDestinations.push_back(dest);
+ return success();
+ }
+
+ // Otherwise, expect 'catch(<attr> : <type>) : ^block'.
+ // The 'catch(...)' wrapper allows the attribute to include its type
+ // without conflicting with the ':' used for the block destination.
+ if (parser.parseKeyword("catch").failed())
+ return failure();
+
+ if (parser.parseLParen().failed())
+ return failure();
+
+ mlir::Attribute catchTypeAttr;
+ if (parser.parseAttribute(catchTypeAttr).failed())
+ return failure();
+ handlerTypes.push_back(catchTypeAttr);
+
+ if (parser.parseRParen().failed())
+ return failure();
+
+ if (parser.parseColon().failed())
+ return failure();
+
+ Block *dest;
+ if (parser.parseSuccessor(dest).failed())
+ return failure();
+ catchDestinations.push_back(dest);
+ return success();
+ };
+
+ if (parser.parseCommaSeparatedList(parseHandler).failed())
+ return failure();
+
+ if (parser.parseRSquare().failed())
+ return failure();
+
+ // Verify we have catch_all or unwind.
+ if (!hasCatchAll && !hasUnwind)
+ return parser.emitError(parser.getCurrentLocation(),
+ "must have either 'catch_all' or 'unwind' handler");
+
+ // Add attributes and successors.
+ if (!handlerTypes.empty())
+ catchTypes = parser.getBuilder().getArrayAttr(handlerTypes);
+
+ if (hasCatchAll)
+ defaultIsCatchAll = parser.getBuilder().getUnitAttr();
+
+ return success();
+}
+
+void cir::printEhDispatchDestinations(OpAsmPrinter &p, cir::EhDispatchOp op,
+ mlir::ArrayAttr catchTypes,
+ SuccessorRange catchDestinations,
+ Block *defaultDestination,
+ mlir::UnitAttr defaultIsCatchAll) {
+ p << " [";
+ p.printNewline();
+
+ // If we have at least one catch type, print them.
+ if (catchTypes) {
+ // Print type handlers using 'catch(<attr>) : ^block' syntax.
+ llvm::interleave(
+ llvm::zip(catchTypes, catchDestinations),
+ [&](auto i) {
+ mlir::Attribute typeAttr = std::get<0>(i);
+ if (auto filterAttr = mlir::dyn_cast<cir::EhFilterAttr>(typeAttr)) {
+ p << " filter(";
+ llvm::interleaveComma(
+ filterAttr.getPermittedTypes(), p,
+ [&](mlir::Attribute sym) { p.printAttribute(sym); });
+ } else {
+ p << " catch(";
+ p.printAttribute(typeAttr);
+ }
+ p << ") : ";
+ p.printSuccessor(std::get<1>(i));
+ },
+ [&] {
+ p << ',';
+ p.printNewline();
+ });
+
+ p << ", ";
+ p.printNewline();
+ }
+
+ // Print catch_all or unwind handler.
+ if (defaultIsCatchAll)
+ p << " catch_all : ";
+ else
+ p << " unwind : ";
+ p.printSuccessor(defaultDestination);
+ p.printNewline();
+
+ p << "]";
+}
+
+//===----------------------------------------------------------------------===//
+// TableGen-generated op definitions
+//===----------------------------------------------------------------------===//
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsEH.cpp.inc"
diff --git a/clang/lib/CIR/Dialect/IR/CIROpDefinitions.cpp b/clang/lib/CIR/Dialect/IR/CIROpsMath.cpp
similarity index 51%
rename from clang/lib/CIR/Dialect/IR/CIROpDefinitions.cpp
rename to clang/lib/CIR/Dialect/IR/CIROpsMath.cpp
index 629fcb53bbbe4e..909f13d69da8aa 100644
--- a/clang/lib/CIR/Dialect/IR/CIROpDefinitions.cpp
+++ b/clang/lib/CIR/Dialect/IR/CIROpsMath.cpp
@@ -1,4 +1,4 @@
-//===- CIROpDefinitions.cpp - Generated CIR op definitions ----------------===//
+//===- CIROpsMath.cpp - CIR ops in partition `Math` -----------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
@@ -6,13 +6,12 @@
//
//===----------------------------------------------------------------------===//
//
-// This file is the template for the CIROpDefinitions.<N>.cpp shards. Each shard
-// defines GET_OP_DEFS_<N> and includes the TableGen-generated definitions of
-// one subset of the CIR operations.
+// This file implements the CIR ops in partition `Math`, which are all
+// TableGen-generated.
//
//===----------------------------------------------------------------------===//
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIROpsMath.h"
#include "CIROpDefinitions.h"
@@ -21,4 +20,9 @@
using namespace mlir;
using namespace cir;
-#include "clang/CIR/Dialect/IR/CIROps.cpp.inc"
+//===----------------------------------------------------------------------===//
+// TableGen-generated op definitions
+//===----------------------------------------------------------------------===//
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsMath.cpp.inc"
diff --git a/clang/lib/CIR/Dialect/IR/CIROpsSymbols.cpp b/clang/lib/CIR/Dialect/IR/CIROpsSymbols.cpp
new file mode 100644
index 00000000000000..9c1e36156afa7c
--- /dev/null
+++ b/clang/lib/CIR/Dialect/IR/CIROpsSymbols.cpp
@@ -0,0 +1,1536 @@
+//===- CIROpsSymbols.cpp - CIR ops in partition `Symbols` -----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the CIR ops that define or refer to symbols.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/CIR/Dialect/IR/CIROpsSymbols.h"
+#include "clang/CIR/Dialect/IR/CIROpsControlFlow.h"
+#include "clang/CIR/Dialect/IR/CIROpsCoroutine.h"
+
+#include "CIROpDefinitions.h"
+
+#include "mlir/IR/DialectImplementation.h"
+#include "mlir/Interfaces/FunctionImplementation.h"
+#include "clang/CIR/Dialect/IR/CIRDataLayout.h"
+#include "llvm/ADT/SetOperations.h"
+#include "llvm/ADT/SmallSet.h"
+
+using namespace mlir;
+using namespace cir;
+
+//===----------------------------------------------------------------------===//
+// Helpers
+//===----------------------------------------------------------------------===//
+
+// Parses one of the keywords provided in the list `keywords` and returns the
+// position of the parsed keyword in the list. If none of the keywords from the
+// list is parsed, returns -1.
+static int parseOptionalKeywordAlternative(AsmParser &parser,
+ ArrayRef<llvm::StringRef> keywords) {
+ for (auto en : llvm::enumerate(keywords)) {
+ if (succeeded(parser.parseOptionalKeyword(en.value())))
+ return en.index();
+ }
+ return -1;
+}
+
+namespace {
+template <typename Ty> struct EnumTraits {};
+
+#define REGISTER_ENUM_TYPE(Ty) \
+ template <> struct EnumTraits<cir::Ty> { \
+ static llvm::StringRef stringify(cir::Ty value) { \
+ return stringify##Ty(value); \
+ } \
+ static unsigned getMaxEnumVal() { return cir::getMaxEnumValFor##Ty(); } \
+ }
+
+REGISTER_ENUM_TYPE(GlobalLinkageKind);
+REGISTER_ENUM_TYPE(VisibilityKind);
+REGISTER_ENUM_TYPE(CallingConv);
+} // namespace
+
+/// Parse an enum from the keyword, or default to the provided default value.
+/// The return type is the enum type by default, unless overriden with the
+/// second template argument.
+template <typename EnumTy, typename RetTy = EnumTy>
+static RetTy parseOptionalCIRKeyword(AsmParser &parser, EnumTy defaultValue) {
+ llvm::SmallVector<llvm::StringRef, 10> names;
+ for (unsigned i = 0, e = EnumTraits<EnumTy>::getMaxEnumVal(); i <= e; ++i)
+ names.push_back(EnumTraits<EnumTy>::stringify(static_cast<EnumTy>(i)));
+
+ int index = parseOptionalKeywordAlternative(parser, names);
+ if (index == -1)
+ return static_cast<RetTy>(defaultValue);
+ return static_cast<RetTy>(index);
+}
+
+/// Parse an enum from the keyword, return failure if the keyword is not found.
+template <typename EnumTy, typename RetTy = EnumTy>
+static ParseResult parseCIRKeyword(AsmParser &parser, RetTy &result) {
+ llvm::SmallVector<llvm::StringRef, 10> names;
+ for (unsigned i = 0, e = EnumTraits<EnumTy>::getMaxEnumVal(); i <= e; ++i)
+ names.push_back(EnumTraits<EnumTy>::stringify(static_cast<EnumTy>(i)));
+
+ int index = parseOptionalKeywordAlternative(parser, names);
+ if (index == -1)
+ return failure();
+ result = static_cast<RetTy>(index);
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// InlineKindAttr (FIXME: remove once FuncOp uses assembly format)
+//===----------------------------------------------------------------------===//
+
+static ParseResult parseInlineKindAttr(OpAsmParser &parser,
+ cir::InlineKindAttr &inlineKindAttr) {
+ // Static list of possible inline kind keywords
+ static constexpr llvm::StringRef keywords[] = {"no_inline", "always_inline",
+ "inline_hint"};
+
+ // Parse the inline kind keyword (optional)
+ llvm::StringRef keyword;
+ if (parser.parseOptionalKeyword(&keyword, keywords).failed()) {
+ // Not an inline kind keyword, leave inlineKindAttr empty
+ return success();
+ }
+
+ // Parse the enum value from the keyword
+ auto inlineKindResult = ::cir::symbolizeEnum<::cir::InlineKind>(keyword);
+ if (!inlineKindResult) {
+ return parser.emitError(parser.getCurrentLocation(), "expected one of [")
+ << llvm::join(llvm::ArrayRef(keywords), ", ")
+ << "] for inlineKind, got: " << keyword;
+ }
+
+ inlineKindAttr =
+ ::cir::InlineKindAttr::get(parser.getContext(), *inlineKindResult);
+ return success();
+}
+
+static void printInlineKindAttr(OpAsmPrinter &p,
+ cir::InlineKindAttr inlineKindAttr) {
+ if (inlineKindAttr) {
+ p << " " << stringifyInlineKind(inlineKindAttr.getValue());
+ }
+}
+
+//===----------------------------------------------------------------------===//
+// LocalInitOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult
+cir::LocalInitOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
+ cir::GlobalOp global = getReferencedGlobal(symbolTable);
+ if (!global)
+ return emitOpError("'")
+ << getGlobalName() << "' does not reference a valid cir.global";
+
+ if (getTls() && !global.getTlsModel())
+ return emitOpError("access to global not marked thread local");
+
+ if (!global.getDynamicInitGuard().has_value())
+ return emitOpError("static_local attribute mismatch");
+
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// RegisterExitDtorOp
+//===----------------------------------------------------------------------===//
+
+/// Returns true if \p op is nested in the ctor region of a cir.global or
+/// cir.local_init without crossing a cir.func or a cir.register_exit_dtor.
+static bool isInVarCtorRegion(mlir::Operation *op) {
+ for (mlir::Operation *parent = op->getParentOp(); parent;
+ parent = parent->getParentOp()) {
+ mlir::Region *ctorRegion = nullptr;
+ if (auto global = mlir::dyn_cast<cir::GlobalOp>(parent))
+ ctorRegion = &global.getCtorRegion();
+ else if (auto localInit = mlir::dyn_cast<cir::LocalInitOp>(parent))
+ ctorRegion = &localInit.getCtorRegion();
+ else if (mlir::isa<cir::FuncOp, cir::RegisterExitDtorOp>(parent))
+ return false;
+ if (ctorRegion)
+ return ctorRegion->isAncestor(op->getParentRegion());
+ }
+ return false;
+}
+
+LogicalResult cir::RegisterExitDtorOp::verify() {
+ mlir::Block &body = getBody().front();
+ if (body.without_terminator().empty())
+ return emitOpError("body must destroy the object");
+ if (body.getTerminator()->getNumOperands())
+ return emitOpError("body must not yield a value");
+ if (!isInVarCtorRegion(*this))
+ return emitOpError("must be in the ctor region of a cir.global or "
+ "cir.local_init");
+ return success();
+}
+
+LogicalResult
+cir::RegisterExitDtorOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
+ auto global = symbolTable.lookupNearestSymbolFrom<cir::GlobalOp>(
+ *this, getObjectAttr());
+ if (!global)
+ return emitOpError("'")
+ << getObject() << "' does not reference a valid cir.global";
+ if (!global.getCtorRegion().empty() || !global.getDtorRegion().empty())
+ return emitOpError("object must not have ctor or dtor regions");
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// CallOp
+//===----------------------------------------------------------------------===//
+
+mlir::OperandRange cir::CallOp::getArgOperands() {
+ if (isIndirect())
+ return getArgs().drop_front(1);
+ return getArgs();
+}
+
+mlir::MutableOperandRange cir::CallOp::getArgOperandsMutable() {
+ mlir::MutableOperandRange args = getArgsMutable();
+ if (isIndirect())
+ return args.slice(1, args.size() - 1);
+ return args;
+}
+
+mlir::Value cir::CallOp::getIndirectCall() {
+ assert(isIndirect());
+ return getOperand(0);
+}
+
+/// Return the operand at index 'i'.
+Value cir::CallOp::getArgOperand(unsigned i) {
+ if (isIndirect())
+ ++i;
+ return getOperand(i);
+}
+
+/// Return the number of operands.
+unsigned cir::CallOp::getNumArgOperands() {
+ if (isIndirect())
+ return this->getOperation()->getNumOperands() - 1;
+ return this->getOperation()->getNumOperands();
+}
+
+static mlir::ParseResult
+parseTryCallDestinations(mlir::OpAsmParser &parser,
+ mlir::OperationState &result) {
+ mlir::Block *normalDestSuccessor;
+ if (parser.parseSuccessor(normalDestSuccessor))
+ return mlir::failure();
+
+ if (parser.parseComma())
+ return mlir::failure();
+
+ mlir::Block *unwindDestSuccessor;
+ if (parser.parseSuccessor(unwindDestSuccessor))
+ return mlir::failure();
+
+ result.addSuccessors(normalDestSuccessor);
+ result.addSuccessors(unwindDestSuccessor);
+ return mlir::success();
+}
+
+/// Reject an attribute in an explicit attribute dictionary whose value is not
+/// the attribute class expected for its name. Where an op declares the
+/// attribute it is stored as a property, so a value of the wrong class would
+/// be dropped without a diagnostic.
+static ParseResult checkEffectAttrKinds(mlir::OpAsmParser &parser,
+ llvm::SMLoc loc,
+ const mlir::NamedAttrList &attrs) {
+ if (mlir::Attribute effects =
+ attrs.get(CIRDialect::getMemoryEffectsAttrName()))
+ if (!mlir::isa<cir::MemoryEffectsAttr>(effects))
+ return parser.emitError(loc, "attribute '")
+ << CIRDialect::getMemoryEffectsAttrName()
+ << "' must be a #cir.memory_effects attribute";
+
+ if (mlir::Attribute uwtable = attrs.get(CIRDialect::getUwtableAttrName()))
+ if (!mlir::isa<cir::UnwindTableKindAttr>(uwtable))
+ return parser.emitError(loc, "attribute '")
+ << CIRDialect::getUwtableAttrName()
+ << "' must be a #cir.uwtable attribute";
+
+ if (mlir::Attribute framePointer =
+ attrs.get(CIRDialect::getFramePointerAttrName()))
+ if (!mlir::isa<cir::FramePointerKindAttr>(framePointer))
+ return parser.emitError(loc, "attribute '")
+ << CIRDialect::getFramePointerAttrName()
+ << "' must be a #cir.frame_pointer attribute";
+
+ for (llvm::StringRef name :
+ {CIRDialect::getNoUnwindAttrName(), CIRDialect::getWillReturnAttrName(),
+ CIRDialect::getMustProgressAttrName()})
+ if (mlir::Attribute flag = attrs.get(name))
+ if (!mlir::isa<mlir::UnitAttr>(flag))
+ return parser.emitError(loc, "attribute '")
+ << name << "' must be a unit attribute";
+
+ return mlir::success();
+}
+
+static mlir::ParseResult parseCallCommon(mlir::OpAsmParser &parser,
+ mlir::OperationState &result,
+ bool hasDestinationBlocks = false) {
+ llvm::SmallVector<mlir::OpAsmParser::UnresolvedOperand, 4> ops;
+ llvm::SMLoc opsLoc;
+ mlir::FlatSymbolRefAttr calleeAttr;
+
+ // If we cannot parse a string callee, it means this is an indirect call.
+ if (!parser
+ .parseOptionalAttribute(calleeAttr, CIRDialect::getCalleeAttrName(),
+ result.attributes)
+ .has_value()) {
+ OpAsmParser::UnresolvedOperand indirectVal;
+ // Do not resolve right now, since we need to figure out the type
+ if (parser.parseOperand(indirectVal).failed())
+ return failure();
+ ops.push_back(indirectVal);
+ }
+
+ if (parser.parseLParen())
+ return mlir::failure();
+
+ opsLoc = parser.getCurrentLocation();
+ if (parser.parseOperandList(ops))
+ return mlir::failure();
+ if (parser.parseRParen())
+ return mlir::failure();
+
+ if (hasDestinationBlocks &&
+ parseTryCallDestinations(parser, result).failed()) {
+ return ::mlir::failure();
+ }
+
+ if (parser.parseOptionalKeyword("cc").succeeded()) {
+ cir::CallingConv callingConv;
+ if (parser.parseLParen().failed())
+ return failure();
+ if (parseCIRKeyword<cir::CallingConv>(parser, callingConv).failed())
+ return parser.emitError(parser.getCurrentLocation(),
+ "unknown calling convention");
+ if (parser.parseRParen().failed())
+ return failure();
+ result.addAttribute(
+ CIRDialect::getCallingConvAttrName(),
+ cir::CallingConvAttr::get(parser.getContext(), callingConv));
+ }
+
+ if (parser.parseOptionalKeyword("musttail").succeeded())
+ result.addAttribute(CIRDialect::getMustTailAttrName(),
+ mlir::UnitAttr::get(parser.getContext()));
+
+ if (parser.parseOptionalKeyword("nothrow").succeeded())
+ result.addAttribute(CIRDialect::getNoThrowAttrName(),
+ mlir::UnitAttr::get(parser.getContext()));
+
+ if (parser.parseOptionalKeyword("nounwind").succeeded())
+ result.addAttribute(CIRDialect::getNoUnwindAttrName(),
+ mlir::UnitAttr::get(parser.getContext()));
+
+ if (parser.parseOptionalKeyword("willreturn").succeeded())
+ result.addAttribute(CIRDialect::getWillReturnAttrName(),
+ mlir::UnitAttr::get(parser.getContext()));
+
+ llvm::SMLoc attrsLoc = parser.getCurrentLocation();
+ if (parser.parseOptionalAttrDict(result.attributes))
+ return ::mlir::failure();
+
+ if (checkEffectAttrKinds(parser, attrsLoc, result.attributes).failed())
+ return ::mlir::failure();
+
+ if (parser.parseColon())
+ return ::mlir::failure();
+
+ SmallVector<Type> argTypes;
+ SmallVector<DictionaryAttr> argAttrs;
+ SmallVector<Type> resultTypes;
+ SmallVector<DictionaryAttr> resultAttrs;
+ if (call_interface_impl::parseFunctionSignature(parser, argTypes, argAttrs,
+ resultTypes, resultAttrs))
+ return mlir::failure();
+
+ if (resultTypes.size() > 1 || resultAttrs.size() > 1)
+ return parser.emitError(
+ parser.getCurrentLocation(),
+ "functions with multiple return types are not supported");
+
+ result.addTypes(resultTypes);
+
+ if (parser.resolveOperands(ops, argTypes, opsLoc, result.operands))
+ return mlir::failure();
+
+ if (!resultAttrs.empty() && resultAttrs[0])
+ result.addAttribute(
+ CIRDialect::getResAttrsAttrName(),
+ mlir::ArrayAttr::get(parser.getContext(), {resultAttrs[0]}));
+
+ // ArrayAttr requires a vector of 'Attribute', so we have to do the conversion
+ // here into a separate collection.
+ llvm::SmallVector<Attribute> convertedArgAttrs;
+ bool argAttrsEmpty = true;
+
+ llvm::transform(argAttrs, std::back_inserter(convertedArgAttrs),
+ [&](DictionaryAttr da) -> mlir::Attribute {
+ if (da)
+ argAttrsEmpty = false;
+ return da;
+ });
+
+ if (!argAttrsEmpty) {
+ llvm::ArrayRef argAttrsRef = convertedArgAttrs;
+ if (!calleeAttr) {
+ // Fixup for indirect calls, which get an extra entry in the 'args' for
+ // the indirect type, which doesn't get attributes.
+ argAttrsRef = argAttrsRef.drop_front();
+ }
+ result.addAttribute(CIRDialect::getArgAttrsAttrName(),
+ mlir::ArrayAttr::get(parser.getContext(), argAttrsRef));
+ }
+
+ return mlir::success();
+}
+
+static void printCallCommon(mlir::Operation *op,
+ mlir::FlatSymbolRefAttr calleeSym,
+ mlir::Value indirectCallee,
+ mlir::OpAsmPrinter &printer, bool isNothrow,
+ ArrayAttr argAttrs, ArrayAttr resAttrs,
+ mlir::Block *normalDest = nullptr,
+ mlir::Block *unwindDest = nullptr) {
+ printer << ' ';
+
+ auto callLikeOp = mlir::cast<cir::CIRCallOpInterface>(op);
+ auto ops = callLikeOp.getArgOperands();
+
+ if (calleeSym) {
+ // Direct calls
+ printer.printAttributeWithoutType(calleeSym);
+ } else {
+ // Indirect calls
+ assert(indirectCallee);
+ printer << indirectCallee;
+ }
+
+ printer << "(" << ops << ")";
+
+ if (normalDest) {
+ assert(unwindDest && "expected two successors");
+ auto tryCall = cast<cir::TryCallOp>(op);
+ printer << ' ' << tryCall.getNormalDest();
+ printer << ",";
+ printer << ' ';
+ printer << tryCall.getUnwindDest();
+ }
+
+ cir::CallingConv callingConv = callLikeOp.getCallingConv();
+ if (callingConv != cir::CallingConv::C)
+ printer << " cc(" << stringifyCallingConv(callingConv) << ")";
+
+ if (op->hasAttr(CIRDialect::getMustTailAttrName()))
+ printer << " musttail";
+
+ if (isNothrow)
+ printer << " nothrow";
+
+ if (op->hasAttr(CIRDialect::getNoUnwindAttrName()))
+ printer << " nounwind";
+
+ if (op->hasAttr(CIRDialect::getWillReturnAttrName()))
+ printer << " willreturn";
+
+ llvm::StringRef elidedAttrs[] = {
+ CIRDialect::getCalleeAttrName(),
+ CIRDialect::getCallingConvAttrName(),
+ CIRDialect::getMustTailAttrName(),
+ CIRDialect::getNoThrowAttrName(),
+ CIRDialect::getNoUnwindAttrName(),
+ CIRDialect::getWillReturnAttrName(),
+ CIRDialect::getOperandSegmentSizesAttrName(),
+ "res_attrs",
+ "arg_attrs",
+ };
+ // TODO: Split inherent and discardable attribute printing instead of
+ // materializing a single dictionary that mixes the two storage classes.
+ llvm::SmallVector<mlir::NamedAttribute> attrs;
+ for (mlir::NamedAttribute attr : op->getDiscardableAttrs())
+ if (!llvm::is_contained(elidedAttrs, attr.getName()))
+ attrs.push_back(attr);
+ op->getName().walkInherentAttrs(op, [&](llvm::StringRef name,
+ mlir::Attribute &attr) {
+ if (!llvm::is_contained(elidedAttrs, name))
+ attrs.emplace_back(mlir::StringAttr::get(op->getContext(), name), attr);
+ });
+ llvm::sort(attrs, [](mlir::NamedAttribute lhs, mlir::NamedAttribute rhs) {
+ return lhs.getName().strref() < rhs.getName().strref();
+ });
+ printer.printOptionalAttrDict(attrs);
+ printer << " : ";
+ if (calleeSym || !argAttrs) {
+ call_interface_impl::printFunctionSignature(
+ printer, op->getOperands().getTypes(), argAttrs,
+ /*isVariadic=*/false, op->getResultTypes(), resAttrs);
+ } else {
+ // indirect function calls use an 'arg' type for the type of its indirect
+ // argument. However, we don't store a similar attribute collection. In
+ // order to make `printFunctionSignature` have the attributes line up, we
+ // have to make a 'shimmed' copy of the attributes that have a blank set of
+ // attributes for the indirect argument.
+ llvm::SmallVector<Attribute> shimmedArgAttrs;
+ shimmedArgAttrs.push_back(mlir::DictionaryAttr::get(op->getContext(), {}));
+ shimmedArgAttrs.append(argAttrs.begin(), argAttrs.end());
+ call_interface_impl::printFunctionSignature(
+ printer, op->getOperands().getTypes(),
+ mlir::ArrayAttr::get(op->getContext(), shimmedArgAttrs),
+ /*isVariadic=*/false, op->getResultTypes(), resAttrs);
+ }
+}
+
+mlir::ParseResult cir::CallOp::parse(mlir::OpAsmParser &parser,
+ mlir::OperationState &result) {
+ return parseCallCommon(parser, result);
+}
+
+void cir::CallOp::print(mlir::OpAsmPrinter &p) {
+ mlir::Value indirectCallee = isIndirect() ? getIndirectCall() : nullptr;
+ printCallCommon(*this, getCalleeAttr(), indirectCallee, p, getNothrow(),
+ getArgAttrsAttr(), getResAttrsAttr());
+}
+
+static LogicalResult
+verifyCallCommInSymbolUses(mlir::Operation *op,
+ SymbolTableCollection &symbolTable) {
+ auto callIf = cast<cir::CIRCallOpInterface>(op);
+
+ // An indirect call is checked against the function type its callee pointer
+ // points to, and a direct call against its callee's declaration.
+ cir::FuncType fnType;
+ bool hasPrototype = true;
+ auto fnAttr =
+ op->getAttrOfType<FlatSymbolRefAttr>(CIRDialect::getCalleeAttrName());
+ if (!fnAttr) {
+ if (op->getNumOperands() == 0)
+ return op->emitOpError("indirect call requires a callee operand");
+ mlir::Type calleeTy = callIf.getIndirectCall().getType();
+ if (auto ptrTy = mlir::dyn_cast<cir::PointerType>(calleeTy))
+ fnType = mlir::dyn_cast<cir::FuncType>(ptrTy.getPointee());
+ if (!fnType)
+ return op->emitOpError()
+ << "indirect callee must be a pointer to a function, but has type "
+ << calleeTy;
+ } else {
+ auto fn = symbolTable.lookupNearestSymbolFrom<cir::FuncOp>(op, fnAttr);
+ if (!fn)
+ return op->emitOpError() << "'" << fnAttr.getValue()
+ << "' does not reference a valid function";
+ fnType = fn.getFunctionType();
+ hasPrototype = !fn.getNoProto();
+ }
+
+ // Verify that the operand and result types match the callee. Operands are
+ // not checked for a direct call to a function declared no_proto.
+ if (hasPrototype) {
+ unsigned numCallOperands = callIf.getNumArgOperands();
+ unsigned numFnOpOperands = fnType.getNumInputs();
+
+ if (!fnType.isVarArg() && numCallOperands != numFnOpOperands)
+ return op->emitOpError("incorrect number of operands for callee");
+ if (fnType.isVarArg() && numCallOperands < numFnOpOperands)
+ return op->emitOpError("too few operands for callee");
+
+ for (unsigned i = 0, e = numFnOpOperands; i != e; ++i)
+ if (callIf.getArgOperand(i).getType() != fnType.getInput(i))
+ return op->emitOpError("operand type mismatch: expected operand type ")
+ << fnType.getInput(i) << ", but provided "
+ << callIf.getArgOperand(i).getType() << " for operand number "
+ << i;
+ }
+
+ assert(!cir::MissingFeatures::opCallCallConv());
+
+ // Void function must not return any results.
+ if (fnType.hasVoidReturn() && op->getNumResults() != 0)
+ return op->emitOpError("callee returns void but call has results");
+
+ // Non-void function calls must return exactly one result.
+ if (!fnType.hasVoidReturn() && op->getNumResults() != 1)
+ return op->emitOpError("incorrect number of results for callee");
+
+ // Parent function and return value types must match.
+ if (!fnType.hasVoidReturn() &&
+ op->getResultTypes().front() != fnType.getReturnType()) {
+ return op->emitOpError("result type mismatch: expected ")
+ << fnType.getReturnType() << ", but provided "
+ << op->getResult(0).getType();
+ }
+
+ return mlir::success();
+}
+
+LogicalResult
+cir::CallOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
+ return verifyCallCommInSymbolUses(*this, symbolTable);
+}
+
+//===----------------------------------------------------------------------===//
+// TryCallOp
+//===----------------------------------------------------------------------===//
+
+mlir::OperandRange cir::TryCallOp::getArgOperands() {
+ if (isIndirect())
+ return getArgs().drop_front(1);
+ return getArgs();
+}
+
+mlir::MutableOperandRange cir::TryCallOp::getArgOperandsMutable() {
+ mlir::MutableOperandRange args = getArgsMutable();
+ if (isIndirect())
+ return args.slice(1, args.size() - 1);
+ return args;
+}
+
+mlir::Value cir::TryCallOp::getIndirectCall() {
+ assert(isIndirect());
+ return getOperand(0);
+}
+
+/// Return the operand at index 'i'.
+Value cir::TryCallOp::getArgOperand(unsigned i) {
+ if (isIndirect())
+ ++i;
+ return getOperand(i);
+}
+
+/// Return the number of operands.
+unsigned cir::TryCallOp::getNumArgOperands() {
+ if (isIndirect())
+ return this->getOperation()->getNumOperands() - 1;
+ return this->getOperation()->getNumOperands();
+}
+
+LogicalResult
+cir::TryCallOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
+ return verifyCallCommInSymbolUses(*this, symbolTable);
+}
+
+mlir::ParseResult cir::TryCallOp::parse(mlir::OpAsmParser &parser,
+ mlir::OperationState &result) {
+ return parseCallCommon(parser, result, /*hasDestinationBlocks=*/true);
+}
+
+void cir::TryCallOp::print(::mlir::OpAsmPrinter &p) {
+ mlir::Value indirectCallee = isIndirect() ? getIndirectCall() : nullptr;
+ printCallCommon(*this, getCalleeAttr(), indirectCallee, p, getNothrow(),
+ getArgAttrsAttr(), getResAttrsAttr(), getNormalDest(),
+ getUnwindDest());
+}
+
+//===----------------------------------------------------------------------===//
+// GlobalOp
+//===----------------------------------------------------------------------===//
+
+static ParseResult parseConstantValue(OpAsmParser &parser,
+ mlir::Attribute &valueAttr) {
+ NamedAttrList attr;
+ return parser.parseAttribute(valueAttr, "value", attr);
+}
+
+static void printConstant(OpAsmPrinter &p, Attribute value) {
+ p.printAttribute(value);
+}
+
+mlir::LogicalResult cir::GlobalOp::verify() {
+ // A function is not an object, so it cannot be the type of a global. A
+ // global that holds a function's address carries a pointer type instead.
+ if (mlir::isa<cir::FuncType>(getSymType()))
+ return emitOpError("global type cannot be a function type");
+
+ // Verify that the initial value, if present, is either a unit attribute or
+ // an attribute CIR supports.
+ if (getInitialValue().has_value()) {
+ if (checkConstantTypes(getOperation(), getSymType(), *getInitialValue())
+ .failed())
+ return failure();
+ }
+
+ // The initial value can't be recomputed from the type once the ctor is
+ // lowered (e.g. a null member pointer becomes -1 after CXXABILowering).
+ if (!getCtorRegion().empty() && !getInitialValue().has_value())
+ return emitOpError(
+ "cannot have a constructor region without an initial value");
+
+ std::optional<DynamicInitInfoAttr> info = getDynamicInitInfo();
+
+ if ((getDynamicInitGuard().has_value()) && (!info || info->getLocal()) &&
+ (!getCtorRegion().empty() || !getDtorRegion().empty()))
+ return emitOpError(
+ "Cannot have a static-local global-op with a constructor or "
+ "destructor, they require in-function initialization via LocalInitOp");
+
+ // CIRGen emits 'dynamic_init_guard' and 'dynamic_init_info' together and
+ // they are only meaningful together: the guard drives lowering, which reads
+ // the info. Require both or neither so malformed .cir can carry neither a
+ // guard without info nor a dangling info nothing will read.
+ if (getDynamicInitGuard().has_value() != getDynamicInitInfo().has_value())
+ return emitOpError("'dynamic_init_guard' and 'dynamic_init_info' must be "
+ "present together");
+
+ if (getTlsRefs()) {
+ // 'Unordered' TLS globals (variable template instantiations) legitimately
+ // carry both: they can't use the shared __tls_init guard, so CIRGen also
+ // gives them their own dynamic-init guard, mangled the same as
+ // 'tls_refs's own guard name. 'Ordered' TLS globals share __tls_init and
+ // have no per-variable guard name, so a dynamic-init guard there would be
+ // a mismatch.
+ if (getDynamicInitGuard().has_value() && !getTlsRefs()->getGuardName())
+ return emitOpError("cannot have a dynamic-init guard combined with "
+ "ordered tls references");
+ if (!getTlsModel())
+ return emitOpError("'tls_refs' only valid for tls");
+ }
+
+ if (getAliasee().has_value()) {
+ if (getInitialValue().has_value() || !getCtorRegion().empty() ||
+ !getDtorRegion().empty())
+ return emitOpError("global alias shall not have an initializer or "
+ "constructor/destructor regions");
+ }
+
+ // TODO(CIR): Many other checks for properties that haven't been upstreamed
+ // yet.
+
+ return success();
+}
+
+void cir::GlobalOp::build(
+ OpBuilder &odsBuilder, OperationState &odsState, llvm::StringRef sym_name,
+ mlir::Type sym_type, bool isConstant,
+ mlir::ptr::MemorySpaceAttrInterface addrSpace,
+ cir::GlobalLinkageKind linkage,
+ function_ref<void(OpBuilder &, Location)> ctorBuilder,
+ function_ref<void(OpBuilder &, Location)> dtorBuilder) {
+ odsState.addAttribute(getSymNameAttrName(odsState.name),
+ odsBuilder.getStringAttr(sym_name));
+ odsState.addAttribute(getSymTypeAttrName(odsState.name),
+ mlir::TypeAttr::get(sym_type));
+ auto &properties = odsState.getOrAddProperties<cir::GlobalOp::Properties>();
+ properties.setConstant(isConstant);
+
+ addrSpace = normalizeDefaultAddressSpace(addrSpace);
+ if (addrSpace)
+ odsState.addAttribute(getAddrSpaceAttrName(odsState.name), addrSpace);
+
+ cir::GlobalLinkageKindAttr linkageAttr =
+ cir::GlobalLinkageKindAttr::get(odsBuilder.getContext(), linkage);
+ odsState.addAttribute(getLinkageAttrName(odsState.name), linkageAttr);
+
+ Region *ctorRegion = odsState.addRegion();
+ if (ctorBuilder) {
+ odsBuilder.createBlock(ctorRegion);
+ ctorBuilder(odsBuilder, odsState.location);
+ }
+
+ Region *dtorRegion = odsState.addRegion();
+ if (dtorBuilder) {
+ odsBuilder.createBlock(dtorRegion);
+ dtorBuilder(odsBuilder, odsState.location);
+ }
+}
+
+/// Given the region at `index`, or the parent operation if `index` is None,
+/// return the successor regions. These are the regions that may be selected
+/// during the flow of control. `operands` is a set of optional attributes that
+/// correspond to a constant value for each operand, or null if that operand is
+/// not a constant.
+void cir::GlobalOp::getSuccessorRegions(
+ mlir::RegionBranchPoint point, SmallVectorImpl<RegionSuccessor> ®ions) {
+ // The `ctor` and `dtor` regions always branch back to the parent operation.
+ if (!point.isParent()) {
+ regions.emplace_back(getOperation());
+ return;
+ }
+
+ // Don't consider the ctor region if it is empty.
+ Region *ctorRegion = &this->getCtorRegion();
+ if (ctorRegion->empty())
+ ctorRegion = nullptr;
+
+ // Don't consider the dtor region if it is empty.
+ Region *dtorRegion = &this->getDtorRegion();
+ if (dtorRegion->empty())
+ dtorRegion = nullptr;
+
+ // If the condition isn't constant, both regions may be executed.
+ if (ctorRegion)
+ regions.push_back(RegionSuccessor(ctorRegion));
+ if (dtorRegion)
+ regions.push_back(RegionSuccessor(dtorRegion));
+}
+
+static void printComdatName(OpAsmPrinter &p, StringAttr comdat) {
+ if (!comdat)
+ return;
+ p << "comdat";
+ if (!comdat.getValue().empty())
+ p << "(\"" << comdat.getValue() << "\")";
+}
+
+void cir::printComdatName(OpAsmPrinter &p, cir::GlobalOp op,
+ StringAttr comdat) {
+ ::printComdatName(p, comdat);
+}
+
+ParseResult cir::parseComdatName(OpAsmParser &parser, StringAttr &comdatAttr) {
+ if (parser.parseOptionalKeyword("comdat").failed())
+ return success();
+ std::string comdatKey;
+ if (succeeded(parser.parseOptionalLParen())) {
+ if (parser.parseString(&comdatKey).failed() ||
+ parser.parseRParen().failed())
+ return failure();
+ }
+ comdatAttr = parser.getBuilder().getStringAttr(comdatKey);
+ return success();
+}
+
+void cir::printGlobalOpTypeAndInitialValue(OpAsmPrinter &p, cir::GlobalOp op,
+ TypeAttr type, Attribute initAttr,
+ mlir::Region &ctorRegion,
+ mlir::Region &dtorRegion) {
+ auto printType = [&]() { p << ": " << type; };
+ // Aliases are definitions but they have no initial value or ctor/dtor; the
+ // assembly prints them like declarations (`: type`).
+ if (op.isDeclaration() || op.getAliasee()) {
+ printType();
+ return;
+ }
+
+ p << "= ";
+ if (initAttr)
+ printConstant(p, initAttr);
+
+ if (!ctorRegion.empty()) {
+ p << " ctor ";
+ p.printRegion(ctorRegion,
+ /*printEntryBlockArgs=*/false,
+ /*printBlockTerminators=*/false);
+ }
+
+ if (!dtorRegion.empty()) {
+ p << " dtor ";
+ p.printRegion(dtorRegion,
+ /*printEntryBlockArgs=*/false,
+ /*printBlockTerminators=*/false);
+ }
+}
+
+ParseResult cir::parseGlobalOpTypeAndInitialValue(OpAsmParser &parser,
+ TypeAttr &typeAttr,
+ Attribute &initialValueAttr,
+ mlir::Region &ctorRegion,
+ mlir::Region &dtorRegion) {
+ mlir::Type opTy;
+ if (parser.parseOptionalEqual().failed()) {
+ // Absence of equal means a declaration, so we need to parse the type.
+ // cir.global @a : !cir.int<s, 32>
+ if (parser.parseColonType(opTy))
+ return failure();
+ } else {
+ // Parse constant with initializer, examples:
+ // cir.global @y = 3.400000e+00 : f32
+ // cir.global @rgb = #cir.const_array<[...] : !cir.array<!s8i x 3>>
+ if (parseConstantValue(parser, initialValueAttr).failed())
+ return failure();
+
+ assert(mlir::isa<mlir::TypedAttr>(initialValueAttr) &&
+ "Non-typed attrs shouldn't appear here.");
+ opTy = mlir::cast<mlir::TypedAttr>(initialValueAttr).getType();
+
+ // Parse constructor, example:
+ // cir.global @rgb = #cir.zero : type ctor { ... }
+ if (!parser.parseOptionalKeyword("ctor")) {
+ auto parseLoc = parser.getCurrentLocation();
+ if (parser.parseRegion(ctorRegion, /*arguments=*/{}, /*argTypes=*/{}))
+ return failure();
+ if (ensureRegionTerm(parser, ctorRegion, parseLoc).failed())
+ return failure();
+ }
+
+ // Parse destructor, example:
+ // dtor { ... }
+ if (!parser.parseOptionalKeyword("dtor")) {
+ auto parseLoc = parser.getCurrentLocation();
+ if (parser.parseRegion(dtorRegion, /*arguments=*/{}, /*argTypes=*/{}))
+ return failure();
+ if (ensureRegionTerm(parser, dtorRegion, parseLoc).failed())
+ return failure();
+ }
+ }
+
+ typeAttr = TypeAttr::get(opTy);
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// GetGlobalOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult
+cir::GetGlobalOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
+ // Verify that the result type underlying pointer type matches the type of
+ // the referenced cir.global or cir.func op.
+ mlir::Operation *op =
+ symbolTable.lookupNearestSymbolFrom(*this, getNameAttr());
+ if (op == nullptr || !(isa<GlobalOp>(op) || isa<FuncOp>(op)))
+ return emitOpError("'")
+ << getName()
+ << "' does not reference a valid cir.global or cir.func";
+
+ mlir::Type symTy;
+ mlir::ptr::MemorySpaceAttrInterface symAddrSpaceAttr{};
+ if (auto g = dyn_cast<GlobalOp>(op)) {
+ symTy = g.getSymType();
+ symAddrSpaceAttr = g.getAddrSpaceAttr();
+ // Verify that for thread local global access, the global needs to
+ // be marked with tls bits.
+ if (getTls() && !g.getTlsModel())
+ return emitOpError("access to global not marked thread local");
+
+ // Verify that the static_local attribute on GetGlobalOp matches the
+ // dynamic_init_guard attribute on GlobalOp. GetGlobalOp uses a UnitAttr,
+ // GlobalOp uses DynamicInitGuardAttr. Both should be present, or neither.
+ bool getGlobalIsStaticLocal = getStaticLocal();
+ bool globalIsStaticLocal =
+ g.getDynamicInitGuard().has_value() &&
+ (!g.getDynamicInitInfo() || g.getDynamicInitInfo()->getLocal());
+
+ if (getGlobalIsStaticLocal != globalIsStaticLocal &&
+ !getOperation()->getParentOfType<cir::GlobalOp>())
+ return emitOpError("static_local attribute mismatch");
+ } else if (auto f = dyn_cast<FuncOp>(op)) {
+ symTy = f.getFunctionType();
+ } else {
+ llvm_unreachable("Unexpected operation for GetGlobalOp");
+ }
+
+ auto resultType = dyn_cast<PointerType>(getAddr().getType());
+ if (!resultType || symTy != resultType.getPointee())
+ return emitOpError("result type pointee type '")
+ << resultType.getPointee() << "' does not match type " << symTy
+ << " of the global @" << getName();
+
+ // A function address may also be taken in the program address space.
+ auto targetAS =
+ dyn_cast_if_present<TargetAddressSpaceAttr>(resultType.getAddrSpace());
+ bool isProgramAS =
+ isa<FuncOp>(op) && targetAS &&
+ targetAS.getValue() ==
+ CIRDataLayout(getOperation()->getParentOfType<ModuleOp>())
+ .getProgramAddressSpace();
+ if (!isProgramAS && symAddrSpaceAttr != resultType.getAddrSpace()) {
+ return emitOpError()
+ << "result type address space does not match the address "
+ "space of the global @"
+ << getName();
+ }
+
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// FuncOp
+//===----------------------------------------------------------------------===//
+
+/// Returns the name used for the linkage attribute. This *must* correspond to
+/// the name of the attribute in ODS.
+static llvm::StringRef getLinkageAttrNameString() { return "linkage"; }
+
+/// Declared cir.func attributes with no dedicated syntax, which print in and
+/// parse from the attribute dictionary instead.
+static bool isFuncAttrInDict(llvm::StringRef name) {
+ return name == CIRDialect::getMemoryEffectsAttrName() ||
+ name == CIRDialect::getUwtableAttrName() ||
+ name == CIRDialect::getFramePointerAttrName();
+}
+
+void cir::FuncOp::build(OpBuilder &builder, OperationState &result,
+ StringRef name, FuncType type,
+ GlobalLinkageKind linkage, CallingConv callingConv) {
+ result.addRegion();
+ result.addAttribute(getSymNameAttrName(result.name),
+ builder.getStringAttr(name));
+ result.addAttribute(getFunctionTypeAttrName(result.name),
+ TypeAttr::get(type));
+ result.addAttribute(
+ getLinkageAttrNameString(),
+ GlobalLinkageKindAttr::get(builder.getContext(), linkage));
+ result.addAttribute(getCallingConvAttrName(result.name),
+ CallingConvAttr::get(builder.getContext(), callingConv));
+ // A function starts out written in source-level types, and CallConvLowering
+ // rewrites it to the target ABI later.
+ result.addAttribute(getRequiresCallConvLoweringAttrName(result.name),
+ builder.getUnitAttr());
+}
+
+ParseResult cir::FuncOp::parse(OpAsmParser &parser, OperationState &state) {
+ llvm::SMLoc loc = parser.getCurrentLocation();
+ mlir::Builder &builder = parser.getBuilder();
+
+ mlir::StringAttr builtinNameAttr = getBuiltinAttrName(state.name);
+ mlir::StringAttr coroutineKindNameAttr = getCoroutineKindAttrName(state.name);
+ mlir::StringAttr inlineKindNameAttr = getInlineKindAttrName(state.name);
+ mlir::StringAttr lambdaNameAttr = getLambdaAttrName(state.name);
+ mlir::StringAttr noProtoNameAttr = getNoProtoAttrName(state.name);
+ mlir::StringAttr requiresCallConvLoweringNameAttr =
+ getRequiresCallConvLoweringAttrName(state.name);
+ mlir::StringAttr comdatNameAttr = getComdatAttrName(state.name);
+ mlir::StringAttr alignmentNameAttr = getAlignmentAttrName(state.name);
+ mlir::StringAttr preferredAlignmentNameAttr =
+ getPreferredAlignmentAttrName(state.name);
+ mlir::StringAttr visNameAttr = getSymVisibilityAttrName(state.name);
+ mlir::StringAttr dsoLocalNameAttr = getDsoLocalAttrName(state.name);
+ mlir::StringAttr funcInfoNameAttr = getFuncInfoAttrName(state.name);
+
+ if (::mlir::succeeded(parser.parseOptionalKeyword(builtinNameAttr.strref())))
+ state.addAttribute(builtinNameAttr, parser.getBuilder().getUnitAttr());
+
+ // Parse optional coroutine kind attribute
+ {
+ static constexpr llvm::StringRef keywords[] = {"coroutine",
+ "flattened_coroutine"};
+ llvm::StringRef keyword;
+ if (parser.parseOptionalKeyword(&keyword, keywords).succeeded())
+ state.addAttribute(
+ coroutineKindNameAttr,
+ cir::CoroutineKindAttr::get(builder.getContext(),
+ *cir::symbolizeCoroutineKind(keyword)));
+ }
+
+ // Parse optional inline kind attribute
+ cir::InlineKindAttr inlineKindAttr;
+ if (failed(parseInlineKindAttr(parser, inlineKindAttr)))
+ return failure();
+ if (inlineKindAttr)
+ state.addAttribute(inlineKindNameAttr, inlineKindAttr);
+
+ if (::mlir::succeeded(parser.parseOptionalKeyword(lambdaNameAttr.strref())))
+ state.addAttribute(lambdaNameAttr, parser.getBuilder().getUnitAttr());
+ if (parser.parseOptionalKeyword(noProtoNameAttr).succeeded())
+ state.addAttribute(noProtoNameAttr, parser.getBuilder().getUnitAttr());
+
+ {
+ StringAttr comdatAttr;
+ if (parseComdatName(parser, comdatAttr).failed())
+ return failure();
+ if (comdatAttr)
+ state.addAttribute(comdatNameAttr, comdatAttr);
+ }
+
+ auto parseAlignmentBody = [&](int64_t &value) {
+ if (parser.parseLParen().failed() || parser.parseInteger(value).failed() ||
+ parser.parseRParen().failed())
+ return failure();
+
+ if (value <= 0)
+ return static_cast<LogicalResult>(parser.emitError(
+ loc, "function alignment must be a positive integer"));
+
+ return success();
+ };
+
+ if (parser.parseOptionalKeyword(alignmentNameAttr).succeeded()) {
+ int64_t value;
+ if (parseAlignmentBody(value).failed())
+ return failure();
+ state.addAttribute(alignmentNameAttr, builder.getI64IntegerAttr(value));
+ }
+
+ if (parser.parseOptionalKeyword(preferredAlignmentNameAttr).succeeded()) {
+ int64_t value;
+ if (parseAlignmentBody(value).failed())
+ return failure();
+ state.addAttribute(preferredAlignmentNameAttr,
+ builder.getI64IntegerAttr(value));
+ }
+
+ // Default to external linkage if no keyword is provided.
+ state.addAttribute(getLinkageAttrNameString(),
+ GlobalLinkageKindAttr::get(
+ parser.getContext(),
+ parseOptionalCIRKeyword<GlobalLinkageKind>(
+ parser, GlobalLinkageKind::ExternalLinkage)));
+
+ ::llvm::StringRef visAttrStr;
+ if (parser.parseOptionalKeyword(&visAttrStr, {"private", "public", "nested"})
+ .succeeded()) {
+ state.addAttribute(visNameAttr,
+ parser.getBuilder().getStringAttr(visAttrStr));
+ }
+
+ state.getOrAddProperties<cir::FuncOp::Properties>().global_visibility =
+ parseOptionalCIRKeyword(parser, cir::VisibilityKind::Default);
+
+ if (parser.parseOptionalKeyword(dsoLocalNameAttr).succeeded())
+ state.addAttribute(dsoLocalNameAttr, parser.getBuilder().getUnitAttr());
+
+ StringAttr nameAttr;
+ if (parser.parseSymbolName(nameAttr, getSymNameAttrName(state.name),
+ state.attributes))
+ return failure();
+ llvm::SmallVector<OpAsmParser::Argument, 8> arguments;
+ llvm::SmallVector<mlir::Type> resultTypes;
+ llvm::SmallVector<DictionaryAttr> resultAttrs;
+ bool isVariadic = false;
+ if (function_interface_impl::parseFunctionSignatureWithArguments(
+ parser, /*allowVariadic=*/true, arguments, isVariadic, resultTypes,
+ resultAttrs))
+ return failure();
+ llvm::SmallVector<mlir::Type> argTypes;
+ llvm::SmallVector<mlir::Attribute> argAttrs;
+ bool argAttrsEmpty = true;
+ for (OpAsmParser::Argument &arg : arguments) {
+ argTypes.push_back(arg.type);
+ // Add the 'empty' attribute anyway to make sure the arity matches, but we
+ // only want to 'set' the attribute at the top level if there is SOME data
+ // along the way.
+ argAttrs.push_back(arg.attrs);
+ if (arg.attrs)
+ argAttrsEmpty = false;
+ }
+
+ // These should be in sync anyway, but test both of them anyway.
+ if (resultTypes.size() > 1 || resultAttrs.size() > 1)
+ return parser.emitError(
+ loc, "functions with multiple return types are not supported");
+
+ mlir::Type returnType =
+ (resultTypes.empty() ? cir::VoidType::get(builder.getContext())
+ : resultTypes.front());
+
+ cir::FuncType fnType =
+ cir::FuncType::getChecked([&]() { return parser.emitError(loc); },
+ argTypes, returnType, isVariadic);
+ if (!fnType)
+ return failure();
+
+ state.addAttribute(getFunctionTypeAttrName(state.name),
+ TypeAttr::get(fnType));
+
+ if (!resultAttrs.empty() && resultAttrs[0])
+ state.addAttribute(
+ getResAttrsAttrName(state.name),
+ mlir::ArrayAttr::get(parser.getContext(), {resultAttrs[0]}));
+
+ if (!argAttrsEmpty)
+ state.addAttribute(getArgAttrsAttrName(state.name),
+ mlir::ArrayAttr::get(parser.getContext(), argAttrs));
+
+ bool hasAlias = false;
+ mlir::StringAttr aliaseeNameAttr = getAliaseeAttrName(state.name);
+ if (parser.parseOptionalKeyword("alias").succeeded()) {
+ if (parser.parseLParen().failed())
+ return failure();
+ mlir::StringAttr aliaseeAttr;
+ if (parser.parseOptionalSymbolName(aliaseeAttr).failed())
+ return failure();
+ state.addAttribute(aliaseeNameAttr, FlatSymbolRefAttr::get(aliaseeAttr));
+ if (parser.parseRParen().failed())
+ return failure();
+ hasAlias = true;
+ }
+
+ mlir::StringAttr personalityNameAttr = getPersonalityAttrName(state.name);
+ if (parser.parseOptionalKeyword("personality").succeeded()) {
+ if (parser.parseLParen().failed())
+ return failure();
+ mlir::StringAttr personalityAttr;
+ if (parser.parseOptionalSymbolName(personalityAttr).failed())
+ return failure();
+ state.addAttribute(personalityNameAttr,
+ FlatSymbolRefAttr::get(personalityAttr));
+ if (parser.parseRParen().failed())
+ return failure();
+ }
+
+ // Default to C calling convention if no keyword is provided.
+ mlir::StringAttr callConvNameAttr = getCallingConvAttrName(state.name);
+ cir::CallingConv callConv = cir::CallingConv::C;
+ if (parser.parseOptionalKeyword("cc").succeeded()) {
+ if (parser.parseLParen().failed())
+ return failure();
+ if (parseCIRKeyword<cir::CallingConv>(parser, callConv).failed())
+ return parser.emitError(loc) << "unknown calling convention";
+ if (parser.parseRParen().failed())
+ return failure();
+ }
+ state.addAttribute(callConvNameAttr,
+ cir::CallingConvAttr::get(parser.getContext(), callConv));
+
+ auto parseGlobalDtorCtor =
+ [&](StringRef keyword,
+ llvm::function_ref<void(std::optional<int> prio)> createAttr)
+ -> mlir::LogicalResult {
+ if (mlir::succeeded(parser.parseOptionalKeyword(keyword))) {
+ std::optional<int> priority;
+ if (mlir::succeeded(parser.parseOptionalLParen())) {
+ auto parsedPriority = mlir::FieldParser<int>::parse(parser);
+ if (mlir::failed(parsedPriority))
+ return parser.emitError(parser.getCurrentLocation(),
+ "failed to parse 'priority', of type 'int'");
+ priority = parsedPriority.value_or(int());
+ // Parse literal ')'
+ if (parser.parseRParen())
+ return failure();
+ }
+ createAttr(priority);
+ }
+ return success();
+ };
+
+ // Parse the func_info attribute
+ if (parser.parseOptionalKeyword("func_info").succeeded()) {
+ if (parser.parseLess().failed())
+ return failure();
+
+ llvm::SMLoc attrLoc = parser.getCurrentLocation();
+ mlir::Attribute attr;
+ if (parser.parseAttribute(attr).failed())
+ return failure();
+ if (!mlir::isa<cir::CXXCtorAttr, cir::CXXDtorAttr, cir::CXXAssignAttr,
+ cir::FuncIdentityAttr>(attr))
+ return parser.emitError(attrLoc,
+ "expected a function info attribute, got ")
+ << attr;
+ state.addAttribute(funcInfoNameAttr, attr);
+
+ if (parser.parseGreater().failed())
+ return failure();
+ }
+
+ if (parseGlobalDtorCtor("global_ctor", [&](std::optional<int> priority) {
+ mlir::IntegerAttr globalCtorPriorityAttr =
+ builder.getI32IntegerAttr(priority.value_or(65535));
+ state.addAttribute(getGlobalCtorPriorityAttrName(state.name),
+ globalCtorPriorityAttr);
+ }).failed())
+ return failure();
+
+ if (parseGlobalDtorCtor("global_dtor", [&](std::optional<int> priority) {
+ mlir::IntegerAttr globalDtorPriorityAttr =
+ builder.getI32IntegerAttr(priority.value_or(65535));
+ state.addAttribute(getGlobalDtorPriorityAttrName(state.name),
+ globalDtorPriorityAttr);
+ }).failed())
+ return failure();
+
+ // Parse optional annotations attribute (an ArrayAttr of AnnotationAttr).
+ mlir::StringAttr annotationsNameAttr = getAnnotationsAttrName(state.name);
+ mlir::ArrayAttr annotationsAttr;
+ if (parser.parseOptionalAttribute(annotationsAttr).has_value() &&
+ annotationsAttr)
+ state.addAttribute(annotationsNameAttr, annotationsAttr);
+
+ if (parser.parseOptionalKeyword(requiresCallConvLoweringNameAttr).succeeded())
+ state.addAttribute(requiresCallConvLoweringNameAttr,
+ parser.getBuilder().getUnitAttr());
+
+ // Parse the rest of the attributes.
+ NamedAttrList parsedAttrs;
+ llvm::SMLoc attrsLoc = parser.getCurrentLocation();
+ if (parser.parseOptionalAttrDictWithKeyword(parsedAttrs))
+ return failure();
+
+ // Every other declared attribute has dedicated syntax above.
+ for (StringRef disallowed : cir::FuncOp::getAttributeNames()) {
+ if (isFuncAttrInDict(disallowed))
+ continue;
+ if (parsedAttrs.get(disallowed))
+ return parser.emitError(loc, "attribute '")
+ << disallowed
+ << "' should not be specified in the explicit attribute list";
+ }
+
+ if (checkEffectAttrKinds(parser, attrsLoc, parsedAttrs).failed())
+ return failure();
+
+ state.attributes.append(parsedAttrs);
+
+ // Parse the optional function body.
+ auto *body = state.addRegion();
+ OptionalParseResult parseResult = parser.parseOptionalRegion(
+ *body, arguments, /*enableNameShadowing=*/false);
+ if (parseResult.has_value()) {
+ if (hasAlias)
+ return parser.emitError(loc, "function alias shall not have a body");
+ if (failed(*parseResult))
+ return failure();
+ // Function body was parsed, make sure its not empty.
+ if (body->empty())
+ return parser.emitError(loc, "expected non-empty function body");
+ }
+
+ return success();
+}
+
+// This function corresponds to `llvm::GlobalValue::isDeclaration` and should
+// have a similar implementation. We don't currently ifuncs or materializable
+// functions, but those should be handled here as they are implemented.
+bool cir::FuncOp::isDeclaration() {
+ assert(!cir::MissingFeatures::supportIFuncAttr());
+
+ std::optional<StringRef> aliasee = getAliasee();
+ if (!aliasee)
+ return getFunctionBody().empty();
+
+ // Aliases are always definitions.
+ return false;
+}
+
+bool cir::FuncOp::isCXXSpecialMemberFunction() {
+ // The func_info union can grow forms that are not special members, so the
+ // check names the concrete forms rather than testing for presence.
+ mlir::Attribute attr = getFuncInfoAttr();
+ return attr && mlir::isa<CXXCtorAttr, CXXDtorAttr, CXXAssignAttr>(attr);
+}
+
+bool cir::FuncOp::isCxxConstructor() {
+ auto attr = getFuncInfoAttr();
+ return attr && dyn_cast<CXXCtorAttr>(attr);
+}
+
+bool cir::FuncOp::isCxxDestructor() {
+ auto attr = getFuncInfoAttr();
+ return attr && dyn_cast<CXXDtorAttr>(attr);
+}
+
+bool cir::FuncOp::isCxxSpecialAssignment() {
+ auto attr = getFuncInfoAttr();
+ return attr && dyn_cast<CXXAssignAttr>(attr);
+}
+
+std::optional<CtorKind> cir::FuncOp::getCxxConstructorKind() {
+ mlir::Attribute attr = getFuncInfoAttr();
+ if (attr) {
+ if (auto ctor = dyn_cast<CXXCtorAttr>(attr))
+ return ctor.getCtorKind();
+ }
+ return std::nullopt;
+}
+
+std::optional<AssignKind> cir::FuncOp::getCxxSpecialAssignKind() {
+ mlir::Attribute attr = getFuncInfoAttr();
+ if (attr) {
+ if (auto assign = dyn_cast<CXXAssignAttr>(attr))
+ return assign.getAssignKind();
+ }
+ return std::nullopt;
+}
+
+bool cir::FuncOp::isCxxTrivialMemberFunction() {
+ mlir::Attribute attr = getFuncInfoAttr();
+ if (attr) {
+ if (auto ctor = dyn_cast<CXXCtorAttr>(attr))
+ return ctor.getIsTrivial();
+ if (auto dtor = dyn_cast<CXXDtorAttr>(attr))
+ return dtor.getIsTrivial();
+ if (auto assign = dyn_cast<CXXAssignAttr>(attr))
+ return assign.getIsTrivial();
+ }
+ return false;
+}
+
+mlir::Region *cir::FuncOp::getCallableRegion() {
+ // Declarations and aliases have no body to analyze or inline. Returning the
+ // empty region would make interprocedural analyses (e.g. SCCP) treat calls
+ // to them as having no returns, leaving their results uninitialized.
+ if (getBody().empty())
+ return nullptr;
+ return &getBody();
+}
+
+void cir::FuncOp::print(OpAsmPrinter &p) {
+ if (getBuiltin())
+ p << " builtin";
+
+ if (std::optional<cir::CoroutineKind> coroKind = getCoroutineKind())
+ p << ' ' << cir::stringifyCoroutineKind(*coroKind);
+
+ printInlineKindAttr(p, getInlineKindAttr());
+
+ if (getLambda())
+ p << " lambda";
+
+ if (getNoProto())
+ p << " no_proto";
+
+ if (getComdatAttr()) {
+ p << ' ';
+ ::printComdatName(p, getComdatAttr());
+ }
+
+ if (getAlignment())
+ p << " alignment(" << *getAlignment() << ')';
+
+ if (getPreferredAlignment())
+ p << " preferred_alignment(" << *getPreferredAlignment() << ')';
+
+ if (getLinkage() != GlobalLinkageKind::ExternalLinkage)
+ p << ' ' << stringifyGlobalLinkageKind(getLinkage());
+
+ mlir::SymbolTable::Visibility vis = getVisibility();
+ if (vis != mlir::SymbolTable::Visibility::Public)
+ p << ' ' << vis;
+
+ if (getGlobalVisibility() != cir::VisibilityKind::Default)
+ p << ' ' << stringifyVisibilityKind(getGlobalVisibility());
+
+ if (getDsoLocal())
+ p << " dso_local";
+
+ p << ' ';
+ p.printSymbolName(getSymName());
+ cir::FuncType fnType = getFunctionType();
+ function_interface_impl::printFunctionSignature(
+ p, *this, fnType.getInputs(), fnType.isVarArg(), fnType.getReturnTypes());
+
+ if (std::optional<StringRef> aliaseeName = getAliasee()) {
+ p << " alias(";
+ p.printSymbolName(*aliaseeName);
+ p << ")";
+ }
+
+ if (getCallingConv() != cir::CallingConv::C) {
+ p << " cc(";
+ p << stringifyCallingConv(getCallingConv());
+ p << ")";
+ }
+
+ if (std::optional<StringRef> personalityName = getPersonality()) {
+ p << " personality(";
+ p.printSymbolName(*personalityName);
+ p << ")";
+ }
+
+ if (mlir::Attribute funcInfo = getFuncInfoAttr()) {
+ p << " func_info<";
+ p.printAttribute(funcInfo);
+ p << '>';
+ }
+
+ if (auto globalCtorPriority = getGlobalCtorPriority()) {
+ p << " global_ctor";
+ if (globalCtorPriority.value() != 65535)
+ p << "(" << globalCtorPriority.value() << ")";
+ }
+
+ if (auto globalDtorPriority = getGlobalDtorPriority()) {
+ p << " global_dtor";
+ if (globalDtorPriority.value() != 65535)
+ p << "(" << globalDtorPriority.value() << ")";
+ }
+
+ if (mlir::ArrayAttr annotations = getAnnotationsAttr()) {
+ p << ' ';
+ p.printAttribute(annotations);
+ }
+
+ if (getRequiresCallConvLowering())
+ p << ' ' << getRequiresCallConvLoweringAttrName().getValue();
+
+ // Every other declared attribute is printed by the syntax above.
+ llvm::SmallVector<llvm::StringRef> elidedAttrs;
+ for (llvm::StringRef name : cir::FuncOp::getAttributeNames())
+ if (!isFuncAttrInDict(name))
+ elidedAttrs.push_back(name);
+ function_interface_impl::printFunctionAttributes(p, *this, elidedAttrs);
+
+ // Print the body if this is not an external function.
+ Region &body = getOperation()->getRegion(0);
+ if (!body.empty()) {
+ p << ' ';
+ p.printRegion(body, /*printEntryBlockArgs=*/false,
+ /*printBlockTerminators=*/true);
+ }
+}
+
+mlir::LogicalResult cir::FuncOp::verify() {
+
+ if (!isDeclaration() && getCoroutineKind() == cir::CoroutineKind::Coroutine) {
+ int coroutineOpCount = 0;
+ this->walk([&](Operation *op) {
+ if (isa<CoroutineOp>(op)) {
+ coroutineOpCount++;
+ if (coroutineOpCount > 1) {
+ return mlir::WalkResult::interrupt();
+ }
+ }
+ return mlir::WalkResult::advance();
+ });
+ if (coroutineOpCount != 1)
+ return emitOpError()
+ << "coroutine function must have exactly one cir.coroutine op";
+ }
+
+ llvm::SmallSet<llvm::StringRef, 16> labels;
+ llvm::SmallSet<llvm::StringRef, 16> gotos;
+ llvm::SmallSet<llvm::StringRef, 16> blockAddresses;
+ bool invalidBlockAddress = false;
+ getOperation()->walk([&](mlir::Operation *op) {
+ if (auto lab = dyn_cast<cir::LabelOp>(op)) {
+ labels.insert(lab.getLabel());
+ } else if (auto goTo = dyn_cast<cir::GotoOp>(op)) {
+ gotos.insert(goTo.getLabel());
+ } else if (auto blkAdd = dyn_cast<cir::BlockAddressOp>(op)) {
+ if (blkAdd.getBlockAddrInfoAttr().getFunc().getAttr() != getSymName()) {
+ // Stop the walk early, no need to continue
+ invalidBlockAddress = true;
+ return mlir::WalkResult::interrupt();
+ }
+ blockAddresses.insert(blkAdd.getBlockAddrInfoAttr().getLabel());
+ }
+ return mlir::WalkResult::advance();
+ });
+
+ if (invalidBlockAddress)
+ return emitOpError() << "blockaddress references a different function";
+
+ llvm::SmallSet<llvm::StringRef, 16> mismatched;
+ if (!labels.empty() || !gotos.empty()) {
+ mismatched = llvm::set_difference(gotos, labels);
+
+ if (!mismatched.empty())
+ return emitOpError() << "goto/label mismatch";
+ }
+
+ mismatched.clear();
+
+ if (!labels.empty() || !blockAddresses.empty()) {
+ mismatched = llvm::set_difference(blockAddresses, labels);
+
+ if (!mismatched.empty())
+ return emitOpError()
+ << "expects an existing label target in the referenced function";
+ }
+
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// TableGen'd op method definitions
+//===----------------------------------------------------------------------===//
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsSymbols.cpp.inc"
diff --git a/clang/lib/CIR/Dialect/IR/CIROpsVectorMatrix.cpp b/clang/lib/CIR/Dialect/IR/CIROpsVectorMatrix.cpp
new file mode 100644
index 00000000000000..e67ab674a73731
--- /dev/null
+++ b/clang/lib/CIR/Dialect/IR/CIROpsVectorMatrix.cpp
@@ -0,0 +1,426 @@
+//===- CIROpsVectorMatrix.cpp - CIR ops in partition `VectorMatrix` -------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the CIR ops in partition `VectorMatrix`: the
+// hand-written parts first, then the TableGen-generated definitions.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/CIR/Dialect/IR/CIROpsVectorMatrix.h"
+#include "clang/CIR/Dialect/IR/CIROpsCore.h"
+
+#include "CIROpDefinitions.h"
+
+#include "mlir/IR/DialectImplementation.h"
+
+using namespace mlir;
+using namespace cir;
+
+//===----------------------------------------------------------------------===//
+// VecCreateOp
+//===----------------------------------------------------------------------===//
+
+OpFoldResult cir::VecCreateOp::fold(FoldAdaptor adaptor) {
+ if (llvm::any_of(getElements(), [](mlir::Value value) {
+ return !value.getDefiningOp<cir::ConstantOp>();
+ }))
+ return {};
+
+ return cir::ConstVectorAttr::get(
+ getType(), mlir::ArrayAttr::get(getContext(), adaptor.getElements()));
+}
+
+LogicalResult cir::VecCreateOp::verify() {
+ // Verify that the number of arguments matches the number of elements in the
+ // vector, and that the type of all the arguments matches the type of the
+ // elements in the vector.
+ const cir::VectorType vecTy = getType();
+ if (vecTy.isScalable())
+ return emitOpError() << "scalable vectors are not supported";
+
+ if (getElements().size() != vecTy.getFixedSize()) {
+ return emitOpError() << "operand count of " << getElements().size()
+ << " doesn't match vector type " << vecTy
+ << " element count of " << vecTy.getFixedSize();
+ }
+
+ const mlir::Type elementType = vecTy.getElementType();
+ for (const mlir::Value element : getElements()) {
+ if (element.getType() != elementType) {
+ return emitOpError() << "operand type " << element.getType()
+ << " doesn't match vector element type "
+ << elementType;
+ }
+ }
+
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// VecExtractOp
+//===----------------------------------------------------------------------===//
+
+OpFoldResult cir::VecExtractOp::fold(FoldAdaptor adaptor) {
+ const auto vectorAttr =
+ llvm::dyn_cast_if_present<cir::ConstVectorAttr>(adaptor.getVec());
+ if (!vectorAttr)
+ return {};
+
+ const auto indexAttr =
+ llvm::dyn_cast_if_present<cir::IntAttr>(adaptor.getIndex());
+ if (!indexAttr)
+ return {};
+
+ const mlir::ArrayAttr elements = vectorAttr.getElts();
+ const uint64_t index = indexAttr.getUInt();
+ if (index >= elements.size())
+ return {};
+
+ return elements[index];
+}
+
+//===----------------------------------------------------------------------===//
+// VecCmpOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::VecCmpOp::verify() {
+ if (getFenvAttr() && !cir::isFPOrVectorOfFPType(getLhs().getType()))
+ return emitOpError()
+ << "'fenv' is only valid for floating-point comparisons";
+ return success();
+}
+
+OpFoldResult cir::VecCmpOp::fold(FoldAdaptor adaptor) {
+ // Do not fold when fenv is present.
+ // TODO(cir): This is overly conservative. We could fold comparisons as long
+ // as we can prove that the operation wouldn't raise exceptions or
+ // when the fenv attribute does not require strict exception
+ // semantics.
+ if (getFenvAttr())
+ return {};
+
+ auto lhsVecAttr =
+ mlir::dyn_cast_if_present<cir::ConstVectorAttr>(adaptor.getLhs());
+ auto rhsVecAttr =
+ mlir::dyn_cast_if_present<cir::ConstVectorAttr>(adaptor.getRhs());
+ if (!lhsVecAttr || !rhsVecAttr)
+ return {};
+
+ mlir::Type inputElemTy =
+ mlir::cast<cir::VectorType>(lhsVecAttr.getType()).getElementType();
+ if (!isAnyIntegerOrFloatingPointType(inputElemTy))
+ return {};
+
+ cir::CmpOpKind opKind = adaptor.getKind();
+ mlir::ArrayAttr lhsVecElhs = lhsVecAttr.getElts();
+ mlir::ArrayAttr rhsVecElhs = rhsVecAttr.getElts();
+ uint64_t vecSize = lhsVecElhs.size();
+
+ SmallVector<mlir::Attribute, 16> elements(vecSize);
+ bool isIntAttr = vecSize && mlir::isa<cir::IntAttr>(lhsVecElhs[0]);
+ bool isUnsignedInt =
+ isIntAttr && mlir::cast<cir::IntType>(inputElemTy).isUnsigned();
+ for (uint64_t i = 0; i < vecSize; i++) {
+ mlir::Attribute lhsAttr = lhsVecElhs[i];
+ mlir::Attribute rhsAttr = rhsVecElhs[i];
+ bool cmpResult = false;
+ switch (opKind) {
+ case cir::CmpOpKind::lt: {
+ if (isIntAttr) {
+ if (isUnsignedInt)
+ cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getUInt() <
+ mlir::cast<cir::IntAttr>(rhsAttr).getUInt();
+ else
+ cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getSInt() <
+ mlir::cast<cir::IntAttr>(rhsAttr).getSInt();
+ } else {
+ cmpResult = mlir::cast<cir::FPAttr>(lhsAttr).getValue() <
+ mlir::cast<cir::FPAttr>(rhsAttr).getValue();
+ }
+ break;
+ }
+ case cir::CmpOpKind::le: {
+ if (isIntAttr) {
+ if (isUnsignedInt)
+ cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getUInt() <=
+ mlir::cast<cir::IntAttr>(rhsAttr).getUInt();
+ else
+ cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getSInt() <=
+ mlir::cast<cir::IntAttr>(rhsAttr).getSInt();
+ } else {
+ cmpResult = mlir::cast<cir::FPAttr>(lhsAttr).getValue() <=
+ mlir::cast<cir::FPAttr>(rhsAttr).getValue();
+ }
+ break;
+ }
+ case cir::CmpOpKind::gt: {
+ if (isIntAttr) {
+ if (isUnsignedInt)
+ cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getUInt() >
+ mlir::cast<cir::IntAttr>(rhsAttr).getUInt();
+ else
+ cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getSInt() >
+ mlir::cast<cir::IntAttr>(rhsAttr).getSInt();
+ } else {
+ cmpResult = mlir::cast<cir::FPAttr>(lhsAttr).getValue() >
+ mlir::cast<cir::FPAttr>(rhsAttr).getValue();
+ }
+ break;
+ }
+ case cir::CmpOpKind::ge: {
+ if (isIntAttr) {
+ if (isUnsignedInt)
+ cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getUInt() >=
+ mlir::cast<cir::IntAttr>(rhsAttr).getUInt();
+ else
+ cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getSInt() >=
+ mlir::cast<cir::IntAttr>(rhsAttr).getSInt();
+ } else {
+ cmpResult = mlir::cast<cir::FPAttr>(lhsAttr).getValue() >=
+ mlir::cast<cir::FPAttr>(rhsAttr).getValue();
+ }
+ break;
+ }
+ case cir::CmpOpKind::eq: {
+ if (isIntAttr) {
+ cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getSInt() ==
+ mlir::cast<cir::IntAttr>(rhsAttr).getSInt();
+ } else {
+ cmpResult = mlir::cast<cir::FPAttr>(lhsAttr).getValue() ==
+ mlir::cast<cir::FPAttr>(rhsAttr).getValue();
+ }
+ break;
+ }
+ case cir::CmpOpKind::ne: {
+ if (isIntAttr) {
+ cmpResult = mlir::cast<cir::IntAttr>(lhsAttr).getSInt() !=
+ mlir::cast<cir::IntAttr>(rhsAttr).getSInt();
+ } else {
+ cmpResult = mlir::cast<cir::FPAttr>(lhsAttr).getValue() !=
+ mlir::cast<cir::FPAttr>(rhsAttr).getValue();
+ }
+ break;
+ }
+ case cir::CmpOpKind::one: {
+ llvm::APFloat::cmpResult cr =
+ mlir::cast<cir::FPAttr>(lhsAttr).getValue().compare(
+ mlir::cast<cir::FPAttr>(rhsAttr).getValue());
+ cmpResult =
+ cr != llvm::APFloat::cmpUnordered && cr != llvm::APFloat::cmpEqual;
+ break;
+ }
+ case cir::CmpOpKind::uno: {
+ cmpResult = mlir::cast<cir::FPAttr>(lhsAttr).getValue().compare(
+ mlir::cast<cir::FPAttr>(rhsAttr).getValue()) ==
+ llvm::APFloat::cmpUnordered;
+ break;
+ }
+ }
+
+ // A true result is all bits set (-1 in two's complement), and a false
+ // result is all bits clear. For a 1-bit element type these are the same
+ // bit pattern as 1 and 0, respectively.
+ elements[i] =
+ cir::IntAttr::get(getType().getElementType(), cmpResult ? -1LL : 0LL);
+ }
+
+ return cir::ConstVectorAttr::get(
+ getType(), mlir::ArrayAttr::get(getContext(), elements));
+}
+
+//===----------------------------------------------------------------------===//
+// VecShuffleOp
+//===----------------------------------------------------------------------===//
+
+OpFoldResult cir::VecShuffleOp::fold(FoldAdaptor adaptor) {
+ auto vec1Attr =
+ mlir::dyn_cast_if_present<cir::ConstVectorAttr>(adaptor.getVec1());
+ auto vec2Attr =
+ mlir::dyn_cast_if_present<cir::ConstVectorAttr>(adaptor.getVec2());
+ if (!vec1Attr || !vec2Attr)
+ return {};
+
+ mlir::Type vec1ElemTy =
+ mlir::cast<cir::VectorType>(vec1Attr.getType()).getElementType();
+
+ mlir::ArrayAttr vec1Elts = vec1Attr.getElts();
+ mlir::ArrayAttr vec2Elts = vec2Attr.getElts();
+ mlir::ArrayAttr indicesElts = adaptor.getIndices();
+
+ SmallVector<mlir::Attribute, 16> elements;
+ elements.reserve(indicesElts.size());
+
+ uint64_t vec1Size = vec1Elts.size();
+ for (const auto &idxAttr : indicesElts.getAsRange<cir::IntAttr>()) {
+ if (idxAttr.getSInt() == -1) {
+ elements.push_back(cir::UndefAttr::get(vec1ElemTy));
+ continue;
+ }
+
+ uint64_t idxValue = idxAttr.getUInt();
+ elements.push_back(idxValue < vec1Size ? vec1Elts[idxValue]
+ : vec2Elts[idxValue - vec1Size]);
+ }
+
+ return cir::ConstVectorAttr::get(
+ getType(), mlir::ArrayAttr::get(getContext(), elements));
+}
+
+LogicalResult cir::VecShuffleOp::verify() {
+ if (getVec1().getType().isScalable() || getVec2().getType().isScalable() ||
+ getResult().getType().isScalable())
+ return emitOpError() << "scalable vectors are not supported";
+
+ // The number of elements in the indices array must match the number of
+ // elements in the result type.
+ if (getIndices().size() != getResult().getType().getFixedSize()) {
+ return emitOpError() << ": the number of elements in " << getIndices()
+ << " and " << getResult().getType() << " don't match";
+ }
+
+ // The element types of the two input vectors and of the result type must
+ // match.
+ if (getVec1().getType().getElementType() !=
+ getResult().getType().getElementType()) {
+ return emitOpError() << ": element types of " << getVec1().getType()
+ << " and " << getResult().getType() << " don't match";
+ }
+
+ const uint64_t maxValidIndex = getVec1().getType().getFixedSize() +
+ getVec2().getType().getFixedSize() - 1;
+ if (llvm::any_of(
+ getIndices().getAsRange<cir::IntAttr>(), [&](cir::IntAttr idxAttr) {
+ return idxAttr.getSInt() != -1 && idxAttr.getUInt() > maxValidIndex;
+ })) {
+ return emitOpError() << ": index for __builtin_shufflevector must be "
+ "less than the total number of vector elements";
+ }
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// VecShuffleDynamicOp
+//===----------------------------------------------------------------------===//
+
+OpFoldResult cir::VecShuffleDynamicOp::fold(FoldAdaptor adaptor) {
+ mlir::Attribute vec = adaptor.getVec();
+ mlir::Attribute indices = adaptor.getIndices();
+ if (mlir::isa_and_nonnull<cir::ConstVectorAttr>(vec) &&
+ mlir::isa_and_nonnull<cir::ConstVectorAttr>(indices)) {
+ auto vecAttr = mlir::cast<cir::ConstVectorAttr>(vec);
+ auto indicesAttr = mlir::cast<cir::ConstVectorAttr>(indices);
+
+ mlir::ArrayAttr vecElts = vecAttr.getElts();
+ mlir::ArrayAttr indicesElts = indicesAttr.getElts();
+
+ const uint64_t numElements = vecElts.size();
+
+ SmallVector<mlir::Attribute, 16> elements;
+ elements.reserve(numElements);
+
+ const uint64_t maskBits = llvm::NextPowerOf2(numElements - 1) - 1;
+ for (const auto &idxAttr : indicesElts.getAsRange<cir::IntAttr>()) {
+ uint64_t idxValue = idxAttr.getUInt();
+ uint64_t newIdx = idxValue & maskBits;
+ elements.push_back(vecElts[newIdx]);
+ }
+
+ return cir::ConstVectorAttr::get(
+ getType(), mlir::ArrayAttr::get(getContext(), elements));
+ }
+
+ return {};
+}
+
+LogicalResult cir::VecShuffleDynamicOp::verify() {
+ // The number of elements in the two input vectors must match.
+ if (getVec().getType().getSize() !=
+ mlir::cast<cir::VectorType>(getIndices().getType()).getSize()) {
+ return emitOpError() << ": the number of elements in " << getVec().getType()
+ << " and " << getIndices().getType() << " don't match";
+ }
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// VecTernaryOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::VecTernaryOp::verify() {
+ // Verify that the condition operand has the same number of elements as the
+ // other operands. (The automatic verification already checked that all
+ // operands are vector types and that the second and third operands are the
+ // same type.)
+ if (getCond().getType().getSize() != getLhs().getType().getSize()) {
+ return emitOpError() << ": the number of elements in "
+ << getCond().getType() << " and " << getLhs().getType()
+ << " don't match";
+ }
+ return success();
+}
+
+OpFoldResult cir::VecTernaryOp::fold(FoldAdaptor adaptor) {
+ mlir::Attribute cond = adaptor.getCond();
+ mlir::Attribute lhs = adaptor.getLhs();
+ mlir::Attribute rhs = adaptor.getRhs();
+
+ if (!mlir::isa_and_nonnull<cir::ConstVectorAttr>(cond) ||
+ !mlir::isa_and_nonnull<cir::ConstVectorAttr>(lhs) ||
+ !mlir::isa_and_nonnull<cir::ConstVectorAttr>(rhs))
+ return {};
+ auto condVec = mlir::cast<cir::ConstVectorAttr>(cond);
+ auto lhsVec = mlir::cast<cir::ConstVectorAttr>(lhs);
+ auto rhsVec = mlir::cast<cir::ConstVectorAttr>(rhs);
+
+ mlir::ArrayAttr condElts = condVec.getElts();
+
+ SmallVector<mlir::Attribute, 16> elements;
+ elements.reserve(condElts.size());
+
+ for (const auto &[idx, condAttr] :
+ llvm::enumerate(condElts.getAsRange<cir::IntAttr>())) {
+ if (condAttr.getSInt()) {
+ elements.push_back(lhsVec.getElts()[idx]);
+ } else {
+ elements.push_back(rhsVec.getElts()[idx]);
+ }
+ }
+
+ cir::VectorType vecTy = getLhs().getType();
+ return cir::ConstVectorAttr::get(
+ vecTy, mlir::ArrayAttr::get(getContext(), elements));
+}
+
+//===----------------------------------------------------------------------===//
+// MatrixTransposeOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult cir::MatrixTransposeOp::verify() {
+ cir::MatrixType valueTy = getValue().getType();
+ cir::MatrixType resultTy = getResult().getType();
+
+ if ((valueTy.getElementType() != resultTy.getElementType()) ||
+ (valueTy.getNumRows() != resultTy.getNumColumns()) ||
+ (valueTy.getNumColumns() != resultTy.getNumRows())) {
+ auto expectedTy =
+ cir::MatrixType::get(valueTy.getElementType(), valueTy.getNumColumns(),
+ valueTy.getNumRows());
+ emitOpError() << "operand type " << valueTy << " expects result type of "
+ << expectedTy << " but got " << resultTy;
+ return failure();
+ }
+
+ return success();
+}
+
+//===----------------------------------------------------------------------===//
+// TableGen-generated op definitions
+//===----------------------------------------------------------------------===//
+
+#define GET_OP_CLASSES
+#include "clang/CIR/Dialect/IR/CIROpsVectorMatrix.cpp.inc"
diff --git a/clang/lib/CIR/Dialect/IR/CIRTypes.cpp b/clang/lib/CIR/Dialect/IR/CIRTypes.cpp
index 40ba3bb4989776..27dbb619a90b7c 100644
--- a/clang/lib/CIR/Dialect/IR/CIRTypes.cpp
+++ b/clang/lib/CIR/Dialect/IR/CIRTypes.cpp
@@ -19,8 +19,9 @@
#include "mlir/Support/LLVM.h"
#include "clang/Basic/AddressSpaces.h"
#include "clang/CIR/Dialect/IR/CIRAttrs.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "clang/CIR/Dialect/IR/CIROpsEnums.h"
+#include "clang/CIR/Dialect/IR/CIROpsSymbols.h"
#include "clang/CIR/Dialect/IR/CIRTypesDetails.h"
#include "clang/CIR/MissingFeatures.h"
#include "llvm/ADT/APFloat.h"
diff --git a/clang/lib/CIR/Dialect/IR/CMakeLists.txt b/clang/lib/CIR/Dialect/IR/CMakeLists.txt
index 8b2afddfb3536f..653681c6b7efa5 100644
--- a/clang/lib/CIR/Dialect/IR/CMakeLists.txt
+++ b/clang/lib/CIR/Dialect/IR/CMakeLists.txt
@@ -1,31 +1,25 @@
-set(LLVM_OPTIONAL_SOURCES
- CIROpDefinitions.cpp
- )
-
-# Compile the TableGen-generated op definitions in shards to reduce the
-# compile time of the largest CIR translation unit.
-set(LLVM_TARGET_DEFINITIONS CIROpDefinitions.cpp)
-math(EXPR CIR_LAST_SHARD_INDEX "${CLANG_CIR_OP_SHARD_COUNT} - 1")
-foreach(index RANGE ${CIR_LAST_SHARD_INDEX})
- set(CIR_SHARDED_SRC CIROpDefinitions.${index}.cpp)
- list(APPEND CIR_SHARDED_SRCS ${CIR_SHARDED_SRC})
- mlir_src_sharder(${CIR_SHARDED_SRC}
- -op-shard-index=${index})
-endforeach()
-add_public_tablegen_target(MLIRCIROpsShardGen)
-
add_clang_library(MLIRCIR
CIRAttrs.cpp
CIRDialect.cpp
CIRDialectBytecode.cpp
CIRMemorySlot.cpp
+ CIROpsArith.cpp
+ CIROpsBuiltins.cpp
+ CIROpsClassMembers.cpp
+ CIROpsComplex.cpp
+ CIROpsControlFlow.cpp
+ CIROpsCore.cpp
+ CIROpsCoroutine.cpp
+ CIROpsEH.cpp
+ CIROpsMath.cpp
+ CIROpsSymbols.cpp
+ CIROpsVectorMatrix.cpp
CIRTypes.cpp
CIRDataLayout.cpp
CIROpenCLAttrs.cpp
- ${CIR_SHARDED_SRCS}
DEPENDS
- MLIRCIROpsShardGen
+ MLIRCIROpsIncGen
MLIRCIRDialectBytecodeIncGen
MLIRCIREnumsGen
mlir-headers
diff --git a/clang/lib/CIR/Dialect/OpenACC/RegisterOpenACCExtensions.cpp b/clang/lib/CIR/Dialect/OpenACC/RegisterOpenACCExtensions.cpp
index 3dfe7aeb6b1d61..99246e12f11abf 100644
--- a/clang/lib/CIR/Dialect/OpenACC/RegisterOpenACCExtensions.cpp
+++ b/clang/lib/CIR/Dialect/OpenACC/RegisterOpenACCExtensions.cpp
@@ -11,7 +11,7 @@
//===----------------------------------------------------------------------===//
#include "clang/CIR/Dialect/OpenACC/RegisterOpenACCExtensions.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "clang/CIR/Dialect/IR/CIRTypes.h"
#include "clang/CIR/Dialect/OpenACC/CIROpenACCTypeInterfaces.h"
diff --git a/clang/lib/CIR/Dialect/Transforms/LibOpt.cpp b/clang/lib/CIR/Dialect/Transforms/LibOpt.cpp
index 71db03827d3c65..11bbe593af970d 100644
--- a/clang/lib/CIR/Dialect/Transforms/LibOpt.cpp
+++ b/clang/lib/CIR/Dialect/Transforms/LibOpt.cpp
@@ -16,7 +16,7 @@
#include "mlir/IR/Region.h"
#include "clang/Basic/Module.h"
#include "clang/CIR/Dialect/Builder/CIRBaseBuilder.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "clang/CIR/Dialect/Passes.h"
#include "llvm/ADT/SmallVector.h"
#include "llvm/ADT/StringMap.h"
diff --git a/clang/lib/CIR/FrontendAction/CIRGenAction.cpp b/clang/lib/CIR/FrontendAction/CIRGenAction.cpp
index 9acaa6025fa073..17744597b9216c 100644
--- a/clang/lib/CIR/FrontendAction/CIRGenAction.cpp
+++ b/clang/lib/CIR/FrontendAction/CIRGenAction.cpp
@@ -18,7 +18,7 @@
#include "clang/Basic/DiagnosticFrontend.h"
#include "clang/CIR/CIRGenerator.h"
#include "clang/CIR/CIRToCIRPasses.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "clang/CIR/InitAllDialects.h"
#include "clang/CIR/LowerToLLVM.h"
#include "clang/CodeGen/BackendUtil.h"
diff --git a/clang/lib/CIR/Interfaces/CIRLoopOpInterface.cpp b/clang/lib/CIR/Interfaces/CIRLoopOpInterface.cpp
index 1dad9acb62e489..9e65cc3668cd11 100644
--- a/clang/lib/CIR/Interfaces/CIRLoopOpInterface.cpp
+++ b/clang/lib/CIR/Interfaces/CIRLoopOpInterface.cpp
@@ -8,7 +8,8 @@
#include "clang/CIR/Interfaces/CIRLoopOpInterface.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
+#include "clang/CIR/Dialect/IR/CIROpsControlFlow.h"
#include "clang/CIR/Interfaces/CIRLoopOpInterface.cpp.inc"
#include "llvm/Support/ErrorHandling.h"
diff --git a/clang/lib/CIR/Lowering/CIRPasses.cpp b/clang/lib/CIR/Lowering/CIRPasses.cpp
index 24276411df6f9b..9a6543eff54ea4 100644
--- a/clang/lib/CIR/Lowering/CIRPasses.cpp
+++ b/clang/lib/CIR/Lowering/CIRPasses.cpp
@@ -13,7 +13,7 @@
#include "mlir/IR/BuiltinOps.h"
#include "mlir/Pass/PassManager.h"
#include "clang/Basic/LangOptions.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "clang/CIR/Dialect/Passes.h"
#include "llvm/Support/TimeProfiler.h"
#include "llvm/TargetParser/Triple.h"
diff --git a/clang/lib/CIR/RegisterAllDialects.cpp b/clang/lib/CIR/RegisterAllDialects.cpp
index 706635049afbe4..8aa1932e780e0c 100644
--- a/clang/lib/CIR/RegisterAllDialects.cpp
+++ b/clang/lib/CIR/RegisterAllDialects.cpp
@@ -16,7 +16,7 @@
#include "mlir/IR/BuiltinDialect.h"
#include "mlir/IR/MLIRContext.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "clang/CIR/Dialect/OpenACC/RegisterOpenACCExtensions.h"
#include "clang/CIR/Dialect/OpenMP/RegisterOpenMPExtensions.h"
diff --git a/clang/tools/cir-translate/cir-translate.cpp b/clang/tools/cir-translate/cir-translate.cpp
index c581076fc16dbd..a1a26b8e0e8fed 100644
--- a/clang/tools/cir-translate/cir-translate.cpp
+++ b/clang/tools/cir-translate/cir-translate.cpp
@@ -34,7 +34,7 @@
#include "clang/Basic/DiagnosticOptions.h"
#include "clang/Basic/TargetInfo.h"
#include "clang/CIR/CIRDataLayoutSpec.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "clang/CIR/Dialect/Passes.h"
#include "clang/CIR/InitAllDialects.h"
#include "clang/CIR/LowerToLLVM.h"
diff --git a/clang/unittests/CIR/GetFloatingPointTypeTest.cpp b/clang/unittests/CIR/GetFloatingPointTypeTest.cpp
index de9f34e1c05394..c93bb951b67c1b 100644
--- a/clang/unittests/CIR/GetFloatingPointTypeTest.cpp
+++ b/clang/unittests/CIR/GetFloatingPointTypeTest.cpp
@@ -11,7 +11,7 @@
//===----------------------------------------------------------------------===//
#include "mlir/IR/MLIRContext.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "clang/CIR/Dialect/IR/CIRTypes.h"
#include "llvm/ADT/APFloat.h"
#include "gtest/gtest.h"
diff --git a/clang/unittests/CIR/IntTypeABIAlignTest.cpp b/clang/unittests/CIR/IntTypeABIAlignTest.cpp
index 5fe0dc5bd2b4bc..8179288d00117f 100644
--- a/clang/unittests/CIR/IntTypeABIAlignTest.cpp
+++ b/clang/unittests/CIR/IntTypeABIAlignTest.cpp
@@ -12,7 +12,7 @@
#include "mlir/IR/BuiltinOps.h"
#include "mlir/IR/MLIRContext.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "clang/CIR/Dialect/IR/CIRTypes.h"
#include "llvm/Support/MathExtras.h"
#include "gtest/gtest.h"
diff --git a/clang/unittests/CIR/RecordMemberKindTest.cpp b/clang/unittests/CIR/RecordMemberKindTest.cpp
index 7f6843356a1292..f26aa115faf797 100644
--- a/clang/unittests/CIR/RecordMemberKindTest.cpp
+++ b/clang/unittests/CIR/RecordMemberKindTest.cpp
@@ -13,7 +13,7 @@
#include "mlir/IR/BuiltinOps.h"
#include "mlir/IR/Diagnostics.h"
#include "mlir/IR/MLIRContext.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "clang/CIR/Dialect/IR/CIRTypes.h"
#include "gtest/gtest.h"
diff --git a/clang/unittests/CIR/RecordTypeMetadataTest.cpp b/clang/unittests/CIR/RecordTypeMetadataTest.cpp
index 3e7b0d9054970f..25ee9ac8714ffe 100644
--- a/clang/unittests/CIR/RecordTypeMetadataTest.cpp
+++ b/clang/unittests/CIR/RecordTypeMetadataTest.cpp
@@ -13,7 +13,7 @@
#include "mlir/IR/BuiltinOps.h"
#include "mlir/IR/MLIRContext.h"
#include "clang/CIR/Dialect/IR/CIRAttrs.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "clang/CIR/Dialect/IR/CIRTypes.h"
#include "gtest/gtest.h"
diff --git a/clang/unittests/CIR/RecordTypeVerifyTest.cpp b/clang/unittests/CIR/RecordTypeVerifyTest.cpp
index dbbdf23407d02b..64b549718cc74a 100644
--- a/clang/unittests/CIR/RecordTypeVerifyTest.cpp
+++ b/clang/unittests/CIR/RecordTypeVerifyTest.cpp
@@ -14,7 +14,7 @@
#include "mlir/IR/AttrTypeSubElements.h"
#include "mlir/IR/BuiltinTypes.h"
#include "mlir/IR/MLIRContext.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "clang/CIR/Dialect/IR/CIRTypes.h"
#include "gtest/gtest.h"
diff --git a/clang/unittests/CIR/UnionTypeSizeTest.cpp b/clang/unittests/CIR/UnionTypeSizeTest.cpp
index b115980aa4ab96..8a5fc5668a3028 100644
--- a/clang/unittests/CIR/UnionTypeSizeTest.cpp
+++ b/clang/unittests/CIR/UnionTypeSizeTest.cpp
@@ -12,7 +12,7 @@
#include "mlir/IR/BuiltinOps.h"
#include "mlir/IR/MLIRContext.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "clang/CIR/Dialect/IR/CIRTypes.h"
#include "gtest/gtest.h"
diff --git a/clang/unittests/CIR/VectorTypeABIAlignTest.cpp b/clang/unittests/CIR/VectorTypeABIAlignTest.cpp
index 798ea2153ad62d..abf71657508bcc 100644
--- a/clang/unittests/CIR/VectorTypeABIAlignTest.cpp
+++ b/clang/unittests/CIR/VectorTypeABIAlignTest.cpp
@@ -12,7 +12,7 @@
#include "mlir/IR/BuiltinOps.h"
#include "mlir/IR/MLIRContext.h"
-#include "clang/CIR/Dialect/IR/CIRDialect.h"
+#include "clang/CIR/Dialect/IR/CIRDialectBase.h"
#include "clang/CIR/Dialect/IR/CIRTypes.h"
#include "gtest/gtest.h"
More information about the llvm-branch-commits
mailing list