[clang] [CIR] Accept fixed-width vectors in x86_64 callconv lowering (PR #215118)
Adam Smith via cfe-commits
cfe-commits at lists.llvm.org
Thu Aug 13 15:51:11 PDT 2026
https://github.com/adams381 updated https://github.com/llvm/llvm-project/pull/215118
>From 77536bbe2747a41713fe2638f75fc4519a456968 Mon Sep 17 00:00:00 2001
From: Adam Smith <adams at nvidia.com>
Date: Sun, 9 Aug 2026 10:05:12 -0700
Subject: [PATCH 1/3] [CIR] Accept fixed-width vectors in x86_64 callconv
lowering
The CallConvLowering bridge rejects a vector in a parameter or return position,
so a function taking one fails the pass. It also never reads the AVX level,
which is what decides whether a vector wider than 128 bits reaches a register.
A vector is accepted now where the classifier and clang size it the same way,
which means a whole-byte element and a power-of-two width. Scalable vectors and
the other widths stay rejected. The module's AVX level comes from the target
ABI name, as CodeGenModule does. A classifier per level lets a target attribute
raise it for one function. An ABI older than the rule pins every function back
to the module's level. A direct call takes its callee's level, and an indirect
call the level of the function containing it.
CIRGen records target features on a definition but not on a declaration, so a
declaration carrying the attribute is classified at the module's level until
Assisted-by: Cursor / claude-opus-5
---
clang/include/clang/CIR/Dialect/Passes.h | 7 +-
clang/include/clang/CIR/Dialect/Passes.td | 4 +
.../Transforms/CallConvLoweringPass.cpp | 164 +++++++++++++----
clang/lib/CIR/Lowering/CIRPasses.cpp | 28 ++-
.../call-conv-lowering-x86_64-abi-compat.c | 62 ++++++-
.../CodeGen/call-conv-lowering-x86_64-avx.c | 165 ++++++++++++++++++
.../CIR/CodeGen/call-conv-lowering-x86_64.c | 77 ++++++++
.../abi-lowering/x86_64-aggregate-nyi.cir | 43 +++++
.../abi-lowering/x86_64-variadic-call.cir | 71 +++++---
.../Transforms/abi-lowering/x86_64-vector.cir | 75 +++++++-
10 files changed, 615 insertions(+), 81 deletions(-)
create mode 100644 clang/test/CIR/CodeGen/call-conv-lowering-x86_64-avx.c
diff --git a/clang/include/clang/CIR/Dialect/Passes.h b/clang/include/clang/CIR/Dialect/Passes.h
index 888e7b833b1cf..6c8bec15c3838 100644
--- a/clang/include/clang/CIR/Dialect/Passes.h
+++ b/clang/include/clang/CIR/Dialect/Passes.h
@@ -36,10 +36,9 @@ std::unique_ptr<Pass> createCIREHABILoweringPass();
std::unique_ptr<Pass> createCXXABILoweringPass();
std::unique_ptr<Pass> createTargetLoweringPass();
std::unique_ptr<Pass> createCallConvLoweringPass();
-std::unique_ptr<Pass>
-createCallConvLoweringPass(cir::CallConvTarget target,
- llvm::abi::X86AVXABILevel x86AvxAbiLevel,
- const llvm::abi::ABICompatInfo &x86AbiCompat);
+std::unique_ptr<Pass> createCallConvLoweringPass(
+ cir::CallConvTarget target, llvm::abi::X86AVXABILevel x86AvxAbiLevel,
+ bool x86TargetAttrAvx, const llvm::abi::ABICompatInfo &x86AbiCompat);
std::unique_ptr<Pass> createHoistAllocasPass();
std::unique_ptr<Pass> createLoweringPreparePass();
std::unique_ptr<Pass> createLoweringPreparePass(clang::ASTContext *astCtx);
diff --git a/clang/include/clang/CIR/Dialect/Passes.td b/clang/include/clang/CIR/Dialect/Passes.td
index 04ff1c92cb7f7..bc1b99da01440 100644
--- a/clang/include/clang/CIR/Dialect/Passes.td
+++ b/clang/include/clang/CIR/Dialect/Passes.td
@@ -260,6 +260,10 @@ def CallConvLowering : Pass<"cir-call-conv-lowering", "mlir::ModuleOp"> {
clEnumValN(llvm::abi::X86AVXABILevel::AVX, "avx", "AVX"),
clEnumValN(llvm::abi::X86AVXABILevel::AVX512, "avx512", "AVX512")
)}]>,
+ Option<"x86TargetAttrAvx", "x86-target-attr-avx", "bool",
+ /*default=*/"true",
+ "Let a function's own target features raise its AVX ABI level "
+ "above x86-avx-abi-level">,
];
}
diff --git a/clang/lib/CIR/Dialect/Transforms/CallConvLoweringPass.cpp b/clang/lib/CIR/Dialect/Transforms/CallConvLoweringPass.cpp
index 4e3c169f86d9d..5f25e827b9cd2 100644
--- a/clang/lib/CIR/Dialect/Transforms/CallConvLoweringPass.cpp
+++ b/clang/lib/CIR/Dialect/Transforms/CallConvLoweringPass.cpp
@@ -47,8 +47,13 @@
#include "llvm/ABI/FunctionInfo.h"
#include "llvm/ABI/TargetInfo.h"
#include "llvm/ABI/Types.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/TypeSwitch.h"
#include "llvm/IR/CallingConv.h"
+#include "llvm/Support/MathExtras.h"
+
+#include <algorithm>
+#include <array>
using namespace mlir;
using namespace mlir::abi;
@@ -68,10 +73,11 @@ namespace {
// SysV x86_64 classifier, and converts the result back into the
// dialect-agnostic mlir::abi::FunctionClassification that CIRABIRewriteContext
// consumes. Integer (including `_BitInt` up to 128 bits) / pointer / bool /
-// floating-point scalars are handled, as are struct / union / array aggregates
-// and `_Complex`. Vectors, packed or padded records, and a union no member of
-// which spans its declared size are reported NYI by classifyX86_64Function so
-// an unsupported signature fails the pass instead of being misclassified.
+// floating-point scalars are handled, as are struct / union / array aggregates,
+// `_Complex`, and a fixed-width vector whose width is a power of two. Other
+// vectors, packed or padded records, and a union no member of which spans its
+// declared size are reported NYI by classifyX86_64Function so an unsupported
+// signature fails the pass instead of being misclassified.
//===----------------------------------------------------------------------===//
/// Whether a struct's declared argument-passing kind (from the module's
@@ -103,8 +109,9 @@ static llvm::Align recordDeclaredAlign(ModuleOp modOp, cir::RecordType recTy,
/// bits (including `_BitInt` and `__int128`), pointer, bool, void, or any
/// floating-point type. Aggregates: a complete struct or union whose members
/// are all themselves supported, or an array of a supported element type.
-/// Also a `_Complex` of a supported element type. Everything else is reported
-/// NYI at the reject() choke point in classifyX86_64Function.
+/// Also a `_Complex`, or a fixed-width vector, of a supported element type.
+/// Everything else is reported NYI at the reject() choke point in
+/// classifyX86_64Function.
static bool isSupportedType(mlir::Type ty, const DataLayout &dl) {
// A pointer is only handled in the default address space (null) or an
// already-lowered target address space. A LangAddressSpaceAttr must be
@@ -135,6 +142,34 @@ static bool isSupportedType(mlir::Type ty, const DataLayout &dl) {
}
if (auto complexTy = dyn_cast<cir::ComplexType>(ty))
return isSupportedType(complexTy.getElementType(), dl);
+ if (auto vecTy = dyn_cast<cir::VectorType>(ty)) {
+ // A scalable vector has no size the SysV eightbyte rules can read, and
+ // x86_64 has no calling convention for one.
+ if (vecTy.getIsScalable())
+ return false;
+ // The classifier sizes a vector as element count times element width, so
+ // an element is only usable where that width is the one clang gives it.
+ // It is not for bool (a bit to clang, a byte here), for a _BitInt narrower
+ // than a byte (clang rounds to the storage container), or for x87 long
+ // double (80 bits here against clang's 128). A pointer is excluded for a
+ // different reason, its pointee being what abiTypeToCIR drops.
+ mlir::Type elemTy = vecTy.getElementType();
+ if (auto elemInt = dyn_cast<cir::IntType>(elemTy)) {
+ if (elemInt.getWidth() % 8)
+ return false;
+ } else if (auto elemFp = dyn_cast<cir::FPTypeInterface>(elemTy)) {
+ if (&elemFp.getFloatSemantics() == &llvm::APFloat::x87DoubleExtended())
+ return false;
+ } else {
+ return false;
+ }
+ // Clang also rounds the vector's own width up to a power of two, and the
+ // classifier branches on the exact width, so a three-char vector would be
+ // classified at 24 bits where clang uses 32.
+ if (!llvm::isPowerOf2_64(dl.getTypeSizeInBits(ty).getFixedValue()))
+ return false;
+ return isSupportedType(elemTy, dl);
+ }
if (auto arrTy = dyn_cast<cir::ArrayType>(ty))
return isSupportedType(arrTy.getElementType(), dl);
if (auto recTy = dyn_cast<cir::RecordType>(ty)) {
@@ -256,6 +291,14 @@ static const llvm::abi::Type *mapCIRType(mlir::Type type,
mapCIRType(complexTy.getElementType(), typeMapper, dl, modOp),
llvm::Align(dl.getTypeABIAlignment(type)));
})
+ .Case([&](cir::VectorType vecTy) {
+ // isSupportedType rejects a scalable vector, so the element count is
+ // always fixed here.
+ return tb.getVectorType(
+ mapCIRType(vecTy.getElementType(), typeMapper, dl, modOp),
+ llvm::ElementCount::getFixed(vecTy.getSize()),
+ llvm::Align(dl.getTypeABIAlignment(type)));
+ })
.Case([&](cir::ArrayType arrTy) {
const llvm::abi::Type *elemAbi =
mapCIRType(arrTy.getElementType(), typeMapper, dl, modOp);
@@ -334,10 +377,10 @@ convertABIArgInfo(const llvm::abi::ArgInfo &info, MLIRContext *ctx,
// type, so a non-null coerce does not by itself mean a rewrite is needed.
const llvm::abi::Type *coerceAbi = info.getCoerceToType();
bool isAggregate = isa_and_present<cir::RecordType, cir::ArrayType>(origTy);
- // For a _Complex the classifier's coerce is only sometimes the natural
- // type, so it has to be read rather than assumed.
+ // For a _Complex or a vector the classifier's coerce is only sometimes the
+ // natural type, so it has to be read rather than assumed.
bool comparesAgainstCoerce =
- coerceAbi && isa_and_present<cir::ComplexType>(origTy);
+ coerceAbi && isa_and_present<cir::ComplexType, cir::VectorType>(origTy);
bool coerceIsRegisterTuple =
isa_and_present<llvm::abi::RecordType>(coerceAbi);
// Compare widths rather than identity: a coerce no wider than the natural
@@ -381,8 +424,7 @@ convertABIArgInfo(const llvm::abi::ArgInfo &info, MLIRContext *ctx,
/// Where \p fnTy's declared parameters end and its ellipsis arguments begin.
///
/// The only x86_64 rule that reads this boundary sends an unnamed vector wider
-/// than 128 bits to memory, and isSupportedType rejects every vector, so no
-/// input this bridge accepts can observe the difference.
+/// than 128 bits to memory.
static llvm::abi::RequiredArgs requiredArgs(cir::FuncType fnTy) {
if (!fnTy.isVarArg())
return llvm::abi::RequiredArgs::All;
@@ -465,6 +507,34 @@ static std::optional<FunctionClassification> classifyX86_64Signature(
return fc;
}
+/// The AVX level in force at \p func, following getEffectiveX86AVXABILevel in
+/// clang/lib/CodeGen/Targets/X86.cpp. Classic reads the target attribute
+/// itself, where this reads the feature list CIRGen recorded from it. Two
+/// consequences: a declaration gets \p base, CIRGen recording features only on
+/// definitions, and target_clones and cpu_specific will need excluding once
+/// multiversioning is implemented, classic leaving those at the module level.
+static llvm::abi::X86AVXABILevel funcAvxLevel(cir::FuncOp func,
+ llvm::abi::X86AVXABILevel base) {
+ auto features = func->getAttrOfType<mlir::StringAttr>("cir.target-features");
+ if (!features)
+ return base;
+ // A '-' entry disables the feature, so match a whole '+' entry rather than
+ // searching for the name.
+ auto enabled = [&](llvm::StringRef name) {
+ for (llvm::StringRef feature : llvm::split(features.getValue(), ','))
+ if (feature.consume_front("+") && feature == name)
+ return true;
+ return false;
+ };
+ // avx512f implies avx, so both entries are present. Test the wider name
+ // first or the AVX512 branch is unreachable.
+ if (enabled("avx512f"))
+ return std::max(base, llvm::abi::X86AVXABILevel::AVX512);
+ if (enabled("avx"))
+ return std::max(base, llvm::abi::X86AVXABILevel::AVX);
+ return base;
+}
+
/// Classify a cir.func for x86_64 SysV using the LLVM ABI library. Returns
/// std::nullopt and emits an NYI error if the signature uses a type the bridge
/// does not handle yet.
@@ -634,16 +704,33 @@ void CallConvLoweringPass::runOnOperation() {
CIRABIRewriteContext rewriteCtx(moduleOp, dl);
SymbolTable symbolTable(moduleOp);
- // For the x86_64 target, build the LLVM ABI library classifier once and
- // reuse it (and its type mapper) across every function.
+ // A per-function target attribute can raise the AVX level, so one classifier
+ // per module would misclassify a wide vector in such a function.
+ static constexpr unsigned numAvxLevels =
+ static_cast<unsigned>(llvm::abi::X86AVXABILevel::AVX512) + 1;
+ bool isX86 = target == cir::CallConvTarget::X86_64;
std::optional<mlir::abi::ABITypeMapper> x86TypeMapper;
- std::unique_ptr<llvm::abi::TargetInfo> x86Target;
- if (target == cir::CallConvTarget::X86_64) {
+ std::array<std::unique_ptr<llvm::abi::TargetInfo>, numAvxLevels> x86Targets;
+ if (isX86)
x86TypeMapper.emplace(dl);
- x86Target = llvm::abi::createX86_64TargetInfo(
- x86TypeMapper->getTypeBuilder(), x86AvxAbiLevel.getValue(),
- /*Has64BitPointers=*/true, x86AbiCompat);
- }
+ auto x86TargetFor =
+ [&](llvm::abi::X86AVXABILevel level) -> const llvm::abi::TargetInfo & {
+ assert(static_cast<unsigned>(level) < numAvxLevels &&
+ "a new X86AVXABILevel needs a slot in x86Targets");
+ std::unique_ptr<llvm::abi::TargetInfo> &slot =
+ x86Targets[static_cast<unsigned>(level)];
+ if (!slot)
+ slot = llvm::abi::createX86_64TargetInfo(
+ x86TypeMapper->getTypeBuilder(), level,
+ /*Has64BitPointers=*/true, x86AbiCompat);
+ return *slot;
+ };
+ llvm::abi::X86AVXABILevel baseAvxLevel = x86AvxAbiLevel.getValue();
+ auto avxLevelFor = [&](cir::FuncOp func) -> llvm::abi::X86AVXABILevel {
+ if (!x86TargetAttrAvx || !func)
+ return baseAvxLevel;
+ return funcAvxLevel(func, baseAvxLevel);
+ };
// Classify every cir.func up front. No IR mutation happens here, so
// later walks can consult any function's classification regardless of
@@ -652,8 +739,9 @@ void CallConvLoweringPass::runOnOperation() {
bool anyFailed = false;
moduleOp.walk([&](cir::FuncOp f) {
std::optional<FunctionClassification> fc;
- if (x86Target)
- fc = classifyX86_64Function(f, dl, *x86TypeMapper, *x86Target, moduleOp);
+ if (isX86)
+ fc = classifyX86_64Function(f, dl, *x86TypeMapper,
+ x86TargetFor(avxLevelFor(f)), moduleOp);
else
fc = classifyFunction(f, dl, target, classificationAttr);
if (!fc) {
@@ -694,7 +782,7 @@ void CallConvLoweringPass::runOnOperation() {
// drivers the classification comes from a fixed per-function source, so
// such a call stays short a classification and rewriteCallSite reports it.
cir::FuncType calleeTy = callee.getFunctionType();
- if (!x86Target || call.getNumArgOperands() <= calleeTy.getNumInputs())
+ if (!isX86 || call.getNumArgOperands() <= calleeTy.getNumInputs())
return;
// A callee declared without a prototype also takes more operands than it
// declares, and the verifier allows it. Those extra arguments are named
@@ -706,8 +794,14 @@ void CallConvLoweringPass::runOnOperation() {
anyFailed = true;
return;
}
- std::optional<FunctionClassification> fc = classifyX86_64VariadicCall(
- call, calleeTy, dl, *x86TypeMapper, *x86Target, moduleOp);
+ // The callee's level, not the caller's: this pass rewrites the definition
+ // and its call sites from one classification, so they have to agree.
+ // Classic instead arranges every call site from the caller and reports a
+ // caller whose level disagrees with its callee in checkFunctionCallABI,
+ // which has no equivalent here yet.
+ std::optional<FunctionClassification> fc =
+ classifyX86_64VariadicCall(call, calleeTy, dl, *x86TypeMapper,
+ x86TargetFor(avxLevelFor(callee)), moduleOp);
if (!fc) {
anyFailed = true;
return;
@@ -807,11 +901,15 @@ void CallConvLoweringPass::runOnOperation() {
cir::FuncType funcTy = indirectCalleeType(c);
auto classifySignature =
[&](mlir::TypeRange argTypes) -> std::optional<FunctionClassification> {
- if (x86Target)
- return classifyX86_64Signature(funcTy.getReturnType(), argTypes,
- requiredArgs(funcTy), ctx, dl,
- *x86TypeMapper, *x86Target, moduleOp,
- [&]() { return c->emitOpError(); });
+ // A callee resolved at run time carries no features of its own, so the
+ // level comes from the function containing the call, which is the
+ // declaration classic arranges every call site from.
+ if (isX86)
+ return classifyX86_64Signature(
+ funcTy.getReturnType(), argTypes, requiredArgs(funcTy), ctx, dl,
+ *x86TypeMapper,
+ x86TargetFor(avxLevelFor(c->getParentOfType<cir::FuncOp>())),
+ moduleOp, [&]() { return c->emitOpError(); });
return withReturnVoidness(
mlir::abi::test::classify(argTypes, funcTy.getReturnType(), dl),
funcTy.getReturnType());
@@ -857,12 +955,12 @@ std::unique_ptr<Pass> mlir::createCallConvLoweringPass() {
return std::make_unique<CallConvLoweringPass>();
}
-std::unique_ptr<Pass>
-mlir::createCallConvLoweringPass(cir::CallConvTarget target,
- llvm::abi::X86AVXABILevel x86AvxAbiLevel,
- const llvm::abi::ABICompatInfo &x86AbiCompat) {
+std::unique_ptr<Pass> mlir::createCallConvLoweringPass(
+ cir::CallConvTarget target, llvm::abi::X86AVXABILevel x86AvxAbiLevel,
+ bool x86TargetAttrAvx, const llvm::abi::ABICompatInfo &x86AbiCompat) {
CallConvLoweringOptions options;
options.target = target;
options.x86AvxAbiLevel = x86AvxAbiLevel;
+ options.x86TargetAttrAvx = x86TargetAttrAvx;
return std::make_unique<CallConvLoweringPass>(options, x86AbiCompat);
}
diff --git a/clang/lib/CIR/Lowering/CIRPasses.cpp b/clang/lib/CIR/Lowering/CIRPasses.cpp
index dad17f2ef8659..972dc6d1a4a8b 100644
--- a/clang/lib/CIR/Lowering/CIRPasses.cpp
+++ b/clang/lib/CIR/Lowering/CIRPasses.cpp
@@ -29,6 +29,26 @@ static CallConvTarget getCallConvTarget(const llvm::Triple &triple) {
return CallConvTarget::None;
}
+/// The AVX level the classifier uses to size a native vector. Read from the
+/// target ABI name, as CodeGenModule does.
+static llvm::abi::X86AVXABILevel getX86AVXABILevel(llvm::StringRef abi) {
+ if (abi == "avx512")
+ return llvm::abi::X86AVXABILevel::AVX512;
+ if (abi == "avx")
+ return llvm::abi::X86AVXABILevel::AVX;
+ return llvm::abi::X86AVXABILevel::None;
+}
+
+/// Whether `__attribute__((target(...)))` on a function may raise its AVX ABI
+/// level above the command line's. Mirrors getEffectiveX86AVXABILevel in
+/// clang/lib/CodeGen/Targets/X86.cpp, which keeps a target that opts out, and
+/// any ABI older than the rule, at the module level.
+static bool allowsX86TargetAttrAvx(const clang::ASTContext &astContext) {
+ return !astContext.getTargetInfo().getTriple().isPS() &&
+ astContext.getLangOpts().getClangABICompat() >
+ clang::LangOptions::ClangABI::Ver23;
+}
+
/// The x86_64 ABI-compatibility flags, derived from the target and the
/// requested compatibility version. Every flag defaults to true in the ABI
/// library, which is not what any target computes: Clang11Compat is false for a
@@ -95,12 +115,12 @@ runCIRToCIRPasses(mlir::ModuleOp theModule, mlir::MLIRContext &mlirContext,
// so it must run after CXXABILowering has lowered C++ ABI types to plain
// records the classifier can handle. Only the x86_64 System V classifier
// is implemented; other targets are left unchanged.
- CallConvTarget target =
- getCallConvTarget(astContext.getTargetInfo().getTriple());
+ const clang::TargetInfo &targetInfo = astContext.getTargetInfo();
+ CallConvTarget target = getCallConvTarget(targetInfo.getTriple());
if (target != CallConvTarget::None)
pm.addPass(mlir::createCallConvLoweringPass(
- target, llvm::abi::X86AVXABILevel::None,
- getX86ABICompatInfo(astContext)));
+ target, getX86AVXABILevel(targetInfo.getABI()),
+ allowsX86TargetAttrAvx(astContext), getX86ABICompatInfo(astContext)));
}
pm.addPass(mlir::createLoweringPreparePass(&astContext));
diff --git a/clang/test/CIR/CodeGen/call-conv-lowering-x86_64-abi-compat.c b/clang/test/CIR/CodeGen/call-conv-lowering-x86_64-abi-compat.c
index f5cf3c1d49da1..8eb03df7fa5f9 100644
--- a/clang/test/CIR/CodeGen/call-conv-lowering-x86_64-abi-compat.c
+++ b/clang/test/CIR/CodeGen/call-conv-lowering-x86_64-abi-compat.c
@@ -1,13 +1,32 @@
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fclangir -emit-llvm %s -o %t-cir.ll
-// RUN: FileCheck --check-prefix=LINUX-CIR --input-file=%t-cir.ll %s
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm %s -o %t.ll
-// RUN: FileCheck --check-prefix=LINUX-OGCG --input-file=%t.ll %s
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx -fclangir -emit-llvm %s -o %t-cir.ll
+// RUN: FileCheck --check-prefixes=LINUX,LINUX-CIR --input-file=%t-cir.ll %s
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx -emit-llvm %s -o %t.ll
+// RUN: FileCheck --check-prefixes=LINUX,LINUX-OGCG --input-file=%t.ll %s
-// RUN: %clang_cc1 -triple x86_64-apple-darwin -fclangir -emit-llvm %s -o %t-darwin-cir.ll
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx -fclang-abi-compat=3.8 -fclangir -emit-llvm %s -o %t-38-cir.ll
+// RUN: FileCheck --check-prefix=LINUX38 --input-file=%t-38-cir.ll %s
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx -fclang-abi-compat=3.8 -emit-llvm %s -o %t-38.ll
+// RUN: FileCheck --check-prefix=LINUX38 --input-file=%t-38.ll %s
+
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx -fclang-abi-compat=9 -fclangir -emit-llvm %s -o %t-9-cir.ll
+// RUN: FileCheck --check-prefix=LINUX9 --input-file=%t-9-cir.ll %s
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx -fclang-abi-compat=9 -emit-llvm %s -o %t-9.ll
+// RUN: FileCheck --check-prefix=LINUX9 --input-file=%t-9.ll %s
+
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx -fclang-abi-compat=11 -fclangir -emit-llvm %s -o %t-11-cir.ll
+// RUN: FileCheck --check-prefixes=LINUX11-CIR --input-file=%t-11-cir.ll %s
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx -fclang-abi-compat=11 -emit-llvm %s -o %t-11.ll
+// RUN: FileCheck --check-prefixes=LINUX11-OGCG --input-file=%t-11.ll %s
+
+// RUN: %clang_cc1 -triple x86_64-apple-darwin -target-feature +avx -fclangir -emit-llvm %s -o %t-darwin-cir.ll
// RUN: FileCheck --check-prefix=DARWIN --input-file=%t-darwin-cir.ll %s
-// RUN: %clang_cc1 -triple x86_64-apple-darwin -emit-llvm %s -o %t-darwin.ll
+// RUN: %clang_cc1 -triple x86_64-apple-darwin -target-feature +avx -emit-llvm %s -o %t-darwin.ll
// RUN: FileCheck --check-prefix=DARWIN --input-file=%t-darwin.ll %s
+typedef long long v1ll __attribute__((vector_size(8)));
+typedef __int128 v2i128 __attribute__((vector_size(32)));
+typedef float v8f __attribute__((vector_size(32)));
+
// The 0.98 ABI revision sends an eightbyte pair to memory when the high half is
// X87UP and the low half is not X87. Darwin exempts itself for binary
// compatibility with older GCC, so the same union passes in registers there.
@@ -18,3 +37,34 @@ void rev98(ULongDouble u) { (void)u; }
// LINUX-CIR: define dso_local void @rev98(ptr noalias noundef byval(%union.ULongDouble) align 16 %{{[^,)]+}})
// LINUX-OGCG: define dso_local void @rev98(ptr noundef byval(%union.ULongDouble) align 16 %{{[^,)]+}})
// DARWIN: define void @rev98(i64 %{{[^,)]+}}, double %{{[^,)]+}})
+
+// GCC classifies a 64-bit vector of a 64-bit integer as SSE. Clang 3.8 and
+// older did not, and Darwin, FreeBSD and PlayStation still do not.
+void mmx(v1ll v) { (void)v; }
+
+// LINUX: define dso_local void @mmx(double noundef %{{[^,)]+}})
+// LINUX38: define dso_local void @mmx(i64 noundef %{{[^,)]+}})
+// LINUX9: define dso_local void @mmx(double noundef %{{[^,)]+}})
+// DARWIN: define void @mmx(i64 noundef %{{[^,)]+}})
+
+// GCC classifies a vector of __int128 as memory. Clang 9 and older did not,
+// and only Linux and NetBSD follow it. AVX is on above so that this vector
+// would otherwise reach a register, which is what makes the rule observable.
+void wide_int128(v2i128 v) { (void)v; }
+
+// LINUX-CIR: define dso_local void @wide_int128(ptr noalias noundef byval(<2 x i128>) align 32 %{{[^,)]+}})
+// LINUX-OGCG: define dso_local void @wide_int128(ptr noundef byval(<2 x i128>) align 32 %{{[^,)]+}})
+// LINUX38: define dso_local void @wide_int128(<2 x i128> noundef %{{[^,)]+}})
+// LINUX9: define dso_local void @wide_int128(<2 x i128> noundef %{{[^,)]+}})
+// DARWIN: define void @wide_int128(<2 x i128> noundef %{{[^,)]+}})
+
+// A union larger than an eightbyte is classified from the member spanning its
+// size, so it reaches registers once the level admits the vector. Clang 11 and
+// older instead treated every member as spanning, which sends this to memory
+// because the float member does not.
+union UnionWideVector { v8f v; float f; };
+void take_union_wide_vector(union UnionWideVector u) { (void)u; }
+
+// LINUX: define dso_local void @take_union_wide_vector(<4 x double> %{{[^,)]+}})
+// LINUX11-CIR: define dso_local void @take_union_wide_vector(ptr noalias noundef byval(%union.UnionWideVector) align 32 %{{[^,)]+}})
+// LINUX11-OGCG: define dso_local void @take_union_wide_vector(ptr noundef byval(%union.UnionWideVector) align 32 %{{[^,)]+}})
diff --git a/clang/test/CIR/CodeGen/call-conv-lowering-x86_64-avx.c b/clang/test/CIR/CodeGen/call-conv-lowering-x86_64-avx.c
new file mode 100644
index 0000000000000..0c733c34f4d9d
--- /dev/null
+++ b/clang/test/CIR/CodeGen/call-conv-lowering-x86_64-avx.c
@@ -0,0 +1,165 @@
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fclangir -clangir-enable-call-conv-lowering -emit-cir %s -o %t.cir
+// RUN: FileCheck --check-prefixes=CIR,CIR-SSE --input-file=%t.cir %s
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fclangir -clangir-enable-call-conv-lowering -emit-llvm %s -o %t-cir.ll
+// RUN: FileCheck --check-prefixes=LLVM,LLVM-CIR-SSE --input-file=%t-cir.ll %s
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm %s -o %t.ll
+// RUN: FileCheck --check-prefixes=LLVM,LLVM-OGCG-SSE --input-file=%t.ll %s
+
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx -fclangir -clangir-enable-call-conv-lowering -emit-cir %s -o %t-avx.cir
+// RUN: FileCheck --check-prefixes=CIR,CIR-AVX --input-file=%t-avx.cir %s
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx -fclangir -clangir-enable-call-conv-lowering -emit-llvm %s -o %t-avx-cir.ll
+// RUN: FileCheck --check-prefixes=LLVM,LLVM-AVX,LLVM-CIR-AVX --input-file=%t-avx-cir.ll %s
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx -emit-llvm %s -o %t-avx.ll
+// RUN: FileCheck --check-prefixes=LLVM,LLVM-AVX,LLVM-OGCG-AVX --input-file=%t-avx.ll %s
+
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx512f -fclangir -clangir-enable-call-conv-lowering -emit-llvm %s -o %t-avx512-cir.ll
+// RUN: FileCheck --check-prefixes=LLVM,LLVM-AVX512 --input-file=%t-avx512-cir.ll %s
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx512f -emit-llvm %s -o %t-avx512.ll
+// RUN: FileCheck --check-prefixes=LLVM,LLVM-AVX512 --input-file=%t-avx512.ll %s
+
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fclang-abi-compat=23 -fclangir -clangir-enable-call-conv-lowering -emit-llvm %s -o %t-compat23-cir.ll
+// RUN: FileCheck --check-prefix=LLVM-CIR-PINNED --input-file=%t-compat23-cir.ll %s
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fclang-abi-compat=23 -emit-llvm %s -o %t-compat23.ll
+// RUN: FileCheck --check-prefix=LLVM-OGCG-PINNED --input-file=%t-compat23.ll %s
+
+// RUN: %clang_cc1 -triple x86_64-scei-ps4 -fclangir -clangir-enable-call-conv-lowering -emit-llvm %s -o %t-ps4-cir.ll
+// RUN: FileCheck --check-prefix=LLVM-CIR-PINNED --input-file=%t-ps4-cir.ll %s
+// RUN: %clang_cc1 -triple x86_64-scei-ps4 -emit-llvm %s -o %t-ps4.ll
+// RUN: FileCheck --check-prefix=LLVM-OGCG-PINNED --input-file=%t-ps4.ll %s
+
+typedef float v4f __attribute__((vector_size(16)));
+typedef float v8f __attribute__((vector_size(32)));
+typedef float v16f __attribute__((vector_size(64)));
+
+// A 128-bit vector is at or below the native vector size at every AVX level,
+// so it always passes in a register.
+void take_v128(v4f v) { (void)v; }
+
+// CIR: cir.func {{.*}}@take_v128(%arg0: !cir.vector<4 x !cir.float>{{.*}})
+// LLVM: define dso_local void @take_v128(<4 x float> noundef %{{[^,)]+}})
+
+// A 256-bit vector only reaches a register once the ABI level is AVX. Below
+// that it is passed byval, aligned to its size.
+void take_v256(v8f v) { (void)v; }
+
+// CIR-SSE: cir.func {{.*}}@take_v256(%arg0: !cir.ptr<!cir.vector<8 x !cir.float>> {{.*}}llvm.align = 32 : i64{{.*}}llvm.byval = !cir.vector<8 x !cir.float>{{.*}})
+// CIR-AVX: cir.func {{.*}}@take_v256(%arg0: !cir.vector<8 x !cir.float>{{.*}})
+// LLVM-CIR-SSE: define dso_local void @take_v256(ptr noalias noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+// LLVM-OGCG-SSE: define dso_local void @take_v256(ptr noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+// LLVM-AVX: define dso_local void @take_v256(<8 x float> noundef %{{[^,)]+}})
+// LLVM-AVX512: define dso_local void @take_v256(<8 x float> noundef %{{[^,)]+}})
+
+// The register-versus-memory split applies to arguments only. A 256-bit
+// vector comes back in registers at every level.
+v8f ret_v256(void) { v8f z = {0}; return z; }
+
+// LLVM: define dso_local <8 x float> @ret_v256()
+
+// Below AVX the caller has to build the byval slot, which is the memory copy a
+// vector argument needs on a non-variadic call.
+void call_v256(v8f v) { take_v256(v); }
+
+// LLVM-CIR-SSE: define dso_local void @call_v256(ptr noalias noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+// LLVM-CIR-SSE: call void @take_v256(ptr noalias noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+// LLVM-OGCG-SSE: define dso_local void @call_v256(ptr noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+// LLVM-OGCG-SSE: call void @take_v256(ptr noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+// LLVM-AVX: define dso_local void @call_v256(<8 x float> noundef %{{[^,)]+}})
+// LLVM-AVX: call void @take_v256(<8 x float> noundef %{{[^,)]+}})
+// LLVM-AVX512: define dso_local void @call_v256(<8 x float> noundef %{{[^,)]+}})
+// LLVM-AVX512: call void @take_v256(<8 x float> noundef %{{[^,)]+}})
+
+// A 512-bit vector needs AVX512 for the same treatment.
+void take_v512(v16f v) { (void)v; }
+
+// CIR-SSE: cir.func {{.*}}@take_v512(%arg0: !cir.ptr<!cir.vector<16 x !cir.float>> {{.*}}llvm.align = 64 : i64{{.*}}llvm.byval = !cir.vector<16 x !cir.float>{{.*}})
+// CIR-AVX: cir.func {{.*}}@take_v512(%arg0: !cir.ptr<!cir.vector<16 x !cir.float>> {{.*}}llvm.align = 64 : i64{{.*}}llvm.byval = !cir.vector<16 x !cir.float>{{.*}})
+// LLVM-CIR-SSE: define dso_local void @take_v512(ptr noalias noundef byval(<16 x float>) align 64 %{{[^,)]+}})
+// LLVM-OGCG-SSE: define dso_local void @take_v512(ptr noundef byval(<16 x float>) align 64 %{{[^,)]+}})
+// LLVM-CIR-AVX: define dso_local void @take_v512(ptr noalias noundef byval(<16 x float>) align 64 %{{[^,)]+}})
+// LLVM-OGCG-AVX: define dso_local void @take_v512(ptr noundef byval(<16 x float>) align 64 %{{[^,)]+}})
+// LLVM-AVX512: define dso_local void @take_v512(<16 x float> noundef %{{[^,)]+}})
+
+// Disabling a feature the command line enabled is the only way a function's
+// feature list carries a '-' entry, and the level still cannot fall below the
+// module's, so this is byval only where the module itself lacks AVX512.
+__attribute__((target("no-avx512f"))) void take_v512_no_avx512(v16f v) { (void)v; }
+
+// LLVM-CIR-SSE: define dso_local void @take_v512_no_avx512(ptr noalias noundef byval(<16 x float>) align 64 %{{[^,)]+}})
+// LLVM-OGCG-SSE: define dso_local void @take_v512_no_avx512(ptr noundef byval(<16 x float>) align 64 %{{[^,)]+}})
+// LLVM-CIR-AVX: define dso_local void @take_v512_no_avx512(ptr noalias noundef byval(<16 x float>) align 64 %{{[^,)]+}})
+// LLVM-OGCG-AVX: define dso_local void @take_v512_no_avx512(ptr noundef byval(<16 x float>) align 64 %{{[^,)]+}})
+// LLVM-AVX512: define dso_local void @take_v512_no_avx512(<16 x float> noundef %{{[^,)]+}})
+
+// A target attribute raises the level for one function, so these two classify
+// their vector in a register at every configuration above. An ABI older than
+// the rule pins them back to the module's level, as does a target that opts out
+// of the per-function rule entirely.
+__attribute__((target("avx"))) void take_v256_tgt(v8f v) { (void)v; }
+
+// CIR: cir.func {{.*}}@take_v256_tgt(%arg0: !cir.vector<8 x !cir.float>{{.*}})
+// LLVM: define dso_local void @take_v256_tgt(<8 x float> noundef %{{[^,)]+}})
+// LLVM-CIR-PINNED: define dso_local void @take_v256_tgt(ptr noalias noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+// LLVM-OGCG-PINNED: define dso_local void @take_v256_tgt(ptr noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+
+__attribute__((target("avx512f"))) void take_v512_tgt(v16f v) { (void)v; }
+
+// CIR: cir.func {{.*}}@take_v512_tgt(%arg0: !cir.vector<16 x !cir.float>{{.*}})
+// LLVM: define dso_local void @take_v512_tgt(<16 x float> noundef %{{[^,)]+}})
+// LLVM-CIR-PINNED: define dso_local void @take_v512_tgt(ptr noalias noundef byval(<16 x float>) align 64 %{{[^,)]+}})
+// LLVM-OGCG-PINNED: define dso_local void @take_v512_tgt(ptr noundef byval(<16 x float>) align 64 %{{[^,)]+}})
+
+// A call site has to agree with the callee it resolves to. Both sides carry
+// the attribute here, which is the case classic accepts. A caller whose level
+// disagrees with its callee is diagnosed by checkFunctionCallABI in classic
+// CodeGen, and nothing diagnoses it here yet.
+__attribute__((target("avx"))) void call_v256_tgt(v8f v) { take_v256_tgt(v); }
+
+// LLVM: define dso_local void @call_v256_tgt(<8 x float> noundef %{{[^,)]+}})
+// LLVM: call void @take_v256_tgt(<8 x float> noundef %{{[^,)]+}})
+// LLVM-CIR-PINNED: define dso_local void @call_v256_tgt(ptr noalias noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+// LLVM-CIR-PINNED: call void @take_v256_tgt(ptr noalias noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+// LLVM-OGCG-PINNED: define dso_local void @call_v256_tgt(ptr noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+// LLVM-OGCG-PINNED: call void @take_v256_tgt(ptr noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+
+// A callee resolved at run time has no features of its own, so the level comes
+// from the function containing the call. The pair differs only in the
+// attribute, so it is the attribute that has to move the argument.
+typedef void (*v8f_fn)(v8f);
+__attribute__((target("avx"))) void call_indirect(v8f_fn p, v8f v) { p(v); }
+
+// LLVM: define dso_local void @call_indirect(ptr noundef %{{[^,)]+}}, <8 x float> noundef %{{[^,)]+}})
+// LLVM: call void %{{[0-9]+}}(<8 x float> noundef %{{[^,)]+}})
+// LLVM-CIR-PINNED: define dso_local void @call_indirect(ptr noundef %{{[^,)]+}}, ptr noalias noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+// LLVM-OGCG-PINNED: define dso_local void @call_indirect(ptr noundef %{{[^,)]+}}, ptr noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+
+void call_indirect_plain(v8f_fn p, v8f v) { p(v); }
+
+// LLVM-CIR-SSE: define dso_local void @call_indirect_plain(ptr noundef %{{[^,)]+}}, ptr noalias noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+// LLVM-CIR-SSE: call void %{{[0-9]+}}(ptr noalias noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+// LLVM-OGCG-SSE: define dso_local void @call_indirect_plain(ptr noundef %{{[^,)]+}}, ptr noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+// LLVM-AVX: define dso_local void @call_indirect_plain(ptr noundef %{{[^,)]+}}, <8 x float> noundef %{{[^,)]+}})
+// LLVM-AVX: call void %{{[0-9]+}}(<8 x float> noundef %{{[^,)]+}})
+
+// Only a named argument can reach a register, so the struct that passes
+// directly as a declared parameter goes to memory at the ellipsis. This is
+// the rule that reads the declared-parameter boundary, and it is observable
+// only where the level admits the vector.
+typedef struct { v8f v; } StructWideVector;
+int variadic(const char *f, ...);
+void named_swv(StructWideVector s) { (void)s; }
+void pass_swv(StructWideVector s) { variadic("x", s); }
+
+// LLVM-AVX: define dso_local void @named_swv(<8 x float> %{{[^,)]+}})
+// LLVM-AVX: define dso_local void @pass_swv(<8 x float> %{{[^,)]+}})
+// LLVM-CIR-AVX: call i32 (ptr, ...) @variadic(ptr noundef @.str, ptr noalias noundef byval(%struct.StructWideVector) align 32 %{{[^,)]+}})
+// LLVM-OGCG-AVX: call i32 (ptr, ...) @variadic(ptr noundef @.str, ptr noundef byval(%struct.StructWideVector) align 32 %{{[^,)]+}})
+
+// A union whose widest member is a 256-bit vector is classified from that
+// member, so it reaches registers once the level admits the vector.
+union UnionWideVector { v8f v; float f; };
+void take_union_wide_vector(union UnionWideVector u) { (void)u; }
+
+// LLVM-CIR-SSE: define dso_local void @take_union_wide_vector(ptr noalias noundef byval(%union.UnionWideVector) align 32 %{{[^,)]+}})
+// LLVM-OGCG-SSE: define dso_local void @take_union_wide_vector(ptr noundef byval(%union.UnionWideVector) align 32 %{{[^,)]+}})
+// LLVM-AVX: define dso_local void @take_union_wide_vector(<4 x double> %{{[^,)]+}})
+// LLVM-AVX512: define dso_local void @take_union_wide_vector(<4 x double> %{{[^,)]+}})
diff --git a/clang/test/CIR/CodeGen/call-conv-lowering-x86_64.c b/clang/test/CIR/CodeGen/call-conv-lowering-x86_64.c
index 50c9eedfa9086..d3ba0ebb2a9b0 100644
--- a/clang/test/CIR/CodeGen/call-conv-lowering-x86_64.c
+++ b/clang/test/CIR/CodeGen/call-conv-lowering-x86_64.c
@@ -264,6 +264,83 @@ _Complex long long complex_longlong(_Complex long long c) { return c; }
// LLVM-CIR: define dso_local { i64, i64 } @complex_longlong(i64 %{{[^,)]+}}, i64 %{{[^,)]+}})
// LLVM-OGCG: define dso_local { i64, i64 } @complex_longlong(i64 noundef %{{[^,)]+}}, i64 noundef %{{[^,)]+}})
+// A 128-bit vector fills one xmm register and passes in its own type.
+typedef float v4f __attribute__((vector_size(16)));
+v4f vector128(v4f v) { return v; }
+
+// CIR: cir.func {{.*}}@vector128(%arg0: !cir.vector<4 x !cir.float> {{.*}}) -> !cir.vector<4 x !cir.float>
+// LLVM: define dso_local <4 x float> @vector128(<4 x float> noundef %{{[^,)]+}})
+
+// The classifier does not look inside a vector when picking the type for an SSE
+// eightbyte, so a 64-bit vector coerces to double rather than to a two-element
+// vector.
+typedef float v2f __attribute__((vector_size(8)));
+v2f vector64(v2f v) { return v; }
+
+// CIR: cir.func {{.*}}@vector64(%arg0: !cir.double {{.*}}) -> !cir.double
+// LLVM: define dso_local double @vector64(double noundef %{{[^,)]+}})
+
+// A vector at or below 32 bits classifies INTEGER, so it passes as the integer
+// covering it rather than in an xmm register.
+typedef int v1i __attribute__((vector_size(4)));
+v1i vector32(v1i v) { return v; }
+
+// CIR: cir.func {{.*}}@vector32(%arg0: !u32i {{.*}}) -> !u32i
+// LLVM: define dso_local i32 @vector32(i32 noundef %{{[^,)]+}})
+
+// An eightbyte holding a 16-bit float vector coerces to double, the same as
+// any other all-float eightbyte, but at 128 bits the format is preserved.
+typedef _Float16 v4h __attribute__((ext_vector_type(4)));
+typedef _Float16 v8h __attribute__((ext_vector_type(8)));
+void take_v4h(v4h v) { (void)v; }
+void take_v8h(v8h v) { (void)v; }
+
+// CIR: cir.func {{.*}}@take_v4h(%arg0: !cir.double{{.*}})
+// CIR: cir.func {{.*}}@take_v8h(%arg0: !cir.vector<8 x !cir.f16>{{.*}})
+// LLVM: define dso_local void @take_v4h(double noundef %{{[^,)]+}})
+// LLVM: define dso_local void @take_v8h(<8 x half> noundef %{{[^,)]+}})
+
+typedef __bf16 v4bf __attribute__((ext_vector_type(4)));
+typedef __bf16 v8bf __attribute__((ext_vector_type(8)));
+void take_v4bf(v4bf v) { (void)v; }
+void take_v8bf(v8bf v) { (void)v; }
+
+// CIR: cir.func {{.*}}@take_v4bf(%arg0: !cir.double{{.*}})
+// CIR: cir.func {{.*}}@take_v8bf(%arg0: !cir.vector<8 x !cir.bf16>{{.*}})
+// LLVM: define dso_local void @take_v4bf(double noundef %{{[^,)]+}})
+// LLVM: define dso_local void @take_v8bf(<8 x bfloat> noundef %{{[^,)]+}})
+
+// An integer element narrower than an eightbyte does not change the rule: the
+// eightbyte is what gets classified, and here it is SSE.
+typedef char v8c __attribute__((ext_vector_type(8)));
+void take_v8c(v8c v) { (void)v; }
+
+// CIR: cir.func {{.*}}@take_v8c(%arg0: !cir.double{{.*}})
+// LLVM: define dso_local void @take_v8c(double noundef %{{[^,)]+}})
+
+// A vector of a 64-bit element keeps its own type, and a one-element vector of
+// __int128 coerces to the pair of eightbytes it occupies.
+typedef double v2d __attribute__((vector_size(16)));
+void take_v2d(v2d v) { (void)v; }
+
+// CIR: cir.func {{.*}}@take_v2d(%arg0: !cir.vector<2 x !cir.double>{{.*}})
+// LLVM: define dso_local void @take_v2d(<2 x double> noundef %{{[^,)]+}})
+
+typedef __int128 v1i128 __attribute__((vector_size(16)));
+void take_v1i128(v1i128 v) { (void)v; }
+
+// CIR: cir.func {{.*}}@take_v1i128(%arg0: !cir.vector<2 x !u64i>{{.*}})
+// LLVM: define dso_local void @take_v1i128(<2 x i64> noundef %{{[^,)]+}})
+
+// gcc passes a one-element vector of a 64-bit float in memory. Its byval
+// alignment is the greater of the vector's own alignment and 8.
+typedef double v1d __attribute__((vector_size(8)));
+void take_v1d(v1d v) { (void)v; }
+
+// CIR: cir.func {{.*}}@take_v1d(%arg0: !cir.ptr<!cir.vector<1 x !cir.double>> {{.*}}llvm.align = 8 : i64{{.*}}llvm.byval = !cir.vector<1 x !cir.double>{{.*}})
+// LLVM-CIR: define dso_local void @take_v1d(ptr noalias noundef byval(<1 x double>) align 8 %{{[^,)]+}})
+// LLVM-OGCG: define dso_local void @take_v1d(ptr noundef byval(<1 x double>) align 8 %{{[^,)]+}})
+
// A _Complex reaches the classifier as a record member too, not just on its
// own, so these cover the field walk rather than the top-level mapping.
typedef struct { _Complex float c; } WrapComplexFloat;
diff --git a/clang/test/CIR/Transforms/abi-lowering/x86_64-aggregate-nyi.cir b/clang/test/CIR/Transforms/abi-lowering/x86_64-aggregate-nyi.cir
index 89bc54433d586..ff7eacbe3f608 100644
--- a/clang/test/CIR/Transforms/abi-lowering/x86_64-aggregate-nyi.cir
+++ b/clang/test/CIR/Transforms/abi-lowering/x86_64-aggregate-nyi.cir
@@ -112,4 +112,47 @@ module attributes {
// CHECK: not yet implemented for type '!cir.struct<padded {{.*}}pad !cir.array<!cir.int<s, 8> x 2>}>'
+ // A scalable vector has no size the eightbyte rules can read.
+ cir.func @take_scalable(%arg0: !cir.vector<[4] x !cir.float>) {
+ cir.return
+ }
+
+ // CHECK: not yet implemented for type '!cir.vector<[4] x !cir.float>
+
+ // Clang pads a vector's type width up to a power of two and the classifier
+ // branches on the exact width, so a three-byte vector would be classified at
+ // 24 bits where clang uses 32. The ABI library's vector type cannot carry
+ // the padded width, so this is rejected rather than misclassified.
+ cir.func @take_odd_width(%arg0: !cir.vector<3 x !s8i>) {
+ cir.return
+ }
+
+ // CHECK: not yet implemented for type '!cir.vector<3 x !cir.int<s, 8>>
+
+ // A pointer element loses its pointee on the way back from the classifier,
+ // which would rewrite the signature to a vector of void pointers. No C
+ // vector has a pointer element, so this is rejected instead.
+ cir.func @take_ptr_vector(%arg0: !cir.vector<2 x !cir.ptr<!s32i>>) {
+ cir.return
+ }
+
+ // CHECK: not yet implemented for type '!cir.vector<2 x !cir.ptr<!cir.int<s, 32>>>
+
+ // An element narrower than a byte is rounded up to its storage container by
+ // clang but not by the classifier, so the two would size the vector
+ // differently. Classic passes this one in an xmm register from its 128-bit
+ // width, where the classifier would read 32 bits and coerce to an integer.
+ cir.func @take_sub_byte_element(%arg0: !cir.vector<16 x !cir.int<s, 2, bitint>>) {
+ cir.return
+ }
+
+ // CHECK: not yet implemented for type '!cir.vector<16 x !cir.int<s, 2, bitint>>
+
+ // An x87 long double is 80 bits to the classifier and 128 to clang, so the
+ // two would size this vector differently for the same reason.
+ cir.func @take_x87_element(%arg0: !cir.vector<2 x !cir.long_double<!cir.f80>>) {
+ cir.return
+ }
+
+ // CHECK: not yet implemented for type '!cir.vector<2 x !cir.long_double<!cir.f80>>
}
diff --git a/clang/test/CIR/Transforms/abi-lowering/x86_64-variadic-call.cir b/clang/test/CIR/Transforms/abi-lowering/x86_64-variadic-call.cir
index b8f6422cd5c8a..f0cb16463c8d5 100644
--- a/clang/test/CIR/Transforms/abi-lowering/x86_64-variadic-call.cir
+++ b/clang/test/CIR/Transforms/abi-lowering/x86_64-variadic-call.cir
@@ -120,6 +120,18 @@ module attributes {
// CHECK: cir.func{{.*}} @pass_all_float(%arg0: !cir.ptr<!s8i>, %arg1: !cir.vector<2 x !cir.float>)
// CHECK: cir.call @variadic(%arg0, %{{.*}}) : (!cir.ptr<!s8i>, !cir.vector<2 x !cir.float>) -> !s32i
+ // At this level a 256-bit vector is too wide for a register whether it is
+ // named or not, so both the declared parameter and the ellipsis argument
+ // carry the same byval slot.
+ cir.func @pass_wide_vector(%arg0: !cir.ptr<!s8i>, %arg1: !cir.vector<8 x !cir.float>) {
+ %0 = cir.call @variadic(%arg0, %arg1)
+ : (!cir.ptr<!s8i>, !cir.vector<8 x !cir.float>) -> !s32i
+ cir.return
+ }
+
+ // CHECK: cir.func{{.*}} @pass_wide_vector(%arg0: !cir.ptr<!s8i>, %arg1: !cir.ptr<!cir.vector<8 x !cir.float>> {llvm.align = 32 : i64, llvm.byval = !cir.vector<8 x !cir.float>, llvm.noalias, llvm.noundef})
+ // CHECK: cir.call @variadic(%arg0, %{{.*}}) : (!cir.ptr<!s8i>, !cir.ptr<!cir.vector<8 x !cir.float>> {llvm.align = 32 : i64, llvm.byval = !cir.vector<8 x !cir.float>, llvm.noalias, llvm.noundef}) -> !s32i
+
// A declared parameter is coerced the same way whether or not the call also
// passes ellipsis arguments.
cir.func @pass_declared_coerced(%arg0: !rec_Pair, %arg1: !s32i) {
@@ -181,44 +193,47 @@ module attributes {
// LLVM: declare i32 @variadic_pair(i64, ...)
// LLVM: declare void @variadic_big(ptr dead_on_unwind writable sret(%struct.Big) align 8, ptr, ...)
-// LLVM: define void @pass_none(ptr %{{.+}})
-// LLVM: call i32 (ptr, ...) @variadic(ptr %{{.+}})
+// LLVM: define void @pass_none(ptr %{{[^,)]+}})
+// LLVM: call i32 (ptr, ...) @variadic(ptr %{{[^,)]+}})
+
+// LLVM: define void @pass_scalar(ptr %{{[^,)]+}}, i32 %{{[^,)]+}})
+// LLVM: call i32 (ptr, ...) @variadic(ptr %{{[^,)]+}}, i32 %{{[^,)]+}})
-// LLVM: define void @pass_scalar(ptr %{{.+}}, i32 %{{.+}})
-// LLVM: call i32 (ptr, ...) @variadic(ptr %{{.+}}, i32 %{{.+}})
+// LLVM: define void @pass_narrow(ptr %{{[^,)]+}}, i8 signext %{{[^,)]+}})
+// LLVM: call i32 (ptr, ...) @variadic(ptr %{{[^,)]+}}, i8 signext %{{[^,)]+}})
-// LLVM: define void @pass_narrow(ptr %{{.+}}, i8 signext %{{.+}})
-// LLVM: call i32 (ptr, ...) @variadic(ptr %{{.+}}, i8 signext %{{.+}})
+// LLVM: define void @pass_double(ptr %{{[^,)]+}}, double %{{[^,)]+}})
+// LLVM: call i32 (ptr, ...) @variadic(ptr %{{[^,)]+}}, double %{{[^,)]+}})
-// LLVM: define void @pass_double(ptr %{{.+}}, double %{{.+}})
-// LLVM: call i32 (ptr, ...) @variadic(ptr %{{.+}}, double %{{.+}})
+// LLVM: define void @pass_two_early(ptr %{{[^,)]+}}, i64 %{{[^,)]+}}, i64 %{{[^,)]+}})
+// LLVM: call i32 (ptr, ...) @variadic(ptr %{{[^,)]+}}, i64 %{{[^,)]+}}, i64 %{{[^,)]+}})
-// LLVM: define void @pass_two_early(ptr %{{.+}}, i64 %{{.+}}, i64 %{{.+}})
-// LLVM: call i32 (ptr, ...) @variadic(ptr %{{.+}}, i64 %{{.+}}, i64 %{{.+}})
+// LLVM: define void @pass_two_exhausted(ptr %{{[^,)]+}}, i32 %{{[^,)]+}}, i64 %{{[^,)]+}}, i64 %{{[^,)]+}})
+// LLVM: call i32 (ptr, ...) @variadic(ptr %{{[^,)]+}}, i32 %{{[^,)]+}}, i32 %{{[^,)]+}}, i32 %{{[^,)]+}}, i32 %{{[^,)]+}}, i32 %{{[^,)]+}}, ptr noalias noundef byval(%struct.Two) align 8 %{{[^,)]+}})
-// LLVM: define void @pass_two_exhausted(ptr %{{.+}}, i32 %{{.+}}, i64 %{{.+}}, i64 %{{.+}})
-// LLVM: call i32 (ptr, ...) @variadic(ptr %{{.+}}, i32 %{{.+}}, i32 %{{.+}}, i32 %{{.+}}, i32 %{{.+}}, i32 %{{.+}}, ptr noalias noundef byval(%struct.Two) align 8 %{{.+}})
+// LLVM: define void @pass_big(ptr %{{[^,)]+}}, ptr noalias noundef byval(%struct.Big) align 8 %{{[^,)]+}})
+// LLVM: call i32 (ptr, ...) @variadic(ptr %{{[^,)]+}}, ptr noalias noundef byval(%struct.Big) align 8 %{{[^,)]+}})
-// LLVM: define void @pass_big(ptr %{{.+}}, ptr noalias noundef byval(%struct.Big) align 8 %{{.+}})
-// LLVM: call i32 (ptr, ...) @variadic(ptr %{{.+}}, ptr noalias noundef byval(%struct.Big) align 8 %{{.+}})
+// LLVM: define void @pass_empty(ptr %{{[^,)]+}}, i32 %{{[^,)]+}})
+// LLVM: call i32 (ptr, ...) @variadic(ptr %{{[^,)]+}}, i32 %{{[^,)]+}})
-// LLVM: define void @pass_empty(ptr %{{.+}}, i32 %{{.+}})
-// LLVM: call i32 (ptr, ...) @variadic(ptr %{{.+}}, i32 %{{.+}})
+// LLVM: define void @pass_all_float(ptr %{{[^,)]+}}, <2 x float> %{{[^,)]+}})
+// LLVM: call i32 (ptr, ...) @variadic(ptr %{{[^,)]+}}, <2 x float> %{{[^,)]+}})
-// LLVM: define void @pass_all_float(ptr %{{.+}}, <2 x float> %{{.+}})
-// LLVM: call i32 (ptr, ...) @variadic(ptr %{{.+}}, <2 x float> %{{.+}})
+// LLVM: define void @pass_wide_vector(ptr %{{[^,)]+}}, ptr noalias noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+// LLVM: call i32 (ptr, ...) @variadic(ptr %{{[^,)]+}}, ptr noalias noundef byval(<8 x float>) align 32 %{{[^,)]+}})
-// LLVM: define void @pass_declared_coerced(i64 %{{.+}}, i32 %{{.+}})
-// LLVM: call i32 (i64, ...) @variadic_pair(i64 %{{.+}}, i32 %{{.+}})
+// LLVM: define void @pass_declared_coerced(i64 %{{[^,)]+}}, i32 %{{[^,)]+}})
+// LLVM: call i32 (i64, ...) @variadic_pair(i64 %{{[^,)]+}}, i32 %{{[^,)]+}})
-// LLVM: define void @pass_sret_return(ptr dead_on_unwind noalias writable sret(%struct.Big) align 8 %{{.+}}, ptr %{{.+}}, i32 %{{.+}})
-// LLVM: call void (ptr, ptr, ...) @variadic_big(ptr dead_on_unwind writable sret(%struct.Big) align 8 %{{.+}}, ptr %{{.+}}, i32 %{{.+}})
+// LLVM: define void @pass_sret_return(ptr dead_on_unwind noalias writable sret(%struct.Big) align 8 %{{[^,)]+}}, ptr %{{[^,)]+}}, i32 %{{[^,)]+}})
+// LLVM: call void (ptr, ptr, ...) @variadic_big(ptr dead_on_unwind writable sret(%struct.Big) align 8 %{{[^,)]+}}, ptr %{{[^,)]+}}, i32 %{{[^,)]+}})
-// LLVM: define void @indirect_scalar(ptr %{{.+}}, ptr %{{.+}}, i32 %{{.+}})
-// LLVM: call i32 (ptr, ...) %{{.+}}(ptr %{{.+}}, i32 %{{.+}})
+// LLVM: define void @indirect_scalar(ptr %{{[^,)]+}}, ptr %{{[^,)]+}}, i32 %{{[^,)]+}})
+// LLVM: call i32 (ptr, ...) %{{[^,)]+}}(ptr %{{[^,)]+}}, i32 %{{[^,)]+}})
-// LLVM: define void @indirect_void(ptr %{{.+}}, ptr %{{.+}}, i32 %{{.+}})
-// LLVM: call void (ptr, ...) %{{.+}}(ptr %{{.+}}, i32 %{{.+}})
+// LLVM: define void @indirect_void(ptr %{{[^,)]+}}, ptr %{{[^,)]+}}, i32 %{{[^,)]+}})
+// LLVM: call void (ptr, ...) %{{[^,)]+}}(ptr %{{[^,)]+}}, i32 %{{[^,)]+}})
-// LLVM: define void @indirect_coerced(ptr %{{.+}}, i64 %{{.+}})
-// LLVM: call i32 (i64, ...) %{{.+}}(i64 %{{.+}})
+// LLVM: define void @indirect_coerced(ptr %{{[^,)]+}}, i64 %{{[^,)]+}})
+// LLVM: call i32 (i64, ...) %{{[^,)]+}}(i64 %{{[^,)]+}})
diff --git a/clang/test/CIR/Transforms/abi-lowering/x86_64-vector.cir b/clang/test/CIR/Transforms/abi-lowering/x86_64-vector.cir
index 1c8d705f6d9ac..b1aa989585e10 100644
--- a/clang/test/CIR/Transforms/abi-lowering/x86_64-vector.cir
+++ b/clang/test/CIR/Transforms/abi-lowering/x86_64-vector.cir
@@ -1,8 +1,13 @@
// RUN: cir-opt %s -cir-call-conv-lowering=target=x86_64 | FileCheck %s
+// RUN: cir-opt %s -cir-call-conv-lowering='target=x86_64 x86-avx-abi-level=avx' \
+// RUN: | FileCheck %s --check-prefix=AVX
+// RUN: cir-opt %s -cir-call-conv-lowering='target=x86_64 x86-avx-abi-level=avx512' \
+// RUN: | FileCheck %s --check-prefix=AVX512
// RUN: cir-opt %s -cir-call-conv-lowering=target=x86_64 -cir-to-llvm -o - 2>/dev/null \
// RUN: | mlir-translate -mlir-to-llvmir --allow-unregistered-dialect \
// RUN: | FileCheck %s --check-prefix=LLVM
+!s32i = !cir.int<s, 32>
!rec_FF = !cir.struct<"FF" {data !cir.float, data !cir.float}>
!rec_F3 = !cir.struct<"F3" {data !cir.float, data !cir.float, data !cir.float}>
!rec_UFloats = !cir.union<"UFloats" {data !cir.array<!cir.float x 2>, data !cir.array<!cir.float x 2>}>
@@ -15,6 +20,52 @@ module attributes {
#dlti.dl_entry<f64, dense<64>: vector<2xi64>>>
} {
+ // A 128-bit vector fills one xmm register and passes in its own type, so no
+ // coercion is inserted.
+ cir.func @v128(%arg0: !cir.vector<4 x !cir.float>) -> !cir.vector<4 x !cir.float> {
+ cir.return %arg0 : !cir.vector<4 x !cir.float>
+ }
+
+ // CHECK: cir.func{{.*}} @v128(%arg0: !cir.vector<4 x !cir.float>) -> !cir.vector<4 x !cir.float>
+ // CHECK-NEXT: cir.return %arg0
+
+ // The classifier does not look inside a vector when picking the type for an
+ // SSE eightbyte, so it falls back to double rather than to the two-element
+ // vector that take_ff below gets for the same eightbyte.
+ cir.func @v64(%arg0: !cir.vector<2 x !cir.float>) -> !cir.vector<2 x !cir.float> {
+ cir.return %arg0 : !cir.vector<2 x !cir.float>
+ }
+
+ // CHECK: cir.func{{.*}} @v64(%arg0: !cir.double) -> !cir.double
+
+ // A 32-bit vector is passed as an integer, matching what gcc does.
+ cir.func @v32(%arg0: !cir.vector<1 x !s32i>) -> !cir.vector<1 x !s32i> {
+ cir.return %arg0 : !cir.vector<1 x !s32i>
+ }
+
+ // CHECK: cir.func{{.*}} @v32(%arg0: !u32i) -> !u32i
+
+ // Without AVX a 256-bit vector exceeds the native vector size and goes to
+ // memory, and with AVX it stays in a register. The byval alignment is the
+ // greater of the vector's own alignment and 8.
+ cir.func @v256(%arg0: !cir.vector<8 x !cir.float>) {
+ cir.return
+ }
+
+ // CHECK: cir.func{{.*}} @v256(%arg0: !cir.ptr<!cir.vector<8 x !cir.float>> {{.*}}llvm.align = 32 : i64{{.*}}llvm.byval = !cir.vector<8 x !cir.float>{{.*}})
+ // AVX: cir.func{{.*}} @v256(%arg0: !cir.vector<8 x !cir.float>)
+ // AVX512: cir.func{{.*}} @v256(%arg0: !cir.vector<8 x !cir.float>)
+
+ // A 512-bit vector needs the top level before it reaches a register, so this
+ // is the case that distinguishes avx from avx512.
+ cir.func @v512(%arg0: !cir.vector<16 x !cir.float>) {
+ cir.return
+ }
+
+ // CHECK: cir.func{{.*}} @v512(%arg0: !cir.ptr<!cir.vector<16 x !cir.float>> {{.*}}llvm.align = 64 : i64{{.*}}llvm.byval = !cir.vector<16 x !cir.float>{{.*}})
+ // AVX: cir.func{{.*}} @v512(%arg0: !cir.ptr<!cir.vector<16 x !cir.float>> {{.*}}llvm.align = 64 : i64{{.*}}llvm.byval = !cir.vector<16 x !cir.float>{{.*}})
+ // AVX512: cir.func{{.*}} @v512(%arg0: !cir.vector<16 x !cir.float>)
+
// Two floats in one eightbyte classify SSE, and the coercion type the
// classifier picks for that eightbyte is a vector.
cir.func @take_ff(%arg0: !rec_FF) {
@@ -46,17 +97,29 @@ module attributes {
// CHECK: cir.func{{.*}} @take_f3(%arg0: !cir.vector<2 x !cir.float>, %arg1: !cir.float)
- // A union whose highest-aligned member is an all-float array reaches the same
- // vector coercion through the union path.
+ // A union whose widest member is an all-float array reaches the same vector
+ // coercion through the union path.
cir.func @take_union_floats(%arg0: !rec_UFloats) {
cir.return
}
// CHECK: cir.func{{.*}} @take_union_floats(%arg0: !cir.vector<2 x !cir.float>)
+
+ cir.func @call_v128(%arg0: !cir.vector<4 x !cir.float>) -> !cir.vector<4 x !cir.float> {
+ %0 = cir.call @v128(%arg0) : (!cir.vector<4 x !cir.float>) -> !cir.vector<4 x !cir.float>
+ cir.return %0 : !cir.vector<4 x !cir.float>
+ }
+
+ // CHECK: cir.func{{.*}} @call_v128(%arg0: !cir.vector<4 x !cir.float>) -> !cir.vector<4 x !cir.float>
+ // CHECK: cir.call @v128(%arg0) : (!cir.vector<4 x !cir.float>) -> !cir.vector<4 x !cir.float>
}
-// LLVM: define void @take_ff(<2 x float> %{{.+}})
+// LLVM: define <4 x float> @v128(<4 x float> %{{[^,)]+}})
+// LLVM: define double @v64(double %{{[^,)]+}})
+// LLVM: define i32 @v32(i32 %{{[^,)]+}})
+// LLVM: define void @v256(ptr noalias noundef byval(<8 x float>) align 32 %{{[^,)]+}})
+// LLVM: define void @take_ff(<2 x float> %{{[^,)]+}})
// LLVM: define <2 x float> @ret_ff()
-// LLVM: define void @take_farr(<2 x float> %{{.+}})
-// LLVM: define void @take_f3(<2 x float> %{{.+}}, float %{{.+}})
-// LLVM: define void @take_union_floats(<2 x float> %{{.+}})
+// LLVM: define void @take_farr(<2 x float> %{{[^,)]+}})
+// LLVM: define void @take_f3(<2 x float> %{{[^,)]+}}, float %{{[^,)]+}})
+// LLVM: define void @take_union_floats(<2 x float> %{{[^,)]+}})
>From 5d8843194e589a08e1712a53bceed31171df0509 Mon Sep 17 00:00:00 2001
From: Adam Smith <adams at nvidia.com>
Date: Wed, 12 Aug 2026 13:44:14 -0700
Subject: [PATCH 2/3] [CIR] Drop the call-conv lowering flag from the AVX test
The AVX test this branch adds asks for the pass with
`-clangir-enable-call-conv-lowering`, which no longer exists now that
this branch has picked up #215026 and #215117. The pass runs by
default, so the RUN lines do not need a flag at all and get the same
lowering without one.
Assisted-by: Cursor / claude-opus-5
---
.../CIR/CodeGen/call-conv-lowering-x86_64-avx.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/clang/test/CIR/CodeGen/call-conv-lowering-x86_64-avx.c b/clang/test/CIR/CodeGen/call-conv-lowering-x86_64-avx.c
index 0c733c34f4d9d..d632744e9c647 100644
--- a/clang/test/CIR/CodeGen/call-conv-lowering-x86_64-avx.c
+++ b/clang/test/CIR/CodeGen/call-conv-lowering-x86_64-avx.c
@@ -1,28 +1,28 @@
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fclangir -clangir-enable-call-conv-lowering -emit-cir %s -o %t.cir
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fclangir -emit-cir %s -o %t.cir
// RUN: FileCheck --check-prefixes=CIR,CIR-SSE --input-file=%t.cir %s
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fclangir -clangir-enable-call-conv-lowering -emit-llvm %s -o %t-cir.ll
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fclangir -emit-llvm %s -o %t-cir.ll
// RUN: FileCheck --check-prefixes=LLVM,LLVM-CIR-SSE --input-file=%t-cir.ll %s
// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm %s -o %t.ll
// RUN: FileCheck --check-prefixes=LLVM,LLVM-OGCG-SSE --input-file=%t.ll %s
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx -fclangir -clangir-enable-call-conv-lowering -emit-cir %s -o %t-avx.cir
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx -fclangir -emit-cir %s -o %t-avx.cir
// RUN: FileCheck --check-prefixes=CIR,CIR-AVX --input-file=%t-avx.cir %s
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx -fclangir -clangir-enable-call-conv-lowering -emit-llvm %s -o %t-avx-cir.ll
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx -fclangir -emit-llvm %s -o %t-avx-cir.ll
// RUN: FileCheck --check-prefixes=LLVM,LLVM-AVX,LLVM-CIR-AVX --input-file=%t-avx-cir.ll %s
// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx -emit-llvm %s -o %t-avx.ll
// RUN: FileCheck --check-prefixes=LLVM,LLVM-AVX,LLVM-OGCG-AVX --input-file=%t-avx.ll %s
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx512f -fclangir -clangir-enable-call-conv-lowering -emit-llvm %s -o %t-avx512-cir.ll
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx512f -fclangir -emit-llvm %s -o %t-avx512-cir.ll
// RUN: FileCheck --check-prefixes=LLVM,LLVM-AVX512 --input-file=%t-avx512-cir.ll %s
// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -target-feature +avx512f -emit-llvm %s -o %t-avx512.ll
// RUN: FileCheck --check-prefixes=LLVM,LLVM-AVX512 --input-file=%t-avx512.ll %s
-// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fclang-abi-compat=23 -fclangir -clangir-enable-call-conv-lowering -emit-llvm %s -o %t-compat23-cir.ll
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fclang-abi-compat=23 -fclangir -emit-llvm %s -o %t-compat23-cir.ll
// RUN: FileCheck --check-prefix=LLVM-CIR-PINNED --input-file=%t-compat23-cir.ll %s
// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fclang-abi-compat=23 -emit-llvm %s -o %t-compat23.ll
// RUN: FileCheck --check-prefix=LLVM-OGCG-PINNED --input-file=%t-compat23.ll %s
-// RUN: %clang_cc1 -triple x86_64-scei-ps4 -fclangir -clangir-enable-call-conv-lowering -emit-llvm %s -o %t-ps4-cir.ll
+// RUN: %clang_cc1 -triple x86_64-scei-ps4 -fclangir -emit-llvm %s -o %t-ps4-cir.ll
// RUN: FileCheck --check-prefix=LLVM-CIR-PINNED --input-file=%t-ps4-cir.ll %s
// RUN: %clang_cc1 -triple x86_64-scei-ps4 -emit-llvm %s -o %t-ps4.ll
// RUN: FileCheck --check-prefix=LLVM-OGCG-PINNED --input-file=%t-ps4.ll %s
>From 394b3b5dc71903ea63fd4a07dce6c5a5f49240a2 Mon Sep 17 00:00:00 2001
From: Adam Smith <adams at nvidia.com>
Date: Wed, 12 Aug 2026 15:14:54 -0700
Subject: [PATCH 3/3] [CIR] Search the feature list once and assert on
multiversioning
Review feedback on #215118.
funcAvxLevel walked the whole feature list once per name and was called
twice, so it now makes a single pass.
It also has no equivalent of the `hasAttr<TargetAttr>()` gate classic
uses. A multiversioned function's feature list carries the same `+avx`
entry a plain `target` attribute produces, so this would take a raised
level where classic keeps the module's. CIRGen rejects multiversioning
today, so an assert on opFuncMultiVersioning marks the gate that has to
go in when that changes.
Comments naming a function or file in classic CodeGen came out. They go
stale when that code moves, and one already had, claiming a declaration
carries no features. x86TargetAttrAvx is now allowsX86TargetAttrAvx,
after the function that produces it.
Assisted-by: Cursor / claude-opus-5
---
clang/include/clang/CIR/Dialect/Passes.h | 2 +-
clang/include/clang/CIR/Dialect/Passes.td | 2 +-
.../Transforms/CallConvLoweringPass.cpp | 44 +++++++++----------
clang/lib/CIR/Lowering/CIRPasses.cpp | 14 +++---
4 files changed, 28 insertions(+), 34 deletions(-)
diff --git a/clang/include/clang/CIR/Dialect/Passes.h b/clang/include/clang/CIR/Dialect/Passes.h
index 6c8bec15c3838..674f2180c27ab 100644
--- a/clang/include/clang/CIR/Dialect/Passes.h
+++ b/clang/include/clang/CIR/Dialect/Passes.h
@@ -38,7 +38,7 @@ std::unique_ptr<Pass> createTargetLoweringPass();
std::unique_ptr<Pass> createCallConvLoweringPass();
std::unique_ptr<Pass> createCallConvLoweringPass(
cir::CallConvTarget target, llvm::abi::X86AVXABILevel x86AvxAbiLevel,
- bool x86TargetAttrAvx, const llvm::abi::ABICompatInfo &x86AbiCompat);
+ bool allowsX86TargetAttrAvx, const llvm::abi::ABICompatInfo &x86AbiCompat);
std::unique_ptr<Pass> createHoistAllocasPass();
std::unique_ptr<Pass> createLoweringPreparePass();
std::unique_ptr<Pass> createLoweringPreparePass(clang::ASTContext *astCtx);
diff --git a/clang/include/clang/CIR/Dialect/Passes.td b/clang/include/clang/CIR/Dialect/Passes.td
index bc1b99da01440..b983cfe59112a 100644
--- a/clang/include/clang/CIR/Dialect/Passes.td
+++ b/clang/include/clang/CIR/Dialect/Passes.td
@@ -260,7 +260,7 @@ def CallConvLowering : Pass<"cir-call-conv-lowering", "mlir::ModuleOp"> {
clEnumValN(llvm::abi::X86AVXABILevel::AVX, "avx", "AVX"),
clEnumValN(llvm::abi::X86AVXABILevel::AVX512, "avx512", "AVX512")
)}]>,
- Option<"x86TargetAttrAvx", "x86-target-attr-avx", "bool",
+ Option<"allowsX86TargetAttrAvx", "x86-target-attr-avx", "bool",
/*default=*/"true",
"Let a function's own target features raise its AVX ABI level "
"above x86-avx-abi-level">,
diff --git a/clang/lib/CIR/Dialect/Transforms/CallConvLoweringPass.cpp b/clang/lib/CIR/Dialect/Transforms/CallConvLoweringPass.cpp
index 5f25e827b9cd2..6fcd149f61cd4 100644
--- a/clang/lib/CIR/Dialect/Transforms/CallConvLoweringPass.cpp
+++ b/clang/lib/CIR/Dialect/Transforms/CallConvLoweringPass.cpp
@@ -44,6 +44,7 @@
#include "mlir/Pass/Pass.h"
#include "clang/CIR/Dialect/IR/CIRDialect.h"
#include "clang/CIR/Dialect/Passes.h"
+#include "clang/CIR/MissingFeatures.h"
#include "llvm/ABI/FunctionInfo.h"
#include "llvm/ABI/TargetInfo.h"
#include "llvm/ABI/Types.h"
@@ -507,32 +508,29 @@ static std::optional<FunctionClassification> classifyX86_64Signature(
return fc;
}
-/// The AVX level in force at \p func, following getEffectiveX86AVXABILevel in
-/// clang/lib/CodeGen/Targets/X86.cpp. Classic reads the target attribute
-/// itself, where this reads the feature list CIRGen recorded from it. Two
-/// consequences: a declaration gets \p base, CIRGen recording features only on
-/// definitions, and target_clones and cpu_specific will need excluding once
-/// multiversioning is implemented, classic leaving those at the module level.
+/// The AVX level to classify \p func at: \p base, raised if the function's own
+/// recorded feature list enables a wider vector.
static llvm::abi::X86AVXABILevel funcAvxLevel(cir::FuncOp func,
llvm::abi::X86AVXABILevel base) {
+ // Only a `target` attribute may raise the level. A multiversioned function
+ // carries a raised feature list too, and must stay at the module's level.
+ assert(!cir::MissingFeatures::opFuncMultiVersioning());
+
auto features = func->getAttrOfType<mlir::StringAttr>("cir.target-features");
if (!features)
return base;
// A '-' entry disables the feature, so match a whole '+' entry rather than
- // searching for the name.
- auto enabled = [&](llvm::StringRef name) {
- for (llvm::StringRef feature : llvm::split(features.getValue(), ','))
- if (feature.consume_front("+") && feature == name)
- return true;
- return false;
- };
- // avx512f implies avx, so both entries are present. Test the wider name
- // first or the AVX512 branch is unreachable.
- if (enabled("avx512f"))
- return std::max(base, llvm::abi::X86AVXABILevel::AVX512);
- if (enabled("avx"))
- return std::max(base, llvm::abi::X86AVXABILevel::AVX);
- return base;
+ // searching for the name. avx512f implies avx, so both entries are present:
+ // return on the wider name so one pass suffices.
+ bool avx = false;
+ for (llvm::StringRef feature : llvm::split(features.getValue(), ',')) {
+ if (!feature.consume_front("+"))
+ continue;
+ if (feature == "avx512f")
+ return std::max(base, llvm::abi::X86AVXABILevel::AVX512);
+ avx |= feature == "avx";
+ }
+ return avx ? std::max(base, llvm::abi::X86AVXABILevel::AVX) : base;
}
/// Classify a cir.func for x86_64 SysV using the LLVM ABI library. Returns
@@ -727,7 +725,7 @@ void CallConvLoweringPass::runOnOperation() {
};
llvm::abi::X86AVXABILevel baseAvxLevel = x86AvxAbiLevel.getValue();
auto avxLevelFor = [&](cir::FuncOp func) -> llvm::abi::X86AVXABILevel {
- if (!x86TargetAttrAvx || !func)
+ if (!allowsX86TargetAttrAvx || !func)
return baseAvxLevel;
return funcAvxLevel(func, baseAvxLevel);
};
@@ -957,10 +955,10 @@ std::unique_ptr<Pass> mlir::createCallConvLoweringPass() {
std::unique_ptr<Pass> mlir::createCallConvLoweringPass(
cir::CallConvTarget target, llvm::abi::X86AVXABILevel x86AvxAbiLevel,
- bool x86TargetAttrAvx, const llvm::abi::ABICompatInfo &x86AbiCompat) {
+ bool allowsX86TargetAttrAvx, const llvm::abi::ABICompatInfo &x86AbiCompat) {
CallConvLoweringOptions options;
options.target = target;
options.x86AvxAbiLevel = x86AvxAbiLevel;
- options.x86TargetAttrAvx = x86TargetAttrAvx;
+ options.allowsX86TargetAttrAvx = allowsX86TargetAttrAvx;
return std::make_unique<CallConvLoweringPass>(options, x86AbiCompat);
}
diff --git a/clang/lib/CIR/Lowering/CIRPasses.cpp b/clang/lib/CIR/Lowering/CIRPasses.cpp
index 972dc6d1a4a8b..1d1fdaf42aaa4 100644
--- a/clang/lib/CIR/Lowering/CIRPasses.cpp
+++ b/clang/lib/CIR/Lowering/CIRPasses.cpp
@@ -29,8 +29,8 @@ static CallConvTarget getCallConvTarget(const llvm::Triple &triple) {
return CallConvTarget::None;
}
-/// The AVX level the classifier uses to size a native vector. Read from the
-/// target ABI name, as CodeGenModule does.
+/// The AVX level the classifier uses to size a native vector, read from the
+/// target ABI name.
static llvm::abi::X86AVXABILevel getX86AVXABILevel(llvm::StringRef abi) {
if (abi == "avx512")
return llvm::abi::X86AVXABILevel::AVX512;
@@ -40,9 +40,8 @@ static llvm::abi::X86AVXABILevel getX86AVXABILevel(llvm::StringRef abi) {
}
/// Whether `__attribute__((target(...)))` on a function may raise its AVX ABI
-/// level above the command line's. Mirrors getEffectiveX86AVXABILevel in
-/// clang/lib/CodeGen/Targets/X86.cpp, which keeps a target that opts out, and
-/// any ABI older than the rule, at the module level.
+/// level above the command line's. A target that opts out, and any ABI older
+/// than the rule, stay at the module level.
static bool allowsX86TargetAttrAvx(const clang::ASTContext &astContext) {
return !astContext.getTargetInfo().getTriple().isPS() &&
astContext.getLangOpts().getClangABICompat() >
@@ -53,10 +52,7 @@ static bool allowsX86TargetAttrAvx(const clang::ASTContext &astContext) {
/// requested compatibility version. Every flag defaults to true in the ABI
/// library, which is not what any target computes: Clang11Compat is false for a
/// modern Linux target, so leaving it at the default classifies a union larger
-/// than an eightbyte as though every member spanned its size. Mirrors the
-/// predicates in clang/lib/CodeGen/Targets/X86.cpp and the derivation in
-/// CodeGenModule::getLLVMABITargetInfo, which computes the same five flags for
-/// the classic path.
+/// than an eightbyte as though every member spanned its size.
static llvm::abi::ABICompatInfo
getX86ABICompatInfo(const clang::ASTContext &astContext) {
const llvm::Triple &triple = astContext.getTargetInfo().getTriple();
More information about the cfe-commits
mailing list