[clang] [clang] Compile always_inline functions with the target features of their callers (PR #241171)
via cfe-commits
cfe-commits at lists.llvm.org
Wed Nov 11 00:11:37 PST 2026
https://github.com/higher-performance created https://github.com/llvm/llvm-project/pull/241171
AI-generated fix for #129398.
>From d479112aba26e86024f1a72b02ba046b6cc55df2 Mon Sep 17 00:00:00 2001
From: higher-performance <higher.performance.github at gmail.com>
Date: Wed, 11 Nov 2026 03:10:18 -0500
Subject: [PATCH] [clang] Compile always_inline functions with the target
features of their callers
---
clang/docs/ReleaseNotes.md | 9 +
clang/include/clang/Basic/AttrDocs.td | 22 ++
clang/lib/CodeGen/CGCall.cpp | 47 ++-
clang/lib/CodeGen/CodeGenFunction.cpp | 5 +-
clang/lib/CodeGen/CodeGenModule.cpp | 345 ++++++++++++++++++
clang/lib/CodeGen/CodeGenModule.h | 48 +++
clang/lib/CodeGen/Targets/X86.cpp | 12 +-
.../CodeGenCXX/target-features-inferred.cpp | 113 ++++++
8 files changed, 574 insertions(+), 27 deletions(-)
create mode 100644 clang/test/CodeGenCXX/target-features-inferred.cpp
diff --git a/clang/docs/ReleaseNotes.md b/clang/docs/ReleaseNotes.md
index 0f9e7e01636d71..04f8a0e240cb8c 100644
--- a/clang/docs/ReleaseNotes.md
+++ b/clang/docs/ReleaseNotes.md
@@ -436,6 +436,15 @@ documentation](CIR/index.md) for more information regarding ClangIR's status and
### Attribute Changes in Clang
+- An `always_inline` function without a `target` attribute is now compiled with
+ the target features that its callers have in common, provided that other
+ translation units cannot call it (for example because it is `static`,
+ `inline` or a template) and that its only uses are direct calls. Generic
+ helpers can thus be shared between functions with different `target`
+ attributes and can call target intrinsics without naming the features
+ themselves; previously this was rejected with "always_inline function requires
+ target feature" and "changes the ABI" errors.
+
- Clang now properly propagates attributes on class and variable templates to their redeclarations, which will result in redeclarations not interfering with diagnostics. (#GH209812)
- Clang now recognizes the `[[gnu::flag_enum]]` attribute and treats it equivalent to `[[clang::flag_enum]]`
diff --git a/clang/include/clang/Basic/AttrDocs.td b/clang/include/clang/Basic/AttrDocs.td
index 2300c8456ef164..575a02207dab41 100644
--- a/clang/include/clang/Basic/AttrDocs.td
+++ b/clang/include/clang/Basic/AttrDocs.td
@@ -3426,6 +3426,28 @@ the front end.
Note that this attribute does not apply transitively to nested functions such
as blocks or C++ lambdas.
+A function that is marked `always_inline` and has no `target` attribute is
+compiled with the target features that all of its callers have in common, so
+that it can be inlined into each of them. It can therefore call `always_inline`
+functions that require more features than were enabled on the command line,
+such as target intrinsics, as long as every function that calls it enables
+them:
+
+```c++
+__attribute__((always_inline))
+static __m256 twice(__m256 x) { return _mm256_add_ps(x, x); }
+
+__attribute__((target("avx")))
+__m256 twice_avx(__m256 x) { return twice(x); } // OK, twice is AVX code here.
+```
+
+This only holds for functions that cannot run in any other way: the definition
+must not be callable from other translation units (as is the case for `static`
+and `inline` functions and for implicit template instantiations), its only uses
+must be direct calls from other functions of the translation unit, and these
+must all be compiled for the same CPU. Any other `always_inline` function is
+compiled with the features that were passed on the command line.
+
Additionally, this attribute supports function multiversioning for ELF based
x86/x86-64 targets, which can be used to create multiple implementations of the
same function that will be resolved at runtime based on the priority of their
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index 1d7f9f742b87e9..0924671fb56b5b 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5719,28 +5719,32 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
llvm::FunctionType *IRFuncTy = getTypes().GetFunctionType(CallInfo);
const Decl *TargetDecl = Callee.getAbstractInfo().getCalleeDecl().getDecl();
- if (const FunctionDecl *FD = dyn_cast_or_null<FunctionDecl>(TargetDecl)) {
- // We can only guarantee that a function is called from the correct
- // context/function based on the appropriate target attributes,
- // so only check in the case where we have both always_inline and target
- // since otherwise we could be making a conditional call after a check for
- // the proper cpu features (and it won't cause code generation issues due to
- // function based code generation).
- if ((TargetDecl->hasAttr<AlwaysInlineAttr>() &&
- (TargetDecl->hasAttr<TargetAttr>() ||
- (CurFuncDecl && CurFuncDecl->hasAttr<TargetAttr>()))) ||
- (CurFuncDecl && CurFuncDecl->hasAttr<FlattenAttr>() &&
- (CurFuncDecl->hasAttr<TargetAttr>() ||
- TargetDecl->hasAttr<TargetAttr>())))
- checkTargetFeatures(Loc, FD);
- }
-
- // Some architectures (such as x86-64) have the ABI changed based on
- // attribute-target/features. Give them a chance to diagnose.
const FunctionDecl *CallerDecl = dyn_cast_or_null<FunctionDecl>(CurCodeDecl);
const FunctionDecl *CalleeDecl = dyn_cast_or_null<FunctionDecl>(TargetDecl);
- CGM.getTargetCodeGenInfo().checkFunctionCallABI(CGM, Loc, CallerDecl,
- CalleeDecl, CallArgs, RetTy);
+ // We can only guarantee that a function is called from the correct
+ // context/function based on the appropriate target attributes,
+ // so only check in the case where we have both always_inline and target
+ // since otherwise we could be making a conditional call after a check for
+ // the proper cpu features (and it won't cause code generation issues due to
+ // function based code generation).
+ bool CheckFeatures =
+ CalleeDecl && ((TargetDecl->hasAttr<AlwaysInlineAttr>() &&
+ (TargetDecl->hasAttr<TargetAttr>() ||
+ (CurFuncDecl && CurFuncDecl->hasAttr<TargetAttr>()))) ||
+ (CurFuncDecl && CurFuncDecl->hasAttr<FlattenAttr>() &&
+ (CurFuncDecl->hasAttr<TargetAttr>() ||
+ TargetDecl->hasAttr<TargetAttr>())));
+ bool DeferTargetChecks =
+ CGM.shouldDeferCallTargetChecks(CallerDecl, CalleeDecl);
+ if (!DeferTargetChecks) {
+ if (CheckFeatures)
+ checkTargetFeatures(Loc, CalleeDecl);
+
+ // Some architectures (such as x86-64) have the ABI changed based on
+ // attribute-target/features. Give them a chance to diagnose.
+ CGM.getTargetCodeGenInfo().checkFunctionCallABI(
+ CGM, Loc, CallerDecl, CalleeDecl, CallArgs, RetTy);
+ }
// 1. Set up the arguments.
@@ -6431,6 +6435,9 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
BundleList);
EmitBlock(Cont);
}
+ if (DeferTargetChecks)
+ CGM.deferCallTargetChecks(Loc, CallerDecl, CalleeDecl, CI, CallArgs, RetTy,
+ CheckFeatures);
if (CI->getCalledFunction() && CI->getCalledFunction()->hasName() &&
CI->getCalledFunction()->getName().starts_with("_Z4sqrt")) {
SetSqrtFPAccuracy(CI);
diff --git a/clang/lib/CodeGen/CodeGenFunction.cpp b/clang/lib/CodeGen/CodeGenFunction.cpp
index 33434526d86325..a3ddc6ae128335 100644
--- a/clang/lib/CodeGen/CodeGenFunction.cpp
+++ b/clang/lib/CodeGen/CodeGenFunction.cpp
@@ -1499,7 +1499,10 @@ void CodeGenFunction::GenerateCode(GlobalDecl GD, llvm::Function *Fn,
FunctionArgList Args;
QualType ResTy = BuildFunctionArgList(GD, Args);
- CGM.getTargetCodeGenInfo().checkFunctionABI(CGM, FD);
+ if (CGM.shouldDeferCallTargetChecks(/*Caller=*/nullptr, FD))
+ CGM.deferFunctionABICheck(FD);
+ else
+ CGM.getTargetCodeGenInfo().checkFunctionABI(CGM, FD);
if (FD->isInlineBuiltinDeclaration()) {
// When generating code for a builtin with an inline declaration, use a
diff --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index bdfbdeb1059384..f6e926d4ac27a4 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -49,11 +49,13 @@
#include "clang/Basic/Version.h"
#include "clang/CodeGen/BackendUtil.h"
#include "clang/CodeGen/ConstantInitBuilder.h"
+#include "clang/CodeGenUtils/FunctionUtils.h"
#include "clang/CodeGenUtils/ModuleUtils.h"
#include "clang/Lex/Preprocessor.h"
#include "llvm/ABI/IRTypeMapper.h"
#include "llvm/ABI/TargetInfo.h"
#include "llvm/ADT/APFloat.h"
+#include "llvm/ADT/BitVector.h"
#include "llvm/ADT/STLExtras.h"
#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringSwitch.h"
@@ -1358,6 +1360,8 @@ void CodeGenModule::Release() {
addCompilerUsedGlobal(GV);
}
emitLLVMUsed();
+ // This has to know every use of the functions that have been emitted.
+ emitDeferredCallTargetChecks();
if (SanStats)
SanStats->finish();
@@ -3617,6 +3621,347 @@ bool CodeGenModule::GetCPUAndFeaturesAttributes(GlobalDecl GD,
return AddedAttr;
}
+namespace {
+/// The target CPU and features that the IR attributes of a function give it.
+struct FunctionTargetFeatures {
+ StringRef CPU;
+ std::string Features;
+ llvm::BitVector Enabled;
+ llvm::BitVector Disabled;
+};
+} // namespace
+
+/// Whether \p FD is declared such that it could be compiled with the target
+/// features of its callers. Whether it is also only used in ways that allow
+/// this is not known before the end of the module.
+static bool mayInferTargetFeatures(const FunctionDecl *FD) {
+ FD = FD->getMostRecentDecl();
+ // A flatten function is checked against the features of all of its callees,
+ // which the inference does not model.
+ return FD->hasAttr<AlwaysInlineAttr>() && !FD->getBuiltinID() &&
+ !FD->hasAttr<TargetAttr>() && !FD->hasAttr<TargetVersionAttr>() &&
+ !FD->hasAttr<TargetClonesAttr>() && !FD->hasAttr<CPUSpecificAttr>() &&
+ !FD->hasAttr<CPUDispatchAttr>() && !FD->hasAttr<FlattenAttr>();
+}
+
+bool CodeGenModule::shouldDeferCallTargetChecks(
+ const FunctionDecl *Caller, const FunctionDecl *Callee) const {
+ // HIP stdpar device compilations leave the feature checks to the backend.
+ return Callee && !(LangOpts.HIPStdPar && LangOpts.CUDAIsDevice) &&
+ ((Caller && mayInferTargetFeatures(Caller)) ||
+ mayInferTargetFeatures(Callee));
+}
+
+void CodeGenModule::deferCallTargetChecks(
+ SourceLocation Loc, const FunctionDecl *Caller, const FunctionDecl *Callee,
+ llvm::CallBase *Call, const CallArgList &Args, QualType ReturnType,
+ bool CheckFeatures) {
+ DeferredCallTargetChecks.push_back(
+ {Loc, Caller, Callee, Call, ReturnType,
+ static_cast<unsigned>(DeferredCallArgTypes.size()),
+ static_cast<unsigned>(Args.size()), CheckFeatures});
+ for (const CallArg &Arg : Args)
+ DeferredCallArgTypes.push_back(Arg.getType());
+}
+
+void CodeGenModule::getFunctionFeatureMap(llvm::StringMap<bool> &FeatureMap,
+ const FunctionDecl *FD) const {
+ auto Inferred = FD ? InferredTargetFeatures.find(FD->getCanonicalDecl())
+ : InferredTargetFeatures.end();
+ if (Inferred != InferredTargetFeatures.end())
+ FeatureMap = Inferred->second;
+ else
+ Context.getFunctionFeatureMap(FeatureMap, FD);
+}
+
+void CodeGenModule::emitDeferredCallTargetChecks() {
+ // An always_inline function without a target attribute is compiled with the
+ // target features that its callers have in common, provided that nothing but
+ // direct calls from functions of this module can run it. It can then be
+ // inlined into all of its callers, and the always_inline functions that it
+ // calls can be inlined into it as long as each of its callers has the
+ // features that they need. Any other function keeps the features it was
+ // declared with.
+ struct RequiredFeatures {
+ llvm::BitVector Features;
+ /// Those that target attributes name, which diagnostics mention first.
+ SmallVector<unsigned, 4> Named;
+ };
+ struct InferringFunction {
+ llvm::SmallPtrSet<llvm::Function *, 2> Fns;
+ std::optional<FunctionTargetFeatures> Features;
+ /// What the always_inline functions that it calls need.
+ RequiredFeatures Required;
+ bool Valid = true;
+ };
+ llvm::MapVector<const FunctionDecl *, InferringFunction> Inferring;
+ llvm::DenseMap<const llvm::CallBase *, const FunctionDecl *> CalleeOfCall;
+
+ auto LiveCall = [](const DeferredCallTargetCheck &Check) {
+ llvm::Value *Call = Check.Call;
+ return dyn_cast_or_null<llvm::CallBase>(Call);
+ };
+ auto CalledFunction = [](const llvm::CallBase *Call) {
+ return Call ? dyn_cast<llvm::Function>(Call->getCalledOperand()) : nullptr;
+ };
+
+ for (const DeferredCallTargetCheck &Check : DeferredCallTargetChecks) {
+ llvm::CallBase *Call = LiveCall(Check);
+ llvm::Function *Fn = CalledFunction(Call);
+ // Nothing checks the target features of code outside of functions, so a
+ // call from there counts as an unknown use below.
+ if (!Fn || !Check.Caller || !mayInferTargetFeatures(Check.Callee))
+ continue;
+ const FunctionDecl *Callee = Check.Callee->getCanonicalDecl();
+ Inferring[Callee].Fns.insert(Fn);
+ CalleeOfCall[Call] = Callee;
+ }
+
+ llvm::DenseMap<const llvm::Function *, const FunctionDecl *> DeclOfFn;
+ for (auto &Entry : Inferring) {
+ const FunctionDecl *FD = Entry.first;
+ InferringFunction &Callee = Entry.second;
+ const FunctionDecl *Def = FD->getDefinition();
+ // The body of an available_externally function is not the one that runs
+ // when a call to it is not inlined.
+ GVALinkage Linkage =
+ Def ? Context.GetGVALinkageForFunction(Def) : GVA_StrongExternal;
+ Callee.Valid = (Linkage == GVA_Internal || Linkage == GVA_DiscardableODR) &&
+ llvm::all_of(Callee.Fns, [&](const llvm::Function *Fn) {
+ return !Fn->isDeclaration() &&
+ llvm::all_of(Fn->uses(), [&](const llvm::Use &U) {
+ const auto *Call =
+ dyn_cast<llvm::CallBase>(U.getUser());
+ return Call && Call->isCallee(&U) &&
+ CalleeOfCall.lookup(Call) == FD;
+ });
+ });
+ if (Callee.Valid)
+ for (const llvm::Function *Fn : Callee.Fns)
+ DeclOfFn[Fn] = FD;
+ }
+ auto FindDecl = [&](const FunctionDecl *FD) -> InferringFunction * {
+ auto It = FD ? Inferring.find(FD->getCanonicalDecl()) : Inferring.end();
+ return It != Inferring.end() && It->second.Valid ? &It->second : nullptr;
+ };
+ auto Find = [&](const llvm::Function *Fn) {
+ return FindDecl(DeclOfFn.lookup(Fn));
+ };
+
+ llvm::StringMap<unsigned> FeatureIDs;
+ SmallVector<StringRef, 0> FeatureNames;
+ auto FeatureID = [&](StringRef Name) {
+ auto [It, Inserted] = FeatureIDs.try_emplace(Name, FeatureNames.size());
+ if (Inserted)
+ FeatureNames.push_back(It->getKey());
+ return It->second;
+ };
+ auto Set = [](llvm::BitVector &Bits, unsigned ID) {
+ if (Bits.size() <= ID)
+ Bits.resize(ID + 1);
+ Bits.set(ID);
+ };
+ auto FeatureString = [&](const FunctionTargetFeatures &Features) {
+ std::vector<std::string> Strings;
+ for (unsigned ID : Features.Enabled.set_bits())
+ Strings.push_back(("+" + FeatureNames[ID]).str());
+ for (unsigned ID : Features.Disabled.set_bits())
+ Strings.push_back(("-" + FeatureNames[ID]).str());
+ llvm::sort(Strings);
+ return llvm::join(Strings, ",");
+ };
+
+ // The IR attributes of the calling functions are what decides whether a call
+ // can be inlined, and unlike the declarations they also cover function
+ // versions and thunks.
+ llvm::DenseMap<std::pair<StringRef, StringRef>, FunctionTargetFeatures>
+ ParsedFeatures;
+ auto FeaturesOf =
+ [&](const llvm::Function *Fn) -> const FunctionTargetFeatures * {
+ if (InferringFunction *Caller = Find(Fn))
+ return Caller->Features ? &*Caller->Features : nullptr;
+ StringRef CPU = Fn->getFnAttribute("target-cpu").getValueAsString();
+ StringRef Features =
+ Fn->getFnAttribute("target-features").getValueAsString();
+ auto [It, Inserted] = ParsedFeatures.try_emplace(std::pair(CPU, Features));
+ if (Inserted) {
+ It->second.CPU = CPU;
+ It->second.Features = Features.str();
+ for (StringRef Feature : llvm::split(Features, ','))
+ if (!Feature.empty())
+ Set(Feature[0] == '+' ? It->second.Enabled : It->second.Disabled,
+ FeatureID(Feature.substr(1)));
+ }
+ return &It->second;
+ };
+
+ for (bool Retry = true; Retry;) {
+ Retry = false;
+ for (auto &Entry : Inferring)
+ Entry.second.Features.reset();
+ for (bool Changed = true; Changed;) {
+ Changed = false;
+ for (const DeferredCallTargetCheck &Check : DeferredCallTargetChecks) {
+ llvm::CallBase *Call = LiveCall(Check);
+ InferringFunction *Callee = Find(CalledFunction(Call));
+ const FunctionTargetFeatures *Caller =
+ Callee ? FeaturesOf(Call->getFunction()) : nullptr;
+ if (!Caller)
+ continue;
+ if (!Callee->Features) {
+ Callee->Features = *Caller;
+ Changed = true;
+ } else if (Callee->Features->CPU != Caller->CPU) {
+ // The features that only the CPUs imply are not known here, so
+ // there is no telling which of them the callers have in common.
+ Callee->Valid = false;
+ Retry = true;
+ } else if (Callee->Features->Enabled.test(Caller->Enabled) ||
+ Caller->Disabled.test(Callee->Features->Disabled)) {
+ // A feature that only some callers mention is left to the CPU,
+ // which is the same for all of them.
+ Callee->Features->Enabled &= Caller->Enabled;
+ Callee->Features->Disabled |= Caller->Disabled;
+ Callee->Features->Features = FeatureString(*Callee->Features);
+ Changed = true;
+ }
+ }
+ }
+ // Functions that only call each other are not reachable from any function
+ // with known features.
+ for (auto &Entry : Inferring)
+ if (Entry.second.Valid && !Entry.second.Features) {
+ Entry.second.Valid = false;
+ Retry = true;
+ }
+ }
+
+ llvm::DenseMap<const FunctionDecl *, RequiredFeatures> DeclFeatures;
+ auto FeaturesOfDecl =
+ [&](const FunctionDecl *FD) -> const RequiredFeatures & {
+ auto [It, Inserted] = DeclFeatures.try_emplace(FD);
+ if (Inserted) {
+ llvm::StringMap<bool> FeatureMap;
+ Context.getFunctionFeatureMap(FeatureMap, FD);
+ for (const auto &Feature : FeatureMap)
+ if (Feature.getValue())
+ Set(It->second.Features, FeatureID(Feature.getKey()));
+ if (const auto *TD = FD->getAttr<TargetAttr>();
+ TD && !FD->isMultiVersion())
+ for (const std::string &Feature :
+ Context.filterFunctionTargetAttrs(TD).Features)
+ if (Feature[0] == '+' &&
+ FeatureMap.lookup(StringRef(Feature).substr(1)))
+ It->second.Named.push_back(FeatureID(StringRef(Feature).substr(1)));
+ }
+ return It->second;
+ };
+
+ for (bool Changed = true; Changed;) {
+ Changed = false;
+ // Callees are mostly emitted after their callers, so walking backwards
+ // tends to converge in a single pass.
+ for (const DeferredCallTargetCheck &Check :
+ llvm::reverse(DeferredCallTargetChecks)) {
+ llvm::CallBase *Call = LiveCall(Check);
+ InferringFunction *Caller = Call ? Find(Call->getFunction()) : nullptr;
+ if (!Caller || !Check.Callee->hasAttr<AlwaysInlineAttr>())
+ continue;
+ InferringFunction *Callee = Find(CalledFunction(Call));
+ const RequiredFeatures &Required =
+ Callee ? Callee->Required : FeaturesOfDecl(Check.Callee);
+ if (Required.Features.test(Caller->Required.Features)) {
+ Caller->Required.Features |= Required.Features;
+ Changed = true;
+ }
+ for (unsigned ID : Required.Named)
+ if (!llvm::is_contained(Caller->Required.Named, ID)) {
+ Caller->Required.Named.push_back(ID);
+ Changed = true;
+ }
+ }
+ }
+
+ for (auto &Entry : Inferring) {
+ InferringFunction &Callee = Entry.second;
+ if (!Callee.Valid)
+ continue;
+ for (llvm::Function *Fn : Callee.Fns) {
+ if (Fn->getFnAttribute("target-cpu").getValueAsString() ==
+ Callee.Features->CPU &&
+ Fn->getFnAttribute("target-features").getValueAsString() ==
+ Callee.Features->Features)
+ continue;
+ Fn->removeFnAttr("target-cpu");
+ Fn->removeFnAttr("target-features");
+ if (!Callee.Features->CPU.empty())
+ Fn->addFnAttr("target-cpu", Callee.Features->CPU);
+ if (!Callee.Features->Features.empty())
+ Fn->addFnAttr("target-features", Callee.Features->Features);
+ // Other translation units may give their definitions of the function
+ // other features, so the linker must not merge them with this one.
+ if (!Fn->hasLocalLinkage()) {
+ Fn->setComdat(nullptr);
+ Fn->setLinkage(llvm::Function::InternalLinkage);
+ }
+ llvm::StringMap<bool> &FeatureMap = InferredTargetFeatures[Entry.first];
+ for (unsigned ID : Callee.Features->Enabled.set_bits())
+ FeatureMap[FeatureNames[ID]] = true;
+ }
+ }
+
+ for (const DeferredCallTargetCheck &Check : DeferredCallTargetChecks) {
+ llvm::CallBase *Call = LiveCall(Check);
+ InferringFunction *Caller =
+ Call ? Find(Call->getFunction()) : FindDecl(Check.Caller);
+ InferringFunction *Callee =
+ Call ? Find(CalledFunction(Call)) : FindDecl(Check.Callee);
+ // A call that was removed again is neither inlined nor run.
+ if (!Call && (Caller || Callee))
+ continue;
+
+ // What an inferring caller needs is checked where it is called, up to the
+ // first caller that does not infer its features.
+ if (Callee && !Caller) {
+ llvm::BitVector Missing = Callee->Required.Features;
+ Missing.reset(FeaturesOfDecl(Check.Caller).Features);
+ if (Missing.any()) {
+ StringRef Feature;
+ for (unsigned ID : Missing.set_bits())
+ if (Feature.empty() || FeatureNames[ID] < Feature)
+ Feature = FeatureNames[ID];
+ if (const unsigned *Named =
+ llvm::find_if(Callee->Required.Named,
+ [&](unsigned ID) { return Missing.test(ID); });
+ Named != Callee->Required.Named.end())
+ Feature = FeatureNames[*Named];
+ getDiags().Report(Check.Loc, diag::err_function_needs_feature)
+ << Check.Caller->getDeclName() << Check.Callee->getDeclName()
+ << Feature;
+ }
+ } else if (!Caller && Check.CheckFeatures) {
+ CodeGenUtils::checkTargetFeatures(Context, getDiags(), getLangOpts(),
+ Check.Loc, Check.Caller, Check.Callee);
+ }
+
+ // The targets only look at the types of the arguments.
+ CallArgList Args;
+ for (QualType Type : ArrayRef<QualType>(DeferredCallArgTypes)
+ .slice(Check.FirstArgType, Check.NumArgTypes))
+ Args.add(RValue::get(nullptr), Type);
+ getTargetCodeGenInfo().checkFunctionCallABI(
+ *this, Check.Loc, Check.Caller, Check.Callee, Args, Check.ReturnType);
+ }
+
+ for (const FunctionDecl *FD : DeferredFunctionABIChecks)
+ getTargetCodeGenInfo().checkFunctionABI(*this, FD);
+
+ DeferredCallTargetChecks.clear();
+ DeferredCallArgTypes.clear();
+ DeferredFunctionABIChecks.clear();
+ InferredTargetFeatures.clear();
+}
+
void CodeGenModule::setNonAliasAttributes(GlobalDecl GD,
llvm::GlobalObject *GO) {
const Decl *D = GD.getDecl();
diff --git a/clang/lib/CodeGen/CodeGenModule.h b/clang/lib/CodeGen/CodeGenModule.h
index e2bd6bc4701fda..1b38d91e397244 100644
--- a/clang/lib/CodeGen/CodeGenModule.h
+++ b/clang/lib/CodeGen/CodeGenModule.h
@@ -101,6 +101,7 @@ class CGOpenMPRuntime;
class CGCUDARuntime;
class CGHLSLRuntime;
class CGFunctionInfo;
+class CallArgList;
class CoverageMappingModuleGen;
class QualTypeMapper;
class TargetCodeGenInfo;
@@ -424,6 +425,28 @@ class CodeGenModule : public CodeGenTypeCache {
addEmittedDeferredDecl(GD);
}
+ /// A call whose target feature and ABI checks wait for the end of the module.
+ /// See shouldDeferCallTargetChecks.
+ struct DeferredCallTargetCheck {
+ SourceLocation Loc;
+ const FunctionDecl *Caller;
+ const FunctionDecl *Callee;
+ llvm::WeakTrackingVH Call;
+ QualType ReturnType;
+ /// The argument types are a range of DeferredCallArgTypes.
+ unsigned FirstArgType;
+ unsigned NumArgTypes;
+ bool CheckFeatures;
+ };
+ std::vector<DeferredCallTargetCheck> DeferredCallTargetChecks;
+ std::vector<QualType> DeferredCallArgTypes;
+ std::vector<const FunctionDecl *> DeferredFunctionABIChecks;
+
+ /// The target features of the always_inline functions that were given those
+ /// of their callers, keyed by canonical declaration.
+ llvm::DenseMap<const FunctionDecl *, llvm::StringMap<bool>>
+ InferredTargetFeatures;
+
/// Decls that were DeferredDecls and have now been emitted.
llvm::DenseMap<llvm::StringRef, GlobalDecl> EmittedDeferredDecls;
@@ -742,6 +765,26 @@ class CodeGenModule : public CodeGenTypeCache {
/// Finalize LLVM code generation.
void Release();
+ /// Whether the target feature and ABI checks of a call, or with a null
+ /// \p Caller those of the definition of \p Callee, have to wait for the end
+ /// of the module. An always_inline function without a target attribute is
+ /// compiled with the target features that its callers have in common, and
+ /// those are only known once every call to it has been emitted.
+ bool shouldDeferCallTargetChecks(const FunctionDecl *Caller,
+ const FunctionDecl *Callee) const;
+ void deferCallTargetChecks(SourceLocation Loc, const FunctionDecl *Caller,
+ const FunctionDecl *Callee, llvm::CallBase *Call,
+ const CallArgList &Args, QualType ReturnType,
+ bool CheckFeatures);
+ void deferFunctionABICheck(const FunctionDecl *FD) {
+ DeferredFunctionABIChecks.push_back(FD);
+ }
+
+ /// Like ASTContext::getFunctionFeatureMap, but with the target features that
+ /// an always_inline function was given because its callers have them.
+ void getFunctionFeatureMap(llvm::StringMap<bool> &FeatureMap,
+ const FunctionDecl *FD) const;
+
/// Get the current Atomic options.
AtomicOptions getAtomicOpts() { return AtomicOpts; }
@@ -2095,6 +2138,11 @@ class CodeGenModule : public CodeGenTypeCache {
/// Emit any needed decls for which code generation was deferred.
void EmitDeferred();
+ /// Gives the always_inline functions that allow it the target features of
+ /// their callers, then performs the checks of the calls for which
+ /// shouldDeferCallTargetChecks held.
+ void emitDeferredCallTargetChecks();
+
/// Try to emit external vtables as available_externally if they have emitted
/// all inlined virtual functions. It runs after EmitDeferred() and therefore
/// is not allowed to create new references to things that need to be emitted
diff --git a/clang/lib/CodeGen/Targets/X86.cpp b/clang/lib/CodeGen/Targets/X86.cpp
index 5324c8d366b829..c35273e5bf0fdc 100644
--- a/clang/lib/CodeGen/Targets/X86.cpp
+++ b/clang/lib/CodeGen/Targets/X86.cpp
@@ -1622,7 +1622,7 @@ class X86_64TargetCodeGenInfo : public TargetCodeGenInfo {
};
} // namespace
-static void initFeatureMaps(const ASTContext &Ctx,
+static void initFeatureMaps(const CodeGenModule &CGM,
llvm::StringMap<bool> &CallerMap,
const FunctionDecl *Caller,
llvm::StringMap<bool> &CalleeMap,
@@ -1631,8 +1631,8 @@ static void initFeatureMaps(const ASTContext &Ctx,
// The caller is potentially nullptr in the case where the call isn't in a
// function. In this case, the getFunctionFeatureMap ensures we just get
// the TU level setting (since it cannot be modified by 'target'..
- Ctx.getFunctionFeatureMap(CallerMap, Caller);
- Ctx.getFunctionFeatureMap(CalleeMap, Callee);
+ CGM.getFunctionFeatureMap(CallerMap, Caller);
+ CGM.getFunctionFeatureMap(CalleeMap, Callee);
}
}
@@ -1711,7 +1711,7 @@ void X86_64TargetCodeGenInfo::checkFunctionABI(CodeGenModule &CGM,
CGM.getContext().getTypeSize(Ty) > 256 ? "avx512f" : "avx";
llvm::StringMap<bool> FeatureMap;
- CGM.getContext().getFunctionFeatureMap(FeatureMap, FD);
+ CGM.getFunctionFeatureMap(FeatureMap, FD);
if (!FeatureMap.lookup(Feature)) {
CGM.getDiags().Report(Loc, diag::warn_avx_calling_convention)
<< !IsReturn << Ty << Feature;
@@ -1765,7 +1765,7 @@ void X86_64TargetCodeGenInfo::checkFunctionCallABI(CodeGenModule &CGM,
// change its ABI with attribute-target after this call.
if (Arg.getType()->isVectorType() &&
CGM.getContext().getTypeSize(Arg.getType()) > 128) {
- initFeatureMaps(CGM.getContext(), CallerMap, Caller, CalleeMap, Callee);
+ initFeatureMaps(CGM, CallerMap, Caller, CalleeMap, Callee);
QualType Ty = Arg.getType();
// The CallArg seems to have desugared the type already, so for clearer
// diagnostics, replace it with the type in the FunctionDecl if possible.
@@ -1783,7 +1783,7 @@ void X86_64TargetCodeGenInfo::checkFunctionCallABI(CodeGenModule &CGM,
// whether this value is used, tail-called, etc.
if (Callee->getReturnType()->isVectorType() &&
CGM.getContext().getTypeSize(Callee->getReturnType()) > 128) {
- initFeatureMaps(CGM.getContext(), CallerMap, Caller, CalleeMap, Callee);
+ initFeatureMaps(CGM, CallerMap, Caller, CalleeMap, Callee);
checkAVXParam(CGM.getDiags(), CGM.getContext(), CallLoc, *Callee, CallerMap,
CalleeMap, Callee->getReturnType(),
/*IsArgument*/ false);
diff --git a/clang/test/CodeGenCXX/target-features-inferred.cpp b/clang/test/CodeGenCXX/target-features-inferred.cpp
new file mode 100644
index 00000000000000..cbd214ea75a149
--- /dev/null
+++ b/clang/test/CodeGenCXX/target-features-inferred.cpp
@@ -0,0 +1,113 @@
+// RUN: %clang_cc1 %s -triple=x86_64-linux-gnu -emit-llvm -disable-llvm-passes -verify=clean -o - | FileCheck %s
+// RUN: %clang_cc1 %s -triple=x86_64-linux-gnu -emit-llvm -o /dev/null -verify -DERRORS
+
+// An always_inline function that has no target attribute and that only direct
+// calls in this translation unit can reach is compiled with the target
+// features that its callers have in common.
+
+// clean-no-diagnostics
+
+typedef float v8f __attribute__((vector_size(32)));
+
+template <class> struct Zero;
+template <> struct Zero<v8f> {
+ [[gnu::target("avx"), gnu::always_inline]] inline v8f operator()() const {
+ return v8f{};
+ }
+};
+
+template <class T>
+[[gnu::always_inline]] static inline auto direct(T gen) { return gen(); }
+
+// Unlike direct, this is not instantiated until the end of the translation
+// unit, after its caller has been emitted.
+template <class T>
+[[gnu::always_inline]] static inline v8f indirect(T gen) { return direct(gen); }
+
+// Passing a vector on must not be diagnosed as a change of the ABI either.
+[[gnu::always_inline]] static inline v8f pass(v8f x) { return x; }
+
+struct Vec {
+ v8f v;
+ [[gnu::always_inline]] Vec(v8f v) : v(v) {}
+};
+
+[[gnu::target("avx")]] v8f avx_caller() {
+ return Vec(pass(indirect(Zero<v8f>()))).v;
+}
+
+[[gnu::target("avx2"), gnu::always_inline]] static inline int leaf() {
+ return 1;
+}
+[[gnu::always_inline]] static inline int mid() { return leaf(); }
+[[gnu::always_inline]] static inline int top() { return mid(); }
+
+[[gnu::target("avx2")]] int avx2_caller() {
+ return top() + []() __attribute__((always_inline)) { return leaf(); }();
+}
+
+[[gnu::always_inline]] static inline int common() { return 0; }
+[[gnu::target("avx")]] int common_avx() { return common(); }
+[[gnu::target("avx2")]] int common_avx2() { return common(); }
+
+// A call that is never emitted requires nothing of the callers.
+[[gnu::always_inline]] static inline int folded() {
+ if (0)
+ return leaf();
+ return 0;
+}
+int default_caller() { return folded(); }
+
+// Other translation units may give their definitions other features, so the
+// linker must not merge them with this one.
+[[gnu::always_inline]] inline int odr() { return leaf(); }
+[[gnu::target("avx2")]] int odr_caller() { return odr(); }
+
+[[gnu::always_inline]] inline int odr_default() { return 0; }
+int odr_default_caller() { return odr_default(); }
+
+// CHECK-DAG: define dso_local {{.*}} @_Z10avx_callerv() #[[AVX_CALLER:[0-9]+]]
+// CHECK-DAG: define internal {{.*}} @_ZL4passDv8_f({{.*}}) #[[AVX_VECTOR:[0-9]+]]
+// CHECK-DAG: define internal {{.*}} @_ZL8indirectI4ZeroIDv8_fEES1_T_() #[[AVX_VECTOR]]
+// CHECK-DAG: define internal {{.*}} @_ZL6directI4ZeroIDv8_fEEDaT_() #[[AVX_VECTOR]]
+// CHECK-DAG: define internal void @_ZN3VecC1EDv8_f({{.*}}) unnamed_addr #[[AVX:[0-9]+]]
+// CHECK-DAG: define internal void @_ZN3VecC2EDv8_f({{.*}}) unnamed_addr #[[AVX]]
+// CHECK-DAG: define dso_local {{.*}} @_Z11avx2_callerv() #[[AVX2_CALLER:[0-9]+]]
+// CHECK-DAG: define internal {{.*}} @_ZL3topv() #[[AVX2:[0-9]+]]
+// CHECK-DAG: define internal {{.*}} @_ZL3midv() #[[AVX2]]
+// CHECK-DAG: define internal {{.*}} @"_ZZ11avx2_callervENK3$_0clEv"({{.*}}) #[[AVX2]]
+// CHECK-DAG: define internal {{.*}} @_ZL6commonv() #[[AVX]]
+// CHECK-DAG: define dso_local {{.*}} @_Z14default_callerv() #[[DEFAULT_CALLER:[0-9]+]]
+// CHECK-DAG: define internal {{.*}} @_ZL6foldedv() #[[DEFAULT:[0-9]+]]
+// CHECK-DAG: define internal {{.*}} @_Z3odrv() #[[AVX2]]
+// CHECK-DAG: define linkonce_odr {{.*}} @_Z11odr_defaultv() #[[DEFAULT]] comdat
+// CHECK-DAG: attributes #[[AVX_CALLER]] = { {{.*}}"target-features"="[[AVX_FEATURES:[^"]+]]"
+// CHECK-DAG: attributes #[[AVX_VECTOR]] = { alwaysinline {{.*}}"target-features"="[[AVX_FEATURES]]"
+// CHECK-DAG: attributes #[[AVX]] = { alwaysinline {{.*}}"target-features"="[[AVX_FEATURES]]"
+// CHECK-DAG: attributes #[[AVX2_CALLER]] = { {{.*}}"target-features"="[[AVX2_FEATURES:[^"]+]]"
+// CHECK-DAG: attributes #[[AVX2]] = { alwaysinline {{.*}}"target-features"="[[AVX2_FEATURES]]"
+// CHECK-DAG: attributes #[[DEFAULT_CALLER]] = { {{.*}}"target-features"="[[DEFAULT_FEATURES:[^"]+]]"
+// CHECK-DAG: attributes #[[DEFAULT]] = { alwaysinline {{.*}}"target-features"="[[DEFAULT_FEATURES]]"
+
+#ifdef ERRORS
+int disabled() {
+ return top(); // expected-error {{always_inline function 'top' requires target feature 'avx2', but would be inlined into function 'disabled' that is compiled without support for 'avx2'}}
+}
+
+// Functions that can run without having been inlined into a function that was
+// checked keep the features they were declared with.
+[[gnu::always_inline]] static inline int escaping() {
+ return leaf(); // expected-error {{always_inline function 'leaf' requires target feature 'avx2', but would be inlined into function 'escaping' that is compiled without support for 'avx2'}}
+}
+int (*pointer())() { return escaping; }
+
+[[gnu::always_inline]] static inline int initializer() {
+ return leaf(); // expected-error {{always_inline function 'leaf' requires target feature 'avx2', but would be inlined into function 'initializer' that is compiled without support for 'avx2'}}
+}
+int global = initializer();
+
+[[gnu::always_inline]] int external() {
+ return leaf(); // expected-error {{always_inline function 'leaf' requires target feature 'avx2', but would be inlined into function 'external' that is compiled without support for 'avx2'}}
+}
+[[gnu::target("avx2")]] int external_caller() { return external(); }
+#endif
More information about the cfe-commits
mailing list