[clang] [clang] Compile always_inline functions with the target features of their callers (PR #241171)

via cfe-commits cfe-commits at lists.llvm.org
Wed Nov 11 00:11:37 PST 2026


https://github.com/higher-performance created https://github.com/llvm/llvm-project/pull/241171

AI-generated fix for #129398.

>From d479112aba26e86024f1a72b02ba046b6cc55df2 Mon Sep 17 00:00:00 2001
From: higher-performance <higher.performance.github at gmail.com>
Date: Wed, 11 Nov 2026 03:10:18 -0500
Subject: [PATCH] [clang] Compile always_inline functions with the target
 features of their callers

---
 clang/docs/ReleaseNotes.md                    |   9 +
 clang/include/clang/Basic/AttrDocs.td         |  22 ++
 clang/lib/CodeGen/CGCall.cpp                  |  47 ++-
 clang/lib/CodeGen/CodeGenFunction.cpp         |   5 +-
 clang/lib/CodeGen/CodeGenModule.cpp           | 345 ++++++++++++++++++
 clang/lib/CodeGen/CodeGenModule.h             |  48 +++
 clang/lib/CodeGen/Targets/X86.cpp             |  12 +-
 .../CodeGenCXX/target-features-inferred.cpp   | 113 ++++++
 8 files changed, 574 insertions(+), 27 deletions(-)
 create mode 100644 clang/test/CodeGenCXX/target-features-inferred.cpp

diff --git a/clang/docs/ReleaseNotes.md b/clang/docs/ReleaseNotes.md
index 0f9e7e01636d71..04f8a0e240cb8c 100644
--- a/clang/docs/ReleaseNotes.md
+++ b/clang/docs/ReleaseNotes.md
@@ -436,6 +436,15 @@ documentation](CIR/index.md) for more information regarding ClangIR's status and
 
 ### Attribute Changes in Clang
 
+- An `always_inline` function without a `target` attribute is now compiled with
+  the target features that its callers have in common, provided that other
+  translation units cannot call it (for example because it is `static`,
+  `inline` or a template) and that its only uses are direct calls. Generic
+  helpers can thus be shared between functions with different `target`
+  attributes and can call target intrinsics without naming the features
+  themselves; previously this was rejected with "always_inline function requires
+  target feature" and "changes the ABI" errors.
+
 - Clang now properly propagates attributes on class and variable templates to their redeclarations, which will result in redeclarations not interfering with diagnostics. (#GH209812)
 
 - Clang now recognizes the `[[gnu::flag_enum]]` attribute and treats it equivalent to `[[clang::flag_enum]]`
diff --git a/clang/include/clang/Basic/AttrDocs.td b/clang/include/clang/Basic/AttrDocs.td
index 2300c8456ef164..575a02207dab41 100644
--- a/clang/include/clang/Basic/AttrDocs.td
+++ b/clang/include/clang/Basic/AttrDocs.td
@@ -3426,6 +3426,28 @@ the front end.
 Note that this attribute does not apply transitively to nested functions such
 as blocks or C++ lambdas.
 
+A function that is marked `always_inline` and has no `target` attribute is
+compiled with the target features that all of its callers have in common, so
+that it can be inlined into each of them. It can therefore call `always_inline`
+functions that require more features than were enabled on the command line,
+such as target intrinsics, as long as every function that calls it enables
+them:
+
+```c++
+__attribute__((always_inline))
+static __m256 twice(__m256 x) { return _mm256_add_ps(x, x); }
+
+__attribute__((target("avx")))
+__m256 twice_avx(__m256 x) { return twice(x); } // OK, twice is AVX code here.
+```
+
+This only holds for functions that cannot run in any other way: the definition
+must not be callable from other translation units (as is the case for `static`
+and `inline` functions and for implicit template instantiations), its only uses
+must be direct calls from other functions of the translation unit, and these
+must all be compiled for the same CPU. Any other `always_inline` function is
+compiled with the features that were passed on the command line.
+
 Additionally, this attribute supports function multiversioning for ELF based
 x86/x86-64 targets, which can be used to create multiple implementations of the
 same function that will be resolved at runtime based on the priority of their
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index 1d7f9f742b87e9..0924671fb56b5b 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -5719,28 +5719,32 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
   llvm::FunctionType *IRFuncTy = getTypes().GetFunctionType(CallInfo);
 
   const Decl *TargetDecl = Callee.getAbstractInfo().getCalleeDecl().getDecl();
-  if (const FunctionDecl *FD = dyn_cast_or_null<FunctionDecl>(TargetDecl)) {
-    // We can only guarantee that a function is called from the correct
-    // context/function based on the appropriate target attributes,
-    // so only check in the case where we have both always_inline and target
-    // since otherwise we could be making a conditional call after a check for
-    // the proper cpu features (and it won't cause code generation issues due to
-    // function based code generation).
-    if ((TargetDecl->hasAttr<AlwaysInlineAttr>() &&
-         (TargetDecl->hasAttr<TargetAttr>() ||
-          (CurFuncDecl && CurFuncDecl->hasAttr<TargetAttr>()))) ||
-        (CurFuncDecl && CurFuncDecl->hasAttr<FlattenAttr>() &&
-         (CurFuncDecl->hasAttr<TargetAttr>() ||
-          TargetDecl->hasAttr<TargetAttr>())))
-      checkTargetFeatures(Loc, FD);
-  }
-
-  // Some architectures (such as x86-64) have the ABI changed based on
-  // attribute-target/features. Give them a chance to diagnose.
   const FunctionDecl *CallerDecl = dyn_cast_or_null<FunctionDecl>(CurCodeDecl);
   const FunctionDecl *CalleeDecl = dyn_cast_or_null<FunctionDecl>(TargetDecl);
-  CGM.getTargetCodeGenInfo().checkFunctionCallABI(CGM, Loc, CallerDecl,
-                                                  CalleeDecl, CallArgs, RetTy);
+  // We can only guarantee that a function is called from the correct
+  // context/function based on the appropriate target attributes,
+  // so only check in the case where we have both always_inline and target
+  // since otherwise we could be making a conditional call after a check for
+  // the proper cpu features (and it won't cause code generation issues due to
+  // function based code generation).
+  bool CheckFeatures =
+      CalleeDecl && ((TargetDecl->hasAttr<AlwaysInlineAttr>() &&
+                      (TargetDecl->hasAttr<TargetAttr>() ||
+                       (CurFuncDecl && CurFuncDecl->hasAttr<TargetAttr>()))) ||
+                     (CurFuncDecl && CurFuncDecl->hasAttr<FlattenAttr>() &&
+                      (CurFuncDecl->hasAttr<TargetAttr>() ||
+                       TargetDecl->hasAttr<TargetAttr>())));
+  bool DeferTargetChecks =
+      CGM.shouldDeferCallTargetChecks(CallerDecl, CalleeDecl);
+  if (!DeferTargetChecks) {
+    if (CheckFeatures)
+      checkTargetFeatures(Loc, CalleeDecl);
+
+    // Some architectures (such as x86-64) have the ABI changed based on
+    // attribute-target/features. Give them a chance to diagnose.
+    CGM.getTargetCodeGenInfo().checkFunctionCallABI(
+        CGM, Loc, CallerDecl, CalleeDecl, CallArgs, RetTy);
+  }
 
   // 1. Set up the arguments.
 
@@ -6431,6 +6435,9 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
                               BundleList);
     EmitBlock(Cont);
   }
+  if (DeferTargetChecks)
+    CGM.deferCallTargetChecks(Loc, CallerDecl, CalleeDecl, CI, CallArgs, RetTy,
+                              CheckFeatures);
   if (CI->getCalledFunction() && CI->getCalledFunction()->hasName() &&
       CI->getCalledFunction()->getName().starts_with("_Z4sqrt")) {
     SetSqrtFPAccuracy(CI);
diff --git a/clang/lib/CodeGen/CodeGenFunction.cpp b/clang/lib/CodeGen/CodeGenFunction.cpp
index 33434526d86325..a3ddc6ae128335 100644
--- a/clang/lib/CodeGen/CodeGenFunction.cpp
+++ b/clang/lib/CodeGen/CodeGenFunction.cpp
@@ -1499,7 +1499,10 @@ void CodeGenFunction::GenerateCode(GlobalDecl GD, llvm::Function *Fn,
   FunctionArgList Args;
   QualType ResTy = BuildFunctionArgList(GD, Args);
 
-  CGM.getTargetCodeGenInfo().checkFunctionABI(CGM, FD);
+  if (CGM.shouldDeferCallTargetChecks(/*Caller=*/nullptr, FD))
+    CGM.deferFunctionABICheck(FD);
+  else
+    CGM.getTargetCodeGenInfo().checkFunctionABI(CGM, FD);
 
   if (FD->isInlineBuiltinDeclaration()) {
     // When generating code for a builtin with an inline declaration, use a
diff --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index bdfbdeb1059384..f6e926d4ac27a4 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -49,11 +49,13 @@
 #include "clang/Basic/Version.h"
 #include "clang/CodeGen/BackendUtil.h"
 #include "clang/CodeGen/ConstantInitBuilder.h"
+#include "clang/CodeGenUtils/FunctionUtils.h"
 #include "clang/CodeGenUtils/ModuleUtils.h"
 #include "clang/Lex/Preprocessor.h"
 #include "llvm/ABI/IRTypeMapper.h"
 #include "llvm/ABI/TargetInfo.h"
 #include "llvm/ADT/APFloat.h"
+#include "llvm/ADT/BitVector.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringSwitch.h"
@@ -1358,6 +1360,8 @@ void CodeGenModule::Release() {
     addCompilerUsedGlobal(GV);
   }
   emitLLVMUsed();
+  // This has to know every use of the functions that have been emitted.
+  emitDeferredCallTargetChecks();
   if (SanStats)
     SanStats->finish();
 
@@ -3617,6 +3621,347 @@ bool CodeGenModule::GetCPUAndFeaturesAttributes(GlobalDecl GD,
   return AddedAttr;
 }
 
+namespace {
+/// The target CPU and features that the IR attributes of a function give it.
+struct FunctionTargetFeatures {
+  StringRef CPU;
+  std::string Features;
+  llvm::BitVector Enabled;
+  llvm::BitVector Disabled;
+};
+} // namespace
+
+/// Whether \p FD is declared such that it could be compiled with the target
+/// features of its callers. Whether it is also only used in ways that allow
+/// this is not known before the end of the module.
+static bool mayInferTargetFeatures(const FunctionDecl *FD) {
+  FD = FD->getMostRecentDecl();
+  // A flatten function is checked against the features of all of its callees,
+  // which the inference does not model.
+  return FD->hasAttr<AlwaysInlineAttr>() && !FD->getBuiltinID() &&
+         !FD->hasAttr<TargetAttr>() && !FD->hasAttr<TargetVersionAttr>() &&
+         !FD->hasAttr<TargetClonesAttr>() && !FD->hasAttr<CPUSpecificAttr>() &&
+         !FD->hasAttr<CPUDispatchAttr>() && !FD->hasAttr<FlattenAttr>();
+}
+
+bool CodeGenModule::shouldDeferCallTargetChecks(
+    const FunctionDecl *Caller, const FunctionDecl *Callee) const {
+  // HIP stdpar device compilations leave the feature checks to the backend.
+  return Callee && !(LangOpts.HIPStdPar && LangOpts.CUDAIsDevice) &&
+         ((Caller && mayInferTargetFeatures(Caller)) ||
+          mayInferTargetFeatures(Callee));
+}
+
+void CodeGenModule::deferCallTargetChecks(
+    SourceLocation Loc, const FunctionDecl *Caller, const FunctionDecl *Callee,
+    llvm::CallBase *Call, const CallArgList &Args, QualType ReturnType,
+    bool CheckFeatures) {
+  DeferredCallTargetChecks.push_back(
+      {Loc, Caller, Callee, Call, ReturnType,
+       static_cast<unsigned>(DeferredCallArgTypes.size()),
+       static_cast<unsigned>(Args.size()), CheckFeatures});
+  for (const CallArg &Arg : Args)
+    DeferredCallArgTypes.push_back(Arg.getType());
+}
+
+void CodeGenModule::getFunctionFeatureMap(llvm::StringMap<bool> &FeatureMap,
+                                          const FunctionDecl *FD) const {
+  auto Inferred = FD ? InferredTargetFeatures.find(FD->getCanonicalDecl())
+                     : InferredTargetFeatures.end();
+  if (Inferred != InferredTargetFeatures.end())
+    FeatureMap = Inferred->second;
+  else
+    Context.getFunctionFeatureMap(FeatureMap, FD);
+}
+
+void CodeGenModule::emitDeferredCallTargetChecks() {
+  // An always_inline function without a target attribute is compiled with the
+  // target features that its callers have in common, provided that nothing but
+  // direct calls from functions of this module can run it. It can then be
+  // inlined into all of its callers, and the always_inline functions that it
+  // calls can be inlined into it as long as each of its callers has the
+  // features that they need. Any other function keeps the features it was
+  // declared with.
+  struct RequiredFeatures {
+    llvm::BitVector Features;
+    /// Those that target attributes name, which diagnostics mention first.
+    SmallVector<unsigned, 4> Named;
+  };
+  struct InferringFunction {
+    llvm::SmallPtrSet<llvm::Function *, 2> Fns;
+    std::optional<FunctionTargetFeatures> Features;
+    /// What the always_inline functions that it calls need.
+    RequiredFeatures Required;
+    bool Valid = true;
+  };
+  llvm::MapVector<const FunctionDecl *, InferringFunction> Inferring;
+  llvm::DenseMap<const llvm::CallBase *, const FunctionDecl *> CalleeOfCall;
+
+  auto LiveCall = [](const DeferredCallTargetCheck &Check) {
+    llvm::Value *Call = Check.Call;
+    return dyn_cast_or_null<llvm::CallBase>(Call);
+  };
+  auto CalledFunction = [](const llvm::CallBase *Call) {
+    return Call ? dyn_cast<llvm::Function>(Call->getCalledOperand()) : nullptr;
+  };
+
+  for (const DeferredCallTargetCheck &Check : DeferredCallTargetChecks) {
+    llvm::CallBase *Call = LiveCall(Check);
+    llvm::Function *Fn = CalledFunction(Call);
+    // Nothing checks the target features of code outside of functions, so a
+    // call from there counts as an unknown use below.
+    if (!Fn || !Check.Caller || !mayInferTargetFeatures(Check.Callee))
+      continue;
+    const FunctionDecl *Callee = Check.Callee->getCanonicalDecl();
+    Inferring[Callee].Fns.insert(Fn);
+    CalleeOfCall[Call] = Callee;
+  }
+
+  llvm::DenseMap<const llvm::Function *, const FunctionDecl *> DeclOfFn;
+  for (auto &Entry : Inferring) {
+    const FunctionDecl *FD = Entry.first;
+    InferringFunction &Callee = Entry.second;
+    const FunctionDecl *Def = FD->getDefinition();
+    // The body of an available_externally function is not the one that runs
+    // when a call to it is not inlined.
+    GVALinkage Linkage =
+        Def ? Context.GetGVALinkageForFunction(Def) : GVA_StrongExternal;
+    Callee.Valid = (Linkage == GVA_Internal || Linkage == GVA_DiscardableODR) &&
+                   llvm::all_of(Callee.Fns, [&](const llvm::Function *Fn) {
+                     return !Fn->isDeclaration() &&
+                            llvm::all_of(Fn->uses(), [&](const llvm::Use &U) {
+                              const auto *Call =
+                                  dyn_cast<llvm::CallBase>(U.getUser());
+                              return Call && Call->isCallee(&U) &&
+                                     CalleeOfCall.lookup(Call) == FD;
+                            });
+                   });
+    if (Callee.Valid)
+      for (const llvm::Function *Fn : Callee.Fns)
+        DeclOfFn[Fn] = FD;
+  }
+  auto FindDecl = [&](const FunctionDecl *FD) -> InferringFunction * {
+    auto It = FD ? Inferring.find(FD->getCanonicalDecl()) : Inferring.end();
+    return It != Inferring.end() && It->second.Valid ? &It->second : nullptr;
+  };
+  auto Find = [&](const llvm::Function *Fn) {
+    return FindDecl(DeclOfFn.lookup(Fn));
+  };
+
+  llvm::StringMap<unsigned> FeatureIDs;
+  SmallVector<StringRef, 0> FeatureNames;
+  auto FeatureID = [&](StringRef Name) {
+    auto [It, Inserted] = FeatureIDs.try_emplace(Name, FeatureNames.size());
+    if (Inserted)
+      FeatureNames.push_back(It->getKey());
+    return It->second;
+  };
+  auto Set = [](llvm::BitVector &Bits, unsigned ID) {
+    if (Bits.size() <= ID)
+      Bits.resize(ID + 1);
+    Bits.set(ID);
+  };
+  auto FeatureString = [&](const FunctionTargetFeatures &Features) {
+    std::vector<std::string> Strings;
+    for (unsigned ID : Features.Enabled.set_bits())
+      Strings.push_back(("+" + FeatureNames[ID]).str());
+    for (unsigned ID : Features.Disabled.set_bits())
+      Strings.push_back(("-" + FeatureNames[ID]).str());
+    llvm::sort(Strings);
+    return llvm::join(Strings, ",");
+  };
+
+  // The IR attributes of the calling functions are what decides whether a call
+  // can be inlined, and unlike the declarations they also cover function
+  // versions and thunks.
+  llvm::DenseMap<std::pair<StringRef, StringRef>, FunctionTargetFeatures>
+      ParsedFeatures;
+  auto FeaturesOf =
+      [&](const llvm::Function *Fn) -> const FunctionTargetFeatures * {
+    if (InferringFunction *Caller = Find(Fn))
+      return Caller->Features ? &*Caller->Features : nullptr;
+    StringRef CPU = Fn->getFnAttribute("target-cpu").getValueAsString();
+    StringRef Features =
+        Fn->getFnAttribute("target-features").getValueAsString();
+    auto [It, Inserted] = ParsedFeatures.try_emplace(std::pair(CPU, Features));
+    if (Inserted) {
+      It->second.CPU = CPU;
+      It->second.Features = Features.str();
+      for (StringRef Feature : llvm::split(Features, ','))
+        if (!Feature.empty())
+          Set(Feature[0] == '+' ? It->second.Enabled : It->second.Disabled,
+              FeatureID(Feature.substr(1)));
+    }
+    return &It->second;
+  };
+
+  for (bool Retry = true; Retry;) {
+    Retry = false;
+    for (auto &Entry : Inferring)
+      Entry.second.Features.reset();
+    for (bool Changed = true; Changed;) {
+      Changed = false;
+      for (const DeferredCallTargetCheck &Check : DeferredCallTargetChecks) {
+        llvm::CallBase *Call = LiveCall(Check);
+        InferringFunction *Callee = Find(CalledFunction(Call));
+        const FunctionTargetFeatures *Caller =
+            Callee ? FeaturesOf(Call->getFunction()) : nullptr;
+        if (!Caller)
+          continue;
+        if (!Callee->Features) {
+          Callee->Features = *Caller;
+          Changed = true;
+        } else if (Callee->Features->CPU != Caller->CPU) {
+          // The features that only the CPUs imply are not known here, so
+          // there is no telling which of them the callers have in common.
+          Callee->Valid = false;
+          Retry = true;
+        } else if (Callee->Features->Enabled.test(Caller->Enabled) ||
+                   Caller->Disabled.test(Callee->Features->Disabled)) {
+          // A feature that only some callers mention is left to the CPU,
+          // which is the same for all of them.
+          Callee->Features->Enabled &= Caller->Enabled;
+          Callee->Features->Disabled |= Caller->Disabled;
+          Callee->Features->Features = FeatureString(*Callee->Features);
+          Changed = true;
+        }
+      }
+    }
+    // Functions that only call each other are not reachable from any function
+    // with known features.
+    for (auto &Entry : Inferring)
+      if (Entry.second.Valid && !Entry.second.Features) {
+        Entry.second.Valid = false;
+        Retry = true;
+      }
+  }
+
+  llvm::DenseMap<const FunctionDecl *, RequiredFeatures> DeclFeatures;
+  auto FeaturesOfDecl =
+      [&](const FunctionDecl *FD) -> const RequiredFeatures & {
+    auto [It, Inserted] = DeclFeatures.try_emplace(FD);
+    if (Inserted) {
+      llvm::StringMap<bool> FeatureMap;
+      Context.getFunctionFeatureMap(FeatureMap, FD);
+      for (const auto &Feature : FeatureMap)
+        if (Feature.getValue())
+          Set(It->second.Features, FeatureID(Feature.getKey()));
+      if (const auto *TD = FD->getAttr<TargetAttr>();
+          TD && !FD->isMultiVersion())
+        for (const std::string &Feature :
+             Context.filterFunctionTargetAttrs(TD).Features)
+          if (Feature[0] == '+' &&
+              FeatureMap.lookup(StringRef(Feature).substr(1)))
+            It->second.Named.push_back(FeatureID(StringRef(Feature).substr(1)));
+    }
+    return It->second;
+  };
+
+  for (bool Changed = true; Changed;) {
+    Changed = false;
+    // Callees are mostly emitted after their callers, so walking backwards
+    // tends to converge in a single pass.
+    for (const DeferredCallTargetCheck &Check :
+         llvm::reverse(DeferredCallTargetChecks)) {
+      llvm::CallBase *Call = LiveCall(Check);
+      InferringFunction *Caller = Call ? Find(Call->getFunction()) : nullptr;
+      if (!Caller || !Check.Callee->hasAttr<AlwaysInlineAttr>())
+        continue;
+      InferringFunction *Callee = Find(CalledFunction(Call));
+      const RequiredFeatures &Required =
+          Callee ? Callee->Required : FeaturesOfDecl(Check.Callee);
+      if (Required.Features.test(Caller->Required.Features)) {
+        Caller->Required.Features |= Required.Features;
+        Changed = true;
+      }
+      for (unsigned ID : Required.Named)
+        if (!llvm::is_contained(Caller->Required.Named, ID)) {
+          Caller->Required.Named.push_back(ID);
+          Changed = true;
+        }
+    }
+  }
+
+  for (auto &Entry : Inferring) {
+    InferringFunction &Callee = Entry.second;
+    if (!Callee.Valid)
+      continue;
+    for (llvm::Function *Fn : Callee.Fns) {
+      if (Fn->getFnAttribute("target-cpu").getValueAsString() ==
+              Callee.Features->CPU &&
+          Fn->getFnAttribute("target-features").getValueAsString() ==
+              Callee.Features->Features)
+        continue;
+      Fn->removeFnAttr("target-cpu");
+      Fn->removeFnAttr("target-features");
+      if (!Callee.Features->CPU.empty())
+        Fn->addFnAttr("target-cpu", Callee.Features->CPU);
+      if (!Callee.Features->Features.empty())
+        Fn->addFnAttr("target-features", Callee.Features->Features);
+      // Other translation units may give their definitions of the function
+      // other features, so the linker must not merge them with this one.
+      if (!Fn->hasLocalLinkage()) {
+        Fn->setComdat(nullptr);
+        Fn->setLinkage(llvm::Function::InternalLinkage);
+      }
+      llvm::StringMap<bool> &FeatureMap = InferredTargetFeatures[Entry.first];
+      for (unsigned ID : Callee.Features->Enabled.set_bits())
+        FeatureMap[FeatureNames[ID]] = true;
+    }
+  }
+
+  for (const DeferredCallTargetCheck &Check : DeferredCallTargetChecks) {
+    llvm::CallBase *Call = LiveCall(Check);
+    InferringFunction *Caller =
+        Call ? Find(Call->getFunction()) : FindDecl(Check.Caller);
+    InferringFunction *Callee =
+        Call ? Find(CalledFunction(Call)) : FindDecl(Check.Callee);
+    // A call that was removed again is neither inlined nor run.
+    if (!Call && (Caller || Callee))
+      continue;
+
+    // What an inferring caller needs is checked where it is called, up to the
+    // first caller that does not infer its features.
+    if (Callee && !Caller) {
+      llvm::BitVector Missing = Callee->Required.Features;
+      Missing.reset(FeaturesOfDecl(Check.Caller).Features);
+      if (Missing.any()) {
+        StringRef Feature;
+        for (unsigned ID : Missing.set_bits())
+          if (Feature.empty() || FeatureNames[ID] < Feature)
+            Feature = FeatureNames[ID];
+        if (const unsigned *Named =
+                llvm::find_if(Callee->Required.Named,
+                              [&](unsigned ID) { return Missing.test(ID); });
+            Named != Callee->Required.Named.end())
+          Feature = FeatureNames[*Named];
+        getDiags().Report(Check.Loc, diag::err_function_needs_feature)
+            << Check.Caller->getDeclName() << Check.Callee->getDeclName()
+            << Feature;
+      }
+    } else if (!Caller && Check.CheckFeatures) {
+      CodeGenUtils::checkTargetFeatures(Context, getDiags(), getLangOpts(),
+                                        Check.Loc, Check.Caller, Check.Callee);
+    }
+
+    // The targets only look at the types of the arguments.
+    CallArgList Args;
+    for (QualType Type : ArrayRef<QualType>(DeferredCallArgTypes)
+                             .slice(Check.FirstArgType, Check.NumArgTypes))
+      Args.add(RValue::get(nullptr), Type);
+    getTargetCodeGenInfo().checkFunctionCallABI(
+        *this, Check.Loc, Check.Caller, Check.Callee, Args, Check.ReturnType);
+  }
+
+  for (const FunctionDecl *FD : DeferredFunctionABIChecks)
+    getTargetCodeGenInfo().checkFunctionABI(*this, FD);
+
+  DeferredCallTargetChecks.clear();
+  DeferredCallArgTypes.clear();
+  DeferredFunctionABIChecks.clear();
+  InferredTargetFeatures.clear();
+}
+
 void CodeGenModule::setNonAliasAttributes(GlobalDecl GD,
                                           llvm::GlobalObject *GO) {
   const Decl *D = GD.getDecl();
diff --git a/clang/lib/CodeGen/CodeGenModule.h b/clang/lib/CodeGen/CodeGenModule.h
index e2bd6bc4701fda..1b38d91e397244 100644
--- a/clang/lib/CodeGen/CodeGenModule.h
+++ b/clang/lib/CodeGen/CodeGenModule.h
@@ -101,6 +101,7 @@ class CGOpenMPRuntime;
 class CGCUDARuntime;
 class CGHLSLRuntime;
 class CGFunctionInfo;
+class CallArgList;
 class CoverageMappingModuleGen;
 class QualTypeMapper;
 class TargetCodeGenInfo;
@@ -424,6 +425,28 @@ class CodeGenModule : public CodeGenTypeCache {
     addEmittedDeferredDecl(GD);
   }
 
+  /// A call whose target feature and ABI checks wait for the end of the module.
+  /// See shouldDeferCallTargetChecks.
+  struct DeferredCallTargetCheck {
+    SourceLocation Loc;
+    const FunctionDecl *Caller;
+    const FunctionDecl *Callee;
+    llvm::WeakTrackingVH Call;
+    QualType ReturnType;
+    /// The argument types are a range of DeferredCallArgTypes.
+    unsigned FirstArgType;
+    unsigned NumArgTypes;
+    bool CheckFeatures;
+  };
+  std::vector<DeferredCallTargetCheck> DeferredCallTargetChecks;
+  std::vector<QualType> DeferredCallArgTypes;
+  std::vector<const FunctionDecl *> DeferredFunctionABIChecks;
+
+  /// The target features of the always_inline functions that were given those
+  /// of their callers, keyed by canonical declaration.
+  llvm::DenseMap<const FunctionDecl *, llvm::StringMap<bool>>
+      InferredTargetFeatures;
+
   /// Decls that were DeferredDecls and have now been emitted.
   llvm::DenseMap<llvm::StringRef, GlobalDecl> EmittedDeferredDecls;
 
@@ -742,6 +765,26 @@ class CodeGenModule : public CodeGenTypeCache {
   /// Finalize LLVM code generation.
   void Release();
 
+  /// Whether the target feature and ABI checks of a call, or with a null
+  /// \p Caller those of the definition of \p Callee, have to wait for the end
+  /// of the module. An always_inline function without a target attribute is
+  /// compiled with the target features that its callers have in common, and
+  /// those are only known once every call to it has been emitted.
+  bool shouldDeferCallTargetChecks(const FunctionDecl *Caller,
+                                   const FunctionDecl *Callee) const;
+  void deferCallTargetChecks(SourceLocation Loc, const FunctionDecl *Caller,
+                             const FunctionDecl *Callee, llvm::CallBase *Call,
+                             const CallArgList &Args, QualType ReturnType,
+                             bool CheckFeatures);
+  void deferFunctionABICheck(const FunctionDecl *FD) {
+    DeferredFunctionABIChecks.push_back(FD);
+  }
+
+  /// Like ASTContext::getFunctionFeatureMap, but with the target features that
+  /// an always_inline function was given because its callers have them.
+  void getFunctionFeatureMap(llvm::StringMap<bool> &FeatureMap,
+                             const FunctionDecl *FD) const;
+
   /// Get the current Atomic options.
   AtomicOptions getAtomicOpts() { return AtomicOpts; }
 
@@ -2095,6 +2138,11 @@ class CodeGenModule : public CodeGenTypeCache {
   /// Emit any needed decls for which code generation was deferred.
   void EmitDeferred();
 
+  /// Gives the always_inline functions that allow it the target features of
+  /// their callers, then performs the checks of the calls for which
+  /// shouldDeferCallTargetChecks held.
+  void emitDeferredCallTargetChecks();
+
   /// Try to emit external vtables as available_externally if they have emitted
   /// all inlined virtual functions.  It runs after EmitDeferred() and therefore
   /// is not allowed to create new references to things that need to be emitted
diff --git a/clang/lib/CodeGen/Targets/X86.cpp b/clang/lib/CodeGen/Targets/X86.cpp
index 5324c8d366b829..c35273e5bf0fdc 100644
--- a/clang/lib/CodeGen/Targets/X86.cpp
+++ b/clang/lib/CodeGen/Targets/X86.cpp
@@ -1622,7 +1622,7 @@ class X86_64TargetCodeGenInfo : public TargetCodeGenInfo {
 };
 } // namespace
 
-static void initFeatureMaps(const ASTContext &Ctx,
+static void initFeatureMaps(const CodeGenModule &CGM,
                             llvm::StringMap<bool> &CallerMap,
                             const FunctionDecl *Caller,
                             llvm::StringMap<bool> &CalleeMap,
@@ -1631,8 +1631,8 @@ static void initFeatureMaps(const ASTContext &Ctx,
     // The caller is potentially nullptr in the case where the call isn't in a
     // function.  In this case, the getFunctionFeatureMap ensures we just get
     // the TU level setting (since it cannot be modified by 'target'..
-    Ctx.getFunctionFeatureMap(CallerMap, Caller);
-    Ctx.getFunctionFeatureMap(CalleeMap, Callee);
+    CGM.getFunctionFeatureMap(CallerMap, Caller);
+    CGM.getFunctionFeatureMap(CalleeMap, Callee);
   }
 }
 
@@ -1711,7 +1711,7 @@ void X86_64TargetCodeGenInfo::checkFunctionABI(CodeGenModule &CGM,
         CGM.getContext().getTypeSize(Ty) > 256 ? "avx512f" : "avx";
 
     llvm::StringMap<bool> FeatureMap;
-    CGM.getContext().getFunctionFeatureMap(FeatureMap, FD);
+    CGM.getFunctionFeatureMap(FeatureMap, FD);
     if (!FeatureMap.lookup(Feature)) {
       CGM.getDiags().Report(Loc, diag::warn_avx_calling_convention)
           << !IsReturn << Ty << Feature;
@@ -1765,7 +1765,7 @@ void X86_64TargetCodeGenInfo::checkFunctionCallABI(CodeGenModule &CGM,
     // change its ABI with attribute-target after this call.
     if (Arg.getType()->isVectorType() &&
         CGM.getContext().getTypeSize(Arg.getType()) > 128) {
-      initFeatureMaps(CGM.getContext(), CallerMap, Caller, CalleeMap, Callee);
+      initFeatureMaps(CGM, CallerMap, Caller, CalleeMap, Callee);
       QualType Ty = Arg.getType();
       // The CallArg seems to have desugared the type already, so for clearer
       // diagnostics, replace it with the type in the FunctionDecl if possible.
@@ -1783,7 +1783,7 @@ void X86_64TargetCodeGenInfo::checkFunctionCallABI(CodeGenModule &CGM,
   // whether this value is used, tail-called, etc.
   if (Callee->getReturnType()->isVectorType() &&
       CGM.getContext().getTypeSize(Callee->getReturnType()) > 128) {
-    initFeatureMaps(CGM.getContext(), CallerMap, Caller, CalleeMap, Callee);
+    initFeatureMaps(CGM, CallerMap, Caller, CalleeMap, Callee);
     checkAVXParam(CGM.getDiags(), CGM.getContext(), CallLoc, *Callee, CallerMap,
                   CalleeMap, Callee->getReturnType(),
                   /*IsArgument*/ false);
diff --git a/clang/test/CodeGenCXX/target-features-inferred.cpp b/clang/test/CodeGenCXX/target-features-inferred.cpp
new file mode 100644
index 00000000000000..cbd214ea75a149
--- /dev/null
+++ b/clang/test/CodeGenCXX/target-features-inferred.cpp
@@ -0,0 +1,113 @@
+// RUN: %clang_cc1 %s -triple=x86_64-linux-gnu -emit-llvm -disable-llvm-passes -verify=clean -o - | FileCheck %s
+// RUN: %clang_cc1 %s -triple=x86_64-linux-gnu -emit-llvm -o /dev/null -verify -DERRORS
+
+// An always_inline function that has no target attribute and that only direct
+// calls in this translation unit can reach is compiled with the target
+// features that its callers have in common.
+
+// clean-no-diagnostics
+
+typedef float v8f __attribute__((vector_size(32)));
+
+template <class> struct Zero;
+template <> struct Zero<v8f> {
+  [[gnu::target("avx"), gnu::always_inline]] inline v8f operator()() const {
+    return v8f{};
+  }
+};
+
+template <class T>
+[[gnu::always_inline]] static inline auto direct(T gen) { return gen(); }
+
+// Unlike direct, this is not instantiated until the end of the translation
+// unit, after its caller has been emitted.
+template <class T>
+[[gnu::always_inline]] static inline v8f indirect(T gen) { return direct(gen); }
+
+// Passing a vector on must not be diagnosed as a change of the ABI either.
+[[gnu::always_inline]] static inline v8f pass(v8f x) { return x; }
+
+struct Vec {
+  v8f v;
+  [[gnu::always_inline]] Vec(v8f v) : v(v) {}
+};
+
+[[gnu::target("avx")]] v8f avx_caller() {
+  return Vec(pass(indirect(Zero<v8f>()))).v;
+}
+
+[[gnu::target("avx2"), gnu::always_inline]] static inline int leaf() {
+  return 1;
+}
+[[gnu::always_inline]] static inline int mid() { return leaf(); }
+[[gnu::always_inline]] static inline int top() { return mid(); }
+
+[[gnu::target("avx2")]] int avx2_caller() {
+  return top() + []() __attribute__((always_inline)) { return leaf(); }();
+}
+
+[[gnu::always_inline]] static inline int common() { return 0; }
+[[gnu::target("avx")]] int common_avx() { return common(); }
+[[gnu::target("avx2")]] int common_avx2() { return common(); }
+
+// A call that is never emitted requires nothing of the callers.
+[[gnu::always_inline]] static inline int folded() {
+  if (0)
+    return leaf();
+  return 0;
+}
+int default_caller() { return folded(); }
+
+// Other translation units may give their definitions other features, so the
+// linker must not merge them with this one.
+[[gnu::always_inline]] inline int odr() { return leaf(); }
+[[gnu::target("avx2")]] int odr_caller() { return odr(); }
+
+[[gnu::always_inline]] inline int odr_default() { return 0; }
+int odr_default_caller() { return odr_default(); }
+
+// CHECK-DAG: define dso_local {{.*}} @_Z10avx_callerv() #[[AVX_CALLER:[0-9]+]]
+// CHECK-DAG: define internal {{.*}} @_ZL4passDv8_f({{.*}}) #[[AVX_VECTOR:[0-9]+]]
+// CHECK-DAG: define internal {{.*}} @_ZL8indirectI4ZeroIDv8_fEES1_T_() #[[AVX_VECTOR]]
+// CHECK-DAG: define internal {{.*}} @_ZL6directI4ZeroIDv8_fEEDaT_() #[[AVX_VECTOR]]
+// CHECK-DAG: define internal void @_ZN3VecC1EDv8_f({{.*}}) unnamed_addr #[[AVX:[0-9]+]]
+// CHECK-DAG: define internal void @_ZN3VecC2EDv8_f({{.*}}) unnamed_addr #[[AVX]]
+// CHECK-DAG: define dso_local {{.*}} @_Z11avx2_callerv() #[[AVX2_CALLER:[0-9]+]]
+// CHECK-DAG: define internal {{.*}} @_ZL3topv() #[[AVX2:[0-9]+]]
+// CHECK-DAG: define internal {{.*}} @_ZL3midv() #[[AVX2]]
+// CHECK-DAG: define internal {{.*}} @"_ZZ11avx2_callervENK3$_0clEv"({{.*}}) #[[AVX2]]
+// CHECK-DAG: define internal {{.*}} @_ZL6commonv() #[[AVX]]
+// CHECK-DAG: define dso_local {{.*}} @_Z14default_callerv() #[[DEFAULT_CALLER:[0-9]+]]
+// CHECK-DAG: define internal {{.*}} @_ZL6foldedv() #[[DEFAULT:[0-9]+]]
+// CHECK-DAG: define internal {{.*}} @_Z3odrv() #[[AVX2]]
+// CHECK-DAG: define linkonce_odr {{.*}} @_Z11odr_defaultv() #[[DEFAULT]] comdat
+// CHECK-DAG: attributes #[[AVX_CALLER]] = { {{.*}}"target-features"="[[AVX_FEATURES:[^"]+]]"
+// CHECK-DAG: attributes #[[AVX_VECTOR]] = { alwaysinline {{.*}}"target-features"="[[AVX_FEATURES]]"
+// CHECK-DAG: attributes #[[AVX]] = { alwaysinline {{.*}}"target-features"="[[AVX_FEATURES]]"
+// CHECK-DAG: attributes #[[AVX2_CALLER]] = { {{.*}}"target-features"="[[AVX2_FEATURES:[^"]+]]"
+// CHECK-DAG: attributes #[[AVX2]] = { alwaysinline {{.*}}"target-features"="[[AVX2_FEATURES]]"
+// CHECK-DAG: attributes #[[DEFAULT_CALLER]] = { {{.*}}"target-features"="[[DEFAULT_FEATURES:[^"]+]]"
+// CHECK-DAG: attributes #[[DEFAULT]] = { alwaysinline {{.*}}"target-features"="[[DEFAULT_FEATURES]]"
+
+#ifdef ERRORS
+int disabled() {
+  return top(); // expected-error {{always_inline function 'top' requires target feature 'avx2', but would be inlined into function 'disabled' that is compiled without support for 'avx2'}}
+}
+
+// Functions that can run without having been inlined into a function that was
+// checked keep the features they were declared with.
+[[gnu::always_inline]] static inline int escaping() {
+  return leaf(); // expected-error {{always_inline function 'leaf' requires target feature 'avx2', but would be inlined into function 'escaping' that is compiled without support for 'avx2'}}
+}
+int (*pointer())() { return escaping; }
+
+[[gnu::always_inline]] static inline int initializer() {
+  return leaf(); // expected-error {{always_inline function 'leaf' requires target feature 'avx2', but would be inlined into function 'initializer' that is compiled without support for 'avx2'}}
+}
+int global = initializer();
+
+[[gnu::always_inline]] int external() {
+  return leaf(); // expected-error {{always_inline function 'leaf' requires target feature 'avx2', but would be inlined into function 'external' that is compiled without support for 'avx2'}}
+}
+[[gnu::target("avx2")]] int external_caller() { return external(); }
+#endif



More information about the cfe-commits mailing list