<div dir="ltr">Hi Peter,<br><br>One of added tests shows inconsistent results on expensive-checks builder.<br>It fails time to time:<br><br><a href="http://lab.llvm.org:8011/builders/llvm-clang-x86_64-expensive-checks-win/builds/9404">http://lab.llvm.org:8011/builders/llvm-clang-x86_64-expensive-checks-win/builds/9404</a><br><a href="http://lab.llvm.org:8011/builders/llvm-clang-x86_64-expensive-checks-win/builds/9407">http://lab.llvm.org:8011/builders/llvm-clang-x86_64-expensive-checks-win/builds/9407</a><br>etc.<br>the last:<br><a href="http://lab.llvm.org:8011/builders/llvm-clang-x86_64-expensive-checks-win/builds/9426">http://lab.llvm.org:8011/builders/llvm-clang-x86_64-expensive-checks-win/builds/9426</a><br><br>. . . <br>********************<br>Failing Tests (1):<br>    LLVM :: Transforms/LowerTypeTests/icall-branch-funnel.ll<br><br>Please have a look at this?<br><br>Thanks<br><br>Galina<br><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 9, 2018 at 11:11 AM, Peter Collingbourne via llvm-commits <span dir="ltr"><<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">Author: pcc<br>
Date: Fri Mar  9 11:11:44 2018<br>
New Revision: 327163<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=327163&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project?rev=327163&view=rev</a><br>
Log:<br>
Use branch funnels for virtual calls when retpoline mitigation is enabled.<br>
<br>
The retpoline mitigation for variant 2 of CVE-2017-5715 inhibits the<br>
branch predictor, and as a result it can lead to a measurable loss of<br>
performance. We can reduce the performance impact of retpolined virtual<br>
calls by replacing them with a special construct known as a branch<br>
funnel, which is an instruction sequence that implements virtual calls<br>
to a set of known targets using a binary tree of direct branches. This<br>
allows the processor to speculately execute valid implementations of the<br>
virtual function without allowing for speculative execution of of calls<br>
to arbitrary addresses.<br>
<br>
This patch extends the whole-program devirtualization pass to replace<br>
certain virtual calls with calls to branch funnels, which are<br>
represented using a new llvm.icall.jumptable intrinsic. It also extends<br>
the LowerTypeTests pass to recognize the new intrinsic, generate code<br>
for the branch funnels (x86_64 only for now) and lay out virtual tables<br>
as required for each branch funnel.<br>
<br>
The implementation supports full LTO as well as ThinLTO, and extends the<br>
ThinLTO summary format used for whole-program devirtualization to<br>
support branch funnels.<br>
<br>
For more details see RFC:<br>
<a href="http://lists.llvm.org/pipermail/llvm-dev/2018-January/120672.html" rel="noreferrer" target="_blank">http://lists.llvm.org/<wbr>pipermail/llvm-dev/2018-<wbr>January/120672.html</a><br>
<br>
Differential Revision: <a href="https://reviews.llvm.org/D42453" rel="noreferrer" target="_blank">https://reviews.llvm.org/<wbr>D42453</a><br>
<br>
</div></div>Added:<br>
  Â  llvm/trunk/test/CodeGen/X86/<wbr>icall-branch-funnel.ll<br>
  Â  llvm/trunk/test/Transforms/<wbr>LowerTypeTests/icall-branch-<wbr>funnel.ll<br>
  Â  llvm/trunk/test/Transforms/<wbr>WholeProgramDevirt/Inputs/<wbr>import-branch-funnel.yaml<br>
  Â  llvm/trunk/test/Transforms/<wbr>WholeProgramDevirt/Inputs/<wbr>import-vcp-branch-funnel.yaml<br>
  Â  llvm/trunk/test/Transforms/<wbr>WholeProgramDevirt/branch-<wbr>funnel.ll<br>
Modified:<br>
  Â  llvm/trunk/include/llvm/ADT/<wbr>PointerUnion.h<br>
  Â  llvm/trunk/include/llvm/<wbr>CodeGen/TargetOpcodes.def<br>
  Â  llvm/trunk/include/llvm/IR/<wbr>Intrinsics.td<br>
  Â  llvm/trunk/include/llvm/IR/<wbr>ModuleSummaryIndex.h<br>
  Â  llvm/trunk/include/llvm/IR/<wbr>ModuleSummaryIndexYAML.h<br>
  Â  llvm/trunk/include/llvm/<wbr>Target/Target.td<br>
  Â  llvm/trunk/lib/CodeGen/<wbr>SelectionDAG/<wbr>SelectionDAGBuilder.cpp<br>
  Â  llvm/trunk/lib/IR/Verifier.cpp<br>
  Â  llvm/trunk/lib/Target/X86/<wbr>X86ExpandPseudo.cpp<br>
  Â  llvm/trunk/lib/Transforms/IPO/<wbr>LowerTypeTests.cpp<br>
  Â  llvm/trunk/lib/Transforms/IPO/<wbr>WholeProgramDevirt.cpp<br>
  Â  llvm/trunk/test/Transforms/<wbr>WholeProgramDevirt/import.ll<br>
<br>
Modified: llvm/trunk/include/llvm/ADT/<wbr>PointerUnion.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/ADT/PointerUnion.h?rev=327163&r1=327162&r2=327163&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/include/<wbr>llvm/ADT/PointerUnion.h?rev=<wbr>327163&r1=327162&r2=327163&<wbr>view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/include/llvm/ADT/<wbr>PointerUnion.h (original)<br>
+++ llvm/trunk/include/llvm/ADT/<wbr>PointerUnion.h Fri Mar  9 11:11:44 2018<br>
@@ -346,6 +346,12 @@ struct PointerLikeTypeTraits<<wbr>PointerUnio<br>
  Â };<br>
 };<br>
<br>
+template <typename PT1, typename PT2, typename PT3><br>
+bool operator<(PointerUnion3<PT1, PT2, PT3> lhs,<br>
+  Â  Â  Â  Â  Â  Â  Â PointerUnion3<PT1, PT2, PT3> rhs) {<br>
+  return lhs.getOpaqueValue() < rhs.getOpaqueValue();<br>
+}<br>
+<br>
 /// A pointer union of four pointer types. See documentation for PointerUnion<br>
 /// for usage.<br>
 template <typename PT1, typename PT2, typename PT3, typename PT4><br>
<br>
Modified: llvm/trunk/include/llvm/<wbr>CodeGen/TargetOpcodes.def<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/TargetOpcodes.def?rev=327163&r1=327162&r2=327163&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/include/<wbr>llvm/CodeGen/TargetOpcodes.<wbr>def?rev=327163&r1=327162&r2=<wbr>327163&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/include/llvm/<wbr>CodeGen/TargetOpcodes.def (original)<br>
+++ llvm/trunk/include/llvm/<wbr>CodeGen/TargetOpcodes.def Fri Mar  9 11:11:44 2018<br>
@@ -187,6 +187,8 @@ HANDLE_TARGET_OPCODE(<wbr>PATCHABLE_TAIL_CALL<br>
 /// patched to insert instrumentation instructions.<br>
 HANDLE_TARGET_OPCODE(<wbr>PATCHABLE_EVENT_CALL)<br>
<br>
+HANDLE_TARGET_OPCODE(ICALL_<wbr>BRANCH_FUNNEL)<br>
+<br>
 /// The following generic opcodes are not supposed to appear after ISel.<br>
 /// This is something we might want to relax, but for now, this is convenient<br>
 /// to produce diagnostics.<br>
<br>
Modified: llvm/trunk/include/llvm/IR/<wbr>Intrinsics.td<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/Intrinsics.td?rev=327163&r1=327162&r2=327163&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/include/<wbr>llvm/IR/Intrinsics.td?rev=<wbr>327163&r1=327162&r2=327163&<wbr>view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/include/llvm/IR/<wbr>Intrinsics.td (original)<br>
+++ llvm/trunk/include/llvm/IR/<wbr>Intrinsics.td Fri Mar  9 11:11:44 2018<br>
@@ -874,6 +874,10 @@ def int_type_checked_load : Intrinsic<[l<br>
  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â [llvm_ptr_ty, llvm_i32_ty, llvm_metadata_ty],<br>
  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â [IntrNoMem]>;<br>
<br>
+// Create a branch funnel that implements an indirect call to a limited set of<br>
+// callees. This needs to be a musttail call.<br>
+def int_icall_branch_funnel : Intrinsic<[], [llvm_vararg_ty], []>;<br>
+<br>
 def int_load_relative: Intrinsic<[llvm_ptr_ty], [llvm_ptr_ty, llvm_anyint_ty],<br>
  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  [IntrReadMem, IntrArgMemOnly]>;<br>
<br>
<br>
Modified: llvm/trunk/include/llvm/IR/<wbr>ModuleSummaryIndex.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/ModuleSummaryIndex.h?rev=327163&r1=327162&r2=327163&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/include/<wbr>llvm/IR/ModuleSummaryIndex.h?<wbr>rev=327163&r1=327162&r2=<wbr>327163&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/include/llvm/IR/<wbr>ModuleSummaryIndex.h (original)<br>
+++ llvm/trunk/include/llvm/IR/<wbr>ModuleSummaryIndex.h Fri Mar  9 11:11:44 2018<br>
@@ -656,8 +656,11 @@ struct TypeTestResolution {<br>
<br>
 struct WholeProgramDevirtResolution {<br>
  Â enum Kind {<br>
-  Â  Indir,  Â  Â  ///< Just do a regular virtual call<br>
-  Â  SingleImpl, ///< Single implementation devirtualization<br>
+  Â  Indir,  Â  Â  Â  ///< Just do a regular virtual call<br>
+  Â  SingleImpl,  Â ///< Single implementation devirtualization<br>
+  Â  BranchFunnel, ///< When retpoline mitigation is enabled, use a branch funnel<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  ///< that is defined in the merged module. Otherwise same as<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  ///< Indir.<br>
  Â } TheKind = Indir;<br>
<br>
  Â std::string SingleImplName;<br>
<br>
Modified: llvm/trunk/include/llvm/IR/<wbr>ModuleSummaryIndexYAML.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/ModuleSummaryIndexYAML.h?rev=327163&r1=327162&r2=327163&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/include/<wbr>llvm/IR/<wbr>ModuleSummaryIndexYAML.h?rev=<wbr>327163&r1=327162&r2=327163&<wbr>view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/include/llvm/IR/<wbr>ModuleSummaryIndexYAML.h (original)<br>
+++ llvm/trunk/include/llvm/IR/<wbr>ModuleSummaryIndexYAML.h Fri Mar  9 11:11:44 2018<br>
@@ -98,6 +98,8 @@ template <> struct ScalarEnumerationTrai<br>
  Â static void enumeration(IO &io, WholeProgramDevirtResolution::<wbr>Kind &value) {<br>
  Â  Â io.enumCase(value, "Indir", WholeProgramDevirtResolution::<wbr>Indir);<br>
  Â  Â io.enumCase(value, "SingleImpl", WholeProgramDevirtResolution::<wbr>SingleImpl);<br>
+  Â  io.enumCase(value, "BranchFunnel",<br>
+  Â  Â  Â  Â  Â  Â  Â  WholeProgramDevirtResolution::<wbr>BranchFunnel);<br>
  Â }<br>
 };<br>
<br>
<br>
Modified: llvm/trunk/include/llvm/<wbr>Target/Target.td<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Target/Target.td?rev=327163&r1=327162&r2=327163&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/include/<wbr>llvm/Target/Target.td?rev=<wbr>327163&r1=327162&r2=327163&<wbr>view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/include/llvm/<wbr>Target/Target.td (original)<br>
+++ llvm/trunk/include/llvm/<wbr>Target/Target.td Fri Mar  9 11:11:44 2018<br>
@@ -1140,6 +1140,12 @@ def FENTRY_CALL : StandardPseudoInstruct<br>
  Â let mayStore = 1;<br>
  Â let hasSideEffects = 1;<br>
 }<br>
+def ICALL_BRANCH_FUNNEL : StandardPseudoInstruction {<br>
+  let OutOperandList = (outs unknown:$dst);<br>
+  let InOperandList = (ins variable_ops);<br>
+  let AsmString = "";<br>
+  let hasSideEffects = 1;<br>
+}<br>
<br>
 // Generic opcodes used in GlobalISel.<br>
 include "llvm/Target/GenericOpcodes.<wbr>td"<br>
<br>
Modified: llvm/trunk/lib/CodeGen/<wbr>SelectionDAG/<wbr>SelectionDAGBuilder.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp?rev=327163&r1=327162&r2=327163&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/lib/<wbr>CodeGen/SelectionDAG/<wbr>SelectionDAGBuilder.cpp?rev=<wbr>327163&r1=327162&r2=327163&<wbr>view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/lib/CodeGen/<wbr>SelectionDAG/<wbr>SelectionDAGBuilder.cpp (original)<br>
+++ llvm/trunk/lib/CodeGen/<wbr>SelectionDAG/<wbr>SelectionDAGBuilder.cpp Fri Mar  9 11:11:44 2018<br>
@@ -6043,6 +6043,60 @@ SelectionDAGBuilder::<wbr>visitIntrinsicCall(<br>
  Â case Intrinsic::experimental_<wbr>vector_reduce_fmin:<br>
  Â  Â visitVectorReduce(I, Intrinsic);<br>
  Â  Â return nullptr;<br>
+<br>
+  case Intrinsic::icall_branch_<wbr>funnel: {<br>
+  Â  SmallVector<SDValue, 16> Ops;<br>
+  Â  Ops.push_back(DAG.getRoot());<br>
+  Â  Ops.push_back(getValue(I.<wbr>getArgOperand(0)));<br>
+<br>
+  Â  int64_t Offset;<br>
+  Â  auto *Base = dyn_cast<GlobalObject>(<wbr>GetPointerBaseWithConstantOffs<wbr>et(<br>
+  Â  Â  Â  I.getArgOperand(1), Offset, DAG.getDataLayout()));<br>
+  Â  if (!Base)<br>
+  Â  Â  report_fatal_error(<br>
+  Â  Â  Â  Â  "llvm.icall.branch.funnel operand must be a GlobalValue");<br>
+  Â  Ops.push_back(DAG.<wbr>getTargetGlobalAddress(Base, getCurSDLoc(), MVT::i64, 0));<br>
+<br>
+  Â  struct BranchFunnelTarget {<br>
+  Â  Â  int64_t Offset;<br>
+  Â  Â  SDValue Target;<br>
+  Â  };<br>
+  Â  SmallVector<<wbr>BranchFunnelTarget, 8> Targets;<br>
+<br>
+  Â  for (unsigned Op = 1, N = I.getNumArgOperands(); Op != N; Op += 2) {<br>
+  Â  Â  auto *ElemBase = dyn_cast<GlobalObject>(<wbr>GetPointerBaseWithConstantOffs<wbr>et(<br>
+  Â  Â  Â  Â  I.getArgOperand(Op), Offset, DAG.getDataLayout()));<br>
+  Â  Â  if (ElemBase != Base)<br>
+  Â  Â  Â  report_fatal_error("all llvm.icall.branch.funnel operands must refer "<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â "to the same GlobalValue");<br>
+<br>
+  Â  Â  SDValue Val = getValue(I.getArgOperand(Op + 1));<br>
+  Â  Â  auto *GA = dyn_cast<GlobalAddressSDNode>(<wbr>Val);<br>
+  Â  Â  if (!GA)<br>
+  Â  Â  Â  report_fatal_error(<br>
+  Â  Â  Â  Â  Â  "llvm.icall.branch.funnel operand must be a GlobalValue");<br>
+  Â  Â  Targets.push_back({Offset, DAG.getTargetGlobalAddress(<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â GA->getGlobal(), getCurSDLoc(),<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â Val.getValueType(), GA->getOffset())});<br>
+  Â  }<br>
+  Â  std::sort(Targets.begin(), Targets.end(),<br>
+  Â  Â  Â  Â  Â  Â  [](const BranchFunnelTarget &T1, const BranchFunnelTarget &T2) {<br>
+  Â  Â  Â  Â  Â  Â  Â  return T1.Offset < T2.Offset;<br>
+  Â  Â  Â  Â  Â  Â  });<br>
+<br>
+  Â  for (auto &T : Targets) {<br>
+  Â  Â  Ops.push_back(DAG.<wbr>getTargetConstant(T.Offset, getCurSDLoc(), MVT::i32));<br>
+  Â  Â  Ops.push_back(T.Target);<br>
+  Â  }<br>
+<br>
+  Â  SDValue N(DAG.getMachineNode(<wbr>TargetOpcode::ICALL_BRANCH_<wbr>FUNNEL,<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â getCurSDLoc(), MVT::Other, Ops),<br>
+  Â  Â  Â  Â  Â  Â  0);<br>
+  Â  DAG.setRoot(N);<br>
+  Â  setValue(&I, N);<br>
+  Â  HasTailCall = true;<br>
+  Â  return nullptr;<br>
+  }<br>
  Â }<br>
 }<br>
<br>
<br>
Modified: llvm/trunk/lib/IR/Verifier.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/IR/Verifier.cpp?rev=327163&r1=327162&r2=327163&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/lib/IR/<wbr>Verifier.cpp?rev=327163&r1=<wbr>327162&r2=327163&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/lib/IR/Verifier.cpp (original)<br>
+++ llvm/trunk/lib/IR/Verifier.cpp Fri Mar  9 11:11:44 2018<br>
@@ -2864,17 +2864,20 @@ void Verifier::verifyMustTailCall(<wbr>CallIn<br>
  Â Function *F = CI.getParent()->getParent();<br>
  Â FunctionType *CallerTy = F->getFunctionType();<br>
  Â FunctionType *CalleeTy = CI.getFunctionType();<br>
-  Assert(CallerTy->getNumParams(<wbr>) == CalleeTy->getNumParams(),<br>
-  Â  Â  Â  Â "cannot guarantee tail call due to mismatched parameter counts", &CI);<br>
+  if (!CI.getCalledFunction() || !CI.getCalledFunction()-><wbr>isIntrinsic()) {<br>
+  Â  Assert(CallerTy->getNumParams(<wbr>) == CalleeTy->getNumParams(),<br>
+  Â  Â  Â  Â  Â "cannot guarantee tail call due to mismatched parameter counts",<br>
+  Â  Â  Â  Â  Â &CI);<br>
+  Â  for (int I = 0, E = CallerTy->getNumParams(); I != E; ++I) {<br>
+  Â  Â  Assert(<br>
+  Â  Â  Â  Â  isTypeCongruent(CallerTy-><wbr>getParamType(I), CalleeTy->getParamType(I)),<br>
+  Â  Â  Â  Â  "cannot guarantee tail call due to mismatched parameter types", &CI);<br>
+  Â  }<br>
+  }<br>
  Â Assert(CallerTy->isVarArg() == CalleeTy->isVarArg(),<br>
  Â  Â  Â  Â  "cannot guarantee tail call due to mismatched varargs", &CI);<br>
  Â Assert(isTypeCongruent(<wbr>CallerTy->getReturnType(), CalleeTy->getReturnType()),<br>
  Â  Â  Â  Â  "cannot guarantee tail call due to mismatched return types", &CI);<br>
-  for (int I = 0, E = CallerTy->getNumParams(); I != E; ++I) {<br>
-  Â  Assert(<br>
-  Â  Â  Â  isTypeCongruent(CallerTy-><wbr>getParamType(I), CalleeTy->getParamType(I)),<br>
-  Â  Â  Â  "cannot guarantee tail call due to mismatched parameter types", &CI);<br>
-  }<br>
<br>
  Â // - The calling conventions of the caller and callee must match.<br>
  Â Assert(F->getCallingConv() == CI.getCallingConv(),<br>
<br>
Modified: llvm/trunk/lib/Target/X86/<wbr>X86ExpandPseudo.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/X86/X86ExpandPseudo.cpp?rev=327163&r1=327162&r2=327163&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/lib/Target/<wbr>X86/X86ExpandPseudo.cpp?rev=<wbr>327163&r1=327162&r2=327163&<wbr>view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/lib/Target/X86/<wbr>X86ExpandPseudo.cpp (original)<br>
+++ llvm/trunk/lib/Target/X86/<wbr>X86ExpandPseudo.cpp Fri Mar  9 11:11:44 2018<br>
@@ -59,12 +59,112 @@ public:<br>
  Â }<br>
<br>
 private:<br>
+  void ExpandICallBranchFunnel(<wbr>MachineBasicBlock *MBB,<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â MachineBasicBlock::iterator MBBI);<br>
+<br>
  Â bool ExpandMI(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI);<br>
  Â bool ExpandMBB(MachineBasicBlock &MBB);<br>
 };<br>
 char X86ExpandPseudo::ID = 0;<br>
 } // End anonymous namespace.<br>
<br>
+void X86ExpandPseudo::<wbr>ExpandICallBranchFunnel(<br>
+  Â  MachineBasicBlock *MBB, MachineBasicBlock::iterator MBBI) {<br>
+  MachineBasicBlock *JTMBB = MBB;<br>
+  MachineInstr *JTInst = &*MBBI;<br>
+  MachineFunction *MF = MBB->getParent();<br>
+  const BasicBlock *BB = MBB->getBasicBlock();<br>
+  auto InsPt = MachineFunction::iterator(MBB)<wbr>;<br>
+  ++InsPt;<br>
+<br>
+  std::vector<std::pair<<wbr>MachineBasicBlock *, unsigned>> TargetMBBs;<br>
+  DebugLoc DL = JTInst->getDebugLoc();<br>
+  MachineOperand Selector = JTInst->getOperand(0);<br>
+  const GlobalValue *CombinedGlobal = JTInst->getOperand(1).<wbr>getGlobal();<br>
+<br>
+  auto CmpTarget = [&](unsigned Target) {<br>
+  Â  BuildMI(*MBB, MBBI, DL, TII->get(X86::LEA64r), X86::R11)<br>
+  Â  Â  Â  .addReg(X86::RIP)<br>
+  Â  Â  Â  .addImm(1)<br>
+  Â  Â  Â  .addReg(0)<br>
+  Â  Â  Â  .addGlobalAddress(<wbr>CombinedGlobal,<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  JTInst->getOperand(2 + 2 * Target).getImm())<br>
+  Â  Â  Â  .addReg(0);<br>
+  Â  BuildMI(*MBB, MBBI, DL, TII->get(X86::CMP64rr))<br>
+  Â  Â  Â  .add(Selector)<br>
+  Â  Â  Â  .addReg(X86::R11);<br>
+  };<br>
+<br>
+  auto CreateMBB = [&]() {<br>
+  Â  auto *NewMBB = MF->CreateMachineBasicBlock(<wbr>BB);<br>
+  Â  MBB->addSuccessor(NewMBB);<br>
+  Â  return NewMBB;<br>
+  };<br>
+<br>
+  auto EmitCondJump = [&](unsigned Opcode, MachineBasicBlock *ThenMBB) {<br>
+  Â  BuildMI(*MBB, MBBI, DL, TII->get(Opcode)).addMBB(<wbr>ThenMBB);<br>
+<br>
+  Â  auto *ElseMBB = CreateMBB();<br>
+  Â  MF->insert(InsPt, ElseMBB);<br>
+  Â  MBB = ElseMBB;<br>
+  Â  MBBI = MBB->end();<br>
+  };<br>
+<br>
+  auto EmitCondJumpTarget = [&](unsigned Opcode, unsigned Target) {<br>
+  Â  auto *ThenMBB = CreateMBB();<br>
+  Â  TargetMBBs.push_back({ThenMBB, Target});<br>
+  Â  EmitCondJump(Opcode, ThenMBB);<br>
+  };<br>
+<br>
+  auto EmitTailCall = [&](unsigned Target) {<br>
+  Â  BuildMI(*MBB, MBBI, DL, TII->get(X86::TAILJMPd64))<br>
+  Â  Â  Â  .add(JTInst->getOperand(3 + 2 * Target));<br>
+  };<br>
+<br>
+  std::function<void(unsigned, unsigned)> EmitBranchFunnel =<br>
+  Â  Â  [&](unsigned FirstTarget, unsigned NumTargets) {<br>
+  Â  if (NumTargets == 1) {<br>
+  Â  Â  EmitTailCall(FirstTarget);<br>
+  Â  Â  return;<br>
+  Â  }<br>
+<br>
+  Â  if (NumTargets == 2) {<br>
+  Â  Â  CmpTarget(FirstTarget + 1);<br>
+  Â  Â  EmitCondJumpTarget(X86::JB_1, FirstTarget);<br>
+  Â  Â  EmitTailCall(FirstTarget + 1);<br>
+  Â  Â  return;<br>
+  Â  }<br>
+<br>
+  Â  if (NumTargets < 6) {<br>
+  Â  Â  CmpTarget(FirstTarget + 1);<br>
+  Â  Â  EmitCondJumpTarget(X86::JB_1, FirstTarget);<br>
+  Â  Â  EmitCondJumpTarget(X86::JE_1, FirstTarget + 1);<br>
+  Â  Â  EmitBranchFunnel(FirstTarget + 2, NumTargets - 2);<br>
+  Â  Â  return;<br>
+  Â  }<br>
+<br>
+  Â  auto *ThenMBB = CreateMBB();<br>
+  Â  CmpTarget(FirstTarget + (NumTargets / 2));<br>
+  Â  EmitCondJump(X86::JB_1, ThenMBB);<br>
+  Â  EmitCondJumpTarget(X86::JE_1, FirstTarget + (NumTargets / 2));<br>
+  Â  EmitBranchFunnel(FirstTarget + (NumTargets / 2) + 1,<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  NumTargets - (NumTargets / 2) - 1);<br>
+<br>
+  Â  MF->insert(InsPt, ThenMBB);<br>
+  Â  MBB = ThenMBB;<br>
+  Â  MBBI = MBB->end();<br>
+  Â  EmitBranchFunnel(FirstTarget, NumTargets / 2);<br>
+  };<br>
+<br>
+  EmitBranchFunnel(0, (JTInst->getNumOperands() - 2) / 2);<br>
+  for (auto P : TargetMBBs) {<br>
+  Â  MF->insert(InsPt, P.first);<br>
+  Â  BuildMI(P.first, DL, TII->get(X86::TAILJMPd64))<br>
+  Â  Â  Â  .add(JTInst->getOperand(3 + 2 * P.second));<br>
+  }<br>
+  JTMBB->erase(JTInst);<br>
+}<br>
+<br>
 /// If \p MBBI is a pseudo instruction, this method expands<br>
 /// it to the corresponding (sequence of) actual instruction(s).<br>
 /// \returns true if \p MBBI has been expanded.<br>
@@ -259,6 +359,9 @@ bool X86ExpandPseudo::ExpandMI(<wbr>MachineBa<br>
  Â  Â MBBI->eraseFromParent();<br>
  Â  Â return true;<br>
  Â }<br>
+  case TargetOpcode::ICALL_BRANCH_<wbr>FUNNEL:<br>
+  Â  ExpandICallBranchFunnel(&MBB, MBBI);<br>
+  Â  return true;<br>
  Â }<br>
  Â llvm_unreachable("Previous switch has a fallthrough?");<br>
 }<br>
<br>
Modified: llvm/trunk/lib/Transforms/IPO/<wbr>LowerTypeTests.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/IPO/LowerTypeTests.cpp?rev=327163&r1=327162&r2=327163&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/lib/<wbr>Transforms/IPO/LowerTypeTests.<wbr>cpp?rev=327163&r1=327162&r2=<wbr>327163&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/lib/Transforms/IPO/<wbr>LowerTypeTests.cpp (original)<br>
+++ llvm/trunk/lib/Transforms/IPO/<wbr>LowerTypeTests.cpp Fri Mar  9 11:11:44 2018<br>
@@ -8,6 +8,8 @@<br>
 //===-------------------------<wbr>------------------------------<wbr>---------------===//<br>
 //<br>
 // This pass lowers type metadata and calls to the llvm.type.test intrinsic.<br>
+// It also ensures that globals are properly laid out for the<br>
+// llvm.icall.branch.funnel intrinsic.<br>
 // See <a href="http://llvm.org/docs/TypeMetadata.html" rel="noreferrer" target="_blank">http://llvm.org/docs/<wbr>TypeMetadata.html</a> for more information.<br>
 //<br>
 //===-------------------------<wbr>------------------------------<wbr>---------------===//<br>
@@ -25,6 +27,7 @@<br>
 #include "llvm/ADT/TinyPtrVector.h"<br>
 #include "llvm/ADT/Triple.h"<br>
 #include "llvm/Analysis/<wbr>TypeMetadataUtils.h"<br>
+#include "llvm/Analysis/ValueTracking.<wbr>h"<br>
 #include "llvm/IR/Attributes.h"<br>
 #include "llvm/IR/BasicBlock.h"<br>
 #include "llvm/IR/Constant.h"<br>
@@ -291,6 +294,29 @@ public:<br>
  Â }<br>
 };<br>
<br>
+struct ICallBranchFunnel final<br>
+  Â  : TrailingObjects<<wbr>ICallBranchFunnel, GlobalTypeMember *> {<br>
+  static ICallBranchFunnel *create(BumpPtrAllocator &Alloc, CallInst *CI,<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â ArrayRef<GlobalTypeMember *> Targets) {<br>
+  Â  auto *Call = static_cast<ICallBranchFunnel *>(<br>
+  Â  Â  Â  Alloc.Allocate(<wbr>totalSizeToAlloc<<wbr>GlobalTypeMember *>(Targets.size()),<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â alignof(ICallBranchFunnel)));<br>
+  Â  Call->CI = CI;<br>
+  Â  Call->NTargets = Targets.size();<br>
+  Â  std::uninitialized_copy(<wbr>Targets.begin(), Targets.end(),<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Call->getTrailingObjects<<wbr>GlobalTypeMember *>());<br>
+  Â  return Call;<br>
+  }<br>
+<br>
+  CallInst *CI;<br>
+  ArrayRef<GlobalTypeMember *> targets() const {<br>
+  Â  return makeArrayRef(<wbr>getTrailingObjects<<wbr>GlobalTypeMember *>(), NTargets);<br>
+  }<br>
+<br>
+private:<br>
+  size_t NTargets;<br>
+};<br>
+<br>
 class LowerTypeTestsModule {<br>
  Â Module &M;<br>
<br>
@@ -372,6 +398,7 @@ class LowerTypeTestsModule {<br>
  Â  Â  Â const DenseMap<GlobalTypeMember *, uint64_t> &GlobalLayout);<br>
  Â Value *lowerTypeTestCall(Metadata *TypeId, CallInst *CI,<br>
  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  const TypeIdLowering &TIL);<br>
+<br>
  Â void buildBitSetsFromGlobalVariable<wbr>s(ArrayRef<Metadata *> TypeIds,<br>
  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  ArrayRef<GlobalTypeMember *> Globals);<br>
  Â unsigned getJumpTableEntrySize();<br>
@@ -383,11 +410,13 @@ class LowerTypeTestsModule {<br>
  Â void buildBitSetsFromFunctions(<wbr>ArrayRef<Metadata *> TypeIds,<br>
  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  ArrayRef<GlobalTypeMember *> Functions);<br>
  Â void buildBitSetsFromFunctionsNativ<wbr>e(ArrayRef<Metadata *> TypeIds,<br>
-  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  ArrayRef<GlobalTypeMember *> Functions);<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â ArrayRef<GlobalTypeMember *> Functions);<br>
  Â void buildBitSetsFromFunctionsWASM(<wbr>ArrayRef<Metadata *> TypeIds,<br>
  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  ArrayRef<GlobalTypeMember *> Functions);<br>
-  void buildBitSetsFromDisjointSet(<wbr>ArrayRef<Metadata *> TypeIds,<br>
-  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â ArrayRef<GlobalTypeMember *> Globals);<br>
+  void<br>
+  buildBitSetsFromDisjointSet(<wbr>ArrayRef<Metadata *> TypeIds,<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  ArrayRef<GlobalTypeMember *> Globals,<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  ArrayRef<ICallBranchFunnel *> ICallBranchFunnels);<br>
<br>
  Â void replaceWeakDeclarationWithJump<wbr>TablePtr(Function *F, Constant *JT);<br>
  Â void moveInitializerToModuleConstru<wbr>ctor(GlobalVariable *GV);<br>
@@ -1462,7 +1491,8 @@ void LowerTypeTestsModule::<wbr>buildBitSetsF<br>
 }<br>
<br>
 void LowerTypeTestsModule::<wbr>buildBitSetsFromDisjointSet(<br>
-  Â  ArrayRef<Metadata *> TypeIds, ArrayRef<GlobalTypeMember *> Globals) {<br>
+  Â  ArrayRef<Metadata *> TypeIds, ArrayRef<GlobalTypeMember *> Globals,<br>
+  Â  ArrayRef<ICallBranchFunnel *> ICallBranchFunnels) {<br>
  Â DenseMap<Metadata *, uint64_t> TypeIdIndices;<br>
  Â for (unsigned I = 0; I != TypeIds.size(); ++I)<br>
  Â  Â TypeIdIndices[TypeIds[I]] = I;<br>
@@ -1471,15 +1501,25 @@ void LowerTypeTestsModule::<wbr>buildBitSetsF<br>
  Â // the type identifier.<br>
  Â std::vector<std::set<uint64_t><wbr>> TypeMembers(TypeIds.size());<br>
  Â unsigned GlobalIndex = 0;<br>
+  DenseMap<GlobalTypeMember *, uint64_t> GlobalIndices;<br>
  Â for (GlobalTypeMember *GTM : Globals) {<br>
  Â  Â for (MDNode *Type : GTM->types()) {<br>
  Â  Â  Â // Type = { offset, type identifier }<br>
-  Â  Â  unsigned TypeIdIndex = TypeIdIndices[Type-><wbr>getOperand(1)];<br>
-  Â  Â  TypeMembers[TypeIdIndex].<wbr>insert(GlobalIndex);<br>
+  Â  Â  auto I = TypeIdIndices.find(Type-><wbr>getOperand(1));<br>
+  Â  Â  if (I != TypeIdIndices.end())<br>
+  Â  Â  Â  TypeMembers[I->second].insert(<wbr>GlobalIndex);<br>
  Â  Â }<br>
+  Â  GlobalIndices[GTM] = GlobalIndex;<br>
  Â  Â GlobalIndex++;<br>
  Â }<br>
<br>
+  for (ICallBranchFunnel *JT : ICallBranchFunnels) {<br>
+  Â  TypeMembers.emplace_back();<br>
+  Â  std::set<uint64_t> &TMSet = TypeMembers.back();<br>
+  Â  for (GlobalTypeMember *T : JT->targets())<br>
+  Â  Â  TMSet.insert(GlobalIndices[T])<wbr>;<br>
+  }<br>
+<br>
  Â // Order the sets of indices by size. The GlobalLayoutBuilder works best<br>
  Â // when given small index sets first.<br>
  Â std::stable_sort(<br>
@@ -1567,8 +1607,11 @@ bool LowerTypeTestsModule::<wbr>runForTesting<br>
 bool LowerTypeTestsModule::lower() {<br>
  Â Function *TypeTestFunc =<br>
  Â  Â  Â M.getFunction(Intrinsic::<wbr>getName(Intrinsic::type_test))<wbr>;<br>
-  if ((!TypeTestFunc || TypeTestFunc->use_empty()) && !ExportSummary &&<br>
-  Â  Â  !ImportSummary)<br>
+  Function *ICallBranchFunnelFunc =<br>
+  Â  Â  M.getFunction(Intrinsic::<wbr>getName(Intrinsic::icall_<wbr>branch_funnel));<br>
+  if ((!TypeTestFunc || TypeTestFunc->use_empty()) &&<br>
+  Â  Â  (!ICallBranchFunnelFunc || ICallBranchFunnelFunc->use_<wbr>empty()) &&<br>
+  Â  Â  !ExportSummary && !ImportSummary)<br>
  Â  Â return false;<br>
<br>
  Â if (ImportSummary) {<br>
@@ -1580,6 +1623,10 @@ bool LowerTypeTestsModule::lower() {<br>
  Â  Â  Â }<br>
  Â  Â }<br>
<br>
+  Â  if (ICallBranchFunnelFunc && !ICallBranchFunnelFunc->use_<wbr>empty())<br>
+  Â  Â  report_fatal_error(<br>
+  Â  Â  Â  Â  "unexpected call to llvm.icall.branch.funnel during import phase");<br>
+<br>
  Â  Â SmallVector<Function *, 8> Defs;<br>
  Â  Â SmallVector<Function *, 8> Decls;<br>
  Â  Â for (auto &F : M) {<br>
@@ -1604,8 +1651,8 @@ bool LowerTypeTestsModule::lower() {<br>
  Â // Equivalence class set containing type identifiers and the globals that<br>
  Â // reference them. This is used to partition the set of type identifiers in<br>
  Â // the module into disjoint sets.<br>
-  using GlobalClassesTy =<br>
-  Â  Â  EquivalenceClasses<<wbr>PointerUnion<GlobalTypeMember *, Metadata *>>;<br>
+  using GlobalClassesTy = EquivalenceClasses<<br>
+  Â  Â  PointerUnion3<GlobalTypeMember *, Metadata *, ICallBranchFunnel *>>;<br>
  Â GlobalClassesTy GlobalClasses;<br>
<br>
  Â // Verify the type metadata and build a few data structures to let us<br>
@@ -1688,14 +1735,13 @@ bool LowerTypeTestsModule::lower() {<br>
  Â  Â }<br>
  Â }<br>
<br>
+  DenseMap<GlobalObject *, GlobalTypeMember *> GlobalTypeMembers;<br>
  Â for (GlobalObject &GO : M.global_objects()) {<br>
  Â  Â if (isa<GlobalVariable>(GO) && GO.isDeclarationForLinker())<br>
  Â  Â  Â continue;<br>
<br>
  Â  Â Types.clear();<br>
  Â  Â GO.getMetadata(LLVMContext::<wbr>MD_type, Types);<br>
-  Â  if (Types.empty())<br>
-  Â  Â  continue;<br>
<br>
  Â  Â bool IsDefinition = !GO.isDeclarationForLinker();<br>
  Â  Â bool IsExported = false;<br>
@@ -1706,6 +1752,7 @@ bool LowerTypeTestsModule::lower() {<br>
<br>
  Â  Â auto *GTM =<br>
  Â  Â  Â  Â GlobalTypeMember::create(<wbr>Alloc, &GO, IsDefinition, IsExported, Types);<br>
+  Â  GlobalTypeMembers[&GO] = GTM;<br>
  Â  Â for (MDNode *Type : Types) {<br>
  Â  Â  Â verifyTypeMDNode(&GO, Type);<br>
  Â  Â  Â auto &Info = TypeIdInfo[Type->getOperand(1)<wbr>];<br>
@@ -1746,6 +1793,43 @@ bool LowerTypeTestsModule::lower() {<br>
  Â  Â }<br>
  Â }<br>
<br>
+  if (ICallBranchFunnelFunc) {<br>
+  Â  for (const Use &U : ICallBranchFunnelFunc->uses()) {<br>
+  Â  Â  if (Arch != Triple::x86_64)<br>
+  Â  Â  Â  report_fatal_error(<br>
+  Â  Â  Â  Â  Â  "llvm.icall.branch.funnel not supported on this target");<br>
+<br>
+  Â  Â  auto CI = cast<CallInst>(U.getUser());<br>
+<br>
+  Â  Â  std::vector<GlobalTypeMember *> Targets;<br>
+  Â  Â  if (CI->getNumArgOperands() % 2 != 1)<br>
+  Â  Â  Â  report_fatal_error("number of arguments should be odd");<br>
+<br>
+  Â  Â  GlobalClassesTy::member_<wbr>iterator CurSet;<br>
+  Â  Â  for (unsigned I = 1; I != CI->getNumArgOperands(); I += 2) {<br>
+  Â  Â  Â  int64_t Offset;<br>
+  Â  Â  Â  auto *Base = dyn_cast<GlobalObject>(<wbr>GetPointerBaseWithConstantOffs<wbr>et(<br>
+  Â  Â  Â  Â  Â  CI->getOperand(I), Offset, M.getDataLayout()));<br>
+  Â  Â  Â  if (!Base)<br>
+  Â  Â  Â  Â  report_fatal_error(<br>
+  Â  Â  Â  Â  Â  Â  "Expected branch funnel operand to be global value");<br>
+<br>
+  Â  Â  Â  GlobalTypeMember *GTM = GlobalTypeMembers[Base];<br>
+  Â  Â  Â  Targets.push_back(GTM);<br>
+  Â  Â  Â  GlobalClassesTy::member_<wbr>iterator NewSet =<br>
+  Â  Â  Â  Â  Â  GlobalClasses.findLeader(<wbr>GlobalClasses.insert(GTM));<br>
+  Â  Â  Â  if (I == 1)<br>
+  Â  Â  Â  Â  CurSet = NewSet;<br>
+  Â  Â  Â  else<br>
+  Â  Â  Â  Â  CurSet = GlobalClasses.unionSets(<wbr>CurSet, NewSet);<br>
+  Â  Â  }<br>
+<br>
+  Â  Â  GlobalClasses.unionSets(<br>
+  Â  Â  Â  Â  CurSet, GlobalClasses.findLeader(<wbr>GlobalClasses.insert(<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  ICallBranchFunnel::create(<wbr>Alloc, CI, Targets))));<br>
+  Â  }<br>
+  }<br>
+<br>
  Â if (ExportSummary) {<br>
  Â  Â DenseMap<GlobalValue::GUID, TinyPtrVector<Metadata *>> MetadataByGUID;<br>
  Â  Â for (auto &P : TypeIdInfo) {<br>
@@ -1798,13 +1882,16 @@ bool LowerTypeTestsModule::lower() {<br>
  Â  Â // Build the list of type identifiers in this disjoint set.<br>
  Â  Â std::vector<Metadata *> TypeIds;<br>
  Â  Â std::vector<GlobalTypeMember *> Globals;<br>
+  Â  std::vector<ICallBranchFunnel *> ICallBranchFunnels;<br>
  Â  Â for (GlobalClassesTy::member_<wbr>iterator MI =<br>
  Â  Â  Â  Â  Â  Â  GlobalClasses.member_begin(S.<wbr>first);<br>
  Â  Â  Â  Â  MI != GlobalClasses.member_end(); ++MI) {<br>
-  Â  Â  if ((*MI).is<Metadata *>())<br>
+  Â  Â  if (MI->is<Metadata *>())<br>
  Â  Â  Â  Â TypeIds.push_back(MI->get<<wbr>Metadata *>());<br>
-  Â  Â  else<br>
+  Â  Â  else if (MI->is<GlobalTypeMember *>())<br>
  Â  Â  Â  Â Globals.push_back(MI->get<<wbr>GlobalTypeMember *>());<br>
+  Â  Â  else<br>
+  Â  Â  Â  ICallBranchFunnels.push_back(<wbr>MI->get<ICallBranchFunnel *>());<br>
  Â  Â }<br>
<br>
  Â  Â // Order type identifiers by global index for determinism. This ordering is<br>
@@ -1814,7 +1901,7 @@ bool LowerTypeTestsModule::lower() {<br>
  Â  Â });<br>
<br>
  Â  Â // Build bitsets for this disjoint set.<br>
-  Â  buildBitSetsFromDisjointSet(<wbr>TypeIds, Globals);<br>
+  Â  buildBitSetsFromDisjointSet(<wbr>TypeIds, Globals, ICallBranchFunnels);<br>
  Â }<br>
<br>
  Â allocateByteArrays();<br>
<br>
Modified: llvm/trunk/lib/Transforms/IPO/<wbr>WholeProgramDevirt.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/IPO/WholeProgramDevirt.cpp?rev=327163&r1=327162&r2=327163&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/lib/<wbr>Transforms/IPO/<wbr>WholeProgramDevirt.cpp?rev=<wbr>327163&r1=327162&r2=327163&<wbr>view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/lib/Transforms/IPO/<wbr>WholeProgramDevirt.cpp (original)<br>
+++ llvm/trunk/lib/Transforms/IPO/<wbr>WholeProgramDevirt.cpp Fri Mar  9 11:11:44 2018<br>
@@ -316,12 +316,17 @@ struct CallSiteInfo {<br>
  Â /// cases we are directly operating on the call sites at the IR level.<br>
  Â std::vector<VirtualCallSite> CallSites;<br>
<br>
+  /// Whether all call sites represented by this CallSiteInfo, including those<br>
+  /// in summaries, have been devirtualized. This starts off as true because a<br>
+  /// default constructed CallSiteInfo represents no call sites.<br>
+  bool AllCallSitesDevirted = true;<br>
+<br>
  Â // These fields are used during the export phase of ThinLTO and reflect<br>
  Â // information collected from function summaries.<br>
<br>
  Â /// Whether any function summary contains an llvm.assume(llvm.type.test) for<br>
  Â /// this slot.<br>
-  bool SummaryHasTypeTestAssumeUsers;<br>
+  bool SummaryHasTypeTestAssumeUsers = false;<br>
<br>
  Â /// CFI-specific: a vector containing the list of function summaries that use<br>
  Â /// the llvm.type.checked.load intrinsic and therefore will require<br>
@@ -337,8 +342,22 @@ struct CallSiteInfo {<br>
  Â  Â  Â  Â  Â  !SummaryTypeCheckedLoadUsers.<wbr>empty();<br>
  Â }<br>
<br>
-  /// As explained in the comment for SummaryTypeCheckedLoadUsers.<br>
-  void markDevirt() { SummaryTypeCheckedLoadUsers.<wbr>clear(); }<br>
+  void markSummaryHasTypeTestAssumeUs<wbr>ers() {<br>
+  Â  SummaryHasTypeTestAssumeUsers = true;<br>
+  Â  AllCallSitesDevirted = false;<br>
+  }<br>
+<br>
+  void addSummaryTypeCheckedLoadUser(<wbr>FunctionSummary *FS) {<br>
+  Â  SummaryTypeCheckedLoadUsers.<wbr>push_back(FS);<br>
+  Â  AllCallSitesDevirted = false;<br>
+  }<br>
+<br>
+  void markDevirt() {<br>
+  Â  AllCallSitesDevirted = true;<br>
+<br>
+  Â  // As explained in the comment for SummaryTypeCheckedLoadUsers.<br>
+  Â  SummaryTypeCheckedLoadUsers.<wbr>clear();<br>
+  }<br>
 };<br>
<br>
 // Call site information collected for a specific VTableSlot.<br>
@@ -373,7 +392,9 @@ CallSiteInfo &VTableSlotInfo::findCallSi<br>
<br>
 void VTableSlotInfo::addCallSite(<wbr>Value *VTable, CallSite CS,<br>
  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  unsigned *NumUnsafeUses) {<br>
-  findCallSiteInfo(CS).<wbr>CallSites.push_back({VTable, CS, NumUnsafeUses});<br>
+  auto &CSI = findCallSiteInfo(CS);<br>
+  CSI.AllCallSitesDevirted = false;<br>
+  CSI.CallSites.push_back({<wbr>VTable, CS, NumUnsafeUses});<br>
 }<br>
<br>
 struct DevirtModule {<br>
@@ -438,6 +459,12 @@ struct DevirtModule {<br>
  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  VTableSlotInfo &SlotInfo,<br>
  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  WholeProgramDevirtResolution *Res);<br>
<br>
+  void applyICallBranchFunnel(<wbr>VTableSlotInfo &SlotInfo, Constant *JT,<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  bool &IsExported);<br>
+  void tryICallBranchFunnel(<wbr>MutableArrayRef<<wbr>VirtualCallTarget> TargetsForSlot,<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  VTableSlotInfo &SlotInfo,<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  WholeProgramDevirtResolution *Res, VTableSlot Slot);<br>
+<br>
  Â bool tryEvaluateFunctionsWithArgs(<br>
  Â  Â  Â MutableArrayRef<<wbr>VirtualCallTarget> TargetsForSlot,<br>
  Â  Â  Â ArrayRef<uint64_t> Args);<br>
@@ -471,6 +498,8 @@ struct DevirtModule {<br>
  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  StringRef Name, IntegerType *IntTy,<br>
  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  uint32_t Storage);<br>
<br>
+  Constant *getMemberAddr(const TypeMemberInfo *M);<br>
+<br>
  Â void applyUniqueRetValOpt(<wbr>CallSiteInfo &CSInfo, StringRef FnName, bool IsOne,<br>
  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â Constant *UniqueMemberAddr);<br>
  Â bool tryUniqueRetValOpt(unsigned BitWidth,<br>
@@ -726,10 +755,9 @@ void DevirtModule::<wbr>applySingleImplDevirt<br>
  Â  Â  Â if (VCallSite.NumUnsafeUses)<br>
  Â  Â  Â  Â --*VCallSite.NumUnsafeUses;<br>
  Â  Â }<br>
-  Â  if (CSInfo.isExported()) {<br>
+  Â  if (CSInfo.isExported())<br>
  Â  Â  Â IsExported = true;<br>
-  Â  Â  CSInfo.markDevirt();<br>
-  Â  }<br>
+  Â  CSInfo.markDevirt();<br>
  Â };<br>
  Â Apply(SlotInfo.CSInfo);<br>
  Â for (auto &P : SlotInfo.ConstCSInfo)<br>
@@ -785,6 +813,134 @@ bool DevirtModule::<wbr>trySingleImplDevirt(<br>
  Â return true;<br>
 }<br>
<br>
+void DevirtModule::<wbr>tryICallBranchFunnel(<br>
+  Â  MutableArrayRef<<wbr>VirtualCallTarget> TargetsForSlot, VTableSlotInfo &SlotInfo,<br>
+  Â  WholeProgramDevirtResolution *Res, VTableSlot Slot) {<br>
+  Triple T(M.getTargetTriple());<br>
+  if (T.getArch() != Triple::x86_64)<br>
+  Â  return;<br>
+<br>
+  const unsigned kBranchFunnelThreshold = 10;<br>
+  if (TargetsForSlot.size() > kBranchFunnelThreshold)<br>
+  Â  return;<br>
+<br>
+  bool HasNonDevirt = !SlotInfo.CSInfo.<wbr>AllCallSitesDevirted;<br>
+  if (!HasNonDevirt)<br>
+  Â  for (auto &P : SlotInfo.ConstCSInfo)<br>
+  Â  Â  if (!P.second.<wbr>AllCallSitesDevirted) {<br>
+  Â  Â  Â  HasNonDevirt = true;<br>
+  Â  Â  Â  break;<br>
+  Â  Â  }<br>
+<br>
+  if (!HasNonDevirt)<br>
+  Â  return;<br>
+<br>
+  FunctionType *FT =<br>
+  Â  Â  FunctionType::get(Type::<wbr>getVoidTy(M.getContext()), {Int8PtrTy}, true);<br>
+  Function *JT;<br>
+  if (isa<MDString>(Slot.TypeID)) {<br>
+  Â  JT = Function::Create(FT, Function::ExternalLinkage,<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  getGlobalName(Slot, {}, "branch_funnel"), &M);<br>
+  Â  JT->setVisibility(GlobalValue:<wbr>:HiddenVisibility);<br>
+  } else {<br>
+  Â  JT = Function::Create(FT, Function::InternalLinkage, "branch_funnel", &M);<br>
+  }<br>
+  JT->addAttribute(1, Attribute::Nest);<br>
+<br>
+  std::vector<Value *> JTArgs;<br>
+  JTArgs.push_back(JT->arg_<wbr>begin());<br>
+  for (auto &T : TargetsForSlot) {<br>
+  Â  JTArgs.push_back(<wbr>getMemberAddr(<a href="http://T.TM" rel="noreferrer" target="_blank">T.TM</a>));<br>
+  Â  JTArgs.push_back(T.Fn);<br>
+  }<br>
+<br>
+  BasicBlock *BB = BasicBlock::Create(M.<wbr>getContext(), "", JT, nullptr);<br>
+  Constant *Intr =<br>
+  Â  Â  Intrinsic::getDeclaration(&M, llvm::Intrinsic::icall_branch_<wbr>funnel, {});<br>
+<br>
+  auto *CI = CallInst::Create(Intr, JTArgs, "", BB);<br>
+  CI->setTailCallKind(CallInst::<wbr>TCK_MustTail);<br>
+  ReturnInst::Create(M.<wbr>getContext(), nullptr, BB);<br>
+<br>
+  bool IsExported = false;<br>
+  applyICallBranchFunnel(<wbr>SlotInfo, JT, IsExported);<br>
+  if (IsExported)<br>
+  Â  Res->TheKind = WholeProgramDevirtResolution::<wbr>BranchFunnel;<br>
+}<br>
+<br>
+void DevirtModule::<wbr>applyICallBranchFunnel(<wbr>VTableSlotInfo &SlotInfo,<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Constant *JT, bool &IsExported) {<br>
+  auto Apply = [&](CallSiteInfo &CSInfo) {<br>
+  Â  if (CSInfo.isExported())<br>
+  Â  Â  IsExported = true;<br>
+  Â  if (CSInfo.AllCallSitesDevirted)<br>
+  Â  Â  return;<br>
+  Â  for (auto &&VCallSite : CSInfo.CallSites) {<br>
+  Â  Â  CallSite CS = VCallSite.CS;<br>
+<br>
+  Â  Â  // Jump tables are only profitable if the retpoline mitigation is enabled.<br>
+  Â  Â  Attribute FSAttr = CS.getCaller()-><wbr>getFnAttribute("target-<wbr>features");<br>
+  Â  Â  if (FSAttr.hasAttribute(<wbr>Attribute::None) ||<br>
+  Â  Â  Â  Â  !FSAttr.getValueAsString().<wbr>contains("+retpoline"))<br>
+  Â  Â  Â  continue;<br>
+<br>
+  Â  Â  if (RemarksEnabled)<br>
+  Â  Â  Â  VCallSite.emitRemark("branch-<wbr>funnel", JT->getName(), OREGetter);<br>
+<br>
+  Â  Â  // Pass the address of the vtable in the nest register, which is r10 on<br>
+  Â  Â  // x86_64.<br>
+  Â  Â  std::vector<Type *> NewArgs;<br>
+  Â  Â  NewArgs.push_back(Int8PtrTy);<br>
+  Â  Â  for (Type *T : CS.getFunctionType()->params()<wbr>)<br>
+  Â  Â  Â  NewArgs.push_back(T);<br>
+  Â  Â  PointerType *NewFT = PointerType::getUnqual(<br>
+  Â  Â  Â  Â  FunctionType::get(CS.<wbr>getFunctionType()-><wbr>getReturnType(), NewArgs,<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  CS.getFunctionType()-><wbr>isVarArg()));<br>
+<br>
+  Â  Â  IRBuilder<> IRB(CS.getInstruction());<br>
+  Â  Â  std::vector<Value *> Args;<br>
+  Â  Â  Args.push_back(IRB.<wbr>CreateBitCast(VCallSite.<wbr>VTable, Int8PtrTy));<br>
+  Â  Â  for (unsigned I = 0; I != CS.getNumArgOperands(); ++I)<br>
+  Â  Â  Â  Args.push_back(CS.<wbr>getArgOperand(I));<br>
+<br>
+  Â  Â  CallSite NewCS;<br>
+  Â  Â  if (CS.isCall())<br>
+  Â  Â  Â  NewCS = IRB.CreateCall(IRB.<wbr>CreateBitCast(JT, NewFT), Args);<br>
+  Â  Â  else<br>
+  Â  Â  Â  NewCS = IRB.CreateInvoke(<br>
+  Â  Â  Â  Â  Â  IRB.CreateBitCast(JT, NewFT),<br>
+  Â  Â  Â  Â  Â  cast<InvokeInst>(CS.<wbr>getInstruction())-><wbr>getNormalDest(),<br>
+  Â  Â  Â  Â  Â  cast<InvokeInst>(CS.<wbr>getInstruction())-><wbr>getUnwindDest(), Args);<br>
+  Â  Â  NewCS.setCallingConv(CS.<wbr>getCallingConv());<br>
+<br>
+  Â  Â  AttributeList Attrs = CS.getAttributes();<br>
+  Â  Â  std::vector<AttributeSet> NewArgAttrs;<br>
+  Â  Â  NewArgAttrs.push_back(<wbr>AttributeSet::get(<br>
+  Â  Â  Â  Â  M.getContext(), ArrayRef<Attribute>{Attribute:<wbr>:get(<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  M.getContext(), Attribute::Nest)}));<br>
+  Â  Â  for (unsigned I = 0; I + 2 <  Attrs.getNumAttrSets(); ++I)<br>
+  Â  Â  Â  NewArgAttrs.push_back(Attrs.<wbr>getParamAttributes(I));<br>
+  Â  Â  NewCS.setAttributes(<br>
+  Â  Â  Â  Â  AttributeList::get(M.<wbr>getContext(), Attrs.getFnAttributes(),<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â Attrs.getRetAttributes(), NewArgAttrs));<br>
+<br>
+  Â  Â  CS->replaceAllUsesWith(NewCS.<wbr>getInstruction());<br>
+  Â  Â  CS->eraseFromParent();<br>
+<br>
+  Â  Â  // This use is no longer unsafe.<br>
+  Â  Â  if (VCallSite.NumUnsafeUses)<br>
+  Â  Â  Â  --*VCallSite.NumUnsafeUses;<br>
+  Â  }<br>
+  Â  // Don't mark as devirtualized because there may be callers compiled without<br>
+  Â  // retpoline mitigation, which would mean that they are lowered to<br>
+  Â  // llvm.type.test and therefore require an llvm.type.test resolution for the<br>
+  Â  // type identifier.<br>
+  };<br>
+  Apply(SlotInfo.CSInfo);<br>
+  for (auto &P : SlotInfo.ConstCSInfo)<br>
+  Â  Apply(P.second);<br>
+}<br>
+<br>
 bool DevirtModule::<wbr>tryEvaluateFunctionsWithArgs(<br>
  Â  Â MutableArrayRef<<wbr>VirtualCallTarget> TargetsForSlot,<br>
  Â  Â ArrayRef<uint64_t> Args) {<br>
@@ -937,6 +1093,12 @@ void DevirtModule::<wbr>applyUniqueRetValOpt(<br>
  Â CSInfo.markDevirt();<br>
 }<br>
<br>
+Constant *DevirtModule::getMemberAddr(<wbr>const TypeMemberInfo *M) {<br>
+  Constant *C = ConstantExpr::getBitCast(M-><wbr>Bits->GV, Int8PtrTy);<br>
+  return ConstantExpr::<wbr>getGetElementPtr(Int8Ty, C,<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  ConstantInt::get(Int64Ty, M->Offset));<br>
+}<br>
+<br>
 bool DevirtModule::<wbr>tryUniqueRetValOpt(<br>
  Â  Â unsigned BitWidth, MutableArrayRef<<wbr>VirtualCallTarget> TargetsForSlot,<br>
  Â  Â CallSiteInfo &CSInfo, WholeProgramDevirtResolution::<wbr>ByArg *Res,<br>
@@ -956,12 +1118,7 @@ bool DevirtModule::<wbr>tryUniqueRetValOpt(<br>
  Â  Â // checked for a uniform return value in tryUniformRetValOpt.<br>
  Â  Â assert(UniqueMember);<br>
<br>
-  Â  Constant *UniqueMemberAddr =<br>
-  Â  Â  Â  ConstantExpr::getBitCast(<wbr>UniqueMember->Bits->GV, Int8PtrTy);<br>
-  Â  UniqueMemberAddr = ConstantExpr::<wbr>getGetElementPtr(<br>
-  Â  Â  Â  Int8Ty, UniqueMemberAddr,<br>
-  Â  Â  Â  ConstantInt::get(Int64Ty, UniqueMember->Offset));<br>
-<br>
+  Â  Constant *UniqueMemberAddr = getMemberAddr(UniqueMember);<br>
  Â  Â if (CSInfo.isExported()) {<br>
  Â  Â  Â Res->TheKind = WholeProgramDevirtResolution::<wbr>ByArg::UniqueRetVal;<br>
  Â  Â  Â Res->Info = IsOne;<br>
@@ -1348,6 +1505,14 @@ void DevirtModule::<wbr>importResolution(VTab<br>
  Â  Â  Â break;<br>
  Â  Â }<br>
  Â }<br>
+<br>
+  if (Res.TheKind == WholeProgramDevirtResolution::<wbr>BranchFunnel) {<br>
+  Â  auto *JT = M.getOrInsertFunction(<wbr>getGlobalName(Slot, {}, "branch_funnel"),<br>
+  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â Type::getVoidTy(M.getContext()<wbr>));<br>
+  Â  bool IsExported = false;<br>
+  Â  applyICallBranchFunnel(<wbr>SlotInfo, JT, IsExported);<br>
+  Â  assert(!IsExported);<br>
+  }<br>
 }<br>
<br>
 void DevirtModule::<wbr>removeRedundantTypeTests() {<br>
@@ -1417,14 +1582,13 @@ bool DevirtModule::run() {<br>
  Â  Â  Â  Â // FIXME: Only add live functions.<br>
  Â  Â  Â  Â for (FunctionSummary::VFuncId VF : FS->type_test_assume_vcalls()) {<br>
  Â  Â  Â  Â  Â for (Metadata *MD : MetadataByGUID[VF.GUID]) {<br>
-  Â  Â  Â  Â  Â  CallSlots[{MD, VF.Offset}].CSInfo.<wbr>SummaryHasTypeTestAssumeUsers =<br>
-  Â  Â  Â  Â  Â  Â  Â  true;<br>
+  Â  Â  Â  Â  Â  CallSlots[{MD, VF.Offset}]<br>
+  Â  Â  Â  Â  Â  Â  Â  .CSInfo.<wbr>markSummaryHasTypeTestAssumeUs<wbr>ers();<br>
  Â  Â  Â  Â  Â }<br>
  Â  Â  Â  Â }<br>
  Â  Â  Â  Â for (FunctionSummary::VFuncId VF : FS->type_checked_load_vcalls()<wbr>) {<br>
  Â  Â  Â  Â  Â for (Metadata *MD : MetadataByGUID[VF.GUID]) {<br>
-  Â  Â  Â  Â  Â  CallSlots[{MD, VF.Offset}]<br>
-  Â  Â  Â  Â  Â  Â  Â  .CSInfo.<wbr>SummaryTypeCheckedLoadUsers.<wbr>push_back(FS);<br>
+  Â  Â  Â  Â  Â  CallSlots[{MD, VF.Offset}].CSInfo.<wbr>addSummaryTypeCheckedLoadUser(<wbr>FS);<br>
  Â  Â  Â  Â  Â }<br>
  Â  Â  Â  Â }<br>
  Â  Â  Â  Â for (const FunctionSummary::ConstVCall &VC :<br>
@@ -1432,7 +1596,7 @@ bool DevirtModule::run() {<br>
  Â  Â  Â  Â  Â for (Metadata *MD : MetadataByGUID[VC.VFunc.GUID]) {<br>
  Â  Â  Â  Â  Â  Â CallSlots[{MD, VC.VFunc.Offset}]<br>
  Â  Â  Â  Â  Â  Â  Â  Â .ConstCSInfo[VC.Args]<br>
-  Â  Â  Â  Â  Â  Â  Â  .SummaryHasTypeTestAssumeUsers = true;<br>
+  Â  Â  Â  Â  Â  Â  Â  .<wbr>markSummaryHasTypeTestAssumeUs<wbr>ers();<br>
  Â  Â  Â  Â  Â }<br>
  Â  Â  Â  Â }<br>
  Â  Â  Â  Â for (const FunctionSummary::ConstVCall &VC :<br>
@@ -1440,7 +1604,7 @@ bool DevirtModule::run() {<br>
  Â  Â  Â  Â  Â for (Metadata *MD : MetadataByGUID[VC.VFunc.GUID]) {<br>
  Â  Â  Â  Â  Â  Â CallSlots[{MD, VC.VFunc.Offset}]<br>
  Â  Â  Â  Â  Â  Â  Â  Â .ConstCSInfo[VC.Args]<br>
-  Â  Â  Â  Â  Â  Â  Â  .SummaryTypeCheckedLoadUsers.<wbr>push_back(FS);<br>
+  Â  Â  Â  Â  Â  Â  Â  .<wbr>addSummaryTypeCheckedLoadUser(<wbr>FS);<br>
  Â  Â  Â  Â  Â }<br>
  Â  Â  Â  Â }<br>
  Â  Â  Â }<br>
@@ -1464,9 +1628,12 @@ bool DevirtModule::run() {<br>
  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  cast<MDString>(S.first.TypeID)<wbr>->getString())<br>
  Â  Â  Â  Â  Â  Â  Â  Â  Â  .WPDRes[S.first.ByteOffset];<br>
<br>
-  Â  Â  if (!trySingleImplDevirt(<wbr>TargetsForSlot, S.second, Res) &&<br>
-  Â  Â  Â  Â  tryVirtualConstProp(<wbr>TargetsForSlot, S.second, Res, S.first))<br>
-  Â  Â  Â  DidVirtualConstProp = true;<br>
+  Â  Â  if (!trySingleImplDevirt(<wbr>TargetsForSlot, S.second, Res)) {<br>
+  Â  Â  Â  DidVirtualConstProp |=<br>
+  Â  Â  Â  Â  Â  tryVirtualConstProp(<wbr>TargetsForSlot, S.second, Res, S.first);<br>
+<br>
+  Â  Â  Â  tryICallBranchFunnel(<wbr>TargetsForSlot, S.second, Res, S.first);<br>
+  Â  Â  }<br>
<br>
  Â  Â  Â // Collect functions devirtualized at least for one call site for stats.<br>
  Â  Â  Â if (RemarksEnabled)<br>
<br>
Added: llvm/trunk/test/CodeGen/X86/<wbr>icall-branch-funnel.ll<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/CodeGen/X86/icall-branch-funnel.ll?rev=327163&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/test/<wbr>CodeGen/X86/icall-branch-<wbr>funnel.ll?rev=327163&view=auto</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/test/CodeGen/X86/<wbr>icall-branch-funnel.ll (added)<br>
+++ llvm/trunk/test/CodeGen/X86/<wbr>icall-branch-funnel.ll Fri Mar  9 11:11:44 2018<br>
@@ -0,0 +1,170 @@<br>
+; RUN: llc -mtriple=x86_64-unknown-linux < %s | FileCheck %s<br>
+<br>
+@g = external global i8<br>
+<br>
+declare void @f0()<br>
+declare void @f1()<br>
+declare void @f2()<br>
+declare void @f3()<br>
+declare void @f4()<br>
+declare void @f5()<br>
+declare void @f6()<br>
+declare void @f7()<br>
+declare void @f8()<br>
+declare void @f9()<br>
+<br>
+declare void @llvm.icall.branch.funnel(...)<br>
+<br>
+define void @jt2(i8* nest, ...) {<br>
+  ; CHECK: jt2:<br>
+  ; CHECK:  Â  Â  leaq g+1(%rip), %r11<br>
+  ; CHECK-NEXT: cmpq %r11, %r10<br>
+  ; CHECK-NEXT: jae .LBB0_1<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: jmp f0<br>
+  ; CHECK-NEXT: .LBB0_1:<br>
+  ; CHECK-NEXT: jmp f1<br>
+  musttail call void (...) @llvm.icall.branch.funnel(<br>
+  Â  Â  i8* %0,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 0), void ()* @f0,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 1), void ()* @f1,<br>
+  Â  Â  ...<br>
+  )<br>
+  ret void<br>
+}<br>
+<br>
+define void @jt3(i8* nest, ...) {<br>
+  ; CHECK: jt3:<br>
+  ; CHECK:  Â  Â  leaq g+1(%rip), %r11<br>
+  ; CHECK-NEXT: cmpq %r11, %r10<br>
+  ; CHECK-NEXT: jae .LBB1_1<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: jmp f0<br>
+  ; CHECK-NEXT: .LBB1_1:<br>
+  ; CHECK-NEXT: jne .LBB1_2<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: jmp f1<br>
+  ; CHECK-NEXT: .LBB1_2:<br>
+  ; CHECK-NEXT: jmp f2<br>
+  musttail call void (...) @llvm.icall.branch.funnel(<br>
+  Â  Â  i8* %0,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 0), void ()* @f0,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 2), void ()* @f2,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 1), void ()* @f1,<br>
+  Â  Â  ...<br>
+  )<br>
+  ret void<br>
+}<br>
+<br>
+define void @jt7(i8* nest, ...) {<br>
+  ; CHECK: jt7:<br>
+  ; CHECK:  Â  Â  leaq g+3(%rip), %r11<br>
+  ; CHECK-NEXT: cmpq %r11, %r10<br>
+  ; CHECK-NEXT: jae .LBB2_1<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: leaq g+1(%rip), %r11<br>
+  ; CHECK-NEXT: cmpq %r11, %r10<br>
+  ; CHECK-NEXT: jae .LBB2_6<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: jmp f0<br>
+  ; CHECK-NEXT: .LBB2_1:<br>
+  ; CHECK-NEXT: jne .LBB2_2<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: jmp f3<br>
+  ; CHECK-NEXT: .LBB2_6:<br>
+  ; CHECK-NEXT: jne .LBB2_7<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: jmp f1<br>
+  ; CHECK-NEXT: .LBB2_2:<br>
+  ; CHECK-NEXT: leaq g+5(%rip), %r11<br>
+  ; CHECK-NEXT: cmpq %r11, %r10<br>
+  ; CHECK-NEXT: jae .LBB2_3<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: jmp f4<br>
+  ; CHECK-NEXT: .LBB2_7:<br>
+  ; CHECK-NEXT: jmp f2<br>
+  ; CHECK-NEXT: .LBB2_3:<br>
+  ; CHECK-NEXT: jne .LBB2_4<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: jmp f5<br>
+  ; CHECK-NEXT: .LBB2_4:<br>
+  ; CHECK-NEXT: jmp f6<br>
+  musttail call void (...) @llvm.icall.branch.funnel(<br>
+  Â  Â  i8* %0,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 0), void ()* @f0,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 1), void ()* @f1,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 2), void ()* @f2,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 3), void ()* @f3,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 4), void ()* @f4,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 5), void ()* @f5,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 6), void ()* @f6,<br>
+  Â  Â  ...<br>
+  )<br>
+  ret void<br>
+}<br>
+<br>
+define void @jt10(i8* nest, ...) {<br>
+  ; CHECK: jt10:<br>
+  ; CHECK:  Â  Â  leaq g+5(%rip), %r11<br>
+  ; CHECK-NEXT: cmpq %r11, %r10<br>
+  ; CHECK-NEXT: jae .LBB3_1<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: leaq g+1(%rip), %r11<br>
+  ; CHECK-NEXT: cmpq %r11, %r10<br>
+  ; CHECK-NEXT: jae .LBB3_7<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: jmp f0<br>
+  ; CHECK-NEXT: .LBB3_1:<br>
+  ; CHECK-NEXT: jne .LBB3_2<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: jmp f5<br>
+  ; CHECK-NEXT: .LBB3_7:<br>
+  ; CHECK-NEXT: jne .LBB3_8<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: jmp f1<br>
+  ; CHECK-NEXT: .LBB3_2:<br>
+  ; CHECK-NEXT: leaq g+7(%rip), %r11<br>
+  ; CHECK-NEXT: cmpq %r11, %r10<br>
+  ; CHECK-NEXT: jae .LBB3_3<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: jmp f6<br>
+  ; CHECK-NEXT: .LBB3_8:<br>
+  ; CHECK-NEXT: leaq g+3(%rip), %r11<br>
+  ; CHECK-NEXT: cmpq %r11, %r10<br>
+  ; CHECK-NEXT: jae .LBB3_9<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: jmp f2<br>
+  ; CHECK-NEXT: .LBB3_3:<br>
+  ; CHECK-NEXT: jne .LBB3_4<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: jmp f7<br>
+  ; CHECK-NEXT: .LBB3_9:<br>
+  ; CHECK-NEXT: jne .LBB3_10<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: jmp f3<br>
+  ; CHECK-NEXT: .LBB3_4:<br>
+  ; CHECK-NEXT: leaq g+9(%rip), %r11<br>
+  ; CHECK-NEXT: cmpq %r11, %r10<br>
+  ; CHECK-NEXT: jae .LBB3_5<br>
+  ; CHECK-NEXT: #<br>
+  ; CHECK-NEXT: jmp f8<br>
+  ; CHECK-NEXT: .LBB3_10:<br>
+  ; CHECK-NEXT: jmp f4<br>
+  ; CHECK-NEXT: .LBB3_5:<br>
+  ; CHECK-NEXT: jmp f9<br>
+  musttail call void (...) @llvm.icall.branch.funnel(<br>
+  Â  Â  i8* %0,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 0), void ()* @f0,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 1), void ()* @f1,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 2), void ()* @f2,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 3), void ()* @f3,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 4), void ()* @f4,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 5), void ()* @f5,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 6), void ()* @f6,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 7), void ()* @f7,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 8), void ()* @f8,<br>
+  Â  Â  i8* getelementptr (i8, i8* @g, i64 9), void ()* @f9,<br>
+  Â  Â  ...<br>
+  )<br>
+  ret void<br>
+}<br>
<br>
Added: llvm/trunk/test/Transforms/<wbr>LowerTypeTests/icall-branch-<wbr>funnel.ll<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LowerTypeTests/icall-branch-funnel.ll?rev=327163&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/test/<wbr>Transforms/LowerTypeTests/<wbr>icall-branch-funnel.ll?rev=<wbr>327163&view=auto</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/test/Transforms/<wbr>LowerTypeTests/icall-branch-<wbr>funnel.ll (added)<br>
+++ llvm/trunk/test/Transforms/<wbr>LowerTypeTests/icall-branch-<wbr>funnel.ll Fri Mar  9 11:11:44 2018<br>
@@ -0,0 +1,46 @@<br>
+; RUN: opt -S -lowertypetests < %s | FileCheck %s<br>
+<br>
+target datalayout = "e-p:64:64"<br>
+target triple = "x86_64-unknown-linux"<br>
+<br>
+; CHECK: @0 = private constant { i32, [0 x i8], i32 } { i32 1, [0 x i8] zeroinitializer, i32 2 }<br>
+; CHECK: @f1 = alias void (), void ()* @.cfi.jumptable<br>
+; CHECK: @f2 = alias void (), bitcast ([8 x i8]* getelementptr inbounds ([2 x [8 x i8]], [2 x [8 x i8]]* bitcast (void ()* @.cfi.jumptable to [2 x [8 x i8]]*), i64 0, i64 1) to void ()*)<br>
+; CHECK: @g1 = alias i32, getelementptr inbounds ({ i32, [0 x i8], i32 }, { i32, [0 x i8], i32 }* @0, i32 0, i32 0)<br>
+; CHECK: @g2 = alias i32, getelementptr inbounds ({ i32, [0 x i8], i32 }, { i32, [0 x i8], i32 }* @0, i32 0, i32 2)<br>
+<br>
+@g1 = constant i32 1<br>
+@g2 = constant i32 2<br>
+<br>
+define void @f1() {<br>
+  ret void<br>
+}<br>
+<br>
+define void @f2() {<br>
+  ret void<br>
+}<br>
+<br>
+declare void @g1f()<br>
+declare void @g2f()<br>
+<br>
+define void @jt2(i8* nest, ...) {<br>
+  musttail call void (...) @llvm.icall.branch.funnel(<br>
+  Â  Â  i8* %0,<br>
+  Â  Â  i32* @g1, void ()* @g1f,<br>
+  Â  Â  i32* @g2, void ()* @g2f,<br>
+  Â  Â  ...<br>
+  )<br>
+  ret void<br>
+}<br>
+<br>
+define void @jt3(i8* nest, ...) {<br>
+  musttail call void (...) @llvm.icall.branch.funnel(<br>
+  Â  Â  i8* %0,<br>
+  Â  Â  void ()* @f1, void ()* @f1,<br>
+  Â  Â  void ()* @f2, void ()* @f2,<br>
+  Â  Â  ...<br>
+  )<br>
+  ret void<br>
+}<br>
+<br>
+declare void @llvm.icall.branch.funnel(...)<br>
<br>
Added: llvm/trunk/test/Transforms/<wbr>WholeProgramDevirt/Inputs/<wbr>import-branch-funnel.yaml<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/WholeProgramDevirt/Inputs/import-branch-funnel.yaml?rev=327163&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/test/<wbr>Transforms/WholeProgramDevirt/<wbr>Inputs/import-branch-funnel.<wbr>yaml?rev=327163&view=auto</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/test/Transforms/<wbr>WholeProgramDevirt/Inputs/<wbr>import-branch-funnel.yaml (added)<br>
+++ llvm/trunk/test/Transforms/<wbr>WholeProgramDevirt/Inputs/<wbr>import-branch-funnel.yaml Fri Mar  9 11:11:44 2018<br>
@@ -0,0 +1,11 @@<br>
+---<br>
+TypeIdMap:<br>
+  typeid1:<br>
+  Â  WPDRes:<br>
+  Â  Â  0:<br>
+  Â  Â  Â  Kind: BranchFunnel<br>
+  typeid2:<br>
+  Â  WPDRes:<br>
+  Â  Â  8:<br>
+  Â  Â  Â  Kind: BranchFunnel<br>
+...<br>
<br>
Added: llvm/trunk/test/Transforms/<wbr>WholeProgramDevirt/Inputs/<wbr>import-vcp-branch-funnel.yaml<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/WholeProgramDevirt/Inputs/import-vcp-branch-funnel.yaml?rev=327163&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/test/<wbr>Transforms/WholeProgramDevirt/<wbr>Inputs/import-vcp-branch-<wbr>funnel.yaml?rev=327163&view=<wbr>auto</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/test/Transforms/<wbr>WholeProgramDevirt/Inputs/<wbr>import-vcp-branch-funnel.yaml (added)<br>
+++ llvm/trunk/test/Transforms/<wbr>WholeProgramDevirt/Inputs/<wbr>import-vcp-branch-funnel.yaml Fri Mar  9 11:11:44 2018<br>
@@ -0,0 +1,23 @@<br>
+---<br>
+TypeIdMap:<br>
+  typeid1:<br>
+  Â  WPDRes:<br>
+  Â  Â  0:<br>
+  Â  Â  Â  Kind: BranchFunnel<br>
+  Â  Â  Â  ResByArg:<br>
+  Â  Â  Â  Â  1:<br>
+  Â  Â  Â  Â  Â  Kind: VirtualConstProp<br>
+  Â  Â  Â  Â  Â  Info: 0<br>
+  Â  Â  Â  Â  Â  Byte: 42<br>
+  Â  Â  Â  Â  Â  Bit: 0<br>
+  typeid2:<br>
+  Â  WPDRes:<br>
+  Â  Â  8:<br>
+  Â  Â  Â  Kind: BranchFunnel<br>
+  Â  Â  Â  ResByArg:<br>
+  Â  Â  Â  Â  3:<br>
+  Â  Â  Â  Â  Â  Kind: VirtualConstProp<br>
+  Â  Â  Â  Â  Â  Info: 0<br>
+  Â  Â  Â  Â  Â  Byte: 43<br>
+  Â  Â  Â  Â  Â  Bit: 128<br>
+...<br>
<br>
Added: llvm/trunk/test/Transforms/<wbr>WholeProgramDevirt/branch-<wbr>funnel.ll<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/WholeProgramDevirt/branch-funnel.ll?rev=327163&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/test/<wbr>Transforms/WholeProgramDevirt/<wbr>branch-funnel.ll?rev=327163&<wbr>view=auto</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/test/Transforms/<wbr>WholeProgramDevirt/branch-<wbr>funnel.ll (added)<br>
+++ llvm/trunk/test/Transforms/<wbr>WholeProgramDevirt/branch-<wbr>funnel.ll Fri Mar  9 11:11:44 2018<br>
@@ -0,0 +1,155 @@<br>
+; RUN: opt -S -wholeprogramdevirt %s | FileCheck --check-prefixes=CHECK,RETP %s<br>
+; RUN: sed -e 's,+retpoline,-retpoline,g' %s | opt -S -wholeprogramdevirt | FileCheck --check-prefixes=CHECK,NORETP %s<br>
+; RUN: opt -wholeprogramdevirt -wholeprogramdevirt-summary-<wbr>action=export -wholeprogramdevirt-read-<wbr>summary=%S/Inputs/export.yaml -wholeprogramdevirt-write-<wbr>summary=%t -S -o - %s | FileCheck --check-prefixes=CHECK,RETP %s<br>
+; RUN: FileCheck --check-prefix=SUMMARY %s < %t<br>
+<br>
+; SUMMARY:  Â  Â  TypeIdMap:  Â  Â  Â <br>
+; SUMMARY-NEXT:  Â typeid1:  Â  Â  Â  Â <br>
+; SUMMARY-NEXT:  Â  Â TTRes:  Â  Â  Â  Â  Â <br>
+; SUMMARY-NEXT:  Â  Â  Â Kind:  Â  Â  Â  Â  Â  Unsat<br>
+; SUMMARY-NEXT:  Â  Â  Â SizeM1BitWidth:  0<br>
+; SUMMARY-NEXT:  Â  Â  Â AlignLog2:  Â  Â  Â 0<br>
+; SUMMARY-NEXT:  Â  Â  Â SizeM1:  Â  Â  Â  Â  0<br>
+; SUMMARY-NEXT:  Â  Â  Â BitMask:  Â  Â  Â  Â 0<br>
+; SUMMARY-NEXT:  Â  Â  Â InlineBits:  Â  Â  0<br>
+; SUMMARY-NEXT:  Â  Â WPDRes:  Â  Â  Â  Â  <br>
+; SUMMARY-NEXT:  Â  Â  Â 0:  Â  Â  Â  Â  Â  Â  Â <br>
+; SUMMARY-NEXT:  Â  Â  Â  Â Kind:  Â  Â  Â  Â  Â  BranchFunnel<br>
+; SUMMARY-NEXT:  Â  Â  Â  Â SingleImplName:  ''<br>
+; SUMMARY-NEXT:  Â  Â  Â  Â ResByArg:  Â  Â  Â  <br>
+; SUMMARY-NEXT:  Â typeid2:  Â  Â  Â  Â <br>
+; SUMMARY-NEXT:  Â  Â TTRes:  Â  Â  Â  Â  Â <br>
+; SUMMARY-NEXT:  Â  Â  Â Kind:  Â  Â  Â  Â  Â  Unsat<br>
+; SUMMARY-NEXT:  Â  Â  Â SizeM1BitWidth:  0<br>
+; SUMMARY-NEXT:  Â  Â  Â AlignLog2:  Â  Â  Â 0<br>
+; SUMMARY-NEXT:  Â  Â  Â SizeM1:  Â  Â  Â  Â  0<br>
+; SUMMARY-NEXT:  Â  Â  Â BitMask:  Â  Â  Â  Â 0<br>
+; SUMMARY-NEXT:  Â  Â  Â InlineBits:  Â  Â  0<br>
+; SUMMARY-NEXT:  Â  Â WPDRes:  Â  Â  Â  Â  <br>
+; SUMMARY-NEXT:  Â  Â  Â 0:  Â  Â  Â  Â  Â  Â  Â <br>
+; SUMMARY-NEXT:  Â  Â  Â  Â Kind:  Â  Â  Â  Â  Â  Indir<br>
+; SUMMARY-NEXT:  Â  Â  Â  Â SingleImplName:  ''<br>
+; SUMMARY-NEXT:  Â  Â  Â  Â ResByArg:  Â  Â  Â  <br>
+; SUMMARY-NEXT:  Â typeid3:  Â  Â  Â  Â <br>
+; SUMMARY-NEXT:  Â  Â TTRes:  Â  Â  Â  Â  Â <br>
+; SUMMARY-NEXT:  Â  Â  Â Kind:  Â  Â  Â  Â  Â  Unsat<br>
+; SUMMARY-NEXT:  Â  Â  Â SizeM1BitWidth:  0<br>
+; SUMMARY-NEXT:  Â  Â  Â AlignLog2:  Â  Â  Â 0<br>
+; SUMMARY-NEXT:  Â  Â  Â SizeM1:  Â  Â  Â  Â  0<br>
+; SUMMARY-NEXT:  Â  Â  Â BitMask:  Â  Â  Â  Â 0<br>
+; SUMMARY-NEXT:  Â  Â  Â InlineBits:  Â  Â  0<br>
+; SUMMARY-NEXT:  Â  Â WPDRes:  Â  Â  Â  Â  <br>
+; SUMMARY-NEXT:  Â  Â  Â 0:  Â  Â  Â  Â  Â  Â  Â <br>
+; SUMMARY-NEXT:  Â  Â  Â  Â Kind:  Â  Â  Â  Â  Â  BranchFunnel<br>
+; SUMMARY-NEXT:  Â  Â  Â  Â SingleImplName:  ''<br>
+; SUMMARY-NEXT:  Â  Â  Â  Â ResByArg:  Â  Â  Â  <br>
+<br>
+target datalayout = "e-p:64:64"<br>
+target triple = "x86_64-unknown-linux-gnu"<br>
+<br>
+@vt1_1 = constant [1 x i8*] [i8* bitcast (i32 (i8*, i32)* @vf1_1 to i8*)], !type !0<br>
+@vt1_2 = constant [1 x i8*] [i8* bitcast (i32 (i8*, i32)* @vf1_2 to i8*)], !type !0<br>
+<br>
+declare i32 @vf1_1(i8* %this, i32 %arg)<br>
+declare i32 @vf1_2(i8* %this, i32 %arg)<br>
+<br>
+@vt2_1 = constant [1 x i8*] [i8* bitcast (i32 (i8*, i32)* @vf2_1 to i8*)], !type !1<br>
+@vt2_2 = constant [1 x i8*] [i8* bitcast (i32 (i8*, i32)* @vf2_2 to i8*)], !type !1<br>
+@vt2_3 = constant [1 x i8*] [i8* bitcast (i32 (i8*, i32)* @vf2_3 to i8*)], !type !1<br>
+@vt2_4 = constant [1 x i8*] [i8* bitcast (i32 (i8*, i32)* @vf2_4 to i8*)], !type !1<br>
+@vt2_5 = constant [1 x i8*] [i8* bitcast (i32 (i8*, i32)* @vf2_5 to i8*)], !type !1<br>
+@vt2_6 = constant [1 x i8*] [i8* bitcast (i32 (i8*, i32)* @vf2_6 to i8*)], !type !1<br>
+@vt2_7 = constant [1 x i8*] [i8* bitcast (i32 (i8*, i32)* @vf2_7 to i8*)], !type !1<br>
+@vt2_8 = constant [1 x i8*] [i8* bitcast (i32 (i8*, i32)* @vf2_8 to i8*)], !type !1<br>
+@vt2_9 = constant [1 x i8*] [i8* bitcast (i32 (i8*, i32)* @vf2_9 to i8*)], !type !1<br>
+@vt2_10 = constant [1 x i8*] [i8* bitcast (i32 (i8*, i32)* @vf2_10 to i8*)], !type !1<br>
+@vt2_11 = constant [1 x i8*] [i8* bitcast (i32 (i8*, i32)* @vf2_11 to i8*)], !type !1<br>
+<br>
+declare i32 @vf2_1(i8* %this, i32 %arg)<br>
+declare i32 @vf2_2(i8* %this, i32 %arg)<br>
+declare i32 @vf2_3(i8* %this, i32 %arg)<br>
+declare i32 @vf2_4(i8* %this, i32 %arg)<br>
+declare i32 @vf2_5(i8* %this, i32 %arg)<br>
+declare i32 @vf2_6(i8* %this, i32 %arg)<br>
+declare i32 @vf2_7(i8* %this, i32 %arg)<br>
+declare i32 @vf2_8(i8* %this, i32 %arg)<br>
+declare i32 @vf2_9(i8* %this, i32 %arg)<br>
+declare i32 @vf2_10(i8* %this, i32 %arg)<br>
+declare i32 @vf2_11(i8* %this, i32 %arg)<br>
+<br>
+@vt3_1 = constant [1 x i8*] [i8* bitcast (i32 (i8*, i32)* @vf3_1 to i8*)], !type !2<br>
+@vt3_2 = constant [1 x i8*] [i8* bitcast (i32 (i8*, i32)* @vf3_2 to i8*)], !type !2<br>
+<br>
+declare i32 @vf3_1(i8* %this, i32 %arg)<br>
+declare i32 @vf3_2(i8* %this, i32 %arg)<br>
+<br>
+@vt4_1 = constant [1 x i8*] [i8* bitcast (i32 (i8*, i32)* @vf4_1 to i8*)], !type !3<br>
+@vt4_2 = constant [1 x i8*] [i8* bitcast (i32 (i8*, i32)* @vf4_2 to i8*)], !type !3<br>
+<br>
+declare i32 @vf4_1(i8* %this, i32 %arg)<br>
+declare i32 @vf4_2(i8* %this, i32 %arg)<br>
+<br>
+; CHECK: define i32 @fn1<br>
+define i32 @fn1(i8* %obj) #0 {<br>
+  %vtableptr = bitcast i8* %obj to [1 x i8*]**<br>
+  %vtable = load [1 x i8*]*, [1 x i8*]** %vtableptr<br>
+  %vtablei8 = bitcast [1 x i8*]* %vtable to i8*<br>
+  %p = call i1 @llvm.type.test(i8* %vtablei8, metadata !"typeid1")<br>
+  call void @llvm.assume(i1 %p)<br>
+  %fptrptr = getelementptr [1 x i8*], [1 x i8*]* %vtable, i32 0, i32 0<br>
+  %fptr = load i8*, i8** %fptrptr<br>
+  %fptr_casted = bitcast i8* %fptr to i32 (i8*, i32)*<br>
+  ; RETP: {{.*}} = bitcast {{.*}} to i8*<br>
+  ; RETP: [[VT1:%.*]] = bitcast {{.*}} to i8*<br>
+  ; RETP: call i32 bitcast (void (i8*, ...)* @__typeid_typeid1_0_branch_<wbr>funnel to i32 (i8*, i8*, i32)*)(i8* nest [[VT1]], i8* %obj, i32 1)<br>
+  %result = call i32 %fptr_casted(i8* %obj, i32 1)<br>
+  ; NORETP: call i32 %<br>
+  ret i32 %result<br>
+}<br>
+<br>
+; CHECK: define i32 @fn2<br>
+define i32 @fn2(i8* %obj) #0 {<br>
+  %vtableptr = bitcast i8* %obj to [1 x i8*]**<br>
+  %vtable = load [1 x i8*]*, [1 x i8*]** %vtableptr<br>
+  %vtablei8 = bitcast [1 x i8*]* %vtable to i8*<br>
+  %p = call i1 @llvm.type.test(i8* %vtablei8, metadata !"typeid2")<br>
+  call void @llvm.assume(i1 %p)<br>
+  %fptrptr = getelementptr [1 x i8*], [1 x i8*]* %vtable, i32 0, i32 0<br>
+  %fptr = load i8*, i8** %fptrptr<br>
+  %fptr_casted = bitcast i8* %fptr to i32 (i8*, i32)*<br>
+  ; CHECK: call i32 %<br>
+  %result = call i32 %fptr_casted(i8* %obj, i32 1)<br>
+  ret i32 %result<br>
+}<br>
+<br>
+; CHECK: define i32 @fn3<br>
+define i32 @fn3(i8* %obj) #0 {<br>
+  %vtableptr = bitcast i8* %obj to [1 x i8*]**<br>
+  %vtable = load [1 x i8*]*, [1 x i8*]** %vtableptr<br>
+  %vtablei8 = bitcast [1 x i8*]* %vtable to i8*<br>
+  %p = call i1 @llvm.type.test(i8* %vtablei8, metadata !4)<br>
+  call void @llvm.assume(i1 %p)<br>
+  %fptrptr = getelementptr [1 x i8*], [1 x i8*]* %vtable, i32 0, i32 0<br>
+  %fptr = load i8*, i8** %fptrptr<br>
+  %fptr_casted = bitcast i8* %fptr to i32 (i8*, i32)*<br>
+  ; RETP: call i32 bitcast (void (i8*, ...)* @branch_funnel to<br>
+  ; NORETP: call i32 %<br>
+  %result = call i32 %fptr_casted(i8* %obj, i32 1)<br>
+  ret i32 %result<br>
+}<br>
+<br>
+; CHECK: define internal void @branch_funnel(i8* nest, ...)<br>
+<br>
+; CHECK: define hidden void @__typeid_typeid1_0_branch_<wbr>funnel(i8* nest, ...)<br>
+; CHECK-NEXT: call void (...) @llvm.icall.branch.funnel(i8* %0, i8* bitcast ([1 x i8*]* @vt1_1 to i8*), i32 (i8*, i32)* @vf1_1, i8* bitcast ([1 x i8*]* @vt1_2 to i8*), i32 (i8*, i32)* @vf1_2, ...)<br>
+<br>
+declare i1 @llvm.type.test(i8*, metadata)<br>
+declare void @llvm.assume(i1)<br>
+<br>
+!0 = !{i32 0, !"typeid1"}<br>
+!1 = !{i32 0, !"typeid2"}<br>
+!2 = !{i32 0, !"typeid3"}<br>
+!3 = !{i32 0, !4}<br>
+!4 = distinct !{}<br>
+<br>
+attributes #0 = { "target-features"="+retpoline" }<br>
<br>
Modified: llvm/trunk/test/Transforms/<wbr>WholeProgramDevirt/import.ll<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/WholeProgramDevirt/import.ll?rev=327163&r1=327162&r2=327163&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/test/<wbr>Transforms/WholeProgramDevirt/<wbr>import.ll?rev=327163&r1=<wbr>327162&r2=327163&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/test/Transforms/<wbr>WholeProgramDevirt/import.ll (original)<br>
+++ llvm/trunk/test/Transforms/<wbr>WholeProgramDevirt/import.ll Fri Mar  9 11:11:44 2018<br>
@@ -1,10 +1,12 @@<br>
 ; RUN: opt -S -wholeprogramdevirt -wholeprogramdevirt-summary-<wbr>action=import -wholeprogramdevirt-read-<wbr>summary=%S/Inputs/import-<wbr>single-impl.yaml < %s | FileCheck --check-prefixes=CHECK,SINGLE-<wbr>IMPL %s<br>
-; RUN: opt -S -wholeprogramdevirt -wholeprogramdevirt-summary-<wbr>action=import -wholeprogramdevirt-read-<wbr>summary=%S/Inputs/import-<wbr>uniform-ret-val.yaml < %s | FileCheck --check-prefixes=CHECK,<wbr>UNIFORM-RET-VAL %s<br>
-; RUN: opt -S -wholeprogramdevirt -wholeprogramdevirt-summary-<wbr>action=import -wholeprogramdevirt-read-<wbr>summary=%S/Inputs/import-<wbr>unique-ret-val0.yaml < %s | FileCheck --check-prefixes=CHECK,UNIQUE-<wbr>RET-VAL0 %s<br>
-; RUN: opt -S -wholeprogramdevirt -wholeprogramdevirt-summary-<wbr>action=import -wholeprogramdevirt-read-<wbr>summary=%S/Inputs/import-<wbr>unique-ret-val1.yaml < %s | FileCheck --check-prefixes=CHECK,UNIQUE-<wbr>RET-VAL1 %s<br>
-; RUN: opt -S -wholeprogramdevirt -wholeprogramdevirt-summary-<wbr>action=import -wholeprogramdevirt-read-<wbr>summary=%S/Inputs/import-vcp.<wbr>yaml < %s | FileCheck --check-prefixes=CHECK,VCP,<wbr>VCP-X86,VCP64 %s<br>
+; RUN: opt -S -wholeprogramdevirt -wholeprogramdevirt-summary-<wbr>action=import -wholeprogramdevirt-read-<wbr>summary=%S/Inputs/import-<wbr>uniform-ret-val.yaml < %s | FileCheck --check-prefixes=CHECK,INDIR,<wbr>UNIFORM-RET-VAL %s<br>
+; RUN: opt -S -wholeprogramdevirt -wholeprogramdevirt-summary-<wbr>action=import -wholeprogramdevirt-read-<wbr>summary=%S/Inputs/import-<wbr>unique-ret-val0.yaml < %s | FileCheck --check-prefixes=CHECK,INDIR,<wbr>UNIQUE-RET-VAL0 %s<br>
+; RUN: opt -S -wholeprogramdevirt -wholeprogramdevirt-summary-<wbr>action=import -wholeprogramdevirt-read-<wbr>summary=%S/Inputs/import-<wbr>unique-ret-val1.yaml < %s | FileCheck --check-prefixes=CHECK,INDIR,<wbr>UNIQUE-RET-VAL1 %s<br>
+; RUN: opt -S -wholeprogramdevirt -wholeprogramdevirt-summary-<wbr>action=import -wholeprogramdevirt-read-<wbr>summary=%S/Inputs/import-vcp.<wbr>yaml < %s | FileCheck --check-prefixes=CHECK,VCP,<wbr>VCP-X86,VCP64,INDIR %s<br>
 ; RUN: opt -S -wholeprogramdevirt -wholeprogramdevirt-summary-<wbr>action=import -wholeprogramdevirt-read-<wbr>summary=%S/Inputs/import-vcp.<wbr>yaml -mtriple=i686-unknown-linux -data-layout=e-p:32:32 < %s | FileCheck --check-prefixes=CHECK,VCP,<wbr>VCP-X86,VCP32 %s<br>
 ; RUN: opt -S -wholeprogramdevirt -wholeprogramdevirt-summary-<wbr>action=import -wholeprogramdevirt-read-<wbr>summary=%S/Inputs/import-vcp.<wbr>yaml -mtriple=armv7-unknown-linux -data-layout=e-p:32:32 < %s | FileCheck --check-prefixes=CHECK,VCP,<wbr>VCP-ARM %s<br>
+; RUN: opt -S -wholeprogramdevirt -wholeprogramdevirt-summary-<wbr>action=import -wholeprogramdevirt-read-<wbr>summary=%S/Inputs/import-vcp-<wbr>branch-funnel.yaml < %s | FileCheck --check-prefixes=CHECK,VCP,<wbr>VCP-X86,VCP64,BRANCH-FUNNEL %s<br>
+; RUN: opt -S -wholeprogramdevirt -wholeprogramdevirt-summary-<wbr>action=import -wholeprogramdevirt-read-<wbr>summary=%S/Inputs/import-<wbr>branch-funnel.yaml < %s | FileCheck --check-prefixes=CHECK,BRANCH-<wbr>FUNNEL,BRANCH-FUNNEL-NOVCP %s<br>
<br>
 target datalayout = "e-p:64:64"<br>
 target triple = "x86_64-unknown-linux-gnu"<br>
@@ -18,7 +20,7 @@ target triple = "x86_64-unknown-linux-gn<br>
 ; constant propagation.<br>
<br>
 ; CHECK: define i32 @call1<br>
-define i32 @call1(i8* %obj) {<br>
+define i32 @call1(i8* %obj) #0 {<br>
  Â %vtableptr = bitcast i8* %obj to [3 x i8*]**<br>
  Â %vtable = load [3 x i8*]*, [3 x i8*]** %vtableptr<br>
  Â %vtablei8 = bitcast [3 x i8*]* %vtable to i8*<br>
@@ -27,16 +29,18 @@ define i32 @call1(i8* %obj) {<br>
  Â %fptrptr = getelementptr [3 x i8*], [3 x i8*]* %vtable, i32 0, i32 0<br>
  Â %fptr = load i8*, i8** %fptrptr<br>
  Â %fptr_casted = bitcast i8* %fptr to i32 (i8*, i32)*<br>
+  ; CHECK: {{.*}} = bitcast {{.*}} to i8*<br>
+  ; VCP: [[VT1:%.*]] = bitcast {{.*}} to i8*<br>
  Â ; SINGLE-IMPL: call i32 bitcast (void ()* @singleimpl1 to i32 (i8*, i32)*)<br>
  Â %result = call i32 %fptr_casted(i8* %obj, i32 1)<br>
  Â ; UNIFORM-RET-VAL: ret i32 42<br>
-  ; VCP: {{.*}} = bitcast {{.*}} to i8*<br>
-  ; VCP: [[VT1:%.*]] = bitcast {{.*}} to i8*<br>
  Â ; VCP-X86: [[GEP1:%.*]] = getelementptr i8, i8* [[VT1]], i32 ptrtoint (i8* @__typeid_typeid1_0_1_byte to i32)<br>
  Â ; VCP-ARM: [[GEP1:%.*]] = getelementptr i8, i8* [[VT1]], i32 42<br>
  Â ; VCP: [[BC1:%.*]] = bitcast i8* [[GEP1]] to i32*<br>
  Â ; VCP: [[LOAD1:%.*]] = load i32, i32* [[BC1]]<br>
  Â ; VCP: ret i32 [[LOAD1]]<br>
+  ; BRANCH-FUNNEL-NOVCP: [[VT1:%.*]] = bitcast {{.*}} to i8*<br>
+  ; BRANCH-FUNNEL-NOVCP: call i32 bitcast (void ()* @__typeid_typeid1_0_branch_<wbr>funnel to i32 (i8*, i8*, i32)*)(i8* nest [[VT1]], i8* %obj, i32 1)<br>
  Â ret i32 %result<br>
 }<br>
<br>
@@ -44,7 +48,8 @@ define i32 @call1(i8* %obj) {<br>
 ; constant propagation.<br>
<br>
 ; CHECK: define i1 @call2<br>
-define i1 @call2(i8* %obj) {<br>
+define i1 @call2(i8* %obj) #0 {<br>
+  ; BRANCH-FUNNEL: [[VT1:%.*]] = bitcast {{.*}} to i8*<br>
  Â %vtableptr = bitcast i8* %obj to [1 x i8*]**<br>
  Â %vtable = load [1 x i8*]*, [1 x i8*]** %vtableptr<br>
  Â %vtablei8 = bitcast [1 x i8*]* %vtable to i8*<br>
@@ -57,9 +62,8 @@ define i1 @call2(i8* %obj) {<br>
 cont:<br>
  Â %fptr_casted = bitcast i8* %fptr to i1 (i8*, i32)*<br>
  Â ; SINGLE-IMPL: call i1 bitcast (void ()* @singleimpl2 to i1 (i8*, i32)*)<br>
-  ; UNIFORM-RET-VAL: call i1 %<br>
-  ; UNIQUE-RET-VAL0: call i1 %<br>
-  ; UNIQUE-RET-VAL1: call i1 %<br>
+  ; INDIR: call i1 %<br>
+  ; BRANCH-FUNNEL: call i1 bitcast (void ()* @__typeid_typeid2_8_branch_<wbr>funnel to i1 (i8*, i8*, i32)*)(i8* nest [[VT1]], i8* %obj, i32 undef)<br>
  Â %result = call i1 %fptr_casted(i8* %obj, i32 undef)<br>
  Â ret i1 %result<br>
<br>
@@ -69,7 +73,7 @@ trap:<br>
 }<br>
<br>
 ; CHECK: define i1 @call3<br>
-define i1 @call3(i8* %obj) {<br>
+define i1 @call3(i8* %obj) #0 {<br>
  Â %vtableptr = bitcast i8* %obj to [1 x i8*]**<br>
  Â %vtable = load [1 x i8*]*, [1 x i8*]** %vtableptr<br>
  Â %vtablei8 = bitcast [1 x i8*]* %vtable to i8*<br>
@@ -91,6 +95,8 @@ cont:<br>
  Â ; VCP-ARM: [[AND2:%.*]] = and i8 [[LOAD2]], -128<br>
  Â ; VCP: [[ICMP2:%.*]] = icmp ne i8 [[AND2]], 0<br>
  Â ; VCP: ret i1 [[ICMP2]]<br>
+  ; BRANCH-FUNNEL-NOVCP: [[VT2:%.*]] = bitcast {{.*}} to i8*<br>
+  ; BRANCH-FUNNEL-NOVCP: call i1 bitcast (void ()* @__typeid_typeid2_8_branch_<wbr>funnel to i1 (i8*, i8*, i32)*)(i8* nest [[VT2]], i8* %obj, i32 3)<br>
  Â ret i1 %result<br>
<br>
 trap:<br>
@@ -111,3 +117,5 @@ declare void @llvm.assume(i1)<br>
 declare void @llvm.trap()<br>
 declare {i8*, i1} @llvm.type.checked.load(i8*, i32, metadata)<br>
 declare i1 @llvm.type.test(i8*, metadata)<br>
+<br>
+attributes #0 = { "target-features"="+retpoline" }<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
______________________________<wbr>_________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/llvm-commits</a><br>
</div></div></blockquote></div><br></div></div>