[clang] [llvm] [not for merge][RFC] Key Instructions front end demo (PR #130943)
Orlando Cazalet-Hyams via cfe-commits
cfe-commits at lists.llvm.org
Wed Mar 12 04:04:40 PDT 2025
https://github.com/OCHyams created https://github.com/llvm/llvm-project/pull/130943
This draft pull request demonstrates our proposed approach to annotating instructions with Key Instructions metadata. It's not fully complete but works as a proof of concept and I welcome and encourage feedback on the approach, as I'm not particularly familiar with Clang.
The Key Instructions project is introduced here https://discourse.llvm.org/t/rfc-improving-is-stmt-placement-for-better-interactive-debugging/82668, which includes a "quick summary" section at the top which adds context for this PR.
I'll walk through the changes first, followed by some questions at the end.
Note: this PR doesn't include the LLVM parts, so you won't be able to try it out locally.
## Overview
We'd like Clang to annotate instructions with additional metadata (bundled inside DILocations) so that LLVM can make better `is_stmt` placement decisions. Specifically we'll add two new fields to DILocations: An "atom group" number for instructions that perform key functionality ("interesting" behaviour from a debugger-user perspective, see _Key Instructions: Solving the Code Location Problem for Optimized Code (C. Tice, . S. L. Graham, 2000)_); and an "atom rank" number which indicates precedence within a set of instructions that have the same atom group number.
In the project prototype we interpreted IR in a pre-optimisation pass to decide the metadata placement. Having Clang perform the annotation comes with several benefits: it allows the front end to be opinionated about stepping locations, it's more performant for the front end to do it, and it hopefully improves the chance that it can be adopted into other front ends in which the prototype approach isn't approprite.
First, here's an example to highlight the new metadata we'll be producing:
`$ cat -n test.cpp`
```
1. void f(int a) {
2. int x[256] = {a};
3. }
```
`clang++ test.cpp -gmlt -emit-llvm -S`
```
define hidden void @_Z1fi(i32 noundef %a) #0 !dbg !11 {
entry:
%a.addr = alloca i32, align 4
%x = alloca [256 x i32], align 16
store i32 %a, ptr %a.addr, align 4
call void @llvm.memset.p0.i64(ptr align 16 %x, i8 0, i64 1024, i1 false), !dbg !14 ; atom 1, rank 1
%arrayinit.begin = getelementptr inbounds [256 x i32], ptr %x, i64 0, i64 0, !dbg !15
%0 = load i32, ptr %a.addr, align 4, !dbg !16 ; atom 1, rank 2
store i32 %0, ptr %arrayinit.begin, align 4, !dbg !17 ; atom 1, rank 1
ret void, !dbg !18 ; atom 2, rank 1
}
...
!14 = !DILocation(line: 2, column: 7, scope: !11, atomGroup: 1, atomRank: 1)
!15 = !DILocation(line: 2, column: 16, scope: !11)
!16 = !DILocation(line: 2, column: 17, scope: !11, atomGroup: 1, atomRank: 2)
!17 = !DILocation(line: 2, column: 16, scope: !11, atomGroup: 1, atomRank: 1)
!18 = !DILocation(line: 3, column: 1, scope: !11, atomGroup: 2, atomRank: 1)
```
Atom 1 represents the aggregate initialization of `x`. The store and memset have rank 1. That gives them precedence the load of `a` with rank 2, which is essentially a "backup instruction" for the purposes of assigning `is_stmt`. If all rank 1 instructions are optimized away, we'll use that instead. Atom 2 represents the implicit return (given the source location of the closing brace).
<details>
<summary>
DWARF emission info for additional context (feel free to ignore).
</summary>
Not shown in this patch - During dwarf emission, for each atom group, the last instruction in a block that shares the lowest rank will be candidates for `is_stmt`. The wording is tricky, but essentially if any rank 1 instructions exist, higher rank (lower precedence) instructions won't be candidates for `is_stmt` even if they're in different blocks, but all the final rank 1 instructions in the group in different blocks will be.
As an optimisation for convinience (mostly to minimise unecessary difference when the feature is enabled, but also to improve variable availability in some cases), we apply a heuristc that "floats" the `is_stmt` up to the first instruction in a contiguous block of instructions with the same line number.
In the example above the `store` in atom 1 is the candidate for `is_stmt` (the `memset` comes before the `store` and the `load` is lower precedence as well as before the `store`). Because of the heuristic described above, the `is_stmt` flag floats up to the memset, as all the preceeding instructions have the same line number.
</details>
## Implementation
We need to annotate assignments, conditional branches, some unconditional branches, and calls as these are instructions implementing "key functionality". The GEP used to compute the offset at which to store a value for an assignment is not interesting, but the store itself is. We also want to annotate "backup instructions". E.g., the instruction computing the value being stored, or the `cmp` used for a conditional branch.
`ApplyAtomGroup` works similarly to and alongside `ApplyDebugLocation` as an RAII wrapper around a "current atom group" number during CodeGen. The current atom number is applied to certain instructions as they're emitted (stores, branches, etc) using `addInstToCurrentSourceAtom`.
Here's how it looks with the aggregate initialisation example from the overview section:
<details>
<summary>
`EmitAutoVarInit` creates a new atom group with `ApplyAtomGroup`.
</summary>
```
> clang::CodeGen::ApplyAtomGroup::ApplyAtomGroup(clang::CodeGen::CodeGenFunction & CGF) Line 184
clang::CodeGen::CodeGenFunction::EmitAutoVarInit(const clang::CodeGen::CodeGenFunction::AutoVarEmission & emission) Line 1958 ++
clang::CodeGen::CodeGenFunction::EmitAutoVarDecl(const clang::VarDecl & D) Line 1358
clang::CodeGen::CodeGenFunction::EmitVarDecl(const clang::VarDecl & D) Line 219
...
```
</details>
<details>
<summary>
`CheckAggExprForMemSetUse` calls `addInstToCurrentSourceAtom` to add the memset to the current atom group.
</summary>
```
> clang::CodeGen::CodeGenFunction::addInstToCurrentSourceAtom(llvm::Instruction * KeyInstruction, llvm::Value * Backup, unsigned char KeyInstRank) Line 2551
CheckAggExprForMemSetUse(clang::CodeGen::AggValueSlot & Slot, const clang::Expr * E, clang::CodeGen::CodeGenFunction & CGF) Line 2026
clang::CodeGen::CodeGenFunction::EmitAggExpr(const clang::Expr * E, clang::CodeGen::AggValueSlot Slot) Line 2043
clang::CodeGen::CodeGenFunction::EmitExprAsInit(const clang::Expr * init, const clang::ValueDecl * D, clang::CodeGen::LValue lvalue, bool capturedByInit) Line 2104
clang::CodeGen::CodeGenFunction::EmitAutoVarInit(const clang::CodeGen::CodeGenFunction::AutoVarEmission & emission) Line 2047
clang::CodeGen::CodeGenFunction::EmitAutoVarDecl(const clang::VarDecl & D) Line 1358
clang::CodeGen::CodeGenFunction::EmitVarDecl(const clang::VarDecl & D) Line 219
...
```
</details>
<details>
<summary>
`EmitStoreOfScalar` does the same for the store (note - this is the same atom group number as the memset).
</summary>
```
> clang::CodeGen::CodeGenFunction::addInstToCurrentSourceAtom(llvm::Instruction * KeyInstruction, llvm::Value * Backup, unsigned char KeyInstRank) Line 2551
clang::CodeGen::CodeGenFunction::EmitStoreOfScalar(llvm::Value * Value, clang::CodeGen::Address Addr, bool Volatile, clang::QualType Ty, clang::CodeGen::LValueBaseInfo BaseInfo, clang::CodeGen::TBAAAccessInfo TBAAInfo, bool isInit, bool isNontemporal) Line 2133
clang::CodeGen::CodeGenFunction::EmitStoreOfScalar(llvm::Value * value, clang::CodeGen::LValue lvalue, bool isInit) Line 2152
clang::CodeGen::CodeGenFunction::EmitStoreThroughLValue(clang::CodeGen::RValue Src, clang::CodeGen::LValue Dst, bool isInit) Line 2466
clang::CodeGen::CodeGenFunction::EmitScalarInit(const clang::Expr * init, const clang::ValueDecl * D, clang::CodeGen::LValue lvalue, bool capturedByInit) Line 803
`anonymous namespace'::AggExprEmitter::EmitInitializationToLValue(clang::Expr * E, clang::CodeGen::LValue LV) Line 1592
`anonymous namespace'::AggExprEmitter::EmitArrayInit(clang::CodeGen::Address DestPtr, llvm::ArrayType * AType, clang::QualType rrayQTy, clang::Expr * ExprToVisit, llvm::ArrayRef<clang::Expr *> Args, clang::Expr * ArrayFiller) Line 614
`anonymous namespace'::AggExprEmitter::VisitCXXParenListOrInitListExpr(clang::Expr * ExprToVisit, llvm::ArrayRef<clang::Expr *> nitExprs, clang::FieldDecl * InitializedFieldInUnion, clang::Expr * ArrayFiller) Line 1672
`anonymous namespace'::AggExprEmitter::VisitInitListExpr(clang::InitListExpr * E) Line 1641
clang::StmtVisitorBase<std::add_pointer,`anonymous namespace'::AggExprEmitter,void>::Visit(clang::Stmt * S) Line 352
`anonymous namespace'::AggExprEmitter::Visit(clang::Expr * E) Line 114
clang::CodeGen::CodeGenFunction::EmitAggExpr(const clang::Expr * E, clang::CodeGen::AggValueSlot Slot) Line 2045
clang::CodeGen::CodeGenFunction::EmitExprAsInit(const clang::Expr * init, const clang::ValueDecl * D, clang::CodeGen::LValue value, bool capturedByInit) Line 2104
clang::CodeGen::CodeGenFunction::EmitAutoVarInit(const clang::CodeGen::CodeGenFunction::AutoVarEmission & emission) Line 2047 C+
clang::CodeGen::CodeGenFunction::EmitAutoVarDecl(const clang::VarDecl & D) Line 1358
clang::CodeGen::CodeGenFunction::EmitVarDecl(const clang::VarDecl & D) Line 219
...
```
</details>
`addInstToCurrentSourceAtom` annotates the instruction that produces the stored value too, with a higher rank than the store (lower precedence).
## Rough edges and questions
* The single-block return statement handling (see changes in `EmitReturnBlock` in clang/lib/CodeGen/CGStmt.cpp and `addRetToOverrideOrNewSourceAtom` usage in `EmitFunctionEpilog`) doesn't fit the RAII model (I've not found any other cases yet though).
* There's some inefficiency: DILocations are attached to instructions then immediately replaced with a new version with Key Instructions metadata.
* It doesn't fail gracefully: if we accidentally fail to annotate instructions that should be key, those instructions won't be candidate for `is_stmt` and therefore will likely be stepped-over while debugging.
* Is there a way to implement this that errs on over-application of annotations?
* Perhaps every statement (in `EmitStmt`) could be assumed to be in an atom group by default, with particular statements opting out, rather than interesting ones opting in?
* I think we could add all stores (and memsets etc) to a "new" atom group by default, unless there's an "active" RAII group already or they've opted-out. That would offer a graceful fallback for stores. It doesn't help so much with control flow (as I think only some branches are interesting, compared to most stores being interesting).
Finally, does this direction look reasonable overall?
>From fbe21a1d8d8f3b60eb920af832718c0874f41e11 Mon Sep 17 00:00:00 2001
From: Orlando Cazalet-Hyams <orlando.hyams at sony.com>
Date: Thu, 6 Mar 2025 11:33:32 +0000
Subject: [PATCH] Key Instructions front end demo
note to self: up to `ea4fabe48df0`.
---
clang/lib/CodeGen/CGBuiltin.cpp | 49 ++++--
clang/lib/CodeGen/CGCall.cpp | 16 +-
clang/lib/CodeGen/CGClass.cpp | 1 +
clang/lib/CodeGen/CGCleanup.cpp | 3 +
clang/lib/CodeGen/CGDebugInfo.cpp | 149 +++++++++++++++++-
clang/lib/CodeGen/CGDebugInfo.h | 51 ++++++
clang/lib/CodeGen/CGDecl.cpp | 53 ++++---
clang/lib/CodeGen/CGDeclCXX.cpp | 4 +-
clang/lib/CodeGen/CGExpr.cpp | 26 ++-
clang/lib/CodeGen/CGExprAgg.cpp | 9 +-
clang/lib/CodeGen/CGExprComplex.cpp | 12 +-
clang/lib/CodeGen/CGExprScalar.cpp | 4 +
clang/lib/CodeGen/CGStmt.cpp | 66 ++++++--
clang/lib/CodeGen/CodeGenFunction.cpp | 47 +++++-
clang/lib/CodeGen/CodeGenFunction.h | 18 ++-
clang/lib/CodeGen/ItaniumCXXABI.cpp | 1 +
clang/test/KeyInstructions/agg-cpy.cpp | 13 ++
.../agg-init-bzero-plus-stores.cpp | 25 +++
clang/test/KeyInstructions/agg-init.cpp | 24 +++
clang/test/KeyInstructions/agg-null-init.cpp | 21 +++
.../KeyInstructions/agg-return-va-arg.cpp | 23 +++
clang/test/KeyInstructions/assign.c | 11 ++
clang/test/KeyInstructions/binop.c | 17 ++
clang/test/KeyInstructions/bitfield.cpp | 15 ++
clang/test/KeyInstructions/cast.cpp | 17 ++
clang/test/KeyInstructions/complex.cpp | 28 ++++
.../test/KeyInstructions/compound-assign.cpp | 13 ++
clang/test/KeyInstructions/do.cpp | 20 +++
clang/test/KeyInstructions/double-assign.cpp | 22 +++
clang/test/KeyInstructions/for.cpp | 34 ++++
clang/test/KeyInstructions/if.cpp | 42 +++++
clang/test/KeyInstructions/inc.cpp | 13 ++
clang/test/KeyInstructions/member-init.cpp | 13 ++
clang/test/KeyInstructions/memset.c | 10 ++
clang/test/KeyInstructions/multi-func.cpp | 28 ++++
clang/test/KeyInstructions/new.cpp | 36 +++++
clang/test/KeyInstructions/ret-agg.cpp | 23 +++
clang/test/KeyInstructions/return-no-loc.c | 26 +++
clang/test/KeyInstructions/return-ref.cpp | 19 +++
clang/test/KeyInstructions/return.c | 36 +++++
clang/test/KeyInstructions/return.cpp | 26 +++
clang/test/KeyInstructions/scalar-init.cpp | 17 ++
clang/test/KeyInstructions/static-init.cpp | 13 ++
clang/test/KeyInstructions/switch.cpp | 48 ++++++
clang/test/KeyInstructions/try-catch.cpp | 20 +++
clang/test/KeyInstructions/while.cpp | 20 +++
llvm/include/llvm/IR/LLVMContext.h | 4 +
llvm/lib/IR/LLVMContext.cpp | 4 +
48 files changed, 1129 insertions(+), 61 deletions(-)
create mode 100644 clang/test/KeyInstructions/agg-cpy.cpp
create mode 100644 clang/test/KeyInstructions/agg-init-bzero-plus-stores.cpp
create mode 100644 clang/test/KeyInstructions/agg-init.cpp
create mode 100644 clang/test/KeyInstructions/agg-null-init.cpp
create mode 100644 clang/test/KeyInstructions/agg-return-va-arg.cpp
create mode 100644 clang/test/KeyInstructions/assign.c
create mode 100644 clang/test/KeyInstructions/binop.c
create mode 100644 clang/test/KeyInstructions/bitfield.cpp
create mode 100644 clang/test/KeyInstructions/cast.cpp
create mode 100644 clang/test/KeyInstructions/complex.cpp
create mode 100644 clang/test/KeyInstructions/compound-assign.cpp
create mode 100644 clang/test/KeyInstructions/do.cpp
create mode 100644 clang/test/KeyInstructions/double-assign.cpp
create mode 100644 clang/test/KeyInstructions/for.cpp
create mode 100644 clang/test/KeyInstructions/if.cpp
create mode 100644 clang/test/KeyInstructions/inc.cpp
create mode 100644 clang/test/KeyInstructions/member-init.cpp
create mode 100644 clang/test/KeyInstructions/memset.c
create mode 100644 clang/test/KeyInstructions/multi-func.cpp
create mode 100644 clang/test/KeyInstructions/new.cpp
create mode 100644 clang/test/KeyInstructions/ret-agg.cpp
create mode 100644 clang/test/KeyInstructions/return-no-loc.c
create mode 100644 clang/test/KeyInstructions/return-ref.cpp
create mode 100644 clang/test/KeyInstructions/return.c
create mode 100644 clang/test/KeyInstructions/return.cpp
create mode 100644 clang/test/KeyInstructions/scalar-init.cpp
create mode 100644 clang/test/KeyInstructions/static-init.cpp
create mode 100644 clang/test/KeyInstructions/switch.cpp
create mode 100644 clang/test/KeyInstructions/try-catch.cpp
create mode 100644 clang/test/KeyInstructions/while.cpp
diff --git a/clang/lib/CodeGen/CGBuiltin.cpp b/clang/lib/CodeGen/CGBuiltin.cpp
index afd9798cd639b..8add3d7296c2e 100644
--- a/clang/lib/CodeGen/CGBuiltin.cpp
+++ b/clang/lib/CodeGen/CGBuiltin.cpp
@@ -13,6 +13,7 @@
#include "ABIInfo.h"
#include "CGCUDARuntime.h"
#include "CGCXXABI.h"
+#include "CGDebugInfo.h"
#include "CGHLSLRuntime.h"
#include "CGObjCRuntime.h"
#include "CGOpenCLRuntime.h"
@@ -39,11 +40,13 @@
#include "llvm/ADT/APFloat.h"
#include "llvm/ADT/APInt.h"
#include "llvm/ADT/FloatingPointMode.h"
+#include "llvm/ADT/ScopeExit.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/StringExtras.h"
#include "llvm/Analysis/ValueTracking.h"
#include "llvm/IR/DataLayout.h"
#include "llvm/IR/InlineAsm.h"
+#include "llvm/IR/Instruction.h"
#include "llvm/IR/Intrinsics.h"
#include "llvm/IR/IntrinsicsAArch64.h"
#include "llvm/IR/IntrinsicsAMDGPU.h"
@@ -97,6 +100,7 @@ static void initializeAlloca(CodeGenFunction &CGF, AllocaInst *AI, Value *Size,
if (CGF.CGM.stopAutoInit())
return;
auto *I = CGF.Builder.CreateMemSet(AI, Byte, Size, AlignmentInBytes);
+ CGF.addInstToCurrentSourceAtom(I, nullptr);
I->addAnnotationMetadata("auto-init");
}
@@ -3543,6 +3547,14 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
}
};
+ // FIXME(OCH): Should any of the maths builtins be key instructions?
+ auto Grp = ApplyAtomGroup(*this);
+ llvm::Instruction *InstForAtomGrp = nullptr;
+ auto Cleanup = llvm::make_scope_exit([&]() {
+ if (InstForAtomGrp)
+ addInstToCurrentSourceAtom(InstForAtomGrp, nullptr);
+ });
+
switch (BuiltinIDIfNoAsmLabel) {
default: break;
case Builtin::BI__builtin___CFStringMakeConstantString:
@@ -3948,6 +3960,8 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
case Builtin::BI_byteswap_ushort:
case Builtin::BI_byteswap_ulong:
case Builtin::BI_byteswap_uint64: {
+ // FIXME(OCH): Should bswap and similar intrinsics be key instructions?
+ // If the result is stored then that will be key - is that enough?
return RValue::get(
emitBuiltinWithOneOverloadedType<1>(*this, E, Intrinsic::bswap));
}
@@ -4080,7 +4094,7 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
return RValue::get(Builder.CreateCall(F, {Begin, End}));
}
case Builtin::BI__builtin_trap:
- EmitTrapCall(Intrinsic::trap);
+ InstForAtomGrp = EmitTrapCall(Intrinsic::trap);
return RValue::get(nullptr);
case Builtin::BI__builtin_verbose_trap: {
llvm::DILocation *TrapLocation = Builder.getCurrentDebugLocation();
@@ -4095,10 +4109,10 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
return RValue::get(nullptr);
}
case Builtin::BI__debugbreak:
- EmitTrapCall(Intrinsic::debugtrap);
+ InstForAtomGrp = EmitTrapCall(Intrinsic::debugtrap);
return RValue::get(nullptr);
case Builtin::BI__builtin_unreachable: {
- EmitUnreachable(E->getExprLoc());
+ InstForAtomGrp = EmitUnreachable(E->getExprLoc());
// We do need to preserve an insertion point.
EmitBlock(createBasicBlock("unreachable.cont"));
@@ -4547,6 +4561,7 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
Matrix, Dst.emitRawPointer(*this),
Align(Dst.getAlignment().getQuantity()), Stride, IsVolatile,
MatrixTy->getNumRows(), MatrixTy->getNumColumns());
+ InstForAtomGrp = cast<llvm::Instruction>(Result);
return RValue::get(Result);
}
@@ -4667,6 +4682,7 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
.getAsAlign();
AllocaInst *AI = Builder.CreateAlloca(Builder.getInt8Ty(), Size);
AI->setAlignment(SuitableAlignmentInBytes);
+ // NOTE(OCH): `initializeAlloca` adds Key Instruction metadata.
if (BuiltinID != Builtin::BI__builtin_alloca_uninitialized)
initializeAlloca(*this, AI, Size, SuitableAlignmentInBytes);
LangAS AAS = getASTAllocaAddressSpace();
@@ -4689,6 +4705,7 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
CGM.getContext().toCharUnitsFromBits(AlignmentInBits).getAsAlign();
AllocaInst *AI = Builder.CreateAlloca(Builder.getInt8Ty(), Size);
AI->setAlignment(AlignmentInBytes);
+ // NOTE(OCH): `initializeAlloca` adds Key Instruction metadata.
if (BuiltinID != Builtin::BI__builtin_alloca_with_align_uninitialized)
initializeAlloca(*this, AI, Size, AlignmentInBytes);
LangAS AAS = getASTAllocaAddressSpace();
@@ -4707,7 +4724,8 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
Value *SizeVal = EmitScalarExpr(E->getArg(1));
EmitNonNullArgCheck(Dest, E->getArg(0)->getType(),
E->getArg(0)->getExprLoc(), FD, 0);
- Builder.CreateMemSet(Dest, Builder.getInt8(0), SizeVal, false);
+ InstForAtomGrp =
+ Builder.CreateMemSet(Dest, Builder.getInt8(0), SizeVal, false);
return RValue::get(nullptr);
}
@@ -4722,7 +4740,7 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
EmitNonNullArgCheck(RValue::get(Dest.emitRawPointer(*this)),
E->getArg(1)->getType(), E->getArg(1)->getExprLoc(), FD,
0);
- Builder.CreateMemMove(Dest, Src, SizeVal, false);
+ InstForAtomGrp = Builder.CreateMemMove(Dest, Src, SizeVal, false);
return RValue::get(nullptr);
}
@@ -4735,7 +4753,7 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
Value *SizeVal = EmitScalarExpr(E->getArg(2));
EmitArgCheck(TCK_Store, Dest, E->getArg(0), 0);
EmitArgCheck(TCK_Load, Src, E->getArg(1), 1);
- Builder.CreateMemCpy(Dest, Src, SizeVal, false);
+ InstForAtomGrp = Builder.CreateMemCpy(Dest, Src, SizeVal, false);
if (BuiltinID == Builtin::BImempcpy ||
BuiltinID == Builtin::BI__builtin_mempcpy)
return RValue::get(Builder.CreateInBoundsGEP(
@@ -4751,7 +4769,7 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
E->getArg(2)->EvaluateKnownConstInt(getContext()).getZExtValue();
EmitArgCheck(TCK_Store, Dest, E->getArg(0), 0);
EmitArgCheck(TCK_Load, Src, E->getArg(1), 1);
- Builder.CreateMemCpyInline(Dest, Src, Size);
+ InstForAtomGrp = Builder.CreateMemCpyInline(Dest, Src, Size);
return RValue::get(nullptr);
}
@@ -4772,7 +4790,7 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
Address Dest = EmitPointerWithAlignment(E->getArg(0));
Address Src = EmitPointerWithAlignment(E->getArg(1));
Value *SizeVal = llvm::ConstantInt::get(Builder.getContext(), Size);
- Builder.CreateMemCpy(Dest, Src, SizeVal, false);
+ InstForAtomGrp = Builder.CreateMemCpy(Dest, Src, SizeVal, false);
return RValue::get(Dest, *this);
}
@@ -4798,7 +4816,7 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
Address Dest = EmitPointerWithAlignment(E->getArg(0));
Address Src = EmitPointerWithAlignment(E->getArg(1));
Value *SizeVal = llvm::ConstantInt::get(Builder.getContext(), Size);
- Builder.CreateMemMove(Dest, Src, SizeVal, false);
+ InstForAtomGrp = Builder.CreateMemMove(Dest, Src, SizeVal, false);
return RValue::get(Dest, *this);
}
@@ -4809,7 +4827,7 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
Value *SizeVal = EmitScalarExpr(E->getArg(2));
EmitArgCheck(TCK_Store, Dest, E->getArg(0), 0);
EmitArgCheck(TCK_Load, Src, E->getArg(1), 1);
- Builder.CreateMemMove(Dest, Src, SizeVal, false);
+ InstForAtomGrp = Builder.CreateMemMove(Dest, Src, SizeVal, false);
return RValue::get(Dest, *this);
}
case Builtin::BImemset:
@@ -4820,7 +4838,7 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
Value *SizeVal = EmitScalarExpr(E->getArg(2));
EmitNonNullArgCheck(Dest, E->getArg(0)->getType(),
E->getArg(0)->getExprLoc(), FD, 0);
- Builder.CreateMemSet(Dest, ByteVal, SizeVal, false);
+ InstForAtomGrp = Builder.CreateMemSet(Dest, ByteVal, SizeVal, false);
return RValue::get(Dest, *this);
}
case Builtin::BI__builtin_memset_inline: {
@@ -4832,7 +4850,7 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
EmitNonNullArgCheck(RValue::get(Dest.emitRawPointer(*this)),
E->getArg(0)->getType(), E->getArg(0)->getExprLoc(), FD,
0);
- Builder.CreateMemSetInline(Dest, ByteVal, Size);
+ InstForAtomGrp = Builder.CreateMemSetInline(Dest, ByteVal, Size);
return RValue::get(nullptr);
}
case Builtin::BI__builtin___memset_chk: {
@@ -4849,10 +4867,13 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
Value *ByteVal = Builder.CreateTrunc(EmitScalarExpr(E->getArg(1)),
Builder.getInt8Ty());
Value *SizeVal = llvm::ConstantInt::get(Builder.getContext(), Size);
- Builder.CreateMemSet(Dest, ByteVal, SizeVal, false);
+ InstForAtomGrp = Builder.CreateMemSet(Dest, ByteVal, SizeVal, false);
return RValue::get(Dest, *this);
}
case Builtin::BI__builtin_wmemchr: {
+ // FIXME(OCH): Probably ok for none of the inline implementation to be key.
+ // If the result is stored, that store should be a stepping location.
+
// The MSVC runtime library does not provide a definition of wmemchr, so we
// need an inline implementation.
if (!getTarget().getTriple().isOSMSVCRT())
@@ -6462,7 +6483,7 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
Value *Val = EmitScalarExpr(E->getArg(0));
Address Address = EmitPointerWithAlignment(E->getArg(1));
Value *HalfVal = Builder.CreateFPTrunc(Val, Builder.getHalfTy());
- Builder.CreateStore(HalfVal, Address);
+ InstForAtomGrp = Builder.CreateStore(HalfVal, Address);
return RValue::get(nullptr);
}
case Builtin::BI__builtin_load_half: {
diff --git a/clang/lib/CodeGen/CGCall.cpp b/clang/lib/CodeGen/CGCall.cpp
index bfcbc273dbda7..9c7286147d564 100644
--- a/clang/lib/CodeGen/CGCall.cpp
+++ b/clang/lib/CodeGen/CGCall.cpp
@@ -17,6 +17,7 @@
#include "CGBlocks.h"
#include "CGCXXABI.h"
#include "CGCleanup.h"
+#include "CGDebugInfo.h"
#include "CGRecordLayout.h"
#include "CodeGenFunction.h"
#include "CodeGenModule.h"
@@ -37,6 +38,7 @@
#include "llvm/IR/CallingConv.h"
#include "llvm/IR/DataLayout.h"
#include "llvm/IR/InlineAsm.h"
+#include "llvm/IR/Instructions.h"
#include "llvm/IR/IntrinsicInst.h"
#include "llvm/IR/Intrinsics.h"
#include "llvm/IR/Type.h"
@@ -3883,7 +3885,8 @@ void CodeGenFunction::EmitFunctionEpilog(const CGFunctionInfo &FI,
// Functions with no result always return void.
if (!ReturnValue.isValid()) {
- Builder.CreateRetVoid();
+ auto *I = Builder.CreateRetVoid();
+ addRetToOverrideOrNewSourceAtom(I, nullptr);
return;
}
@@ -4065,6 +4068,9 @@ void CodeGenFunction::EmitFunctionEpilog(const CGFunctionInfo &FI,
if (RetDbgLoc)
Ret->setDebugLoc(std::move(RetDbgLoc));
+
+ llvm::Value *Backup = RV ? Ret->getOperand(0) : nullptr;
+ addRetToOverrideOrNewSourceAtom(cast<llvm::ReturnInst>(Ret), Backup);
}
void CodeGenFunction::EmitReturnValueCheck(llvm::Value *RV) {
@@ -5829,6 +5835,14 @@ RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
BundleList);
EmitBlock(Cont);
}
+
+ // NOTE(OCH) We only want the group to apply to the call instuction
+ // specifically. N.B. we currently apply is_stmt to all calls at DWARF
+ // emission time. That makes it easy to avoid "over propagating" is_stmt when
+ // calls are lowered. That's easiest, so we continue to do that for now.
+ // FIXME(OCH): Reinstate this once that is no longer the case.
+ // addInstToNewSourceAtom(CI, nullptr);
+
if (CI->getCalledFunction() && CI->getCalledFunction()->hasName() &&
CI->getCalledFunction()->getName().starts_with("_Z4sqrt")) {
SetSqrtFPAccuracy(CI);
diff --git a/clang/lib/CodeGen/CGClass.cpp b/clang/lib/CodeGen/CGClass.cpp
index e54fd543f217b..d69fe1e509ffe 100644
--- a/clang/lib/CodeGen/CGClass.cpp
+++ b/clang/lib/CodeGen/CGClass.cpp
@@ -1339,6 +1339,7 @@ void CodeGenFunction::EmitCtorPrologue(const CXXConstructorDecl *CD,
assert(!Member->isBaseInitializer());
assert(Member->isAnyMemberInitializer() &&
"Delegating initializer on non-delegating constructor");
+ auto Grp = ApplyAtomGroup(*this);
CM.addMemberInitializer(Member);
}
CM.finish();
diff --git a/clang/lib/CodeGen/CGCleanup.cpp b/clang/lib/CodeGen/CGCleanup.cpp
index 7e1c5b7da9552..8462e4a1f3791 100644
--- a/clang/lib/CodeGen/CGCleanup.cpp
+++ b/clang/lib/CodeGen/CGCleanup.cpp
@@ -17,6 +17,7 @@
//===----------------------------------------------------------------------===//
#include "CGCleanup.h"
+#include "CGDebugInfo.h"
#include "CodeGenFunction.h"
#include "llvm/Support/SaveAndRestore.h"
@@ -1118,6 +1119,8 @@ void CodeGenFunction::EmitBranchThroughCleanup(JumpDest Dest) {
// Create the branch.
llvm::BranchInst *BI = Builder.CreateBr(Dest.getBlock());
+ // This is the primary instruction for this atom, acting as a ret.
+ addInstToCurrentSourceAtom(BI, nullptr);
// Calculate the innermost active normal cleanup.
EHScopeStack::stable_iterator
diff --git a/clang/lib/CodeGen/CGDebugInfo.cpp b/clang/lib/CodeGen/CGDebugInfo.cpp
index 0e6daa42ee7bf..8bf5d1418f431 100644
--- a/clang/lib/CodeGen/CGDebugInfo.cpp
+++ b/clang/lib/CodeGen/CGDebugInfo.cpp
@@ -43,6 +43,7 @@
#include "llvm/IR/Constants.h"
#include "llvm/IR/DataLayout.h"
#include "llvm/IR/DerivedTypes.h"
+#include "llvm/IR/Instruction.h"
#include "llvm/IR/Instructions.h"
#include "llvm/IR/Intrinsics.h"
#include "llvm/IR/Metadata.h"
@@ -52,6 +53,7 @@
#include "llvm/Support/SHA1.h"
#include "llvm/Support/SHA256.h"
#include "llvm/Support/TimeProfiler.h"
+#include <cstdint>
#include <optional>
using namespace clang;
using namespace clang::CodeGen;
@@ -119,6 +121,144 @@ CGDebugInfo::~CGDebugInfo() {
"Region stack mismatch, stack not empty!");
}
+void CGDebugInfo::addInstSourceAtomMetadata(llvm::Instruction *I,
+ uint64_t Group, uint8_t Rank) {
+ if (!I->getDebugLoc() || Group == 0 || !I->getDebugLoc()->getLine())
+ return;
+
+ // Saturate the 3-bit rank.
+ Rank = std::min<uint8_t>(Rank, 7);
+
+ const llvm::DebugLoc &DL = I->getDebugLoc();
+
+ // Each instruction can only be attributed to one source atom (as an
+ // limitation of the implementation). If this instruction is already
+ // part of a source atom, pick the group in which it has highest
+ // precedence (lowest rank).
+ // TODO(OCH): Is there a better way to handle merging? Ideally we'd like
+ // to be able to have it attributed to both atoms.
+ if (DL.get()->getAtomGroup() && DL.get()->getAtomRank() &&
+ DL.get()->getAtomRank() < Rank) {
+ Group = DL.get()->getAtomGroup();
+ Rank = DL.get()->getAtomRank();
+ }
+
+ // Update the watermark to indicate this Group ID has been used
+ // in this function.
+ HighestEmittedAtomGroup = std::max(Group, HighestEmittedAtomGroup);
+
+ llvm::DILocation *NewDL = llvm::DILocation::get(
+ I->getContext(), DL.getLine(), DL.getCol(), DL.getScope(),
+ DL.getInlinedAt(), DL.isImplicitCode(), Group, Rank);
+ I->setDebugLoc(NewDL);
+};
+
+void CGDebugInfo::addInstToCurrentSourceAtom(llvm::Instruction *KeyInstruction,
+ llvm::Value *Backup,
+ uint8_t KeyInstRank) {
+ /* TODO(OCH):
+ if (key-instructions-is-not-enabled)
+ return;
+ */
+ uint64_t Group = CurrentAtomGroup;
+ if (!Group)
+ return;
+
+ KeyInstRank += CurrentAtomRankBase;
+ addInstSourceAtomMetadata(KeyInstruction, Group, KeyInstRank);
+
+ llvm::Instruction *BackupI =
+ llvm::dyn_cast_or_null<llvm::Instruction>(Backup);
+ if (!BackupI)
+ return;
+
+ // Add the backup instruction to the group.
+ addInstSourceAtomMetadata(BackupI, Group, /*Rank*/ ++KeyInstRank);
+
+ // Look through chains of casts too, as they're probably going to evaporate.
+ // FIXME(OCH): And other nops like zero length geps?
+ // FIXME(OCH): Should use Cast->isNoopCast()?
+ while (auto *Cast = dyn_cast<llvm::CastInst>(BackupI)) {
+ BackupI = dyn_cast<llvm::Instruction>(Cast->getOperand(0));
+ if (!BackupI)
+ break;
+ addInstSourceAtomMetadata(BackupI, Group, ++KeyInstRank);
+ }
+}
+
+void CGDebugInfo::addRetToOverrideOrNewSourceAtom(llvm::ReturnInst *Ret,
+ llvm::Value *Backup,
+ uint8_t KeyInstRank) {
+ if (RetAtomGroupOverride) {
+ uint64_t CurGrp = CurrentAtomGroup;
+ CurrentAtomGroup = RetAtomGroupOverride;
+ addInstToCurrentSourceAtom(Ret, Backup, KeyInstRank);
+ CurrentAtomGroup = CurGrp;
+ RetAtomGroupOverride = 0;
+ } else {
+ auto Grp = ApplyAtomGroup(this);
+ addInstToCurrentSourceAtom(Ret, Backup, KeyInstRank);
+ }
+}
+
+void CGDebugInfo::setRetInstSourceAtomOverride(uint64_t Group) {
+ assert(RetAtomGroupOverride == 0);
+ RetAtomGroupOverride = Group;
+}
+
+void CGDebugInfo::completeFunction() {
+ // Atoms are identified by a {AtomGroup, InlinedAt} pair, meaning AtomGroup
+ // numbers can be repeated across different functions. LLVM optimisations may
+ // need to assign new AtomGroups. In order to guarentee that those future
+ // transformations keep the numbers within functions unique, we just need to
+ // track the highest number used across all functions.
+ CGM.getLLVMContext().updateAtomGroupWaterline(NextAtomGroup);
+ NextAtomGroup = 1;
+ HighestEmittedAtomGroup = 0;
+ CurrentAtomGroup = 0;
+ RetAtomGroupOverride = 0;
+ CurrentAtomRankBase = 0;
+}
+
+ApplyAtomGroup::ApplyAtomGroup(CodeGenFunction &CGF)
+ : ApplyAtomGroup(CGF.getDebugInfo()) {}
+
+ApplyAtomGroup::ApplyAtomGroup(CGDebugInfo *DI) : DI(DI) {
+ if (!DI)
+ return;
+ OriginalAtomGroup = DI->CurrentAtomGroup;
+ DI->CurrentAtomGroup = DI->NextAtomGroup++;
+ // Reset rank-base as it should only apply to the group it was added to.
+ OriginalRankBase = DI->CurrentAtomRankBase;
+ DI->CurrentAtomRankBase = 0;
+}
+
+ApplyAtomGroup::~ApplyAtomGroup() {
+ if (!DI)
+ return;
+ if (DI->HighestEmittedAtomGroup < DI->NextAtomGroup - 1)
+ DI->NextAtomGroup = DI->HighestEmittedAtomGroup + 1;
+ DI->CurrentAtomGroup = OriginalAtomGroup;
+
+ DI->CurrentAtomRankBase = OriginalRankBase;
+}
+
+IncAtomRank::IncAtomRank(CodeGenFunction &CGF)
+ : IncAtomRank(CGF.getDebugInfo()) {}
+
+IncAtomRank::IncAtomRank(CGDebugInfo *DI) : DI(DI) {
+ if (!DI)
+ return;
+ ++DI->CurrentAtomRankBase;
+}
+
+IncAtomRank::~IncAtomRank() {
+ if (!DI)
+ return;
+ assert(DI->CurrentAtomRankBase);
+ --DI->CurrentAtomRankBase;
+}
+
ApplyDebugLocation::ApplyDebugLocation(CodeGenFunction &CGF,
SourceLocation TemporaryLocation)
: CGF(&CGF) {
@@ -174,8 +314,15 @@ ApplyDebugLocation::ApplyDebugLocation(CodeGenFunction &CGF, llvm::DebugLoc Loc)
return;
}
OriginalLocation = CGF.Builder.getCurrentDebugLocation();
- if (Loc)
+ if (Loc) {
+ // Key Instructions: drop the atom group and rank to avoid accidentally
+ // propagating it around.
+ if (Loc->getAtomGroup())
+ Loc = llvm::DILocation::get(Loc->getContext(), Loc.getLine(),
+ Loc->getColumn(), Loc->getScope(),
+ Loc->getInlinedAt(), Loc.isImplicitCode());
CGF.Builder.SetCurrentDebugLocation(std::move(Loc));
+ }
}
ApplyDebugLocation::~ApplyDebugLocation() {
diff --git a/clang/lib/CodeGen/CGDebugInfo.h b/clang/lib/CodeGen/CGDebugInfo.h
index 38f73eca561b7..997cf0e3db215 100644
--- a/clang/lib/CodeGen/CGDebugInfo.h
+++ b/clang/lib/CodeGen/CGDebugInfo.h
@@ -58,6 +58,9 @@ class CGBlockInfo;
class CGDebugInfo {
friend class ApplyDebugLocation;
friend class SaveAndRestoreLocation;
+ friend class ApplyAtomGroup;
+ friend class IncAtomRank;
+
CodeGenModule &CGM;
const llvm::codegenoptions::DebugInfoKind DebugKind;
bool DebugTypeExtRefs;
@@ -179,6 +182,15 @@ class CGDebugInfo {
/// The key is coroutine real parameters, value is DIVariable in LLVM IR.
Param2DILocTy ParamDbgMappings;
+ /// Source atoms are identified by a {AtomGroup, InlinedAt} pair, meaning
+ /// AtomGroup numbers can be repeated across different functions.
+ uint64_t NextAtomGroup = 1;
+ uint64_t HighestEmittedAtomGroup = 0;
+ uint64_t CurrentAtomGroup = 0;
+ uint64_t RetAtomGroupOverride = 0;
+ uint8_t CurrentAtomRankBase = 0;
+
+private:
/// Helper functions for getOrCreateType.
/// @{
/// Currently the checksum of an interface includes the number of
@@ -636,7 +648,26 @@ class CGDebugInfo {
StringRef Category,
StringRef FailureMsg);
+ // TODO(OCH): comment.
+ void completeFunction();
+
+ // TODO(OCH): Add comment.
+ void addInstToCurrentSourceAtom(llvm::Instruction *KeyInstruction,
+ llvm::Value *Backup, uint8_t KeyInstRank = 1);
+ // TODO(OCH): Add comment.
+ void addRetToOverrideOrNewSourceAtom(llvm::ReturnInst *Ret,
+ llvm::Value *Backup,
+ uint8_t KeyInstRank = 1);
+ void setRetInstSourceAtomOverride(uint64_t Group);
+
private:
+ /// Amend \p I's DebugLoc with \p Group (its source atom group) and \p
+ /// Rank (lower nonzero rank is higher precedence). Does nothing if \p I
+ /// has no DebugLoc, and chooses the atom group in which the instruction
+ /// has the highest precedence if it's already in one.
+ void addInstSourceAtomMetadata(llvm::Instruction *I, uint64_t Group,
+ uint8_t Rank);
+
/// Emit call to llvm.dbg.declare for a variable declaration.
/// Returns a pointer to the DILocalVariable associated with the
/// llvm.dbg.declare, or nullptr otherwise.
@@ -853,6 +884,26 @@ class CGDebugInfo {
}
};
+class ApplyAtomGroup {
+ uint64_t OriginalAtomGroup = 0;
+ uint8_t OriginalRankBase = 0;
+ CGDebugInfo *DI = nullptr;
+
+public:
+ ApplyAtomGroup(CodeGenFunction &CGF);
+ ApplyAtomGroup(CGDebugInfo *DI);
+ ~ApplyAtomGroup();
+};
+
+class IncAtomRank {
+ CGDebugInfo *DI = nullptr;
+
+public:
+ IncAtomRank(CodeGenFunction &CGF);
+ IncAtomRank(CGDebugInfo *DI);
+ ~IncAtomRank();
+};
+
/// A scoped helper to set the current debug location to the specified
/// location or preferred location of the specified Expr.
class ApplyDebugLocation {
diff --git a/clang/lib/CodeGen/CGDecl.cpp b/clang/lib/CodeGen/CGDecl.cpp
index 9cd5885aaae51..6df93822ca21f 100644
--- a/clang/lib/CodeGen/CGDecl.cpp
+++ b/clang/lib/CodeGen/CGDecl.cpp
@@ -428,8 +428,10 @@ void CodeGenFunction::EmitStaticVarDecl(const VarDecl &D,
bool isCudaSharedVar = getLangOpts().CUDA && getLangOpts().CUDAIsDevice &&
D.hasAttr<CUDASharedAttr>();
// If this value has an initializer, emit it.
- if (D.getInit() && !isCudaSharedVar)
+ if (D.getInit() && !isCudaSharedVar) {
+ auto Grp = ApplyAtomGroup(*this);
var = AddInitializerToStaticVarDecl(D, var);
+ }
var->setAlignment(alignment.getAsAlign());
@@ -777,6 +779,7 @@ void CodeGenFunction::EmitScalarInit(const Expr *init, const ValueDecl *D,
Qualifiers::ObjCLifetime lifetime = lvalue.getObjCLifetime();
if (!lifetime) {
llvm::Value *value = EmitScalarExpr(init);
+
if (capturedByInit)
drillIntoBlockVariable(*this, lvalue, cast<VarDecl>(D));
EmitNullabilityCheck(lvalue, value, init->getExprLoc());
@@ -928,7 +931,8 @@ static bool canEmitInitWithFewStoresAfterBZero(llvm::Constant *Init,
/// For inits that canEmitInitWithFewStoresAfterBZero returned true for, emit
/// the scalar stores that would be required.
-static void emitStoresForInitAfterBZero(CodeGenModule &CGM,
+static void emitStoresForInitAfterBZero(CodeGenFunction &CGF,
+ CodeGenModule &CGM,
llvm::Constant *Init, Address Loc,
bool isVolatile, CGBuilderTy &Builder,
bool IsAutoInit) {
@@ -939,6 +943,7 @@ static void emitStoresForInitAfterBZero(CodeGenModule &CGM,
isa<llvm::ConstantVector>(Init) || isa<llvm::BlockAddress>(Init) ||
isa<llvm::ConstantExpr>(Init)) {
auto *I = Builder.CreateStore(Init, Loc, isVolatile);
+ CGF.addInstToCurrentSourceAtom(I, nullptr);
if (IsAutoInit)
I->addAnnotationMetadata("auto-init");
return;
@@ -952,8 +957,8 @@ static void emitStoresForInitAfterBZero(CodeGenModule &CGM,
// If necessary, get a pointer to the element and emit it.
if (!Elt->isNullValue() && !isa<llvm::UndefValue>(Elt))
emitStoresForInitAfterBZero(
- CGM, Elt, Builder.CreateConstInBoundsGEP2_32(Loc, 0, i), isVolatile,
- Builder, IsAutoInit);
+ CGF, CGM, Elt, Builder.CreateConstInBoundsGEP2_32(Loc, 0, i),
+ isVolatile, Builder, IsAutoInit);
}
return;
}
@@ -966,7 +971,7 @@ static void emitStoresForInitAfterBZero(CodeGenModule &CGM,
// If necessary, get a pointer to the element and emit it.
if (!Elt->isNullValue() && !isa<llvm::UndefValue>(Elt))
- emitStoresForInitAfterBZero(CGM, Elt,
+ emitStoresForInitAfterBZero(CGF, CGM, Elt,
Builder.CreateConstInBoundsGEP2_32(Loc, 0, i),
isVolatile, Builder, IsAutoInit);
}
@@ -1169,9 +1174,9 @@ static Address createUnnamedGlobalForMemcpyFrom(CodeGenModule &CGM,
return SrcPtr.withElementType(CGM.Int8Ty);
}
-static void emitStoresForConstant(CodeGenModule &CGM, const VarDecl &D,
- Address Loc, bool isVolatile,
- CGBuilderTy &Builder,
+static void emitStoresForConstant(CodeGenFunction &CGF, CodeGenModule &CGM,
+ const VarDecl &D, Address Loc,
+ bool isVolatile, CGBuilderTy &Builder,
llvm::Constant *constant, bool IsAutoInit) {
auto *Ty = constant->getType();
uint64_t ConstantSize = CGM.getDataLayout().getTypeAllocSize(Ty);
@@ -1194,6 +1199,7 @@ static void emitStoresForConstant(CodeGenModule &CGM, const VarDecl &D,
if (shouldUseBZeroPlusStoresToInitialize(constant, ConstantSize)) {
auto *I = Builder.CreateMemSet(Loc, llvm::ConstantInt::get(CGM.Int8Ty, 0),
SizeVal, isVolatile);
+ CGF.addInstToCurrentSourceAtom(I, nullptr);
if (IsAutoInit)
I->addAnnotationMetadata("auto-init");
@@ -1201,7 +1207,7 @@ static void emitStoresForConstant(CodeGenModule &CGM, const VarDecl &D,
constant->isNullValue() || isa<llvm::UndefValue>(constant);
if (!valueAlreadyCorrect) {
Loc = Loc.withElementType(Ty);
- emitStoresForInitAfterBZero(CGM, constant, Loc, isVolatile, Builder,
+ emitStoresForInitAfterBZero(CGF, CGM, constant, Loc, isVolatile, Builder,
IsAutoInit);
}
return;
@@ -1240,7 +1246,7 @@ static void emitStoresForConstant(CodeGenModule &CGM, const VarDecl &D,
CharUnits::fromQuantity(Layout->getElementOffset(i));
Address EltPtr = Builder.CreateConstInBoundsByteGEP(
Loc.withElementType(CGM.Int8Ty), CurOff);
- emitStoresForConstant(CGM, D, EltPtr, isVolatile, Builder,
+ emitStoresForConstant(CGF, CGM, D, EltPtr, isVolatile, Builder,
constant->getAggregateElement(i), IsAutoInit);
}
return;
@@ -1251,7 +1257,7 @@ static void emitStoresForConstant(CodeGenModule &CGM, const VarDecl &D,
for (unsigned i = 0; i != ATy->getNumElements(); i++) {
Address EltPtr = Builder.CreateConstGEP(
Loc.withElementType(ATy->getElementType()), i);
- emitStoresForConstant(CGM, D, EltPtr, isVolatile, Builder,
+ emitStoresForConstant(CGF, CGM, D, EltPtr, isVolatile, Builder,
constant->getAggregateElement(i), IsAutoInit);
}
return;
@@ -1265,28 +1271,30 @@ static void emitStoresForConstant(CodeGenModule &CGM, const VarDecl &D,
createUnnamedGlobalForMemcpyFrom(
CGM, D, Builder, constant, Loc.getAlignment()),
SizeVal, isVolatile);
+ CGF.addInstToCurrentSourceAtom(I, nullptr);
+
if (IsAutoInit)
I->addAnnotationMetadata("auto-init");
}
-static void emitStoresForZeroInit(CodeGenModule &CGM, const VarDecl &D,
- Address Loc, bool isVolatile,
- CGBuilderTy &Builder) {
+static void emitStoresForZeroInit(CodeGenFunction &CGF, CodeGenModule &CGM,
+ const VarDecl &D, Address Loc,
+ bool isVolatile, CGBuilderTy &Builder) {
llvm::Type *ElTy = Loc.getElementType();
llvm::Constant *constant =
constWithPadding(CGM, IsPattern::No, llvm::Constant::getNullValue(ElTy));
- emitStoresForConstant(CGM, D, Loc, isVolatile, Builder, constant,
+ emitStoresForConstant(CGF, CGM, D, Loc, isVolatile, Builder, constant,
/*IsAutoInit=*/true);
}
-static void emitStoresForPatternInit(CodeGenModule &CGM, const VarDecl &D,
- Address Loc, bool isVolatile,
- CGBuilderTy &Builder) {
+static void emitStoresForPatternInit(CodeGenFunction &CGF, CodeGenModule &CGM,
+ const VarDecl &D, Address Loc,
+ bool isVolatile, CGBuilderTy &Builder) {
llvm::Type *ElTy = Loc.getElementType();
llvm::Constant *constant = constWithPadding(
CGM, IsPattern::Yes, initializationPatternFor(CGM, ElTy));
assert(!isa<llvm::UndefValue>(constant));
- emitStoresForConstant(CGM, D, Loc, isVolatile, Builder, constant,
+ emitStoresForConstant(CGF, CGM, D, Loc, isVolatile, Builder, constant,
/*IsAutoInit=*/true);
}
@@ -1829,7 +1837,7 @@ void CodeGenFunction::emitZeroOrPatternForAutoVarInit(QualType type,
if (trivialAutoVarInitMaxSize > 0 &&
allocSize > trivialAutoVarInitMaxSize)
return;
- emitStoresForZeroInit(CGM, D, Loc, isVolatile, Builder);
+ emitStoresForZeroInit(*this, CGM, D, Loc, isVolatile, Builder);
break;
case LangOptions::TrivialAutoVarInitKind::Pattern:
if (CGM.stopAutoInit())
@@ -1837,7 +1845,7 @@ void CodeGenFunction::emitZeroOrPatternForAutoVarInit(QualType type,
if (trivialAutoVarInitMaxSize > 0 &&
allocSize > trivialAutoVarInitMaxSize)
return;
- emitStoresForPatternInit(CGM, D, Loc, isVolatile, Builder);
+ emitStoresForPatternInit(*this, CGM, D, Loc, isVolatile, Builder);
break;
}
return;
@@ -1921,6 +1929,7 @@ void CodeGenFunction::EmitAutoVarInit(const AutoVarEmission &emission) {
const VarDecl &D = *emission.Variable;
auto DL = ApplyDebugLocation::CreateDefaultArtificial(*this, D.getLocation());
+ auto Grp = ApplyAtomGroup(*this);
QualType type = D.getType();
// If this local has an initializer, emit it now.
@@ -2052,7 +2061,7 @@ void CodeGenFunction::EmitAutoVarInit(const AutoVarEmission &emission) {
return EmitStoreThroughLValue(RValue::get(constant), lv, true);
}
- emitStoresForConstant(CGM, D, Loc.withElementType(CGM.Int8Ty),
+ emitStoresForConstant(*this, CGM, D, Loc.withElementType(CGM.Int8Ty),
type.isVolatileQualified(), Builder, constant,
/*IsAutoInit=*/false);
}
diff --git a/clang/lib/CodeGen/CGDeclCXX.cpp b/clang/lib/CodeGen/CGDeclCXX.cpp
index f5950f03673a1..93468d87c1e67 100644
--- a/clang/lib/CodeGen/CGDeclCXX.cpp
+++ b/clang/lib/CodeGen/CGDeclCXX.cpp
@@ -215,8 +215,10 @@ void CodeGenFunction::EmitCXXGlobalVarDeclInit(const VarDecl &D,
}
bool NeedsDtor =
D.needsDestruction(getContext()) == QualType::DK_cxx_destructor;
- if (PerformInit)
+ if (PerformInit) {
+ auto Grp = ApplyAtomGroup(*this);
EmitDeclInit(*this, D, DeclAddr);
+ }
if (D.getType().isConstantStorage(getContext(), true, !NeedsDtor))
EmitDeclInvariant(*this, D, DeclPtr);
else
diff --git a/clang/lib/CodeGen/CGExpr.cpp b/clang/lib/CodeGen/CGExpr.cpp
index 191912ca7d800..714dd87920e2e 100644
--- a/clang/lib/CodeGen/CGExpr.cpp
+++ b/clang/lib/CodeGen/CGExpr.cpp
@@ -36,6 +36,7 @@
#include "llvm/ADT/ScopeExit.h"
#include "llvm/ADT/StringExtras.h"
#include "llvm/IR/DataLayout.h"
+#include "llvm/IR/Instructions.h"
#include "llvm/IR/Intrinsics.h"
#include "llvm/IR/LLVMContext.h"
#include "llvm/IR/MDBuilder.h"
@@ -2174,6 +2175,8 @@ void CodeGenFunction::EmitStoreOfScalar(llvm::Value *Value, Address Addr,
}
llvm::StoreInst *Store = Builder.CreateStore(Value, Addr, Volatile);
+ addInstToCurrentSourceAtom(Store, Value);
+
if (isNontemporal) {
llvm::MDNode *Node =
llvm::MDNode::get(Store->getContext(),
@@ -2427,8 +2430,9 @@ void CodeGenFunction::EmitStoreThroughLValue(RValue Src, LValue Dst,
// <N x i1> --> <iN>.
Vec = Builder.CreateBitCast(Vec, IRStoreTy);
}
- Builder.CreateStore(Vec, Dst.getVectorAddress(),
- Dst.isVolatileQualified());
+ auto *I = Builder.CreateStore(Vec, Dst.getVectorAddress(),
+ Dst.isVolatileQualified());
+ addInstToCurrentSourceAtom(I, Src.getScalarVal());
return;
}
@@ -2585,7 +2589,10 @@ void CodeGenFunction::EmitStoreThroughBitfieldLValue(RValue Src, LValue Dst,
}
// Write the new value back out.
- Builder.CreateStore(SrcVal, Ptr, Dst.isVolatileQualified());
+ auto *I = Builder.CreateStore(SrcVal, Ptr, Dst.isVolatileQualified());
+ addInstToCurrentSourceAtom(I, SrcVal);
+ // FIXME(OCH): Should the casts below get added to the group? It looks like
+ // we're ok without doing that.
// Return the new value of the bit-field, if requested.
if (Result) {
@@ -3901,7 +3908,7 @@ void CodeGenFunction::EmitCfiCheckFail() {
CGM.addUsedGlobal(F);
}
-void CodeGenFunction::EmitUnreachable(SourceLocation Loc) {
+llvm::UnreachableInst *CodeGenFunction::EmitUnreachable(SourceLocation Loc) {
if (SanOpts.has(SanitizerKind::Unreachable)) {
SanitizerScope SanScope(this);
EmitCheck(std::make_pair(static_cast<llvm::Value *>(Builder.getFalse()),
@@ -3909,7 +3916,7 @@ void CodeGenFunction::EmitUnreachable(SourceLocation Loc) {
SanitizerHandler::BuiltinUnreachable,
EmitCheckSourceLocation(Loc), {});
}
- Builder.CreateUnreachable();
+ return Builder.CreateUnreachable();
}
void CodeGenFunction::EmitTrapCheck(llvm::Value *Checked,
@@ -5772,6 +5779,15 @@ LValue CodeGenFunction::EmitBinaryOperatorLValue(const BinaryOperator *E) {
assert(E->getOpcode() == BO_Assign && "unexpected binary l-value");
+ // This covers both LHS and RHS expressions, though nested RHS
+ // expressions may get subsequently separately grouped.
+ // FIXME(OCH): Not clear yet if we've got fine enough control
+ // to pick and choose when we need to. Currently looks ok:
+ // a = b = c -> Two atoms.
+ // x = new(1) -> One atom (for both addr store and value store).
+ // Complex and agg assignment -> One atom.
+ auto Grp = ApplyAtomGroup(*this);
+
// Note that in all of these cases, __block variables need the RHS
// evaluated first just in case the variable gets moved by the RHS.
diff --git a/clang/lib/CodeGen/CGExprAgg.cpp b/clang/lib/CodeGen/CGExprAgg.cpp
index c8bdda375d1b1..071d647df150b 100644
--- a/clang/lib/CodeGen/CGExprAgg.cpp
+++ b/clang/lib/CodeGen/CGExprAgg.cpp
@@ -12,6 +12,7 @@
#include "CGCXXABI.h"
#include "CGHLSLRuntime.h"
+#include "CGDebugInfo.h"
#include "CGObjCRuntime.h"
#include "CGRecordLayout.h"
#include "CodeGenFunction.h"
@@ -724,6 +725,7 @@ void AggExprEmitter::EmitArrayInit(Address DestPtr, llvm::ArrayType *AType,
// Advance to the start of the rest of the array.
llvm::Value *element = begin;
if (NumInitElements) {
+ // TODO: OCH - test this too.
element = Builder.CreateInBoundsGEP(
llvmElementType, element,
llvm::ConstantInt::get(CGF.SizeTy, NumInitElements),
@@ -2192,7 +2194,9 @@ static void CheckAggExprForMemSetUse(AggValueSlot &Slot, const Expr *E,
llvm::Constant *SizeVal = CGF.Builder.getInt64(Size.getQuantity());
Address Loc = Slot.getAddress().withElementType(CGF.Int8Ty);
- CGF.Builder.CreateMemSet(Loc, CGF.Builder.getInt8(0), SizeVal, false);
+ auto *I =
+ CGF.Builder.CreateMemSet(Loc, CGF.Builder.getInt8(0), SizeVal, false);
+ CGF.addInstToCurrentSourceAtom(I, nullptr);
// Tell the AggExprEmitter that the slot is known zero.
Slot.setZeroed();
@@ -2394,7 +2398,8 @@ void CodeGenFunction::EmitAggregateCopy(LValue Dest, LValue Src, QualType Ty,
}
}
- auto Inst = Builder.CreateMemCpy(DestPtr, SrcPtr, SizeVal, isVolatile);
+ auto *Inst = Builder.CreateMemCpy(DestPtr, SrcPtr, SizeVal, isVolatile);
+ addInstToCurrentSourceAtom(Inst, nullptr);
// Determine the metadata to describe the position of any padding in this
// memcpy, as well as the TBAA tags for the members of the struct, in case
diff --git a/clang/lib/CodeGen/CGExprComplex.cpp b/clang/lib/CodeGen/CGExprComplex.cpp
index ff7c55be246cc..ed29b94f49967 100644
--- a/clang/lib/CodeGen/CGExprComplex.cpp
+++ b/clang/lib/CodeGen/CGExprComplex.cpp
@@ -457,8 +457,12 @@ void ComplexExprEmitter::EmitStoreOfComplex(ComplexPairTy Val, LValue lvalue,
Address RealPtr = CGF.emitAddrOfRealComponent(Ptr, lvalue.getType());
Address ImagPtr = CGF.emitAddrOfImagComponent(Ptr, lvalue.getType());
- Builder.CreateStore(Val.first, RealPtr, lvalue.isVolatileQualified());
- Builder.CreateStore(Val.second, ImagPtr, lvalue.isVolatileQualified());
+ auto *R =
+ Builder.CreateStore(Val.first, RealPtr, lvalue.isVolatileQualified());
+ CGF.addInstToCurrentSourceAtom(R, Val.first);
+ auto *I =
+ Builder.CreateStore(Val.second, ImagPtr, lvalue.isVolatileQualified());
+ CGF.addInstToCurrentSourceAtom(I, Val.second);
}
@@ -1203,6 +1207,8 @@ LValue ComplexExprEmitter::
EmitCompoundAssignLValue(const CompoundAssignOperator *E,
ComplexPairTy (ComplexExprEmitter::*Func)(const BinOpInfo&),
RValue &Val) {
+ auto Grp = ApplyAtomGroup(CGF);
+
TestAndClearIgnoreReal();
TestAndClearIgnoreImag();
QualType LHSTy = E->getLHS()->getType();
@@ -1344,6 +1350,8 @@ LValue ComplexExprEmitter::EmitBinAssignLValue(const BinaryOperator *E,
}
ComplexPairTy ComplexExprEmitter::VisitBinAssign(const BinaryOperator *E) {
+ auto Grp = ApplyAtomGroup(CGF);
+
ComplexPairTy Val;
LValue LV = EmitBinAssignLValue(E, Val);
diff --git a/clang/lib/CodeGen/CGExprScalar.cpp b/clang/lib/CodeGen/CGExprScalar.cpp
index 6646057b9d772..84e339d063f41 100644
--- a/clang/lib/CodeGen/CGExprScalar.cpp
+++ b/clang/lib/CodeGen/CGExprScalar.cpp
@@ -892,6 +892,7 @@ class ScalarExprEmitter
return result; \
} \
Value *VisitBin##OP##Assign(const CompoundAssignOperator *E) { \
+ auto Grp = ApplyAtomGroup(CGF); \
return EmitCompoundAssign(E, &ScalarExprEmitter::Emit##OP); \
}
HANDLEBINOP(Mul)
@@ -2935,6 +2936,7 @@ class OMPLastprivateConditionalUpdateRAII {
llvm::Value *
ScalarExprEmitter::EmitScalarPrePostIncDec(const UnaryOperator *E, LValue LV,
bool isInc, bool isPre) {
+ auto Grp = ApplyAtomGroup(CGF);
OMPLastprivateConditionalUpdateRAII OMPRegion(CGF, E);
QualType type = E->getSubExpr()->getType();
llvm::PHINode *atomicPHI = nullptr;
@@ -4953,6 +4955,7 @@ llvm::Value *CodeGenFunction::EmitWithOriginalRHSBitfieldAssignment(
}
Value *ScalarExprEmitter::VisitBinAssign(const BinaryOperator *E) {
+ auto Grp = ApplyAtomGroup(CGF);
bool Ignore = TestAndClearIgnoreResultAssign();
Value *RHS;
@@ -5720,6 +5723,7 @@ LValue CodeGenFunction::EmitObjCIsaExpr(const ObjCIsaExpr *E) {
LValue CodeGenFunction::EmitCompoundAssignmentLValue(
const CompoundAssignOperator *E) {
+ auto Grp = ApplyAtomGroup(*this);
ScalarExprEmitter Scalar(*this);
Value *Result = nullptr;
switch (E->getOpcode()) {
diff --git a/clang/lib/CodeGen/CGStmt.cpp b/clang/lib/CodeGen/CGStmt.cpp
index abe799af32c6e..7b1eaf464734c 100644
--- a/clang/lib/CodeGen/CGStmt.cpp
+++ b/clang/lib/CodeGen/CGStmt.cpp
@@ -61,6 +61,11 @@ void CodeGenFunction::EmitStmt(const Stmt *S, ArrayRef<const Attr *> Attrs) {
assert(S && "Null statement?");
PGO.setCurrentStmt(S);
+ // FIXME(OCH): Would it be safer to have a stmt-level ApplyAtomGroup(*this)
+ // here, with constructs opting-out rather than opting-in to atom groups?
+ // Note if we do that we've got to be careful to keep atomGroup numbers
+ // dense (minimising unused numbers below the global watermark).
+
// These statements have their own debug info handling.
if (EmitSimpleStmt(S, Attrs))
return;
@@ -679,6 +684,9 @@ void CodeGenFunction::EmitBranch(llvm::BasicBlock *Target) {
// terminated, don't touch it.
} else {
// Otherwise, create a fall-through branch.
+ // In general unconditional branches are "boring" for stepping with
+ // Key Instructions, so outside of special cases we don't put these into
+ // atom groups.
Builder.CreateBr(Target);
}
@@ -1124,7 +1132,9 @@ void CodeGenFunction::EmitWhileStmt(const WhileStmt &S,
if (!Weights && CGM.getCodeGenOpts().OptimizationLevel)
BoolCondVal = emitCondLikelihoodViaExpectIntrinsic(
BoolCondVal, Stmt::getLikelihood(S.getBody()));
- Builder.CreateCondBr(BoolCondVal, LoopBody, ExitBlock, Weights);
+
+ auto *I = Builder.CreateCondBr(BoolCondVal, LoopBody, ExitBlock, Weights);
+ addInstToNewSourceAtom(I, BoolCondVal);
if (ExitBlock != LoopExit.getBlock()) {
EmitBlock(ExitBlock);
@@ -1236,9 +1246,10 @@ void CodeGenFunction::EmitDoStmt(const DoStmt &S,
// As long as the condition is true, iterate the loop.
if (EmitBoolCondBranch) {
uint64_t BackedgeCount = getProfileCount(S.getBody()) - ParentCount;
- Builder.CreateCondBr(
+ auto *I = Builder.CreateCondBr(
BoolCondVal, LoopBody, LoopExit.getBlock(),
createProfileWeightsForLoop(S.getCond(), BackedgeCount));
+ addInstToNewSourceAtom(I, BoolCondVal);
}
LoopStack.pop();
@@ -1303,6 +1314,8 @@ void CodeGenFunction::EmitForStmt(const ForStmt &S,
Continue = getJumpDestInCurrentScope("for.inc");
BreakContinueStack.push_back(BreakContinue(LoopExit, Continue));
+ llvm::BasicBlock *ForBody = nullptr;
+
if (S.getCond()) {
// If the for statement has a condition scope, emit the local variable
// declaration.
@@ -1327,18 +1340,20 @@ void CodeGenFunction::EmitForStmt(const ForStmt &S,
ExitBlock = createBasicBlock("for.cond.cleanup");
// As long as the condition is true, iterate the loop.
- llvm::BasicBlock *ForBody = createBasicBlock("for.body");
+ ForBody = createBasicBlock("for.body");
// C99 6.8.5p2/p4: The first substatement is executed if the expression
// compares unequal to 0. The condition must be a scalar type.
llvm::Value *BoolCondVal = EvaluateExprAsBool(S.getCond());
+
llvm::MDNode *Weights =
createProfileWeightsForLoop(S.getCond(), getProfileCount(S.getBody()));
if (!Weights && CGM.getCodeGenOpts().OptimizationLevel)
BoolCondVal = emitCondLikelihoodViaExpectIntrinsic(
BoolCondVal, Stmt::getLikelihood(S.getBody()));
- Builder.CreateCondBr(BoolCondVal, ForBody, ExitBlock, Weights);
+ auto *I = Builder.CreateCondBr(BoolCondVal, ForBody, ExitBlock, Weights);
+ addInstToNewSourceAtom(I, BoolCondVal);
if (ExitBlock != LoopExit.getBlock()) {
EmitBlock(ExitBlock);
@@ -1392,6 +1407,12 @@ void CodeGenFunction::EmitForStmt(const ForStmt &S,
if (CGM.shouldEmitConvergenceTokens())
ConvergenceTokenStack.pop_back();
+
+ if (ForBody) {
+ // We want the for closing brace to be step-able on to match existing
+ // behaviour. FIXME: invokes?
+ addInstToNewSourceAtom(ForBody->getTerminator(), nullptr);
+ }
}
void
@@ -1439,7 +1460,8 @@ CodeGenFunction::EmitCXXForRangeStmt(const CXXForRangeStmt &S,
if (!Weights && CGM.getCodeGenOpts().OptimizationLevel)
BoolCondVal = emitCondLikelihoodViaExpectIntrinsic(
BoolCondVal, Stmt::getLikelihood(S.getBody()));
- Builder.CreateCondBr(BoolCondVal, ForBody, ExitBlock, Weights);
+ auto *I = Builder.CreateCondBr(BoolCondVal, ForBody, ExitBlock, Weights);
+ addInstToNewSourceAtom(I, BoolCondVal);
if (ExitBlock != LoopExit.getBlock()) {
EmitBlock(ExitBlock);
@@ -1488,6 +1510,13 @@ CodeGenFunction::EmitCXXForRangeStmt(const CXXForRangeStmt &S,
if (CGM.shouldEmitConvergenceTokens())
ConvergenceTokenStack.pop_back();
+
+ // We want the for closing brace to be step-able on to match existing
+ // behaviour.
+ // Note that with exceptions, this doesn't do the right thing, since
+ // it puts the atomgroup on the invoke, rather than the flow-on block
+ // that contains the br we see in no-exception code.
+ addInstToNewSourceAtom(ForBody->getTerminator(), nullptr);
}
void CodeGenFunction::EmitReturnOfRValue(RValue RV, QualType Ty) {
@@ -1545,6 +1574,7 @@ static bool isSwiftAsyncCallee(const CallExpr *CE) {
/// if the function returns void, or may be missing one if the function returns
/// non-void. Fun stuff :).
void CodeGenFunction::EmitReturnStmt(const ReturnStmt &S) {
+ auto Grp = ApplyAtomGroup(*this);
if (requiresReturnValueCheck()) {
llvm::Constant *SLoc = EmitCheckSourceLocation(S.getBeginLoc());
auto *SLocPtr =
@@ -1620,16 +1650,30 @@ void CodeGenFunction::EmitReturnStmt(const ReturnStmt &S) {
// If this function returns a reference, take the address of the expression
// rather than the value.
RValue Result = EmitReferenceBindingToExpr(RV);
- Builder.CreateStore(Result.getScalarVal(), ReturnValue);
+ auto *S = Builder.CreateStore(Result.getScalarVal(), ReturnValue);
+ // Key Instructions: See comment in else block.
+ auto Rnk = IncAtomRank(*this);
+ addInstToCurrentSourceAtom(S, S->getValueOperand());
} else {
+ // Key Instructions: The return-value-alloca store and branch to the
+ // return block should both be considered part of the same return
+ // statement atom. Set KeyInstRank to 2 because the primary instruction is
+ // the branch, which we'll annotate separately below. Using the store (and
+ // the stored value) as a backup location is important to retain a step to
+ // the return statement after inlining.
+ // FIXME(OCH): Possibly fine giving the store and branch both rank 1, and
+ // dropping the `IncAtomRank` class.
+ auto Rnk = IncAtomRank(*this);
switch (getEvaluationKind(RV->getType())) {
case TEK_Scalar: {
llvm::Value *Ret = EmitScalarExpr(RV);
- if (CurFnInfo->getReturnInfo().getKind() == ABIArgInfo::Indirect)
+ if (CurFnInfo->getReturnInfo().getKind() == ABIArgInfo::Indirect) {
EmitStoreOfScalar(Ret, MakeAddrLValue(ReturnValue, RV->getType()),
/*isInit*/ true);
- else
- Builder.CreateStore(Ret, ReturnValue);
+ } else {
+ auto *S = Builder.CreateStore(Ret, ReturnValue);
+ addInstToCurrentSourceAtom(S, Ret, 2);
+ }
break;
}
case TEK_Complex:
@@ -2258,6 +2302,7 @@ void CodeGenFunction::EmitSwitchStmt(const SwitchStmt &S) {
if (S.getConditionVariable())
EmitDecl(*S.getConditionVariable());
+
llvm::Value *CondV = EmitScalarExpr(S.getCond());
// Create basic block to hold stuff that comes after switch
@@ -2265,7 +2310,10 @@ void CodeGenFunction::EmitSwitchStmt(const SwitchStmt &S) {
// explicit case ranges tests can have a place to jump to on
// failure.
llvm::BasicBlock *DefaultBlock = createBasicBlock("sw.default");
+
SwitchInsn = Builder.CreateSwitch(CondV, DefaultBlock);
+ addInstToNewSourceAtom(SwitchInsn, CondV);
+
if (PGO.haveRegionCounts()) {
// Walk the SwitchCase list to find how many there are.
uint64_t DefaultCount = 0;
diff --git a/clang/lib/CodeGen/CodeGenFunction.cpp b/clang/lib/CodeGen/CodeGenFunction.cpp
index 08165e0b28406..b0435c8272797 100644
--- a/clang/lib/CodeGen/CodeGenFunction.cpp
+++ b/clang/lib/CodeGen/CodeGenFunction.cpp
@@ -36,11 +36,12 @@
#include "clang/CodeGen/CGFunctionInfo.h"
#include "clang/Frontend/FrontendDiagnostic.h"
#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
#include "llvm/Frontend/OpenMP/OMPIRBuilder.h"
#include "llvm/IR/DataLayout.h"
#include "llvm/IR/Dominators.h"
#include "llvm/IR/FPEnv.h"
-#include "llvm/IR/Instruction.h"
+#include "llvm/IR/Instructions.h"
#include "llvm/IR/IntrinsicInst.h"
#include "llvm/IR/Intrinsics.h"
#include "llvm/IR/MDBuilder.h"
@@ -339,6 +340,15 @@ llvm::DebugLoc CodeGenFunction::EmitReturnBlock() {
// later by the actual 'ret' instruction.
llvm::DebugLoc Loc = BI->getDebugLoc();
Builder.SetInsertPoint(BI->getParent());
+
+ // Key Instructions: If there's only one `ret` then we want to put the
+ // instruction in the same source atom group as the store to the ret-value
+ // alloca and unconditional `br` to the return block that we're about to
+ // delete. It all comes from the same source (`return (value)`).
+ if (auto *DI = getDebugInfo(); DI && BI->getDebugLoc())
+ DI->setRetInstSourceAtomOverride(
+ BI->getDebugLoc().get()->getAtomGroup());
+
BI->eraseFromParent();
delete ReturnBlock.getBlock();
ReturnBlock = JumpDest();
@@ -1537,6 +1547,12 @@ void CodeGenFunction::GenerateCode(GlobalDecl GD, llvm::Function *Fn,
Bypasses.Init(Body);
}
+ // Finalize function debug info on exit.
+ auto Cleanup = llvm::make_scope_exit([this] {
+ if (CGDebugInfo *DI = getDebugInfo())
+ DI->completeFunction();
+ });
+
// Emit the standard function prologue.
StartFunction(GD, ResTy, Fn, FnInfo, Args, Loc, BodyRange.getBegin());
@@ -2112,6 +2128,8 @@ void CodeGenFunction::EmitBranchOnBoolExpr(
case HLSLControlFlowHintAttr::SpellingNotCalculated:
break;
}
+
+ addInstToNewSourceAtom(BrInst, CondV);
}
/// ErrorUnsupported - Print out an error that codegen doesn't support the
@@ -2232,14 +2250,16 @@ CodeGenFunction::EmitNullInitialization(Address DestPtr, QualType Ty) {
if (vla) return emitNonZeroVLAInit(*this, Ty, DestPtr, SrcPtr, SizeVal);
// Get and call the appropriate llvm.memcpy overload.
- Builder.CreateMemCpy(DestPtr, SrcPtr, SizeVal, false);
+ auto *I = Builder.CreateMemCpy(DestPtr, SrcPtr, SizeVal, false);
+ addInstToCurrentSourceAtom(I, nullptr);
return;
}
// Otherwise, just memset the whole thing to zero. This is legal
// because in LLVM, all default initializers (other than the ones we just
// handled above) are guaranteed to have a bit pattern of all zeros.
- Builder.CreateMemSet(DestPtr, Builder.getInt8(0), SizeVal, false);
+ auto *I = Builder.CreateMemSet(DestPtr, Builder.getInt8(0), SizeVal, false);
+ addInstToCurrentSourceAtom(I, nullptr);
}
llvm::BlockAddress *CodeGenFunction::GetAddrOfLabel(const LabelDecl *L) {
@@ -2584,6 +2604,27 @@ void CodeGenFunction::EmitDeclRefExprDbgValue(const DeclRefExpr *E,
Dbg->EmitGlobalVariable(E->getDecl(), Init);
}
+void CodeGenFunction::addInstToCurrentSourceAtom(
+ llvm::Instruction *KeyInstruction, llvm::Value *Backup,
+ uint8_t KeyInstRank) {
+ if (auto *DI = getDebugInfo())
+ DI->addInstToCurrentSourceAtom(KeyInstruction, Backup, KeyInstRank);
+}
+
+void CodeGenFunction::addInstToNewSourceAtom(llvm::Instruction *KeyInstruction,
+ llvm::Value *Backup,
+ uint8_t KeyInstRank) {
+ auto Grp = ApplyAtomGroup(*this);
+ addInstToCurrentSourceAtom(KeyInstruction, Backup, KeyInstRank);
+}
+
+void CodeGenFunction::addRetToOverrideOrNewSourceAtom(llvm::ReturnInst *Ret,
+ llvm::Value *Backup,
+ uint8_t KeyInstRank) {
+ if (auto *DI = getDebugInfo())
+ DI->addRetToOverrideOrNewSourceAtom(Ret, Backup, KeyInstRank);
+}
+
CodeGenFunction::PeepholeProtection
CodeGenFunction::protectFromPeepholes(RValue rvalue) {
// At the moment, the only aggressive peephole we do in IR gen
diff --git a/clang/lib/CodeGen/CodeGenFunction.h b/clang/lib/CodeGen/CodeGenFunction.h
index 018fc66b72a1e..5335901e05994 100644
--- a/clang/lib/CodeGen/CodeGenFunction.h
+++ b/clang/lib/CodeGen/CodeGenFunction.h
@@ -3018,6 +3018,7 @@ class CodeGenFunction : public CodeGenTypeCache {
/// Emit an aggregate assignment.
void EmitAggregateAssign(LValue Dest, LValue Src, QualType EltTy) {
bool IsVolatile = hasVolatileMember(EltTy);
+ auto Grp = ApplyAtomGroup(*this);
EmitAggregateCopy(Dest, Src, EltTy, AggValueSlot::MayOverlap, IsVolatile);
}
@@ -5255,7 +5256,7 @@ class CodeGenFunction : public CodeGenTypeCache {
/// Emit a reached-unreachable diagnostic if \p Loc is valid and runtime
/// checking is enabled. Otherwise, just emit an unreachable instruction.
- void EmitUnreachable(SourceLocation Loc);
+ llvm::UnreachableInst *EmitUnreachable(SourceLocation Loc);
/// Create a basic block that will call the trap intrinsic, and emit a
/// conditional branch to it, for the -ftrapv checks.
@@ -5313,6 +5314,21 @@ class CodeGenFunction : public CodeGenTypeCache {
unsigned NumElementsDst,
const llvm::Twine &Name = "");
+ // Wrapper, see \p DebugInfo::addInstToCurrentSourceAtom.
+ void addInstToCurrentSourceAtom(llvm::Instruction *KeyInstruction,
+ llvm::Value *Backup, uint8_t KeyInstRank = 1);
+
+ // Wrapper, see \p DebugInfo::addInstToCurrentSourceAtom.
+ // Adds the instruction to a new source atom rather than the existing
+ // scoped one.
+ void addInstToNewSourceAtom(llvm::Instruction *KeyInstruction,
+ llvm::Value *Backup, uint8_t KeyInstRank = 1);
+
+ // TODO(OCH): Comment.
+ void addRetToOverrideOrNewSourceAtom(llvm::ReturnInst *Ret,
+ llvm::Value *Backup,
+ uint8_t KeyInstRank = 1);
+
private:
// Emits a convergence_loop instruction for the given |BB|, with |ParentToken|
// as it's parent convergence instr.
diff --git a/clang/lib/CodeGen/ItaniumCXXABI.cpp b/clang/lib/CodeGen/ItaniumCXXABI.cpp
index b145da0f0ec09..a40d716f95e85 100644
--- a/clang/lib/CodeGen/ItaniumCXXABI.cpp
+++ b/clang/lib/CodeGen/ItaniumCXXABI.cpp
@@ -5054,6 +5054,7 @@ void ItaniumCXXABI::emitBeginCatch(CodeGenFunction &CGF,
// Emit the local.
CodeGenFunction::AutoVarEmission var = CGF.EmitAutoVarAlloca(*CatchParam);
+ auto Grp = ApplyAtomGroup(CGF);
InitCatchParam(CGF, *CatchParam, var.getObjectAddress(CGF), S->getBeginLoc());
CGF.EmitAutoVarCleanups(var);
}
diff --git a/clang/test/KeyInstructions/agg-cpy.cpp b/clang/test/KeyInstructions/agg-cpy.cpp
new file mode 100644
index 0000000000000..88a7e86e3d4b6
--- /dev/null
+++ b/clang/test/KeyInstructions/agg-cpy.cpp
@@ -0,0 +1,13 @@
+
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+struct Struct { int a, b, c; };
+void fun(Struct &a, Struct &b) {
+// CHECK: call void @llvm.memcpy{{.*}}, !dbg [[G1R1:!.*]]
+ a = b;
+// CHECK: ret void, !dbg [[G2R1:!.*]]
+}
+
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
diff --git a/clang/test/KeyInstructions/agg-init-bzero-plus-stores.cpp b/clang/test/KeyInstructions/agg-init-bzero-plus-stores.cpp
new file mode 100644
index 0000000000000..1e4766b76690b
--- /dev/null
+++ b/clang/test/KeyInstructions/agg-init-bzero-plus-stores.cpp
@@ -0,0 +1,25 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+char g;
+void a() {
+// CHECK: _Z1av()
+// CHECK: call void @llvm.memset{{.*}}, !dbg [[G1R1:!.*]]
+// CHECK: store i8 97{{.*}}, !dbg [[G1R1]]
+// CHECK: store i8 98{{.*}}, !dbg [[G1R1]]
+// CHECK: store i8 99{{.*}}, !dbg [[G1R1]]
+// CHECK: store i8 100{{.*}}, !dbg [[G1R1]]
+ char big[65536] = { 'a', 'b', 'c', 'd' };
+
+// CHECK: call void @llvm.memset{{.*}}, !dbg [[G2R1:!.*]]
+// CHECK: [[l:%.*]] = load i8{{.*}}, !dbg [[G2R2:!.*]]
+// CHECK: store i8 [[l]]{{.*}}, !dbg [[G2R1C22:!.*]]
+ char big2[65536] = { g };
+// CHECK: ret void{{.*}}, !dbg [[G3R1:!.*]]
+}
+
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
+// CHECK: [[G2R2]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 2)
+// CHECK: [[G2R1C22]] = !DILocation({{.*}}column: 22{{.*}}, atomGroup: 2, atomRank: 1)
+// CHECK: [[G3R1]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 1)
diff --git a/clang/test/KeyInstructions/agg-init.cpp b/clang/test/KeyInstructions/agg-init.cpp
new file mode 100644
index 0000000000000..11b56275ccf2a
--- /dev/null
+++ b/clang/test/KeyInstructions/agg-init.cpp
@@ -0,0 +1,24 @@
+
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+// The implicit-check-not is important; we don't want the GEPs created for the
+// store locations to be included in the atom group.
+
+int g;
+void a() {
+// CHECK: _Z1av()
+// CHECK: call void @llvm.memcpy{{.*}}, !dbg [[G1R1:!.*]]
+ int A[] = { 1, 2, 3};
+// CHECK: store i32 1, ptr %{{.*}}, !dbg [[G2R1:!.*]]
+// CHECK: store i32 2, ptr %{{.*}}, !dbg [[G2R1]]
+// CHECK: %0 = load i32, ptr @g{{.*}}, !dbg [[G2R2:!.*]]
+// CHECK: store i32 %0, ptr %{{.*}}, !dbg [[G2R1]]
+ int B[] = { 1, 2, g};
+// CHECK: ret{{.*}}, !dbg [[G3R1:!.*]]
+}
+
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
+// CHECK: [[G2R2]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 2)
+// CHECK: [[G3R1]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 1)
diff --git a/clang/test/KeyInstructions/agg-null-init.cpp b/clang/test/KeyInstructions/agg-null-init.cpp
new file mode 100644
index 0000000000000..c1243302e3b4a
--- /dev/null
+++ b/clang/test/KeyInstructions/agg-null-init.cpp
@@ -0,0 +1,21 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+struct S { void *data[3]; };
+
+// CHECK: _Z1av
+// CHECK-NEXT: entry:
+// CHECK-NEXT: call void @llvm.memset{{.*}}, !dbg [[A_G1R2:!.*]]
+// CHECK-NEXT: ret void, !dbg [[A_G1R1:!.*]]
+S a() { return S(); }
+
+// CHECK: _Z1bv
+// CHECK-NEXT: entry:
+// CHECK-NEXT: call void @llvm.memset{{.*}}, !dbg [[B_G1R2:!.*]]
+// CHECK-NEXT: ret void, !dbg [[B_G1R1:!.*]]
+S b() { return S{}; }
+
+// CHECK: [[A_G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[A_G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[B_G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[B_G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
diff --git a/clang/test/KeyInstructions/agg-return-va-arg.cpp b/clang/test/KeyInstructions/agg-return-va-arg.cpp
new file mode 100644
index 0000000000000..76aa112a16e97
--- /dev/null
+++ b/clang/test/KeyInstructions/agg-return-va-arg.cpp
@@ -0,0 +1,23 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+typedef struct {
+ struct{} a;
+ double b;
+} s1;
+
+s1 f(int z, ...) {
+ __builtin_va_list list;
+ __builtin_va_start(list, z);
+// CHECK: vaarg.end:
+// CHECK-NEXT: %vaarg.addr = phi ptr
+// CHECK-NEXT: call void @llvm.memcpy{{.*}}, !dbg [[COL10_G1R2:!.*]]
+// CHECK-NEXT: %3 = getelementptr{{.*}}
+// CHECK-NEXT: %4 = load double, ptr %3{{.*}} !dbg [[COL3_G1R2:!.*]]
+// CHECK-NEXT: ret double %4, !dbg [[G1R1:!.*]]
+ return __builtin_va_arg(list, s1);
+}
+
+// CHECK: [[COL10_G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[COL3_G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
diff --git a/clang/test/KeyInstructions/assign.c b/clang/test/KeyInstructions/assign.c
new file mode 100644
index 0000000000000..2c71084d545ac
--- /dev/null
+++ b/clang/test/KeyInstructions/assign.c
@@ -0,0 +1,11 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+unsigned long long g;
+void fun() { g = 0; }
+
+// CHECK: store i64 0, ptr @g{{.*}}, !dbg [[G1R1:!.*]]
+// CHECK: ret void, !dbg [[G2R1:!.*]]
+
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
diff --git a/clang/test/KeyInstructions/binop.c b/clang/test/KeyInstructions/binop.c
new file mode 100644
index 0000000000000..9629dac4e3d81
--- /dev/null
+++ b/clang/test/KeyInstructions/binop.c
@@ -0,0 +1,17 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+long a;
+void b() {
+ long l;
+ l += a;
+}
+
+// CHECK: %add = add {{.*}}, !dbg [[G1R2:!.*]]
+// CHECK: store {{.*}}, !dbg [[G1R1:!.*]]
+// CHECK: ret void, !dbg [[G2R1:!.*]]
+
+// CHECK: [[G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
+
diff --git a/clang/test/KeyInstructions/bitfield.cpp b/clang/test/KeyInstructions/bitfield.cpp
new file mode 100644
index 0000000000000..a9498e6684a8d
--- /dev/null
+++ b/clang/test/KeyInstructions/bitfield.cpp
@@ -0,0 +1,15 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+struct S { int a:3; };
+
+void foo(int x, S s) {
+// CHECK: %bf.set = or i8 %bf.clear, %bf.value, !dbg [[G1R2:!.*]]
+// CHECK: store i8 %bf.set, ptr %s, align 4, !dbg [[G1R1:!.*]]
+ s.a = x;
+// CHECK: ret void, !dbg [[G2R1:!.*]]
+}
+
+// CHECK: [[G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
diff --git a/clang/test/KeyInstructions/cast.cpp b/clang/test/KeyInstructions/cast.cpp
new file mode 100644
index 0000000000000..81b9b1a6983b1
--- /dev/null
+++ b/clang/test/KeyInstructions/cast.cpp
@@ -0,0 +1,17 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s
+//--implicit-check-not atomGroup --implicit-check-not atomRank
+
+float g;
+void a() {
+// CHECK: %0 = load float, ptr @g{{.*}}, !dbg [[G1R3:!.*]]
+// CHECK: %conv = fptosi float %0 to i32{{.*}}, !dbg [[G1R2:!.*]]
+// CHECK: store i32 %conv, ptr %a{{.*}}, !dbg [[G1R1:!.*]]
+ int a = g;
+// CHECK: ret{{.*}}, !dbg [[G2R1:!.*]]
+}
+
+// CHECK: [[G1R3]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 3)
+// CHECK: [[G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
diff --git a/clang/test/KeyInstructions/complex.cpp b/clang/test/KeyInstructions/complex.cpp
new file mode 100644
index 0000000000000..b18527a473de4
--- /dev/null
+++ b/clang/test/KeyInstructions/complex.cpp
@@ -0,0 +1,28 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+_Complex int ci;
+void test() {
+// CHECK: %ci.real = load i32, ptr @ci{{.*}}, !dbg [[G1R2:!.*]]
+// CHECK: %ci.imag = load i32, ptr getelementptr inbounds ({ i32, i32 }, ptr @ci, i32 0, i32 1){{.*}}, !dbg [[G1R2]]
+// CHECK: store i32 %ci.real, ptr @ci{{.*}}, !dbg [[G1R1:!.*]]
+// CHECK: store i32 %ci.imag, ptr getelementptr inbounds ({ i32, i32 }, ptr @ci, i32 0, i32 1){{.*}}, !dbg [[G1R1]]
+ ci = ci;
+// CHECK: %add.r = add i32 %ci.real3, %ci.real1, !dbg [[G2R2:!.*]]
+// CHECK: %add.i = add i32 %ci.imag4, %ci.imag2, !dbg [[G2R2]]
+// CHECK: store i32 %add.r, ptr @ci{{.*}}, !dbg [[G2R1:!.*]]
+// CHECK: store i32 %add.i, ptr getelementptr inbounds ({ i32, i32 }, ptr @ci, i32 0, i32 1){{.*}}, !dbg [[G2R1]]
+ ci += ci;
+// CHECK: %add = add nsw i32 %0, %1, !dbg [[G3R2:!.*]]
+// CHECK: store i32 %add, ptr getelementptr inbounds ({ i32, i32 }, ptr @ci, i32 0, i32 1){{.*}}, !dbg [[G3R1:!.*]]
+ __imag ci = __imag ci + __imag ci;
+// CHECK: ret void, !dbg [[G4R1:!.*]]
+}
+
+// CHECK: [[G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R2]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 2)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
+// CHECK: [[G3R2]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 2)
+// CHECK: [[G3R1]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 1)
+// CHECK: [[G4R1]] = !DILocation({{.*}}, atomGroup: 4, atomRank: 1)
diff --git a/clang/test/KeyInstructions/compound-assign.cpp b/clang/test/KeyInstructions/compound-assign.cpp
new file mode 100644
index 0000000000000..1e969bb9749b0
--- /dev/null
+++ b/clang/test/KeyInstructions/compound-assign.cpp
@@ -0,0 +1,13 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+unsigned long long g;
+void fun() { g += 60; }
+
+// CHECK: %add = add i64 %0, 60, !dbg [[G1R2:!.*]]
+// CHECK: store i64 %add, ptr @g{{.*}}, !dbg [[G1R1:!.*]]
+// CHECK: ret void, !dbg [[G2R1:!.*]]
+
+// CHECK: [[G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
diff --git a/clang/test/KeyInstructions/do.cpp b/clang/test/KeyInstructions/do.cpp
new file mode 100644
index 0000000000000..c071eb1279085
--- /dev/null
+++ b/clang/test/KeyInstructions/do.cpp
@@ -0,0 +1,20 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+// FIXME: Perennial quesiton: should the dec be its own source atom or not
+// (currently it is).
+
+void a(int A) {
+// CHECK: %dec = add nsw i32 %0, -1, !dbg [[G1R2:!.*]]
+// CHECK: store i32 %dec, ptr %A.addr{{.*}}, !dbg [[G1R1:!.*]]
+// CHECK: %tobool = icmp ne i32 %dec, 0, !dbg [[G2R2:!.*]]
+// CHECK: br i1 %tobool, label %do.body, label %do.end, !dbg [[G2R1:!.*]], !llvm.loop
+ do { } while (--A);
+// CHECK: ret{{.*}}, !dbg [[G3R1:!.*]]
+}
+
+// CHECK: [[G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R2]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 2)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
+// CHECK: [[G3R1]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 1)
diff --git a/clang/test/KeyInstructions/double-assign.cpp b/clang/test/KeyInstructions/double-assign.cpp
new file mode 100644
index 0000000000000..ba644d5fb2aef
--- /dev/null
+++ b/clang/test/KeyInstructions/double-assign.cpp
@@ -0,0 +1,22 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+// FIXME: Question - should these all be in the same atomGroup or not?
+// FIXME: Because of the atomGroup implementation the load can only be
+// associated with one of the two stores, despite being a good backup
+// loction for both.
+
+int g;
+void a() {
+// CHECK: %0 = load i32, ptr @g{{.*}}, !dbg [[G1R2:!.*]]
+// CHECK: store i32 %0, ptr %b{{.*}}, !dbg [[G2R1:!.*]]
+// CHECK: store i32 %0, ptr %a{{.*}}, !dbg [[G1R1:!.*]]
+ int a, b;
+ a = b = g;
+// CHECK: ret{{.*}}, !dbg [[G3R1:!.*]]
+}
+
+// CHECK: [[G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G3R1]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 1)
diff --git a/clang/test/KeyInstructions/for.cpp b/clang/test/KeyInstructions/for.cpp
new file mode 100644
index 0000000000000..09fe285ef2052
--- /dev/null
+++ b/clang/test/KeyInstructions/for.cpp
@@ -0,0 +1,34 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+// FIXME: Perennial quesiton: should the inc be its own source atom or not
+// (currently it is).
+
+void a(int A) {
+// CHECK: entry:
+// CHECK: store i32 0, ptr %i{{.*}}, !dbg [[G1R1:!.*]]
+// CHECK: for.cond:
+// CHECK: %cmp = icmp slt i32 %0, %1, !dbg [[G2R2:!.*]]
+// CHECK: br i1 %cmp, label %for.body, label %for.end, !dbg [[G2R1:!.*]]
+
+// FIXME: Added uncond br group here which is useful for O0, which we're
+// no longer targeting. With optimisations loop rotate puts the condition
+// into for.inc and simplifycfg smooshes that and for.body together, so
+// it's not clear whether it adds any value.
+// CHECK: for.body:
+// CHECK: br label %for.inc, !dbg [[G4R1:!.*]]
+
+// CHECK: for.inc:
+// CHECK: %inc = add{{.*}}, !dbg [[G3R2:!.*]]
+// CHECK: store i32 %inc, ptr %i{{.*}}, !dbg [[G3R1:!.*]]
+ for (int i = 0; i < A; ++i) { }
+// CHECK: ret void, !dbg [[G5R1:!.*]]
+}
+
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R2]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 2)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
+// CHECK: [[G4R1]] = !DILocation({{.*}}, atomGroup: 4, atomRank: 1)
+// CHECK: [[G3R2]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 2)
+// CHECK: [[G3R1]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 1)
+// CHECK: [[G5R1]] = !DILocation({{.*}}, atomGroup: 5, atomRank: 1)
diff --git a/clang/test/KeyInstructions/if.cpp b/clang/test/KeyInstructions/if.cpp
new file mode 100644
index 0000000000000..73e9e0cf08150
--- /dev/null
+++ b/clang/test/KeyInstructions/if.cpp
@@ -0,0 +1,42 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+int g;
+void a(int A) {
+// CHECK: entry:
+// CHECK: %tobool = icmp ne i32 %0, 0{{.*}}, !dbg [[G1R2:!.*]]
+// CHECK: br i1 %tobool, label %if.then, label %if.end{{.*}}, !dbg [[G1R1:!.*]]
+ if (A)
+ ;
+// The assignment in the if currently gets a distinct source atom group.
+// FIXME: Is that the right choice?
+// CHECK: if.end:
+// CHECK: %1 = load i32, ptr %A.addr{{.*}}, !dbg [[G2R2:!.*]]
+// CHECK: store i32 %1, ptr @g{{.*}}, !dbg [[G2R1:!.*]]
+// CHECK: %tobool1 = icmp ne i32 %1, 0{{.*}}, !dbg [[G3R2:!.*]]
+// CHECK: br i1 %tobool1, label %if.then2, label %if.end3{{.*}}, !dbg [[G3R1:!.*]]
+ if ((g = A))
+ ;
+// The assignment in the if currently gets a distinct source atom group.
+// FIXME: Is that the right choice?
+// CHECK: if.end3:
+// CHECK: %2 = load i32, ptr %A.addr{{.*}}, !dbg [[G4R2:!.*]]
+// CHECK: store i32 %2, ptr %B{{.*}}, !dbg [[G4R1:!.*]]
+// CHECK: %tobool4 = icmp ne i32 %3, 0{{.*}}, !dbg [[G5R2:!.*]]
+// CHECK: br i1 %tobool4, label %if.then5, label %if.end6{{.*}}, !dbg [[G5R1:!.*]]
+ if (int B = A; B)
+ ;
+// CHECK: ret{{.*}}, !dbg [[G6R1:!.*]]
+}
+
+// CHECK: [[G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R2]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 2)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
+// CHECK: [[G3R2]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 2)
+// CHECK: [[G3R1]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 1)
+// CHECK: [[G4R2]] = !DILocation({{.*}}, atomGroup: 4, atomRank: 2)
+// CHECK: [[G4R1]] = !DILocation({{.*}}, atomGroup: 4, atomRank: 1)
+// CHECK: [[G5R2]] = !DILocation({{.*}}, atomGroup: 5, atomRank: 2)
+// CHECK: [[G5R1]] = !DILocation({{.*}}, atomGroup: 5, atomRank: 1)
+// CHECK: [[G6R1]] = !DILocation({{.*}}, atomGroup: 6, atomRank: 1)
diff --git a/clang/test/KeyInstructions/inc.cpp b/clang/test/KeyInstructions/inc.cpp
new file mode 100644
index 0000000000000..7018fb5ec67ae
--- /dev/null
+++ b/clang/test/KeyInstructions/inc.cpp
@@ -0,0 +1,13 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+int g;
+void fun() { g++; }
+
+// CHECK: %inc = add nsw i32 %0, 1, !dbg [[G1R2:!.*]]
+// CHECK: store i32 %inc, ptr @g{{.*}}, !dbg [[G1R1:!.*]]
+// CHECK: ret void, !dbg [[G2R1:!.*]]
+
+// CHECK: [[G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
diff --git a/clang/test/KeyInstructions/member-init.cpp b/clang/test/KeyInstructions/member-init.cpp
new file mode 100644
index 0000000000000..b914a21022586
--- /dev/null
+++ b/clang/test/KeyInstructions/member-init.cpp
@@ -0,0 +1,13 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s
+
+class Cls {
+ int x = 1;
+};
+
+void fun() {
+ Cls c;
+}
+
+// CHECK: store i32 1, ptr %x{{.*}}, !dbg [[G1R1:!.*]]
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
diff --git a/clang/test/KeyInstructions/memset.c b/clang/test/KeyInstructions/memset.c
new file mode 100644
index 0000000000000..8e55ef9704f1e
--- /dev/null
+++ b/clang/test/KeyInstructions/memset.c
@@ -0,0 +1,10 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - \
+// RUN: | FileCheck %s
+
+void *memset(void *, int, unsigned long);
+void a(int *P) {
+ memset(P, 1, 8);
+}
+
+// CHECK: call void @llvm.memset{{.*}}, !dbg [[G1R1:!.*]]
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
diff --git a/clang/test/KeyInstructions/multi-func.cpp b/clang/test/KeyInstructions/multi-func.cpp
new file mode 100644
index 0000000000000..5f1c24f447dcd
--- /dev/null
+++ b/clang/test/KeyInstructions/multi-func.cpp
@@ -0,0 +1,28 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+// Check atomGroup is reset to start at 1 in each function.
+
+void a() {
+// CHECK: _Z1av()
+// CHECK: store i32 0, ptr %A{{.*}}, !dbg [[A_G1R1:!.*]]
+ int A = 0;
+// CHECK: store i32 0, ptr %B{{.*}}, !dbg [[A_G2R1:!.*]]
+ int B = 0;
+// CHECK: ret{{.*}}, !dbg [[A_G3R1:!.*]]
+}
+
+void b() {
+// CHECK: _Z1bv()
+// CHECK: store i32 0, ptr %A{{.*}}, !dbg [[B_G1R1:!.*]]
+ int A = 0;
+// CHECK: ret{{.*}}, !dbg [[B_G2R1:!.*]]
+}
+
+// CHECK: [[A:!.*]] = distinct !DISubprogram(name: "a",
+// CHECK: [[A_G1R1]] = !DILocation({{.*}}, scope: [[A]], atomGroup: 1, atomRank: 1)
+// CHECK: [[A_G2R1]] = !DILocation({{.*}}, scope: [[A]], atomGroup: 2, atomRank: 1)
+// CHECK: [[A_G3R1]] = !DILocation({{.*}}, scope: [[A]], atomGroup: 3, atomRank: 1)
+// CHECK: [[B:!.*]] = distinct !DISubprogram(name: "b",
+// CHECK: [[B_G1R1]] = !DILocation({{.*}}, scope: [[B]], atomGroup: 1, atomRank: 1)
+// CHECK: [[B_G2R1]] = !DILocation({{.*}}, scope: [[B]], atomGroup: 2, atomRank: 1)
diff --git a/clang/test/KeyInstructions/new.cpp b/clang/test/KeyInstructions/new.cpp
new file mode 100644
index 0000000000000..9cb82338883fb
--- /dev/null
+++ b/clang/test/KeyInstructions/new.cpp
@@ -0,0 +1,36 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+
+void f(int x) {
+// CHECK: %call = call noalias noundef nonnull ptr @_Znwm{{.*}}, !dbg [[G1R2_C12:!.*]]
+// CHECK: %0 = load i32, ptr %x.addr{{.*}}, !dbg [[G1R2_C20:!.*]]
+// CHECK: store i32 %0, ptr %call{{.*}}, !dbg [[G1R1_C12:!.*]]
+// CHECK: store ptr %call, ptr %{{.*}}, !dbg [[G1R1_C8:!.*]]
+ int *n = new int(x);
+// CHECK: %call1 = call noalias noundef nonnull ptr @_Znwm{{.*}}, !dbg [[G2R2_C7:!.*]]
+// CHECK: %1 = load i32, ptr %x.addr{{.*}}, !dbg [[G2R2_C15:!.*]]
+// CHECK: store i32 %1, ptr %call{{.*}}, !dbg [[G2R1_C7:!.*]]
+// CHECK: store ptr %call1, ptr %{{.*}}, !dbg [[G2R1_C5:!.*]]
+ n = new int(x);
+// CHECK: %2 = load i32, ptr %x.addr{{.*}}, !dbg [[G3R2:!.*]]
+// CHECK: %3 = load ptr, ptr %n
+// CHECK: store i32 %2, ptr %3{{.*}}, !dbg [[G3R1:!.*]]
+ *n = x;
+// CHECK: ret void, !dbg [[G4R1:!.*]]
+}
+
+// CHECK: [[G1R2_C12]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[G1R2_C20]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[G1R1_C12]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G1R1_C8]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+
+// CHECK: [[G2R2_C7]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 2)
+// CHECK: [[G2R2_C15]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 2)
+// CHECK: [[G2R1_C7]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
+// CHECK: [[G2R1_C5]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
+
+// CHECK: [[G3R2]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 2)
+// CHECK: [[G3R1]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 1)
+
+// CHECK: [[G4R1]] = !DILocation({{.*}}, atomGroup: 4, atomRank: 1)
diff --git a/clang/test/KeyInstructions/ret-agg.cpp b/clang/test/KeyInstructions/ret-agg.cpp
new file mode 100644
index 0000000000000..0219a41f2608b
--- /dev/null
+++ b/clang/test/KeyInstructions/ret-agg.cpp
@@ -0,0 +1,23 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+struct S {
+ void* a;
+ void* b;
+};
+S get();
+void fun() {
+// CHECK: %0 = getelementptr inbounds { ptr, ptr }, ptr %s, i32 0, i32 0
+// CHECK: %1 = extractvalue { ptr, ptr } %call, 0, !dbg [[G1R2:!.*]]
+// CHECK: store ptr %1, ptr %0{{.*}}, !dbg [[G1R1:!.*]]
+// CHECK: %2 = getelementptr inbounds { ptr, ptr }, ptr %s, i32 0, i32 1
+// CHECK: %3 = extractvalue { ptr, ptr } %call, 1, !dbg [[G1R2]]
+// CHECK: store ptr %3, ptr %2{{.*}}, !dbg [[G1R1:!.*]]
+ S s = get();
+// CHECK: ret void, !dbg [[G2R1:!.*]]
+}
+
+// CHECK: [[G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
+
diff --git a/clang/test/KeyInstructions/return-no-loc.c b/clang/test/KeyInstructions/return-no-loc.c
new file mode 100644
index 0000000000000..32fc33618db18
--- /dev/null
+++ b/clang/test/KeyInstructions/return-no-loc.c
@@ -0,0 +1,26 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+// The implicit return here get the line number of the closing brace; make it
+// key to match existing behaviour.
+
+int a() {
+ if (a)
+ return 1;
+}
+
+// CHECK: br i1 true{{.*}}, !dbg [[G1R1:!.*]]
+
+// CHECK: if.then:
+// CHECK-NEXT: store i32 1, ptr %retval{{.*}}, !dbg [[G2R2:!.*]]
+// CHECK-NEXT: br label %if.end, !dbg [[G2R1:!.*]]
+
+// CHECK: if.end:
+// CHECK-NEXT: %0 = load i32, ptr %retval{{.*}}, !dbg [[G3R2:!.*]]
+// CHECK-NEXT: ret i32{{.*}}, !dbg [[G3R1:!.*]]
+
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R2]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 2)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
+// CHECK: [[G3R2]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 2)
+// CHECK: [[G3R1]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 1)
diff --git a/clang/test/KeyInstructions/return-ref.cpp b/clang/test/KeyInstructions/return-ref.cpp
new file mode 100644
index 0000000000000..d406cc6e036fd
--- /dev/null
+++ b/clang/test/KeyInstructions/return-ref.cpp
@@ -0,0 +1,19 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s
+
+// Include ctrl-flow to stop ret value store being elided.
+
+int g;
+int &f(int &r) {
+ if (r)
+// CHECK: if.then:
+// CHECK-NEXT: %2 = load ptr, ptr %r.addr{{.*}}, !dbg [[G2R3:!.*]]
+// CHECK-NEXT: store ptr %2, ptr %retval{{.*}}, !dbg [[G2R2:!.*]]
+// CHECK-NEXT: br label %return, !dbg [[G2R1:!.*]]
+ return r;
+ return g;
+}
+
+// CHECK: [[G2R3]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 3)
+// CHECK: [[G2R2]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 2)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
diff --git a/clang/test/KeyInstructions/return.c b/clang/test/KeyInstructions/return.c
new file mode 100644
index 0000000000000..676cf98144290
--- /dev/null
+++ b/clang/test/KeyInstructions/return.c
@@ -0,0 +1,36 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+int g;
+float a() {
+ if (a)
+ return g;
+ return 1;
+}
+
+// CHECK: entry:
+// CHECK: br i1 true{{.*}}, !dbg [[G1R1:!.*]]
+
+// CHECK: if.then:
+// CHECK-NEXT: %0 = load i32, ptr @g{{.*}}, !dbg [[G2R4:!.*]]
+// CHECK-NEXT: %conv = sitofp i32 %0 to float{{.*}}, !dbg [[G2R3:!.*]]
+// CHECK-NEXT: store float %conv, ptr %retval{{.*}}, !dbg [[G2R2:!.*]]
+// CHECK-NEXT: br label %return{{.*}}, !dbg [[G2R1:!.*]]
+
+// CHECK: if.end:
+// CHECK-NEXT: store float 1.000000e+00, ptr %retval{{.*}}, !dbg [[G3R2:!.*]]
+// CHECK-NEXT: br label %return, !dbg [[G3R1:!.*]]
+
+// CHECK: return:
+// CHECK-NEXT: %1 = load float, ptr %retval{{.*}}, !dbg [[G4R2:!.*]]
+// CHECK-NEXT: ret float %1{{.*}}, !dbg [[G4R1:!.*]]
+
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R4]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 4)
+// CHECK: [[G2R3]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 3)
+// CHECK: [[G2R2]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 2)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
+// CHECK: [[G3R2]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 2)
+// CHECK: [[G3R1]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 1)
+// CHECK: [[G4R2]] = !DILocation({{.*}}, atomGroup: 4, atomRank: 2)
+// CHECK: [[G4R1]] = !DILocation({{.*}}, atomGroup: 4, atomRank: 1)
diff --git a/clang/test/KeyInstructions/return.cpp b/clang/test/KeyInstructions/return.cpp
new file mode 100644
index 0000000000000..5d1806e6da141
--- /dev/null
+++ b/clang/test/KeyInstructions/return.cpp
@@ -0,0 +1,26 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+// NOTE: (return) (g = 1) are two separate atoms. FIXME: is that best?
+
+int g;
+
+// CHECK: _Z1av()
+// CHECK: ret void{{.*}}, !dbg [[A_G1R1:!.*]]
+void a() { return; }
+
+// CHECK: _Z1bv()
+// CHECK: %add = add{{.*}}, !dbg [[B_G1R2:!.*]]
+// CHECK: ret i32 %add{{.*}}, !dbg [[B_G1R1:!.*]]
+int b() { return g + 1; }
+
+// CHECK: _Z1cv()
+// CHECK: store{{.*}}, !dbg [[C_G2R1:!.*]]
+// CHECK: ret i32 1{{.*}}, !dbg [[C_G1R1:!.*]]
+int c() { return g = 1; }
+
+// CHECK: [[A_G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[B_G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[B_G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[C_G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
+// CHECK: [[C_G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
diff --git a/clang/test/KeyInstructions/scalar-init.cpp b/clang/test/KeyInstructions/scalar-init.cpp
new file mode 100644
index 0000000000000..b6f6b4b4b147c
--- /dev/null
+++ b/clang/test/KeyInstructions/scalar-init.cpp
@@ -0,0 +1,17 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+void a() {
+// CHECK: _Z1av()
+// CHECK: store i32 0, ptr %A{{.*}}, !dbg [[G1R1:!.*]]
+ int A = 0;
+// CHECK: %add = add {{.*}}, !dbg [[G2R2:!.*]]
+// CHECK: store i32 %add, ptr %B, align 4, !dbg [[G2R1:!.*]]
+ int B = 2 * A + 1;
+// CHECK: ret{{.*}}, !dbg [[G3R1:!.*]]
+}
+
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R2]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 2)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
+// CHECK: [[G3R1]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 1)
diff --git a/clang/test/KeyInstructions/static-init.cpp b/clang/test/KeyInstructions/static-init.cpp
new file mode 100644
index 0000000000000..b92621515db56
--- /dev/null
+++ b/clang/test/KeyInstructions/static-init.cpp
@@ -0,0 +1,13 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+void g(int *a) {
+// CHECK: %2 = load ptr, ptr %a.addr{{.*}}, !dbg [[G1R2:!.*]]
+// CHECK: store ptr %2, ptr @_ZZ1gPiE1b{{.*}}, !dbg [[G1R1:!.*]]
+ static int &b = *a;
+// CHECK: ret void, !dbg [[G2R2:!.*]]
+}
+
+// CHECK: [[G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R2]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
diff --git a/clang/test/KeyInstructions/switch.cpp b/clang/test/KeyInstructions/switch.cpp
new file mode 100644
index 0000000000000..1d02f3ed5ccda
--- /dev/null
+++ b/clang/test/KeyInstructions/switch.cpp
@@ -0,0 +1,48 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+int g;
+void a(int A, int B) {
+// CHECK: entry:
+// The load gets associated with the branch rather than the store.
+// FIXME: Is that the best thing to do?
+// CHECK: %0 = load i32, ptr %A.addr{{.*}}, !dbg [[G2R2:!.*]]
+// CHECK: store i32 %0, ptr @g{{.*}}, !dbg [[G1R1:!.*]]
+// CHECK: switch i32 %0, label %{{.*}} [
+// CHECK: i32 0, label %sw.bb
+// CHECK: i32 1, label %sw.bb1
+// CHECK: ], !dbg [[G2R1:!.*]]
+ switch ((g = A)) {
+ case 0: break;
+ case 1: {
+// CHECK: sw.bb1:
+// CHECK: %1 = load i32, ptr %B.addr{{.*}}, !dbg [[G3R2:!.*]]
+// CHECK: switch i32 %1, label %{{.*}} [
+// CHECK: i32 0, label %sw.bb2
+// CHECK: ], !dbg [[G3R1:!.*]]
+ switch ((B)) {
+ case 0: {
+// Test that assignments in constant-folded switches don't go missing.
+// CHECK: sw.bb2:
+// CHECK: store i32 1, ptr %C{{.*}}, !dbg [[G4R1:!.*]]
+ switch (const int C = 1; C) {
+ case 0: break;
+ case 1: break;
+ default: break;
+ }
+ } break;
+ default: break;
+ }
+ } break;
+ default: break;
+ }
+// CHECK: ret{{.*}}, !dbg [[G5R1:!.*]]
+}
+
+// CHECK: [[G2R2]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 2)
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
+// CHECK: [[G3R2]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 2)
+// CHECK: [[G3R1]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 1)
+// CHECK: [[G4R1]] = !DILocation({{.*}}, atomGroup: 4, atomRank: 1)
+// CHECK: [[G5R1]] = !DILocation({{.*}}, atomGroup: 5, atomRank: 1)
diff --git a/clang/test/KeyInstructions/try-catch.cpp b/clang/test/KeyInstructions/try-catch.cpp
new file mode 100644
index 0000000000000..881ffd85f07df
--- /dev/null
+++ b/clang/test/KeyInstructions/try-catch.cpp
@@ -0,0 +1,20 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -fexceptions -fcxx-exceptions \
+// RUN: | FileCheck %s
+
+void except() {
+ // FIXME(OCH): Should `store i32 32, ptr %exception` be key?
+ throw 32;
+}
+
+void attempt() {
+ try { except(); }
+// CHECK: catch:
+// CHECK: %4 = call ptr @__cxa_begin_catch(ptr %exn)
+// CHECK: %5 = load i32{{.*}}, !dbg [[G1R2:!.*]]
+// CHECK: store i32 %5, ptr %e{{.*}}, !dbg [[G1R1:!.*]]
+// CHECK: call void @__cxa_end_catch()
+ catch (int e) { }
+}
+
+// CHECK: [[G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
diff --git a/clang/test/KeyInstructions/while.cpp b/clang/test/KeyInstructions/while.cpp
new file mode 100644
index 0000000000000..55e8dffb80ee3
--- /dev/null
+++ b/clang/test/KeyInstructions/while.cpp
@@ -0,0 +1,20 @@
+// RUN: %clang %s -gmlt -gcolumn-info -S -emit-llvm -o - -Wno-unused-variable \
+// RUN: | FileCheck %s --implicit-check-not atomGroup --implicit-check-not atomRank
+
+// FIXME: Perennial quesiton: should the `dec` be in its own source atom or not
+// (currently it is).
+
+void a(int A) {
+// CHECK: %dec = add nsw i32 %0, -1, !dbg [[G1R2:!.*]]
+// CHECK: store i32 %dec, ptr %A.addr{{.*}}, !dbg [[G1R1:!.*]]
+// CHECK: %tobool = icmp ne i32 %dec, 0, !dbg [[G2R2:!.*]]
+// CHECK: br i1 %tobool, label %while.body, label %while.end, !dbg [[G2R1:!.*]]
+ while (--A) { };
+// CHECK: ret{{.*}}, !dbg [[G3R1:!.*]]
+}
+
+// CHECK: [[G1R2]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 2)
+// CHECK: [[G1R1]] = !DILocation({{.*}}, atomGroup: 1, atomRank: 1)
+// CHECK: [[G2R2]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 2)
+// CHECK: [[G2R1]] = !DILocation({{.*}}, atomGroup: 2, atomRank: 1)
+// CHECK: [[G3R1]] = !DILocation({{.*}}, atomGroup: 3, atomRank: 1)
diff --git a/llvm/include/llvm/IR/LLVMContext.h b/llvm/include/llvm/IR/LLVMContext.h
index bbd125fd38cf1..c3524a7815fa4 100644
--- a/llvm/include/llvm/IR/LLVMContext.h
+++ b/llvm/include/llvm/IR/LLVMContext.h
@@ -335,6 +335,10 @@ class LLVMContext {
StringRef getDefaultTargetFeatures();
void setDefaultTargetFeatures(StringRef Features);
+ /// Key Instructions: update the highest number Atom Group emitted for any
+ /// function.
+ void updateAtomGroupWaterline(uint64_t V);
+
private:
// Module needs access to the add/removeModule methods.
friend class Module;
diff --git a/llvm/lib/IR/LLVMContext.cpp b/llvm/lib/IR/LLVMContext.cpp
index 447e5d92e0b99..9abf52d438d94 100644
--- a/llvm/lib/IR/LLVMContext.cpp
+++ b/llvm/lib/IR/LLVMContext.cpp
@@ -377,3 +377,7 @@ StringRef LLVMContext::getDefaultTargetFeatures() {
void LLVMContext::setDefaultTargetFeatures(StringRef Features) {
pImpl->DefaultTargetFeatures = Features;
}
+
+void LLVMContext::updateAtomGroupWaterline(uint64_t V) {
+ pImpl->NextAtomGroup = std::max(pImpl->NextAtomGroup, V);
+}
\ No newline at end of file
More information about the cfe-commits
mailing list