[all-commits] [llvm/llvm-project] f3c445: [CSSPGO] IR intrinsic for pseudo-probe block instr...

[Hongtao Yu via All-commits]

  Branch: refs/heads/master
  Home:   https://github.com/llvm/llvm-project
  Commit: f3c445697d23cfc1f09f86e66409d4ff69359732
      https://github.com/llvm/llvm-project/commit/f3c445697d23cfc1f09f86e66409d4ff69359732
  Author: Hongtao Yu <hoy at fb.com>
  Date:   2020-11-20 (Fri, 20 Nov 2020)

  Changed paths:
    M llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
    M llvm/include/llvm/CodeGen/BasicTTIImpl.h
    M llvm/include/llvm/IR/BasicBlock.h
    M llvm/include/llvm/IR/Instruction.h
    M llvm/include/llvm/IR/IntrinsicInst.h
    M llvm/include/llvm/IR/Intrinsics.td
    M llvm/lib/Analysis/AliasSetTracker.cpp
    M llvm/lib/Analysis/InlineCost.cpp
    M llvm/lib/Analysis/ValueTracking.cpp
    M llvm/lib/Analysis/VectorUtils.cpp
    M llvm/lib/CodeGen/Analysis.cpp
    M llvm/lib/CodeGen/CodeGenPrepare.cpp
    M llvm/lib/IR/BasicBlock.cpp
    M llvm/lib/IR/Instruction.cpp
    M llvm/lib/Transforms/Scalar/JumpThreading.cpp
    M llvm/lib/Transforms/Scalar/TailRecursionElimination.cpp
    M llvm/lib/Transforms/Utils/Evaluator.cpp
    M llvm/lib/Transforms/Utils/SimplifyCFG.cpp
    M llvm/lib/Transforms/Vectorize/LoadStoreVectorizer.cpp
    M llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
    M llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

  Log Message:
  -----------
  [CSSPGO] IR intrinsic for pseudo-probe block instrumentation

This change introduces a new IR intrinsic named `llvm.pseudoprobe` for pseudo-probe block instrumentation. Please refer to https://reviews.llvm.org/D86193 for the whole story.

A pseudo probe is used to collect the execution count of the block where the probe is instrumented. This requires a pseudo probe to be persisting. The LLVM PGO instrumentation also instruments in similar places by placing a counter in the form of atomic read/write operations or runtime helper calls. While these operations are very persisting or optimization-resilient, in theory we can borrow the atomic read/write implementation from PGO counters and cut it off at the end of compilation with all the atomics converted into binary data. This was our initial design and we’ve seen promising sample correlation quality with it. However, the atomics approach has a couple issues:

1. IR Optimizations are blocked unexpectedly. Those atomic instructions are not going to be physically present in the binary code, but since they are on the IR till very end of compilation, they can still prevent certain IR optimizations and result in lower code quality.
2. The counter atomics may not be fully cleaned up from the code stream eventually.
3. Extra work is needed for re-targeting.

We choose to implement pseudo probes based on a special LLVM intrinsic, which is expected to have most of the semantics that comes with an atomic operation but does not block desired optimizations as much as possible. More specifically the semantics associated with the new intrinsic enforces a pseudo probe to be virtually executed exactly the same number of times before and after an IR optimization. The intrinsic also comes with certain flags that are carefully chosen so that the places they are probing are not going to be messed up by the optimizer while most of the IR optimizations still work. The core flags given to the special intrinsic is `IntrInaccessibleMemOnly`, which means the intrinsic accesses memory and does have a side effect so that it is not removable, but is does not access memory locations that are accessible by any original instructions. This way the intrinsic does not alias with any original instruction and thus it does not block optimizations as much as an atomic operation does. We also assign a function GUID and a block index to an intrinsic so that they are uniquely identified and not merged in order to achieve good correlation quality.

Let's now look at an example. Given the following LLVM IR:

```
define internal void @foo2(i32 %x, void (i32)* %f) !dbg !4 {
bb0:
  %cmp = icmp eq i32 %x, 0
   br i1 %cmp, label %bb1, label %bb2
bb1:
   br label %bb3
bb2:
   br label %bb3
bb3:
   ret void
}
```

The instrumented IR will look like below. Note that each `llvm.pseudoprobe` intrinsic call represents a pseudo probe at a block, of which the first parameter is the GUID of the probe’s owner function and the second parameter is the probe’s ID.

```
define internal void @foo2(i32 %x, void (i32)* %f) !dbg !4 {
bb0:
   %cmp = icmp eq i32 %x, 0
   call void @llvm.pseudoprobe(i64 837061429793323041, i64 1)
   br i1 %cmp, label %bb1, label %bb2
bb1:
   call void @llvm.pseudoprobe(i64 837061429793323041, i64 2)
   br label %bb3
bb2:
   call void @llvm.pseudoprobe(i64 837061429793323041, i64 3)
   br label %bb3
bb3:
   call void @llvm.pseudoprobe(i64 837061429793323041, i64 4)
   ret void
}

```

Reviewed By: wmi

Differential Revision: https://reviews.llvm.org/D86490