[all-commits] [llvm/llvm-project] 0474e6: [IR] Define GPU wave-profile metadata

[Yaxun (Sam) Liu via All-commits]

  Branch: refs/heads/users/yxsamliu/pgo-wave-metadata
  Home:   https://github.com/llvm/llvm-project
  Commit: 0474e66be18f3de0a85ab8b24eac824c87da5c1a
      https://github.com/llvm/llvm-project/commit/0474e66be18f3de0a85ab8b24eac824c87da5c1a
  Author: Yaxun (Sam) Liu <yaxun.liu at amd.com>
  Date:   2026-09-23 (Wed, 23 Sep 2026)

  Changed paths:
    M llvm/docs/LangRef.md
    M llvm/include/llvm/IR/FixedMetadataKinds.def
    M llvm/include/llvm/IR/ProfDataUtils.h
    M llvm/lib/IR/Instructions.cpp
    M llvm/lib/IR/ProfDataUtils.cpp
    M llvm/lib/IR/Verifier.cpp
    A llvm/test/Bitcode/wave-profile.ll
    A llvm/test/Transforms/InstCombine/wave-profile.ll
    A llvm/test/Verifier/wave-profile.ll
    M llvm/unittests/IR/CMakeLists.txt
    A llvm/unittests/IR/ProfDataUtilsTest.cpp

  Log Message:
  -----------
  [IR] Define GPU wave-profile metadata

Existing offload GPU profile counters measure lane executions, while
GPU instructions execute at wave granularity under an active-lane
mask. A block visited by every wave can therefore look cold when only
a few lanes are active. Scalar branch weights also cannot represent a
divergent wave visiting both successors before reconverging. These
profiles are a poor fit for optimizations that estimate work performed
by a wave.

Lane counters remain useful for measuring per-lane branch selectivity
and estimating work that scales with the number of active lanes.
Wave counters cannot replace them: a visit with one active lane and a
visit with all lanes active both count as one. The two profiles provide
complementary information about instruction execution and lane activity.

Introduce wave.profile and wave.profile.block metadata to represent
measured wave visits to IR blocks. Each dynamic visit with at least one
active lane contributes one to the count. A measured zero is distinct
from an unavailable count. Wave counts do not obey scalar flow
conservation: one wave may visit both sides of a branch and reconverge
once. The original entry count provides a stable normalization point.
These counts are profitability hints, not proofs of uniform execution
or transformation legality.

Spill placement and register-allocation heuristics can use wave counts
to estimate how often spill and reload instructions execute.
Code-motion and if-conversion cost models can also use them to compare
work across divergent paths, while retaining lane counts for costs
that depend on lane activity.

Store counts in a function-level table, with block identities and
recorded successors attached to terminators. Add verification and
shared helpers to read, write and preserve the metadata, rejecting
stale or ambiguous mappings. Optimizer integration is left to
subsequent changes.



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications