[llvm] [llvm-profgen] Support Arm SPE and BRBE branch profiles (PR #209769)
Sergey Shcherbinin via llvm-commits
llvm-commits at lists.llvm.org
Wed Jul 15 07:22:44 PDT 2026
https://github.com/SergeyShch01 created https://github.com/llvm/llvm-project/pull/209769
Profile support:
- Read Arm SPE branch samples by reusing the LBR machinery and modeling each sample as a single-entry LBR stack.
- Process AArch64 BRBE branch stacks through the existing LBR path.
- Add AArch64 tests for SPE and BRBE profiles.
New options:
- --spe-br-prof selects Arm SPE input.
- --perf-binary selects the perf executable.
- --page-size and --use-system-page-size control profiling page size.
- --warn-not-symbolized reports symbolization failures only for addresses covered by recorded MMAP ranges.
Bug fixes:
- Preserve a zero-valued first PT_LOAD address.
- Report perf invocation failures and prevent stale redirected output.
These changes are part of the work to introduce the universal profile-guided loop data prefetcher described in:
https://discourse.llvm.org/t/rfc-universal-profile-guided-automatic-software-loop-data-prefetcher-for-llvm/90916
Assisted by GPT-5
>From 0a80322cc5d267598b8eba101768bab03f3cbad8 Mon Sep 17 00:00:00 2001
From: Sergey Shcherbinin <sscherbinin at nvidia.com>
Date: Tue, 14 Jul 2026 22:59:04 +0400
Subject: [PATCH] [llvm-profgen] Support Arm SPE and BRBE branch profiles
Profile support:
- Read Arm SPE branch samples by reusing the LBR machinery and modeling each
sample as a single-entry LBR stack.
- Process AArch64 BRBE branch stacks through the existing LBR path.
- Add AArch64 tests for SPE and BRBE profiles.
New options:
- --spe-br-prof selects Arm SPE input.
- --perf-binary selects the perf executable.
- --page-size and --use-system-page-size control profiling page size.
- --warn-not-symbolized reports symbolization failures only for addresses
covered by recorded MMAP ranges.
Bug fixes:
- Preserve a zero-valued first PT_LOAD address.
- Report perf invocation failures and prevent stale redirected output.
These changes are part of the work to introduce the universal profile-guided
loop data prefetcher described in:
https://discourse.llvm.org/t/rfc-universal-profile-guided-automatic-software-loop-data-prefetcher-for-llvm/90916
Assisted by GPT-5
---
llvm/docs/CommandGuide/llvm-profgen.rst | 37 +++-
.../AArch64/brbe-branch-profile.test | 57 +++++++
.../llvm-profgen/AArch64/brbe-perfdata.test | 127 ++++++++++++++
.../AArch64/first-loadable-address.test | 58 +++++++
.../tools/llvm-profgen/AArch64/page-size.test | 67 ++++++++
.../AArch64/perf-binary-error.test | 46 +++++
.../AArch64/spe-branch-profile.test | 74 ++++++++
.../llvm-profgen/AArch64/spe-perfdata.test | 84 +++++++++
.../llvm-profgen/AArch64/spe-range-gap.test | 66 ++++++++
.../AArch64/warn-not-symbolized-mmap.test | 118 +++++++++++++
llvm/tools/llvm-profgen/PerfReader.cpp | 159 ++++++++++++++++--
llvm/tools/llvm-profgen/PerfReader.h | 22 ++-
llvm/tools/llvm-profgen/ProfiledBinary.cpp | 46 ++++-
llvm/tools/llvm-profgen/ProfiledBinary.h | 50 +++++-
llvm/tools/llvm-profgen/llvm-profgen.cpp | 12 +-
15 files changed, 986 insertions(+), 37 deletions(-)
create mode 100644 llvm/test/tools/llvm-profgen/AArch64/brbe-branch-profile.test
create mode 100644 llvm/test/tools/llvm-profgen/AArch64/brbe-perfdata.test
create mode 100644 llvm/test/tools/llvm-profgen/AArch64/first-loadable-address.test
create mode 100644 llvm/test/tools/llvm-profgen/AArch64/page-size.test
create mode 100644 llvm/test/tools/llvm-profgen/AArch64/perf-binary-error.test
create mode 100644 llvm/test/tools/llvm-profgen/AArch64/spe-branch-profile.test
create mode 100644 llvm/test/tools/llvm-profgen/AArch64/spe-perfdata.test
create mode 100644 llvm/test/tools/llvm-profgen/AArch64/spe-range-gap.test
create mode 100644 llvm/test/tools/llvm-profgen/AArch64/warn-not-symbolized-mmap.test
diff --git a/llvm/docs/CommandGuide/llvm-profgen.rst b/llvm/docs/CommandGuide/llvm-profgen.rst
index 006cdfecf4f88..978aa3446ba44 100644
--- a/llvm/docs/CommandGuide/llvm-profgen.rst
+++ b/llvm/docs/CommandGuide/llvm-profgen.rst
@@ -21,8 +21,10 @@ At least one of the following commands are required:
.. option:: --perfscript=<string[,string,...]>
- Path of perf-script trace created by Linux perf tool with `script`
- command(the raw perf.data should be profiled with -b).
+ Path of a trace created by the Linux ``perf script`` command. For LBR or BRBE
+ input, the raw perf data must contain branch stacks, for example from
+ recording with ``-b``. With ``--spe-br-prof``, the trace must contain Arm SPE
+ branch samples.
.. option:: --etm=<string>
@@ -31,8 +33,9 @@ At least one of the following commands are required:
.. option:: --perfdata=<perfdata>, --pd
- Path of raw perf data created by Linux perf tool (it should be profiled
- with -b).
+ Path of raw perf data created by the Linux perf tool. For LBR or BRBE input,
+ it must contain branch stacks, for example from recording with ``-b``. With
+ ``--spe-br-prof``, it must contain Arm SPE branch samples.
.. option:: --unsymbolized-profile=<unsymbolized profile>, --up
@@ -71,6 +74,31 @@ OPTIONS
Print mmap events.
+.. option:: --spe-br-prof
+
+ Read the input as an Arm SPE branch profile. Requires perf 6.13 or later to
+ process all branch samples; older versions provide only mispredicted
+ branches.
+
+.. option:: --perf-binary=<filename>
+
+ Path to the ``perf`` executable used to convert ``--perfdata`` input. If this
+ option is omitted, ``llvm-profgen`` searches for ``perf`` in ``PATH``.
+
+.. option:: --page-size=<bytes>
+
+ Page size of the system on which the profile was recorded. The value must be
+ a power of two. The default is 4096 bytes.
+
+.. option:: --use-system-page-size
+
+ Use the page size of the system running ``llvm-profgen``. This option cannot
+ be used together with ``--page-size``.
+
+.. option:: --warn-not-symbolized
+
+ Warn when an address covered by a recorded mmap range cannot be symbolized.
+
.. option:: --show-disassembly
Print disassembled code.
@@ -102,4 +130,3 @@ OPTIONS
File path of a Linux perf raw trace (generated by ``perf report -D``)
consisting of memory access events.
-
diff --git a/llvm/test/tools/llvm-profgen/AArch64/brbe-branch-profile.test b/llvm/test/tools/llvm-profgen/AArch64/brbe-branch-profile.test
new file mode 100644
index 0000000000000..918e34f316efd
--- /dev/null
+++ b/llvm/test/tools/llvm-profgen/AArch64/brbe-branch-profile.test
@@ -0,0 +1,57 @@
+# RUN: split-file %s %t
+# RUN: yaml2obj %t/binary.yaml -o %t/brbe-branch-profile.exe
+# RUN: llvm-profgen --binary=%t/brbe-branch-profile.exe \
+# RUN: --perfscript=%t/perfscript --skip-symbolization --use-offset=0 \
+# RUN: --format=text --output=%t/profile
+# RUN: FileCheck %s --input-file=%t/profile
+
+## perf script represents Arm BRBE records using the same branch-stack format
+## as X86 LBR records. Verify that the generic LBR path handles an AArch64
+## binary, canonicalizes runtime addresses, and reconstructs the execution
+## range between consecutive branch records.
+# CHECK: 1
+# CHECK-NEXT: 1008-100c:1
+# CHECK-NEXT: 2
+# CHECK-NEXT: 1000->1008:1
+# CHECK-NEXT: 100c->1000:1
+
+#--- binary.yaml
+--- !ELF
+FileHeader:
+ Class: ELFCLASS64
+ Data: ELFDATA2LSB
+ Type: ET_DYN
+ Machine: EM_AARCH64
+ Entry: 0x1000
+Sections:
+ - Name: .text
+ Type: SHT_PROGBITS
+ Flags: [ SHF_ALLOC, SHF_EXECINSTR ]
+ Address: 0x1000
+ Offset: 0x1000
+ AddressAlign: 0x4
+ ## b 0x1008
+ ## nop
+ ## nop
+ ## b 0x1000
+ ## ret
+ Content: 020000141F2003D51F2003D5FDFFFF17C0035FD6
+ProgramHeaders:
+ - Type: PT_LOAD
+ Flags: [ PF_X, PF_R ]
+ Offset: 0
+ VAddr: 0
+ FileSize: 0x1014
+ MemSize: 0x1014
+ Align: 0x10000
+Symbols:
+ - Name: foo
+ Type: STT_FUNC
+ Section: .text
+ Binding: STB_GLOBAL
+ Value: 0x1000
+ Size: 0x14
+
+#--- perfscript
+PERF_RECORD_MMAP2 1/1: [0x700000000000(0x2000) @ 0 00:00 0 0]: r-xp /tmp/brbe-branch-profile.exe
+70000000100c 0x70000000100c/0x700000001000/P/-/-/0 0x700000001000/0x700000001008/P/-/-/0
diff --git a/llvm/test/tools/llvm-profgen/AArch64/brbe-perfdata.test b/llvm/test/tools/llvm-profgen/AArch64/brbe-perfdata.test
new file mode 100644
index 0000000000000..d510f73be058f
--- /dev/null
+++ b/llvm/test/tools/llvm-profgen/AArch64/brbe-perfdata.test
@@ -0,0 +1,127 @@
+# REQUIRES: system-linux
+
+# RUN: split-file %s %t
+# RUN: yaml2obj %t/binary.yaml -o %t/brbe-perfdata.exe
+# RUN: touch %t/perf.data
+# RUN: chmod +x %t/mock-perf %t/mock-perf-fail
+# RUN: llvm-profgen --binary=%t/brbe-perfdata.exe \
+# RUN: --perfdata=%t/perf.data --perf-binary=%t/mock-perf \
+# RUN: --skip-symbolization --use-offset=0 --format=text --output=%t/profile
+# RUN: FileCheck %s --check-prefix=PROFILE --input-file=%t/profile
+# RUN: not llvm-profgen --binary=%t/brbe-perfdata.exe \
+# RUN: --perfdata=%t/perf.data --perf-binary=%t/mock-perf-fail \
+# RUN: --output=/dev/null 2>&1 \
+# RUN: | FileCheck %s --check-prefix=SECOND-ERROR
+
+## Verify that the non-SPE perfdata path requests ip,brstack. The first mock
+## invocation deliberately writes the complete second output plus a duplicate
+## sample. Without truncating the shared redirect file, that stale sample
+## survives the second invocation and doubles every profile count.
+# PROFILE: 1
+# PROFILE-NEXT: 1008-100c:1
+# PROFILE-NEXT: 2
+# PROFILE-NEXT: 1000->1008:1
+# PROFILE-NEXT: 100c->1000:1
+
+## The first invocation also leaves a longer stderr message. Verify that an
+## error from the second invocation contains only its own stderr.
+# SECOND-ERROR: error: Perf script failed with exit code 7
+# SECOND-ERROR-NEXT: second invocation failed
+# SECOND-ERROR-NOT: STALE-FIRST-STDERR
+
+#--- mock-perf
+#!/bin/sh
+
+if [ "$1" != "script" ] || [ "$2" != "--show-mmap-events" ] || \
+ [ "$3" != "-F" ] || [ "$5" != "-i" ] || [ ! -f "$6" ]; then
+ echo "unexpected perf arguments: $*" >&2
+ exit 2
+fi
+
+emit_sample() {
+ printf '%s\n' 'PERF_RECORD_MMAP2 1/1: [0x700000000000(0x2000) @ 0 00:00 0 0]: r-xp /tmp/brbe-perfdata.exe'
+ printf '%s\n' '70000000100c 0x70000000100c/0x700000001000/P/-/-/0 0x700000001000/0x700000001008/P/-/-/0'
+}
+
+case "$4" in
+ comm,pid)
+ if [ "$#" -ne 6 ]; then
+ echo "unexpected mmap arguments: $*" >&2
+ exit 2
+ fi
+ emit_sample
+ printf '%s\n' '70000000100c 0x70000000100c/0x700000001000/P/-/-/0 0x700000001000/0x700000001008/P/-/-/0'
+ ;;
+ ip,brstack)
+ if [ "$#" -ne 8 ] || [ "$7" != "--pid" ] || [ "$8" != "1" ]; then
+ echo "unexpected sample arguments: $*" >&2
+ exit 2
+ fi
+ emit_sample
+ ;;
+ *)
+ echo "unexpected perf fields: $4" >&2
+ exit 2
+ ;;
+esac
+
+#--- mock-perf-fail
+#!/bin/sh
+
+if [ "$1" != "script" ] || [ "$2" != "--show-mmap-events" ] || \
+ [ "$3" != "-F" ] || [ "$5" != "-i" ] || [ ! -f "$6" ]; then
+ echo "unexpected perf arguments: $*" >&2
+ exit 2
+fi
+
+case "$4" in
+ comm,pid)
+ printf '%s\n' 'PERF_RECORD_MMAP2 1/1: [0x700000000000(0x2000) @ 0 00:00 0 0]: r-xp /tmp/brbe-perfdata.exe'
+ printf '%s\n' 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx STALE-FIRST-STDERR' >&2
+ ;;
+ ip,brstack)
+ printf '%s\n' 'second invocation failed' >&2
+ exit 7
+ ;;
+ *)
+ echo "unexpected perf fields: $4" >&2
+ exit 2
+ ;;
+esac
+
+#--- binary.yaml
+--- !ELF
+FileHeader:
+ Class: ELFCLASS64
+ Data: ELFDATA2LSB
+ Type: ET_DYN
+ Machine: EM_AARCH64
+ Entry: 0x1000
+Sections:
+ - Name: .text
+ Type: SHT_PROGBITS
+ Flags: [ SHF_ALLOC, SHF_EXECINSTR ]
+ Address: 0x1000
+ Offset: 0x1000
+ AddressAlign: 0x4
+ ## b 0x1008
+ ## nop
+ ## nop
+ ## b 0x1000
+ ## ret
+ Content: 020000141F2003D51F2003D5FDFFFF17C0035FD6
+ProgramHeaders:
+ - Type: PT_LOAD
+ Flags: [ PF_X, PF_R ]
+ Offset: 0
+ VAddr: 0
+ FileSize: 0x1014
+ MemSize: 0x1014
+ Align: 0x10000
+Symbols:
+ - Name: foo
+ Type: STT_FUNC
+ Section: .text
+ Binding: STB_GLOBAL
+ Value: 0x1000
+ Size: 0x14
diff --git a/llvm/test/tools/llvm-profgen/AArch64/first-loadable-address.test b/llvm/test/tools/llvm-profgen/AArch64/first-loadable-address.test
new file mode 100644
index 0000000000000..3264d81866b8b
--- /dev/null
+++ b/llvm/test/tools/llvm-profgen/AArch64/first-loadable-address.test
@@ -0,0 +1,58 @@
+# RUN: split-file %s %t
+# RUN: yaml2obj %t/binary.yaml -o %t/first-loadable-address.exe
+# RUN: llvm-profgen --binary=%t/first-loadable-address.exe \
+# RUN: --perfscript=%t/perfscript --skip-symbolization --format=text \
+# RUN: --page-size=65536 --use-offset=1 \
+# RUN: --use-first-loadable-segment-as-base=1 --output=%t/profile
+# RUN: FileCheck %s --input-file=%t/profile
+
+## The first PT_LOAD has a valid zero virtual address. Ensure it is not treated
+## as an unset value and overwritten by the following executable PT_LOAD.
+# CHECK: 0
+# CHECK-NEXT: 1
+# CHECK-NEXT: 11008->11000:1
+
+#--- binary.yaml
+--- !ELF
+FileHeader:
+ Class: ELFCLASS64
+ Data: ELFDATA2LSB
+ Type: ET_DYN
+ Machine: EM_AARCH64
+ Entry: 0x0000000000011000
+Sections:
+ - Name: .text
+ Type: SHT_PROGBITS
+ Flags: [ SHF_ALLOC, SHF_EXECINSTR ]
+ Address: 0x0000000000011000
+ Offset: 0x0000000000001000
+ AddressAlign: 0x4
+ ## nop
+ ## nop
+ ## ret
+ Content: 1F2003D51F2003D5C0035FD6
+ProgramHeaders:
+ - Type: PT_LOAD
+ Flags: [ PF_R ]
+ Offset: 0x0
+ VAddr: 0x0
+ FileSize: 0x1000
+ MemSize: 0x1000
+ Align: 0x10000
+ - Type: PT_LOAD
+ Flags: [ PF_X, PF_R ]
+ VAddr: 0x0000000000011000
+ Align: 0x10000
+ FirstSec: .text
+ LastSec: .text
+Symbols:
+ - Name: foo
+ Type: STT_FUNC
+ Section: .text
+ Binding: STB_GLOBAL
+ Value: 0x0000000000011000
+ Size: 0xC
+
+#--- perfscript
+PERF_RECORD_MMAP2 1/1: [0x700000000000(0x10000) @ 0 00:00 0 0]: r-xp /tmp/first-loadable-address.exe
+700000001000 0x700000001008/0x700000001000/P/-/-/0
diff --git a/llvm/test/tools/llvm-profgen/AArch64/page-size.test b/llvm/test/tools/llvm-profgen/AArch64/page-size.test
new file mode 100644
index 0000000000000..dba631c6ce5d4
--- /dev/null
+++ b/llvm/test/tools/llvm-profgen/AArch64/page-size.test
@@ -0,0 +1,67 @@
+# RUN: split-file %s %t
+# RUN: yaml2obj %t/binary.yaml -o %t/page-size.exe
+
+## A 64 KiB profiling system maps the executable segment from file offset zero.
+# RUN: llvm-profgen --binary=%t/page-size.exe --perfscript=%t/64k.perfscript \
+# RUN: --skip-symbolization --use-offset=0 --format=text --page-size=65536 \
+# RUN: --output=%t/explicit-64k.prof
+# RUN: FileCheck %s --check-prefix=PROFILE --input-file=%t/explicit-64k.prof
+
+# RUN: not llvm-profgen --binary=%t/page-size.exe --show-disassembly-only \
+# RUN: --page-size=3 --output=/dev/null 2>&1 \
+# RUN: | FileCheck %s --check-prefix=INVALID
+# RUN: not llvm-profgen --binary=%t/page-size.exe --show-disassembly-only \
+# RUN: --page-size=4096 --use-system-page-size --output=/dev/null 2>&1 \
+# RUN: | FileCheck %s --check-prefix=CONFLICT
+
+# PROFILE: 0
+# PROFILE-NEXT: 1
+# PROFILE-NEXT: 11008->11000:1
+
+# INVALID: error: --page-size must be a power of two
+# CONFLICT: error: --page-size and --use-system-page-size cannot be used together
+
+#--- binary.yaml
+--- !ELF
+FileHeader:
+ Class: ELFCLASS64
+ Data: ELFDATA2LSB
+ Type: ET_DYN
+ Machine: EM_AARCH64
+ Entry: 0x0000000000011000
+Sections:
+ - Name: .text
+ Type: SHT_PROGBITS
+ Flags: [ SHF_ALLOC, SHF_EXECINSTR ]
+ Address: 0x0000000000011000
+ Offset: 0x0000000000001000
+ AddressAlign: 0x4
+ ## nop
+ ## nop
+ ## ret
+ Content: 1F2003D51F2003D5C0035FD6
+ProgramHeaders:
+ - Type: PT_LOAD
+ Flags: [ PF_R ]
+ Offset: 0x0
+ VAddr: 0x0000000000010000
+ FileSize: 0x1000
+ MemSize: 0x1000
+ Align: 0x10000
+ - Type: PT_LOAD
+ Flags: [ PF_X, PF_R ]
+ VAddr: 0x0000000000011000
+ Align: 0x10000
+ FirstSec: .text
+ LastSec: .text
+Symbols:
+ - Name: foo
+ Type: STT_FUNC
+ Section: .text
+ Binding: STB_GLOBAL
+ Value: 0x0000000000011000
+ Size: 0xC
+
+#--- 64k.perfscript
+PERF_RECORD_MMAP2 1/1: [0x700000000000(0x10000) @ 0 00:00 0 0]: r-xp /tmp/page-size.exe
+700000001000 0x700000001008/0x700000001000/P/-/-/0
diff --git a/llvm/test/tools/llvm-profgen/AArch64/perf-binary-error.test b/llvm/test/tools/llvm-profgen/AArch64/perf-binary-error.test
new file mode 100644
index 0000000000000..5458f205a7bad
--- /dev/null
+++ b/llvm/test/tools/llvm-profgen/AArch64/perf-binary-error.test
@@ -0,0 +1,46 @@
+# RUN: split-file %s %t
+# RUN: yaml2obj %t/binary.yaml -o %t/perf-binary-error.exe
+# RUN: touch %t/perf.data
+
+# RUN: not llvm-profgen --binary=%t/perf-binary-error.exe \
+# RUN: --perfdata=%t/perf.data --perf-binary yaml2obj --output=/dev/null \
+# RUN: 2>&1 | FileCheck %s --check-prefix=EXIT
+# RUN: not llvm-profgen --binary=%t/perf-binary-error.exe \
+# RUN: --perfdata=%t/perf.data --perf-binary=%t/missing-perf \
+# RUN: --output=/dev/null 2>&1 | FileCheck %s --check-prefix=EXEC
+
+# EXIT: error: Perf script failed with exit code 1
+# EXIT: yaml2obj
+# EXEC: error: Failed to execute perf script
+
+#--- binary.yaml
+--- !ELF
+FileHeader:
+ Class: ELFCLASS64
+ Data: ELFDATA2LSB
+ Type: ET_DYN
+ Machine: EM_AARCH64
+Sections:
+ - Name: .text
+ Type: SHT_PROGBITS
+ Flags: [ SHF_ALLOC, SHF_EXECINSTR ]
+ Address: 0x1000
+ Offset: 0x1000
+ AddressAlign: 0x4
+ ## ret
+ Content: C0035FD6
+ProgramHeaders:
+ - Type: PT_LOAD
+ Flags: [ PF_X, PF_R ]
+ Offset: 0
+ VAddr: 0
+ Align: 0x1000
+ FirstSec: .text
+ LastSec: .text
+Symbols:
+ - Name: foo
+ Type: STT_FUNC
+ Section: .text
+ Binding: STB_GLOBAL
+ Value: 0x1000
+ Size: 0x4
diff --git a/llvm/test/tools/llvm-profgen/AArch64/spe-branch-profile.test b/llvm/test/tools/llvm-profgen/AArch64/spe-branch-profile.test
new file mode 100644
index 0000000000000..2c2477dcf41ca
--- /dev/null
+++ b/llvm/test/tools/llvm-profgen/AArch64/spe-branch-profile.test
@@ -0,0 +1,74 @@
+# RUN: split-file %s %t
+# RUN: yaml2obj %t/binary.yaml -o %t/spe-branch-profile.exe
+# RUN: llvm-profgen --binary=%t/spe-branch-profile.exe \
+# RUN: --perfscript=%t/perfscript --spe-br-prof --skip-symbolization \
+# RUN: --use-offset=0 --format=text --output=%t/profile 2>&1 \
+# RUN: | FileCheck --allow-empty %s --check-prefix=NO-WARN
+# RUN: FileCheck %s --input-file=%t/profile
+# RUN: llvm-profgen --binary=%t/spe-branch-profile.exe \
+# RUN: --perfscript=%t/malformed.perfscript --spe-br-prof \
+# RUN: --skip-symbolization --output=/dev/null 2>&1 \
+# RUN: | FileCheck %s --check-prefix=INVALID
+
+## Arm SPE prints each branch as <Dst> <Src>. Use different source and
+## destination addresses to verify their order, canonicalize them through the
+## runtime MMAP, and infer the executed range up to the nearest transfer. The
+## second record is a not-taken conditional branch whose destination is the
+## fall-through instruction rather than the statically encoded branch target.
+# NO-WARN-NOT: warning: {{.*}}branch targets do not match the binary
+
+# CHECK: 2
+# CHECK-NEXT: 1000-1008:1
+# CHECK-NEXT: 100c-100c:1
+# CHECK-NEXT: 2
+# CHECK-NEXT: 1008->1000:1
+# CHECK-NEXT: 1008->100c:1
+
+# INVALID: warning: Invalid Arm SPE branch record at line 2: 70000000100a
+# INVALID-NOT: warning: Invalid Arm SPE branch record
+
+#--- binary.yaml
+--- !ELF
+FileHeader:
+ Class: ELFCLASS64
+ Data: ELFDATA2LSB
+ Type: ET_DYN
+ Machine: EM_AARCH64
+ Entry: 0x1000
+Sections:
+ - Name: .text
+ Type: SHT_PROGBITS
+ Flags: [ SHF_ALLOC, SHF_EXECINSTR ]
+ Address: 0x1000
+ Offset: 0x1000
+ AddressAlign: 0x4
+ ## nop
+ ## nop
+ ## b.eq 0x1000
+ ## ret
+ Content: 1F2003D51F2003D5C0FFFF54C0035FD6
+ProgramHeaders:
+ - Type: PT_LOAD
+ Flags: [ PF_X, PF_R ]
+ Offset: 0
+ VAddr: 0
+ FileSize: 0x1010
+ MemSize: 0x1010
+ Align: 0x10000
+Symbols:
+ - Name: foo
+ Type: STT_FUNC
+ Section: .text
+ Binding: STB_GLOBAL
+ Value: 0x1000
+ Size: 0x10
+
+#--- perfscript
+PERF_RECORD_MMAP2 1/1: [0x700000000000(0x2000) @ 0 00:00 0 0]: r-xp /tmp/spe-branch-profile.exe
+700000001000 700000001008
+70000000100c 700000001008
+
+#--- malformed.perfscript
+PERF_RECORD_MMAP2 1/1: [0x700000000000(0x2000) @ 0 00:00 0 0]: r-xp /tmp/spe-branch-profile.exe
+70000000100a
+700000003000 700000003004
diff --git a/llvm/test/tools/llvm-profgen/AArch64/spe-perfdata.test b/llvm/test/tools/llvm-profgen/AArch64/spe-perfdata.test
new file mode 100644
index 0000000000000..32188d9f0afb5
--- /dev/null
+++ b/llvm/test/tools/llvm-profgen/AArch64/spe-perfdata.test
@@ -0,0 +1,84 @@
+# REQUIRES: system-linux
+
+# RUN: split-file %s %t
+# RUN: yaml2obj %t/binary.yaml -o %t/spe-perfdata.exe
+# RUN: touch %t/perf.data
+# RUN: chmod +x %t/mock-perf
+# RUN: llvm-profgen --binary=%t/spe-perfdata.exe \
+# RUN: --perfdata=%t/perf.data --perf-binary=%t/mock-perf --spe-br-prof \
+# RUN: --skip-symbolization --use-offset=0 --format=text --output=%t/profile
+# RUN: FileCheck %s --input-file=%t/profile
+
+## The mock validates both perf invocations, including the addr field required
+## for Arm SPE, while the profile checks that the second invocation's output
+## is consumed.
+# CHECK: 1
+# CHECK-NEXT: 1000-1008:1
+# CHECK-NEXT: 1
+# CHECK-NEXT: 1008->1000:1
+
+#--- mock-perf
+#!/bin/sh
+
+if [ "$1" != "script" ] || [ "$2" != "--show-mmap-events" ] || \
+ [ "$3" != "-F" ] || [ "$5" != "-i" ] || [ ! -f "$6" ]; then
+ echo "unexpected perf arguments: $*" >&2
+ exit 2
+fi
+
+case "$4" in
+ comm,pid)
+ if [ "$#" -ne 6 ]; then
+ echo "unexpected mmap arguments: $*" >&2
+ exit 2
+ fi
+ printf '%s\n' 'PERF_RECORD_MMAP2 1/1: [0x700000000000(0x2000) @ 0 00:00 0 0]: r-xp /tmp/spe-perfdata.exe'
+ ;;
+ ip,addr)
+ if [ "$#" -ne 8 ] || [ "$7" != "--pid" ] || [ "$8" != "1" ]; then
+ echo "unexpected sample arguments: $*" >&2
+ exit 2
+ fi
+ printf '%s\n' 'PERF_RECORD_MMAP2 1/1: [0x700000000000(0x2000) @ 0 00:00 0 0]: r-xp /tmp/spe-perfdata.exe'
+ printf '%s\n' '700000001000 700000001008'
+ ;;
+ *)
+ echo "unexpected perf fields: $4" >&2
+ exit 2
+ ;;
+esac
+
+#--- binary.yaml
+--- !ELF
+FileHeader:
+ Class: ELFCLASS64
+ Data: ELFDATA2LSB
+ Type: ET_DYN
+ Machine: EM_AARCH64
+ Entry: 0x1000
+Sections:
+ - Name: .text
+ Type: SHT_PROGBITS
+ Flags: [ SHF_ALLOC, SHF_EXECINSTR ]
+ Address: 0x1000
+ Offset: 0x1000
+ AddressAlign: 0x4
+ ## nop
+ ## nop
+ ## b 0x1000
+ Content: 1F2003D51F2003D5FEFFFF17
+ProgramHeaders:
+ - Type: PT_LOAD
+ Flags: [ PF_X, PF_R ]
+ Offset: 0
+ VAddr: 0
+ FileSize: 0x100C
+ MemSize: 0x100C
+ Align: 0x10000
+Symbols:
+ - Name: foo
+ Type: STT_FUNC
+ Section: .text
+ Binding: STB_GLOBAL
+ Value: 0x1000
+ Size: 0xC
diff --git a/llvm/test/tools/llvm-profgen/AArch64/spe-range-gap.test b/llvm/test/tools/llvm-profgen/AArch64/spe-range-gap.test
new file mode 100644
index 0000000000000..534e65c27c4e9
--- /dev/null
+++ b/llvm/test/tools/llvm-profgen/AArch64/spe-range-gap.test
@@ -0,0 +1,66 @@
+# RUN: split-file %s %t
+# RUN: yaml2obj %t/binary.yaml -o %t/spe-range-gap.exe
+# RUN: llvm-profgen --binary=%t/spe-range-gap.exe \
+# RUN: --perfscript=%t/perfscript --spe-br-prof --skip-symbolization \
+# RUN: --use-offset=0 --format=text --output=%t/profile
+# RUN: FileCheck %s --input-file=%t/profile
+
+## The decoded instruction addresses are 0x1000, 0x1004, and 0x2000. Do not
+## extend the inferred range across the address gap to the transfer at 0x2000.
+## The independently recorded branch count must remain unchanged.
+# CHECK: 1
+# CHECK-NEXT: 1000-1004:1
+# CHECK-NEXT: 1
+# CHECK-NEXT: 2000->1000:1
+
+#--- binary.yaml
+--- !ELF
+FileHeader:
+ Class: ELFCLASS64
+ Data: ELFDATA2LSB
+ Type: ET_DYN
+ Machine: EM_AARCH64
+ Entry: 0x1000
+Sections:
+ - Name: .text.first
+ Type: SHT_PROGBITS
+ Flags: [ SHF_ALLOC, SHF_EXECINSTR ]
+ Address: 0x1000
+ Offset: 0x1000
+ AddressAlign: 0x4
+ ## nop
+ ## nop
+ Content: 1F2003D51F2003D5
+ - Name: .text.second
+ Type: SHT_PROGBITS
+ Flags: [ SHF_ALLOC, SHF_EXECINSTR ]
+ Address: 0x2000
+ Offset: 0x2000
+ AddressAlign: 0x4
+ ## b 0x1000
+ Content: 00FCFF17
+ProgramHeaders:
+ - Type: PT_LOAD
+ Flags: [ PF_X, PF_R ]
+ Offset: 0
+ VAddr: 0
+ FileSize: 0x2004
+ MemSize: 0x2004
+ Align: 0x10000
+Symbols:
+ - Name: first
+ Type: STT_FUNC
+ Section: .text.first
+ Binding: STB_GLOBAL
+ Value: 0x1000
+ Size: 0x8
+ - Name: second
+ Type: STT_FUNC
+ Section: .text.second
+ Binding: STB_GLOBAL
+ Value: 0x2000
+ Size: 0x4
+
+#--- perfscript
+PERF_RECORD_MMAP2 1/1: [0x700000000000(0x3000) @ 0 00:00 0 0]: r-xp /tmp/spe-range-gap.exe
+700000001000 700000002000
diff --git a/llvm/test/tools/llvm-profgen/AArch64/warn-not-symbolized-mmap.test b/llvm/test/tools/llvm-profgen/AArch64/warn-not-symbolized-mmap.test
new file mode 100644
index 0000000000000..a6a39b7dae373
--- /dev/null
+++ b/llvm/test/tools/llvm-profgen/AArch64/warn-not-symbolized-mmap.test
@@ -0,0 +1,118 @@
+# RUN: split-file %s %t
+# RUN: yaml2obj %t/binary.yaml -o %t/warn-not-symbolized-mmap.exe
+
+## The LBR pair creates a valid code range inside without_debug, which
+## intentionally has no DWARF. The warning is emitted only when its runtime
+## address is covered by a recorded MMAP range.
+
+## BaseAddress is initially zero. Ensure that a first MMAP at address zero is
+## recorded before the same-address early return.
+# RUN: llvm-profgen --binary=%t/warn-not-symbolized-mmap.exe \
+# RUN: --perfscript=%t/zero.perfscript --warn-not-symbolized \
+# RUN: --output=/dev/null 2>&1 | FileCheck %s --check-prefix=ZERO
+
+## The second MMAP has the same non-zero address but extends the mapped range.
+## Ensure that it is recorded before the same-address early return.
+# RUN: llvm-profgen --binary=%t/warn-not-symbolized-mmap.exe \
+# RUN: --perfscript=%t/repeated.perfscript --warn-not-symbolized \
+# RUN: --output=/dev/null 2>&1 | FileCheck %s --check-prefix=REPEATED
+
+## The MMAP range is half-open. An address at Address + Size is not mapped and
+## must not produce a symbolization warning.
+# RUN: llvm-profgen --binary=%t/warn-not-symbolized-mmap.exe \
+# RUN: --perfscript=%t/outside.perfscript --warn-not-symbolized \
+# RUN: --output=/dev/null 2>&1 \
+# RUN: | FileCheck --allow-empty %s --check-prefix=OUTSIDE
+
+# ZERO: warning: Failed to symbolize address
+# ZERO-SAME: (vaddr= 284)
+# REPEATED: warning: Failed to symbolize address
+# REPEATED-SAME: (vaddr=700000000284)
+# OUTSIDE-NOT: warning: Failed to symbolize address
+
+#--- binary.yaml
+--- !ELF
+FileHeader:
+ Class: ELFCLASS64
+ Data: ELFDATA2LSB
+ Type: ET_DYN
+ Machine: EM_AARCH64
+ProgramHeaders:
+ - Type: PT_LOAD
+ Flags: [ PF_X, PF_R ]
+ VAddr: 0
+ Offset: 0
+ Align: 0x1000
+ FirstSec: .text
+ LastSec: .text
+Sections:
+ - Name: .text
+ Type: SHT_PROGBITS
+ Flags: [ SHF_ALLOC, SHF_EXECINSTR ]
+ Address: 0x27C
+ Offset: 0x27C
+ AddressAlign: 0x4
+ ## with_debug: add w0, w0, #1; ret
+ ## without_debug: add w0, w0, #2; ret
+ Content: 00040011C0035FD600080011C0035FD6
+ - Name: .debug_abbrev
+ Type: SHT_PROGBITS
+ AddressAlign: 0x1
+ Content: 011101252513050325721710171B25111B120673170000022E00111B1206401803253A0B3B0B49133F19000003240003253E0B0B0B000000
+ - Name: .debug_info
+ Type: SHT_PROGBITS
+ AddressAlign: 0x1
+ Content: 33000000050001080000000001001D0001080000000000000002000800000008000000020008000000016F030001320000000304050400
+ - Name: .debug_str_offsets
+ Type: SHT_PROGBITS
+ AddressAlign: 0x1
+ Content: 18000000050000000200000007000000000000001D00000003000000
+ - Name: .debug_line
+ Type: SHT_PROGBITS
+ AddressAlign: 0x1
+ Content: 460000000500080025000000010101FB0E0D00010101010000000100000101011F010000000002011F020F0102000000000400050C0A0009027C020000000000001305034B0204000101
+ - Name: .debug_line_str
+ Type: SHT_PROGBITS
+ Flags: [ SHF_MERGE, SHF_STRINGS ]
+ AddressAlign: 0x1
+ EntSize: 0x1
+ Content: 2F007761726E2D6E6F742D73796D626F6C697A65642E6300
+Symbols:
+ - Name: with_debug
+ Type: STT_FUNC
+ Section: .text
+ Binding: STB_GLOBAL
+ Value: 0x27C
+ Size: 0x8
+ - Name: without_debug
+ Type: STT_FUNC
+ Section: .text
+ Binding: STB_GLOBAL
+ Value: 0x284
+ Size: 0x8
+DWARF:
+ debug_str:
+ - '/'
+ - ''
+ - int
+ - warn-not-symbolized.c
+ - with_debug
+ debug_addr:
+ - Length: 0xC
+ Version: 0x5
+ AddressSize: 0x8
+ Entries:
+ - Address: 0x27C
+
+#--- zero.perfscript
+PERF_RECORD_MMAP2 1/1: [0x0(0x1000) @ 0 00:00 0 0]: r-xp /tmp/warn-not-symbolized-mmap.exe
+284 0x288/0x27c/P/-/-/0 0x280/0x284/P/-/-/0
+
+#--- repeated.perfscript
+PERF_RECORD_MMAP2 1/1: [0x700000000000(0x200) @ 0 00:00 0 0]: r-xp /tmp/warn-not-symbolized-mmap.exe
+PERF_RECORD_MMAP2 1/1: [0x700000000000(0x1000) @ 0 00:00 0 0]: r-xp /tmp/warn-not-symbolized-mmap.exe
+700000000284 0x700000000288/0x70000000027c/P/-/-/0 0x700000000280/0x700000000284/P/-/-/0
+
+#--- outside.perfscript
+PERF_RECORD_MMAP2 1/1: [0x700000000000(0x284) @ 0 00:00 0 0]: r-xp /tmp/warn-not-symbolized-mmap.exe
+700000000284 0x700000000288/0x70000000027c/P/-/-/0 0x700000000280/0x700000000284/P/-/-/0
diff --git a/llvm/tools/llvm-profgen/PerfReader.cpp b/llvm/tools/llvm-profgen/PerfReader.cpp
index 62b24a288dfc4..7aeb1fb70bd69 100644
--- a/llvm/tools/llvm-profgen/PerfReader.cpp
+++ b/llvm/tools/llvm-profgen/PerfReader.cpp
@@ -18,6 +18,7 @@
#include "llvm/Support/Process.h"
#include "llvm/Support/Timer.h"
#include "llvm/Support/ToolOutputFile.h"
+#include "llvm/Support/raw_ostream.h"
#include "llvm/TargetParser/Triple.h"
#define DEBUG_TYPE "perf-reader"
@@ -69,6 +70,19 @@ cl::opt<bool> TimeProfGen("time-profgen", cl::desc("Time llvm-profgen phases"),
static const char *TimerGroupName = "profgen";
static const char *TimerGroupDesc = "llvm-profgen";
+static cl::opt<bool>
+ ReadArmSPEBrProfile("spe-br-prof",
+ cl::desc("Read the input as an Arm SPE branch profile. "
+ "Requires perf 6.13 or later to process all "
+ "branch samples; older versions provide only "
+ "mispredicted branches."),
+ cl::cat(ProfGenCategory));
+
+static cl::opt<std::string> PerfPath("perf-binary",
+ cl::desc("Path to perf binary"),
+ cl::value_desc("filename"),
+ cl::cat(ProfGenCategory));
+
namespace sampleprof {
void VirtualUnwinder::unwindCall(UnwindState &State) {
@@ -363,11 +377,13 @@ PerfReaderBase::create(ProfiledBinary *Binary, InputFile &Input,
return PerfReader;
}
+ StringRef ExtraSampleFields = ReadArmSPEBrProfile ? "addr" : "brstack";
+
// For perf data input, we need to convert them into perf script first.
// If this is a kernel perf file, there is no need for retrieving PIDs.
if (Input.Format == InputFormat::PerfData)
- Input = PerfScriptReader::convertPerfDataToTrace(Binary, Binary->isKernel(),
- Input, PIDFilter);
+ Input = PerfScriptReader::convertPerfDataToTrace(
+ Binary, Binary->isKernel(), Input, PIDFilter, ExtraSampleFields);
assert((Input.Format == InputFormat::PerfScript) &&
"Should be a perfscript!");
@@ -377,7 +393,13 @@ PerfReaderBase::create(ProfiledBinary *Binary, InputFile &Input,
PerfReader.reset(
new HybridPerfReader(Binary, Input.InputFilePath, PIDFilter));
} else if (Input.Content == PerfContent::LBR) {
- PerfReader.reset(new LBRPerfReader(Binary, Input.InputFilePath, PIDFilter));
+ if (ReadArmSPEBrProfile) {
+ PerfReader.reset(
+ new ArmSPEReader(Binary, Input.InputFilePath, PIDFilter));
+ } else {
+ PerfReader.reset(
+ new LBRPerfReader(Binary, Input.InputFilePath, PIDFilter));
+ }
} else {
exitWithError("Unsupported perfscript!");
}
@@ -456,17 +478,18 @@ Error PerfReaderBase::parseDataAccessPerfTraces(
return Error::success();
}
-InputFile
-PerfScriptReader::convertPerfDataToTrace(ProfiledBinary *Binary, bool SkipPID,
- InputFile &File,
- std::optional<int32_t> PIDFilter) {
+InputFile PerfScriptReader::convertPerfDataToTrace(
+ ProfiledBinary *Binary, bool SkipPID, InputFile &File,
+ std::optional<int32_t> PIDFilter, StringRef ExtraSampleFields) {
StringRef PerfData = File.InputFilePath;
// Run perf script to retrieve PIDs matching binary we're interested in.
- auto PerfExecutable = sys::Process::FindInEnvPath("PATH", "perf");
+ auto PerfExecutable = PerfPath.getNumOccurrences()
+ ? PerfPath
+ : sys::Process::FindInEnvPath("PATH", "perf");
if (!PerfExecutable) {
exitWithError("Perf not found.");
}
- std::string PerfPath = *PerfExecutable;
+ std::string PerfExecutablePath = *PerfExecutable;
SmallString<128> PerfTraceFile;
sys::fs::createUniquePath("perf-script-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%.tmp",
PerfTraceFile, /*MakeAbsolute=*/true);
@@ -477,12 +500,53 @@ PerfScriptReader::convertPerfDataToTrace(ProfiledBinary *Binary, bool SkipPID,
PerfScriptReader::TempFileCleanups.emplace_back(PerfTraceFile);
PerfScriptReader::TempFileCleanups.emplace_back(ErrorFile);
+ auto RunPerfScript = [&](ArrayRef<StringRef> Args) {
+ // ExecuteAndWait does not truncate redirected output files on Unix. Remove
+ // both files so a shorter invocation cannot retain output from the
+ // previous perf script invocation.
+ for (StringRef Path : {StringRef(PerfTraceFile), StringRef(ErrorFile)}) {
+ if (std::error_code EC = sys::fs::remove(Path))
+ exitWithError(EC, Path);
+ }
+
+ std::string ExecutionError;
+ bool ExecutionFailed = false;
+ int ExitCode =
+ sys::ExecuteAndWait(PerfExecutablePath, Args, std::nullopt, Redirects,
+ /*SecondsToWait=*/0, /*MemoryLimit=*/0,
+ &ExecutionError, &ExecutionFailed);
+ if (!ExecutionFailed && ExitCode == 0)
+ return;
+
+ std::string Message;
+ raw_string_ostream OS(Message);
+ if (ExecutionFailed || ExitCode == -1)
+ OS << "Failed to execute perf script";
+ else if (ExitCode == -2)
+ OS << "Perf script terminated abnormally";
+ else
+ OS << "Perf script failed with exit code " << ExitCode;
+ if (!ExecutionError.empty())
+ OS << ": " << ExecutionError;
+
+ if (auto ErrorBuffer = MemoryBuffer::getFile(ErrorFile)) {
+ StringRef Stderr = ErrorBuffer.get()->getBuffer().trim();
+ if (!Stderr.empty())
+ OS << "\n" << Stderr;
+ }
+ exitWithError(OS.str());
+ };
+
std::string PIDs;
if (!SkipPID) {
- StringRef ScriptMMapArgs[] = {PerfPath, "script", "--show-mmap-events",
- "-F", "comm,pid", "-i",
+ StringRef ScriptMMapArgs[] = {PerfExecutablePath,
+ "script",
+ "--show-mmap-events",
+ "-F",
+ "comm,pid",
+ "-i",
PerfData};
- sys::ExecuteAndWait(PerfPath, ScriptMMapArgs, std::nullopt, Redirects);
+ RunPerfScript(ScriptMMapArgs);
// Collect the PIDs
TraceStream TraceIt(PerfTraceFile);
@@ -509,18 +573,19 @@ PerfScriptReader::convertPerfDataToTrace(ProfiledBinary *Binary, bool SkipPID,
// Run perf script again to retrieve events for PIDs collected above
SmallVector<StringRef, 8> ScriptSampleArgs;
- ScriptSampleArgs.push_back(PerfPath);
+ ScriptSampleArgs.push_back(PerfExecutablePath);
ScriptSampleArgs.push_back("script");
ScriptSampleArgs.push_back("--show-mmap-events");
ScriptSampleArgs.push_back("-F");
- ScriptSampleArgs.push_back("ip,brstack");
+ std::string FieldsStr = "ip," + ExtraSampleFields.str();
+ ScriptSampleArgs.push_back(FieldsStr);
ScriptSampleArgs.push_back("-i");
ScriptSampleArgs.push_back(PerfData);
if (!PIDs.empty()) {
ScriptSampleArgs.push_back("--pid");
ScriptSampleArgs.push_back(PIDs);
}
- sys::ExecuteAndWait(PerfPath, ScriptSampleArgs, std::nullopt, Redirects);
+ RunPerfScript(ScriptSampleArgs);
return {std::string(PerfTraceFile), InputFormat::PerfScript,
PerfContent::UnknownContent};
@@ -552,6 +617,8 @@ void PerfScriptReader::updateBinaryAddress(const MMapEvent &Event) {
if (PIDFilter && Event.PID != *PIDFilter)
return;
+ Binary->addMMapRange(Event.Address, Event.Size);
+
// Drop the event if its image is loaded at the same address
if (Event.Address == Binary->getBaseAddress()) {
Binary->setIsLoadedByMMap(true);
@@ -1024,6 +1091,11 @@ void PerfScriptReader::computeCounterFromLBR(const PerfSample *Sample,
Counter.recordBranchCount(SourceAddress, TargetAddress, Repeat);
}
+ // For ARM SPE find range on our own
+ if (ReadArmSPEBrProfile) {
+ EndAddress = Binary->findNearestTransfer(TargetAddress);
+ }
+
// If this not the first LBR, update the range count between TO of current
// LBR and FROM of next LBR.
uint64_t StartAddress = TargetAddress;
@@ -1045,6 +1117,44 @@ void LBRPerfReader::parseSample(TraceStream &TraceIt, uint64_t Count) {
}
}
+bool ArmSPEReader::extractLBRStack(TraceStream &TraceIt,
+ SmallVectorImpl<LBREntry> &LBRStack) {
+ uint64_t Src = 0, Dst = 0;
+ StringRef Line = TraceIt.getCurrentLine().ltrim();
+ bool Invalid = true;
+
+ // Format: <Dst> <Src>
+ if (!Line.consumeInteger(16, Dst)) {
+ Line = Line.ltrim();
+ Invalid = Line.consumeInteger(16, Src) || !Line.ltrim().empty();
+ }
+ if (Invalid) {
+ WithColor::warning() << "Invalid Arm SPE branch record at line "
+ << TraceIt.getLineNumber() << ": "
+ << TraceIt.getCurrentLine() << "\n";
+ TraceIt.advance();
+ return false;
+ }
+
+ // Canonicalize to use preferred load address as base address.
+ Src = Binary->canonicalizeVirtualAddress(Src);
+ Dst = Binary->canonicalizeVirtualAddress(Dst);
+ bool SrcIsInternal = Binary->addressIsCode(Src);
+ bool DstIsInternal = Binary->addressIsCode(Dst);
+ if (!SrcIsInternal)
+ Src = ExternalAddr;
+ if (!DstIsInternal)
+ Dst = ExternalAddr;
+
+ TraceIt.advance();
+ // Filter external-to-external branches without treating them as malformed.
+ if (!SrcIsInternal && !DstIsInternal)
+ return false;
+
+ LBRStack.emplace_back(LBREntry(Src, Dst));
+ return true;
+}
+
void PerfScriptReader::generateUnsymbolizedProfile() {
// There is no context for LBR only sample, so initialize one entry with
// fake "empty" context key.
@@ -1211,6 +1321,8 @@ bool PerfScriptReader::isMMapEvent(StringRef Line) {
// checking whether there is a non-empty call stack immediately followed by
// a LBR sample
PerfContent PerfScriptReader::checkPerfScriptType(StringRef FileName) {
+ if (ReadArmSPEBrProfile)
+ return PerfContent::LBR;
TraceStream TraceIt(FileName);
uint64_t FrameAddr = 0;
while (!TraceIt.isAtEoF()) {
@@ -1264,6 +1376,8 @@ void PerfScriptReader::warnTruncatedStack() {
}
void PerfScriptReader::warnInvalidRange() {
+ if (ReadArmSPEBrProfile)
+ return;
DenseMap<std::pair<uint64_t, uint64_t>, uint64_t> Ranges;
for (const auto &Item : AggregatedSamples) {
@@ -1393,9 +1507,18 @@ void PerfScriptReader::warnIfBranchTargetMismatch() {
if (Binary->addressIsIndirectBranch(Source)) {
if (!Binary->addressIsCode(Target))
MismatchedIndirectTargets++;
- } else if (!Binary->addressIsBranchTarget(Target) &&
- !Binary->findFuncRangeForStartAddr(Target))
- MismatchedTargets++;
+ } else {
+ // Arm SPE also records not-taken conditional branches. Their executed
+ // target is the fall-through instruction rather than the statically
+ // encoded branch target.
+ uint64_t InstSize = Binary->getInstSize(Source);
+ bool IsSPEFallThrough = ReadArmSPEBrProfile && InstSize &&
+ Binary->addressIsTransfer(Source) &&
+ Target > Source && Target - Source == InstSize;
+ if (!Binary->addressIsBranchTarget(Target) &&
+ !Binary->findFuncRangeForStartAddr(Target) && !IsSPEFallThrough)
+ MismatchedTargets++;
+ }
}
}
diff --git a/llvm/tools/llvm-profgen/PerfReader.h b/llvm/tools/llvm-profgen/PerfReader.h
index b9af0f19cb5d3..68419a38bbd6a 100644
--- a/llvm/tools/llvm-profgen/PerfReader.h
+++ b/llvm/tools/llvm-profgen/PerfReader.h
@@ -622,7 +622,8 @@ class PerfScriptReader : public PerfReaderBase {
// Generate perf script from perf data
static InputFile convertPerfDataToTrace(ProfiledBinary *Binary, bool SkipPID,
InputFile &File,
- std::optional<int32_t> PIDFilter);
+ std::optional<int32_t> PIDFilter,
+ StringRef ExtraSampleFields);
// Extract perf script type by peaking at the input
static PerfContent checkPerfScriptType(StringRef FileName);
@@ -656,8 +657,8 @@ class PerfScriptReader : public PerfReaderBase {
bool extractCallstack(TraceStream &TraceIt,
SmallVectorImpl<uint64_t> &CallStack);
// Extract LBR stack from one perf trace line
- bool extractLBRStack(TraceStream &TraceIt,
- SmallVectorImpl<LBREntry> &LBRStack);
+ virtual bool extractLBRStack(TraceStream &TraceIt,
+ SmallVectorImpl<LBREntry> &LBRStack);
uint64_t parseAggregatedCount(TraceStream &TraceIt);
// Parse one sample from multiple perf lines, override this for different
// sample type
@@ -765,6 +766,21 @@ class ETMReader {
ContextSampleCounterMap Counters;
};
+/*
+ The reader of ARM SPE perf script to fill branch profile data.
+*/
+class ArmSPEReader : public LBRPerfReader {
+public:
+ ArmSPEReader(ProfiledBinary *Binary, StringRef PerfTrace,
+ std::optional<int32_t> PID)
+ : LBRPerfReader(Binary, PerfTrace, PID) {}
+
+protected:
+ // Extract LBR stack with one entry from SPE perf trace line
+ bool extractLBRStack(TraceStream &TraceIt,
+ SmallVectorImpl<LBREntry> &LBRStack) override;
+};
+
} // end namespace sampleprof
} // end namespace llvm
diff --git a/llvm/tools/llvm-profgen/ProfiledBinary.cpp b/llvm/tools/llvm-profgen/ProfiledBinary.cpp
index 08fd6917c9f3a..e266f1941ce7f 100644
--- a/llvm/tools/llvm-profgen/ProfiledBinary.cpp
+++ b/llvm/tools/llvm-profgen/ProfiledBinary.cpp
@@ -24,6 +24,7 @@
#include "llvm/Support/CommandLine.h"
#include "llvm/Support/Debug.h"
#include "llvm/Support/Format.h"
+#include "llvm/Support/Process.h"
#include "llvm/Support/TargetSelect.h"
#include "llvm/TargetParser/Triple.h"
#include <optional>
@@ -81,6 +82,20 @@ static cl::opt<bool>
cl::desc("Generate the profile for Linux kernel binary."),
cl::cat(ProfGenCategory));
+static cl::opt<uint32_t>
+ ProfPageSize("page-size", cl::desc("Page size of a profiling system"),
+ cl::init(0x1000), cl::cat(ProfGenCategory));
+
+static cl::opt<bool> UseSystemPageSize(
+ "use-system-page-size",
+ cl::desc("Use the page size of the system running llvm-profgen"),
+ cl::cat(ProfGenCategory));
+
+static cl::opt<bool> WarnNotSymbolized(
+ "warn-not-symbolized", cl::init(false),
+ cl::desc("Generate warnings for not symbolized addresses"),
+ cl::cat(ProfGenCategory));
+
namespace sampleprof {
static const Target *getTarget(const ObjectFile *Obj) {
@@ -362,17 +377,28 @@ template <class ELFT>
void ProfiledBinary::setPreferredTextSegmentAddresses(const ELFFile<ELFT> &Obj,
StringRef FileName) {
const auto &PhdrRange = unwrapOrError(Obj.program_headers(), FileName);
- // FIXME: This should be the page size of the system running profiling.
- // However such info isn't available at post-processing time, assuming
- // 4K page now. Note that we don't use EXEC_PAGESIZE from <linux/param.h>
- // because we may build the tools on non-linux.
- uint64_t PageSize = 0x1000;
+ // FIXME: Read the page size from the perf.data header when available.
+ // Keep the default independent of the post-processing host so profiles can
+ // be processed on a system with a different page size.
+ // Note that we don't use EXEC_PAGESIZE from <linux/param.h> because we may
+ // build the tools on non-linux.
+ if (UseSystemPageSize && ProfPageSize.getNumOccurrences())
+ exitWithError(
+ "--page-size and --use-system-page-size cannot be used together");
+ uint64_t PageSize = UseSystemPageSize
+ ? llvm::sys::Process::getPageSizeEstimate()
+ : ProfPageSize;
+ if (!isPowerOf2_64(PageSize))
+ exitWithError("--page-size must be a power of two");
+ bool SeenFirstLoadableSegment = false;
for (const typename ELFT::Phdr &Phdr : PhdrRange) {
if (Phdr.p_type == ELF::PT_INTERP)
HasInterp = true;
if (Phdr.p_type == ELF::PT_LOAD) {
- if (!FirstLoadableAddress)
+ if (!SeenFirstLoadableSegment) {
FirstLoadableAddress = Phdr.p_vaddr & ~(PageSize - 1U);
+ SeenFirstLoadableSegment = true;
+ }
if (Phdr.p_flags & ELF::PF_X) {
// Segments will always be loaded at a page boundary.
PreferredTextSegmentAddresses.push_back(Phdr.p_vaddr &
@@ -1179,6 +1205,14 @@ SampleContextFrameVector ProfiledBinary::symbolize(const InstructionPointer &IP,
CallStack.emplace_back(FunctionId(It.first->getKey()), Line);
}
+ if (WarnNotSymbolized && CallStack.empty()) {
+ uint64_t VAddr = IP.Address + BaseAddress - getPreferredBaseAddress();
+ if (isVaddrMMapped(VAddr))
+ WithColor::warning() << "Failed to symbolize address "
+ << format("%8" PRIx64, IP.Address)
+ << " (vaddr=" << format("%8" PRIx64, VAddr) << ")\n";
+ }
+
return CallStack;
}
diff --git a/llvm/tools/llvm-profgen/ProfiledBinary.h b/llvm/tools/llvm-profgen/ProfiledBinary.h
index d1dea857f3043..e4b0480cf439f 100644
--- a/llvm/tools/llvm-profgen/ProfiledBinary.h
+++ b/llvm/tools/llvm-profgen/ProfiledBinary.h
@@ -201,6 +201,12 @@ struct MMapEvent {
StringRef BinaryPath;
};
+// Runtime address range covered by an MMAP event for the profiled binary.
+struct MMapRange {
+ uint64_t Address = 0;
+ uint64_t Size = 0;
+};
+
class ProfiledBinary {
// The executable binary file.
object::OwningBinary<object::Binary> OBinary;
@@ -314,6 +320,9 @@ class ProfiledBinary {
// MMap events for PT_LOAD segments without 'x' memory protection flag.
std::map<uint64_t, MMapEvent, std::greater<uint64_t>> NonTextMMapEvents;
+ // Address ranges mapped for the profiled binary.
+ SmallVector<MMapRange, 8> MMapRanges;
+
// Records the file offset, file size and virtual address of program headers.
struct PhdrInfo {
uint64_t FileOffset;
@@ -499,7 +508,7 @@ class ProfiledBinary {
bool addressIsIndirectBranch(uint64_t Address) const {
return IndirectBranchAddressSet.count(Address);
}
- bool addressIsTransfer(uint64_t Address) {
+ bool addressIsTransfer(uint64_t Address) const {
return BranchAddressSet.count(Address) || RetAddressSet.count(Address) ||
CallAddressSet.count(Address);
}
@@ -546,6 +555,32 @@ class ProfiledBinary {
return 0;
}
+ /// Find the address of the nearest control transfer instruction at or after
+ /// the given address. Returns the address itself if it's already a transfer.
+ /// Returns the last valid code address if no transfer is found before the
+ /// current contiguous code range ends.
+ /// Returns 0 if StartAddr is not valid code.
+ uint64_t findNearestTransfer(uint64_t StartAddr) const {
+ if (!addressIsCode(StartAddr))
+ return 0;
+
+ InstructionPointer IP(this, StartAddr);
+ uint64_t LastValidAddr = IP.Address;
+ for (;;) {
+ if (addressIsTransfer(IP.Address))
+ return IP.Address;
+
+ LastValidAddr = IP.Address;
+ uint64_t Size = getInstSize(IP.Address);
+ if (!Size || !IP.advance())
+ return LastValidAddr;
+
+ // Do not invent a range across a gap in CodeAddressVec.
+ if (IP.Address <= LastValidAddr || IP.Address - LastValidAddr != Size)
+ return LastValidAddr;
+ }
+ }
+
FuncRange *findFuncRangeForStartAddr(uint64_t Address) {
auto I = StartAddrToFuncRangeMap.find(Address);
if (I == StartAddrToFuncRangeMap.end())
@@ -728,6 +763,19 @@ class ProfiledBinary {
return Error::success();
}
+ void addMMapRange(uint64_t Address, uint64_t Size) {
+ MMapRanges.push_back({Address, Size});
+ }
+
+ // Check if a given virtual address is covered by any of the mmap ranges for
+ // the profiled binary.
+ bool isVaddrMMapped(uint64_t VAddr) const {
+ for (const MMapRange &Range : MMapRanges)
+ if (Range.Address <= VAddr && VAddr - Range.Address < Range.Size)
+ return true;
+ return false;
+ }
+
// Given a non-text runtime address, canonicalize it to the virtual address in
// the binary.
// TODO: Consider unifying the canonicalization of text and non-text addresses
diff --git a/llvm/tools/llvm-profgen/llvm-profgen.cpp b/llvm/tools/llvm-profgen/llvm-profgen.cpp
index cba5f1cd3d878..79a7987a4d623 100644
--- a/llvm/tools/llvm-profgen/llvm-profgen.cpp
+++ b/llvm/tools/llvm-profgen/llvm-profgen.cpp
@@ -31,8 +31,10 @@ cl::OptionCategory ProfGenCategory("ProfGen Options");
static cl::opt<std::string> PerfScriptFilename(
"perfscript", cl::value_desc("perfscript"),
- cl::desc("Path of perf-script trace created by Linux perf tool with "
- "`script` command(the raw perf.data should be profiled with -b). "
+ cl::desc("Path of a trace created by the Linux `perf script` command. For "
+ "LBR or BRBE input, the raw perf data must contain branch "
+ "stacks, for example from recording with -b. With "
+ "--spe-br-prof, the trace must contain Arm SPE branch samples. "
"Cannot be used with --perfdata, --unsymbolized-profile, or "
"--llvm-sample-profile."),
cl::cat(ProfGenCategory));
@@ -41,8 +43,10 @@ static cl::alias PSA("ps", cl::desc("Alias for --perfscript"),
static cl::opt<std::string> PerfDataFilename(
"perfdata", cl::value_desc("perfdata"),
- cl::desc("Path of raw perf data created by Linux perf tool (it should be "
- "profiled with -b). Cannot be used with --perfscript, "
+ cl::desc("Path of raw perf data created by the Linux perf tool. For LBR or "
+ "BRBE input, it must contain branch stacks, for example from "
+ "recording with -b. With --spe-br-prof, it must contain Arm SPE "
+ "branch samples. Cannot be used with --perfscript, "
"--unsymbolized-profile, or --llvm-sample-profile."),
cl::cat(ProfGenCategory));
static cl::alias PDA("pd", cl::desc("Alias for --perfdata"),
More information about the llvm-commits
mailing list