[llvm] [offload] Fix kernel record/replay and add extensible mechanism (PR #190588)

Wed Apr 15 18:27:11 PDT 2026

llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-ubuntu-22-cmake-build-only` running on `rocm-docker-ubu-22` while building `offload` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/203/builds/43604

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
[21/56] Generating OffloadEntryPoints.inc
[22/56] Linking CXX shared library /home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/build/lib/clang/23/lib/x86_64-unknown-linux-gnu/libclang_rt.memprof.so
[23/56] Linking CXX shared library /home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/build/lib/clang/23/lib/x86_64-unknown-linux-gnu/libclang_rt.asan.so
[24/56] Generating /home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/build/compile_commands.json
[25/56] Building CXX object offload/plugins-nextgen/amdgpu/CMakeFiles/omptarget.rtl.amdgpu.dir/dynamic_hsa/hsa.cpp.o
[26/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/OpenMP/OMPT/Callback.cpp.o
[27/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/DeviceImage.cpp.o
[28/56] Building CXX object offload/tools/kernelreplay/CMakeFiles/llvm-omp-kernel-replay.dir/llvm-omp-kernel-replay.cpp.o
[29/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/LegacyAPI.cpp.o
[30/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/omptarget.cpp.o
FAILED: offload/libomptarget/CMakeFiles/omptarget.dir/omptarget.cpp.o 
/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/build/./bin/clang++ --target=x86_64-unknown-linux-gnu -DDEBUG_PREFIX=\"omptarget\" -DOMPT_SUPPORT=1 -DTARGET_NAME=omptarget -D_DEBUG -D_GLIBCXX_ASSERTIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/llvm-project/llvm/include -I/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/build/include -I/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/build/runtimes/runtimes-bins/openmp/runtime/src -I/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/llvm-project/offload/include -I/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/build/runtimes/runtimes-bins/offload/include -I/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/llvm-project/offload/plugins-nextgen/common/include -I/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/build/runtimes/runtimes-bins/offload/plugins-nextgen/common/include -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Xclang -fno-pch-timestamp -O3 -DNDEBUG -fPIC -UNDEBUG -fno-exceptions -funwind-tables -fno-rtti -Werror=global-constructors -MD -MT offload/libomptarget/CMakeFiles/omptarget.dir/omptarget.cpp.o -MF offload/libomptarget/CMakeFiles/omptarget.dir/omptarget.cpp.o.d -o offload/libomptarget/CMakeFiles/omptarget.dir/omptarget.cpp.o -c /home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/llvm-project/offload/libomptarget/omptarget.cpp
/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/llvm-project/offload/libomptarget/omptarget.cpp:2425:12: error: use of undeclared identifier 'TM'; did you mean 'SymbolDataTy::TM'?
 2425 |     assert(TM->Table->TargetsTable.size() > (size_t)DeviceId &&
      |            ^~
      |            SymbolDataTy::TM
/usr/include/assert.h:93:27: note: expanded from macro 'assert'
   93 |      (static_cast <bool> (expr)                                         \
      |                           ^~~~
/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/llvm-project/offload/libomptarget/omptarget.cpp:2404:15: note: 'SymbolDataTy::TM' declared here
 2404 |     TableMap *TM = nullptr;
      |               ^
/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/llvm-project/offload/libomptarget/omptarget.cpp:2425:12: error: invalid use of non-static data member 'TM'
 2425 |     assert(TM->Table->TargetsTable.size() > (size_t)DeviceId &&
      |            ^~
2 errors generated.
[31/56] Building CXX object offload/plugins-nextgen/common/CMakeFiles/PluginCommon.dir/src/GlobalHandler.cpp.o
[32/56] Building CXX object offload/plugins-nextgen/common/CMakeFiles/PluginCommon.dir/src/RecordReplay.cpp.o
/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/llvm-project/offload/plugins-nextgen/common/src/RecordReplay.cpp:318:20: warning: comparison of integers of different signs: 'uint64_t' (aka 'unsigned long') and '__ptrdiff_t' (aka 'long') [-Wsign-compare]
  318 |   assert(TotalSize ==
      |          ~~~~~~~~~ ^
  319 |              utils::getPtrDiff(BufferPtr, GlobalsMB->get()->getBufferStart()) &&
      |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/assert.h:93:27: note: expanded from macro 'assert'
   93 |      (static_cast <bool> (expr)                                         \
      |                           ^~~~
1 warning generated.
[33/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/OffloadRTL.cpp.o
[34/56] Building CXX object offload/plugins-nextgen/host/CMakeFiles/omptarget.rtl.host.dir/src/rtl.cpp.o
[35/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/OpenMP/InteropAPI.cpp.o
[36/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/OpenMP/Mapping.cpp.o
[37/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/interface.cpp.o
[38/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/device.cpp.o
[39/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/OpenMP/API.cpp.o
[40/56] Building CXX object offload/liboffload/CMakeFiles/LLVMOffload.dir/src/OffloadLib.cpp.o
[41/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/PluginManager.cpp.o
[42/56] Building CXX object offload/liboffload/CMakeFiles/LLVMOffload.dir/src/OffloadImpl.cpp.o
[43/56] Building CXX object offload/plugins-nextgen/common/CMakeFiles/PluginCommon.dir/src/PluginInterface.cpp.o
[44/56] Building CXX object offload/plugins-nextgen/cuda/CMakeFiles/omptarget.rtl.cuda.dir/src/rtl.cpp.o
Step 7 (build cmake config) failure: build cmake config (failure)
...
[21/56] Generating OffloadEntryPoints.inc
[22/56] Linking CXX shared library /home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/build/lib/clang/23/lib/x86_64-unknown-linux-gnu/libclang_rt.memprof.so
[23/56] Linking CXX shared library /home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/build/lib/clang/23/lib/x86_64-unknown-linux-gnu/libclang_rt.asan.so
[24/56] Generating /home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/build/compile_commands.json
[25/56] Building CXX object offload/plugins-nextgen/amdgpu/CMakeFiles/omptarget.rtl.amdgpu.dir/dynamic_hsa/hsa.cpp.o
[26/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/OpenMP/OMPT/Callback.cpp.o
[27/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/DeviceImage.cpp.o
[28/56] Building CXX object offload/tools/kernelreplay/CMakeFiles/llvm-omp-kernel-replay.dir/llvm-omp-kernel-replay.cpp.o
[29/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/LegacyAPI.cpp.o
[30/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/omptarget.cpp.o
FAILED: offload/libomptarget/CMakeFiles/omptarget.dir/omptarget.cpp.o 
/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/build/./bin/clang++ --target=x86_64-unknown-linux-gnu -DDEBUG_PREFIX=\"omptarget\" -DOMPT_SUPPORT=1 -DTARGET_NAME=omptarget -D_DEBUG -D_GLIBCXX_ASSERTIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/llvm-project/llvm/include -I/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/build/include -I/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/build/runtimes/runtimes-bins/openmp/runtime/src -I/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/llvm-project/offload/include -I/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/build/runtimes/runtimes-bins/offload/include -I/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/llvm-project/offload/plugins-nextgen/common/include -I/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/build/runtimes/runtimes-bins/offload/plugins-nextgen/common/include -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Xclang -fno-pch-timestamp -O3 -DNDEBUG -fPIC -UNDEBUG -fno-exceptions -funwind-tables -fno-rtti -Werror=global-constructors -MD -MT offload/libomptarget/CMakeFiles/omptarget.dir/omptarget.cpp.o -MF offload/libomptarget/CMakeFiles/omptarget.dir/omptarget.cpp.o.d -o offload/libomptarget/CMakeFiles/omptarget.dir/omptarget.cpp.o -c /home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/llvm-project/offload/libomptarget/omptarget.cpp
/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/llvm-project/offload/libomptarget/omptarget.cpp:2425:12: error: use of undeclared identifier 'TM'; did you mean 'SymbolDataTy::TM'?
 2425 |     assert(TM->Table->TargetsTable.size() > (size_t)DeviceId &&
      |            ^~
      |            SymbolDataTy::TM
/usr/include/assert.h:93:27: note: expanded from macro 'assert'
   93 |      (static_cast <bool> (expr)                                         \
      |                           ^~~~
/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/llvm-project/offload/libomptarget/omptarget.cpp:2404:15: note: 'SymbolDataTy::TM' declared here
 2404 |     TableMap *TM = nullptr;
      |               ^
/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/llvm-project/offload/libomptarget/omptarget.cpp:2425:12: error: invalid use of non-static data member 'TM'
 2425 |     assert(TM->Table->TargetsTable.size() > (size_t)DeviceId &&
      |            ^~
2 errors generated.
[31/56] Building CXX object offload/plugins-nextgen/common/CMakeFiles/PluginCommon.dir/src/GlobalHandler.cpp.o
[32/56] Building CXX object offload/plugins-nextgen/common/CMakeFiles/PluginCommon.dir/src/RecordReplay.cpp.o
/home/botworker/bbot/amdgpu-offload-ubuntu-22-cmake-build-only/llvm-project/offload/plugins-nextgen/common/src/RecordReplay.cpp:318:20: warning: comparison of integers of different signs: 'uint64_t' (aka 'unsigned long') and '__ptrdiff_t' (aka 'long') [-Wsign-compare]
  318 |   assert(TotalSize ==
      |          ~~~~~~~~~ ^
  319 |              utils::getPtrDiff(BufferPtr, GlobalsMB->get()->getBufferStart()) &&
      |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/assert.h:93:27: note: expanded from macro 'assert'
   93 |      (static_cast <bool> (expr)                                         \
      |                           ^~~~
1 warning generated.
[33/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/OffloadRTL.cpp.o
[34/56] Building CXX object offload/plugins-nextgen/host/CMakeFiles/omptarget.rtl.host.dir/src/rtl.cpp.o
[35/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/OpenMP/InteropAPI.cpp.o
[36/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/OpenMP/Mapping.cpp.o
[37/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/interface.cpp.o
[38/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/device.cpp.o
[39/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/OpenMP/API.cpp.o
[40/56] Building CXX object offload/liboffload/CMakeFiles/LLVMOffload.dir/src/OffloadLib.cpp.o
[41/56] Building CXX object offload/libomptarget/CMakeFiles/omptarget.dir/PluginManager.cpp.o
[42/56] Building CXX object offload/liboffload/CMakeFiles/LLVMOffload.dir/src/OffloadImpl.cpp.o
[43/56] Building CXX object offload/plugins-nextgen/common/CMakeFiles/PluginCommon.dir/src/PluginInterface.cpp.o
[44/56] Building CXX object offload/plugins-nextgen/cuda/CMakeFiles/omptarget.rtl.cuda.dir/src/rtl.cpp.o

```

</details>

https://github.com/llvm/llvm-project/pull/190588