[all-commits] [llvm/llvm-project] 05634f: [BOLT] Move from RuntimeDyld to JITLink

[Job Noorman via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 05634f7346a59f6dab89cde53f39b40d9a70b9c9
      https://github.com/llvm/llvm-project/commit/05634f7346a59f6dab89cde53f39b40d9a70b9c9
  Author: Job Noorman <jnoorman at igalia.com>
  Date:   2023-06-15 (Thu, 15 Jun 2023)

  Changed paths:
    M bolt/include/bolt/Core/BinaryContext.h
    M bolt/include/bolt/Core/BinarySection.h
    A bolt/include/bolt/Core/Linker.h
    M bolt/include/bolt/Rewrite/ExecutableFileMemoryManager.h
    A bolt/include/bolt/Rewrite/JITLinkLinker.h
    M bolt/include/bolt/Rewrite/MachORewriteInstance.h
    M bolt/include/bolt/Rewrite/RewriteInstance.h
    M bolt/include/bolt/RuntimeLibs/HugifyRuntimeLibrary.h
    M bolt/include/bolt/RuntimeLibs/InstrumentationRuntimeLibrary.h
    M bolt/include/bolt/RuntimeLibs/RuntimeLibrary.h
    M bolt/lib/Core/BinaryContext.cpp
    M bolt/lib/Core/BinaryEmitter.cpp
    M bolt/lib/Core/DebugData.cpp
    M bolt/lib/Rewrite/CMakeLists.txt
    M bolt/lib/Rewrite/ExecutableFileMemoryManager.cpp
    A bolt/lib/Rewrite/JITLinkLinker.cpp
    M bolt/lib/Rewrite/MachORewriteInstance.cpp
    M bolt/lib/Rewrite/RewriteInstance.cpp
    M bolt/lib/RuntimeLibs/CMakeLists.txt
    M bolt/lib/RuntimeLibs/HugifyRuntimeLibrary.cpp
    M bolt/lib/RuntimeLibs/InstrumentationRuntimeLibrary.cpp
    M bolt/lib/RuntimeLibs/RuntimeLibrary.cpp

  Log Message:
  -----------
  [BOLT] Move from RuntimeDyld to JITLink

RuntimeDyld has been deprecated in favor of JITLink. [1] This patch
replaces all uses of RuntimeDyld in BOLT with JITLink.

Care has been taken to minimize the impact on the code structure in
order to ease the inspection of this (rather large) changeset. Since
BOLT relied on the RuntimeDyld API in multiple places, this wasn't
always possible though and I'll explain the changes in code structure
first.

Design note: BOLT uses a JIT linker to perform what essentially is
static linking. No linked code is ever executed; the result of linking
is simply written back to an executable file. For this reason, I
restricted myself to the use of the core JITLink library and avoided ORC
as much as possible.

RuntimeDyld contains methods for loading objects (loadObject) and symbol
lookup (getSymbol). Since JITLink doesn't provide a class with a similar
interface, the BOLTLinker abstract class was added to implement it. It
was added to Core since both the Rewrite and RuntimeLibs libraries make
use of it. Wherever a RuntimeDyld object was used before, it was
replaced with a BOLTLinker object.

There is one major difference between the RuntimeDyld and BOLTLinker
interfaces: in JITLink, section allocation and the application of fixups
(relocation) happens in a single call (jitlink::link). That is, there is
no separate method like finalizeWithMemoryManagerLocking in RuntimeDyld.
BOLT used to remap sections between allocating (loadObject) and linking
them (finalizeWithMemoryManagerLocking). This doesn't work anymore with
JITLink. Instead, BOLTLinker::loadObject accepts a callback that is
called before fixups are applied which is used to remap sections.

The actual implementation of the BOLTLinker interface lives in the
JITLinkLinker class in the Rewrite library. It's the only part of the
BOLT code that should directly interact with the JITLink API.

For loading object, JITLinkLinker first creates a LinkGraph
(jitlink::createLinkGraphFromObject) and then links it (jitlink::link).
For the latter, it uses a custom JITLinkContext with the following
properties:
- Use BOLT's ExecutableFileMemoryManager. This one was updated to
  implement the JITLinkMemoryManager interface. Since BOLT never
  executes code, its finalization step is a no-op.
- Pass config: don't use the default target passes since they modify
  DWARF sections in a way that seems incompatible with BOLT. Also run a
  custom pre-prune pass that makes sure sections without symbols are not
  pruned by JITLink.
- Implement symbol lookup. This used to be implemented by
  BOLTSymbolResolver.
- Call the section mapper callback before the final linking step.
- Copy symbol values when the LinkGraph is resolved. Symbols are stored
  inside JITLinkLinker to ensure that later objects (i.e.,
  instrumentation libraries) can find them. This functionality used to
  be provided by RuntimeDyld but I did not find a way to use JITLink
  directly for this.

Some more minor points of interest:
- BinarySection::SectionID: JITLink doesn't have something equivalent to
  RuntimeDyld's Section IDs. Instead, sections can only be referred to
  by name. Hence, SectionID was updated to a string.
- There seem to be no tests for Mach-O. I've tested a small hello-world
  style binary but not more than that.
- On Mach-O, JITLink "normalizes" section names to include the segment
  name. I had to parse the section name back from this manually which
  feels slightly hacky.

[1] https://reviews.llvm.org/D145686#4222642

Reviewed By: rafauler

Differential Revision: https://reviews.llvm.org/D147544