[PATCH] D58704: Initial (incomplete) implementation of JITLink - A replacement for RuntimeDyld.

[Lang Hames via Phabricator via llvm-commits]

lhames created this revision.
lhames added reviewers: dblaikie, andrew.w.kaylor, anarazel, reames, loladiro, mtrent, pete.
Herald added subscribers: jdoerfert, kristina, mgrang, mgorny.
Herald added a project: LLVM.

JITLink is a jit-linker and performs the same high-level task as RuntimeDyld:
it parses relocatable object files to make their contents runnable in a target
process.

JITLink aims to improve on RuntimeDyld in two primary ways:

(1) A clear design intended to maximize code-sharing while minimizing coupling.

RuntimeDyld has been developed in an ad-hoc fashion for a number of years and
this had led to intermingling of code for multiple architectures (e.g. in
RuntimeDyldELF::processRelocationRef) in a way that makes the code more
difficult to read, reason about, extend.

(2) Native support for asynchronous linking, where the symbol resolution and
finalization steps call back to linker continuations to complete the linker's
work. This allows for cleaner interoperation with the new concurrent ORC JIT
APIs, while still being easily implementable in synchronous style where
asynchrony is not needed.

To maximise sharing, the design has a hierarchy of common code:

(1) Generic atom-graph data structure and algorithms (e.g. dead stripping and

| memory allocation) that are intended to be shared by all architectures. |
|
|

+ -- (2) Shared per-format code that utilizes (1), e.g. Generic MachO to

  |  atom-graph parsing.
  |
  + -- (3) Architecture specific code that uses (1) and (2). E.g.
           JITLinkerMachO_x86_64, which adds x86-64 specific relocation
           support to (2) to build and patch up the atom graph.

To support asynchronous symbol resolution and finalization, the callbacks for
these operations take continuations as arguments:

  using JITLinkAsyncLookupContinuation =
      std::function<void(Expected<AsyncLookupResult> LR)>;
  
  using JITLinkAsyncLookupFunction =
      std::function<void(const DenseSet<StringRef> &Symbols,
                         JITLinkAsyncLookupContinuation LookupContinuation)>;
  
  using FinalizeContinuation = std::function<void(Error)>;
  
  virtual void finalizeAsync(FinalizeContinuation OnFinalize);

In addition to supporting the top-level design goals, JITLink also aims to fix
a number of nitpicks. The first of these (which appears in this patch) is GOT
and stub handling: RuntimeDyld's limited 'stub management' utility allows for
the creation of stubs OR GOT entries but not both, preventing the small-code
model from being implemented on some architectures (e.g. x86-64). JITLink
addresses this by allowing mutations on the atom graph prior to finalization,
which can be used to build both GOT entries and stubs.

This patch is incomplete, but I'm posting it now to allow for early feedback.
Once it is ready to land in-tree, my aim will be to develop JITLink to the point
where it could replace RuntimeDyld for MachO, then (time permitting) look at
tackling COFF and ELF too, though I will be hoping to enlist support for those
formats as I am less familiar with them.


Repository:
  rL LLVM

https://reviews.llvm.org/D58704

Files:
  include/llvm/ExecutionEngine/JITLink/JITLink.h
  include/llvm/ExecutionEngine/JITLink/JITLinkMachO.h
  include/llvm/Support/Memory.h
  lib/ExecutionEngine/CMakeLists.txt
  lib/ExecutionEngine/JITLink/CMakeLists.txt
  lib/ExecutionEngine/JITLink/JITLink.cpp
  lib/ExecutionEngine/JITLink/JITLinkGeneric.cpp
  lib/ExecutionEngine/JITLink/JITLinkGeneric.h
  lib/ExecutionEngine/JITLink/JITLinkMachO_x86_64.cpp
  lib/ExecutionEngine/JITLink/LLVMBuild.txt
  lib/ExecutionEngine/JITLink/MachOAtomGraphBuilder.cpp
  lib/ExecutionEngine/JITLink/MachOAtomGraphBuilder.h
  lib/ExecutionEngine/LLVMBuild.txt
  lib/Support/Memory.cpp
  unittests/ExecutionEngine/CMakeLists.txt
  unittests/ExecutionEngine/JITLink/CMakeLists.txt
  unittests/ExecutionEngine/JITLink/JITLinkTestCommon.cpp
  unittests/ExecutionEngine/JITLink/JITLinkTestCommon.h
  unittests/ExecutionEngine/JITLink/JITLink_MachO_x86_64_Tests.cpp

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D58704.188488.patch
Type: text/x-patch
Size: 102132 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20190227/f736d962/attachment-0001.bin>