[lld] r361923 - Add IR support, ELF section and user documentation for partitioning feature.

Tue May 28 20:29:01 PDT 2019

Author: pcc
Date: Tue May 28 20:29:01 2019
New Revision: 361923

URL: http://llvm.org/viewvc/llvm-project?rev=361923&view=rev
Log:
Add IR support, ELF section and user documentation for partitioning feature.

The partitioning feature was proposed here:
http://lists.llvm.org/pipermail/llvm-dev/2019-February/130583.html

This is mostly just documentation. The feature itself will be contributed
in subsequent patches.

Differential Revision: https://reviews.llvm.org/D60242

Added:
    lld/trunk/docs/Partitions.rst
    lld/trunk/docs/partitions.dot
    lld/trunk/docs/partitions.svg
Modified:
    lld/trunk/docs/index.rst

Added: lld/trunk/docs/Partitions.rst
URL: http://llvm.org/viewvc/llvm-project/lld/trunk/docs/Partitions.rst?rev=361923&view=auto
==============================================================================

--- lld/trunk/docs/Partitions.rst (added)
+++ lld/trunk/docs/Partitions.rst Tue May 28 20:29:01 2019
@@ -0,0 +1,117 @@
+Partitions
+==========
+
+.. warning::
+
+  This feature has not yet fully landed in LLD. This document describes how
+  the feature is intended to work once it lands. Furthermore, the feature
+  is currently experimental, and its interface is subject to change.
+
+LLD's partitioning feature allows a program (which may be an executable
+or a shared library) to be split into multiple pieces, or partitions. A
+partitioned program consists of a main partition together with a number of
+loadable partitions. The loadable partitions depend on the main partition
+in a similar way to a regular ELF shared object dependency, but unlike a
+shared object, the main partition and the loadable partitions share a virtual
+address space at link time, and each loadable partition is assigned a fixed
+offset from the main partition. This allows the loadable partitions to refer
+to code and data in the main partition directly without the binary size and
+performance overhead of PLTs, GOTs or symbol table entries.
+
+Usage
+-----
+
+A program that uses the partitioning feature must decide which symbols are
+going to be used as the "entry points" for each partition. An entry point
+could, for example, be the equivalent of the partition's ``main`` function, or
+there could be a group of functions that expose the functionality implemented
+by the partition. The intent is that in order to use a loadable partition,
+the program will use ``dlopen``/``dlsym`` or similar functions to dynamically
+load the partition at its assigned address, look up an entry point by name
+and call it. Note, however, that the standard ``dlopen`` function does not
+allow specifying a load address. On Android, the ``android_dlopen_ext``
+function may be used together with the ``ANDROID_DLEXT_RESERVED_ADDRESS``
+flag to load a shared object at a specific address.
+
+Once the entry points have been decided, the translation unit(s)
+containing the entry points should be compiled using the Clang compiler flag
+``-fsymbol-partition=<soname>``, where ``<soname>`` is the intended soname
+of the partition. The resulting object files are passed to the linker in
+the usual way.
+
+The linker will then use these entry points to automatically split the program
+into partitions according to which sections of the program are reachable from
+which entry points, similarly to how ``--gc-sections`` removes unused parts of
+a program. Any sections that are only reachable from a loadable partition's
+entry point are assigned to that partition, while all other sections are
+assigned to the main partition, including sections only reachable from
+loadable partitions.
+
+The following diagram illustrates how sections are assigned to partitions. Each
+section is colored according to its assigned partition.
+
+.. image:: partitions.svg
+
+The result of linking a program that uses partitions is essentially an
+ELF file with all of the partitions concatenated together. This file is
+referred to as a combined output file. To extract a partition from the
+combined output file, the ``llvm-objcopy`` tool should be used together
+with the flag ``--extract-main-partition`` to extract the main partition, or
+``-extract-partition=<soname>`` to extract one of the loadable partitions.
+An example command sequence is shown below:
+
+.. code-block:: shell
+
+  # Compile the main program.
+  clang -ffunction-sections -fdata-sections -c main.c
+
+  # Compile a feature to be placed in a loadable partition.
+  # Note that this is likely to be a separate build step to the main partition.
+  clang -ffunction-sections -fdata-sections -fsymbol-partition=libfeature.so -c feature.c
+
+  # Link the combined output file.
+  clang main.o feature.o -fuse-ld=lld -shared -o libcombined.so -Wl,-soname,libmain.so -Wl,--gc-sections
+
+  # Extract the partitions.
+  llvm-objcopy libcombined.so libmain.so --extract-main-partition
+  llvm-objcopy libcombined.so libfeature.so --extract-partition=libfeature.so
+
+In order to allow a program to discover the names of its loadable partitions
+and the locations of their reserved regions, the linker creates a partition
+index, which is an array of structs with the following definition:
+
+.. code-block:: c
+
+  struct partition_index_entry {
+    int32_t name_offset;
+    int32_t addr_offset;
+    uint32_t size;
+  };
+
+The ``name_offset`` field is a relative pointer to a null-terminated string
+containing the soname of the partition, the ``addr_offset`` field is a
+relative pointer to its load address and the ``size`` field contains the
+size of the region reserved for the partition. To derive an absolute pointer
+from the relative pointer fields in this data structure, the address of the
+field should be added to the value stored in the field.
+
+The program may discover the location of the partition index using the
+linker-defined symbols ``__part_index_begin`` and ``__part_index_end``.
+
+Restrictions
+------------
+
+This feature is currently only supported in the ELF linker.
+
+The partitioning feature may not currently be used together with the
+``SECTIONS`` or ``PHDRS`` linker script features, nor may it be used with the
+``--section-start``, ``-Ttext``, ``-Tdata`` or ``-Tbss`` flags. All of these
+features assume a single set of output sections and/or program headers, which
+makes their semantics ambiguous in the presence of more than one partition.
+
+The partitioning feature may not currently be used on the MIPS architecture
+because it is unclear whether the MIPS multi-GOT ABI is compatible with
+partitions.
+
+The current implementation only supports creating up to 254 partitions due
+to implementation limitations. This limit may be relaxed in the future.

Modified: lld/trunk/docs/index.rst
URL: http://llvm.org/viewvc/llvm-project/lld/trunk/docs/index.rst?rev=361923&r1=361922&r2=361923&view=diff
==============================================================================
--- lld/trunk/docs/index.rst (original)
+++ lld/trunk/docs/index.rst Tue May 28 20:29:01 2019
@@ -174,4 +174,5 @@ document soon.
    WebAssembly
    windows_support
    missingkeyfunction
+   Partitions
    ReleaseNotes

Added: lld/trunk/docs/partitions.dot
URL: http://llvm.org/viewvc/llvm-project/lld/trunk/docs/partitions.dot?rev=361923&view=auto
==============================================================================
--- lld/trunk/docs/partitions.dot (added)
+++ lld/trunk/docs/partitions.dot Tue May 28 20:29:01 2019
@@ -0,0 +1,22 @@
+digraph G {
+  part_main [label="Main partition",shape=plaintext];
+  part1 [label="Loadable partition 1",shape=plaintext];
+  part2 [label="Loadable partition 2",shape=plaintext];
+  main [style=filled,fillcolor=lightblue];
+  f1 [style=filled,fillcolor=lightsalmon];
+  f2 [style=filled,fillcolor=palegreen];
+  f3 [style=filled,fillcolor=lightblue];
+  f4 [style=filled,fillcolor=lightsalmon];
+  f5 [style=filled,fillcolor=lightblue];
+  f6 [style=filled,fillcolor=palegreen];
+  part_main -> main;
+  main -> f3;
+  part1 -> f1;
+  f1 -> f3;
+  f1 -> f4;
+  f1 -> f5;
+  part2 -> f2;
+  f2 -> f3;
+  f2 -> f5;
+  f2 -> f6;
+}

Added: lld/trunk/docs/partitions.svg
URL: http://llvm.org/viewvc/llvm-project/lld/trunk/docs/partitions.svg?rev=361923&view=auto
==============================================================================
--- lld/trunk/docs/partitions.svg (added)
+++ lld/trunk/docs/partitions.svg Tue May 28 20:29:01 2019
@@ -0,0 +1,110 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
+ "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by graphviz version 2.38.0 (20140413.2041)
+ -->
+<!-- Title: G Pages: 1 -->
+<svg width="393pt" height="188pt"
+ viewBox="0.00 0.00 393.00 188.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
+<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 184)">
+<title>G</title>
+<polygon fill="white" stroke="none" points="-4,4 -4,-184 389,-184 389,4 -4,4"/>
+<!-- part_main -->
+<g id="node1" class="node"><title>part_main</title>
+<text text-anchor="middle" x="47.5" y="-158.3" font-family="Times,serif" font-size="14.00">Main partition</text>
+</g>
+<!-- main -->
+<g id="node4" class="node"><title>main</title>
+<ellipse fill="lightblue" stroke="black" cx="75.5" cy="-90" rx="28.6953" ry="18"/>
+<text text-anchor="middle" x="75.5" y="-86.3" font-family="Times,serif" font-size="14.00">main</text>
+</g>
+<!-- part_main->main -->
+<g id="edge1" class="edge"><title>part_main->main</title>
+<path fill="none" stroke="black" d="M54.4214,-143.697C57.6218,-135.696 61.492,-126.02 65.0381,-117.155"/>
+<polygon fill="black" stroke="black" points="68.3868,-118.207 68.8511,-107.622 61.8874,-115.607 68.3868,-118.207"/>
+</g>
+<!-- part1 -->
+<g id="node2" class="node"><title>part1</title>
+<text text-anchor="middle" x="176.5" y="-158.3" font-family="Times,serif" font-size="14.00">Loadable partition 1</text>
+</g>
+<!-- f1 -->
+<g id="node5" class="node"><title>f1</title>
+<ellipse fill="lightsalmon" stroke="black" cx="176.5" cy="-90" rx="27" ry="18"/>
+<text text-anchor="middle" x="176.5" y="-86.3" font-family="Times,serif" font-size="14.00">f1</text>
+</g>
+<!-- part1->f1 -->
+<g id="edge3" class="edge"><title>part1->f1</title>
+<path fill="none" stroke="black" d="M176.5,-143.697C176.5,-135.983 176.5,-126.712 176.5,-118.112"/>
+<polygon fill="black" stroke="black" points="180,-118.104 176.5,-108.104 173,-118.104 180,-118.104"/>
+</g>
+<!-- part2 -->
+<g id="node3" class="node"><title>part2</title>
+<text text-anchor="middle" x="321.5" y="-158.3" font-family="Times,serif" font-size="14.00">Loadable partition 2</text>
+</g>
+<!-- f2 -->
+<g id="node6" class="node"><title>f2</title>
+<ellipse fill="palegreen" stroke="black" cx="284.5" cy="-90" rx="27" ry="18"/>
+<text text-anchor="middle" x="284.5" y="-86.3" font-family="Times,serif" font-size="14.00">f2</text>
+</g>
+<!-- part2->f2 -->
+<g id="edge7" class="edge"><title>part2->f2</title>
+<path fill="none" stroke="black" d="M312.354,-143.697C307.97,-135.403 302.636,-125.311 297.813,-116.187"/>
+<polygon fill="black" stroke="black" points="300.801,-114.35 293.034,-107.145 294.612,-117.621 300.801,-114.35"/>
+</g>
+<!-- f3 -->
+<g id="node7" class="node"><title>f3</title>
+<ellipse fill="lightblue" stroke="black" cx="104.5" cy="-18" rx="27" ry="18"/>
+<text text-anchor="middle" x="104.5" y="-14.3" font-family="Times,serif" font-size="14.00">f3</text>
+</g>
+<!-- main->f3 -->
+<g id="edge2" class="edge"><title>main->f3</title>
+<path fill="none" stroke="black" d="M82.3726,-72.411C85.7675,-64.2164 89.9422,-54.1395 93.7473,-44.9548"/>
+<polygon fill="black" stroke="black" points="97.0828,-46.0481 97.6767,-35.4699 90.6158,-43.3689 97.0828,-46.0481"/>
+</g>
+<!-- f1->f3 -->
+<g id="edge4" class="edge"><title>f1->f3</title>
+<path fill="none" stroke="black" d="M161.93,-74.8345C151.75,-64.9376 137.976,-51.5462 126.469,-40.3591"/>
+<polygon fill="black" stroke="black" points="128.905,-37.8461 119.296,-33.3847 124.026,-42.865 128.905,-37.8461"/>
+</g>
+<!-- f4 -->
+<g id="node8" class="node"><title>f4</title>
+<ellipse fill="lightsalmon" stroke="black" cx="176.5" cy="-18" rx="27" ry="18"/>
+<text text-anchor="middle" x="176.5" y="-14.3" font-family="Times,serif" font-size="14.00">f4</text>
+</g>
+<!-- f1->f4 -->
+<g id="edge5" class="edge"><title>f1->f4</title>
+<path fill="none" stroke="black" d="M176.5,-71.6966C176.5,-63.9827 176.5,-54.7125 176.5,-46.1124"/>
+<polygon fill="black" stroke="black" points="180,-46.1043 176.5,-36.1043 173,-46.1044 180,-46.1043"/>
+</g>
+<!-- f5 -->
+<g id="node9" class="node"><title>f5</title>
+<ellipse fill="lightblue" stroke="black" cx="248.5" cy="-18" rx="27" ry="18"/>
+<text text-anchor="middle" x="248.5" y="-14.3" font-family="Times,serif" font-size="14.00">f5</text>
+</g>
+<!-- f1->f5 -->
+<g id="edge6" class="edge"><title>f1->f5</title>
+<path fill="none" stroke="black" d="M191.07,-74.8345C201.25,-64.9376 215.024,-51.5462 226.531,-40.3591"/>
+<polygon fill="black" stroke="black" points="228.974,-42.865 233.704,-33.3847 224.095,-37.8461 228.974,-42.865"/>
+</g>
+<!-- f2->f3 -->
+<g id="edge8" class="edge"><title>f2->f3</title>
+<path fill="none" stroke="black" d="M260.806,-81.0022C232.063,-71.1346 182.266,-53.5073 140.5,-36 138.683,-35.2385 136.825,-34.4358 134.957,-33.6106"/>
+<polygon fill="black" stroke="black" points="136.231,-30.3452 125.68,-29.3829 133.328,-36.7149 136.231,-30.3452"/>
+</g>
+<!-- f2->f5 -->
+<g id="edge9" class="edge"><title>f2->f5</title>
+<path fill="none" stroke="black" d="M276.15,-72.7646C271.788,-64.2831 266.353,-53.7144 261.459,-44.1974"/>
+<polygon fill="black" stroke="black" points="264.49,-42.4395 256.804,-35.1473 258.265,-45.6409 264.49,-42.4395"/>
+</g>
+<!-- f6 -->
+<g id="node10" class="node"><title>f6</title>
+<ellipse fill="palegreen" stroke="black" cx="320.5" cy="-18" rx="27" ry="18"/>
+<text text-anchor="middle" x="320.5" y="-14.3" font-family="Times,serif" font-size="14.00">f6</text>
+</g>
+<!-- f2->f6 -->
+<g id="edge10" class="edge"><title>f2->f6</title>
+<path fill="none" stroke="black" d="M292.85,-72.7646C297.212,-64.2831 302.647,-53.7144 307.541,-44.1974"/>
+<polygon fill="black" stroke="black" points="310.735,-45.6409 312.196,-35.1473 304.51,-42.4395 310.735,-45.6409"/>
+</g>
+</g>
+</svg>


