[llvm-dev] [RFC] LLVM Busybox Proposal

[Fangrui Song via llvm-dev]

On 2021-06-21, Fangrui Song wrote:
>On 2021-06-21, Leonard Chan via llvm-dev wrote:
>>Hello all,
>>
>>When building LLVM tools, including Clang and lld, it's currently possible
>>to use either static or shared linking for LLVM libraries. The latter can
>>significantly reduce the size of the toolchain since we aren't duplicating
>>the same code in every binary, but the dynamic relocations can affect
>>performance. The former doesn't affect performance but significantly
>>increases the size of our toolchain.
>
>The dynamic relocation claim is not true.
>
>A thin executable using just -Bsymbolic libLLVM-13git.so is almost
>identical to a mostly statically linked PIE.
>
>I added -Bsymbolic-functions to libLLVM.so and libclang-cpp.so which
>has claimed most of the -Bsymbolic benefits.
>
>The shared object approach *can be* inferior to static linking plus
>-Wl,--gc-sections because with libLLVM.so and libclang-cpp.so we are
>making many many API dynamic and that inhibits the --gc-sections
>benefits. However, if clang and lld are shipped together with
>llvm-objdump/llvm-readobj/llvm-objcopy/.... , I expect the non-GCable
>code due to shared objects will be significantly smaller.
>
>I am conservative on adding yet another mechanism.
>
>>We would like to implement a support for a third approach which we call,
>>for a lack of better term, "busybox" feature, where everything is compiled
>>into a single binary which then dispatches into an appropriate tool
>>depending on the first command. This approach can significantly reduce the
>>size by deduplicating all of the shared code without affecting the
>>performance.
>>
>>In terms of implementation, the build would produce a single binary called
>>`llvm` and the first command would identify the tool. For example, instead
>>of invoking `llvm-nm` you'd invoke `llvm nm`. Ideally we would also support
>>creation of `llvm-nm` symlink which redirects to `llvm` for backwards
>>compatibility.
>>This functionality would ideally be implemented as an option in the CMake
>>build that toolchain vendors can opt into.
>>
>>The implementation would have to replace `main` function of each tool with
>>an entrypoint regular function which is registered into a tool registry.
>>This could be wrapped in a macro for convenience. When the "busybox"
>>feature is disabled, the macro would expand to a `main` function as before
>>and redirect to the entrypoint function. When the "busybox" feature is
>>enabled, it would register the entrypoint function into the registry, which
>>would be responsible for the dispatching based on the tool name. Ideally,
>>toolchain maintainers would also be able to control which tools they could
>>add to the "busybox" binary via CMake build options, so toolchains will
>>only include the tools they use.
>>
>>One implementation detail we think will be an issue is merging arguments in
>>individual tools that use `cl::opt`. `cl::opt` works by maintaining a
>>global state of flags, but we aren’t confident of what the resulting
>>behavior will be when merging them together in the dispatching `main`. What
>>we would like to avoid is having flags used by one specific tool available
>>on other tools. To address this issue, we would like to migrate all tools
>>to use `OptTable` which doesn't have this issue and has been the general
>>direction most tools have been already moving into.
>>
>>A second issue would be resolving symlinks. For example, llvm-objcopy will
>>check argv[0] and behave as llvm-strip (ie. use the right flags +
>>configuration) if it is called via a symlink that “looks like” a strip
>>tool, but for all other cases it will run under the default objcopy mode.
>>The “looks like” function is usually an `Is` function copied in multiple
>>tools that is essentially a substring check: so symlinks like `llvm-strip`,
>>strip.exe, and `gnu-llvm-strip-10` all result in using the strip “mode”
>>while all other names use the objcopy mode. To replicate the same behavior,
>>we will need to take great care in making sure symlinks to the busybox tool
>>dispatch correctly to the appropriate llvm tool, which might mean exposing
>>and merging these `Is` functions.
>>
>>Some open questions:
>>- People's initial thoughts/opinions?
>>- Are there existing tools in LLVM that already do this?
>>- Other implementation details/global states that we would also need to
>>account for?
>
>crunchgen. As you said, argv[0] checking code needs to be taken care of.
>We should make these executables' main file not have colliding symbols.
>I have cleaned up a lot of files.
>
>>- Leonard

A few points.

In an ideal ELF world only external function calls need PLT entries.
Currently shared objects have PLT entries for in-dso function calls
because default visibility non-local symbols are preemptible by default
and the linker will produce PLT entries. -Bsymbolic-functions suppresses
PLT entries for in-dso symbols.

---

I have an approach for users whose libLLVM.so libclang-cpp are closed
sets and want to get GC benefits. I'll use libLLVM.so as an example.

* Identify the list of executables which link against libLLVM.so: exe0, exe1, exe2.
* For each exe, do a relocatable link of its own code (usually llvm/tools/llvm-foobar/*.o). Get the undefined symbol list.
* Take the union of the undefined symbol lists of all exe. Create a version script file with these symbols `global:` and `local: *`.
* Re-link libLLVM.so with --version-script.

The resulting libLLVM.so only provides dynamic symbols needed by these executables.

This is still tricky and I am not sure how much it can decrease the size.