[llvm-dev] [RFC] LLVM Busybox Proposal

Tue Jun 22 17:00:04 PDT 2021

On 2021-06-22, Leonard Chan via llvm-dev wrote:
>Small update: I have a WIP prototype of the tool at
>https://reviews.llvm.org/D104686. The prototype only includes llvm-objcopy
>and llvm-objdump packed together, but we're seeing size benefits from
>busyboxing those two compared against having two separate tools. (More
>details in the prototype's description.) I don't plan on landing this as-is
>anytime soon and there's still some things I'd like to improve/change and
>get feedback on.
>
>To answer some replies:
>
>- Ideally, we could start off with an incremental approach and not package
>large tools like clang/lld off the bat. The llvm-* tools seem like a good
>place to start since they're generally a bunch of relatively small binaries
>that all share a subset of functions in libLLVM, but don't necessarily use
>all of libLLVM, so statically linking them together (with --gc-sections)
>can help dedup a lot of shared components vs having separate statically
>compiled tools. In my measurements, the busybox tool containing
>llvm-objcopy+objdump is negligibly larger than llvm-objdump on its own (a
>couple KB difference) indicating a lot of shared code between objdump and
>objcopy.
>
>- Will Dietz's multiplexing tool looks like a good place to start from. The
>only concern I can see though is mostly the amount of work needed to update
>it to LLVM 13.
>
>- We don't have plans for windows support now, but it's not off the table.
>(Been mostly focusing on *nix for now). Depending on overall traction for
>this idea, we could approach incrementally and add support for different
>platforms over time.

-DLLVM_LINK_LLVM_DYLIB=on -DCLANG_LINK_CLANG_DYLIB=on -DLLVM_TARGETS_TO_BUILD=X86 (custom1)
vs
-DLLVM_TARGETS_TO_BUILD=X86 (custom2)

# This is the lower bound for any multiplexing approach. clang is the largest executable.
% stat -c %s /tmp/out/custom2/bin/clang-13
102900408

I have built clang, lld and a bunch of ELF binary utilities.

% stat -c %s /tmp/out/custom1/lib/libLLVM-13git.so /tmp/out/custom1/lib/libclang-cpp.so.13git /tmp/out/custom1/bin/{clang-13,lld,llvm-{ar,cov,cxxfilt,nm,objcopy,objdump,readobj,size,strings,symbolizer}} | awk '{s+=$1}END{print s}'
138896544

% stat -c %s /tmp/out/custom2/bin/{clang-13,lld,llvm-{ar,cov,cxxfilt,nm,objcopy,objdump,readobj,size,strings,symbolizer}} | awk '{s+=$1}END{print s}'
209054440

The -DLLVM_LINK_LLVM_DYLIB=on -DCLANG_LINK_CLANG_DYLIB=on build is doing a really good job.

A multiplexing approach can squeeze some bytes from 138896544 toward 102900408,
but how much can it do?

>- I'm starting to think the `cl::opt` to `OptTable` issue might be
>orthogonal to the busybox implementation. The tool essentially dispatches
>to different "main" functions in different tools, but as long as we don't
>do anything within busybox after exiting that tool's main, then the global
>state issues we weren't sure of with `cl::opt` might not be of any concern
>now. It may be an issue down the line if, let's say, the tool flags moved
>from being "owned" by the tools themselves to instead being "owned" by
>busybox, and then we'd have to merge similarly-named flags together. In
>that case, migrating these tools to use `OptTable` may be necessary since
>(I think) `OptTable` should handle this. This may be a tedious task, but
>this is just to say that busybox won't need to be immediately blocked on it.

Such improvement is useful even if we don't do multiplexing.
I switched llvm-symbolizer. thakis switched llvm-objdump.
I can look at some binary utilities.

>- I haven't seen any issues with colliding symbols when linking (although
>I've only merged two tools for now). I suspect that with small-ish llvm-*
>tools, the bulk of their code is shared from libLLVM, and they have their
>own distinct logic built on top of it, which could mean a low chance of
>conflicting internal ABIs.
>
>On Mon, Jun 21, 2021 at 10:54 AM Leonard Chan <leonardchan at google.com>
>wrote:
>
>> Hello all,
>>
>> When building LLVM tools, including Clang and lld, it's currently possible
>> to use either static or shared linking for LLVM libraries. The latter can
>> significantly reduce the size of the toolchain since we aren't duplicating
>> the same code in every binary, but the dynamic relocations can affect
>> performance. The former doesn't affect performance but significantly
>> increases the size of our toolchain.
>>
>> We would like to implement a support for a third approach which we call,
>> for a lack of better term, "busybox" feature, where everything is compiled
>> into a single binary which then dispatches into an appropriate tool
>> depending on the first command. This approach can significantly reduce the
>> size by deduplicating all of the shared code without affecting the
>> performance.
>>
>> In terms of implementation, the build would produce a single binary called
>> `llvm` and the first command would identify the tool. For example, instead
>> of invoking `llvm-nm` you'd invoke `llvm nm`. Ideally we would also support
>> creation of `llvm-nm` symlink which redirects to `llvm` for backwards
>> compatibility.
>> This functionality would ideally be implemented as an option in the CMake
>> build that toolchain vendors can opt into.
>>
>> The implementation would have to replace `main` function of each tool with
>> an entrypoint regular function which is registered into a tool registry.
>> This could be wrapped in a macro for convenience. When the "busybox"
>> feature is disabled, the macro would expand to a `main` function as before
>> and redirect to the entrypoint function. When the "busybox" feature is
>> enabled, it would register the entrypoint function into the registry, which
>> would be responsible for the dispatching based on the tool name. Ideally,
>> toolchain maintainers would also be able to control which tools they could
>> add to the "busybox" binary via CMake build options, so toolchains will
>> only include the tools they use.
>>
>> One implementation detail we think will be an issue is merging arguments
>> in individual tools that use `cl::opt`. `cl::opt` works by maintaining a
>> global state of flags, but we aren’t confident of what the resulting
>> behavior will be when merging them together in the dispatching `main`. What
>> we would like to avoid is having flags used by one specific tool available
>> on other tools. To address this issue, we would like to migrate all tools
>> to use `OptTable` which doesn't have this issue and has been the general
>> direction most tools have been already moving into.
>>
>> A second issue would be resolving symlinks. For example, llvm-objcopy will
>> check argv[0] and behave as llvm-strip (ie. use the right flags +
>> configuration) if it is called via a symlink that “looks like” a strip
>> tool, but for all other cases it will run under the default objcopy mode.
>> The “looks like” function is usually an `Is` function copied in multiple
>> tools that is essentially a substring check: so symlinks like `llvm-strip`,
>> strip.exe, and `gnu-llvm-strip-10` all result in using the strip “mode”
>> while all other names use the objcopy mode. To replicate the same behavior,
>> we will need to take great care in making sure symlinks to the busybox tool
>> dispatch correctly to the appropriate llvm tool, which might mean exposing
>> and merging these `Is` functions.
>>
>> Some open questions:
>> - People's initial thoughts/opinions?
>> - Are there existing tools in LLVM that already do this?
>> - Other implementation details/global states that we would also need to
>> account for?
>>
>> - Leonard
>>

>_______________________________________________
>LLVM Developers mailing list
>llvm-dev at lists.llvm.org
>https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev