[llvm-dev] RFC [ThinLTO]: Promoting more aggressively in order to reduce incremental link time and allow sharing between linkage units

[Peter Collingbourne via llvm-dev]

Hi all,

I'd like to propose changes to how we do promotion of global values in
ThinLTO. The goal here is to make it possible to pre-compile parts of the
translation unit to native code at compile time. For example, if we know
that:

1) A function is a leaf function, so it will never import any other
functions, and
2) The function's instruction count falls above a threshold specified at
compile time, so it will never be imported.
or
3) The compile-time threshold is zero, so there is no possibility of
functions being imported (What's the utility of this? Consider a program
transformation that requires whole-program information, such as CFI. During
development, the import threshold may be set to zero in order to minimize
the incremental link time while still providing the same CFI enforcement
that would be used in production builds of the application.)

then the function's body will not be affected by link-time decisions, and
we might as well produce its object code at compile time. This will also
allow the object code to be shared between linkage units (this should
hopefully help solve a major scalability problem for Chromium, as that
project contains a large number of test binaries based on common libraries).

This can be done with a change to the intermediate object file format. We
can represent object files as native code containing statically compiled
functions and global data in the .text,. data, .rodata (etc.) sections,
with an .llvmbc section (or, I suppose, "__LLVM, __bitcode" when targeting
Mach-O) containing bitcode for functions to be compiled at link time.

In order to make this work, we need to make sure that references from
link-time compiled functions to statically compiled functions work
correctly in the case where the statically compiled function has internal
linkage. We can do this by promoting every global value with internal
linkage, using a hash of the external names (as I mentioned in [1]).

I imagine that for some linkers, it may not be possible to deal with this
scheme. For example, I did some investigation last year and discovered that
I could not use the gold plugin interface to load a native object file if
we had already claimed it as an IR file. I wouldn't be surprised to learn
that ld64 has similar problems.

In cases where we completely control the linker (e.g. lld), we can easily
support this scheme, as the linker can directly do whatever it wants. But
for linkers that cannot support this, I suggest that we promote
consistently under ThinLTO rather than having different promotion schemes
for different linkers, in order to reduce overall complexity.

Thanks for your feedback!

Thanks,
-- 
-- 
Peter

[1] http://lists.llvm.org/pipermail/llvm-dev/2016-April/098062.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160406/5c8768cf/attachment.html>