[LLVMdev] Updated RFC: ThinLTO Implementation Plan

[Teresa Johnson]

On Thu, Jun 4, 2015 at 10:11 AM, Philip Reames
<listmail at philipreames.com> wrote:
> On 06/04/2015 07:43 AM, Teresa Johnson wrote:
>>
>> On Wed, Jun 3, 2015 at 1:29 PM, Sean Silva <chisophugis at gmail.com> wrote:
>>>
>>>
>>> On Mon, Jun 1, 2015 at 6:34 AM, Teresa Johnson <tejohnson at google.com>
>>> wrote:
>>>>
>>>> On Fri, May 29, 2015 at 6:15 PM, Sean Silva <chisophugis at gmail.com>
>>>> wrote:
>>>>>
>>>>>
>>>>> On Fri, May 29, 2015 at 8:01 AM, Teresa Johnson <tejohnson at google.com>
>>>>> wrote:
>>>>>>
>>>>>> On Fri, May 29, 2015 at 6:56 AM, Alex Rosenberg <alexr at leftfield.org>
>>>>>> wrote:
>>>>>>>
>>>>>>> My earlier statement about wrapping things in a native object file
>>>>>>> held
>>>>>>> in that it is controversial. It appears to be still central to your
>>>>>>> design.
>>>>>>>
>>>>>>> It may help to look at the problem from a different viewpoint: LLVM
>>>>>>> is
>>>>>>> not a compiler. It is a framework that can be used to make
>>>>>>> compiler-like
>>>>>>> tools.
>>>>>>>
>>>>>>>  From that view, it no longer makes sense to discuss "the plugin," or
>>>>>>> gold, or $AR, because there isn't just one of any of those things.
>>>>>>> ld64
>>>>>>> isn't the only outlier linker to consider. We have our own linker at
>>>>>>> Sony,
>>>>>>> for example. From this perspective, then it makes more sense to
>>>>>>> consider
>>>>>>> replacing the binary utilities with ones that support bitcode,
>>>>>>> because from
>>>>>>> a user-perspective, all of the linkers already transparently support
>>>>>>> bitcode
>>>>>>> directly today, as do ar, nm, etc. This has been necessary for the
>>>>>>> regular
>>>>>>> LTO process.
>>>>>>
>>>>>> Hi Alex,
>>>>>>
>>>>>> It's true that the LLVM versions of these tools support bitcode
>>>>>> transparently, but not all build systems use LLVM versions of these
>>>>>> tools, particularly build systems that support a variety of compilers,
>>>>>> or legacy build systems.
>>>>>
>>>>>
>>>>> If a build system can do
>>>>> CC=clang
>>>>> why wouldn't it be able to do
>>>>> AR=llvm-ar
>>>>> ?
>>>>
>>>> That assumes that the LLVM tools are all deployed in the build system,
>>>> and adds a requirement for using clang in this mode that wasn't there
>>>> before when using clang for -O2. We are trying to make the transition
>>>> from clang -O2 to clang -O2 + ThinLTO as seamless as possible.
>>>
>>>
>>> I'd just like the point out that downthread serious suggestions are being
>>> fielded to use a nonstandard ELF header or nonstandard bits marking the
>>> header. This "adds a requirement".
>>>
>>> At this point at least 1 (the only?) concrete deployment use case
>>> (besides
>>> yours) that has been brought up in the ThinLTO RFC threads is
>>> inconvenienced
>>> by this design decision. This suggests that native object wrapping
>>> doesn't
>>> offer as much seamlessness as it seems.
>>
>> One big goal is to make it as painless as possible to transition from
>> plain -O2 to -O2+thinlto. Users of clang who don't already use LTO
>> have not had to use/deploy llvm versions of all of these tools
>> (llvm-nm, llvm-objcopy, llvm-ar, llvm-ranlib, etc), or the plugins for
>> the native versions of these tools, because they weren't dealing with
>> bitcode files. That is why we are prioritizing the native wrapped
>> approach for the initial implementation. For users of the gold linker
>> (which uses the LTOModule interfaces) this should make it much easier
>> to enable ThinLTO (from my browsing of ld64 source, it looks like
>> native-wrapped bitcode should be handled already there too due to the
>> handling being hidden behind the lto_module API).
>>
>> Teresa
>
> Quick question: Is the word required to support ThinLTO using llvm's native
> tools orthogonal to that required to supporting non-llvm tools?  If not,
> would it make sense to start with a deployment of entirely LLVM based tools
> - since there seems to be general interest in that - and then come back to
> the non-llvm based tools separately?
>
> Personally, I see both sides here.  I can understand why you want to
> minimize build configuration changes - they tend to be painful - but I also
> am reluctant to design a major enhancement to LLVM under the assumption that
> LLVM's own tools aren't adequate for the purpose. That seems like it would
> be majorly problematic from the perspective of the project as a whole.
>
> (I realize that LLVM's tools could simply extract the bitcode out of the
> wrapper file, but that seems unnecessarily complex for an *initial* LLVM
> only solution.)

Hi Phillip,

You are right, we can hide most of the details of the input format
behind an API, and make most of the ThinLTO implementation blissfully
unaware of what format was used. Given that, I am working on figuring
out what the right API is first.

One other thing about the bitcode-only approach is that we need to
define how the function index etc will be represented in bitcode,
which will presumably require some IR changes. So in order to test the
distributed build integration and deploy more quickly, I was thinking
of the following proposal:
1) Define API first
2) Define and implement native object format support (specifically ELF
first since I need that to test the distributed build integration)
3) While 2 is in progress, propose bitcode extensions to handle the
ThinLTO info in bitcode
4) Implement bitcode writing/reading support for ThinLTO sections
5) Implement support for other native-object formats: COFF/MachO

Interspersed with 2-5 is the rest of the ThinLTO support that is
independent of the format as it uses the API from 1.

Thanks,
Teresa

>
> Philip
>



-- 
Teresa Johnson | Software Engineer | tejohnson at google.com | 408-460-2413