[LLVMdev] Updated RFC: ThinLTO Implementation Plan

[Philip Reames]

On 06/04/2015 07:43 AM, Teresa Johnson wrote:
> On Wed, Jun 3, 2015 at 1:29 PM, Sean Silva <chisophugis at gmail.com> wrote:
>>
>> On Mon, Jun 1, 2015 at 6:34 AM, Teresa Johnson <tejohnson at google.com> wrote:
>>> On Fri, May 29, 2015 at 6:15 PM, Sean Silva <chisophugis at gmail.com> wrote:
>>>>
>>>> On Fri, May 29, 2015 at 8:01 AM, Teresa Johnson <tejohnson at google.com>
>>>> wrote:
>>>>> On Fri, May 29, 2015 at 6:56 AM, Alex Rosenberg <alexr at leftfield.org>
>>>>> wrote:
>>>>>> My earlier statement about wrapping things in a native object file
>>>>>> held
>>>>>> in that it is controversial. It appears to be still central to your
>>>>>> design.
>>>>>>
>>>>>> It may help to look at the problem from a different viewpoint: LLVM
>>>>>> is
>>>>>> not a compiler. It is a framework that can be used to make
>>>>>> compiler-like
>>>>>> tools.
>>>>>>
>>>>>>  From that view, it no longer makes sense to discuss "the plugin," or
>>>>>> gold, or $AR, because there isn't just one of any of those things.
>>>>>> ld64
>>>>>> isn't the only outlier linker to consider. We have our own linker at
>>>>>> Sony,
>>>>>> for example. From this perspective, then it makes more sense to
>>>>>> consider
>>>>>> replacing the binary utilities with ones that support bitcode,
>>>>>> because from
>>>>>> a user-perspective, all of the linkers already transparently support
>>>>>> bitcode
>>>>>> directly today, as do ar, nm, etc. This has been necessary for the
>>>>>> regular
>>>>>> LTO process.
>>>>> Hi Alex,
>>>>>
>>>>> It's true that the LLVM versions of these tools support bitcode
>>>>> transparently, but not all build systems use LLVM versions of these
>>>>> tools, particularly build systems that support a variety of compilers,
>>>>> or legacy build systems.
>>>>
>>>> If a build system can do
>>>> CC=clang
>>>> why wouldn't it be able to do
>>>> AR=llvm-ar
>>>> ?
>>> That assumes that the LLVM tools are all deployed in the build system,
>>> and adds a requirement for using clang in this mode that wasn't there
>>> before when using clang for -O2. We are trying to make the transition
>>> from clang -O2 to clang -O2 + ThinLTO as seamless as possible.
>>
>> I'd just like the point out that downthread serious suggestions are being
>> fielded to use a nonstandard ELF header or nonstandard bits marking the
>> header. This "adds a requirement".
>>
>> At this point at least 1 (the only?) concrete deployment use case (besides
>> yours) that has been brought up in the ThinLTO RFC threads is inconvenienced
>> by this design decision. This suggests that native object wrapping doesn't
>> offer as much seamlessness as it seems.
> One big goal is to make it as painless as possible to transition from
> plain -O2 to -O2+thinlto. Users of clang who don't already use LTO
> have not had to use/deploy llvm versions of all of these tools
> (llvm-nm, llvm-objcopy, llvm-ar, llvm-ranlib, etc), or the plugins for
> the native versions of these tools, because they weren't dealing with
> bitcode files. That is why we are prioritizing the native wrapped
> approach for the initial implementation. For users of the gold linker
> (which uses the LTOModule interfaces) this should make it much easier
> to enable ThinLTO (from my browsing of ld64 source, it looks like
> native-wrapped bitcode should be handled already there too due to the
> handling being hidden behind the lto_module API).
>
> Teresa
Quick question: Is the word required to support ThinLTO using llvm's 
native tools orthogonal to that required to supporting non-llvm tools?  
If not, would it make sense to start with a deployment of entirely LLVM 
based tools - since there seems to be general interest in that - and 
then come back to the non-llvm based tools separately?

Personally, I see both sides here.  I can understand why you want to 
minimize build configuration changes - they tend to be painful - but I 
also am reluctant to design a major enhancement to LLVM under the 
assumption that LLVM's own tools aren't adequate for the purpose. That 
seems like it would be majorly problematic from the perspective of the 
project as a whole.

(I realize that LLVM's tools could simply extract the bitcode out of the 
wrapper file, but that seems unnecessarily complex for an *initial* LLVM 
only solution.)

Philip