[llvm-dev] RFC: LTO should use -disable-llvm-verifier

Tue Sep 8 10:52:29 PDT 2015

> On 2015-Sep-04, at 15:53, Peter Collingbourne via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> On Fri, Sep 04, 2015 at 03:11:39PM -0700, Mehdi Amini wrote:
>> 
>>> On Sep 4, 2015, at 11:38 AM, Peter Collingbourne <peter at pcc.me.uk> wrote:
>>> 
>>> On Fri, Sep 04, 2015 at 11:13:43AM -0700, Mehdi Amini wrote:
>>>> 
>>>>> On Sep 4, 2015, at 11:03 AM, Eric Christopher <echristo at gmail.com> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>> On Fri, Sep 4, 2015 at 12:48 AM Mehdi Amini <mehdi.amini at apple.com <mailto:mehdi.amini at apple.com>> wrote:
>>>>>> On Sep 4, 2015, at 12:22 AM, Eric Christopher <echristo at gmail.com <mailto:echristo at gmail.com>> wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Thu, Sep 3, 2015 at 11:45 PM Mehdi Amini <mehdi.amini at apple.com <mailto:mehdi.amini at apple.com>> wrote:
>>>>>> Hi,
>>>>>> 
>>>>>>> On Sep 2, 2015, at 7:31 PM, Peter Collingbourne via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>>>>>> 
>>>>>>> On Thu, Sep 03, 2015 at 01:10:42AM +0000, Eric Christopher wrote:
>>>>>>>> On Tue, Sep 1, 2015 at 10:43 AM Duncan P. N. Exon Smith <
>>>>>>>> dexonsmith at apple.com <mailto:dexonsmith at apple.com>> wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On 2015-Aug-31, at 18:09, Eric Christopher <echristo at gmail.com <mailto:echristo at gmail.com>> wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Mon, Aug 31, 2015 at 5:50 PM Duncan P. N. Exon Smith <
>>>>>>>>> dexonsmith at apple.com <mailto:dexonsmith at apple.com>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> On 2015-Aug-31, at 12:21, Eric Christopher <echristo at gmail.com <mailto:echristo at gmail.com>> wrote:
>>>>>>>>>>> Yep. This is where I was going :)
>>>>>>>>>> 
>>>>>>>>>> Glad I found consensus, but I want to double-check that this makes
>>>>>>>>>> sense to add to the driver.  I didn't quite think through the
>>>>>>>>>> implications myself.
>>>>>>>>>> 
>>>>>>>>>> Since the driver doesn't know if there's any bitcode, or if LTO is
>>>>>>>>>> going to be invoked, it seems like I'll have to change the noasserts
>>>>>>>>>> driver to *always* pass the option to the linker just in case we are
>>>>>>>>>> doing LTO.  Is this reasonable?
>>>>>>>>>> 
>>>>>>>>>> Also, I realized that passing `-mllvm -disable-llvm-verifier` to ld64
>>>>>>>>>> is redundant... so I'm thinking `-mllvm -disable-verify`.  Make
>>>>>>>>>> sense?
>>>>>>>>>> 
>>>>>>>>>> *sigh* Reasons to hate the driver interface again...
>>>>>>>>>> 
>>>>>>>>>> I guess this is ok. Could possibly add it to the existing terrible
>>>>>>>>> "enable this pass" interface on liblto as well.
>>>>>>>>> 
>>>>>>>>> The linker doesn't know whether clang was built with asserts, though.
>>>>>>>>> 
>>>>>>>>> We could just make it implicit: move the decision to libLTO itself.  Given
>>>>>>>>> that clang and libLTO.dylib are different executables anyway -- and you
>>>>>>>>> might be interposing an asserts libLTO.dylib to use with an installed clang
>>>>>>>>> -- maybe it's even better?
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> *nod* We could do that. Seems better than the alternative.
>>>>>>> 
>>>>>>> +1
>>>>>>> 
>>>>>>>>>> I don't suppose ld64 could move to a model like we're talking about with
>>>>>>>>> lld that pcc is working on?
>>>>>>>>> 
>>>>>>>>> What specifically?
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Ah, using the C++ interface to handle everything and not using libLTO at
>>>>>>>> all.
>>>>>>>> 
>>>>>>>> He can speak more to this though.
>>>>>>> 
>>>>>>> The C++ interface is much more convenient for a C++ program to use, but
>>>>>>> clients need to revlock themselves to LLVM in order to use it.
>>>>>>> 
>>>>>>> In theory the same does not apply to libLTO, but we do end up changing it
>>>>>>> occasionally to add new APIs which ld64 promptly starts using, so it isn't
>>>>>>> clear how much ld64 gains by relying on libLTO,
>>>>>> 
>>>>>> Backward/Forward compatibility?
>>>>>> Drop any version of clang/libLTO and still be able to use the system provided linker on any version of OS X?
>>>>>> Sounds like a valuable feature to me, which is what I believe was (is?) sought by the C API in general (but that’s another story).
>>>>>> 
>>>>>> 
>>>>>> Static linking against the parts of llvm you care about? There's nothing in the ld64 build system that means it can't do this :)
>>>>> 
>>>>> I’m not sure how is that supposed to work?
>>>>> I drop a new clang/libLTO on the system, clang generates bitcode, and libLTO handles it.
>>>>> 
>>>>> If ld64 was statically linked against LLVM, it couldn’t read the new bitcode right?
>>>>> 
>>>>> 
>>>>> Well sure, that's what Peter meant when he said revlocked. :)
>>>> 
>>>> I’m not sure we’re talking about the same thing, I may have misunderstood his point (“In theory the same does not apply to libLTO, but we do end up changing it
>>>> occasionally to add new APIs which ld64 promptly starts using, so it isn’t clear how much ld64 gains by relying on libLTO”) ; my take from that was that since the new ld64 starts using the new libLTO interface, there is a revlock anyway and then there is no advantage in keeping the old libLTO entry points.
>>>> I just described a use case that shows how useful it can be.
>>>> 
>>>> Apologize if I misunderstood the point.
>>> 
>>> That seems like a potentially valid use case, but if it doesn't actually
>>> work right now it doesn't really buy you much.
>> 
>> It is working right now AFAIK. At least my understand is that the current situation is that you can drop a clang/libLTO built today on a machine that has ld64 from a few years ago and it will work.

BTW, a brand new ld64 will work with an old libLTO.dylib, too.  I'm
not aware of any revlock at all.

> Is there anyone who actually needs to do this? Why can't they upgrade
> their linker?

While it's fairly easy for external people (like you) to compile a
new version of clang/libLTO.dylib, without the ld64 sources it would
be hard to compile a new version of the linker.  At the least, ld64's
use of the stable LTO C API enables compiler hackers to hack away.

We take advantage of this internally, since it gives us freedom to
stage ld64 and clang releases independently.

(Once we bring up lld, I think the story can be quite different.)