[llvm-dev] RFC: LTO should use -disable-llvm-verifier

Fri Sep 4 15:11:39 PDT 2015

> On Sep 4, 2015, at 11:38 AM, Peter Collingbourne <peter at pcc.me.uk> wrote:
> 
> On Fri, Sep 04, 2015 at 11:13:43AM -0700, Mehdi Amini wrote:
>> 
>>> On Sep 4, 2015, at 11:03 AM, Eric Christopher <echristo at gmail.com> wrote:
>>> 
>>> 
>>> 
>>> On Fri, Sep 4, 2015 at 12:48 AM Mehdi Amini <mehdi.amini at apple.com <mailto:mehdi.amini at apple.com>> wrote:
>>>> On Sep 4, 2015, at 12:22 AM, Eric Christopher <echristo at gmail.com <mailto:echristo at gmail.com>> wrote:
>>>> 
>>>> 
>>>> 
>>>> On Thu, Sep 3, 2015 at 11:45 PM Mehdi Amini <mehdi.amini at apple.com <mailto:mehdi.amini at apple.com>> wrote:
>>>> Hi,
>>>> 
>>>>> On Sep 2, 2015, at 7:31 PM, Peter Collingbourne via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>>>> 
>>>>> On Thu, Sep 03, 2015 at 01:10:42AM +0000, Eric Christopher wrote:
>>>>>> On Tue, Sep 1, 2015 at 10:43 AM Duncan P. N. Exon Smith <
>>>>>> dexonsmith at apple.com <mailto:dexonsmith at apple.com>> wrote:
>>>>>> 
>>>>>>> 
>>>>>>>> On 2015-Aug-31, at 18:09, Eric Christopher <echristo at gmail.com <mailto:echristo at gmail.com>> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Mon, Aug 31, 2015 at 5:50 PM Duncan P. N. Exon Smith <
>>>>>>> dexonsmith at apple.com <mailto:dexonsmith at apple.com>> wrote:
>>>>>>>> 
>>>>>>>>> On 2015-Aug-31, at 12:21, Eric Christopher <echristo at gmail.com <mailto:echristo at gmail.com>> wrote:
>>>>>>>>> Yep. This is where I was going :)
>>>>>>>> 
>>>>>>>> Glad I found consensus, but I want to double-check that this makes
>>>>>>>> sense to add to the driver.  I didn't quite think through the
>>>>>>>> implications myself.
>>>>>>>> 
>>>>>>>> Since the driver doesn't know if there's any bitcode, or if LTO is
>>>>>>>> going to be invoked, it seems like I'll have to change the noasserts
>>>>>>>> driver to *always* pass the option to the linker just in case we are
>>>>>>>> doing LTO.  Is this reasonable?
>>>>>>>> 
>>>>>>>> Also, I realized that passing `-mllvm -disable-llvm-verifier` to ld64
>>>>>>>> is redundant... so I'm thinking `-mllvm -disable-verify`.  Make
>>>>>>>> sense?
>>>>>>>> 
>>>>>>>> *sigh* Reasons to hate the driver interface again...
>>>>>>>> 
>>>>>>>> I guess this is ok. Could possibly add it to the existing terrible
>>>>>>> "enable this pass" interface on liblto as well.
>>>>>>> 
>>>>>>> The linker doesn't know whether clang was built with asserts, though.
>>>>>>> 
>>>>>>> We could just make it implicit: move the decision to libLTO itself.  Given
>>>>>>> that clang and libLTO.dylib are different executables anyway -- and you
>>>>>>> might be interposing an asserts libLTO.dylib to use with an installed clang
>>>>>>> -- maybe it's even better?
>>>>>>> 
>>>>>>> 
>>>>>> *nod* We could do that. Seems better than the alternative.
>>>>> 
>>>>> +1
>>>>> 
>>>>>>>> I don't suppose ld64 could move to a model like we're talking about with
>>>>>>> lld that pcc is working on?
>>>>>>> 
>>>>>>> What specifically?
>>>>>> 
>>>>>> 
>>>>>> Ah, using the C++ interface to handle everything and not using libLTO at
>>>>>> all.
>>>>>> 
>>>>>> He can speak more to this though.
>>>>> 
>>>>> The C++ interface is much more convenient for a C++ program to use, but
>>>>> clients need to revlock themselves to LLVM in order to use it.
>>>>> 
>>>>> In theory the same does not apply to libLTO, but we do end up changing it
>>>>> occasionally to add new APIs which ld64 promptly starts using, so it isn't
>>>>> clear how much ld64 gains by relying on libLTO,
>>>> 
>>>> Backward/Forward compatibility?
>>>> Drop any version of clang/libLTO and still be able to use the system provided linker on any version of OS X?
>>>> Sounds like a valuable feature to me, which is what I believe was (is?) sought by the C API in general (but that’s another story).
>>>> 
>>>> 
>>>> Static linking against the parts of llvm you care about? There's nothing in the ld64 build system that means it can't do this :)
>>> 
>>> I’m not sure how is that supposed to work?
>>> I drop a new clang/libLTO on the system, clang generates bitcode, and libLTO handles it.
>>> 
>>> If ld64 was statically linked against LLVM, it couldn’t read the new bitcode right?
>>> 
>>> 
>>> Well sure, that's what Peter meant when he said revlocked. :)
>> 
>> I’m not sure we’re talking about the same thing, I may have misunderstood his point (“In theory the same does not apply to libLTO, but we do end up changing it
>> occasionally to add new APIs which ld64 promptly starts using, so it isn’t clear how much ld64 gains by relying on libLTO”) ; my take from that was that since the new ld64 starts using the new libLTO interface, there is a revlock anyway and then there is no advantage in keeping the old libLTO entry points.
>> I just described a use case that shows how useful it can be.
>> 
>> Apologize if I misunderstood the point.
> 
> That seems like a potentially valid use case, but if it doesn't actually
> work right now it doesn't really buy you much.

It is working right now AFAIK. At least my understand is that the current situation is that you can drop a clang/libLTO built today on a machine that has ld64 from a few years ago and it will work.

> 
> If this is indeed an abolute requirement, there are better ways of achieving
> this kind of thing without creating an API stability burden in LLVM. For
> example you could make libLTO a build product of ld64 rather than of LLVM by
> copying the tools/lto code into ld64 and using the preprocessor to make the
> code LLVM-version-dependent. This seems like it could be relatively easily
> achievable right now; all you'd need to do is switch the Xcode toolchain
> over to shipping the libLTO build product from ld64 instead of from LLVM
> (or even statically linking as Eric suggested). Any other existing clients
> of libLTO who care about this can do the same thing.

It seems I’m missing something: the system ld64 linker will be shipped with a self-contained libLTO, but the new compiler I’ll drop-in will generate “new" bitcode that will be fed to this “old” libLTO, and that’s not gonna play well.
Unless you are only including the C API wrapper in ld64, but in this case I’m not sure I understand "using the preprocessor to make the code LLVM-version-dependent”. You will be able to build ld64 independently of LLVM, but how about at runtime? The compiler needs to ship a dynamic library that will be loaded by the linker, and there is still a need for a stable API there.

— 
Mehdi

> Another advantage of doing this is that you can change the "stable" libLTO
> API without breaking clients, as ld64 would be the only client.
> 
> Thanks,
> -- 
> Peter