[llvm-dev] LLD status update and performance chart

Sat Dec 17 19:43:29 PST 2016

On Sat, Dec 17, 2016 at 6:56 PM, Mehdi Amini <mehdi.amini at apple.com> wrote:

>
> On Dec 17, 2016, at 6:54 PM, Sean Silva <chisophugis at gmail.com> wrote:
>
>
>
> On Sat, Dec 17, 2016 at 6:45 PM, Mehdi Amini <mehdi.amini at apple.com>
> wrote:
>
>>
>> On Dec 17, 2016, at 6:32 PM, Sean Silva via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>
>>
>> On Fri, Dec 16, 2016 at 12:31 PM, Pete Cooper via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>>
>>> On Dec 16, 2016, at 11:46 AM, Rui Ueyama <ruiu at google.com> wrote:
>>>
>>> On Fri, Dec 16, 2016 at 11:18 AM, Pete Cooper <peter_cooper at apple.com>
>>> wrote:
>>>
>>>> Hi Rui
>>>>
>>>> I agree separating the components out in to libraries only makes sense
>>>> when there is a clear reason to do so.  However, just this year there was a
>>>> very involved discussion about what it means to be a library.
>>>> Specifically, I don't think your current 'main-as-library' argument is
>>>> valid while you call exit or (if you) rely on mutable global state.  Having
>>>> a single entry point via a main function is fine, but that function cannot
>>>> then kill the process which its linked in to.
>>>>
>>>
>>> Our main function returns as long as input object files are not
>>> corrupted. If you are doing in-memory linking, I think it is unlikely that
>>> the object files in memory are corrupted (especially when you just created
>>> them using LLVM), so I think this satisfies most users needs in practice.
>>> Do you have a concern about that?
>>>
>>> Ultimately my concern is that there is *any* code path calling exit.  I
>>> would say that this prevents the lld library from being used in-process.
>>> But others opinions may differ, and I honestly don't have a use case in
>>> mind, just that I don't think library code should ever call exit.
>>>
>>
>> I agreed with the sentiment at first, but after thinking about it for a
>> while, I actually have convinced myself that it doesn't hold water under
>> closer inspection.
>>
>> The fundamental thing is that the LLVM libraries actually do have tons of
>> fatal errors; they're just in the form of assert's (or we'll dereference a
>> null pointer, or run off the end of a data structure, or go into an
>> infinite loop, etc.).
>>
>> If you pass a corrupted Module to LLVM through the library API, you can
>> certainly trip tons of "fatal errors" (in the form of failed assertions or
>> UB). The way that LLVM gets around this is by having a policy of "if you
>> pass it corrupted Module that doesn't pass the verifier, it's your fault,
>> you're using our API wrong". Why can't an LLD library API have that same
>> requirement?
>>
>> If it is safe for clang to uses the LLVM library API without running the
>> verifier as its default configuration for non-development builds, why would
>> it be unsafe for (say) clang to pass an object file directly to LLD as a
>> library without verification?
>>
>>
>> It would be OK, but in practice this is not what happens, does it?
>>
>
> That is what happens in practice.
>
>
> I was talking about "clang to pass an object file directly to LLD”
>

Ah, sorry. I misunderstood what you were saying. Currently, no. But my post
above was primarily talking about the in-memory use case; I was just using
clang passing object files through memory to LLD as a hypothetical since
that has come up before.

-- Sean Silva

>
> —
> Mehdi
>
>
>
> Also, even if you opt-in to the verifier (which is enabled by an internal
> option), it uses fatal error handling to report the verifier error.
>
> -- Sean Silva
>
>
>> We run the IR verifier on every LTO input for instance.
>>
>> —
>> Mehdi
>>
>>
>> Like Rui said, it's absolutely possible to create a verifier pass for
>> LLD; it just hasn't been written because most object files we've seen so
>> far seem to come from a small number of well-tested codepaths that always
>> (as in the `assert` meaning of "always") create valid ELF files. In fact,
>> we've added graceful recovery as appropriate (e.g. r259831), which is a
>> step above error handling!
>>
>> Also, I'd like to point out that Clang, even when it does run the LLVM
>> verifier (which is not the default except in development builds), runs it
>> with fatal error handling. Is anybody aware of a program that uses LLVM as
>> a library, produces IR in memory, runs the verifier, and does not simply
>> abort if the verifier fails in non-development builds?
>>
>> -- Sean Silva
>>
>>
>>>
>>> For the situation that you need to handle foreign object files in the
>>> same process (I'd recommend you to sandbox a process in that case though),
>>> we can write a verifier to check for file correctness rigorously so that we
>>> can guarantee that object files are as trustworthy as freshly-created
>>> object files. I think this feature is a reasonable addition to the linker.
>>>
>>> That sounds great.  Having written some parts of the MachO lld linker
>>> and seen Kevin's work on llvm-objdump, I can appreciate that is not easy.
>>> For example, I wrote the logic to process EH FDE's which may need to error
>>> out if invalid.  You don't necessarily want to validate them all up front
>>> as it may be too slow, so I can understand that this isn't necessarily
>>> trivial to handle in a performant way.
>>>
>>>
>>> As to the mutable shared state, my current (unproved) idea is to make
>>> them thread local variables. Since no one yet has come up to say "hey, we
>>> are actually trying to run multiple instances of the linker in the same
>>> process simultaneously but LLD doesn't allow that", that's not implemented
>>> yet, but technically I think it's doable, and that's needless to say a
>>> reasonable feature request.
>>>
>>> LLVM uses the LLVMContext for this (and begs users to look the other way
>>> with regards to cl::opt's).  I don't know if there's been a discussion in
>>> LLVM about whether TLV's would be better there too, but seems like a
>>> reasonable discussion to have.  Certainly I don't think anyone should say
>>> you can't use them without good reason.
>>>
>>>
>>> As I repeatedly said in the thread that speed is not the only goal for
>>> us. Honestly, it's going to be the best selling point of LLD, because most
>>> people do not use that many linker features but just use it to create
>>> executables (and sometimes wait for a long period of time). I reported
>>> about the performance in this thread because I thought people would be
>>> happy to hear the speed improvement we've made this year. Also, because I
>>> was happy about that, I probably emphasized that too much. But that's not
>>> our single goal.
>>>
>>> I meant to commend you for both sending out a summary email, and the
>>> results.  Having this fast a linker on ELF/COFF is going to be a huge win
>>> for developers.  And I personally really like status updates for major
>>> projects/features as it can be hard to follow along with all the email
>>> traffic.  So thank you for doing that.
>>>
>>> My only concern with performance is that I felt like you would be
>>> against changes to the code which make it slower but add functionality.
>>> Error handling is such a use case.  LLVM and clang continue to get bigger
>>> each year and sometimes that means a little slower too.  The linker may be
>>> faster next year than it is now, or it may be slower but have a feature
>>> which makes that a worthwhile tradeoff.  I don't want to slow down any of
>>> the code for any reason, but its natural that sometimes it'll happen with
>>> good reason.
>>>
>>> Thanks,
>>> Pete
>>>
>>>
>>> If you want  context then the relevant piece of the thread is
>>>> http://lists.llvm.org/pipermail/llvm-dev/2016-January/093760.html.
>>>>
>>>> Arseny summarized things very well there, so i'll just quote him at the
>>>> end here.  I understand that you and others want to first write a fast
>>>> linker tool and i don't think anyone has any problem with that, but there
>>>> is also a clear desire from folks to have it be usable as a library and I
>>>> would hope any patches to do so are accepted, even if they make the code
>>>> more complex, or slower.
>>>>
>>>> >>>* On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via llvm-dev <
>>>> *>>>* llvm-dev at lists.llvm.org <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>> wrote:
>>>> *>>>>>>>* In the process of migrating from old lld ELF linker to new (previously
>>>> *>>>>* ELF2) I noticed the interface lost several important features (ordered by
>>>> *>>>>* importance for my use case):
>>>> *>>>>>>>>* 1. Detecting errors in the first place. New linker seems to call
>>>> *>>>>* exit(1) for any error.
>>>> *>>>>>>>>* 2. Reporting messages to non-stderr outputs. Previously all link
>>>> *>>>>* functions had a raw_ostream argument so it was possible to delay the error
>>>> *>>>>* output, aggregate it for multiple linked files, output via a different
>>>> *>>>>* format, etc.
>>>> *>>>>>>>>* 3. Linking multiple outputs in parallel (useful for test drivers) in a
>>>> *>>>>* single process. Not really an interface issue but there are at least two
>>>> *>>>>* global pointers (Config & Driver) that refer to stack variables and are
>>>> *>>>>* used in various places in the code.
>>>> *>>>>>>>>* All of this seems to indicate a departure from the linker being useable
>>>> *>>>>* as a library. To maintain the previous behavior you'd have to use a linker
>>>> *>>>>* binary & popen.*
>>>>
>>>> Pete
>>>>
>>>> On Dec 16, 2016, at 10:15 AM, Rui Ueyama via llvm-dev <
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>> I talked several people and found that this is more like a
>>>> communication issue rather than a technical/philosophical issue. I believe
>>>> communication problems won't solve themselves. As a person who is on the
>>>> owners file of LLD, I think I need to say something about that issue. Also,
>>>> I guess people who were just watching this thread wondered why my happy
>>>> pre-holiday status report suddenly turned into a heated discussion, and
>>>> they are probably still wondering what's wrong with LLD. I want to address
>>>> that, too.
>>>>
>>>> So, as a project, there is no anti-library policy in LLD. I think this
>>>> is the misunderstanding one side had. We already provide main-as-a-library
>>>> feature so that you can embed the linker to your program. We as a project
>>>> welcome other ideas to export linker features at a well-defined boundary.
>>>> For example, I think abstracting the file system access so that you can
>>>> hook file operations could be a well-defined, useful API for those who want
>>>> to do in-memory linking (I expressed that opinion already in this thread).
>>>> Just like LLVM, we won't guarantee API compatibility between releases, and
>>>> we are unlikely to be able to expose deep internals of the linker, but as
>>>> long as you think you found a reasonable coarse API boundary, there should
>>>> be nothing preventing you from bringing that to the table.
>>>>
>>>> On the other hand, as far as I talked, no one who is on the "library"
>>>> side requested LLD expose deep internals. This is the misunderstanding the
>>>> other side had. If we as a project said that LLD should not support any
>>>> library interface at all, they would be upset and speak out loudly, but
>>>> again, that's not a project policy.
>>>>
>>>> So, correct me if I'm wrong, but I don't see no serious conflicts here.
>>>> The conflict I saw in the thread is I believe superficial, and I strongly
>>>> believe that it could have been addressed calmly and nicely if we have used
>>>> more words to explain thoughts instead of small number of strong words.
>>>>
>>>> Hope this helps.
>>>>
>>>> Rui
>>>>
>>>> On Fri, Dec 16, 2016 at 1:40 AM, George Rimar via llvm-dev <
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> >I am on PTO, so slow to respond.
>>>>> >
>>>>> >Some items that are left:
>>>>> >
>>>>> >* Debug fission
>>>>> >* Single file debug fission
>>>>> >* Range extension thunks
>>>>> >* All of freebsd links and works
>>>>> >* Very good performance when all that is in
>>>>>
>>>>> Looks we have initial version of debug fusion implemented.
>>>>> r289790, r289810 commits from yesterday did the rest of main job I
>>>>> believe.
>>>>> I do not know what is "Single file debug fission" ? (quick googling
>>>>> gives nothing and I never heard about that before I think)
>>>>>
>>>>> George.
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20161217/122514af/attachment-0001.html>