[llvm-dev] LLD status update and performance chart

Sean Silva via llvm-dev llvm-dev at lists.llvm.org
Sat Dec 17 18:54:39 PST 2016 

On Sat, Dec 17, 2016 at 6:45 PM, Mehdi Amini <mehdi.amini at apple.com> wrote:

>
> On Dec 17, 2016, at 6:32 PM, Sean Silva via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>
>
> On Fri, Dec 16, 2016 at 12:31 PM, Pete Cooper via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>>
>> On Dec 16, 2016, at 11:46 AM, Rui Ueyama <ruiu at google.com> wrote:
>>
>> On Fri, Dec 16, 2016 at 11:18 AM, Pete Cooper <peter_cooper at apple.com>
>> wrote:
>>
>>> Hi Rui
>>>
>>> I agree separating the components out in to libraries only makes sense
>>> when there is a clear reason to do so.  However, just this year there was a
>>> very involved discussion about what it means to be a library.
>>> Specifically, I don't think your current 'main-as-library' argument is
>>> valid while you call exit or (if you) rely on mutable global state.  Having
>>> a single entry point via a main function is fine, but that function cannot
>>> then kill the process which its linked in to.
>>>
>>
>> Our main function returns as long as input object files are not
>> corrupted. If you are doing in-memory linking, I think it is unlikely that
>> the object files in memory are corrupted (especially when you just created
>> them using LLVM), so I think this satisfies most users needs in practice.
>> Do you have a concern about that?
>>
>> Ultimately my concern is that there is *any* code path calling exit.  I
>> would say that this prevents the lld library from being used in-process.
>> But others opinions may differ, and I honestly don't have a use case in
>> mind, just that I don't think library code should ever call exit.
>>
>
> I agreed with the sentiment at first, but after thinking about it for a
> while, I actually have convinced myself that it doesn't hold water under
> closer inspection.
>
> The fundamental thing is that the LLVM libraries actually do have tons of
> fatal errors; they're just in the form of assert's (or we'll dereference a
> null pointer, or run off the end of a data structure, or go into an
> infinite loop, etc.).
>
> If you pass a corrupted Module to LLVM through the library API, you can
> certainly trip tons of "fatal errors" (in the form of failed assertions or
> UB). The way that LLVM gets around this is by having a policy of "if you
> pass it corrupted Module that doesn't pass the verifier, it's your fault,
> you're using our API wrong". Why can't an LLD library API have that same
> requirement?
>
> If it is safe for clang to uses the LLVM library API without running the
> verifier as its default configuration for non-development builds, why would
> it be unsafe for (say) clang to pass an object file directly to LLD as a
> library without verification?
>
>
> It would be OK, but in practice this is not what happens, does it?
>

That is what happens in practice. Also, even if you opt-in to the verifier
(which is enabled by an internal option), it uses fatal error handling to
report the verifier error.

-- Sean Silva


> We run the IR verifier on every LTO input for instance.
>
> —
> Mehdi
>
>
> Like Rui said, it's absolutely possible to create a verifier pass for LLD;
> it just hasn't been written because most object files we've seen so far
> seem to come from a small number of well-tested codepaths that always (as
> in the `assert` meaning of "always") create valid ELF files. In fact, we've
> added graceful recovery as appropriate (e.g. r259831), which is a step
> above error handling!
>
> Also, I'd like to point out that Clang, even when it does run the LLVM
> verifier (which is not the default except in development builds), runs it
> with fatal error handling. Is anybody aware of a program that uses LLVM as
> a library, produces IR in memory, runs the verifier, and does not simply
> abort if the verifier fails in non-development builds?
>
> -- Sean Silva
>
>
>>
>> For the situation that you need to handle foreign object files in the
>> same process (I'd recommend you to sandbox a process in that case though),
>> we can write a verifier to check for file correctness rigorously so that we
>> can guarantee that object files are as trustworthy as freshly-created
>> object files. I think this feature is a reasonable addition to the linker.
>>
>> That sounds great.  Having written some parts of the MachO lld linker and
>> seen Kevin's work on llvm-objdump, I can appreciate that is not easy.  For
>> example, I wrote the logic to process EH FDE's which may need to error out
>> if invalid.  You don't necessarily want to validate them all up front as it
>> may be too slow, so I can understand that this isn't necessarily trivial to
>> handle in a performant way.
>>
>>
>> As to the mutable shared state, my current (unproved) idea is to make
>> them thread local variables. Since no one yet has come up to say "hey, we
>> are actually trying to run multiple instances of the linker in the same
>> process simultaneously but LLD doesn't allow that", that's not implemented
>> yet, but technically I think it's doable, and that's needless to say a
>> reasonable feature request.
>>
>> LLVM uses the LLVMContext for this (and begs users to look the other way
>> with regards to cl::opt's).  I don't know if there's been a discussion in
>> LLVM about whether TLV's would be better there too, but seems like a
>> reasonable discussion to have.  Certainly I don't think anyone should say
>> you can't use them without good reason.
>>
>>
>> As I repeatedly said in the thread that speed is not the only goal for
>> us. Honestly, it's going to be the best selling point of LLD, because most
>> people do not use that many linker features but just use it to create
>> executables (and sometimes wait for a long period of time). I reported
>> about the performance in this thread because I thought people would be
>> happy to hear the speed improvement we've made this year. Also, because I
>> was happy about that, I probably emphasized that too much. But that's not
>> our single goal.
>>
>> I meant to commend you for both sending out a summary email, and the
>> results.  Having this fast a linker on ELF/COFF is going to be a huge win
>> for developers.  And I personally really like status updates for major
>> projects/features as it can be hard to follow along with all the email
>> traffic.  So thank you for doing that.
>>
>> My only concern with performance is that I felt like you would be against
>> changes to the code which make it slower but add functionality.  Error
>> handling is such a use case.  LLVM and clang continue to get bigger each
>> year and sometimes that means a little slower too.  The linker may be
>> faster next year than it is now, or it may be slower but have a feature
>> which makes that a worthwhile tradeoff.  I don't want to slow down any of
>> the code for any reason, but its natural that sometimes it'll happen with
>> good reason.
>>
>> Thanks,
>> Pete
>>
>>
>> If you want  context then the relevant piece of the thread is
>>> http://lists.llvm.org/pipermail/llvm-dev/2016-January/093760.html.
>>>
>>> Arseny summarized things very well there, so i'll just quote him at the
>>> end here.  I understand that you and others want to first write a fast
>>> linker tool and i don't think anyone has any problem with that, but there
>>> is also a clear desire from folks to have it be usable as a library and I
>>> would hope any patches to do so are accepted, even if they make the code
>>> more complex, or slower.
>>>
>>> >>>* On Thu, Jan 7, 2016 at 7:03 AM, Arseny Kapoulkine via llvm-dev <
>>> *>>>* llvm-dev at lists.llvm.org <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>> wrote:
>>> *>>>>>>>* In the process of migrating from old lld ELF linker to new (previously
>>> *>>>>* ELF2) I noticed the interface lost several important features (ordered by
>>> *>>>>* importance for my use case):
>>> *>>>>>>>>* 1. Detecting errors in the first place. New linker seems to call
>>> *>>>>* exit(1) for any error.
>>> *>>>>>>>>* 2. Reporting messages to non-stderr outputs. Previously all link
>>> *>>>>* functions had a raw_ostream argument so it was possible to delay the error
>>> *>>>>* output, aggregate it for multiple linked files, output via a different
>>> *>>>>* format, etc.
>>> *>>>>>>>>* 3. Linking multiple outputs in parallel (useful for test drivers) in a
>>> *>>>>* single process. Not really an interface issue but there are at least two
>>> *>>>>* global pointers (Config & Driver) that refer to stack variables and are
>>> *>>>>* used in various places in the code.
>>> *>>>>>>>>* All of this seems to indicate a departure from the linker being useable
>>> *>>>>* as a library. To maintain the previous behavior you'd have to use a linker
>>> *>>>>* binary & popen.*
>>>
>>> Pete
>>>
>>> On Dec 16, 2016, at 10:15 AM, Rui Ueyama via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>> I talked several people and found that this is more like a communication
>>> issue rather than a technical/philosophical issue. I believe communication
>>> problems won't solve themselves. As a person who is on the owners file of
>>> LLD, I think I need to say something about that issue. Also, I guess people
>>> who were just watching this thread wondered why my happy pre-holiday status
>>> report suddenly turned into a heated discussion, and they are probably
>>> still wondering what's wrong with LLD. I want to address that, too.
>>>
>>> So, as a project, there is no anti-library policy in LLD. I think this
>>> is the misunderstanding one side had. We already provide main-as-a-library
>>> feature so that you can embed the linker to your program. We as a project
>>> welcome other ideas to export linker features at a well-defined boundary.
>>> For example, I think abstracting the file system access so that you can
>>> hook file operations could be a well-defined, useful API for those who want
>>> to do in-memory linking (I expressed that opinion already in this thread).
>>> Just like LLVM, we won't guarantee API compatibility between releases, and
>>> we are unlikely to be able to expose deep internals of the linker, but as
>>> long as you think you found a reasonable coarse API boundary, there should
>>> be nothing preventing you from bringing that to the table.
>>>
>>> On the other hand, as far as I talked, no one who is on the "library"
>>> side requested LLD expose deep internals. This is the misunderstanding the
>>> other side had. If we as a project said that LLD should not support any
>>> library interface at all, they would be upset and speak out loudly, but
>>> again, that's not a project policy.
>>>
>>> So, correct me if I'm wrong, but I don't see no serious conflicts here.
>>> The conflict I saw in the thread is I believe superficial, and I strongly
>>> believe that it could have been addressed calmly and nicely if we have used
>>> more words to explain thoughts instead of small number of strong words.
>>>
>>> Hope this helps.
>>>
>>> Rui
>>>
>>> On Fri, Dec 16, 2016 at 1:40 AM, George Rimar via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> >I am on PTO, so slow to respond.
>>>> >
>>>> >Some items that are left:
>>>> >
>>>> >* Debug fission
>>>> >* Single file debug fission
>>>> >* Range extension thunks
>>>> >* All of freebsd links and works
>>>> >* Very good performance when all that is in
>>>>
>>>> Looks we have initial version of debug fusion implemented.
>>>> r289790, r289810 commits from yesterday did the rest of main job I
>>>> believe.
>>>> I do not know what is "Single file debug fission" ? (quick googling
>>>> gives nothing and I never heard about that before I think)
>>>>
>>>> George.
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>>>
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20161217/b096c051/attachment.html>

More information about the llvm-dev mailing list