<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Thu, Jan 7, 2016 at 5:19 PM Philip Reames via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
<br>
<br>
<div>On 01/07/2016 05:12 PM, Chandler
Carruth via llvm-dev wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr">On Thu, Jan 7, 2016 at 4:05 PM Rui Ueyama <<a href="mailto:ruiu@google.com" target="_blank"></a><a href="mailto:ruiu@google.com" target="_blank">ruiu@google.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">By organizing it as a library, I'm expecting
something coarse. I don't expect to reorganize the linker
itself as a collection of small libraries, but make the
entire linker available as a library, so that you can link
stuff in-process. More specifically, I expect that the
library would basically export one function,
link(std::vector<StringRef>), which takes command
line arguments, and returns a memory buffer for a newly
created executable. We may want to allow a mix of
StringRef and MemoryBuffer as input, so that you can
directly pass in-memory objects to the linker, but the
basic idea remains the same.
<div><br>
</div>
<div>Are we on the same page?</div>
</div>
</blockquote>
<div><br>
</div>
<div>Let me answer this below, where I think you get to the
core of the problem.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra"><br>
</div>
</div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Thu, Jan 7, 2016 at 3:44 PM,
Chandler Carruth <span dir="ltr"><<a href="mailto:chandlerc@gmail.com" target="_blank"></a><a href="mailto:chandlerc@gmail.com" target="_blank">chandlerc@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_quote"><span>
<div dir="ltr">On Thu, Jan 7, 2016 at 3:18 PM
Rui Ueyama <<a href="mailto:ruiu@google.com" target="_blank">ruiu@google.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Thu, Jan 7,
2016 at 2:56 PM, Chandler Carruth <span dir="ltr"><<a href="mailto:chandlerc@gmail.com" target="_blank"></a><a href="mailto:chandlerc@gmail.com" target="_blank">chandlerc@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_quote"><span>
<div dir="ltr">On Thu, Jan 7,
2016 at 7:18 AM Rui Ueyama
via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank"></a><a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">On
Thu, Jan 7, 2016 at 7:03
AM, Arseny Kapoulkine
via llvm-dev <span dir="ltr"><<a href="mailto:llvm-dev@lists.llvm.org" target="_blank"></a><a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>></span>
wrote:<br>
</div>
</div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">In
the process of
migrating from old
lld ELF linker to
new (previously
ELF2) I noticed
the interface lost
several important
features (ordered
by importance for
my use case):
<div><br>
</div>
<div>
<div>1.
Detecting
errors in the
first place.
New linker
seems to call
exit(1) for
any error.</div>
</div>
<div><br>
</div>
<div>2. Reporting
messages to
non-stderr
outputs.
Previously all
link functions
had
a raw_ostream
argument so it
was possible to
delay the error
output,
aggregate it for
multiple linked
files, output
via a different
format, etc.</div>
<div><br>
</div>
<div>3. Linking
multiple outputs
in parallel
(useful for test
drivers) in a
single process.
Not really an
interface issue
but there are at
least two global
pointers (Config
& Driver)
that refer to
stack variables
and are used in
various places
in the code.</div>
<div><br>
</div>
<div>All of this
seems to
indicate a
departure from
the linker being
useable as a
library. To
maintain the
previous
behavior you'd
have to use a
linker binary
& popen.</div>
<div><br>
</div>
<div>Is this a
conscious design
decision or a
temporary
limitation?</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>That the new ELF
and COFF linkers are
designed as commands
instead of libraries
is very much an
intended design
change.</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>I disagree.</div>
<div><br>
</div>
<div>During the discussion,
there was a *specific*
discussion of both the new
COFF port and ELF port
continuing to be libraries
with a common command line
driver.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>There was a discussion that we
would keep the same entry point for
the old and the new, but I don't
remember if I promised that we were
going to organize the new linker as
a library.</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>Ok, myself and essentially everyone else
thought this was clear. If it isn't lets
clarify:</div>
<div><br>
</div>
<div>I think it is absolutely critical and
important that LLD's architecture remain one
where all functionality is available as a
library. This is *the* design goal of LLVM and
all of LLVM's infrastructure. This applies
just as much to LLD as it does to Clang.</div>
<div><br>
</div>
<div>You say that it isn't compelling to match
Clang's design, but in fact it is. You would
need an overwhelming argument to *diverge*
from Clang's design.</div>
<div><br>
</div>
<div>The fact that it makes the design more
challenging is not compelling at all. Yes,
building libraries that can be re-used and
making the binary calling it equally efficient
is more challenging, but that is the express
mission of LLVM and every project within it.</div>
<span>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> The new one is designed as a
command from day one. (Precisely
speaking, the original code
propagates errors all the way up to
the entry point, so you can call it
and expect it to always return.
Rafael introduced error() function
later and we now depends on that
function does not return.)</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>I think this last was a mistake.</div>
<div><br>
</div>
<div>The fact that the code propagates errors
all the way up is fine, and even good. We
don't necessarily need to be able to *recover*
from link errors and try some other path.</div>
<div><br>
</div>
<div>But we absolutely need the design to be a
*library* that can be embedded into other
programs and tools. I can't even begin to
count the use cases for this.</div>
<div><br>
</div>
<div>So please, let's go back to where we *do
not* rely on never-returning error handling.
That is an absolute mistake.</div>
<span>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_quote">
<div>If you want to consider
changing that, we should have
a fresh (and broad)
discussion, but it goes pretty
firmly against the design of
the entire LLVM project. I
also don't really understand
why it would be beneficial.</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<div dir="ltr">
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">I'm not against
organizing it as a library as long as it
does not make things too complicated</div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>I am certain that it will make things more
complicated, but that is the technical
challenge that we must overcome. It will be
hard, but I am absolutely confident it is
possible to have an elegant library design
here. It may not be as simple as a pure
command line tool, but it will be
*dramatically* more powerful, general, and
broadly applicable.</div>
<div><br>
</div>
<div>The design of LLVM is not the simplest way
to build a compiler. But it is valuable to all
of those working on it precisely because of
this flexibility imparted by its library
oriented design. This is absolutely not
something that we should lose from the linker.</div>
<span>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">, and I guess
reorganizing the existing code as a
library is relatively easy because it's
still pretty small, but I don't really
want to focus on that until it becomes
usable as an alternative to GNU ld or
gold. I want to focus on the linker
features themselves at this moment. Once
it's complete, it becomes more clear how
to organize it.</div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>Ok, now we're talking about something
totally reasonable.</div>
<div><br>
</div>
<div>If it is easier for you all to develop this
first as a command line tool, and then make it
work as a library, sure, go for it. You're
doing the work, I can hardly tell you how to
go about it. ;]</div>
</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>It is not only easier for me to develop but is
also super important for avoiding over-designing the
API of the library. Until we know what we need to do
and what can be done, it is too easy to make mistake
to design API that is supposed to cover everything
-- including hypothetical unrealistic ones. Such API
would slow down the development speed significantly,
and it's a pain when we abandon that when we realize
that that was not needed.</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I'm very sympathetic to the problem of not wanting to
design an API until the concrete use cases for it appear.
That makes perfect sense.</div>
<div><br>
</div>
<div>We just need to be *ready* to extend the library API (and
potentially find a more fine grained layering if one is
actually called for) when a reasonable and real use case
arises for some users of LLD. Once we have people that
actually have a use case and want to introduce a certain
interface to the library that supports it, we need to work
with them to figure out how to effectively support their use
case.</div>
<div><br>
</div>
<div>At the least, we clearly need the super simple
interface[1] that the command line tool would use, but an
in-process linker could also probably use.</div>
<div><br>
</div>
<div>We might need minor extensions to effectively support
Arseny's use case (I think an in-process linker is a *very*
reasonable thing to support, I'd even like to teach the
Clang driver to optionally work that way to be more
efficient on platforms like Windows). But I have to imagine
that the interface for an in-process static linker and the
command line linker are extremely similar if not precisely
the same.</div>
<div><br>
</div>
<div>At some point, it might also make sense to support more
interesting linking scenarios such as linking a PIC "shared
object" that can be mapped into the running process for JIT
users. But I think it is reasonable to build the interface
that those users need when those users are ready to leverage
LLD. That way we can work with them to make sure we don't
build the wrong interface or an overly complicated one (as
you say).</div>
</div>
</div>
</blockquote></div><div text="#000000" bgcolor="#FFFFFF">
I don't disagree with anything Chandler said, but it's worth noting
that we *already* have a specialized in-process linker used to MCJIT
to resolve relocations and patch things like symbolic calls. It'd
be really really nice if the new linker library supported that use
case. <br></div><div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div><br></div></div></div></blockquote></div></blockquote><div>This is, in fact, the goal that Dave and I mentioned :)</div><div><br></div><div>Lang and I have been talking about this wrt MCJIT for a long time.</div><div><br></div><div>-eric</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><blockquote type="cite"><div dir="ltr"><div class="gmail_quote"><div>
</div>
<div>Make sense?</div>
<div>-Chandler</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
LLVM Developers mailing list
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div></div>