<div dir="ltr">On 13 August 2013 18:26, Shuxin Yang <span dir="ltr"><<a href="mailto:shuxin.llvm@gmail.com" target="_blank">shuxin.llvm@gmail.com</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div>
<br>
<div>On 8/13/13 5:32 PM, Nick Lewycky wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">On 12 August 2013 16:41, Shuxin Yang <span dir="ltr"><<a href="mailto:shuxin.llvm@gmail.com" target="_blank">shuxin.llvm@gmail.com</a>></span>
wrote:<br>
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Thank you very much for sharing you concerns. I
read this mail carefully, it seems we had little
miscommunications. <br>
I hope I clarify in this mail:-). See the interleaving
text. <br>
<div> <br>
<br>
On 8/12/13 3:41 PM, Nick Lewycky wrote:<br>
</div>
</div>
<div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div> >which in turn
drives libLTO through the
API.<br>
<br>
</div>
Depending on the what kind
of info "something" else
need to drive the libLTO. <br>
In general it is very bad
idea, if "something else"
need micro-management.</div>
</blockquote>
<div><br>
</div>
<div>libLTO is part of the
linker that uses it. </div>
</div>
</div>
</div>
</blockquote>
<br>
</div>
No! Absolutely not!</div>
</blockquote>
<div><br>
</div>
<div>Fair enough. I meant "libLTO is part of
the linker that uses it" in the same sense
that a networking library is part of the web
browser that uses it. The library shouldn't
be off deciding to do things of its own
accord, it should provide an API that allows
something else to accomplish its task.</div>
</div>
</div>
</div>
</blockquote>
</div>
I don't think such comparison is precise. For app with
networking lib, the "braid" reside at app side because
the app define the behavior. <br>
For linker+libLTO, I believe the "brain" should reside
at libLTO side, as it is far more complicate (although
our current implement is bit simple). <br>
</div>
</blockquote>
<div><br>
</div>
<div>I'm afraid I don't agree. We can agree to disagree on
this point, it isn't necessary for us to agree here to
make forwards progress.</div>
</div>
</div>
</div>
</blockquote></div>
Such discussion will be open-ended. I don't like to waste bandwidth
over here. It is not my focus<br>
at all. Maybe you can understand that point after you try to make
some infrastructure level <br>
change to the lto. If you OSX, you will see that. If you don't, grab
a Linux machine, try to implement <br>
that feature on Linux+gold but *WITHOUT* touching any bit the
tool/gold/*. <br></div></blockquote><div><br></div><div>Does <a href="http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20090202/073164.html" target="_blank">http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20090202/073164.html</a> count? tools/gold/* didn't exist when I started.</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><div><blockquote type="cite">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>From my point of view, the program that the user calls
is ld. That ld is responsible for fulfilling its promises
to the user, who does not know or care what libraries ld
is using under the hood. We've taken the direction that
we'll add new features to libLTO lazily, when there's a
demand from a linker that wants them, but that shouldn't
be confused for deliberately hiding functionality from the
linker.<br></div>
<div><br>
</div>
<div>And it is confusing because there *are* things that
libLTO is deliberately hiding: all the various changes in
LLVM's C++ API.</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">I believe GNU gold
is a good in designing the interface. In the case of
gold+libLTO, the "brain" is embodied by
"tool/gold/*.cpp."<br>
</div>
</blockquote>
<div><br>
</div>
<div>There's very little logic in tools/gold. It's either in
llvm proper, or in gold proper. Both libLTO and LLVMgold
are thin wrappers.</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Anybody can change
to whatever he/she like. Although it dose call APIs, it
dose not have to. I can directly call those c++ code <br>
dancing behind the API. This is the "stable API" I'm
talking about.
</div>
</blockquote>
<div><br>
</div>
<div>Okay. As a matter of terminology, I've been using
"stable" to mean "unchanging", synonymous with ABI locked
or ABI fixed.</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>I don't see this as a very bad idea or as
micro-management.<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</div>
I didn't see that either until I start implementing
stuff. <br>
<div> <br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>In this regard, I don't see a difference
between libLTO and other libraries like
libPNG or netlib or freetype. (There is a
difference in that we want libLTO to be a
very high-level interface instead of
exposing the details of .bc files, entirely
unlike what libPNG does for PNG or freetype
does for font files.)</div>
</div>
</div>
</div>
</blockquote>
<br>
</div>
Similar in concept. Concept only. <br>
<div> <br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>Having a default setting
with the ability to override
it is a sensible convenience
for users of libLTO.</div>
</div>
</div>
</div>
</blockquote>
<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">Take Apple ld
as example, if I want to
change LTO in a way such
that I don't want to load
all module, <br>
I just want to load summary
info. Current APIs are not
sufficient. I have to modify
the API, or add new APIs<br>
to that matter, in the mean
time, I need release the new
ld to the user in order to
accomodate the change. <br>
that is nightmare. </div>
</blockquote>
<div><br>
</div>
<div>The point of libLTO is to
provide an ABI-fixed library,
isolating the linker from
llvm's internals. </div>
</div>
</div>
</div>
</blockquote>
</div>
It is not "fixed", it is changing
constantly.</div>
</blockquote>
<div><br>
</div>
<div>The only reason libLTO exists at all is
to give the linker something to link against
which will have a fixed ABI. Same with
"libclang" on the clang side.</div>
</div>
</div>
</div>
</blockquote>
</div>
It is slightly different in the case of libLTO+linker. <br>
<br>
clang + libclang are tightly bound in terms of release.
If you are not happy with particular API, you can change
to whatever you feel more comfortable. <br>
<br>
The linker is usually release independently. The
compiler has less control, if the API between compiler
and linker is constantly changing.<br>
Keeping backward compatability is a big problem. <br>
</div>
</blockquote>
<div><br>
</div>
<div>Right. libLTO is supposed to be that solution, by
giving the linker a stable ABI (and API) to link against,
and handling whatever changes in the LLVM C++ API behind
the scenes. Put another way, a linker built against libLTO
version X should work with version Y for all Y>X
forever, without even a recompile (assuming the linker was
built to load libLTO dynamically).</div>
</div>
</div>
</div>
</blockquote></div>
The problem is how do you magically define such APIs, once for all?<br></div></blockquote><div><br></div><div>Look, I didn't make up this rule. I'm trying to tell you, as a new contributor, that the rule already existed and you should be aware of it.</div>
<div><br></div>
<div>Here's an old example of Chris mentioning the rule that changes to the C ABI are not allowed (and yes, libLTO's API is part of the C API):</div><div><a href="http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20090720/081858.html" target="_blank">http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20090720/081858.html</a> . I don't think this is the first time it was mentioned either, but I wasn't able to find a better example with a minute of searching.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000">
Currently, we ignore the I/O issue about LTO, can you imagine what
kind of APIs we need in future <br>
to improve the I/Os. And how can magically maintain such backward
compatability? libLTO is <br>
potentially very complex state-machine. <br>
<br>
Why not let linker to expose its interface, and let plugin (LTO) to
work with them. Isn't that easier. <br>
I don't understand you think exposing interface from LTO to linker
is better. Such linker is tightly<br>
bound to a compiler, dedicated to compiler. In order to maintain
that linker, engineer needs in-depth<br>
knowhows about compiler and linker. I don't see why it is better. <br></div></blockquote><div><br></div><div>Let me try to restate what you said in my own words, so we can make sure I understood it. You want llvm-lto to be its own program, and the system linker should be a plugin that llvm-lto loads?</div>
<div><br></div><div>I have not experienced the problem you mention with needing to understand the compiler and the linker. To take a recent example, <a href="http://sourceware.org/ml/binutils/2013-06/msg00139.html">http://sourceware.org/ml/binutils/2013-06/msg00139.html</a> was less than an hour of work for me to determine that the bug was in gold. I certainly am not really familiar with gold (or linkers in general), and I don't think Cary knows much about llvm (though he probably knows a lot about compilers).</div>
<div><br></div><div>I have experienced a different kind of problem due to LTO: writing testcases is a pain! With LTO, you can't just split a testcase into two files to demonstrate the linking problem because LTO will see right through that.</div>
<div><br></div><div>Is your concern theoretical, or has it actually come up?</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div><blockquote type="cite"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div><blockquote type="cite"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Thus far it's LLVM policy that the *whole
and entire* C API is ABI-fixed forever, and
I've argued a few times on the mailing list
that this can't be right, and that only
libLTO and libClang ought to be ABI locked.</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> E.g.
the APIs used to take for granted the
libLTO return only one objects, <br>
now I need to return multiple.<br>
</div>
</blockquote>
<div><br>
</div>
<div>Yes, and that's a problem. Not your
problem really, except to the degree that
you inherited it. The existing APIs in
libLTO weren't nearly forwards-compatible
enough, and now we're in trouble.</div>
<div><br>
</div>
<div>Unless we figure out something clever, we
may have to add a whole new set of functions
to libLTO, and not deprecate the existing
ones (at least, not unless we get consensus
on llvm-dev that it's okay to break our
previous ABI promise).</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>That in turn leads to a few
design decisions. The API is
designed to refer to
high-level concepts instead of
the details of llvm's actual
behaviour. Things like module
lazy loading or setting the
datalayout are excluded from
the API. Flags are even more
private, surely we should be
able to change flags in LLVM's
libraries without worrying
about breaking linkers.</div>
<div><br>
</div>
<div>If the linker needs to do
something where it matters how
llvm is implemented -- you
mention loading summary info,
I'll assume you mean
lazy-loading the module such
that function bodies aren't
loaded -- then the linker
doesn't use libLTO at all, but
uses llvm directly.
Conversely, libLTO knows all
about llvm and will lazy-load
.bc files without being asked
to.</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">Sure,
"something else" can control
the libLTO, if it want. In
my case, if "something else"
want specify <br>
a workdir, then go ahead.
Otherwise, the libLTO use
default one. Is there any
wrong here?</div>
</blockquote>
<div><br>
</div>
<div>At a high level that sounds
fine to me. The wrong part is
using flags to do it.</div>
</div>
</div>
</div>
</blockquote>
</div>
then how to change the behavior for say,
debugging purpose. <br>
</div>
</blockquote>
<div><br>
</div>
<div>Debugging is special. In theory, you
don't even need to commit to upstream for
debugging, but it's fine to add features
that are helpful. We have that sort of thin
all over llvm. libLTO has addDebugOptions to
permit this sort of debugging usage, but it
shouldn't be used in the non-debugging case.</div>
</div>
</div>
</div>
</blockquote>
<br>
</div>
Passing flags LTO is annoying and it is a sort of
high-tech. <br>
Bill's attribute-stuff is a way to pass some flags down
the roads. <br>
<br>
How about passing -O3 -floop-vecotroize to make LTO and
post-LTO code works. (The opt-level is -O2, had
vectorize-flags is off by default). <br>
</div>
</blockquote>
<div><br>
</div>
<div>Right, this is a great example. I would argue that we
absolutely should not offer such control to libLTO, not by
flags or environment variables or C API or anything. Why?
Because it locks the entirety of LLVM to offering a "loop
vectorizer" forever. What happens in LLVM 4.0 when we have
a "global vectorizer" or "common vectorizer" or some other
name? We've removed optimizations before (global common
subexpression elimination, gvnpre, ...) and we should have
the freedom to continue to do so.</div>
<div><br>
</div>
<div>Now, suppose along comes a linker vendor who says "as a
feature, my users can specify -floop-vectorize or other
flags to control which optimizations get run at link
time". Do we refuse? Do we tell them it's okay, but we
reserve the right to break these flags at any time (and
then what, we can't catch typos because they could just be
names of old optz'ns we don't support)? But I actually
don't think giving control of this is a *feature* -- it
can only really used as a workaround for bugs (or exotic
stuff like kernel code where the vector register unit
hasn't been initialized yet, but I am okay with having a
flag to control whether we're allowed to use a certain
class of instructions -- yet not okay with disabling
individual optimizations).</div>
</div>
</div>
</div>
</blockquote>
<br></div>
Those flags are passed as plugin flags. Ld has no idea of it. On
the other hand, such flags are passed by compiler (derived from <br>
-flto and other flags) . They are transparent to users.<br></div></blockquote><div><br></div><div>Sure. It's a way to let the compiler and the compiler-provided LTO plugin collaborate, via "flags" (really any string with limited punctuation).</div>
<div><br></div><div>But that assumes the linker will be run by the compiler, which isn't always true. We want llvm lto to work as a drop-in replacement for the existing tools, and sometimes existing build systems will run ld directly. (Yes, gold will search known paths on the system to find the plugin, it doesn't require a flag.)</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><div><div><blockquote type="cite">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">We may (likely) have
better way in the future to pass these flags to LTO, <br>
but we have to pass the these flags the it to make the
existing code work, at least for the time being. <br>
<div>
<div> <br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Adding
flags to
linker
instead, I
think that is
wrong
direction.
Linker dose
not have data
structure
which libLTO
dose.</blockquote>
<div><br>
</div>
<div>This is the
discussion to
have. What
things do you
need here
which you
don't think
should be
exposed
through the
API, and yet
you want to be
exposed for
you?</div>
</div>
</div>
</div>
</blockquote>
</div>
I actually discuss with
Nick @ Apple before. The
conclusion is linker must
be LTO oblivious, <br>
it should think in
symbol-way, and talk in
symbol way (as with GNU
gold). It would otherwise<br>
very very troublesome
both for linker and
libLTO. <br>
</div>
</blockquote>
<div><br>
</div>
<div>And now you're discussing
it with me. I also agree
that the linker should
communicate primarily in
symbols and about symbols
with libLTO.</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">On the
other hand, we now have
two linkers support LTO.
There are different way to
control <br>
the libLTO (even for
simple task, like save
intermediate files), how
messy?<br>
<br>
I'd like to move all these
stuff to libLTO to have a
unified control. <br>
</div>
</blockquote>
<div><br>
</div>
<div>I have no problem with a
unified control.</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div>libLTO is
intended to be
used as a
library, it
may not get a
chance to
parse flags.<br>
</div>
It has to.
Prior to my
change,
linkers (GNU
linker and
Apple ld) pass
arch to
linker, via a
function<br>
confusingly
called,
something like
"add.*debug.*options".</blockquote>
<div><br>
</div>
<div>Can't. If
we allow this,
every flag in
every part of
LLVM that
libLTO links
against is
baked into the
C ABI forever.</div>
<div><br>
</div>
<div>Of course
addDebugOptions
does allow
this, but it's
named (and I
thought
documented in
the comments)
such that
anybody using
it knows
they're using
a non-stable
non-production
debugging API.
Anybody using
addDebugOptions
for something
other than
debugging
libLTO is
living outside
the ABI
guarantees.</div>
</div>
</div>
</div>
</blockquote>
</div>
addDebugOptions is
misnomer. It is also
passes essential flags
like -arch=x86. Without
such flags, <br>
the LTO dose not even
compile. <br>
</div>
</blockquote>
<div><br>
</div>
<div>That sounds like a nice
bug you've got there!
Wouldn't want anything to
happen to it. It'd be a
shame if breaks before you
manage to add a
liblto_set_arch() function
for it.</div>
</div>
</div>
</div>
</blockquote>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>* Honestly, I looked and
couldn't find a -arch flag
that libLTO would interpret.
How sure are you about this?</div>
</div>
</div>
</div>
</blockquote>
</div>
Perhaps not -arch flags.<br>
But at least some flags are passed this
way. I remember we use this way to pass
-fast-math before Bill's attribute-stuff
is working. <br>
<div> <br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>In case it isn't
completely clear, flags are
absolutely right out. Either
you will revert this patch,
or I will revert it for you.</div>
</div>
</div>
</div>
</blockquote>
</div>
I have no alternative. If I introduce a
workdir, I need to have to way to inform
linker-plugin to get rid of way. <br>
This is another example why those API
sucks.<br>
</div>
</blockquote>
<div><br>
</div>
<div>You don't have the source code to the
linker?</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
I can modify linker source code. The problem is how to
make sure all users get the modified linker to work with
the new compilers. <br>
It going to be very messy. right? <br>
</div>
</blockquote>
<div><br>
</div>
<div>True. You have a deployment problem where instead of
shipping just a new libLTO, you ship a new libLTO (and all
older linkers must continue to work with it), and then
ship a new linker taking advantage of the new libLTO APIs.
Sorry, but I think this is a natural consequence of the
fact that libLTO needs to be ABI-locked.</div>
<div><br>
</div>
<div>(Also, in reality, if you can solve the deployment
problem for libLTO, then you can solve the deployment
problem for libLTO+ld. Yes it'll be more work.)</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">Unlike the clang and
clanglib, they are so "close" in terms of release. We
can change at will. <br>
<div> <br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>Let's focus on this, it sounds like this
is the key problem. What's wrong with
modifying the linker if you want to change
the behaviour of your linker?</div>
</div>
</div>
</div>
</blockquote>
</div>
How often dose a user check if the linker is
up-to-minute? <br>
<div> <br>
<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> I'm sorry you decided to
land three things together in
one patch, please remember not
to do that in the future.<br>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
Ok, tell me how to create temp workding
directory right. How to save temp files
right both for gold and Apple ld. <br>
</div>
</blockquote>
<div><br>
</div>
<div>*Why*? Are you implementing this as a
linker feature you intend to ship in the
real linker? Or is this to debug the innards
of libLTO?</div>
</div>
</div>
</div>
</blockquote>
<br>
</div>
It is not linker's feature, it is absolutely libLTO's
own biz. Creating a workdir is neat way to organize
intermediate files, <br>
we can certainly use a messy way to organize the
intermediate files without creating workdir. <br>
<div> <br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>The only case I *am* okay with flags is
when we all agree they're flags for
debugging the internals of libLTO, </div>
</div>
</div>
</div>
</blockquote>
</div>
To large extend, it is for trouble-shooting purpose. <br>
<div> <br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>and that we don't ship products that rely
on them. </div>
</div>
</div>
</div>
</blockquote>
</div>
The product will not rely on it.</div>
</blockquote>
<div><br>
</div>
<div><br>
</div>
<div>Okay. Got it. So I have a few thoughts on this.</div>
<div><br>
</div>
<div>First of all, why don't we expose the fact that we
produce native .o files? Because we don't want to
necessarily require files. lto_codegen_compile returns a
pointer to the memory containing the file in memory.
Hilariously this means the libLTO writes out to disk,
loads it into memory, hands it off to LLVMgold which
writes it back down to disk again and hands it off to
gold, which reads it back into memory. And yet it's still
the right interface. libLTO should stop writing to disk
and actually produce .o in RAM directly, and gold should
learn to read from RAM directly.*</div>
<div><br>
</div>
</div>
</div>
</div>
</blockquote></div></div>
I remember gold call lto_codegen_compile_to_file while Apple call
lto_codegen_compile.<br>
I don't want to discuss the details here as it deviate from what we
are focusing. <br><div>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>Secondly, if we aren't exposing the fact that we
produce native .o files, should we be exposing a knob that
lets us control the working directory? Probably not, but
it's not unreasonable. If we're going to write to disk
it's polite to let the caller choose where. If we don't
write to disk, the API can be trivially implemented by
doing nothing.</div>
</div>
</div>
</div>
</blockquote></div>
I don't want to expose workdir, it could have lots of junk over
there. I don't like linker penetrate the privacy.</div></blockquote><div><br></div><div>That's true, we don't want anyone to rely on the contents of the workdir. I find it strange that you think it's perfectly fine to expose every flag in llvm, but balk at exposing the workdir. I'd expect people to accidentally bake in flags without realizing it could break on them in the future, but not to accidentally plunder the workdir and rely on its contents without realizing it could break some day.</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><div><blockquote type="cite">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div>You suggest $PWD/unique-tmp-workdir instead of /tmp.
Consider $TMP, or on Windows $TEMP? I don't mind if we're
smart enough to pick good defaults, but I can absolutely
imagine a linker that to keep its temp files in a specific
directory. Suppose a mobile OS that runs the linker on the
phone, where they have strict disk quotas. It's important
to put the files in the right places, so they get counted
against the right quotas. Also, for cleanup in the event
of a crash (assume it wipes the whole directory tree). I
really think lto_set_tempdir would be a good API to have
in libLTO, and poses no risk of being unimplementable in
the future.</div>
</div>
</div>
</div>
</blockquote></div>
This is changed now. 1st try PWD/unique-dir. It was not successful
(e.g. no write-permission), call sys::fs::createUniqueDirectory() <br>
to create dir under $TMP or $TEMP. <br>
<br>
We try $PWD first, because often time, we debug over there. So it is
bit convenient. <br>
There is no strong reason here. <br><div>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div><span style="font-family:arial,sans-serif;font-size:13px">Thirdly,
I'm not convinced that lto_codegen_get_files_need_</span><span style="font-family:arial,sans-serif;font-size:13px">remove
needs to exist. Why not do the file deletion in
lto_codegen_dispose?</span><br>
</div>
</div>
</div>
</div>
</blockquote></div>
You never know long the linker hold the intermediate file. <br>
On the hand, there was a bug in Apple ld, which never call
llto_codegen_dispose. <br></div></blockquote><div><br></div><div>What's the problem? The linker releases the intermediate files at the call to lto_codegen_dispose. Do you want finer-grained control?</div><div><br>
</div><div>Nick<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div><blockquote type="cite"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div>* Actually, Rafael added lto_codegen_compile_to_file in
r128108 and we've had numerous LLVM release since then.
Now we have to support writing to files forever; even if
we support writing to memory, we can't remove the
writing-to-file path. </div>
</div>
</div>
</div>
</blockquote></div>
GNU gold dose not see lto_xxx. GNU-gold and lto_xxx are bridged by
tool/gold/*.cpp. <br>
<br>
Whatever change I made to lto_xxx, dose not impact GNU-gold. <br><div>
<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>Fortunately we can continue to implement this API in
the future by codegen'ing to memory plus a small amount of
code to write to disk. Still, I'm a little bit sad inside.</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>I explicitly called that out. If the only
purpose of these was to implement debugging
features, then I'm sorry for the
miscommunication!</div>
<div><br>
</div>
<div>If the intention is to let libLTO run on
machines that don't have /tmp (this is what
I thought), we should give libLTO an API
that lets the linker decide where the files
go. </div>
</div>
</div>
</div>
</blockquote>
</div>
I choose $PWD/unique-tmp-workdir instead of /tmp/xxx. <br>
I should try /tmp/xxx if $PWD/unique-tmp-workdir dose
not work. <br>
<div> <br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>Maybe it wants to do smart things like
putting it in a directory with the right
permissions, or which is scheduled for
cleanup in the event of a crash.</div>
</div>
</div>
</div>
</blockquote>
</div>
That is in TODO list. I try to install the sig handler,
but the supporting routines ignore directory (it only
delete regular file on signal).<br>
</div>
</blockquote>
<div><br>
</div>
<div>Hold on, what if the linker installs its own signal
handler? If lto_codegen_dispose/lto_module_dispose aren't
safe to call during a signal handler, what do you think
about providing a signal-safe-emergency-shutdown API to
libLTO? It should only be as hard to implement as
factoring out the code you were going to write anyways.</div>
<div><br>
</div>
<br>
</div>
</div>
</div>
</blockquote></div>
I have not yet closely check how the support/file-system stuff are
implemented. <br>
Anyway I add them to list-of-file-need-to-removed to the cleanup
hook. Hoping they will be called on signal. <br>
</div>
</blockquote></div></div><div class="gmail_extra"><br></div></div>