<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Thank you very much for sharing you
concerns. I read this mail carefully, it seems we had little
miscommunications. <br>
I hope I clarify in this mail:-). See the interleaving text. <br>
<br>
<br>
On 8/12/13 3:41 PM, Nick Lewycky wrote:<br>
</div>
<blockquote
cite="mid:CADbEz-ju2ZfW-RmOsj08sf7R3-785TOceOhS-LPf23tMi3SYEA@mail.gmail.com"
type="cite">
<div dir="ltr"><br>
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div class="im"><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div> >which in turn drives libLTO
through the API.<br>
<br>
</div>
Depending on the what kind of info
"something" else need to drive the libLTO.
<br>
In general it is very bad idea, if
"something else" need micro-management.</div>
</blockquote>
<div><br>
</div>
<div>libLTO is part of the linker that uses
it. </div>
</div>
</div>
</div>
</blockquote>
<br>
</div>
No! Absolutely not!</div>
</blockquote>
<div><br>
</div>
<div>Fair enough. I meant "libLTO is part of the linker that
uses it" in the same sense that a networking library is
part of the web browser that uses it. The library
shouldn't be off deciding to do things of its own accord,
it should provide an API that allows something else to
accomplish its task.</div>
</div>
</div>
</div>
</blockquote>
I don't think such comparison is precise. For app with networking
lib, the "braid" reside at app side because the app define the
behavior. <br>
For linker+libLTO, I believe the "brain" should reside at libLTO
side, as it is far more complicate (although our current implement
is bit simple). <br>
<br>
I believe GNU gold is a good in designing the interface. In the
case of gold+libLTO, the "brain" is embodied by "tool/gold/*.cpp."<br>
Anybody can change to whatever he/she like. Although it dose call
APIs, it dose not have to. I can directly call those c++ code <br>
dancing behind the API. This is the "stable API" I'm talking about.
<br>
<br>
<blockquote
cite="mid:CADbEz-ju2ZfW-RmOsj08sf7R3-785TOceOhS-LPf23tMi3SYEA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>I don't see this as a very bad idea or as
micro-management. </div>
</div>
</div>
</div>
</blockquote>
<br>
I didn't see that either until I start implementing stuff. <br>
<br>
<blockquote
cite="mid:CADbEz-ju2ZfW-RmOsj08sf7R3-785TOceOhS-LPf23tMi3SYEA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>In this regard, I don't see a difference between libLTO
and other libraries like libPNG or netlib or freetype.
(There is a difference in that we want libLTO to be a very
high-level interface instead of exposing the details of
.bc files, entirely unlike what libPNG does for PNG or
freetype does for font files.)</div>
</div>
</div>
</div>
</blockquote>
<br>
Similar in concept. Concept only. <br>
<br>
<blockquote
cite="mid:CADbEz-ju2ZfW-RmOsj08sf7R3-785TOceOhS-LPf23tMi3SYEA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div class="im">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>Having a default setting with the ability
to override it is a sensible convenience for
users of libLTO.</div>
</div>
</div>
</div>
</blockquote>
<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">Take
Apple ld as example, if I want to change
LTO in a way such that I don't want to
load all module, <br>
I just want to load summary info. Current
APIs are not sufficient. I have to modify
the API, or add new APIs<br>
to that matter, in the mean time, I need
release the new ld to the user in order to
accomodate the change. <br>
that is nightmare. </div>
</blockquote>
<div><br>
</div>
<div>The point of libLTO is to provide an
ABI-fixed library, isolating the linker from
llvm's internals. </div>
</div>
</div>
</div>
</blockquote>
</div>
It is not "fixed", it is changing constantly.</div>
</blockquote>
<div><br>
</div>
<div>The only reason libLTO exists at all is to give the
linker something to link against which will have a fixed
ABI. Same with "libclang" on the clang side.</div>
</div>
</div>
</div>
</blockquote>
It is slightly different in the case of libLTO+linker. <br>
<br>
clang + libclang are tightly bound in terms of release. If you are
not happy with particular API, you can change to whatever you feel
more comfortable. <br>
<br>
The linker is usually release independently. The compiler has less
control, if the API between compiler and linker is constantly
changing.<br>
Keeping backward compatability is a big problem. <br>
<br>
<br>
<br>
<blockquote
cite="mid:CADbEz-ju2ZfW-RmOsj08sf7R3-785TOceOhS-LPf23tMi3SYEA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>Thus far it's LLVM policy that the *whole and entire* C
API is ABI-fixed forever, and I've argued a few times on
the mailing list that this can't be right, and that only
libLTO and libClang ought to be ABI locked.</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> E.g. the APIs used
to take for granted the libLTO return only one objects,
<br>
now I need to return multiple.<br>
</div>
</blockquote>
<div><br>
</div>
<div>Yes, and that's a problem. Not your problem really,
except to the degree that you inherited it. The existing
APIs in libLTO weren't nearly forwards-compatible enough,
and now we're in trouble.</div>
<div><br>
</div>
<div>Unless we figure out something clever, we may have to
add a whole new set of functions to libLTO, and not
deprecate the existing ones (at least, not unless we get
consensus on llvm-dev that it's okay to break our previous
ABI promise).</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div class="im">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>That in turn leads to a few design
decisions. The API is designed to refer to
high-level concepts instead of the details
of llvm's actual behaviour. Things like
module lazy loading or setting the
datalayout are excluded from the API. Flags
are even more private, surely we should be
able to change flags in LLVM's libraries
without worrying about breaking linkers.</div>
<div><br>
</div>
<div>If the linker needs to do something where
it matters how llvm is implemented -- you
mention loading summary info, I'll assume
you mean lazy-loading the module such that
function bodies aren't loaded -- then the
linker doesn't use libLTO at all, but uses
llvm directly. Conversely, libLTO knows all
about llvm and will lazy-load .bc files
without being asked to.</div>
<div><br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">Sure,
"something else" can control the libLTO,
if it want. In my case, if "something
else" want specify <br>
a workdir, then go ahead. Otherwise, the
libLTO use default one. Is there any wrong
here?</div>
</blockquote>
<div><br>
</div>
<div>At a high level that sounds fine to me.
The wrong part is using flags to do it.</div>
</div>
</div>
</div>
</blockquote>
</div>
then how to change the behavior for say, debugging
purpose. <br>
</div>
</blockquote>
<div><br>
</div>
<div>Debugging is special. In theory, you don't even need to
commit to upstream for debugging, but it's fine to add
features that are helpful. We have that sort of thin all
over llvm. libLTO has addDebugOptions to permit this sort
of debugging usage, but it shouldn't be used in the
non-debugging case.</div>
</div>
</div>
</div>
</blockquote>
<br>
Passing flags LTO is annoying and it is a sort of high-tech. <br>
Bill's attribute-stuff is a way to pass some flags down the roads. <br>
<br>
How about passing -O3 -floop-vecotroize to make LTO and post-LTO
code works. (The opt-level is -O2, had vectorize-flags is off by
default). <br>
<br>
We may (likely) have better way in the future to pass these flags to
LTO, <br>
but we have to pass the these flags the it to make the existing code
work, at least for the time being. <br>
<br>
<br>
<blockquote
cite="mid:CADbEz-ju2ZfW-RmOsj08sf7R3-785TOceOhS-LPf23tMi3SYEA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div class="im">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Adding
flags to linker instead, I
think that is wrong direction.
Linker dose not have data
structure which libLTO dose.</blockquote>
<div><br>
</div>
<div>This is the discussion to
have. What things do you need
here which you don't think
should be exposed through the
API, and yet you want to be
exposed for you?</div>
</div>
</div>
</div>
</blockquote>
</div>
I actually discuss with Nick @ Apple
before. The conclusion is linker must be
LTO oblivious, <br>
it should think in symbol-way, and talk in
symbol way (as with GNU gold). It would
otherwise<br>
very very troublesome both for linker and
libLTO. <br>
</div>
</blockquote>
<div><br>
</div>
<div>And now you're discussing it with me. I
also agree that the linker should
communicate primarily in symbols and about
symbols with libLTO.</div>
<div><br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">On the
other hand, we now have two linkers
support LTO. There are different way to
control <br>
the libLTO (even for simple task, like
save intermediate files), how messy?<br>
<br>
I'd like to move all these stuff to libLTO
to have a unified control. <br>
</div>
</blockquote>
<div><br>
</div>
<div>I have no problem with a unified control.</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div>libLTO is intended to be
used as a library, it may
not get a chance to parse
flags.<br>
</div>
It has to. Prior to my change,
linkers (GNU linker and Apple
ld) pass arch to linker, via a
function<br>
confusingly called, something
like "add.*debug.*options".</blockquote>
<div><br>
</div>
<div>Can't. If we allow this,
every flag in every part of
LLVM that libLTO links against
is baked into the C ABI
forever.</div>
<div><br>
</div>
<div>Of course addDebugOptions
does allow this, but it's
named (and I thought
documented in the comments)
such that anybody using it
knows they're using a
non-stable non-production
debugging API. Anybody using
addDebugOptions for something
other than debugging libLTO is
living outside the ABI
guarantees.</div>
</div>
</div>
</div>
</blockquote>
</div>
addDebugOptions is misnomer. It is also
passes essential flags like -arch=x86.
Without such flags, <br>
the LTO dose not even compile. <br>
</div>
</blockquote>
<div><br>
</div>
<div>That sounds like a nice bug you've got
there! Wouldn't want anything to happen to
it. It'd be a shame if breaks before you
manage to add a liblto_set_arch() function
for it.</div>
</div>
</div>
</div>
</blockquote>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>* Honestly, I looked and couldn't find a
-arch flag that libLTO would interpret. How
sure are you about this?</div>
</div>
</div>
</div>
</blockquote>
</div>
Perhaps not -arch flags.<br>
But at least some flags are passed this way. I remember
we use this way to pass -fast-math before Bill's
attribute-stuff is working. <br>
<div class="im"> <br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>In case it isn't completely clear, flags
are absolutely right out. Either you will
revert this patch, or I will revert it for
you.</div>
</div>
</div>
</div>
</blockquote>
</div>
I have no alternative. If I introduce a workdir, I need
to have to way to inform linker-plugin to get rid of
way. <br>
This is another example why those API sucks.<br>
</div>
</blockquote>
<div><br>
</div>
<div>You don't have the source code to the linker?</div>
</div>
</div>
</div>
</blockquote>
I can modify linker source code. The problem is how to make sure all
users get the modified linker to work with the new compilers. <br>
It going to be very messy. right? <br>
<br>
Unlike the clang and clanglib, they are so "close" in terms of
release. We can change at will. <br>
<br>
<blockquote
cite="mid:CADbEz-ju2ZfW-RmOsj08sf7R3-785TOceOhS-LPf23tMi3SYEA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>Let's focus on this, it sounds like this is the key
problem. What's wrong with modifying the linker if you
want to change the behaviour of your linker?</div>
</div>
</div>
</div>
</blockquote>
How often dose a user check if the linker is up-to-minute? <br>
<br>
<br>
<br>
<blockquote
cite="mid:CADbEz-ju2ZfW-RmOsj08sf7R3-785TOceOhS-LPf23tMi3SYEA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div class="im">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> I'm sorry you decided to land three
things together in one patch, please
remember not to do that in the future.<br>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
Ok, tell me how to create temp workding directory right.
How to save temp files right both for gold and Apple ld.
<br>
</div>
</blockquote>
<div><br>
</div>
<div>*Why*? Are you implementing this as a linker feature
you intend to ship in the real linker? Or is this to debug
the innards of libLTO?</div>
</div>
</div>
</div>
</blockquote>
<br>
It is not linker's feature, it is absolutely libLTO's own biz.
Creating a workdir is neat way to organize intermediate files, <br>
we can certainly use a messy way to organize the intermediate files
without creating workdir. <br>
<br>
<blockquote
cite="mid:CADbEz-ju2ZfW-RmOsj08sf7R3-785TOceOhS-LPf23tMi3SYEA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>The only case I *am* okay with flags is when we all
agree they're flags for debugging the internals of libLTO,
</div>
</div>
</div>
</div>
</blockquote>
To large extend, it is for trouble-shooting purpose. <br>
<br>
<blockquote
cite="mid:CADbEz-ju2ZfW-RmOsj08sf7R3-785TOceOhS-LPf23tMi3SYEA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>and that we don't ship products that rely on them. </div>
</div>
</div>
</div>
</blockquote>
The product will not rely on it. <br>
<br>
<blockquote
cite="mid:CADbEz-ju2ZfW-RmOsj08sf7R3-785TOceOhS-LPf23tMi3SYEA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>I explicitly called that out. If the only purpose of
these was to implement debugging features, then I'm sorry
for the miscommunication!</div>
<div><br>
</div>
<div>If the intention is to let libLTO run on machines that
don't have /tmp (this is what I thought), we should give
libLTO an API that lets the linker decide where the files
go. </div>
</div>
</div>
</div>
</blockquote>
I choose $PWD/unique-tmp-workdir instead of /tmp/xxx. <br>
I should try /tmp/xxx if $PWD/unique-tmp-workdir dose not work. <br>
<br>
<blockquote
cite="mid:CADbEz-ju2ZfW-RmOsj08sf7R3-785TOceOhS-LPf23tMi3SYEA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>Maybe it wants to do smart things like putting it in a
directory with the right permissions, or which is
scheduled for cleanup in the event of a crash.</div>
</div>
</div>
</div>
</blockquote>
That is in TODO list. I try to install the sig handler, but the
supporting routines ignore directory (it only delete regular file on
signal). <br>
<br>
<blockquote
cite="mid:CADbEz-ju2ZfW-RmOsj08sf7R3-785TOceOhS-LPf23tMi3SYEA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>If the intention is to implement something like -r, or
to implement some new feature like -shared but which emits
a shared-bitcode file instead of a shared-object,</div>
</div>
</div>
</div>
</blockquote>
I'm not sure if bit-code can be encapsulated in *.so (*.a sure
work), and if it's good practice to do so. <br>
At least, I'm not happy to see a *.so is not binary, the app will
sure complains.<br>
<br>
<br>
<blockquote
cite="mid:CADbEz-ju2ZfW-RmOsj08sf7R3-785TOceOhS-LPf23tMi3SYEA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> then I don't think that controlling the working
directory then pulling out intermediate files is the right
design anyways.</div>
</div>
</div>
</div>
</blockquote>
<br>
<br>
<blockquote
cite="mid:CADbEz-ju2ZfW-RmOsj08sf7R3-785TOceOhS-LPf23tMi3SYEA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>Or are you doing something else?</div>
</div>
</div>
</div>
</blockquote>
<br>
I guess I'm doing something else. Suppose you are building libmy.so
from a.o b.o. c.o d.o, where c.o d.o is bit-code.<br>
the LTO manage to compile c.o+d.o into t.o. Or if the c.o + d.o is
huge, the LTO split them into several partitions, <br>
the compile partitions in to t1.o t2.o ... tn.o. <br>
<br>
The intermediate files I'm talking about are those t*.o. libLTO hand
these intermediate files to linker, *NOT* know they<br>
they will die. It is up to linker to get rid of them. <br>
<br>
After linker get these intermediate files, and move on the linker
the final libmy.so. Before exit, it remove all these intermediate
files. <br>
<br>
As you can see, we certainly can live without workdir. It is just a
neat way to organize them, and also obviate the need to <br>
create unique name for each intermediate file to avoid name
conflict. This is especially handy if we want to compare <br>
performances etc. <br>
<br>
<br>
<br>
</body>
</html>