<div dir="ltr">Hi all,<div><br></div><div>I updated the patches to remove the native object wrapper format. As suggested we will work on getting the ThinLTO framework in place using bitcode first, and then work on adding the native object support. As noted in this RFC and in the associated patch D11722, for now I have empty ThinLTO blocks with no records, since I wanted to get feedback on the overall block design first. The RFC discusses this in more detail, but one of the main ideas is to leverage the existing value symbol table block in the module to avoid duplicating function symbol strings, e.g.</div><div><br></div><div>I also wanted to call out another important design consideration here, since it is buried in the other RFC (ThinLTO File API and Data Structures), and has a big influence on the way I have designed the ThinLTO index and object file data structures. The ThinLTO index is read in compile/link steps when the rest of the Module IR is not, and vice versa. That is why I have separate data structures for reading/holding the ThinLTO index. The ThinLTO index in the module (generated during the initial -c compile step) is needed by other modules during the later parallel backend compile phase, and therefore it is only used in the linker plugin step to create the combined index file. The rest of the Module IR is not read during this step (eventually we may look at adding heavier weight whole program analysis under an option, but by default the Module, Functions, etc are not read or materialized). When the normal Module IR is read during the parallel backend compile step, the ThinLTO information in its own module is not read, as the importing pass will read the combined (global) index file instead. This is because a module is only interested in the ThinLTO index from other modules that it is considering importing from.</div><div><br></div><div>Right now I have 5 outstanding patches to put in the basic infrastructure/options for reading/writing the ThinLTO function indices:</div><div><br></div><div><span class="phui-object-item-objname" style="color:rgb(75,77,81);font-family:'Segoe UI','Segoe UI Web Regular','Segoe UI Symbol','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;font-weight:bold;line-height:16.0029983520508px;white-space:nowrap">D11721</span><span style="color:rgb(0,0,0);font-family:'Segoe UI','Segoe UI Web Regular','Segoe UI Symbol','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;font-weight:bold;line-height:16.0029983520508px;white-space:nowrap"> </span><a href="http://reviews.llvm.org/D11721" class="phui-object-item-link" title="[ThinLTO] Data structures for holding ThinLTO function index/summary" style="text-decoration:none;color:rgb(24,85,157);display:inline;font-family:'Segoe UI','Segoe UI Web Regular','Segoe UI Symbol','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;font-weight:bold;line-height:16.0029983520508px;white-space:nowrap">[ThinLTO] Data structures for holding ThinLTO function index/summary</a><br></div><div><span class="phui-object-item-objname" style="color:rgb(75,77,81);font-family:'Segoe UI','Segoe UI Web Regular','Segoe UI Symbol','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;font-weight:bold;line-height:16.0029983520508px;white-space:nowrap">D11722</span><span style="color:rgb(0,0,0);font-family:'Segoe UI','Segoe UI Web Regular','Segoe UI Symbol','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;font-weight:bold;line-height:16.0029983520508px;white-space:nowrap"> </span><a href="http://reviews.llvm.org/D11722" class="phui-object-item-link" title="[ThinLTO] Bitcode reading/writing support for ThinLTO function summary/index" style="text-decoration:none;color:rgb(24,85,157);display:inline;font-family:'Segoe UI','Segoe UI Web Regular','Segoe UI Symbol','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;font-weight:bold;line-height:16.0029983520508px;white-space:nowrap">[ThinLTO] Bitcode reading/writing support for ThinLTO function summary/index</a><br></div><div><span class="phui-object-item-objname" style="color:rgb(75,77,81);font-family:'Segoe UI','Segoe UI Web Regular','Segoe UI Symbol','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;font-weight:bold;line-height:16.0029983520508px;white-space:nowrap">D11723</span><span style="color:rgb(0,0,0);font-family:'Segoe UI','Segoe UI Web Regular','Segoe UI Symbol','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;font-weight:bold;line-height:16.0029983520508px;white-space:nowrap"> </span><a href="http://reviews.llvm.org/D11723" class="phui-object-item-link" title="[ThinLTO] ThinLTO object file interfaces" style="color:rgb(24,85,157);display:inline;font-family:'Segoe UI','Segoe UI Web Regular','Segoe UI Symbol','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;font-weight:bold;line-height:16.0029983520508px;white-space:nowrap">[ThinLTO] ThinLTO object file interfaces</a><br></div><div><span class="phui-object-item-objname" style="color:rgb(75,77,81);font-family:'Segoe UI','Segoe UI Web Regular','Segoe UI Symbol','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;font-weight:bold;line-height:16.0029983520508px;white-space:nowrap">D11907</span><span style="color:rgb(0,0,0);font-family:'Segoe UI','Segoe UI Web Regular','Segoe UI Symbol','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;font-weight:bold;line-height:16.0029983520508px;white-space:nowrap"> </span><a href="http://reviews.llvm.org/D11907" class="phui-object-item-link" title="LLVM support for -fthinlto option." style="text-decoration:none;color:rgb(24,85,157);display:inline;font-family:'Segoe UI','Segoe UI Web Regular','Segoe UI Symbol','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;font-weight:bold;line-height:16.0029983520508px;white-space:nowrap">LLVM support for -fthinlto option.</a><br></div><div><span class="phui-object-item-objname" style="color:rgb(75,77,81);font-family:'Segoe UI','Segoe UI Web Regular','Segoe UI Symbol','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;font-weight:bold;line-height:16.0029983520508px;white-space:nowrap">D11908</span><span style="color:rgb(0,0,0);font-family:'Segoe UI','Segoe UI Web Regular','Segoe UI Symbol','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;font-weight:bold;line-height:16.0029983520508px;white-space:nowrap"> </span><a href="http://reviews.llvm.org/D11908" class="phui-object-item-link" title="Clang support for -fthinlto." style="text-decoration:none;color:rgb(24,85,157);display:inline;font-family:'Segoe UI','Segoe UI Web Regular','Segoe UI Symbol','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;font-weight:bold;line-height:16.0029983520508px;white-space:nowrap">Clang support for -fthinlto.</a><br></div><div><br></div><div>Once the basic options support, data structs, and bitcode support goes in I can send patches for generating/emitting the function index and the combined function index (off by defaut, guarded by the -fthinlto option), and subsequently send patches for the function importing during the backend compile step. I've tried to break down the above infrastructure into small pieces for review, and plan to implement the rest via incremental patches.</div><div><br></div><div>Hope this clarifies the approach I'm taking! Looking forward to additional feedback on the approach and the patches.</div><div><br></div><div>Thanks,</div><div>Teresa</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 12, 2015 at 2:09 PM, Teresa Johnson <span dir="ltr"><<a href="mailto:tejohnson@google.com" target="_blank">tejohnson@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Saw that, thanks! Responding now. Will update the patch with some changes and the wrapper stuff removed later today or very early tomorrow.<span class="HOEnZb"><font color="#888888"><div>Teresa</div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 12, 2015 at 2:07 PM, Philip Reames <span dir="ltr"><<a href="mailto:listmail@philipreames.com" target="_blank">listmail@philipreames.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>I went ahead and replied to two of the
review threads. I only considered the parts which would be left
without the native wrapped bitcode support. <br><span><font color="#888888">
<br>
Philip</font></span><div><div><br>
<br>
On 08/12/2015 01:24 PM, Teresa Johnson wrote:<br>
</div></div></div><div><div>
<blockquote type="cite">
<div dir="ltr">I can remove the native wrapper portions of the
associated patch and add that later. Most of the support as I
mentioned is for the bitcode handling anyway, but I wanted to
include a skeleton of the native wrapper part. For the RFC, I
wanted to show the end goal including how the native wrapper
support would look (it in fact mostly leverages the same bitcode
encoding, so there isn't a lot of difference, and hence there
isn't a whole lot of extra code needed to support that). The
bulk of the RFC deals with the bitcode format, and I would love
some feedback on that.
<div><br>
</div>
<div>Thanks,</div>
<div>Teresa<br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Aug 12, 2015 at 11:50 AM,
Philip Reames <span dir="ltr"><<a href="mailto:listmail@philipreames.com" target="_blank">listmail@philipreames.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Alex already made what I consider to be the most
relevant point. I would suggest removing the
unwanted functionality and asking again. From my
perspective, native wrapped bitcode is only
interesting (and thus worth reviewing and/or talking
about) once the native bitcode version is in tree
and functional. Frankly, I consider the native
wrapped bitcode to be an entirely orthogonal
proposal that shouldn't be tied to ThinLTO at all.
<br>
<br>
Fair warning, I'm not going to be particularly
involved either way. This is far enough from my own
immediate interests that I can't spare the cycles.
I would suggest that you collaborate closely with
the Sony and Apple folks who are already *using* LTO
to find a proposal they're happy with. Until you do
that, you are unlikely to make much progress. <br>
<span><font color="#888888"> <br>
Philip</font></span>
<div>
<div><br>
<br>
On 08/12/2015 09:13 AM, Teresa Johnson wrote:<br>
</div>
</div>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr">
<div>Ping. Explicitly adding a few more people
who commented on the earlier (high-level)
ThinLTO RFC. I removed the body of the RFC
here since the original was large and had
trouble getting through the mailer. I also
updated the patch mentioned below so that it
was emailed to llvm-commits properly.<br>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Teresa</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Aug 3, 2015
at 11:59 AM, Teresa Johnson <span dir="ltr"><<a href="mailto:tejohnson@google.com" target="_blank">tejohnson@google.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Alex,
<div><br>
</div>
<div>After outlining some of the
rationale for using native-wrapped,
there were a couple of responses
that indicated native-wrapped
support was reasonable, but they
preferred to see bitcode-only first
(Phillip and Rafael). This is
essentially what this proposal and
the patches do - I've implemented
some of the basic support for
looking for and parsing the
native-wrapped sections, but the
bitcode-only reading/writing support
is more complete.</div>
<div><br>
</div>
<div>In fact, as described in this
RFC, I designed the native-wrapped
format to utilize the same bitcode
encoding for most of the ThinLTO
information, so it uses most of the
same underlying bitcode interfaces
anyway. The additional support
required for native-wrapped is not
tremendous, and is similar to
existing support in the LLVM tree
for reading native-wrapped bitcode.</div>
<div><br>
</div>
<div>We believe that there will be
clang/llvm users who will find
native-wrapped ThinLTO easier to use
for the same reasons (e.g.
compatibility with existing native
toolchains), so I don't expect this
to be Google specific.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Teresa</div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon,
Aug 3, 2015 at 12:26 PM, Alex
Rosenberg <span dir="ltr"><<a href="mailto:alexr@leftfield.org" target="_blank">alexr@leftfield.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div>I think I've read all
the feedback posted
regarding your May
proposal. I have yet to
see a single response that
wants native object
wrapped bitcode.</div>
<div><br>
</div>
<div>If the only use for
native object wrapped
bitcode is for your
project at Google, then it
probably shouldn't go into
the tree against all of
these objections.<br>
<br>
Alex</div>
<div>
<div><br>
On Aug 3, 2015, at 9:19
AM, Teresa Johnson <<a href="mailto:tejohnson@google.com" target="_blank">tejohnson@google.com</a>>
wrote:<br>
<br>
</div>
<blockquote type="cite">
<div dir="ltr">As
discussed in the
high-level ThinLTO
RFC <span style="font-size:12.8000001907349px">(</span><a href="http://lists.cs.uiuc.edu/pipermail/llvmdev/2015-May/086211.html" style="font-size:12.8000001907349px" target="_blank">http://lists.cs.uiuc.edu/pipermail/llvmdev/2015-May/086211.html</a><span style="font-size:12.8000001907349px">), we would like to add support for
native object
wrapped bitcode and
ThinLTO information.
Based on comments on
the mailing list, I
am adding support
for ThinLTO in both
normal bitcode
files, as well as
native-object
wrapped bitcode.</span>
<div><span style="font-size:12.8000001907349px"><br>
</span></div>
<div><span style="font-size:12.8000001907349px">The
following RFC
describes the
planned file
format of ThinLTO
information both
in the
bitcode-only and
native object
wrapped cases. It
doesn't yet define
the exact record
format, as I would
like feedback on
the overall block
design first.</span></div>
<div><span style="font-size:12.8000001907349px"><br>
</span></div>
<div><span style="font-size:12.8000001907349px">I've
also implemented
the support for
reading and
writing the
bitcode blocks in
the following
patch:</span></div>
<div><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__reviews.llvm.org_D11722&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Mfk2qtn1LTDThVkh6-oGglNfMADXfJdty4_bhmuhMHA&m=oUy_PB_mSfRgDO7H7bZOR04gv_DMzX5rPO_lv4PHt60&s=WVxrKkHnjKr75fCQ-UkGke8dk6KpZcFCnLWVrJ3G188&e=" rel="noreferrer" style="font-size:12.8000001907349px" target="_blank">http://reviews.llvm.org/D11722</a></div>
<div><br>
</div>
<div>The ThinLTO data
structures and the
file APIs are
described in a
separate RFC I will
be sending
simultaneously, with
pointers to the
patches implementing
them.</div>
<div><br>
</div>
<div>Looking forward
to your feedback.
Thanks!</div>
<div>Teresa</div>
<div><br>
</div>
<div><span>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-align:center"><br>
</p>
</span></div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div><span style="font-family:Times;font-size:medium">
<table cellpadding="0" cellspacing="0">
<tbody>
<tr style="color:rgb(85,85,85);font-family:sans-serif;font-size:small">
<td style="border-top-style:solid;border-top-color:rgb(213,15,37);border-top-width:2px" nowrap>Teresa Johnson |</td>
<td style="border-top-style:solid;border-top-color:rgb(51,105,232);border-top-width:2px" nowrap> Software
Engineer |</td>
<td style="border-top-style:solid;border-top-color:rgb(0,153,57);border-top-width:2px" nowrap> <a href="mailto:tejohnson@google.com" target="_blank">tejohnson@google.com</a> |</td>
<td style="border-top-style:solid;border-top-color:rgb(238,178,17);border-top-width:2px" nowrap> <a href="tel:408-460-2413" value="+14084602413" target="_blank">408-460-2413</a></td>
</tr>
</tbody>
</table>
</span></div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div><span style="font-family:Times;font-size:medium">
<table cellpadding="0" cellspacing="0">
<tbody>
<tr style="color:rgb(85,85,85);font-family:sans-serif;font-size:small">
<td style="border-top-style:solid;border-top-color:rgb(213,15,37);border-top-width:2px" nowrap>Teresa Johnson |</td>
<td style="border-top-style:solid;border-top-color:rgb(51,105,232);border-top-width:2px" nowrap> Software Engineer |</td>
<td style="border-top-style:solid;border-top-color:rgb(0,153,57);border-top-width:2px" nowrap> <a href="mailto:tejohnson@google.com" target="_blank">tejohnson@google.com</a> |</td>
<td style="border-top-style:solid;border-top-color:rgb(238,178,17);border-top-width:2px" nowrap> <a href="tel:408-460-2413" value="+14084602413" target="_blank">408-460-2413</a></td>
</tr>
</tbody>
</table>
</span></div>
</div>
</div>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div><span style="font-family:Times;font-size:medium"><table cellspacing="0" cellpadding="0"><tbody><tr style="color:rgb(85,85,85);font-family:sans-serif;font-size:small"><td nowrap style="border-top-style:solid;border-top-color:rgb(213,15,37);border-top-width:2px">Teresa Johnson |</td><td nowrap style="border-top-style:solid;border-top-color:rgb(51,105,232);border-top-width:2px"> Software Engineer |</td><td nowrap style="border-top-style:solid;border-top-color:rgb(0,153,57);border-top-width:2px"> <a href="mailto:tejohnson@google.com" target="_blank">tejohnson@google.com</a> |</td><td nowrap style="border-top-style:solid;border-top-color:rgb(238,178,17);border-top-width:2px"> <a href="tel:408-460-2413" value="+14084602413" target="_blank">408-460-2413</a></td></tr></tbody></table></span></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><span style="font-family:Times;font-size:medium"><table cellspacing="0" cellpadding="0"><tbody><tr style="color:rgb(85,85,85);font-family:sans-serif;font-size:small"><td nowrap style="border-top-style:solid;border-top-color:rgb(213,15,37);border-top-width:2px">Teresa Johnson |</td><td nowrap style="border-top-style:solid;border-top-color:rgb(51,105,232);border-top-width:2px"> Software Engineer |</td><td nowrap style="border-top-style:solid;border-top-color:rgb(0,153,57);border-top-width:2px"> <a href="mailto:tejohnson@google.com" target="_blank">tejohnson@google.com</a> |</td><td nowrap style="border-top-style:solid;border-top-color:rgb(238,178,17);border-top-width:2px"> 408-460-2413</td></tr></tbody></table></span></div>
</div>