<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Sep 16, 2015 at 4:08 PM, David Blaikie <span dir="ltr"><<a href="mailto:dblaikie@gmail.com" target="_blank">dblaikie@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Wed, Sep 16, 2015 at 3:46 PM, Rui Ueyama via llvm-commits <span dir="ltr"><<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>This optimization seems very effective. I picked up a program mksnapshot.exe from Chromium and measure the time LLD spends on ICF. The program contains 27645 COMDAT sections, of which 4501 sections are redundant. The output size after ICF is 9MB.</div><div><br></div><div>Without this patch, LLD takes 93 ms to scan and group all COMDAT sections. The number of hash collisions that had to be resolved by compare section contents was 1335211.</div><div><br></div><div>With this patch, LLD takes only 33 ms to do the same thing. There was *no* false hash collision (but we still had to compare contents for positive case because the section hash value is just a CRC32 and not a cryptographic one).</div><div class="gmail_extra"><br></div><div class="gmail_extra">So this is working as we hoped, and that's probably important because ICF is on by default.</div></div></blockquote><div><br></div></span><div>\o/<br><br>Got any comparative numbers with link.exe? (specifically I was curious about your recent fixes to ICF which mention smaller binary size than link.exe & I was wondering if you had any ideas what the difference was there - but also just tracking the exact comdat functions and deduplication & see if we get exactly the same answers as link.exe?)</div></div></div></div></blockquote><div><br></div><div>Ran link.exe with the same command line options, and here's the table. Unit is second. Probably this program is too small to use for benchmark. I'll test this with different programs.</div><div><br></div><div><div><font face="monospace, monospace"> MSVC LLD</font></div><div><font face="monospace, monospace">ICF on 1.43 0.98</font></div><div><font face="monospace, monospace">ICF off 1.41 0.93</font></div></div><div><br></div><div>As to the small difference in size between LLD and link.exe, I have no good explanation yet. I was thinking that that can even be a sign of a hidden bug that LLD reduces sections too aggressively. Or that's just because of a difference of section layouts.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="h5"><div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 4, 2015 at 1:45 PM, Rui Ueyama via llvm-commits <span dir="ltr"><<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Author: ruiu<br>
Date: Fri Sep 4 15:45:50 2015<br>
New Revision: 246869<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=246869&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project?rev=246869&view=rev</a><br>
Log:<br>
COFF: Use section content checksum for ICF.<br>
<br>
Previously, we calculated our own hash values for section contents.<br>
Of coruse that's slow because we had to access all bytes in sections.<br>
Fortunately, COFF objects usually contain hash values for COMDAT<br>
sections. We can use that to speed up Identical COMDAT Folding.<br>
<br>
Modified:<br>
lld/trunk/COFF/Chunks.h<br>
lld/trunk/COFF/ICF.cpp<br>
lld/trunk/COFF/InputFiles.cpp<br>
<br>
Modified: lld/trunk/COFF/Chunks.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/lld/trunk/COFF/Chunks.h?rev=246869&r1=246868&r2=246869&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/lld/trunk/COFF/Chunks.h?rev=246869&r1=246868&r2=246869&view=diff</a><br>
==============================================================================<br>
--- lld/trunk/COFF/Chunks.h (original)<br>
+++ lld/trunk/COFF/Chunks.h Fri Sep 4 15:45:50 2015<br>
@@ -183,6 +183,10 @@ public:<br>
// and this chunk is considrered as dead.<br>
SectionChunk *Ptr;<br>
<br>
+ // The CRC of the contents as described in the COFF spec 4.5.5.<br>
+ // Auxiliary Format 5: Section Definitions. Used for ICF.<br>
+ uint32_t Checksum = 0;<br>
+<br>
private:<br>
ArrayRef<uint8_t> getContents() const;<br>
<br>
<br>
Modified: lld/trunk/COFF/ICF.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/lld/trunk/COFF/ICF.cpp?rev=246869&r1=246868&r2=246869&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/lld/trunk/COFF/ICF.cpp?rev=246869&r1=246868&r2=246869&view=diff</a><br>
==============================================================================<br>
--- lld/trunk/COFF/ICF.cpp (original)<br>
+++ lld/trunk/COFF/ICF.cpp Fri Sep 4 15:45:50 2015<br>
@@ -44,7 +44,7 @@ uint64_t SectionChunk::getHash() const {<br>
NumRelocs,<br>
uint32_t(Header->SizeOfRawData),<br>
std::distance(Relocs.end(), Relocs.begin()),<br>
- hash_combine_range(A.data(), A.data() + A.size()));<br>
+ Checksum);<br>
}<br>
<br>
// Returns true if this and a given chunk are identical COMDAT sections.<br>
@@ -58,6 +58,8 @@ bool SectionChunk::equals(const SectionC<br>
return false;<br>
if (NumRelocs != X->NumRelocs)<br>
return false;<br>
+ if (Checksum != X->Checksum)<br>
+ return false;<br>
<br>
// Compare data<br>
if (getContents() != X->getContents())<br>
<br>
Modified: lld/trunk/COFF/InputFiles.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/lld/trunk/COFF/InputFiles.cpp?rev=246869&r1=246868&r2=246869&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/lld/trunk/COFF/InputFiles.cpp?rev=246869&r1=246868&r2=246869&view=diff</a><br>
==============================================================================<br>
--- lld/trunk/COFF/InputFiles.cpp (original)<br>
+++ lld/trunk/COFF/InputFiles.cpp Fri Sep 4 15:45:50 2015<br>
@@ -225,13 +225,14 @@ Defined *ObjectFile::createDefined(COFFS<br>
if (!SC)<br>
return nullptr;<br>
<br>
- // Handle associative sections<br>
+ // Handle section definitions<br>
if (IsFirst && AuxP) {<br>
auto *Aux = reinterpret_cast<const coff_aux_section_definition *>(AuxP);<br>
if (Aux->Selection == IMAGE_COMDAT_SELECT_ASSOCIATIVE)<br>
if (auto *ParentSC = cast_or_null<SectionChunk>(<br>
SparseChunks[Aux->getNumber(Sym.isBigObj())]))<br>
ParentSC->addAssociative(SC);<br>
+ SC->Checksum = Aux->CheckSum;<br>
}<br>
<br>
auto *B = new (Alloc) DefinedRegular(this, Sym, SC);<br>
<br>
<br>
_______________________________________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br>
</blockquote></div><br></div></div></div></div>
<br>_______________________________________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br>
<br></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div></div>