<div dir="ltr">Yes we've discussed many different ideas for incremental linking, but our conclusion is that you can only get one of Fast|Simple. If you want it to be fast it has to be complicated and if you want it to be simple then it's going to be slow.<div><br></div><div>Consider the case where you edit one .cpp file and change this:</div><div><br></div><div>int x = 0, y = 7;</div><div><br></div><div>to this:</div><div><br></div><div>int x = 0;</div><div>short y = 7;</div><div><br></div><div>Because different instructions operate on shorts vs ints, some of the instruction encodings will be different and potentially of a different size.</div><div><br></div><div>Because of this, the contribution to the .text section from this object file is going to be a different size.</div><div><br></div><div>Because of that, all subsequent object files will start at a different absolute file address in the final executable. </div><div><br></div><div>Because of that, every single symbol in every single object file will need to be updated in the final PDB.</div><div><br></div><div>There are many other things that need to happen as well, but the point is that trivial change to a cpp file can explode into many changes in the final PDB.</div><div><br></div><div>There are ways to handle this, but they're not simple. We have some ideas, but for the moment we are focused on making full linking as fast as possible because it's much easier and still provides benefits. We think we can get it fast enough that it will be acceptable, and that should give us some extra time to do incremental linking properly.</div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Jan 29, 2018 at 1:07 PM Leonardo Santagada <<a href="mailto:santagada@gmail.com">santagada@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">About incremental linking, the only thing from my benchmark that needs to be incremental is the pdb patching as generating the binary seems faster than incremental linking on link.exe, so did anyone propose renaming the current binary, writing a new one and then diffing the coff obj and using that info to just rewriting that part of the pdb. Or another idea is making the build system feed into the linker which files changed so the types/debug information can be compared instead of all of them?<br></div><div class="gmail_extra"></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 29, 2018 at 7:55 PM, Zachary Turner <span dir="ltr"><<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Not a lot.<div><br></div><div>/TIME will show high level timing of the various phases (this is the same option MSVC uses).</div><div><br></div><div>If you want anything more detailed than that, vTune or ETW+WPA (<a href="https://github.com/google/UIforETW/releases" target="_blank">https://github.com/google/UIforETW/releases</a>) are probably what you'll need to do.</div><div><br></div><div>(We'd definitely love patches to improve performance, or even just ideas about how to make things faster. Improving link speed is one of our biggest priorities.)</div></div><div class="m_6677234159946451931HOEnZb"><div class="m_6677234159946451931h5"><br><div class="gmail_quote"><div dir="ltr">On Mon, Jan 29, 2018 at 10:47 AM Leonardo Santagada <<a href="mailto:santagada@gmail.com" target="_blank">santagada@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">Yeah true, is there any switches to profile the linker?</div><div class="gmail_extra"><br><div class="gmail_quote">On 29 Jan 2018 18:43, "Zachary Turner" <<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Part of the reason why lld is so fast is because we map every input file into memory up front and rely on the virtual memory manager in the kernel to make this fast. Generally speaking, this is a lot faster than opening a file, reading it and processing a file, and closing the file. The downside, as you note, is that it uses a lot of memory. <div><br></div><div>But there's a catch. The kernel is smart enough to share the physical memory pages when you map the same file multiple times from multiple processes. So it only looks like the memory usage is high because it reserves a large amount of address space in each process. But the total amount of physical memory used will not increase when additional instances of the same file are mapped.</div><div><div><br><div class="gmail_quote"><div dir="ltr">On Mon, Jan 29, 2018 at 9:24 AM Leonardo Santagada <<a href="mailto:santagada@gmail.com" target="_blank">santagada@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>I cleaned up my tests and figured that the obj file generated with problems was only with msvc 2015, so trying again with msvc 2017 I get: <br></div><div><br></div><div>lld-link: 4s<br></div><div>lld-link /debug: 1m30s and ~20gb of ram</div><div>lld-link /debug:ghash: 59s and ~20gb of ram</div><div>link: 13s<br></div><div>link /debug:fastlink: 43s and 1gb of ram</div><div>link specialpdb: 1m10s and 4gb of ram</div><div>link /debug: 9m16s min and >14gb of ram</div><div><br></div><div>link incremental: 8s when it works.<br></div><div><br></div><div><br></div><div>*specialpdb is created with passing to a set of compilation units (eg a folder) the same pdb to be written to, so it dedups the symbols before the final linking, but that does decrease the concurrency as this step can't be done after linking.</div><div><br></div><div><br></div><div>My question is, in the set of patches you guys haven't upstreamed is there anything that makes compilation uses less memory? Or just asking more directly, when will those patches make to upstream, or can I try them? The memory usage of lld-link is a little worrying as we have around 6-8 binaries that we link for windows and they mostly use the same libraries so 20gb of ram each means we probably can't link them all together anymore.<br></div><div><br></div><div></div><div><br></div><div>Tomorrow I will send my tool and changes to lld so more people can try this out and tell if it helps with their msvc only code.<br></div><div><br></div></div><div class="gmail_extra"></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jan 28, 2018 at 11:22 PM, Zachary Turner <span dir="ltr"><<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I don’t have pgo numbers. When I build using -flto=thin the link time is significantly faster than msvc /ltcg and runtime is slightly faster, but I haven’t tested on a large variety of different workloads, so YMMV. Link time will definitely be faster though <br><div class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846HOEnZb"><div class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846h5"><div class="gmail_quote"><div dir="ltr">On Sun, Jan 28, 2018 at 2:20 PM Leonardo Santagada <<a href="mailto:santagada@gmail.com" target="_blank">santagada@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>This part is only for objects with /Z7 debug information in them right? I think most of the third parties are either: .lib/obj without debug information, the same with information on pdb files. Rewriting all .lib/.obj with /Z7 information seems doable with a small python script, the pdb one is going to be more work, but I always wanted to know how a pdb file is structured so "fun" times ahead. But yeah printing it out, and timing it might be very useful indeed.</div><div><br></div><div>Did anyone tried to compile/link lld-link.exe with LTO+PGO to see how much faster can it get? I might try that as well, as 10% speed improvement might be handy.<br></div></div><div class="gmail_extra"></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jan 28, 2018 at 11:14 PM, Zachary Turner <span dir="ltr"><<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Look for this code in lld/coff/pdb.cpp<br><br><br>if (Config->DebugGHashes) {<br> ArrayRef<GloballyHashedType> Hashes;<br> std::vector<GloballyHashedType> OwnedHashes;<br> if (Optional<ArrayRef<uint8_t>> DebugH = getDebugH(File))<br> Hashes = getHashesFromDebugH(*DebugH);<br> else {<br> OwnedHashes = GloballyHashedType::hashTypes(Types);<br> Hashes = OwnedHashes;<br> }<br><br>In the else block there, add a log message that says “synthesizing .debug$h section for “ + Obj->Name<br><br>See how many of these you get. When I build chrome + all third party libraries this way i get about 100, which is small enough to still see large performance gains.<br><br>If you have many 3rd party libraries, it may be necessary to rewrite the .lib files too, not just the .obj files. Eventually I’ll get around to implementing all of this as well, as well as better heuristics in lld-link to disable ghash if it’s going to be slow <br><div class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331HOEnZb"><div class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331h5"><div class="gmail_quote"><div dir="ltr">On Sun, Jan 28, 2018 at 1:51 PM Leonardo Santagada <<a href="mailto:santagada@gmail.com" target="_blank">santagada@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Ok I went for kind of middle ground solution, I patch in the obj files, but as adding a new section didn't seem to work, I add a "shadow" section, by editing the pointer to line number and the virtual size on the .debug$T section. Although technically broken, both link.exe and lld-link.exe don't seem to mind the alterations and as the shadow .debug$H is not really a section anymore (its just some bytes at the end of the file) it doesn't change anything else that does matter. With that I could do my first test with a subset of our code base, and the results are not good. I found one of our sources that break the ghash computation, I will get more info on this and post a proper bug report, but I guess its type information that is generated only by msvc. The other more alarming problem is that linking is way slower with the ghahes... my guess is that we have a bunch of pdb files for some third party libraries and calculating those ghashes takes more time than actual linking of this small part of the source (it links in 4s in both link.exe and lld-link.exe without ghashes).<br></div><div class="gmail_extra"></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 26, 2018 at 8:52 PM, Leonardo Santagada <span dir="ltr"><<a href="mailto:santagada@gmail.com" target="_blank">santagada@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">We don't generate any .lib as those don't work well with incremental linking (and give zero advantages when linking AFAIK), and it would be pretty easy to have a modern format for having a .ghash for multiple files, something simple like size prefixed name and then size prefixed ghash blobs.<br></div><div class="gmail_extra"><div><div class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331m_-4675903424918970886m_3803827337711028499h5"><br><div class="gmail_quote">On Fri, Jan 26, 2018 at 8:44 PM, Zachary Turner <span dir="ltr"><<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">We considered that early on, but most object files actually end up in .lib files so unless there were a way to connect the objects in the .lib to the corresponding .ghash files, this would disable ghash usage for a large amount of inputs. Supporting both is an option, but it adds a bit of complexity and I’m not totally convinced it’s worth it<div class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331m_-4675903424918970886m_3803827337711028499m_2230044439106237947HOEnZb"><div class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331m_-4675903424918970886m_3803827337711028499m_2230044439106237947h5"><br><div class="gmail_quote"><div dir="ltr">On Fri, Jan 26, 2018 at 11:38 AM Leonardo Santagada <<a href="mailto:santagada@gmail.com" target="_blank">santagada@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>it does.</div><div><br></div><div>I just had an epiphany: why not just write a .ghash file and have lld read those if they exist for an .obj file?</div><div><br></div><div>Seem much simpler than trying to wire up a 20 year old file format. I will try to do this, is something like this acceptable for LLD? The cool thing is that I can generate .ghash for .lib or any obj lying around (maybe even for pdb in the future).<br></div></div><div class="gmail_extra"></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 26, 2018 at 8:32 PM, Zachary Turner <span dir="ltr"><<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">In general, we should be able to accept any MSVC .obj file to LLD. At the very least, we're not aware of any cases that don't work.<div><br></div><div>Does your MSVC .obj file link fine before you add the .debug$H?</div></div><div class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331m_-4675903424918970886m_3803827337711028499m_2230044439106237947m_-8035360309018806218m_-4064037938399874594HOEnZb"><div class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331m_-4675903424918970886m_3803827337711028499m_2230044439106237947m_-8035360309018806218m_-4064037938399874594h5"><br><div class="gmail_quote"><div dir="ltr">On Fri, Jan 26, 2018 at 11:23 AM Leonardo Santagada <<a href="mailto:santagada@gmail.com" target="_blank">santagada@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Okay, apparently coff2yaml and yaml2coff are not in a great place as they both don't deal well with the fact that you can have overlapping sections, which seems to be what clang-cl produces (the .data section points to the same place as a later section). Which is not a big big problem for me particularly because msvc doesn't even generate .data sections in .obj.</div><div><br></div><div>I'm trying to put support for .bss sections in both coff2yaml and yaml2coff... but I still can link just fine with my transformations clang-cl generated files... what does give me problems is msvc .obj files. Have you tried to link one of these?<br></div></div><div class="gmail_extra"></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 26, 2018 at 8:05 PM, Leonardo Santagada <span dir="ltr"><<a href="mailto:santagada@gmail.com" target="_blank">santagada@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331m_-4675903424918970886m_3803827337711028499m_2230044439106237947m_-8035360309018806218m_-4064037938399874594m_4468285616015269320m_-9029411802399524950m_4545078661566976982gmail-ajy"><img class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331m_-4675903424918970886m_3803827337711028499m_2230044439106237947m_-8035360309018806218m_-4064037938399874594m_4468285616015269320m_-9029411802399524950m_4545078661566976982gmail-ajz" id="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331m_-4675903424918970886m_3803827337711028499m_2230044439106237947m_-8035360309018806218m_-4064037938399874594m_4468285616015269320m_-9029411802399524950m_4545078661566976982gmail-:od" src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt=""></div><span>yeah,
apparently .bss has a flag of unitialized data that is not being
respected on the layout of the coff files (it should skip those
sections) but I dunno what to do with .data as it doesn't have a size.<div class="gmail_extra"><br></div></span><div class="gmail_extra">(resending as apparently my pastes generated a ton of hidden html data and this message hit the mailinglist limit of 100k)<span class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331m_-4675903424918970886m_3803827337711028499m_2230044439106237947m_-8035360309018806218m_-4064037938399874594m_4468285616015269320m_-9029411802399524950HOEnZb"><font color="#888888"><br></font></span></div><span class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331m_-4675903424918970886m_3803827337711028499m_2230044439106237947m_-8035360309018806218m_-4064037938399874594m_4468285616015269320m_-9029411802399524950HOEnZb"><font color="#888888"><div class="gmail_extra">-- <br><div class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331m_-4675903424918970886m_3803827337711028499m_2230044439106237947m_-8035360309018806218m_-4064037938399874594m_4468285616015269320m_-9029411802399524950m_4545078661566976982gmail_signature" data-smartmail="gmail_signature"><br>Leonardo Santagada</div>
</div></font></span></div>
</blockquote></div><br><br clear="all"><br></div><div class="gmail_extra">-- <br><div class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331m_-4675903424918970886m_3803827337711028499m_2230044439106237947m_-8035360309018806218m_-4064037938399874594m_4468285616015269320m_-9029411802399524950gmail_signature" data-smartmail="gmail_signature"><br>Leonardo Santagada</div>
</div></blockquote></div>
</div></div></blockquote></div><br><br clear="all"><br></div><div class="gmail_extra">-- <br><div class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331m_-4675903424918970886m_3803827337711028499m_2230044439106237947m_-8035360309018806218m_-4064037938399874594gmail_signature" data-smartmail="gmail_signature"><br>Leonardo Santagada</div>
</div></blockquote></div>
</div></div></blockquote></div><br><br clear="all"><br></div></div><span class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331m_-4675903424918970886m_3803827337711028499HOEnZb"><font color="#888888">-- <br><div class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331m_-4675903424918970886m_3803827337711028499m_2230044439106237947gmail_signature" data-smartmail="gmail_signature"><br>Leonardo Santagada</div>
</font></span></div>
</blockquote></div><br><br clear="all"><br></div><div class="gmail_extra">-- <br><div class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331m_-4675903424918970886m_3803827337711028499gmail_signature" data-smartmail="gmail_signature"><br>Leonardo Santagada</div>
</div></blockquote></div>
</div></div></blockquote></div><br><br clear="all"><br></div><div class="gmail_extra">-- <br><div class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846m_-626744413999063384m_8983887394214012331gmail_signature" data-smartmail="gmail_signature"><br>Leonardo Santagada</div>
</div></blockquote></div>
</div></div></blockquote></div><br><br clear="all"><br></div><div class="gmail_extra">-- <br><div class="m_6677234159946451931m_-801937299320335964m_-6506230472531316519m_6754292886016693205m_7202138444592024370m_1407297647195485846gmail_signature" data-smartmail="gmail_signature"><br>Leonardo Santagada</div>
</div></blockquote></div></div></div></div>
</blockquote></div></div>
</blockquote></div>
</div></div></blockquote></div><br><br clear="all"><br></div><div class="gmail_extra">-- <br><div class="m_6677234159946451931gmail_signature" data-smartmail="gmail_signature"><br>Leonardo Santagada</div>
</div></blockquote></div>