[PATCH] D59025: Add --add-ghashes to llvm-objcopy to append a .debug$H to coff objects

[Alexandre Ganea via llvm-commits]

Just an idea - as for the database, maybe if we revived the lld-link /INCREMENTAL discussion, but only for debug info. At the end of the link, we could keep the ghashes and Type merging hashtable in a transient memory-mapped file. On next link, you would remove from the hashtable the types belonging to any changed OBJ(s), and merge-in the new types. But that requires reference counting per type record, along with SeenRecords and SeenHashes. The PDB could be generated as usual, or incrementally as link.exe.
In this manner, you would save probably most of the time spend in "Type Merging". And further on, we could do the same for symbols.

  Input File Reading:          1391 ms (  6.9%)
  Code Layout:                  449 ms (  2.2%)
  PDB Emission (Cumulative):  18019 ms ( 89.6%)
    Add Objects:              13046 ms ( 64.8%)
>>>   Type Merging:            9851 ms ( 49.0%) <<<<
      Symbol Merging:          3070 ms ( 15.3%)
    TPI Stream Layout:          791 ms (  3.9%)
    Globals Stream Layout:      554 ms (  2.8%)
    Commit to Disk:            2769 ms ( 13.8%)
  Commit Output File:            31 ms (  0.2%)
-------------------------------------------------
Total Link Time:              20121 ms (100.0%)


-----Original Message-----
From: Leonardo Santagada <santagada at gmail.com> 
Sent: Monday, March 18, 2019 3:48 PM
To: Zachary Turner <zturner at google.com>
Cc: reviews+D59025+public+309b7d4311d79b0b at reviews.llvm.org; alexander.v.shaposhnikov at gmail.com; Jordan Rupprecht <rupprecht at google.com>; Alexandre Ganea <alexandre.ganea at ubisoft.com>; martin at martin.st; Reid Kleckner <rnk at google.com>; jh7370 at my.bristol.ac.uk; Jake Ehrlich <jakehehrlich at google.com>; Michał Górny <mgorny at gentoo.org>; llvm-commits at lists.llvm.org
Subject: Re: [PATCH] D59025: Add --add-ghashes to llvm-objcopy to append a .debug$H to coff objects

Are there plans on saving the hashes to some form of database? If not I don't see how it could ever compare in performance as this can be done per file (so its completely parallel) and can be done offline for libraries being used.

For us even with the pre-computed hashes it still takes 20 seconds to link which in comparison to incremental link.exe performance is not great (it varies but can link in 4-8 seconds for small changes).

I think by sharding of the pdb generation we can get it to be closer to 8 seconds always, but not if we would still need to compute the hashes for the types.

On Mon, Mar 18, 2019 at 8:40 PM Zachary Turner <zturner at google.com> wrote:
>
> With recent work focusing on computing ghashes in parallel in the linker, which would ultimately eliminate the need to write them to the object file entirely, does this still have the same value?
> On Mon, Mar 18, 2019 at 12:32 PM Alexandre Ganea via Phabricator <reviews at reviews.llvm.org> wrote:
>>
>> aganea added a comment.
>>
>> In D59025#1433050 <https://reviews.llvm.org/D59025#1433050>, @jhenderson wrote:
>>
>> > 2. There's a real-world use-case (which could be as simple as significantly simplifying our own testing or whatever).
>>
>>
>> There's a strong industrial use-case here, which is to use MSVC for compilation and LLD for linking.
>> MSVC currently doesn't support (yet?) `.debug$H` streams, and this patch provides a workaround. This inherently makes linking with LLD faster, by allowing compilation with MSVC + `.debug$H` computation on remote PCs (when using distributed compilation).
>> Without this patch, `.debug$H` computation is done locally by LLD, which significantly increases link time.
>>
>> In D59025#1428431 <https://reviews.llvm.org/D59025#1428431>, @rupprecht wrote:
>>
>> > I've never heard of ghashes
>>
>>
>> Worth the read: http://blog.llvm.org/2018/01/
>>
>>
>> Repository:
>>   rG LLVM Github Monorepo
>>
>> CHANGES SINCE LAST ACTION
>>   https://reviews.llvm.org/D59025/new/
>>
>> https://reviews.llvm.org/D59025
>>
>>
>>


-- 

Leonardo Santagada