[all-commits] [llvm/llvm-project] 87fb0e: [BOLT][DWARF] Implement new mechanism for DWARFRew...

Mon Jul 10 14:42:42 PDT 2023

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 87fb0ea27eebceaa1aa1ab0b6ede9689265178e0
      https://github.com/llvm/llvm-project/commit/87fb0ea27eebceaa1aa1ab0b6ede9689265178e0
  Author: Rui Zhong <reversezr33 at gmail.com>
  Date:   2023-07-10 (Mon, 10 Jul 2023)

  Changed paths:
    A bolt/include/bolt/Core/DIEBuilder.h
    M bolt/include/bolt/Core/DebugData.h
    M bolt/include/bolt/Rewrite/DWARFRewriter.h
    M bolt/lib/Core/CMakeLists.txt
    A bolt/lib/Core/DIEBuilder.cpp
    M bolt/lib/Core/DebugData.cpp
    M bolt/lib/Rewrite/CMakeLists.txt
    M bolt/lib/Rewrite/DWARFRewriter.cpp
    M bolt/lib/Rewrite/RewriteInstance.cpp
    M bolt/test/X86/asm-func-debug.test
    M bolt/test/X86/dwarf3-lowpc-highpc-convert.s
    M bolt/test/X86/dwarf4-ftypes-dwp-input-dwo-output.test
    M bolt/test/X86/dwarf4-ftypes-dwp-input-dwp-output.test
    M bolt/test/X86/dwarf5-dwarf4-monolithic.test
    M bolt/test/X86/dwarf5-loclist-offset-form.test
    M bolt/test/X86/dwarf5-split-dwarf4-monolithic.test
    M bolt/test/X86/shared-abbrev.s
    M llvm/include/llvm/CodeGen/DIE.h
    M llvm/include/llvm/DebugInfo/DWARF/DWARFFormValue.h
    M llvm/lib/DebugInfo/DWARF/DWARFFormValue.cpp

  Log Message:
  -----------
  [BOLT][DWARF] Implement new mechanism for DWARFRewriter

This revision implement new mechanism for DWARFRewriter.
In the new mechanism, we adopt the same way with DWARFLinker did.
By parsing Debug information into IR, we are allowed to handle debug information more flexible.
Now the debug information updating process relies on IR and IR will be written out to binary once the updating finished.

A new class was added: DIEBuilder. This class is responsible for parsing debug information and raising it to the IR level.
This class is also used to write out the .debug_info and .debug_abbrev sections.
Since we output brand new Abbrev section we won't need to always convert low_pc/high_pc into ranges.
When conversion does happen we can also remove low_pc entry.

Reviewed By: maksfb, ayermolo

Differential Revision: https://reviews.llvm.org/D130315

  Commit: c33536e9c3068ae45f34f6e1c562d51cb8aa702a
      https://github.com/llvm/llvm-project/commit/c33536e9c3068ae45f34f6e1c562d51cb8aa702a
  Author: Alexander Yermolovich <ayermolo at meta.com>
  Date:   2023-07-10 (Mon, 10 Jul 2023)

  Changed paths:
    M bolt/include/bolt/Core/DIEBuilder.h
    M bolt/include/bolt/Core/DebugData.h
    M bolt/include/bolt/Rewrite/DWARFRewriter.h
    M bolt/lib/Core/DIEBuilder.cpp
    M bolt/lib/Core/DebugData.cpp
    M bolt/lib/Rewrite/DWARFRewriter.cpp
    A bolt/test/X86/Inputs/dwarf4-duplicate-types-helper.s
    A bolt/test/X86/Inputs/dwarf4-duplicate-types-main.s
    A bolt/test/X86/Inputs/dwarf4-gdb-index-types-helper.s
    A bolt/test/X86/Inputs/dwarf4-gdb-index-types-main.s
    A bolt/test/X86/Inputs/dwarf4-gdb-index-types-v8.gdb-index
    A bolt/test/X86/Inputs/dwarf4-split-gdb-index-types-helper.s
    A bolt/test/X86/Inputs/dwarf4-split-gdb-index-types-main.s
    A bolt/test/X86/Inputs/dwarf4-split-gdb-index-types-v8.gdb-index
    A bolt/test/X86/Inputs/dwarf4-types-dwarf5-helper.s
    A bolt/test/X86/Inputs/dwarf4-types-dwarf5-main.s
    A bolt/test/X86/Inputs/dwarf4-types-dwarf5-types-helper.s
    A bolt/test/X86/Inputs/dwarf4-types-dwarf5-types-main.s
    A bolt/test/X86/Inputs/dwarf4-types-helper.s
    A bolt/test/X86/Inputs/dwarf4-types-helper2.s
    A bolt/test/X86/Inputs/dwarf4-types-main.s
    A bolt/test/X86/Inputs/dwarf5-dwarf4-gdb-index-types-v8.generted-gdb11.gdb-index
    A bolt/test/X86/Inputs/dwarf5-dwarf4-gdb-index-types-v8.generted-gdb9.gdb-index
    A bolt/test/X86/Inputs/dwarf5-dwarf4-types-backward-forward-cross-reference-helper.s
    A bolt/test/X86/Inputs/dwarf5-dwarf4-types-backward-forward-cross-reference-main.s
    A bolt/test/X86/Inputs/dwarf5-gdb-index-types-helper.s
    A bolt/test/X86/Inputs/dwarf5-gdb-index-types-main.s
    A bolt/test/X86/Inputs/dwarf5-gdb-index-types-v8.generted-gdb11.gdb-index
    A bolt/test/X86/Inputs/dwarf5-gdb-index-types-v8.generted-gdb9.gdb-index
    A bolt/test/X86/Inputs/dwarf5-locexpr-referrence-helper.s
    A bolt/test/X86/Inputs/dwarf5-locexpr-referrence-main.s
    A bolt/test/X86/Inputs/dwarf5-split-gdb-index-types-helper.s
    A bolt/test/X86/Inputs/dwarf5-split-gdb-index-types-main.s
    A bolt/test/X86/Inputs/dwarf5-split-gdb-index-types-v8.gdb-index
    A bolt/test/X86/Inputs/dwarf5-types-helper2.s
    A bolt/test/X86/dwarf4-duplicate-types.test
    A bolt/test/X86/dwarf4-ftypes-dwo-mono-input-dwp-output.test
    A bolt/test/X86/dwarf4-gdb-index-types-gdb-generated.test
    A bolt/test/X86/dwarf4-gdb-index-types-lld-generated.test
    A bolt/test/X86/dwarf4-sibling.s
    A bolt/test/X86/dwarf4-split-gdb-index-types-gdb-generated.test
    A bolt/test/X86/dwarf4-types-dwarf5-types.test
    A bolt/test/X86/dwarf4-types-dwarf5.test
    A bolt/test/X86/dwarf4-types-forward-backward-cross-reference.s
    A bolt/test/X86/dwarf4-types.test
    A bolt/test/X86/dwarf5-dwarf4-gdb-index-types-gdb-generated-gdb11.test
    A bolt/test/X86/dwarf5-dwarf4-gdb-index-types-gdb-generated-gdb9.test
    A bolt/test/X86/dwarf5-dwarf4-gdb-index-types-lld-generated.test
    A bolt/test/X86/dwarf5-dwarf4-types-backward-forward-cross-reference.test
    A bolt/test/X86/dwarf5-ftypes-dwo-mono-input-dwp-output.test
    A bolt/test/X86/dwarf5-gdb-index-types-gdb-generated-gdb11.test
    A bolt/test/X86/dwarf5-gdb-index-types-gdb-generated-gdb9.test
    A bolt/test/X86/dwarf5-gdb-index-types-lld-generated.test
    A bolt/test/X86/dwarf5-locexpr-referrence.test
    A bolt/test/X86/dwarf5-split-gdb-index-types-gdb-generated.test
    A bolt/test/X86/dwarf5-types-backward-cross-reference.s
    A bolt/test/X86/dwarf5-types-forward-cross-reference.s

  Log Message:
  -----------
  [BOLT][DWARF] Numerous fixes for a new DWARFRewriter

* Some cleanup and minor fixes for the new debug information re-writer before moving on
to productatization.

* The new rewriter wasn't handling binary with DWARF5 and DWARF4 with
-fdebug-types-sections.

* Removed dead cross cu reference code.

* Added support for DW_AT_sibling.

* With the new re-writer abbrev number can change which can lead to offset of Type
Units changing. Before we would just copy raw data. Changed to write out Type
Unit List. This is generated by gdb-add-index.

* Fixed how bolt handles gdb-index generated by gdb-11 with types sections.
Simplified logic that handles variations of gdb-index.

* Clang can generate two type units with the same hash, but different content. LLD
does not de-duplicate when ThinLTO is involved. Changed so that TU hash and
offset are used to make TU's unique.

* It is possible to have references within location expression to another DIE.
Fixed it so that relative offset is updated correctly.

* Removed all the code related to patching.

* Removed dead code. Changed how we handling writting out TUs and TU Index. It now
  should fully work for DWARF4 and DWARF5.

* Removed unused arguments from some APIs, changed return type to void, and other
small cleanups.

Reviewed By: maksfb

Differential Revision: https://reviews.llvm.org/D151906

  Commit: 8362418748ae5f5bcad36708cfd9407b5ff88261
      https://github.com/llvm/llvm-project/commit/8362418748ae5f5bcad36708cfd9407b5ff88261
  Author: Alexander Yermolovich <ayermolo at meta.com>
  Date:   2023-07-10 (Mon, 10 Jul 2023)

  Changed paths:
    M bolt/include/bolt/Rewrite/DWARFRewriter.h
    M bolt/lib/Rewrite/DWARFRewriter.cpp

  Log Message:
  -----------
  [BOLT][DWARF] Output DWO files as they are being processed

Changed how we handle writing out .dwo and .dwp files. We now write out DWO
sections sooner and destroy DIEBuilder. This should decrease memory footprint.

Ran on clang-17 build in debug mode with split-dwarf.
before
8:07.49 real,   664.62 user,    69.00 sys,      0 amem, 41601612 mmem
8:07.06 real,   669.60 user,    68.75 sys,      0 amem, 41822588 mmem
8:00.36 real,   664.14 user,    66.36 sys,      0 amem, 41561548 mmem

after
8:21.85 real,   682.23 user,    69.64 sys,      0 amem, 39379880 mmem
8:04.58 real,   671.62 user,    66.50 sys,      0 amem, 39735800 mmem
8:10.02 real,   680.67 user,    67.24 sys,       0 amem, 39662888 mmem

Reviewed By: maksfb

Differential Revision: https://reviews.llvm.org/D151908

  Commit: dcfa2ab534c8f1b4345012a02f130a0a07060bfd
      https://github.com/llvm/llvm-project/commit/dcfa2ab534c8f1b4345012a02f130a0a07060bfd
  Author: Alexander Yermolovich <ayermolo at meta.com>
  Date:   2023-07-10 (Mon, 10 Jul 2023)

  Changed paths:
    M bolt/include/bolt/Core/DIEBuilder.h
    M bolt/include/bolt/Core/DebugData.h
    M bolt/include/bolt/Rewrite/DWARFRewriter.h
    M bolt/lib/Core/DIEBuilder.cpp
    M bolt/lib/Core/DebugData.cpp
    M bolt/lib/Rewrite/DWARFRewriter.cpp
    A bolt/test/X86/Inputs/dwarf4-cross-reference-different-abbrev-dst.s
    A bolt/test/X86/Inputs/dwarf4-cross-reference-different-abbrev-src.s
    M bolt/test/X86/Inputs/dwarf5-helper1-addr-section-reuse.s
    M bolt/test/X86/Inputs/dwarf5-helper2-addr-section-reuse.s
    A bolt/test/X86/dwarf4-cross-cu-backward-different-abbrev.test
    A bolt/test/X86/dwarf4-cross-cu-forward-different-abbrev.test
    M bolt/test/X86/dwarf4-types-dwarf5-types.test
    M bolt/test/X86/dwarf5-gdb-index-types-gdb-generated-gdb11.test
    M bolt/test/X86/dwarf5-gdb-index-types-gdb-generated-gdb9.test
    M bolt/test/X86/dwarf5-gdb-index-types-lld-generated.test
    M bolt/test/X86/dwarf5-locexpr-referrence.test

  Log Message:
  -----------
  [BOLT][DWARF] Change to process and write out TUs first then CUs in batches

To reduce memory footprint changed so that we process and write out TUs first,
reset DIEBuilder and process CUs. CUs are processed in buckets. First bucket
contains all the CUs with cross CU references. Rest processd one at a time.

clang-17 build in debug mode, by clang-17.
before
8:25.81 real, 834.37 user, 86.03 sys, 0 amem, 79525064 mmem
8:02.20 real, 820.46 user, 81.81 sys, 0 amem, 79501616 mmem
7:52.69 real, 802.01 user, 83.99 sys, 0 amem, 79534392 mmem

after
7:49.35 real, 822.04 user, 66.19 sys, 0 amem, 34934260 mmem
7:42.16 real, 825.46 user, 63.52 sys, 0 amem, 34951660 mmem
7:46.71 real, 821.11 user, 63.14 sys, 0 amem, 34981164 mmem

Reviewed By: maksfb

Differential Revision: https://reviews.llvm.org/D151909

Compare: https://github.com/llvm/llvm-project/compare/c1885d2dfa95...dcfa2ab534c8