[llvm-dev] GSoC'20 Add DWARF support to yaml2obj

[Xing GUO via llvm-dev]

Hi folks,

The GSoC'20 comes to an end. My project is adding DWARF support to
yaml2obj. Now, the usability of yaml2obj is improved. Some of the
outstanding improvements are listed below:

* The Length fields of DWARF sections are replaced with Format and Length.

At first, we have to hardcode the TotalLength and TotalLength64 fields
to instruct the tool to emit a proper DWARF32 or a DWARF64 section,
e.g.,

```
## To handcraft a DWARF32 debug_aranges section.
debug_aranges:
  - Length:
      TotalLength: 0x1234
    Version: 2
    ...

## To handcraft a DWARF64 debug_aranges section.
debug_aranges:
  - Length:
      TotalLength: 0xffffffff
      TotalLength64: 0x1234
    Version: 2
    ...
```

Now, yaml2obj emits DWARF32 sections by default and the Length field
can be omitted, yaml2obj will calculate it for us (Patches that
address this issue: https://reviews.llvm.org/D82622 ,
https://reviews.llvm.org/D85880 , https://reviews.llvm.org/D81063 ,
https://reviews.llvm.org/D84008 , https://reviews.llvm.org/D86590 ,
https://reviews.llvm.org/D84911 ).

* yaml2obj supports emitting multiple abbrev tables.

yaml2obj only supported emitting a single abbrev table and multiple
compilation units had to share the same abbrev table before
https://reviews.llvm.org/D86194 and https://reviews.llvm.org/D83116 .
Now, yaml2obj is able to emit multiple abbrev tables and compilation
units can be linked to any one of them. We add an optional field ID to
abbrev tables and an optional field AbbrevTableID to compilation
units. Compilation units can use AbbrevTableID to link the abbrev
table with the same ID. Besides, the AbbrOffset field of compilation
units which corresponds to the debug_abbrev_offset field doesn't need
to be specified anymore. yaml2obj is able to calculate it for us after
https://reviews.llvm.org/D86614 .

```
debug_abbrev:
  - ID: 0
    Table:
      ...
  - ID: 1
    Table:
      ...
debug_info:
  - ...
    AbbrevTableID: 1 ## Reference the second abbrev table.
    ## AbbrOffset: ... yaml2obj will take care of it.
  - ...
    AbbrevTableID: 0 ## Reference the first abbrev table.
    ## AbbrOffset: ... yaml2obj will take care of it.
```

* The debug_str_offsets, debug_addr, debug_rnglists and debug_loclists
are supported in yaml2obj.

Check out https://reviews.llvm.org/D83624 ,
https://reviews.llvm.org/D84234 , https://reviews.llvm.org/D81541 and
https://reviews.llvm.org/D83853 for more information!

* The DWARF support is added to elf2yaml and improved in macho2yaml.

At first, the output of macho2yaml is noisy. It dumps the DWARF
sections twice, one in the Sections: entry and one in the DWARF:
entry, e.g.,

```
## The content of the debug_str section is dumped twice!
Sections:
  - sectname: __debug_str
    ...
    content: 6D61696E00 ## "main\0"
DWARF:
  debug_str:
    - main
```

After https://reviews.llvm.org/D85506 , if the DWARF parser fails to
parse the DWARF sections into the DWARF: entry, obj2yaml will dump
them as raw content sections, otherwise, they will be presented as
structured DWARF sections in the DWARF: entry. Besides,
https://reviews.llvm.org/D85094 adds DWARF support to elf2yaml.
Although it only supports dumping the debug_aranges section, we can
easily extend it in the future.

==========================================================

Unfinished tasks are listed below. I’m not going to leave the
community and I will improve them in the future.

* Allow users to describe DIEs at a high level.
In my original proposal, we plan to make yaml2obj support describing
DIEs at a high level. However, yaml2obj didn’t support emitting
multiple abbrev tables at that time and I spent some time on enabling
it to emit multiple abbrev tables and link compilation units with
them.

==========================================================

My username on Phabricator is @Higuoxing
(https://reviews.llvm.org/p/Higuoxing/). Please feel free to ping me
if you have trouble in or encountering bugs in crafting DWARF test
cases in YAML. I’m very happy to help.

Last but not least, I would love to express my sincere gratitude to
James Henderson for mentoring me during this project. Besides, I would
like to thank George Rimar, Fangrui Song, Pavel Labath, David Blaikie,
Adrian Prantl and Paul Robinson for reviewing my patches, patiently
answering my questions and leaving comments to my proposal!

==========================================================

My original proposal:
https://docs.google.com/document/d/13wNr4JbXtzaOly-UsFt7vxI3LKXzik_lVU58ICqslWM/edit?usp=sharing


Best Regards,
Xing