<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 26 Aug 2020 at 15:01, Alexey <<a href="mailto:avl.lapshin@gmail.com">avl.lapshin@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p><br>
</p>
<div>On 26.08.2020 10:58, James Henderson
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>In principle, this sounds reasonable to me. I don't know
enough about dsymutil's interface to know whether it makes
sense to try to make it multi-format compatible or not. If it
doesn't I'm perfectly happy for a new tool to be added using
the DWARFLinker library.</div>
<div><br>
</div>
<div>Some more general thoughts:</div>
<div>1) Assuming the proposal is accepted, this should be
introduced piecemeal into LLVM from the beginning as it is
developed, rather than having a separate step 4 in the
roadmap.</div>
<div>2) The default tombstone values used for dead debug data
should be those produced by LLD, in my opinion. In an ideal
world, we'd factor them into some shared constant. Note that
at the time of writing, I believe LLD is currently using
BFD-style tombstones, not the new -1/-2.</div>
</div>
</blockquote>
<p>agreed.</p>
<blockquote type="cite">
<div dir="ltr">
<div>3) Does the DWARFLinker library already support
multi-threading? If not, it might be a lot of work making
things thread-safe.</div>
</div>
</blockquote>
<p>It does, but in a limited way. It can parallelize analyzing and
cloning stages. i.e. the maximal speedup is two times.</p>
<p>To have a greater performance impact it could probably be
parallelized per compilation unit basis.</p>
<p>Another thing is that dsymutil currently loads all DIEs from
source object file into the memory. And releases them after object
file is processed. For non-linked binary this works OK(big
binaries usually compiled from several object files). For linked
binary that means all DIEs are loaded into the memory. In the
result it requires a lot of memory resources. The solution for
this problem could be changing splitting of source data from the
file to the compilation unit basis.</p>
<p>yes, making dsymutil/dwarfutil to work on compilation unit basis
supporting multi-threading is a quite a big piece of work. It
looks like it would be good for both dsymutil and dwarfutil.<br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div>4) Given that DWARF v6 doesn't exist yet, I wouldn't
include that as an option name just yet...!</div>
</div>
</blockquote>
<p>Would "maxpc" be OK? --tombstone=maxpc ?<br></p></div></blockquote><div>"maxpc" sounds reasonable for an initial stab at a name. I'm sure there's something better out there, but I can't think of it, so no need to worry, if you don't come up with anything better!<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><p>
</p>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Thanks for looking at this! Please keep me involved in any
related reviews etc.</div>
</div>
</blockquote>
<p>sure. Thank you for the comments.</p>
<p>Alexey.<br>
</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>James<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, 25 Aug 2020 at 15:29,
Alexey via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
We propose llvm-dwarfutil - a dsymutil-like tool for ELF.<br>
Any thoughts on this?<br>
Thanks in advance, Alexey.<br>
<br>
======================================================================<br>
<br>
llvm-dwarfutil(Apndx A) - is a tool that is used for
processing debug <br>
info(DWARF)<br>
located in built binary files to improve debug info quality,<br>
reduce debug info size and accelerate debug info processing.<br>
Supported object files formats: ELF, MachO(Apndx B),
COFF(Apndx C), <br>
WASM(Apndx C).<br>
<br>
======================================================================<br>
<br>
Specifically, the tool would do:<br>
<br>
- Remove obsolete debug info which refers to code deleted
by the linker<br>
doing the garbage collection (gc-sections).<br>
<br>
- Deduplicate debug type definitions for reducing resulting
size of <br>
binary.<br>
<br>
- Build accelerator/index tables.<br>
= .debug_aranges, .debug_names, .gdb_index,
.debug_pubnames, <br>
.debug_pubtypes.<br>
<br>
- Strip unneeded tables.<br>
= .debug_aranges, .debug_names, .gdb_index,
.debug_pubnames, <br>
.debug_pubtypes.<br>
<br>
- Compress or decompress debug info as requested.<br>
<br>
Possible feature:<br>
<br>
- Join split dwarf .dwo files in a single file containing
all debug info<br>
(convert split DWARF into monolithic DWARF).<br>
<br>
======================================================================<br>
<br>
User interface:<br>
<br>
OVERVIEW: A tool for optimizing debug info located in the
built binary.<br>
<br>
USAGE: llvm-dwarfutil [options] input output<br>
<br>
OPTIONS: (Apndx E)<br>
<br>
======================================================================<br>
<br>
Implementation notes:<br>
<br>
1. Removing obsolete debug info would be done using
DWARFLinker llvm <br>
library.<br>
<br>
2. Data types deduplication would be done using DWARFLinker
llvm library.<br>
<br>
3. Accelerator/index tables would be generated using
DWARFLinker llvm <br>
library.<br>
<br>
4. Interface of DWARFLinker library would be changed in such
way that it<br>
would be possible to switch on/off various stages:<br>
<br>
class DWARFLinker {<br>
setDoRemoveObsoleteInfo ( bool DoRemoveObsoleteInfo =
false);<br>
<br>
setDoAppleNames ( bool DoAppleNames = false );<br>
setDoAppleNamespaces ( bool DoAppleNamespaces = false );<br>
setDoAppleTypes ( bool DoAppleTypes = false );<br>
setDoObjC ( bool DoObjC = false );<br>
setDoDebugPubNames ( bool DoDebugPubNames = false );<br>
setDoDebugPubTypes ( bool DoDebugPubTypes = false );<br>
<br>
setDoDebugNames (bool DoDebugNames = false);<br>
setDoGDBIndex (bool DoGDBIndex = false);<br>
}<br>
<br>
5. Copying source file contents, stripping tables, <br>
compressing/decompressing tables<br>
would be done by ObjCopy llvm library(extracted from
llvm-objcopy):<br>
<br>
Error executeObjcopyOnBinary(const CopyConfig &Config,<br>
object::COFFObjectFile &In,
Buffer &Out);<br>
Error executeObjcopyOnBinary(const CopyConfig &Config,<br>
object::ELFObjectFileBase
&In, Buffer &Out);<br>
Error executeObjcopyOnBinary(const CopyConfig &Config,<br>
object::MachOObjectFile &In,
Buffer &Out);<br>
Error executeObjcopyOnBinary(const CopyConfig &Config,<br>
object::WasmObjectFile &In,
Buffer &Out);<br>
<br>
6. Address ranges and single addresses pointing to removed
code should <br>
be marked<br>
with tombstone value in the input file:<br>
<br>
-2 for .debug_ranges and .debug_loc.<br>
-1 for other .debug* tables.<br>
<br>
7. Prototype implementation - <a href="https://reviews.llvm.org/D86539" rel="noreferrer" target="_blank">https://reviews.llvm.org/D86539</a>.<br>
<br>
======================================================================<br>
<br>
Roadmap:<br>
<br>
1. Refactor llvm-objcopy to extract it`s implementation into
separate <br>
library<br>
ObjCopy(in LLVM tree).<br>
<br>
2. Create a command line utility using existed DWARFLinker and
ObjCopy<br>
implementation. First version is supposed to work with
only ELF <br>
input object files.<br>
It would take input ELF file with unoptimized debug info
and create <br>
output<br>
ELF file with optimized debug info. That version would be
done out <br>
of the llvm tree.<br>
<br>
3. Make a tool to be able to work in multi-thread mode.<br>
<br>
4. Consider it to be included into LLVM tree.<br>
<br>
5. Support DWARF5 tables.<br>
<br>
======================================================================<br>
<br>
Appendix A. Should this tool be implemented as a new tool or
as an extension<br>
to dsymutil/llvm-objcopy?<br>
<br>
There already exists a tool which removes obsolete debug
info on <br>
darwin - dsymutil.<br>
Why create another tool instead of extending the already
existed <br>
dsymutil/llvm-objcopy?<br>
<br>
The main functionality of dsymutil is located in a
separate library <br>
- DWARFLinker.<br>
Thus, dsymutil utility is a command-line interface for
DWARFLinker. <br>
dsymutil has<br>
another type of input/output data: it takes several object
files and <br>
address map<br>
as input and creates a .dSYM bundle with linked debug info
as <br>
output. llvm-dwarfutil<br>
would take a built executable as input and create an
optimized <br>
executable as output.<br>
Additionally, there would be many command-line options
specific for <br>
only one utility.<br>
This means that these utilities(implementing command line
interface) <br>
would significantly<br>
differ. It makes sense not to put another command-line
utility <br>
inside existing dsymutil,<br>
but make it as a separate utility. That is the reason why
<br>
llvm-dwarfutil suggested to be<br>
implemented not as sub-part of dsymutil but as a separate
tool.<br>
<br>
Please share your preference: whether llvm-dwarfutil
should be<br>
separate utility, or a variant of dsymutil compiled for
ELF?<br>
<br>
======================================================================<br>
<br>
Appendix B. The machO object file format is already supported
by dsymutil.<br>
Depending on the decision whether llvm-dwarfutil would be
done as a <br>
subproject<br>
of dsymutil or as a separate utility - machO would be
supported or not.<br>
<br>
======================================================================<br>
<br>
Appendix C. Support for the COFF and WASM object file formats
presented as<br>
possible future improvement. It would be quite easy to
add them <br>
assuming<br>
that llvm-objcopy already supports these formats. It also
would require<br>
supporting DWARF6-suggested tombstone values(-1/-2).<br>
<br>
======================================================================<br>
<br>
Appendix D. Documentation.<br>
<br>
- proposal for DWARF6 which suggested -1/-2 values for
marking bad <br>
addresses<br>
<a href="http://www.dwarfstd.org/ShowIssue.php?issue=200609.1" rel="noreferrer" target="_blank">http://www.dwarfstd.org/ShowIssue.php?issue=200609.1</a><br>
- dsymutil tool <a href="https://llvm.org/docs/CommandGuide/dsymutil.html" rel="noreferrer" target="_blank">https://llvm.org/docs/CommandGuide/dsymutil.html</a>.<br>
- proposal "Remove obsolete debug info in lld."<br>
<a href="http://lists.llvm.org/pipermail/llvm-dev/2020-May/141468.html" rel="noreferrer" target="_blank">http://lists.llvm.org/pipermail/llvm-dev/2020-May/141468.html</a><br>
<br>
======================================================================<br>
<br>
Appendix E. Possible command line options:<br>
<br>
DwarfUtil Options:<br>
<br>
--build-aranges - generate .debug_aranges table.<br>
--build-debug-names - generate .debug_names table.<br>
--build-debug-pubnames - generate .debug_pubnames table.<br>
--build-debug-pubtypes - generate .debug_pubtypes table.<br>
--build-gdb-index - generate .gdb_index table.<br>
--compress - Compress debug tables.<br>
--decompress - Decompress debug tables.<br>
--deduplicate-types - Do ODR deduplication for debug
types.<br>
--garbage-collect - Do garbage collecting for debug
info.<br>
--num-threads=<n> - Specify the maximum
number (n) of <br>
simultaneous threads<br>
to use when optimizing input
file.<br>
Defaults to the number of cores
on the <br>
current machine.<br>
--strip-all - Strip all debug tables.<br>
--strip=<name1,name2> - Strip specified debug
info tables.<br>
--strip-unoptimized-debug - Strip all unoptimized debug
tables.<br>
--tombstone=<value> - Tombstone value used as a
marker of <br>
invalid address.<br>
=bfd - BFD default value<br>
=dwarf6 - Dwarf v6.<br>
--verbose - Enable verbose logging and
encoding details.<br>
<br>
Generic Options:<br>
<br>
--help - Display available options
(--help-hidden <br>
for more)<br>
--version - Display the version of this
program<br>
<br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote>
</div>
</blockquote>
</div>
</blockquote></div></div>