[llvm-dev] Adding accelerator tables to existing linked DWARF files

Mon Mar 2 16:42:53 PST 2020

Is there/could you further explain the use-case for adding an index to an
existing binary? Certainly not the worst idea/could come in handy
sometimes, but you mention benchmarking - is the benefit of not
recompiling/relinking that significant to such experiments?

If it's not for use in a common workflow, but only in a compiler/debugger
development workflow, it doesn't seem so important to me.

On Mon, Mar 2, 2020 at 3:50 PM Greg Clayton via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> We could specify ELF support for the --update feature only for now, which
> adds the accelerator tables adding support. For full linking to work, we
> will need some sort of ELF specific of stand alone debug map file that can
> be read for real linking, though that won't be too hard.
>

Not sure I follow here - --update would be given a fully linked binary,
yes? So why would it need a debug map? It'd have the debug info & the
linked executable code available, so you'd be able to see which bits of the
executable code are referred to by which bits of debug info.

>
> On Mar 2, 2020, at 3:44 PM, Eric Christopher <echristo at gmail.com> wrote:
>
> I'd like it... Adrian? Fred?
>
> -eric
>
> On Mon, Mar 2, 2020 at 3:44 PM Greg Clayton <clayborg at gmail.com> wrote:
>
>> Yes. I am fine with adding ELF support to llvm-dsymutil if that is the
>> way people think we should go?
>>
> Feels like a bit of a weird fit to me (equally llvm-objcopy seems like a
weird fit too) - given the specific name & nature of Darwin debug info
distribution being a bit different (reading object files, having input from
the linker, etc) & the specific name being pretty uniquely applied to that
model/output.

(does that way lie moving dwp functionality to llvm-dsymutil too? )

But don't feel super strongly about any of it.

>
>> On Mar 2, 2020, at 3:33 PM, Eric Christopher <echristo at gmail.com> wrote:
>>
>> Which seems like what we'd want dsymutil to do anyhow?
>>
>> -eric
>>
>> On Mon, Mar 2, 2020 at 3:21 PM Greg Clayton via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> On other options would be to make a new "llvm-dwarfld" tool, where most
>>> of the functionality would exist llvm/lib/DwarfLinker and other locations.
>>> The idea would be to do any post processing to DWARF using this tool. For
>>> accelerator tables, it could just create the new sections and then call
>>> "llvm-objcopy" to add them to the binary.
>>>
>>> This tool could eventually be used to optimize DWARF (dead strip code,
>>> remove unused types, unique types with ODR like llvm-dsymutil, etc).
>>>
>>>
>>> On Mar 2, 2020, at 1:48 PM, Greg Clayton <clayborg at gmail.com> wrote:
>>>
>>>
>>>
>>> On Feb 28, 2020, at 11:25 PM, Fangrui Song via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>> On 2020-02-28, Greg Clayton via llvm-dev wrote:
>>>
>>> I am looking to create a tool that can add Apple or DWARF5 accelerator
>>> tables to fully linked executables that contain DWARF. This will help us
>>> benchmark how much accelerator tables can improve the debugging experience
>>> as debuggers don't need to manually index all of the debug info during
>>> debugging.
>>>
>>> Is it for ELF, Mach-O, wasm, COFF, or any of the combinations?
>>>
>>>
>>> Yes, for any object files that LLVM currently supports. But I am looking
>>> to support ELF first as MachO already has these tables available since
>>> dsymutil already creates either Apple or DWARF accelerator tables. COFF and
>>> Wasm can come later.
>>>
>>>
>>> Looking at how accelerator tables are currently emitted, they seem to be
>>> built up as DWARF is being created or linked, and then emitted using a
>>> subclass of DWARFEmitter. The only subclass if this right now that I see is
>>> one in dsymutil which ends up emitting everything using an AsmPrinter by
>>> eventually emitAppleAccelTable(...) from
>>> llvm/include/llvm/CodeGen/AccelTable.h.
>>>
>>> I spoke briefly with Shoaib on this subject and he suggested adding code
>>> to llvm-objcopy. I briefly looked through the code and from what I can
>>> tell, llvm-objcopy doesn't seem to have any DWARF abilities other than
>>> compressing DWARF sections. If we do add functionality to llvm-objcopy, are
>>> we ok pulling in DebugInfoDWARF and the LLVM object model? AFAICT the code
>>> for this tools has its own object file layer which doesn't match the full
>>> layer inside of LLVM (llvm::ObjectFile and DWARFContext). Also, no
>>> AsmPrinter objects are used in this codebase either.
>>>
>>>
>>> llvm-objcopy supports various ad-hoc binary manipulation features where
>>> each feature does a very
>>> simple task. Neither llvm-objcopy nor GNU objcopy knows DWARF.
>>> --strip-debug,
>>> --compress-debug-sections, --add-gnu-debuglink and --only-keep-debug
>>> have "debug" in their names but
>>> these features don't need to parse DWARF. (GNU objcopy has a --debugging
>>> but that only works for
>>> a.out and coff, not elf).
>>>
>>> Do we have a more suitable tool for such debugging functionality?
>>> dsymutil for ELF?
>>>
>>>
>>> dsymutil is such a tool, but it uses the llvm::ObjectFile layer and the
>>> llvm targets, so if you open a file that contains "armv7" architecture and
>>> the ARM target hasn't been built into your bistro, it will fail to open
>>> this binary with an error that says:
>>>
>>> No available targets are compatible with triple "arm-unknown-unknown"
>>>
>>> I ran into this with a recent gsym patch that is trying to fix the
>>> buildbots for testing, but it fails when the ARM targets are not enabled
>>> and I try to load the DWARF from an object file:
>>>
>>>
>>> http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-ubuntu-fast/builds/63473/steps/test-check-all/logs/stdio
>>>
>>> And this is part of the reason for this email. I would love to not
>>> require llvm-gsymutil to require all LLVM targets to be there.
>>> DebugInfoDWARF doesn't need the targets, it just needs to know address byte
>>> size and endianness and it can parse the debug info in the DWARF.
>>>
>>> So my main question stands: do we want all tools that must manipulate
>>> DWARF to require the llvm::ObjectFile layer and all of the targets to be
>>> enabled just so that the object files can be parsed, or do we want to make
>>> lighter layer available, akin to what llvm-objcopy has, so more tools can
>>> take advantage of this lighter weight layer.
>>>
>>>
>>> Looking at lld sources, is seems to use the DebugInfoDWARF library to
>>> some extent already. Not sure if this tool uses the standard LLVM object
>>> model or has all of its own emitters. Does lld use AsmPrinter at all? I
>>> don't see any mention of it in there.
>>>
>>>
>>> --gdb-index and diagnostics (line tables) use DebugInfoDWARF. I have a
>>> plan to implement .debug_names, which is similar to --gdb-index.
>>>
>>>
>>> That is great, and we will share code for this of course between the
>>> tool I write and the modifications to lld. Does lld use the
>>> llvm::ObjectFile layer? or does it have its own lighter weight layer?
>>>
>>>
>>> dsymutil has a --update feature which seems to load all of the DWARF and
>>> pretend to link it all the while generating the new accelerator table data,
>>> but I fear using this would pull it way too much code (AsmPrinter, all
>>> targets required to load all object files types, the standard llvm object
>>> file model (not the lld or llvm-objcopy versions), targets, etc).
>>>
>>> My initial thoughts are:
>>> 1 - load a DWARFContext and iterate through the DWARF and build
>>> accelerator table data
>>> 2 - create the sections for the accelerator tables and either keep in
>>> memory or save to disk
>>> 3 - call functions to add the newly created sections to the binary
>>>
>>> #1 should be easy as long as I can use a DWARFContext from
>>> DebugInfoDWARF.
>>> #2 might need to be re-implemented using something other than an
>>> AsmPrinter?
>>> #3 can use llvm-objcopy code if needed since it can add sections?
>>>
>>> Any advice on how this can or should be implemented would be appreciated
>>> from anyone with experience.
>>>
>>> Greg Clayton
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200302/6d41ef17/attachment.html>