[llvm-dev] Adding accelerator tables to existing linked DWARF files

Mon Mar 2 18:50:56 PST 2020

On Mon, Mar 2, 2020, 4:43 PM David Blaikie <dblaikie at gmail.com> wrote:

> Is there/could you further explain the use-case for adding an index to an
> existing binary? Certainly not the worst idea/could come in handy
> sometimes, but you mention benchmarking - is the benefit of not
> recompiling/relinking that significant to such experiments?
>
> If it's not for use in a common workflow, but only in a compiler/debugger
> development workflow, it doesn't seem so important to me.
>

It could be useful if you want to, say, minimize the maximum RAM usage
during a link and you're okay with the link taking effectively two steps?

> On Mon, Mar 2, 2020 at 3:50 PM Greg Clayton via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> We could specify ELF support for the --update feature only for now, which
>> adds the accelerator tables adding support. For full linking to work, we
>> will need some sort of ELF specific of stand alone debug map file that can
>> be read for real linking, though that won't be too hard.
>>
>
> Not sure I follow here - --update would be given a fully linked binary,
> yes? So why would it need a debug map? It'd have the debug info & the
> linked executable code available, so you'd be able to see which bits of the
> executable code are referred to by which bits of debug info.
>
>
>>
>> On Mar 2, 2020, at 3:44 PM, Eric Christopher <echristo at gmail.com> wrote:
>>
>> I'd like it... Adrian? Fred?
>>
>> -eric
>>
>> On Mon, Mar 2, 2020 at 3:44 PM Greg Clayton <clayborg at gmail.com> wrote:
>>
>>> Yes. I am fine with adding ELF support to llvm-dsymutil if that is the
>>> way people think we should go?
>>>
>> Feels like a bit of a weird fit to me (equally llvm-objcopy seems like a
> weird fit too) - given the specific name & nature of Darwin debug info
> distribution being a bit different (reading object files, having input from
> the linker, etc) & the specific name being pretty uniquely applied to that
> model/output.
>
> (does that way lie moving dwp functionality to llvm-dsymutil too? )
>

Yep. That's the idea.

-eric

> But don't feel super strongly about any of it.
>
>>
>>> On Mar 2, 2020, at 3:33 PM, Eric Christopher <echristo at gmail.com> wrote:
>>>
>>> Which seems like what we'd want dsymutil to do anyhow?
>>>
>>> -eric
>>>
>>> On Mon, Mar 2, 2020 at 3:21 PM Greg Clayton via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> On other options would be to make a new "llvm-dwarfld" tool, where most
>>>> of the functionality would exist llvm/lib/DwarfLinker and other locations.
>>>> The idea would be to do any post processing to DWARF using this tool. For
>>>> accelerator tables, it could just create the new sections and then call
>>>> "llvm-objcopy" to add them to the binary.
>>>>
>>>> This tool could eventually be used to optimize DWARF (dead strip code,
>>>> remove unused types, unique types with ODR like llvm-dsymutil, etc).
>>>>
>>>>
>>>> On Mar 2, 2020, at 1:48 PM, Greg Clayton <clayborg at gmail.com> wrote:
>>>>
>>>>
>>>>
>>>> On Feb 28, 2020, at 11:25 PM, Fangrui Song via llvm-dev <
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>> On 2020-02-28, Greg Clayton via llvm-dev wrote:
>>>>
>>>> I am looking to create a tool that can add Apple or DWARF5 accelerator
>>>> tables to fully linked executables that contain DWARF. This will help us
>>>> benchmark how much accelerator tables can improve the debugging experience
>>>> as debuggers don't need to manually index all of the debug info during
>>>> debugging.
>>>>
>>>> Is it for ELF, Mach-O, wasm, COFF, or any of the combinations?
>>>>
>>>>
>>>> Yes, for any object files that LLVM currently supports. But I am
>>>> looking to support ELF first as MachO already has these tables available
>>>> since dsymutil already creates either Apple or DWARF accelerator tables.
>>>> COFF and Wasm can come later.
>>>>
>>>>
>>>> Looking at how accelerator tables are currently emitted, they seem to
>>>> be built up as DWARF is being created or linked, and then emitted using a
>>>> subclass of DWARFEmitter. The only subclass if this right now that I see is
>>>> one in dsymutil which ends up emitting everything using an AsmPrinter by
>>>> eventually emitAppleAccelTable(...) from
>>>> llvm/include/llvm/CodeGen/AccelTable.h.
>>>>
>>>> I spoke briefly with Shoaib on this subject and he suggested adding
>>>> code to llvm-objcopy. I briefly looked through the code and from what I can
>>>> tell, llvm-objcopy doesn't seem to have any DWARF abilities other than
>>>> compressing DWARF sections. If we do add functionality to llvm-objcopy, are
>>>> we ok pulling in DebugInfoDWARF and the LLVM object model? AFAICT the code
>>>> for this tools has its own object file layer which doesn't match the full
>>>> layer inside of LLVM (llvm::ObjectFile and DWARFContext). Also, no
>>>> AsmPrinter objects are used in this codebase either.
>>>>
>>>>
>>>> llvm-objcopy supports various ad-hoc binary manipulation features where
>>>> each feature does a very
>>>> simple task. Neither llvm-objcopy nor GNU objcopy knows DWARF.
>>>> --strip-debug,
>>>> --compress-debug-sections, --add-gnu-debuglink and --only-keep-debug
>>>> have "debug" in their names but
>>>> these features don't need to parse DWARF. (GNU objcopy has a
>>>> --debugging but that only works for
>>>> a.out and coff, not elf).
>>>>
>>>> Do we have a more suitable tool for such debugging functionality?
>>>> dsymutil for ELF?
>>>>
>>>>
>>>> dsymutil is such a tool, but it uses the llvm::ObjectFile layer and the
>>>> llvm targets, so if you open a file that contains "armv7" architecture and
>>>> the ARM target hasn't been built into your bistro, it will fail to open
>>>> this binary with an error that says:
>>>>
>>>> No available targets are compatible with triple "arm-unknown-unknown"
>>>>
>>>> I ran into this with a recent gsym patch that is trying to fix the
>>>> buildbots for testing, but it fails when the ARM targets are not enabled
>>>> and I try to load the DWARF from an object file:
>>>>
>>>>
>>>> http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-ubuntu-fast/builds/63473/steps/test-check-all/logs/stdio
>>>>
>>>> And this is part of the reason for this email. I would love to not
>>>> require llvm-gsymutil to require all LLVM targets to be there.
>>>> DebugInfoDWARF doesn't need the targets, it just needs to know address byte
>>>> size and endianness and it can parse the debug info in the DWARF.
>>>>
>>>> So my main question stands: do we want all tools that must manipulate
>>>> DWARF to require the llvm::ObjectFile layer and all of the targets to be
>>>> enabled just so that the object files can be parsed, or do we want to make
>>>> lighter layer available, akin to what llvm-objcopy has, so more tools can
>>>> take advantage of this lighter weight layer.
>>>>
>>>>
>>>> Looking at lld sources, is seems to use the DebugInfoDWARF library to
>>>> some extent already. Not sure if this tool uses the standard LLVM object
>>>> model or has all of its own emitters. Does lld use AsmPrinter at all? I
>>>> don't see any mention of it in there.
>>>>
>>>>
>>>> --gdb-index and diagnostics (line tables) use DebugInfoDWARF. I have a
>>>> plan to implement .debug_names, which is similar to --gdb-index.
>>>>
>>>>
>>>> That is great, and we will share code for this of course between the
>>>> tool I write and the modifications to lld. Does lld use the
>>>> llvm::ObjectFile layer? or does it have its own lighter weight layer?
>>>>
>>>>
>>>> dsymutil has a --update feature which seems to load all of the DWARF
>>>> and pretend to link it all the while generating the new accelerator table
>>>> data, but I fear using this would pull it way too much code (AsmPrinter,
>>>> all targets required to load all object files types, the standard llvm
>>>> object file model (not the lld or llvm-objcopy versions), targets, etc).
>>>>
>>>> My initial thoughts are:
>>>> 1 - load a DWARFContext and iterate through the DWARF and build
>>>> accelerator table data
>>>> 2 - create the sections for the accelerator tables and either keep in
>>>> memory or save to disk
>>>> 3 - call functions to add the newly created sections to the binary
>>>>
>>>> #1 should be easy as long as I can use a DWARFContext from
>>>> DebugInfoDWARF.
>>>> #2 might need to be re-implemented using something other than an
>>>> AsmPrinter?
>>>> #3 can use llvm-objcopy code if needed since it can add sections?
>>>>
>>>> Any advice on how this can or should be implemented would be
>>>> appreciated from anyone with experience.
>>>>
>>>> Greg Clayton
>>>>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200302/6dee6c83/attachment-0001.html>