<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On May 1, 2015, at 5:25 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com" class="">dblaikie@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><br class="Apple-interchange-newline"><br style="font-family: LucidaGrande; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote" style="font-family: LucidaGrande; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">On Fri, May 1, 2015 at 5:19 PM, Adrian Prantl<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:aprantl@apple.com" target="_blank" class="">aprantl@apple.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><br class=""><div class=""><div class=""><div class="h5"><blockquote type="cite" class=""><div class="">On May 1, 2015, at 4:55 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank" class="">dblaikie@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Fri, May 1, 2015 at 4:39 PM, Adrian Prantl<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:aprantl@apple.com" target="_blank" class="">aprantl@apple.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div class=""><div class=""><br class="">> On May 1, 2015, at 10:01 AM, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank" class="">dblaikie@gmail.com</a>> wrote:<br class="">><br class="">><br class="">><br class="">> On Fri, May 1, 2015 at 9:52 AM, Adrian Prantl <<a href="mailto:aprantl@apple.com" target="_blank" class="">aprantl@apple.com</a>> wrote:<br class="">>><br class="">>>> On May 1, 2015, at 9:23 AM, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank" class="">dblaikie@gmail.com</a>> wrote:<br class="">>>><br class="">>>><br class="">>>><br class="">>>> On Thu, Apr 30, 2015 at 5:21 PM, Adrian Prantl <<a href="mailto:aprantl@apple.com" target="_blank" class="">aprantl@apple.com</a>> wrote:<br class="">>>><br class="">>>> > On Apr 30, 2015, at 4:55 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank" class="">dblaikie@gmail.com</a>> wrote:<br class="">>>> ><br class="">>>> ><br class="">>>> ><br class="">>>> > On Thu, Apr 30, 2015 at 4:31 PM, Adrian Prantl <<a href="mailto:aprantl@apple.com" target="_blank" class="">aprantl@apple.com</a>> wrote:<br class="">>>> >><br class="">>>> >> > On Mar 19, 2015, at 5:37 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank" class="">dblaikie@gmail.com</a>> wrote:<br class="">>>> >> ><br class="">>>> >> ><br class="">>>> >> ><br class="">>>> >> > On Thu, Mar 19, 2015 at 5:24 PM, Adrian Prantl <<a href="mailto:aprantl@apple.com" target="_blank" class="">aprantl@apple.com</a>> wrote:<br class="">>>> >> >><br class="">>>> >> >> > On Mar 16, 2015, at 2:55 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank" class="">dblaikie@gmail.com</a>> wrote:<br class="">>>> >> >> ><br class="">>>> >> >> ><br class="">>>> >> >> ><br class="">>>> >> >> >> On Mon, Mar 16, 2015 at 2:45 PM, Robinson, Paul <<a href="mailto:Paul_Robinson@playstation.sony.com" target="_blank" class="">Paul_Robinson@playstation.sony.com</a>> wrote:<br class="">>>> >> >> > Beyond the above (that using a new tag would mean this would go from 'free' to 'not free' for GDB) having a new top level tag is pretty substantial (we only have two at the moment, and with our talk of modules being a "bag of dwarf" might go back to having one top level tag? (it's not clear to me from DWARF4 whether DW_TAG_module is currently a top-level tag, I don't think it is?)<br class="">>>> >> >> ><br class="">>>> >> >> >> The .debug_info section contains one or more compilation units, partial units, or in DWARF 5, type units. DW_TAG_module isn't a unit, if you want it to be handled independently then it would need to be wrapped in a DW_TAG_partial_unit. You would probably then use DW_TAG_imported_unit to refer to it, rather than DW_TAG_imported_module.<br class="">>>> >> >> >><br class="">>>> >> >> ><br class="">>>> >> >> > This makes a fair bit of sense - though the terminology's never going to quite line up with modules, I suspect, and this would still require modifying existing consumers (well, GDB) that can handle split-dwarf today, I suspect (not sure how it'd handle partial_unit - maybe that does work? - and still don't know how existing consumers would handle imported_unit either - could be worth some testing, as it sounds sort of right out of several less right options).<br class="">>>> >> >><br class="">>>> >> >> Thanks for all the input so far!<br class="">>>> >> >> To concretize this end of the discussion up let’s sketch some dwarf of how this could look like in practice.<br class="">>>> >> >><br class="">>>> >> >> ELF (no imports)<br class="">>>> >> >> ----------------<br class="">>>> >> >><br class="">>>> >> >> On ELF or COFF a foo.c referencing types from the module Foundation looks like this:<br class="">>>> >> >><br class="">>>> >> >> .debug_info:<br class="">>>> >> >> DW_TAG_compile_unit<br class="">>>> >> >> DW_AT_name(“foo.c”)<br class="">>>> >> >><br class="">>>> >> >> .debug_info.dwo (on ELF: group 0x1234ABCDE, comdat)<br class="">>>> >> >> DW_TAG_partial_unit<br class="">>>> >> ><br class="">>>> >> > For now I'd suggest we use compile_unit - that way it'll just work with existing split-dwarf consumers. We can see about standardizing a top-level DW_TAG_module or using DW_TAG_partial_unit here later, perhaps? I'm not sure.<br class="">>>> >> ><br class="">>>> >> >> DW_AT_dwo_name(“/tmp/org.llvm.clang/ModuleCache/1234ABCDE/Foundation.pcm”)<br class="">>>> >> >> DW_AT_dwo_id(“0x1234ABCDE”)<br class="">>>> >> >><br class="">>>> >> >><br class="">>>> >> >> Side question: Is .debug_info.dwo the right section to put the module skeleton in, or should it be a .debug_info section like normal fission skeletons?<br class="">>>> >> ><br class="">>>> >> > Skeletons go in .debug_info, the dwo sections are just for the .dwo file (or the module file, in our new case - the extension isn't actually important).<br class="">>>> >> ><br class="">>>> >> > It might be worth you compiling an example or two of split-dwarf to see how this all works hands-on.<br class="">>>> >> ><br class="">>>> >> >> Mach-O (no comdat, no imports)<br class="">>>> >> >> ------------------------------<br class="">>>> >> >><br class="">>>> >> >> Mach-O doesn’t do comdat, so with -split-dwarf=Disable (not sure if that option is the best discriminator) this could look like:<br class="">>>> >> >><br class="">>>> >> >> .debug_info:<br class="">>>> >> >> DW_TAG_compile_unit<br class="">>>> >> >> DW_AT_name(“foo.c”)<br class="">>>> >> >> DW_TAG_partial_unit<br class="">>>> >> >> DW_AT_dwo_name(“/tmp/org.llvm.clang/ModuleCache/1234ABCDE/Foundation.pcm”)<br class="">>>> >> >> DW_AT_dwo_id(“0x1234ABCDE”)<br class="">>>> >> >><br class="">>>> >> >><br class="">>>> >> >> Mach-O (no comdat, with imports)<br class="">>>> >> >> ------------------------------<br class="">>>> >> >><br class="">>>> >> >> If we add the module import information to this, we get:<br class="">>>> >> >><br class="">>>> >> >> .debug_info:<br class="">>>> >> >> DW_TAG_compile_unit<br class="">>>> >> >> DW_AT_name(“foo.c”)<br class="">>>> >> >> DW_TAG_imported_module<br class="">>>> >> >> DW_AT_import(DW_FORM_ref_addr 0x10)<br class="">>>> >> ><br class="">>>> >> > Since we got went down the tangent of explaining split-dwarf many emails ago, I've forgotten (& can't readily find) what we were discussing about what ways the imported_module could work.<br class="">>>> >> ><br class="">>>> >> > The simplest representation I can think of would be to have it reference, by signature, the module unit (whatever tag it uses) - DW_FORM_ref_sig8, seems the simplest thing to do.<br class="">>>> >> ><br class="">>>> >> >><br class="">>>> >> >> DW_TAG_partial_unit<br class="">>>> >> >> DW_AT_dwo_name(“/tmp/org.llvm.clang/ModuleCache/1234ABCDE/Foundation.pcm”)<br class="">>>> >> >> DW_AT_dwo_id(“0x1234ABCDE”)<br class="">>>> >> >><br class="">>>> >> >> 0x10:<br class="">>>> >> ><br class="">>>> >> > This is inside the partial unit? I figured we'd just put these attributes on the top level (compile_unit, or whatever it might be later) - potentially conditionalized on platform, sure.<br class="">>>> >> ><br class="">>>> >> >> DW_TAG_module<br class="">>>> >> >> DW_AT_name(“Foundation”)<br class="">>>> >> >> DW_AT_LLVM_sysroot(“/“)<br class="">>>> >> >> DW_AT_LLVM_include_dir(“”)<br class="">>>> >> >> DW_AT_LLVM_macros(“-DNDEBUG”)<br class="">>>> >> >> ...<br class="">>>> >> >><br class="">>>> >> >><br class="">>>> >> >> ELF (comdat, with imports)<br class="">>>> >> >> --------------------------<br class="">>>> >> >><br class="">>>> >> >> But now let’s go back to ELF. Since the skeleton with the partial unit is comdat'd, I assume that this breaks the FORM_ref_addr used in the DW_AT_import. We could reuse the module hash as a signature for the module:<br class="">>>> >> >><br class="">>>> >> >> .debug_info:<br class="">>>> >> >> DW_TAG_compile_unit<br class="">>>> >> >> DW_AT_name(“foo.c”)<br class="">>>> >> >> DW_TAG_imported_module<br class="">>>> >> >> DW_AT_import(DW_FORM_ref_addr 0x1234ABCDE)<br class="">>>> >> ><br class="">>>> >> > Still only really need these imported_modules for lldb, right? I'd consider having them off-by-default for non-darwin, but I'm not strictly wedded to that notion. Wouldn't mind seeing size impact numbers of some kind - if it's really fractional % increase & GDB doesn't fall over when it sees them (in whatever FORM/tag/etc we decide on) then that's not the end of the world.<br class="">>>> >> ><br class="">>>> >> > Just seems nice if the default mode is the nice, standard, split-dwarf output. Doesn't need anything fancy.<br class="">>>> >> ><br class="">>>> >> ><br class="">>>> >> >> .debug_info.dwo (group 0x1234ABCDE, comdat)<br class="">>>> >> >> DW_TAG_partial_unit<br class="">>>> >> >> DW_AT_dwo_name(“/tmp/org.llvm.clang/ModuleCache/1234ABCDE/Foundation.pcm”)<br class="">>>> >> >> DW_AT_dwo_id(“0x1234ABCDE”)<br class="">>>> >> >><br class="">>>> >> >> DW_TAG_module<br class="">>>> >> >> DW_AT_signature(“0x1234ABCDE”)<br class="">>>> >> >> DW_AT_name(“Foundation”)<br class="">>>> >> ><br class="">>>> >> ><br class="">>>> >> > The thing you haven't covered is the actual .dwo sections (.debug_info.dwo (we'll probably need a simple stub compile_unit to make this correct split-dwarf) and .debug_types.dwo being important - but all the supporting .dwo sections will be necessary) that go in the module file.<br class="">>>> >> ><br class="">>>> >> >> This is bending the definition of DW_AT_signature, but I guess it could be made to work. Or we could say that for now, users have to choose between the comdat optimization and having the module imports recorded in Dwarf, since GDB wouldn’t know what to do with that information anyway.<br class="">>>> >><br class="">>>> >> Sorry for the long delay. Here’s a more complete example that should include all the suggestions made so far. For context I also included external type references in the example although admittedly this is a bit out of scope for this thread:<br class="">>>> >><br class="">>>> >> ELF (typeunits, comdats, with imports)<br class="">>>> >> --------------------------------------<br class="">>>> >><br class="">>>> >> On ELF or COFF a bar.c referencing type Foo from the module FooLib looks like this:<br class="">>>> >><br class="">>>> >> bar.o<br class="">>>> >> ~~~~~<br class="">>>> >><br class="">>>> >> // To keep this example focussed/readable, I'm assuming that bar.o itself was not compiled with fission.<br class="">>>> >> .debug_info:<br class="">>>> >> DW_TAG_compile_unit<br class="">>>> >> DW_AT_name(“bar.c”)<br class="">>>> >> ...<br class="">>>> >><br class="">>>> >> DW_TAG_imported_module // <- This could be optional on ELF.<br class="">>>> >> DW_AT_import [DW_FORM_ref_sig8] (0xABCD1234)<br class="">>>> >><br class="">>>> >> DW_TAG_variable<br class="">>>> >> DW_AT_name(“MyFoo”)<br class="">>>> >> DW_AT_type [DW_FORM_ref4] 0x20<br class="">>>> >> 0x20:<br class="">>>> >> DW_TAG_structure_type<br class="">>>> >> DW_AT_declaration (true)<br class="">>>> >> DW_AT_signature [DW_FORM_ref_sig8] (0xF00)<br class="">>>> >><br class="">>>> >><br class="">>>> >> // Split DWARF skeleton CU for the module Foo.<br class="">>>> >> DW_TAG_compile_unit<br class="">>>> >> DW_AT_dwo_name(“/tmp/org.llvm.clang/ModuleCache/1234ABCDE/FooLib-XYZ.pcm”)<br class="">>>> >> DW_AT_dwo_id(“0xFEDB9876”)<br class="">>>> >> ...<br class="">>>> >><br class="">>>> >> // Comdat’d partial unit containing the optional module descriptor.<br class="">>>> >> .debug_info, group 0xABCD1234, comdat<br class="">>>> >> DW_TAG_partial_unit<br class="">>>> >> DW_TAG_module<br class="">>>> >> DW_AT_name(“FooLib”)<br class="">>>> >> DW_AT_LLVM_sysroot(“/“)<br class="">>>> >> DW_AT_LLVM_include_dirs(“-I/path”)<br class="">>>> >> DW_AT_LLVM_macros(“-DNDEBUG”)<br class="">>>> >> ...<br class="">>>> >><br class="">>>> >> FooLib-XYZ.pcm<br class="">>>> >> ~~~~~~~~~~~~~~<br class="">>>> >><br class="">>>> >> .debug_info.dwo<br class="">>>> >> DW_TAG_compile_unit<br class="">>>> >> DW_AT_dwo_id(“0xFEDB9876”)<br class="">>>> >> ...<br class="">>>> >><br class="">>>> >> // Type unit for the type Foo.<br class="">>>> >> .debug_types.dwo, group 0xF00, comdat<br class="">>>> >> DW_TAG_type_unit<br class="">>>> >> DW_TAG_structure_type<br class="">>>> >> DW_AT_name (“Foo”)<br class="">>>> >> ...<br class="">>>> >><br class="">>>> >><br class="">>>> >> I think it awkward to have both the skeleton compile_unit in .debug_info and the partial_unit containing the TAG_module. Personally I’d prefer putting the TAG_module into the skeleton CU and then just refer to it via a FORM_ref_addr; but if we want to put the TAG_module into a comdat section, it looks like that’s what’s necessary.<br class="">>>> ><br class="">>>> > It's been a while & I've probably lost all the context, but I think my original theory was to have the skeleton compile_unit be comdat'd so they'd deduplicate on linking (so we'd only have one reference to the module.dwo in the linked binary). I don't recall there being a need for a separate partial_unit - I imagine we'd just put the LLDB/LLVM extension attributes on the skeleton compile_unit and expect debuggers that didn't understand them, to ignore them.<br class="">>>> ><br class="">>>> > Was there some reason this didn't work/make sense? Because you need a DW_TAG_module to import with DW_TAG_imported_module?<br class="">>>> Using DW_TAG_module was the best practice that was recommended on dwarf-discuss.<br class="">>>><br class="">>>> Did they have any ideas on how to reference it without duplicating it in every CU?<br class="">>><br class="">>> We didn’t touch the deduplication issue.<br class="">>><br class="">>>> Once we've got the "Bag O Dwarf" stuff (rather than the narrower type units) this would be easier - (I suppose we could do a partial solution/abuse of type units - use a type unit header (perhaps with Eric's merged type/compile unit work) and a DW_FORM_ref_sig8 value for the DW_AT_module in the DW_TAG_imported_module.<br class="">>>><br class="">>>> Though I suppose if we're going to have DW_TAG_imported_module in every CU that references a module, it might not be that big of a deal to include the DW_TAG_module itself there too... while I don't care about this scheme immediately, Google's growing LLDB investment in various platforms, so I am vaguely concerned about getting this right & it's not immediately obvious to me what that right answer is.<br class="">>><br class="">>> Maybe the best path forward is to stage this by initially putting the DW_TAG_module into the main CU and leave the deduplication as an optimization to be implemented once the bag’o dwarf is more fleshed out. This way we won’t do anything that would confuse consumers (assuming they ignore unknown tags) and the extra overhead is likely not even going to be noticeable, since all the string attributes inside the TAG_module can already be deduplicated by traditional means.<br class="">><br class="">> Perhaps. I'd still like to think through/document what this looks like a bit more. Where the data ends up, what it's used for, etc. Sorry to draw this out.<br class="">><br class="">> :/ *ponders*<br class=""><br class=""><br class=""></div></div>Let’s construct this:<br class=""><br class="">The most straightforward representation is to not unique the TAG_module and place it into the main CU.<br class=""><br class="">bar.o<br class="">~~~~~<br class=""><br class="">.debug_info:<br class=""> <span class="Apple-converted-space"> </span>DW_TAG_compile_unit<br class=""> <span class="Apple-converted-space"> </span>...<br class=""> <span class="Apple-converted-space"> </span>DW_TAG_imported_module<br class=""> <span class="Apple-converted-space"> </span>DW_AT_import [DW_FORM_ref4] (0x20)<br class="">0x20:<br class=""><span class=""> <span class="Apple-converted-space"> </span>DW_TAG_module<br class=""> <span class="Apple-converted-space"> </span>DW_AT_name(“FooLib”)<br class=""> <span class="Apple-converted-space"> </span>DW_AT_LLVM_sysroot(“/“)<br class=""> <span class="Apple-converted-space"> </span>DW_AT_LLVM_include_dirs(“-I/path”)<br class=""> <span class="Apple-converted-space"> </span>DW_AT_LLVM_macros(“-DNDEBUG”)<br class=""></span></blockquote><div class=""><br class="">Might as well put all these LLVM attributes on the skeleton CU, though - so they can be deduplicated (& just put the dwo_id in this module somewhere, perhaps just using the DW_AT_dwo_id attribute - possibly that's the only attribute the DW_TAG_module would need, ideally). Unless we need to consider the submodule issue (in which case the skeleton unit would reference the whole module but the submodules would reference/describe the respective submodules?)?</div></div></div></div></div></blockquote><div class=""><br class=""></div></div></div>We cannot put them into the skeleton CU if the skeleton CU is going to be comdat’d, because we’d then have to refer to it via a signature and that leads us directly to the can of worms discussed in the next paragraph :-)<span class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><span class=""> <span class="Apple-converted-space"> </span>...<br class=""><br class=""></span>// Split DWARF skeleton, comdat'd.<br class=""><span class="">.debug_info, group 0xFEDB9876, comdat<br class=""> <span class="Apple-converted-space"> </span>DW_TAG_compile_unit<br class=""> <span class="Apple-converted-space"> </span>DW_AT_dwo_name(“/tmp/org.llvm.clang/ModuleCache/1234ABCDE/FooLib-XYZ.pcm”)<br class=""> <span class="Apple-converted-space"> </span>DW_AT_dwo_id(“0xFEDB9876”)<br class=""> <span class="Apple-converted-space"> </span>...<br class=""><br class=""></span>On Mach-O the split DWARF skeleton would not be a comdat’d, but llvm-dsymutil can just ignore it.<br class=""><br class=""><br class="">If we want to dedup the TAG_module we need to refer to it via signature. This means we need to wrap it in a type_unit or a DWARF5 TAG_type_unit. We might as well throw it in with the skeleton CU.<br class=""><br class="">.debug_info:<br class=""> <span class="Apple-converted-space"> </span>DW_TAG_compile_unit<br class=""> <span class="Apple-converted-space"> </span>...<br class=""> <span class="Apple-converted-space"> </span>DW_TAG_imported_module<br class=""> <span class="Apple-converted-space"> </span>DW_AT_import [DW_FORM_ref_sig8] (0xABCD1234)<br class=""><br class="">// Split DWARF skeleton, comdat'd.<br class=""><span class="">.debug_info, group 0xFEDB9876, comdat<br class=""> <span class="Apple-converted-space"> </span>DW_TAG_compile_unit<br class=""> <span class="Apple-converted-space"> </span>DW_AT_dwo_name(“/tmp/org.llvm.clang/ModuleCache/1234ABCDE/FooLib-XYZ.pcm”)<br class=""> <span class="Apple-converted-space"> </span>DW_AT_dwo_id(“0xFEDB9876”)<br class=""> <span class="Apple-converted-space"> </span>...<br class=""></span> <span class="Apple-converted-space"> </span>DW_TAG_type_unit (signature: 0xABCD1234)<br class=""></blockquote><div class=""><br class="">Can't really put a type_unit inside a compile_unit - it'd need to be top-level with an appropriate type unit header, etc. & then we'd need two different units/headers, could still comdat them, but it's a weird abuse of type units & would probably confuse consumers. I don't know whether that's worth the effort.<br class=""></div></div></div></div></div></blockquote></span><div class="">Oh right.</div><span class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><span class=""> <span class="Apple-converted-space"> </span>DW_TAG_module<br class=""> <span class="Apple-converted-space"> </span>DW_AT_name(“FooLib”)<br class=""> <span class="Apple-converted-space"> </span>DW_AT_LLVM_sysroot(“/“)<br class=""> <span class="Apple-converted-space"> </span>DW_AT_LLVM_include_dirs(“-I/path”)<br class=""> <span class="Apple-converted-space"> </span>DW_AT_LLVM_macros(“-DNDEBUG”)<br class=""> <span class="Apple-converted-space"> </span>...<br class=""><br class=""></span>Now that raises the question about what happens with multiple modules within one PCM.<span class="Apple-converted-space"> </span></blockquote><div class=""><br class="">Is the right term "submodule"? it's sort of confusing to talk about multiple modules within a pcm.<br class=""></div></div></div></div></div></blockquote><div class=""><br class=""></div></span><div class="">Yes, a module with nested submodules.</div><div class=""><a href="http://clang.llvm.org/docs/Modules.html#submodule-declaration" target="_blank" class="">http://clang.llvm.org/docs/Modules.html#submodule-declaration</a></div><span class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">Assuming that the ELF linker is linking and deduping all the non-.dwo sections, we may loose some of the TAG_modules (if not every CU imports all submodules) in the binary, but that wouldn’t matter because the consumer would find all TAG_modules by signature in the .pcm</blockquote><div class=""><br class="">Is there any reason we need to reference the submodules individually, rather than just reference the whole module<br class=""></div></div></div></div></div></blockquote><div class=""><br class=""></div></span><div class="">My assumption is that an AST-aware debugger will want to import the exact submodules that were imported by the CU before dropping into the expression evaluator to replicate the environment of the CU as much as possible.</div></div></div></blockquote><div class=""><br class="">I'm just not picturing that. It seems pretty likely that a debugger user is more likely to treat the whole set of names in the program, not just those syntactically valid at that point in the source file.<br class=""></div></div></div></blockquote><div><br class=""></div><div>Module imports only work if the debugger has the precise list of models imported by the current CU. Clang modules are not namespaces, and any two modules may conflict.</div><div>The cool thing is that with the imported modules the debugger effectively becomes clang and have the entire world visible to the current CU available, including any types and functions that never made it into the debug info because they were optimized out, or because there were uninstantiated templates that cannot be represented by DWARF.</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div class="gmail_quote" style="font-family: LucidaGrande; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class="">A simple example would be if I'm debugging LLVM and I'm in some generic optimization pass, but I want to cast my Instruction pointer to some specific instruction type to examine it in more detail - even though this pass doesn't care about that specific Instruction type nor include the header in which it's declared.<br class=""></div></div></div></blockquote><div><br class=""></div><div>If, however, the type lookup fails, the debugger can still fall back to the traditional behavior, find the type in the accelerator tables and reconstruct it from DWARF (if it is there).</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div class="gmail_quote" style="font-family: LucidaGrande; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><span class=""><div class=""><br class=""></div><div class=""></div><blockquote type="cite" class=""><div class=""> (& have just a single, whole module in the pcm)?</div></blockquote><div class=""><br class=""></div></span><div class="">That’s probably not what you meant, but just to be sure: The pcm will always have the entire module with all submodules in it. But the debugger may choose to import only a subset of those.</div><span class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">file referred to by whichever skeleton CU makes it into the binary:<br class=""><span class=""><br class="">FooLib-XYZ.pcm<br class="">~~~~~~~~~~~~~~<br class=""><br class="">.debug_info.dwo<br class=""> DW_TAG_compile_unit<br class=""> DW_AT_dwo_id(“0xFEDB9876”)<br class=""> ...<br class=""><br class=""></span> DW_TAG_type_unit (signature: 0xABCD1234)<br class=""> DW_TAG_module<br class=""> DW_AT_name(“FooLib”)<br class=""> ...<br class=""> DW_TAG_type_unit (signature: 0xCDEF3456)<br class=""> DW_TAG_module<br class=""> DW_AT_name(“FooLib”)<br class=""> DW_TAG_module<br class=""> DW_AT_name(“SubFoo”)<br class=""> ...<br class=""><br class="">So.. this should work as long as nobody points out that a module isn’t really a type.<br class=""></blockquote><div class=""><br class="">Yeah, probably worth waiting for "Bag O DWARF".<br class=""><br class="">For now, as you mentioned earlier, maybe just putting the imported_module and the module into the compile_unit when tuning for LLDB (so Darwin by default, and anywhere else where someone tunes for LLDB in the future) & leave them out otherwise.<br class=""></div></div></div></div></div></blockquote><div class=""><br class=""></div></span>Sounds prefectly reasonable.<span class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><br class="">Could you remind me why LLDB wants to know which modules are referenced from a CU? (rather than just all the modules used by a program overall?)<br class=""></div></div></div></div></div></blockquote><div class=""><br class=""></div></span><div class="">LLDB uses clang for the expression evaluation. Traditionally it would look up a type in DWARF, build a clang AST out of it and then import it. With this it could directly import the clang modules and have access to everything in the module. But, clang modules are not namespaces, so modules can conflict (and that would probably manifest as a crash in libclang).<span class="Apple-converted-space"> </span></div></div></div></blockquote><div class=""><br class="">What's an example of such a conflict? Is that valid (or is it just in ODR violations) - as mentioned above, it seems to me that only importing the things lexically available in this source file isn't what a debugger user would really want. I certainly think I'd trip over that a lot.<br class=""></div></div></div></blockquote><div><br class=""></div><div>Keep in mind that Objective-C (and C) do not have an ODR, so it’s not just “just” :-)</div><div>Being able to import modules does not mean that the debugger cannot still fall back to loading types from DWARF; in fact it will have to do that for all local types anyway.</div><div><br class=""></div><div>-- adrian</div><br class=""><blockquote type="cite" class=""><div class=""><div class="gmail_quote" style="font-family: LucidaGrande; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class="">It therefore needs to know which modules are imported in the current CU before dropping into the expression evaluator.</div><span class="HOEnZb"><font color="#888888" class=""><div class=""><div class=""><br class=""></div><div class="">- adrian</div></div></font></span><div class=""><div class="h5"><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><br class=""><br class=""><br class="">On Macho-O, in the absence of comdats, we have:<br class=""><br class="">bar.o<br class="">~~~~~<br class=""><br class="">.debug_info:<br class=""> <span class="Apple-converted-space"> </span>DW_TAG_compile_unit<br class=""> <span class="Apple-converted-space"> </span>...<br class=""> <span class="Apple-converted-space"> </span>DW_TAG_imported_module<br class=""> <span class="Apple-converted-space"> </span>DW_AT_import [DW_FORM_ref4] (0x20)<br class=""><br class=""> <span class="Apple-converted-space"> </span>DW_TAG_module // uniqued by dsymutil.<br class=""><span class=""> <span class="Apple-converted-space"> </span>DW_AT_name(“FooLib”)<br class=""> <span class="Apple-converted-space"> </span>DW_AT_LLVM_sysroot(“/“)<br class=""> <span class="Apple-converted-space"> </span>DW_AT_LLVM_include_dirs(“-I/path”)<br class=""> <span class="Apple-converted-space"> </span>DW_AT_LLVM_macros(“-DNDEBUG”)<br class=""> <span class="Apple-converted-space"> </span>...<br class=""><br class=""></span>// Split DWARF skeleton, thrown out by dsymutil.<br class=""></blockquote><div class=""><br class="">Thrown out? Because it's going to read everything in from the module and merge it in to a single linked debug info blob, I take it?<br class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><span class="">.debug_info, group 0xFEDB9876, comdat<br class=""> <span class="Apple-converted-space"> </span>DW_TAG_compile_unit<br class=""> <span class="Apple-converted-space"> </span>DW_AT_dwo_name(“/tmp/org.llvm.clang/ModuleCache/1234ABCDE/FooLib-XYZ.pcm”)<br class=""> <span class="Apple-converted-space"> </span>DW_AT_dwo_id(“0xFEDB9876”)<br class=""> <span class="Apple-converted-space"> </span>...<br class=""><br class=""></span><span class="">FooLib-XYZ.pcm<br class="">~~~~~~~~~~~~~~<br class=""><br class="">.debug_info:<br class=""> <span class="Apple-converted-space"> </span>DW_TAG_compile_unit<br class=""> <span class="Apple-converted-space"> </span>DW_AT_dwo_id(“0xFEDB9876”)<br class=""></span><span class=""> <span class="Apple-converted-space"> </span>...<br class=""><br class=""> <span class="Apple-converted-space"> </span>DW_TAG_module<br class=""> <span class="Apple-converted-space"> </span>DW_AT_name(“FooLib”)<br class=""></span> <span class="Apple-converted-space"> </span>DW_TAG_module<br class=""> <span class="Apple-converted-space"> </span>DW_AT_name(“SubFoo”)<br class=""> <span class="Apple-converted-space"> </span>...<br class=""><span class=""><font color="#888888" class=""><br class="">-- adrian<br class=""></font></span><div class=""><div class=""><br class="">><br class="">>><br class="">>>><br class="">>>> > If it turns out that's the right way to get a target for the imported_module, we could put both the skeleton CU and the partial unit in the same comdat and dedup them both together.<br class="">>>><br class="">>>> I think this works as long as we only have one TAG_module per .pcm file (because we need to refer to it via signature).<br class="">>>><br class="">>>> Not quite following here - why would we have more than one module per pcm - a pcm is a module, right?<br class="">>><br class="">>> Clang modules may have submodules and a compile unit could import two submodules that live in the same .pcm file. For example on Darwin there is a module Darwin.pcm that contains a submodule “C" that contains the submodule “stdio".<br class="">><br class="">> OK, so this bit's relevant to your use case in LLDB of loading the right things for the right context, but not relevant to the context-less debuggers like GDB that will just treat everything as one big namespace (except for file-local things, etc). So it's important for your imported modules but not for the basic Fission style debug reference.<br class="">><br class="">> Well, maybe - I'm not sure what you're picturing in terms of the DWARF in the module for submodules? If you want that granularity we'll have to talk about how to split the DWARF in the module into chunks per submodule?<br class="">><br class="">>><br class="">>>><br class="">>>> But if we don’t mind having duplicate dwo_* references in the same .o file this would also work with more than one TAG_module (or submodules).<br class="">>>><br class="">>>><br class="">>>> .debug_info:<br class="">>>> DW_TAG_compile_unit<br class="">>>> DW_AT_name(“bar.c”)<br class="">>>> ...<br class="">>>><br class="">>>> DW_TAG_imported_module // <- This could be optional on ELF.<br class="">>>> DW_AT_import [DW_FORM_ref_sig8] (0xFEDB9876)<br class="">>>><br class="">>>> ...<br class="">>>><br class="">>>> // Comdat’d split DWARF skeleton CU for the module Foo.<br class="">>>> .debug_info, group 0xFEDB9876, comdat<br class="">>>> DW_TAG_compile_unit<br class="">>>> DW_AT_dwo_name(“/tmp/org.llvm.clang/ModuleCache/1234ABCDE/FooLib-XYZ.pcm”)<br class="">>>> DW_AT_dwo_id(“0xFEDB9876”)<br class="">>>> ...<br class="">>>><br class="">>>> DW_TAG_module<br class="">>>> DW_AT_name(“FooLib”)<br class="">>>> DW_AT_LLVM_sysroot(“/“)<br class="">>>> DW_AT_LLVM_include_dirs(“-I/path”)<br class="">>>> DW_AT_LLVM_macros(“-DNDEBUG”)<br class="">>>> ...<br class="">>>><br class="">>>><br class="">>>> ><br class="">>>> > But this gets into complicated territory when the original binary is built with fission... which will be relevant for modules on ELF with LLDB. Hmm, maybe it's not too complicated - the partial_unit would end up in the .dwo file (maybe we'd have to teach the .dwo file to deduplicate these too - the same way it does for type units... - might require a new header to include the hash, etc :/)... would be tricky to have the dwp tool resolve the relocations to these things. Cross-unit references as you've got there aren't something that every DWARF consumer is totally cool with, I don't think?<br class="">>>><br class="">>>> Ah. I thought the deduplication happens because all ELF sections sharing the same group are uniqued based on the group id.<br class="">>>><br class="">>>> COMDAT groups deduplicate for a normal non-fission build, but fission linking doesn't require the .dwo file to use/contain COMDATs as it uses a DWARF-aware tool (so you don't bother putting the type units in COMDAT groups, for example - the fission linker knows how to parse debug_types, find the type unit headers and their hashes and deduplicates them that way).<br class="">>><br class="">>> Ok that makes sense.<br class="">>><br class="">>> -- adrian<br class="">>><br class="">>>><br class="">>>> It certainly would be nice if we could avoid introducing a new .debug_info header...<br class="">>>><br class="">>>> ><br class="">>>> > Sort of inclined to have the imported module stuff just for LLDB, but I've lost some of the context for that in the ensuing weeks.<br class="">>>><br class="">>>> -- adrian<br class="">>>><br class="">>>> ><br class="">>>> >><br class="">>>> >><br class="">>>> >><br class="">>>> >><br class="">>>> >> MachO (no typeunits, no comdats, with imports)<br class="">>>> >> ----------------------------------------------<br class="">>>> >><br class="">>>> >> Since we don’t have comdat sections in Mach-O and we don’t have the tool support for type units, the way that external types can be referenced necessarily needs to be a bit different. The design that Greg and I came up with for Mach-O relies on llvm-dsymutil to fix up the DWARF for non-module-aware consumers. Just as ELF DWARF consumers need not be able to tell the difference between module debugging an split DWARF, on Mach-O the .dSYM bundle generated by llvm-dsymutil looks like traditional DWARF.<br class="">>>> >><br class="">>>> >> There are three differences in the DWARF output that make this possible:<br class="">>>> >> - Refer to external types by UID rather than by type signature.<br class="">>>> >> (This doubles as the key that allows a debugger to look import the type<br class="">>>> >> directly from the AST and protects us against hash collisions)<br class="">>>> >> - Add an index to the .o file that maps UID -> module file.<br class="">>>> >> (Fast lookup + UIDs for C and ObjC are only unique within a module)<br class="">>>> >> - Add an entry for each type’s UID to the types accelerator table.<br class="">>>> >> (Fast lookup)<br class="">>>> >><br class="">>>> >> bar.o<br class="">>>> >> ~~~~~<br class="">>>> >><br class="">>>> >> .debug_info:<br class="">>>> >> DW_TAG_compile_unit<br class="">>>> >> DW_AT_name(“bar.c”)<br class="">>>> >> DW_TAG_imported_module<br class="">>>> >> DW_AT_import(DW_FORM_ref_addr 0x40)<br class="">>>> >><br class="">>>> >> DW_TAG_variable<br class="">>>> >> DW_AT_name(“MyFoo”)<br class="">>>> >> DW_AT_type [DW_FORM_strp] (“_ZTS3Foo”) // We could use a custom FORM here<br class="">>>> >><br class="">>>> >> // Skeleton unit.<br class="">>>> >> DW_TAG_compile_unit<br class="">>>> >> DW_AT_dwo_name(“/tmp/org.llvm.clang/ModuleCache/1234ABCDE/FooLib-XYZ.pcm”)<br class="">>>> >> DW_AT_dwo_id(“0xFEDB9876”)<br class="">>>> >> ...<br class="">>>> >> 0x40:<br class="">>>> >> DW_TAG_module<br class="">>>> >> DW_AT_name(“FooLib”)<br class="">>>> >> DW_AT_LLVM_sysroot(“/“)<br class="">>>> >> DW_AT_LLVM_include_dirs(“-I/path”)<br class="">>>> >> DW_AT_LLVM_macros(“-DNDEBUG”)<br class="">>>> >><br class="">>>> >> // This index uses the usual accelerator table format.<br class="">>>> >> .apple_exttypes:<br class="">>>> >> { “_ZTS3Foo” => debug_str offset of ”/tmp/org.llvm.clang/ModuleCache/1234ABCDE/FooLib-XYZ.pcm” }<br class="">>>> >><br class="">>>> >> FooLib-XYZ.pcm<br class="">>>> >> ~~~~~~~~~~~~~~<br class="">>>> >><br class="">>>> >> .debug_info<br class="">>>> >> DW_TAG_compile_unit<br class="">>>> >> DW_AT_dwo_id(“0xFEDB9876”)<br class="">>>> >><br class="">>>> >> 0x80:<br class="">>>> >> DW_TAG_structure_type<br class="">>>> >> DW_AT_name (“Foo”)<br class="">>>> >> DW_AT_signature<br class="">>>> >> ...<br class="">>>> >><br class="">>>> >> // In addition to the entry for “Foo”, there is also an entry for the type’s UID “_ZTS3Foo” pointing to the type definition DIE.<br class="">>>> >> .apple_types<br class="">>>> >> { “Foo” => 0x80 }<br class="">>>> >> { “_ZTS3Foo” => 0x80 }<br class="">>>> >><br class="">>>> >><br class="">>>> >><br class="">>>> >> When the debug info linker (llvm-dsymutil) is run, it first pulls in the .debug_info section from the clang module and fixes up all the DW_FORM_strp external type references by turning them into a DW_FORM_ref_addr that references the type in the DW_TAG_compile_unit pulled in from the module. To find the correct type DIE it looks up the UID in the .apple_exttypes index, finds the module, looks up the UID in the regular .apple_types accelerator table and replaces the temporary DW_FROM_strp with a DW_FORM_ref_addr (which incidentally takes up the same amount of space in the DIE).<br class="">>>> >><br class="">>>> >><br class="">>>> >> Thoughts?<br class="">>>> >> --<br class="">>>> >> adrian<br class="">>>> >><br class="">>>> ><br class="">>>><br class="">>><br class="">>><br class="">></div></div></blockquote></div></div></div></div></blockquote></div></div></div></div></blockquote></div></div></blockquote></div><br class=""></body></html>