<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On May 3, 2017, at 2:59 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com" class="">dblaikie@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""><br class=""><div class="gmail_quote"><div dir="ltr" class="">On Wed, May 3, 2017 at 2:09 PM Adrian Prantl <<a href="mailto:aprantl@apple.com" class="">aprantl@apple.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><br class="">> On May 3, 2017, at 2:00 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank" class="">dblaikie@gmail.com</a>> wrote:<br class="">><br class="">> So Dehao and I have been dealing with some of the nitty gritty details of debug info with ThinLTO, specifically with Fission(Split DWARF).<br class="">><br class="">> This applies to LTO as well, so I won't single out ThinLTO here.<br class="">><br class="">> 1) Multiple CUs in a .dwo file<br class="">> Clang/LLVM produces a CU for each original source file - these CUs are kept through IR linking (thin or full) and produced as distinct CUs in the resulting DWARF.<br class="">> This preserves semantics as much as possible - eg: file-local functions and types (those in anonymous namespaces or declared file-static) con co-exist even if their names collide.<br class="">> GDB does good things with this, except with Fission.<br class=""><br class="">Can you lay out the terminology you are using here? Is Fission as used here different/more than creating .dwo files?<br class=""></blockquote><div class=""><br class="">Fission as used in that sentence means .dwo files and anything beyond (.dwp files - <a href="https://gcc.gnu.org/wiki/DebugFissionDWP" class="">https://gcc.gnu.org/wiki/DebugFissionDWP</a> ). GDB warns/errors/complains if two CUs are in a single .dwo, and ignores all but the first.<br class=""></div></div></div></div></blockquote><div><br class=""></div><div>It sounds like having more than one CU per .dwo is outside of the intention of the DWARF specification (though not explicitly forbidden), since there is an implied 1-1 relationship between skeleton CU and .dwo.</div><div><br class=""></div><div>What do you think is the *right* solution here: </div><div>- Allowing more than one CU in gdb?</div><div>- Emitting many little .dwo files for the cross-inlined functions' CUs?</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><br class="">> Binutils DWP produces usable DWPs from DWO files with multiple CUs<br class="">><br class="">> 2) Cross-CU references<br class="">> This is where it gets trickier.<br class="">> LLVM produces cross-CU references (DW_FORM_ref_addr) to refer to types or functions defined in one CU used from another. This only happens with (thin or plain) LTO. LLVM's had this for a while.<br class="">> This helps fully express interesting cases outlined in (1) - it's possible that a file-local function is inlined into another CU - by using ref_addr the semantics described in (1) can be preserved, while also explaining the inlining that has occurred. (similar cases can arise with intra-CU type references too)<br class="">> GDB handles this, except with Fission (I mean, it doesn't get this far - but even with patches to handle (1)+Fission, it's still not enough - needs more work)<br class="">> Binutils DWP and the DWP format in general... maybe can't cope with this.<br class="">><br class="">> Talking about (2)+DWP:<br class=""><br class="">You mean (2)+DWO+DWP?<br class=""></blockquote><div class=""><br class="">DWP is a package containing the contents of multiple DWOs - so, yes and no? Not sure how to answer.<br class=""><br class="">(2)+DWO could be made to work without changing much in the contents/format/etc, I think - consumers could be made to understand the 'obvious' form (no change to producers I think would be needed).<br class=""><br class="">But once you go to a DWP file, then there are repreesntational problems that make it currently impossible to use cross-CU references. The semantics (& possibly the syntax) of DWP files would have to change to enable this functionality.<br class=""></div></div></div></div></blockquote><div><br class=""></div>Why can a .dwo cope with cross-CU-references but not a .dwp?<br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><br class="">> It looks like this may require adjusting at least the semantics, if not the syntax of the indexes in the DWP file. I've started a thread on dwarf-discuss to start trying to hash that out.<br class="">><br class="">><br class="">><br class="">> So, to cut a long story short: Fission+LTO is currently unusable and may take a while to make the DWARF side of this usable.<br class="">><br class="">> In the interim, I'd like to propose adding a flag to LLVM to support something not very nice: When merging modules (or importing things into a module in ThinLTO), do not maintain separate CUs but redirect the cu pointer in any DISubprogram (& probably DIGlobalVariable too - I forget if ThinLTO can import them) metadata to refer to the original CU in the destination module. (in the full LTO case, it'd also be necessary to import any retained types)<br class="">><br class="">> This flag could be passed to the task doing the merging/importing, or could be passed to the frontend compile and stashed in per-CU metadata (where it would be respected when importing/merging from that CU).<br class="">><br class="">> The intent would be to support this flag until the DWARF functionality can be decided on and implemented - and kept for some time after that for backwards compatibility for those who want to use Fission but haven't got the latest toolchains with the fixes/improvements in them.<br class="">><br class=""></blockquote></div></div></div></blockquote><div><br class=""></div><div>If the other tools cannot be fixed quickly, then this would be a pragmatic interim solution.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">><br class="">><br class="">> How's this sound to everyone? Reasonable? Unreasonable? Need more details about the impact of these changes, etc? (I have examples with DWARF output, GDB behavior, etc)<br class=""><br class="">If you can post an example, that would help.<br class=""></blockquote><div class=""><br class="">OK, so, here's an example of how GDB, without fission, benefits from the two distinct CUs (rather than bundling all the types & functions into a single CU):<br class=""><br class="">given:<br class=""><font face="monospace" class=""> a.cpp:<br class=""></font><div class=""><font face="monospace" class=""> namespace {</font></div><div class=""><font face="monospace" class=""> struct foo {</font></div><div class=""><font face="monospace" class=""> int i;</font></div><div class=""><font face="monospace" class=""> };</font></div><div class=""><font face="monospace" class=""> }</font></div><div class=""><font face="monospace" class=""> static void f1() {</font></div><div class=""><font face="monospace" class=""> }</font></div><div class=""><font face="monospace" class=""> void f2() {</font></div><div class=""><font face="monospace" class=""> f1();</font></div><div class=""><font face="monospace" class=""> }<br class=""> b.cpp:<br class=""></font><div class=""><font face="monospace" class=""> namespace {</font></div><div class=""><font face="monospace" class=""> struct foo {</font></div><div class=""><font face="monospace" class=""> float f;</font></div><div class=""><font face="monospace" class=""> };</font></div><div class=""><font face="monospace" class=""> }</font></div><div class=""><font face="monospace" class=""> static void f1() {</font></div><div class=""><font face="monospace" class=""> }</font></div><div class=""><font face="monospace" class=""> void f2();</font></div><div class=""><font face="monospace" class=""> int main() {</font></div><div class=""><font face="monospace" class=""> f1();</font></div><div class=""><font face="monospace" class=""> f2();</font></div><div class=""><font face="monospace" class=""> }<br class=""> $ clang++ {a,b}.cpp -g -c -emit-llvm -S<br class=""> $ llvm-link {a,b}.ll -S -o ab.ll</font></div><div class=""><font face="monospace" class=""> $ clang++-tot ab.ll<br class=""> $ gdb a.out<br class=""> (gdb) start</font></div><div class=""><span style="font-family: monospace;" class=""> ..., main () at b.cpp:11</span></div><div class=""><font face="monospace" class=""> 11 f1();</font></div><div class=""><font face="monospace" class=""> (gdb) ptype foo</font></div><div class=""><font face="monospace" class=""> type = struct (anonymous namespace)::foo {</font></div><div class=""><font face="monospace" class=""> float f;</font></div><div class=""><font face="monospace" class=""> }</font></div><div class=""><font face="monospace" class=""> (gdb) p &f1</font></div><div class=""><font face="monospace" class=""> $1 = (void (*)(void)) 0x400530 <f1()></font></div><div class=""><font face="monospace" class=""> (gdb) n</font></div><div class=""><font face="monospace" class=""> 12 f2();</font></div><div class=""><font face="monospace" class=""> (gdb) s</font></div><div class=""><font face="monospace" class=""> f2 () at a.cpp:10</font></div><div class=""><font face="monospace" class=""> 10 f1();</font></div><div class=""><font face="monospace" class=""> (gdb) ptype foo</font></div><div class=""><font face="monospace" class=""> type = struct (anonymous namespace)::foo {</font></div><div class=""><font face="monospace" class=""> int i;</font></div><div class=""><font face="monospace" class=""> }</font></div><div class=""><font face="monospace" class=""> (gdb) p &f1</font></div><div class=""><font face="monospace" class=""> $2 = (void (*)(void)) 0x400500 <f1()></font><br class=""><br class="">So in that case, the foo type and f1 function are properly identified depending on the context in which the expression is evaluated.<br class=""><br class="">If the types and subprograms are tied into the same CU (with a bit of manual IR editing), two distinct types and functions are emitted into the DWARF, but GDB only finds the first one, every time, in both contexts.<br class=""><br class=""><br class="">To look into the larger motivation, cross-CU inlining and how it interacts with Fission:<br class=""><br class=""><font face="monospace" class=""> a.cpp:</font></div><div class=""><font face="monospace" class=""> __attribute__((optnone)) void f1() {</font><div class=""><font face="monospace" class=""> }</font></div><div class=""><font face="monospace" class=""> __attribute__((always_inline)) void f2() {</font></div><div class=""><font face="monospace" class=""> f1();</font></div><div class=""><font face="monospace" class=""> };</font></div><font face="monospace" class=""> b.cpp:<br class=""></font><div class=""><font face="monospace" class=""> void f2();</font></div><div class=""><font face="monospace" class=""> void f3() {</font></div><div class=""><font face="monospace" class=""> f2();</font></div><div class=""><font face="monospace" class=""> }<br class=""> c.cpp:<br class=""></font><div class=""><font face="monospace" class=""> void f3();</font></div><div class=""><font face="monospace" class=""> int main() {</font></div><div class=""><font face="monospace" class=""> f3();</font></div><div class=""><font face="monospace" class=""> }</font></div></div></div></div></div><div class=""><font face="monospace" class=""> $ clang++ {a,b}.cpp -g -c -emit-llvm -S<br class=""> $ llvm-link {a,b}.ll -S -o ab.ll<br class=""> $ clang++-tot c.cpp ab.ll -gsplit-dwarf<br class=""> $ dwp {c,ab}.dwo -o cab.dwp<br class=""><br class="">This produces two dwo files - one per object file (c.dwo, ab.dwo). If we look at the contents, they make a fair bit of sense:<br class=""><br class=""> 0x0b: DW_TAG_compile_unit [1] *</font></div><div class=""><font face="monospace" class=""> 0x19: DW_TAG_subprogram [2] </font></div><div class=""><font face="monospace" class=""> 0x25: DW_TAG_subprogram [3] </font></div><div class=""><font face="monospace" class=""> 0x31: DW_TAG_subprogram [4] </font></div><div class=""><font face="monospace" class=""> 0x43: DW_TAG_compile_unit [1] *</font></div><div class=""><font face="monospace" class=""> 0x51: DW_TAG_subprogram [5] *</font></div><div class=""><font face="monospace" class=""> 0x5d: DW_TAG_inlined_subroutine [6] </font></div><div class=""><font face="monospace" class=""> DW_AT_abstract_origin [DW_FORM_ref_addr] (0x31 "_Z2f2v")</font><br class=""><br class="">The ref_addr can be resolved relative to this whole .dwo file & the abstract subprogram is found. Yay.<br class=""><br class="">So GDB can't currently handle this:<br class=""><br class=""><font face="monospace" class=""> Could not find DWO CU ab.dwo(0x32dd6d7121dd1d9a) referenced by CU at offset 0x66 [in module a.out]</font><br class=""><br class="">And that's after some local fixes I have so it doesn't error out on the two-CUs-in-a-dwo situation (that fix at least allows the first example (local foo/f1, etc) to work with GDB - but fixing ref_addr to resolve correctly would require deeper changes I haven't figured out/prototyped yet).<br class=""><br class="">So, up until this point I was pretty satisfied there would be a way forward... <br class=""><br class="">Then I hit DWP files.<br class=""><br class="">DWP files represent multiple DWO files stacked together - mostly to deduplicate strings and type units as a sort of archival (like dsym) format.<br class=""><br class="">The way DWPs work, roughly (full/more details here: <a href="https://gcc.gnu.org/wiki/DebugFissionDWP" class="">https://gcc.gnu.org/wiki/DebugFissionDWP</a> ) is that all the sections are concatenated together, except the debug_str and debug_str_offsets sections which have special handling to do what you'd expect - deduplicate strings, and adjust the str_offsets to point to the right offsets in the deduplicated string section.<br class=""><br class="">The other thing that DWP has, is an index, or two. cu_index and tu_index. We'll just look at cu_index.<br class=""><br class="">A cu_index for the cab.dwp above, is:<br class=""><br class=""><div style="color: rgb(33, 33, 33); font-size: 13px;" class=""><font face="monospace" class="">Index Signature INFO ABBR LINE STR_OFF </font></div><div style="color: rgb(33, 33, 33); font-size: 13px;" class=""><font face="monospace" class="">----- ------------------ -------- -------- -------- --------</font></div><div style="color: rgb(33, 33, 33); font-size: 13px;" class=""><font face="monospace" class=""> 2 0x7bd765349b7e7631 [2d, 65) [38, ae) [11, 22) [14, 3c) </font></div><div style="color: rgb(33, 33, 33); font-size: 13px;" class=""><font face="monospace" class=""> 8 0x66f4e160661d2687 [00, 2d) [00, 38) [00, 11) [00, 14) </font></div><div style="color: rgb(33, 33, 33); font-size: 13px;" class=""><font face="monospace" class=""> 11 0x32dd6d7121dd1d9a [65, 98) [38, ae) [11, 22) [14, 3c)</font></div><br class="inbox-inbox-Apple-interchange-newline">What this is for is to tell the consumer, which portions of each section relate to which CUs - since the DIEs in the CU don't contain any relocations, for example the abbrev offset in the CU header is probably zero. It must be resolved relative to the ABBR section in the above table to find the chunk of the debug_abbrev that came from that CU.<br class=""><br class="">So, this is all good and well, except that the INFO range isn't the whole range of the debug info from the DWO file - it's /just/ the range of this CU. Which means there's no way to resolve the ref_addr relative to the whole range, you don't know where it starts.<br class=""><br class=""></div></div></div></div></blockquote><div><br class=""></div>Ah thanks, that explains my question above!<br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote"><div class=""><br class=""><br class="">So this leads us to a few options - the 'simplest', that changes the semantics but not the syntax of the table - would be to widen the INFO range to cover the whole portion that came from the original DWO file.<br class=""><br class="">Consumers would have to be adjusted to cope with the fact that the INFO contribution for a given DWO ID would be a range that contains the CU somewhere, along with other CUs - and they'd have to search (in full LTO, this would mean searching through /all/ the CUs).<br class=""></div></div></div></div></blockquote><div><br class=""></div><div>Would generating a .dwo per CU help with this?</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote"><div class="">This doesn't cover all the use cases I'd have in mind, but it would at least be possible to implement & support ref_addr. The full depth of design choices I'll likely leave to the dwarf-discuss thread, hopefully.</div></div></div></div></blockquote></div><br class=""><div class="">Yes, given all the non-LLVM tools that will need to support this, that is the right forum to discuss this.</div><div class=""><br class=""></div><div class="">-- adrian</div></body></html>