<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Paul, thanks for the clarification. Yes, that was my fault. The LTO object DWARF has multiple top-level compile unit tags and only the first one has a DW_AT_ranges attribute.<div class=""><br class=""></div><div class="">One more thought here: In DWARFDebugInfo::GetCompileUnitAranges() LLDB currently tries to read .debug_aranges and if it can’t, it falls back to reading DW_AT_ranges from the compile unit tags. Would you have objections from constructing that range from DW_AT_low_pc and DW_AT_high_pc here, if they exist? IIUC code for a single compile unit is not guaranteed to be compact, but in practice it usually is right?</div><div class=""><br class=""></div><div class="">Thanks</div><div class="">Stefan</div><div class=""><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 18. Sep 2018, at 15:01, <a href="mailto:paul.robinson@sony.com" class="">paul.robinson@sony.com</a> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Are the nested DW_TAG_compile_unit tags in my LTO object missing their DW_AT_ranges or is that expected?<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><a name="_MailEndCompose" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">Compile units should never be nested. That's invalid DWARF. A unit is the root of the DIE tree described by each unit header.<o:p class=""></o:p></span></a></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">Whatever is producing one unit contained inside another is doing something wrong.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">--paulr<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="border-style: none none none solid; border-left-width: 1.5pt; border-left-color: blue; padding: 0in 0in 0in 4pt;" class=""><div class=""><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 10pt; font-family: Tahoma, sans-serif;" class="">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif;" class=""><span class="Apple-converted-space"> </span>lldb-dev [<a href="mailto:lldb-dev-bounces@lists.llvm.org" class="">mailto:lldb-dev-bounces@lists.llvm.org</a>]<span class="Apple-converted-space"> </span><b class="">On Behalf Of<span class="Apple-converted-space"> </span></b>Stefan Gränitz via lldb-dev<br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Friday, September 14, 2018 11:57 AM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span>LLDB Dev<br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [lldb-dev] Extract compile unit infos from OSO entries for LTO objects<o:p class=""></o:p></span></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">I made some progress on this and next to many small issues, I found that DWARFDebugAranges::FindAddress() returns 0 as the CU offset for my file addresses and wonder if this is expected?<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">DWARFDebugAranges::FindAddress() is supposed to read from .debug_aranges section or DW_AT_ranges attributes right? When dumping with llvm-dwarfdump, it looks like regular objects have exactly one DW_TAG_compile_unit and it has a DW_AT_ranges attribute. Same for the top-level DW_TAG_compile_unit in LTO objects, but all further DW_TAG_compile_unit tags have no DW_AT_ranges attribute here. The comment in DWARFUnit::BuildAddressRangeTable() states that Clang “emits DW_AT_ranges for DW_TAG_compile_units”.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Are the nested DW_TAG_compile_unit tags in my LTO object missing their DW_AT_ranges or is that expected?<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">It looks like FindAddress() would return the offset correctly if it was there and the fix may get simpler.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 7.5pt; font-family: Menlo, serif;" class="">Top-level:</span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 7.5pt; font-family: Menlo, serif;" class="">0x0000000b: DW_TAG_compile_unit</span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 7.5pt; font-family: Menlo, serif;" class=""> ...<br class=""> DW_AT_low_pc. (0x0000000000000000)<br class=""> DW_AT_ranges. (0x00000040<br class=""> [0x0000000000000000, 0x00000000000007de)<br class=""> [0x00000000000007f0, 0x00000000000033cd))</span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 10pt; font-family: Menlo, serif;" class="">Nested:</span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 10pt; font-family: Menlo, serif;" class="">0x000130b3: DW_TAG_compile_unit</span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 7.5pt; font-family: Menlo, serif;" class=""> ...<br class=""> DW_AT_low_pc (0x0000000000003f50)<br class=""> DW_AT_high_pc. (0x000000000000dd0f)</span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><br class=""><br class=""><o:p class=""></o:p></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">As far as I can tell, we need the actual number of CUs only after we discovered plugins.<o:p class=""></o:p></div></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Split that off into HasCompileUnits() and GetNumCompileUnits(). It works well so far. Also managed to extract the single CUs with correct offsets in the LTO object file once GetNumCompileUnits() calls InitOSO(). I kept the logic for associating the CUs with line ranges in the Symtab, so first/last_symbol_id/index are the same for all those CUs. Maybe the code needs a few more adjustments to support this, but so far I don’t see showstoppers.<o:p class=""></o:p></div></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Short recap for what works after the simple changes:<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">* We iterate over the actual CUs in BreakpointResolverFileLine::SearchCallback().<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">* For the matching CU, we get the actual list of support files and find the correct file_idx and line_entry in CompileUnit::ResolveSymbolContext().<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">* We correctly link the OSO line table for this CU (with some overhead due to overlapping symbol ranges, but IIUC it’s fine as we only pick the ones for the CU).<o:p class=""></o:p></div></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">IF you do make a patch, please remove any functions in SymbolFileDWARFDebugMap that are dead code. SymbolFileDWARFDebugMap::GetModuleByOSOIndex() seems to be dead. If that is dead thenvSymbolFileDWARFDebugMap::GetObjectFileByOSOIndex() seems to be dead also.<o:p class=""></o:p></div></div></div></div></blockquote></div></blockquote></div></div></blockquote><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Yes both functions are dead and yes sure, I can include that in my patch. Another one is SymbolFileDWARFDebugMap::PrivateFindGlobalVariables(), which is one of the clients for symbol slices/ranges. The remaining ones are SymbolFileDWARFDebugMap::SymbolContainsSymbolWithID/Index() and SymbolFileDWARFDebugMap::CompileUnitInfo::GetFileRangeMap(), which are all alive.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><br class=""><br class=""><o:p class=""></o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">On 11. Sep 2018, at 20:13, Stefan Gränitz via lldb-dev <<a href="mailto:lldb-dev@lists.llvm.org" style="color: purple; text-decoration: underline;" class="">lldb-dev@lists.llvm.org</a>> wrote:<o:p class=""></o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div><div class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Thanks for your replies!<o:p class=""></o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><br class=""><br class=""><o:p class=""></o:p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Right now we assume that each .o file has only on CU so we don't need to open all .o files in SymbolFileDWARFDebugMap::CalculateAbilities() which is something that gets run when we are trying to figure out which SymbolFile plug-in to load. [...] The problem used to be that even if we had a dSYM file, the loop that selected the symbol file plug-in would give each and every symbol file plugin a chance ...<o:p class=""></o:p></div></div></div></blockquote><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Ok makes total sense, kind of historical reason.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">But that being said, in the past I re-ordered how the SymbolFile plug-ins were initialized to always put SymbolFileDWARF first and if it finds DWARF debug info or a dSYM file and has all the abilities then we stop looking for symbol file plug-ins that can load the current file. [...] So as soon as a SymbolFile plug-in can do everything we now stop.<o:p class=""></o:p></div></div></div></blockquote></blockquote><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Good to know!<o:p class=""></o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Note that counting the number of compile units can be done extremely cheaply (of course, except the cost of opening the file). Each unit as a header that contain its length (which gives you the next unit). I’m not sure we have a primitive that does this without parsing the DWARF, but it should be easy to add.<o:p class=""></o:p></div></div></blockquote></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Right, so we only need to parse the CU headers. That should be fast.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Opening each candidate .o file for the relatively rare case of having multiple CUs still sounds expensive, assuming that “thousands of .o files” may actually happen.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">CalculateAbilities() does indeed call GetNumCompileUnits(), but what it really wants to know at this time is “do we have any CU in there”:<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><br class="">uint32_t SymbolFileDWARFDebugMap::CalculateAbilities() {<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""> ...<br class=""> const uint32_t oso_index_count = GetNumCompileUnits();<br class=""> if (oso_index_count > 0) {<o:p class=""></o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""> InitOSO();<br class=""> if (!m_compile_unit_infos.empty()) {<br class=""> return SymbolFile::CompileUnits | SymbolFile::Functions | …;<o:p class=""></o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""> }<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">}<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">As far as I can tell, we need the actual number of CUs only after we discovered plugins. In my case it’s during breakpoint resolution (i.e. BreakpointResolverFileLine::SearchCallback()). If we separated these two concerns conceptually (into HasCompileUnits() and GetNumCompileUnits()), couldn’t we then also do InitOSO() in two steps? Especially since lazy init is used everywhere already. This would avoid impact on CalculateAbilities() entirely.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">That said, I don’t really know how big the change would get then. And it probably adds complexity, while the implementation is quite complex already.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Anyway, for now what do you think about the idea?<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Do we really need this CU <-> Symbol mapping?<o:p class=""></o:p></div></div></blockquote><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">It’s used in SymbolFileDWARFDebugMap::SymbolContainsSymbolWithIndex(), which looks like dead code.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">It’s also used in SymbolFileDWARFDebugMap::CompileUnitInfo::GetFileRangeMap(), which initialises the OSO range map once. In order to do that it iterates over all CUs, so changing this or adding a special case here seems possible.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><a href="https://github.com/llvm-mirror/lldb/blob/59608853be9b52d3c01609196d152b3e3dbb4dac/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDebugMap.cpp#L172" style="color: purple; text-decoration: underline;" class="">https://github.com/llvm-mirror/lldb/blob/59608853be9b52d3c01609196d152b3e3dbb4dac/source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDebugMap.cpp#L172</a><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">What do you think?<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Best<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Stefan<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><br class=""><br class=""><o:p class=""></o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">On 11. Sep 2018, at 18:03, Frédéric Riss <<a href="mailto:friss@apple.com" style="color: purple; text-decoration: underline;" class="">friss@apple.com</a>> wrote:<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><br class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br style="caret-color: rgb(0, 0, 0); font-variant-caps: normal; text-align: start; -webkit-text-stroke-width: 0px; word-spacing: 0px;" class=""><br class=""></span><o:p class=""></o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">On Sep 11, 2018, at 8:48 AM, Greg Clayton <<a href="mailto:clayborg@gmail.com" style="color: purple; text-decoration: underline;" class="">clayborg@gmail.com</a>> wrote:<o:p class=""></o:p></span></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div><div class=""><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class="">On Sep 11, 2018, at 2:55 AM, Stefan Gränitz <<a href="mailto:sgraenitz@apple.com" style="color: purple; text-decoration: underline;" class="">sgraenitz@apple.com</a>> wrote:<o:p class=""></o:p></span></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Hello everyone<o:p class=""></o:p></span></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">I am investigating a bug that prevents correct breakpoint resolution in LTO objects with embedded DWARF (no separate dSYM file) and tracked it down to the initialization of SymbolFileDWARFDebugMap. This code seems to assume there is only one compile unit per object file, but LTO objects have more than that:<o:p class=""></o:p></span></div></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class="">void SymbolFileDWARFDebugMap::InitOSO() {</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class="">...</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class=""> const uint32_t oso_index_count =</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class=""> symtab->AppendSymbolIndexesWithTypeAndFlagsValue(</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class=""> eSymbolTypeObjectFile, k_oso_symbol_flags_value, oso_indexes);</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class="">...</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class=""> m_compile_unit_infos.resize(oso_index_count); // <—— one CU per OSO entry in the Symtab</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class=""> for (uint32_t i = 0; i < oso_index_count; ++i) {</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class=""> const uint32_t so_idx = oso_indexes[i] - 1;</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class=""> const uint32_t oso_idx = oso_indexes[i];</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class=""> const Symbol *so_symbol = symtab->SymbolAtIndex(so_idx);</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class=""> const Symbol *oso_symbol = symtab->SymbolAtIndex(oso_idx);</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class="">...</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class=""> const Symbol *last_symbol = symtab->SymbolAtIndex(sibling_idx - 1);</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class=""> m_compile_unit_infos[i].first_symbol_index = so_idx;</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class=""> m_compile_unit_infos[i].last_symbol_index = sibling_idx - 1;</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class=""> m_compile_unit_infos[i].first_symbol_id = so_symbol->GetID();</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 8.5pt; font-family: Menlo, serif;" class=""> m_compile_unit_infos[i].last_symbol_id = last_symbol->GetID();</span><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">The symptom is that LLDB will only read debug_line for one CU and miss all the rest. Thus, breakpoints in other CUs can’t be associated with line information.<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">I wonder if there is a good way to populate the correct number of compile units from the OSO entry at this point?<o:p class=""></o:p></span></div></div></div></div></blockquote><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">The reason it is like this is we don't want to have to open all .o files when we parse the debug map in order to figure out a compile unit index. Right now the compile unit UserID is just the index of the .o file in the debug map. Opening thousands of .o files can impose a performance issue.<br class=""><br class=""><o:p class=""></o:p></span></div><div class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">The situation may appear similar to an archive file with a number of objects, but then we have separate OSO entries like “path/to/lib/libLLVMMCParser.a(AsmLexer.cpp.o)”. Furthermore LTO objects have one common symtab for all compile units and it was probably mixed up by optimization, so we cannot simply say that CUs start/end at certain symbol indexes as in the above code. The latter is used rarely and only in SymbolFileDWARFDebugMap, so there may be a workaround, but first I have to figure out the initial question:<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">How to get more information about compile units in an LTO object? Any ideas welcome!<o:p class=""></o:p></span></div></div></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">The only way is to open each .o file and see how many compile units they contain. Right now we assume that each .o file has only on CU so we don't need to open all .o files in SymbolFileDWARFDebugMap::CalculateAbilities() which is something that gets run when we are trying to figure out which SymbolFile plug-in to load. But that being said, in the past I re-ordered how the SymbolFile plug-ins were initialized to always put SymbolFileDWARF first and if it finds DWARF debug info or a dSYM file and has all the abilities then we stop looking for symbol file plug-ins that can load the current file. The problem used to be that even if we had a dSYM file, the loop that selected the symbol file plug-in would give each and every symbol file plugin a chance to tell us how much info they could extract via a call to SymbolFile::CalculateAbilities() and that would cause us to open all .o files just to say "I parse all debug info" just like a previous plug-in could. So as soon as a SymbolFile plug-in can do everything we now stop.<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">If that’s not possible, I may find another way to fix it further down the road, but then the name m_compile_unit_infos seems not exactly correct here. It’s rather something like m_referenced_object_infos, right?<o:p class=""></o:p></span></div></div></div></div></blockquote><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">So now that that change has been in for a while, it might be ok to open each .o file and see how many compile units they contain and then populate m_compile_unit_infos as needed.<o:p class=""></o:p></span></div></div></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Note that counting the number of compile units can be done extremely cheaply (of course, except the cost of opening the file). Each unit as a header that contain its length (which gives you the next unit). I’m not sure we have a primitive that does this without parsing the DWARF, but it should be easy to add. <o:p class=""></o:p></span></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br style="caret-color: rgb(0, 0, 0); font-variant-caps: normal; text-align: start; -webkit-text-stroke-width: 0px; word-spacing: 0px;" class=""><br class=""></span><o:p class=""></o:p></div><div class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">You will need to watch for any usage of m_compile_unit_infos and make sure it does the correct thing.<o:p class=""></o:p></span></div></div></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">That’s the part I was worried about. The structure of m_compile_unit_infos makes the assumption that we can associate slices of symbols to a compile unit. I don’t think this is a correct assumption to make in the LTO case, and even if it were, we’d need to parse the DWARF and do some pretty heavy processing to extract the information. Do we really need this CU <-> Symbol mapping?<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Fred <o:p class=""></o:p></span></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br style="caret-color: rgb(0, 0, 0); font-variant-caps: normal; text-align: start; -webkit-text-stroke-width: 0px; word-spacing: 0px;" class=""><br class=""></span><o:p class=""></o:p></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class=""><br class=""><o:p class=""></o:p></span></div><div class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Btw.: My first attempt was a workaround for the symptom (see<span class="apple-converted-space"> </span><a href="https://reviews.llvm.org/D51546" style="color: purple; text-decoration: underline;" class="">https://reviews.llvm.org/D51546</a>). It simply reads all debug_lines for a single CU, but I’d really appreciate a better solution.<o:p class=""></o:p></span></div></div></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">The fix in D51546 seems wrong because the only way we get to a line table is via the DW_AT_stmt_list from a compile unit. If we can fix the LTO case to load all compile units from the LTO.o files with multiple CU's this fix won't be needed.<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">So the correct solution is to detect how many compile units are in each .o file and then make sure to find all places that were assuming anything about the OSO index being the compile unit UserID are fixed. Now that plug-in loading stops after a SymbolFile says it can handle everything we can probably do a bit more work. One issue is that .o files might have been cleaned up or removed, so be sure to test any solution by removing the .o files and seeing how we do.<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">I will be happy to review any patch you have. I can't think of any other reason the the OSO index needs to be the compile unit index. IF you do make a patch, please remove any functions in SymbolFileDWARFDebugMap that are dead code. SymbolFileDWARFDebugMap::GetModuleByOSOIndex() seems to be dead. If that is dead thenvSymbolFileDWARFDebugMap::GetObjectFileByOSOIndex() seems to be dead also.<o:p class=""></o:p></span></div></div></div></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">_______________________________________________<br class="">lldb-dev mailing list<br class=""><a href="mailto:lldb-dev@lists.llvm.org" style="color: purple; text-decoration: underline;" class="">lldb-dev@lists.llvm.org</a><br class=""><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev</a></div></div></div></div></div></div></blockquote></div><br class=""></div></div></body></html>