[lldb-dev] Loading debug symbols (C++, Ubuntu 14.04)

[Stefan Kratochwil via lldb-dev]

Hi Pavel,

I stepped through SymbolVendorELF::CreateInstance(), and it seems that 
the problem lies within Symbols::LocateExecutableSymbolFile 
(Host/common/Symbols.cpp). This method looks at the correct locations:

(gdb) print debug_file_search_paths
$35 = {m_files = std::vector of length 3, capacity 4 = {
   {m_directory = {m_string = 0x7994f8 "/usr/lib"},
    m_filename = {m_string = 0x7fffe400ba88 "x86_64-linux-gnu"},
    m_is_resolved = true,
    m_syntax = lldb_private::FileSpec::ePathSyntaxPosix},

   {m_directory = {m_string = 0x7fffe400bae0 
"/home/stefan/git/dsu-ld/libdsu-project"},
    m_filename = {m_string = 0x7fffe400bac8 "."},
    m_is_resolved = true,
    m_syntax = lldb_private::FileSpec::ePathSyntaxPosix},

   {m_directory = {m_string = 0x7994f8 "/usr/lib"},
    m_filename = {m_string = 0x7fffe400bb18 "debug"},
    m_is_resolved = true,
    m_syntax = lldb_private::FileSpec::ePathSyntaxPosix}}}


But it looks for the wrong file names. Under Ubuntu, the contents of a 
*-dbg packge are stored under /usr/lib/debug, using the exact same file 
names as their regular executable counterparts. In the case of 
apache2-dbg, the files under /usr/lib/debug are (for example):

/usr/lib/debug/usr/bin/rotatelogs
/usr/lib/debug/usr/sbin/apache2
/usr/lib/debug/usr/lib/apache2/modules/mod_ssl.so

and so on.



If i now look at the file names lldb looks for, I find for 
/usr/lib/x86_64-linux-gnu:
(gdb) print files
$36 = std::vector of length 4, capacity 4 = {
   "/usr/lib/x86_64-linux-gnu/9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug",
 
"/usr/lib/x86_64-linux-gnu/.debug/9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug",
 
"/usr/lib/x86_64-linux-gnu/.build-id/B5/9D72EE0B01E9B2947C7F95E7B4EBBBF05A8AA5.debug",
 
"/usr/lib/x86_64-linux-gnu/usr/lib/x86_64-linux-gnu/9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug"}

For my /home/stefan/git/dsu-ld/libdsu-project/:
(gdb) print files
$38 = std::vector of length 4, capacity 4 = {
 
"/home/stefan/git/dsu-ld/libdsu-project/./9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug",
 
"/home/stefan/git/dsu-ld/libdsu-project/./.debug/9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug",
 
"/home/stefan/git/dsu-ld/libdsu-project/./.build-id/B5/9D72EE0B01E9B2947C7F95E7B4EBBBF05A8AA5.debug", 

 
"/home/stefan/git/dsu-ld/libdsu-project/./usr/lib/x86_64-linux-gnu/9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug"}


And finally for the /usr/lib/debug directory:
(gdb) print files
$39 = std::vector of length 4, capacity 4 = {
   "/usr/lib/debug/9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug",
   "/usr/lib/debug/.debug/9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug",
 
"/usr/lib/debug/.build-id/B5/9D72EE0B01E9B2947C7F95E7B4EBBBF05A8AA5.debug",
 
"/usr/lib/debug/usr/lib/x86_64-linux-gnu/9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug"}


So, lldb expects that the uuid/build of the Module it is about to load 
is also contained in the symbol file name.


Imho, we also have to take the FileSpec contents of the ModuleSpec we 
are currently looking at, and add its path to the files vector. But I am 
not that familiar with the whole lldb backend to tell for sure what 
would happen then.


I also checked the current HEAD of the lldb source tree to see if there 
are any changes, compared with my older revision. There is only a slight 
difference in how Symbols::LoadExecutableSymbolFile uses the 
/usr/lib/debug directory:

// Some debug files may stored in the module directory like this:
//   /usr/lib/debug/usr/lib/library.so.debug
if (!file_dir.IsEmpty())
     files.push_back (dirname + file_dir.AsCString() + "/" ++ 
symbol_filename);



This would be in fact the solution to my problem, if it wouldn't use the 
symbol_filename (which contains a uuid, something the *-dbg packages 
don't have).


Well, what do you think?


Cheers,
Stefan





On 05/09/2016 11:23 AM, Pavel Labath wrote:
> You can also use SBCommandInterpreter::HandleCommand, which gives you
> a bit more control over what happens.
>
> good luck with your thesis.
>
> pl
>
> On 9 May 2016 at 10:00, Stefan Kratochwil <kratochwil at teco.edu> wrote:
>> Hi Pavel,
>>
>> thanks for your quick reply. I am currently using a release build, so it may
>> take a moment until I can step through there. I'll report back as soon as
>> possible (my masters thesis somehow depends on that ;) ).
>>
>> I was just about to implement an addon for the API tonight, when I found
>> SBDebugger::HandleCommand("target symbols add ..."). Yeah, it's a little
>> weird way of using an API, and it's a one way command with stdout as reply
>> channel, but it works ^^
>>
>> Cheers,
>> Stefan
>>
>>
>>
>> On 05/09/2016 10:22 AM, Pavel Labath wrote:
>>>
>>> Hi Stefan,
>>>
>>> what you are describing should work out of the box, so the fact that
>>> you are having to add the symbols manually is a bug. I haven't tried
>>> it with the apache binary specifically, but lldb can certainly find
>>> external debug symbols for libc (definitely on ubuntu 14.04, as that's
>>> what I use). We need to figure out what is different in your case. The
>>> most important function here is SymbolVendorELF::CreateInstance. Could
>>> you step through that function, and see why it fails to find the file?
>>>
>>> On 7 May 2016 at 19:45, Stefan Kratochwil via lldb-dev
>>> <lldb-dev at lists.llvm.org> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I need to gather information about variables on the stack frames of a
>>>> given
>>>> program, lets say apache2 (httpd), using the C++ flavour of the lldb
>>>> scripting bridge.
>>>>
>>>> Now, if I attach to the corresponding process id, lldb does not find any
>>>> debug symbols, although I have installed the debug symbols package for
>>>> apache2 (apache2-dbg).
>>>>
>>>> If I use the lldb cli, I have to use
>>>> 'target symbols load /usr/lib/debug/usr/sbin/apache2'
>>>> (or any other path under /usr/lib/debug, depending on the stack frame /
>>>> module I am currently looking at) to load the corresponding debug
>>>> symbols.
>>>>
>>>> However, if I try the same with gdb, then it automagically uses the
>>>> binaries
>>>> under /usr/lib/debug (the non-stripped binaries from any *-dbg package
>>>> are
>>>> located here) for symbolification.
>>>>
>>>>
>>>> Now, there are two questions for me:
>>>>
>>>> 1.: Is there a way to let lldb automagically load debug symbols from
>>>> /usr/lib/debug, if there are any?
>>>>
>>>> 2.: If not, how can I do this manually using the C++ API?
>>>> The SBTarget class does not have suitable methods for this - the closest
>>>> thing is SBTarget#AddModule(), but this gives me a _new_ module, instead
>>>> of
>>>> associating a symbolfile with an existing one. The code path of
>>>> CommandObjectTarget.cpp#AddModuleSymbols() seems not to be reflected in
>>>> the
>>>> scripting bridge, or at least I haven't found it yet.
>>>
>>>
>>> You can use SBDebugger::HandleCommand to execute any command which
>>> does have an API equivalent. It's not ideal, but it works. :)
>>>
>>> cheers,
>>> pl
>>>
>>