[Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[Pavel Labath via lldb-commits]

On 25/01/2021 05:30, David Blaikie wrote:
> 
> 
> On Fri, Jan 22, 2021 at 5:37 AM Pavel Labath <pavel at labath.sk 
> <mailto:pavel at labath.sk>> wrote:
> 
>     On 19/01/2021 23:23, David Blaikie wrote:
>      > On Tue, Jan 19, 2021 at 1:12 AM Pavel Labath <pavel at labath.sk
>     <mailto:pavel at labath.sk>> wrote:
>      > Yeah - I have mixed feelings about debugger testing here - it is nice
>      > to have end-to-end tests which makes for handy debugger testing
>      > (though I guess in theory, debuginfo-tests is the place for
>      > intentional end-to-end testing), though also being able to test
>      > different features (DWARF version, split DWARF, dsym V object
>      > debugging, etc) when they're written as end-to-end tests.
> 
>     Yeah, it would be nice if there was a clearer separation between the
>     two
>     categories. The current setup has evolved organically, as the
>     end-to-end
>     API tests used to be the only kinds of tests.
> 
> 
>      >
>      > Can we write non-end-to-end API tests, then?
> 
>     Kind of. There is no fundamental reason why one couldn't run llvm-mc or
>     whatever as a part of an API test. The main issue is that we don't have
>     the infrastructure for that set up right now. I think the reason for
>     that is that once you start dealing with "incomplete" executables which
>     cannot be run on the host platform, the usefulness of interactivity
>     goes
>     down sharply. It is hard for such a test to do something other than
>     load
>     up some executable and query its state. This is a perfect use case
>     for a
>     shell test.
> 
>     There are exceptions though. For example we have a collection of "API"
>     tests which test the gdb-remote communication layer, by mocking one end
>     of the connection. Such tests are necessarily interactive, which is why
>     they ended up in the API category, but they are definitely not
>     end-to-end tests, and they either don't use any executables, or just
>     use
>     a static yaml2objed executable. This is why our API tests have the
>     ability to run yaml2obj and one could add other llvm tools in a similar
>     fashion.
> 
> 
> Though then you get the drawback that appears to be Jim's original/early 
> motivation: Avoiding checking the textual output of the debugger as much 
> as possible (so that updating that output doesn't involve changing every 
> single test case - which can cause stagnation due to it being too 
> expensive to update all the tests, so improvements to the output format 
> are not made). Which I have some sympathy for - though LLVM and clang 
> seem to get by without this to some degree - we do/have significantly 
> changed the LLVM IR format, for instance, and done regex updates to the 
> FileCheck/lit tests to account for that - Clang, well, if we changed the 
> format of error messages we might have a bad time, except that we rarely 
> test the literal warning/error text, instead using clang's testing 
> functionality with the // expected-error, etc, syntax.

Yeah, I am aware of Jim's objections to this. :) I haven't been around 
here as long as he has, and I definitely don't, and definitely haven't 
been around to witness the gdb format change attempt, but with the shell 
tests we have, I haven't seen any indication that they inhibit the 
velocity of lldb development (we haven't done any major format recently 
though -- but if we did, I don't think the shell tests would be a big 
problem). So I tend to not put so much emphasis on this aspect, and 
choose to prioritise other aspects of tests (e.g. ease of writing).

With different priorities it's not surprising (though still unfortunate) 
that we reach different conclusions.

> 
> I guess lldb doesn't have a machine readable form, like gdb's machine 
> interface that might make for a more robust thing to test against most 
> of the time (& then leaving a limited number of tests that test the 
> user-textual output)? Instead the python API is the machine interface?

Fun fact: We used to have an lldb-mi tool in the repository, which 
implemented the gdb-mi interface on top of lldb. Although the 
implementation of the tool itself was bad, the main reason why the tool 
was removed was that its tests were horribly flaky and hard to 
understand&maintain.


> 
>     Another aspect of end-to-endness is being able to test a specific
>     component of lldb, instead of just the debugger as a whole. Here the
>     API
>     tests cannot help because the "API" is the lldb public API.
> 
> 
> Not sure I followed here - you mean the API tests aren't more narrowly 
> targeted than the Shell tests, because the API is the public API, so 
> it's mostly/pretty close to what you can interact with from the Shell 
> anyway - doesn't give you lower-level access akin to unit testing? Fair 
> enough.

What I wanted to say here is that the Shell tests can (and some, do) go 
lower level than the lldb public API, though the "lldb-test" tool. It 
sits below the public api, and exposes some useful features of the 
underlying apis. The idea was for it to be the compromise solution for 
the "have a stringly matchable interface but still be free to change 
lldb's text output", by separating the interface used for testing from 
the one that the user uses.

It hasn't exactly taken the world by storm, I would say due to a 
combination of two factors:
- lldb is very monolithic, so it is hard to expose/test just one 
component without creating/mocking the whole world
- lldb already has a bunch of low-level commands (like the image lookup 
thingy) that can already be used to get to most of the things you'd need.

I like how the object-file subcommand of lldb-test (the first use case) 
has worked out and I think it brings pretty good value for money. The 
others, I fear, are failed experiments, for one reason or another.

pl