[lldb-dev] More tests

[Zachary Turner]

The problem with core files is that the test coverage is reduced.  If a
particular platform doesn't understand a core file, the test won't get
run.  You could argue that that's fine, because unwinding varies between
platforms too, but broader test coverage is always a good thing.  For
example, what if there's no buildbot set up for a platform, then you have 0
automated test coverage for that codepath.

If we use LLVM's assembler to just produce an object file with the correct
backend, then I believe this could tested without a core file.  Broader
coverage.  Of course, exposing unwind information through the SB API is
definitely a great idea, and would open up a lot more possibilities for
heavier tests.  We should definitely do that also.  But I don't think it's
a pure substitute.  Having broader test coverage means that, for example,
if 3 bots are red because of some unrelated breaks, but 2 bots are green,
any of those 2 can pick up failures even if the path being tested only ever
occurs in practice on a specific architecture.

So in short: I agree that core files are the best way to test the full
stack, but I think it would be great if we can come up with a way to break
this down into separate components and figure out really what absolutely
can't be tested without running the bits of code on a particular
architecture, and what can we abstract away to produce smaller, lighter
weight tests that can give us broader coverage.

On Thu Feb 19 2015 at 2:50:56 PM Greg Clayton <gclayton at apple.com> wrote:

>
> > On Feb 19, 2015, at 10:33 AM, Zachary Turner <zturner at google.com> wrote:
> >
> > Yesterday a change went up for review to add support for mips64 in the
> unwinder.  In the review, I proposed adding some tests.  There was some
> discussion in the thread, but I don't want to pollute someone's review
> thread with a long disucssion, and I thought maybe more people would be
> interested.  So I'm moving it over here.
> >
> > For some context, I proposed the idea of testing this type of thing via
> unit tests.  I certainly thing that do a full live-system debugging test
> would be useful for this type of thing, but I think this is a great example
> of where a unit test would provide a lot of value.  For starters, the full
> scale tests are very heavyweight.  There's a lot of setup involved, they're
> slow, they're platform specific.
> >
> > I think for tests like this we can write them as UnitTests in a way that
> they're faster, lighter weight, more targeted, and platform independent.
> One of the issues brought up in the commits thread linked above is that the
> test would only be able to run on mips hardware.  I disagree.  The test
> could come with a file containing hand-written assembly.  We can use the
> LLVM assembler to target mips, which will generate mips bytecode no matter
> which platform you're on.  Then we can just feed the bytes to the unwinder.
> >
> > Of course, you need a RegisterContext.  We would probably have to make
> some changes to the unwinder so that the RegisterContext could be
> abstracted out.  This way, a test could provide its own mock register
> context and target memory reader.
> >
> > You still need a full-scale integration test to confirm that unwinding
> actually works in context.  But I think we should consider unit tests for
> things like this.  It's difficult to write unit tests right now because the
> code in many places is not well testable (for example, not being able to
> abstract the registercontext so that a test could provide its own).
> >
> > But I want to just toss this out there and see how much pushback I get.
> If it's something people are interested in, then it's something we could
> work towards gradually.  I think we can all agree that checking in code
> with 0 tests sucks, so hopefully we can work towards something that gets us
> into a better state.  Getting more full-scale public API tests is one angle
> we can approach it from.  But I think unit tests would be some nice
> additional ammo we could throw at the problem, if we work towards better
> abstractions and more testable code.
> >
> > Thoughts?
>
> For the unwinder we could just test that certain assembly code produces
> the expected unwind rows. This would involve exposing the unwinder through
> the lldb::SB API. This would help test that the correct rows are being
> produced for exact code.
>
> I think exposing unwinding info through the lldb::SB API is a great idea,
> though we really need to make sure the right API is exposed.
>
> Core files are really the best way to test if unwinding works as expected.
> I am guessing it wouldn't be too hard to make small core files that test
> one specific unwinding scenario. Maybe a python script could created that
> takes an assembly file and a triple and it produces an ELF or MachO core
> file with a specified or default register context? As long as the core
> files are small they could be checked in as tests. Full blown actual core
> files are way too big and will cause major problems in a GIT repository if
> they get changed, but small targeted core files could be small enough that
> they don't require changes.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150219/e94437a2/attachment.html>