[lldb-dev] LLDB Evolution

Tue Aug 9 08:42:26 PDT 2016

There are a lot of reasons for the lack of tests.  Off the top of my head,
two of the biggest ones are:

1) Some areas of LLDB have been historically hard to test.  The unwinder
and core dumps come to mind.  You can't really just check in an 800MB core
dump into the repo.
2) Tests are very heavyweight.  You have to write a Makefile.  You have to
write a Python script that uses the SB API.  you have to write a C
program.  Then you have to figure out the right incantation of dotest.py to
run your test.  Once you've done this many times it becomes easier, but the
barrier to entry is high.

#1 can be improved through the use of unit tests and fuzzing.  Sure, it's
hard to generate a full blown executable that contains every edge case the
unwinder might ever experience and then have it try to unwind through
there.  Especially considering that the unwinder behaves differently on
every platform.  But it's much more manageable to write a unit test that
constructs a particular sequence of bytes in memory, passes it to the
unwinder, and checks the return value of some function that is supposed to
handle that.  It's still not entirely simple, since the unwinder is
partially heuristic, but at least it's more manageable.

There are also ways we can write IR by hand and have llc generate some byte
code for us and then pass that to those same functions.  Again, it's not
like we can just start doing this overnight, but there are ways.  The
question is just how serious of an effort are we prepared to make and how
much time are we prepared to put into making this testable versus
implementing new features, fixing bugs, etc.

#2 could potentially be improved by lit style tests.  As I mentioned in my
last post, think lldbinline style tests.  Not appropriate for everything,
but certainly for a lot of things.  If you only had to have one file which
is a C program with some annotations in the file, that is a much lower
barrier to entry.

Again, the real question is just how much effort are we actually prepared
to put into this?  I'd love it if there were entire days or weeks that were
just testing weeks, where all we did was add new tests (or refactor code to
make it more testable) and people didn't work on anything else.  I've been
inactive for a while because I've had to prioritize work on some things in
LLVM, but I could make time for something like that.

On Mon, Aug 8, 2016 at 6:12 PM Vedant Kumar via lldb-dev <
lldb-dev at lists.llvm.org> wrote:

> FWIW, as a happy lldb user, it's exciting to read about these planned
> developments.
>
> I have two follow-up questions about the section on 'Testing Strategy
> Evaluation': (1) what is lldb's policy on including test updates with bug
> fix
> commits and functional changes, and (2) how is this policy enforced?
>
> AFAICT, it seems that lldb is not as strict about its test policy as other
> llvm
> sub-projects. That could be to its detriment. Here are some very rough
> numbers
> on the number of commits which include test updates [1]:
>
>   - lldb: 287 of the past 1000 commits
>   - llvm: 511 of the past 1000 commits
>   - clang: 622 of the past 1000 commits
>   - compiler-rt: 543 of the past 1000 commits
>
> NFC commits make these numbers a bit noisy. But, unless lldb has a much
> higher
> ratio of NFC commits to functional changes as compared to other llvm
> sub-projects, this is a concerning statistic.
>
> best,
> vedant
>
> [1] Based on ToT = r278069.
>
> HAS_TEST=0
> TOTAL=1000
> for HASH in $(git log --oneline -n$TOTAL | cut -d' ' -f1); do
>   git show --stat $HASH | grep "|" | grep -q test &&
> HAS_TEST=$((HAS_TEST+1))
> done
> echo $HAS_TEST "/" $TOTAL
>
> > On Aug 8, 2016, at 2:57 PM, Zachary Turner via lldb-dev <
> lldb-dev at lists.llvm.org> wrote:
> >
> >
> >
> > On Mon, Aug 8, 2016 at 2:40 PM Kate Stone via lldb-dev <
> lldb-dev at lists.llvm.org> wrote:
> > LLDB has come a long way since the project was first announced.  As a
> robust debugger for C-family languages and Swift, LLDB is constantly in use
> by millions of developers.  It has also become a foundation for bringing up
> debugger support for other languages like Go and RenderScript.  In addition
> to the original macOS implementation the Linux LLDB port is in active use
> and Windows support has made significant strides.  IDE and editor
> integration via both SB APIs and MI have made LLDB available to even more
> users.  It’s definitely a project every contributor can be proud of and I’d
> like to take a moment to thank everyone who has been involved in one way or
> another.
> >
> > It’s also a project that shows some signs of strain due to its rapid
> growth.  We’ve accumulated some technical debt that must be paid off, and
> in general it seems like a good time to reflect on where we'll be headed
> next.  We’ve outlined a few goals for discussion below as well as one more
> short-term action.  Discussion is very much encouraged.
> >
> > Forward-Looking Goals
> >
> >    1. Testing Strategy Evaluation
> >
> > Keeping our code base healthy is next to impossible without a robust
> testing strategy.  Our existing suite of tests is straightforward to run
> locally, and serves as a foundation for continuous integration.  That said,
> it is definitely not exhaustive.  Obvious priorities for improvement
> include gathering coverage information, investing in more conventional unit
> tests in addition to the suite of end-to-end tests, and introducing tests
> in code bases where we depend on debugger-specific behavior (e.g.: for
> expression evaluation.)
> > I know this is going to be controversial, but I think we should at least
> do a serious evaluation of whether using the lit infrastructure would work
> for LLDB.  Conventional wisdom is that it won't work because LLDB tests are
> fundamentally different than LLVM tests.  I actually completely agree with
> the latter part.  They are fundamentally different.
> >
> > However, we've seen some effort to move towards lldb inline tests, and
> in a sense that's conceptually exactly what lit tests are.  My concern is
> that nobody with experience working on LLDB has a sufficient understanding
> of what lit is capable of to really figure this out.
> >
> > I know when I mentioned this some months ago Jonathan Roelofs chimed in
> and said that he believes lit is extensible enough to support LLDB's use
> case.  The argument -- if I remember it correctly -- is that the
> traditional view of what a lit test (i.e. a sequence of commands that
> checks the output of a program against expected output) is one particular
> implementation of a lit-style test.  But that you can make your own which
> do whatever you want.
> >
> > This would not just be busy work either.  I think everyone involved with
> LLDB has experienced flakiness in the test suite.  Sometimes it's flakiness
> in LLDB itself, but sometimes it is flakiness in the test infrastructure.
> It would be nice to completely eliminate one source of flakiness.
> >
> > I think it would be worth having some LLDB experts sit down in person
> with some lit experts and brainstorm ways to make LLDB use lit.
> >
> > Certainly it's worth a serious look, even if nothing comes of it.
> >
> >
> >    4. Good Citizenship in the LLVM Community
> >
> > Last, but definitely not least, LLDB should endeavor to be a good
> citizen of the LLVM community.  We should encourage developers to think of
> the technology stack as a coherent effort, where common code should be
> introduced at an appropriate level in the stack.  Opportunities to factor
> reusable aspects of the LLDB code base up the stack into LLVM will be
> pursued.
> >
> > One arbitrary source of inconsistency at present is LLDB’s coding
> standard.  That brings us to…
> >
> > Near-Term Goal: Standardizing on LLVM-style clang-format Rules
> >
> > We’ve heard from several in the community that would prefer to have a
> single code formatting style to further unify the two code bases.  Using
> clang-format with the default LLVM conventions would simplify code
> migration, editor configuration, and coding habits for developers who work
> in multiple LLVM projects.  There are non-trivial implications to
> reformatting a code base with this much history.  It can obfuscate history
> and impact downstream projects by complicating merges.  Ideally, it should
> be done once with as much advance notice as is practical.  Here’s the
> timeline we’re proposing:
> >
> > Today - mechanical reformatting proposed, comment period begins
> >
> > To get a preview of what straightforward reformatting of the code looks
> like, just follow these steps to get a clean copy of the repository and
> reformat it:
> >       • Check out a clean copy of the existing repository
> >       • Edit .clang-format in the root of the tree, remove all but the
> line “BasedOnStyle: LLVM”
> >       • Change your current working directory to the root of the tree to
> reformat
> >       • Double-check to make sure you did step 3 ;-)
> >       • Run the following shell command: "find . -name "*.[c,cpp,h]
> -exec clang-format -i {} +"
> > Very excited about this one, personally.  While I have my share of
> qualms with LLVM's style, the benefit of having consistency is hard to
> overstate.  It greatly reduces the effort to switch between codebases, a
> direct consequence of which is that it encourages people with LLVM
> expertise to jump into the LLDB codebase, which hopefully can help to tear
> down the invisible wall between the two.
> >
> > As a personal aside, this allows me to go back to my normal workflow of
> having 3 edit source files opened simultaneously and tiled horizontally,
> which is very nice.
> >
> > _______________________________________________
> > lldb-dev mailing list
> > lldb-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20160809/15abd22a/attachment-0001.html>