[lldb-dev] LLDB Evolution

Tue Aug 9 09:08:34 PDT 2016

I fully welcome this move and agree with the timeline. Woohoo.

I also agree with assessment of the situation regarding testing. I
think we're going to need to devote thought to testing if we're going
to move closer towards llvm.

pl

On 9 August 2016 at 16:42, Zachary Turner via lldb-dev
<lldb-dev at lists.llvm.org> wrote:
> There are a lot of reasons for the lack of tests.  Off the top of my head,
> two of the biggest ones are:
>
> 1) Some areas of LLDB have been historically hard to test.  The unwinder and
> core dumps come to mind.  You can't really just check in an 800MB core dump
> into the repo.
> 2) Tests are very heavyweight.  You have to write a Makefile.  You have to
> write a Python script that uses the SB API.  you have to write a C program.
> Then you have to figure out the right incantation of dotest.py to run your
> test.  Once you've done this many times it becomes easier, but the barrier
> to entry is high.
>
> #1 can be improved through the use of unit tests and fuzzing.  Sure, it's
> hard to generate a full blown executable that contains every edge case the
> unwinder might ever experience and then have it try to unwind through there.
> Especially considering that the unwinder behaves differently on every
> platform.  But it's much more manageable to write a unit test that
> constructs a particular sequence of bytes in memory, passes it to the
> unwinder, and checks the return value of some function that is supposed to
> handle that.  It's still not entirely simple, since the unwinder is
> partially heuristic, but at least it's more manageable.
>
> There are also ways we can write IR by hand and have llc generate some byte
> code for us and then pass that to those same functions.  Again, it's not
> like we can just start doing this overnight, but there are ways.  The
> question is just how serious of an effort are we prepared to make and how
> much time are we prepared to put into making this testable versus
> implementing new features, fixing bugs, etc.
>
> #2 could potentially be improved by lit style tests.  As I mentioned in my
> last post, think lldbinline style tests.  Not appropriate for everything,
> but certainly for a lot of things.  If you only had to have one file which
> is a C program with some annotations in the file, that is a much lower
> barrier to entry.
>
> Again, the real question is just how much effort are we actually prepared to
> put into this?  I'd love it if there were entire days or weeks that were
> just testing weeks, where all we did was add new tests (or refactor code to
> make it more testable) and people didn't work on anything else.  I've been
> inactive for a while because I've had to prioritize work on some things in
> LLVM, but I could make time for something like that.
>
> On Mon, Aug 8, 2016 at 6:12 PM Vedant Kumar via lldb-dev
> <lldb-dev at lists.llvm.org> wrote:
>>
>> FWIW, as a happy lldb user, it's exciting to read about these planned
>> developments.
>>
>> I have two follow-up questions about the section on 'Testing Strategy
>> Evaluation': (1) what is lldb's policy on including test updates with bug
>> fix
>> commits and functional changes, and (2) how is this policy enforced?
>>
>> AFAICT, it seems that lldb is not as strict about its test policy as other
>> llvm
>> sub-projects. That could be to its detriment. Here are some very rough
>> numbers
>> on the number of commits which include test updates [1]:
>>
>>   - lldb: 287 of the past 1000 commits
>>   - llvm: 511 of the past 1000 commits
>>   - clang: 622 of the past 1000 commits
>>   - compiler-rt: 543 of the past 1000 commits
>>
>> NFC commits make these numbers a bit noisy. But, unless lldb has a much
>> higher
>> ratio of NFC commits to functional changes as compared to other llvm
>> sub-projects, this is a concerning statistic.
>>
>> best,
>> vedant
>>
>> [1] Based on ToT = r278069.
>>
>> HAS_TEST=0
>> TOTAL=1000
>> for HASH in $(git log --oneline -n$TOTAL | cut -d' ' -f1); do
>>   git show --stat $HASH | grep "|" | grep -q test &&
>> HAS_TEST=$((HAS_TEST+1))
>> done
>> echo $HAS_TEST "/" $TOTAL
>>
>> > On Aug 8, 2016, at 2:57 PM, Zachary Turner via lldb-dev
>> > <lldb-dev at lists.llvm.org> wrote:
>> >
>> >
>> >
>> > On Mon, Aug 8, 2016 at 2:40 PM Kate Stone via lldb-dev
>> > <lldb-dev at lists.llvm.org> wrote:
>> > LLDB has come a long way since the project was first announced.  As a
>> > robust debugger for C-family languages and Swift, LLDB is constantly in use
>> > by millions of developers.  It has also become a foundation for bringing up
>> > debugger support for other languages like Go and RenderScript.  In addition
>> > to the original macOS implementation the Linux LLDB port is in active use
>> > and Windows support has made significant strides.  IDE and editor
>> > integration via both SB APIs and MI have made LLDB available to even more
>> > users.  It’s definitely a project every contributor can be proud of and I’d
>> > like to take a moment to thank everyone who has been involved in one way or
>> > another.
>> >
>> > It’s also a project that shows some signs of strain due to its rapid
>> > growth.  We’ve accumulated some technical debt that must be paid off, and in
>> > general it seems like a good time to reflect on where we'll be headed next.
>> > We’ve outlined a few goals for discussion below as well as one more
>> > short-term action.  Discussion is very much encouraged.
>> >
>> > Forward-Looking Goals
>> >
>> >    1. Testing Strategy Evaluation
>> >
>> > Keeping our code base healthy is next to impossible without a robust
>> > testing strategy.  Our existing suite of tests is straightforward to run
>> > locally, and serves as a foundation for continuous integration.  That said,
>> > it is definitely not exhaustive.  Obvious priorities for improvement include
>> > gathering coverage information, investing in more conventional unit tests in
>> > addition to the suite of end-to-end tests, and introducing tests in code
>> > bases where we depend on debugger-specific behavior (e.g.: for expression
>> > evaluation.)
>> > I know this is going to be controversial, but I think we should at least
>> > do a serious evaluation of whether using the lit infrastructure would work
>> > for LLDB.  Conventional wisdom is that it won't work because LLDB tests are
>> > fundamentally different than LLVM tests.  I actually completely agree with
>> > the latter part.  They are fundamentally different.
>> >
>> > However, we've seen some effort to move towards lldb inline tests, and
>> > in a sense that's conceptually exactly what lit tests are.  My concern is
>> > that nobody with experience working on LLDB has a sufficient understanding
>> > of what lit is capable of to really figure this out.
>> >
>> > I know when I mentioned this some months ago Jonathan Roelofs chimed in
>> > and said that he believes lit is extensible enough to support LLDB's use
>> > case.  The argument -- if I remember it correctly -- is that the traditional
>> > view of what a lit test (i.e. a sequence of commands that checks the output
>> > of a program against expected output) is one particular implementation of a
>> > lit-style test.  But that you can make your own which do whatever you want.
>> >
>> > This would not just be busy work either.  I think everyone involved with
>> > LLDB has experienced flakiness in the test suite.  Sometimes it's flakiness
>> > in LLDB itself, but sometimes it is flakiness in the test infrastructure.
>> > It would be nice to completely eliminate one source of flakiness.
>> >
>> > I think it would be worth having some LLDB experts sit down in person
>> > with some lit experts and brainstorm ways to make LLDB use lit.
>> >
>> > Certainly it's worth a serious look, even if nothing comes of it.
>> >
>> >
>> >    4. Good Citizenship in the LLVM Community
>> >
>> > Last, but definitely not least, LLDB should endeavor to be a good
>> > citizen of the LLVM community.  We should encourage developers to think of
>> > the technology stack as a coherent effort, where common code should be
>> > introduced at an appropriate level in the stack.  Opportunities to factor
>> > reusable aspects of the LLDB code base up the stack into LLVM will be
>> > pursued.
>> >
>> > One arbitrary source of inconsistency at present is LLDB’s coding
>> > standard.  That brings us to…
>> >
>> > Near-Term Goal: Standardizing on LLVM-style clang-format Rules
>> >
>> > We’ve heard from several in the community that would prefer to have a
>> > single code formatting style to further unify the two code bases.  Using
>> > clang-format with the default LLVM conventions would simplify code
>> > migration, editor configuration, and coding habits for developers who work
>> > in multiple LLVM projects.  There are non-trivial implications to
>> > reformatting a code base with this much history.  It can obfuscate history
>> > and impact downstream projects by complicating merges.  Ideally, it should
>> > be done once with as much advance notice as is practical.  Here’s the
>> > timeline we’re proposing:
>> >
>> > Today - mechanical reformatting proposed, comment period begins
>> >
>> > To get a preview of what straightforward reformatting of the code looks
>> > like, just follow these steps to get a clean copy of the repository and
>> > reformat it:
>> >       • Check out a clean copy of the existing repository
>> >       • Edit .clang-format in the root of the tree, remove all but the
>> > line “BasedOnStyle: LLVM”
>> >       • Change your current working directory to the root of the tree to
>> > reformat
>> >       • Double-check to make sure you did step 3 ;-)
>> >       • Run the following shell command: "find . -name "*.[c,cpp,h]
>> > -exec clang-format -i {} +"
>> > Very excited about this one, personally.  While I have my share of
>> > qualms with LLVM's style, the benefit of having consistency is hard to
>> > overstate.  It greatly reduces the effort to switch between codebases, a
>> > direct consequence of which is that it encourages people with LLVM expertise
>> > to jump into the LLDB codebase, which hopefully can help to tear down the
>> > invisible wall between the two.
>> >
>> > As a personal aside, this allows me to go back to my normal workflow of
>> > having 3 edit source files opened simultaneously and tiled horizontally,
>> > which is very nice.
>> >
>> > _______________________________________________
>> > lldb-dev mailing list
>> > lldb-dev at lists.llvm.org
>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>