[lldb-dev] [RFC] Testsuite in lldb & possible future directions

[Davide Italiano via lldb-dev]

Hi,
in the last couple of months a lot of people put a lot of attentions
and energy into lldb and we're starting to see the first results. I
decided to sit down and write this e-mail to state where we are and
what are some possible future directions for the projects, in terms of
better quality/higher testability.

Current state:

1) We got the testsuite on MacOS to build with zero unexpected
successes and zero failures (modulo one change I'm going to push
tomorrow). This is a collaborative effort and it's very important
because it allows us to push for unexpected successes as failures on
the bots, allowing us to discover issues quicker. Other platform I
think are improving their state as well, mainly thanks to the work of
Pavel and Jan.

2) Adrian is pushing some changes that will allow us to build tests
out of tree. This is particularly good because it doesn't matter the
build directly and indirectly makes the testsuite more reliable as you
can just wipe the build directory when you're done and retry. It also
opens the path for new improvements, e.g. sharing a module cache
across all tests instead of creating a module cache per-test.

3) Vedant spent a lot of time fixing modules issues & ubsan/asan
failures, which. again, improved reliability.

Future:
1) I'm currently exploring ways of testing more akin to what llvm
does. We already have a bunch of unit tests, and I'm considering to
leverage `lldb-test` to experiment testing parts of the compilers that
don't really need interactivity (e.g. expression
parsing/autocomplete). I'm not sure whether this will lead to
anything, but if we can make the process easier for developers coming
from other bits of LLVM, I think we should (at least we should give it
a try).

2) Now that we raised the quality of our standards (we've been with
green bots for a while), I'd love to keep some time and focus on the
standards for future commits.
I think we should adhere to the LLVM policy here
https://llvm.org/docs/DeveloperPolicy.html#quality
So, commits that break tests/break the build/don't include a test
might be reverted. In addition, commits with design decisions not
previously discussed might be reverted if there are concerns about the
general direction.

3) In the short term I plan to remove support for unmaintained
languages (Java/Go/Swift). This allows us to bring them back again (or
bring new languages) but have a better plan for testability &
maintainability.

I'm pretty sure other people have ideas on what they're working
on/want to push/do next, so I'll let them speak.

Conclusions:
The reliability of the suite (and the health of the codebase) is very
important to us. If you have issues, let us know.
In general, I'm looking for any kind of feedback, feel free to speak!

Thanks,

--
Davide