[Lldb-commits] [PATCH] D121900: [lldb] Add more documentation on test variants

[Pavel Labath via Phabricator via lldb-commits]

labath created this revision.
labath added reviewers: JDevlieghere, clayborg, jingham.
Herald added a project: All.
labath requested review of this revision.
Herald added a project: LLDB.

This formalizes some of the discussion on D121631 <https://reviews.llvm.org/D121631>.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D121900

Files:
  lldb/docs/resources/test.rst


Index: lldb/docs/resources/test.rst
===================================================================
--- lldb/docs/resources/test.rst
+++ lldb/docs/resources/test.rst
@@ -185,10 +185,22 @@
 to increase test coverage. It doesn't scale. It's easy to set up, but increases
 the runtime of the tests and has a large ongoing cost.
 
-The key take away is that the different variants don't obviate the need for
-focused tests. So relying on it to test say DWARF5 is a really bad idea.
-Instead you should write tests that check the specific DWARF5 feature, and have
-the variant as a nice-to-have.
+The test variants are most useful when developing a larger feature (e.g. support
+for a new DWARF version). The test suite contains a large number of fairly
+generic tests, so running the test suite with the feature enabled is a good way
+to gain confidence that you haven't missed an important aspect. However, this
+genericness makes them poor regression tests. Because it's not clear what a
+specific test covers, a random modification to the test case can make it start
+(or stop) testing a completely different part of your feature. And since these
+tests tend to look very similar, it's easy for a simple bug to cause hundreds of
+tests to fail in the same way.
+
+For this reason, we recommend using test variants only while developing a new
+feature. This can often be done by running the test suite with different
+arguments -- without any modifications to the code. You can create a focused
+test for any bug found that way. Often, there will be many tests failing, but a
+lot of then will have the same root cause.  These tests will be easier to debug
+and will not put undue burden on all other bots and developers.
 
 In conclusion, you'll want to opt for an API test to test the API itself or
 when you need the expressivity, either for the test case itself or for the


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D121900.416128.patch
Type: text/x-patch
Size: 1879 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/lldb-commits/attachments/20220317/d1c193c1/attachment.bin>