[Lldb-commits] [PATCH] D121900: [lldb] Add more documentation on test variants

[Pavel Labath via Phabricator via lldb-commits]

This revision was automatically updated to reflect the committed changes.
Closed by commit rGc7fc7205bb45: [lldb] Add more documentation on test variants (authored by labath).

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D121900/new/

https://reviews.llvm.org/D121900

Files:
  lldb/docs/resources/test.rst


Index: lldb/docs/resources/test.rst
===================================================================
--- lldb/docs/resources/test.rst
+++ lldb/docs/resources/test.rst
@@ -185,10 +185,22 @@
 to increase test coverage. It doesn't scale. It's easy to set up, but increases
 the runtime of the tests and has a large ongoing cost.
 
-The key take away is that the different variants don't obviate the need for
-focused tests. So relying on it to test say DWARF5 is a really bad idea.
-Instead you should write tests that check the specific DWARF5 feature, and have
-the variant as a nice-to-have.
+The test variants are most useful when developing a larger feature (e.g. support
+for a new DWARF version). The test suite contains a large number of fairly
+generic tests, so running the test suite with the feature enabled is a good way
+to gain confidence that you haven't missed an important aspect. However, this
+genericness makes them poor regression tests. Because it's not clear what a
+specific test covers, a random modification to the test case can make it start
+(or stop) testing a completely different part of your feature. And since these
+tests tend to look very similar, it's easy for a simple bug to cause hundreds of
+tests to fail in the same way.
+
+For this reason, we recommend using test variants only while developing a new
+feature. This can often be done by running the test suite with different
+arguments -- without any modifications to the code. You can create a focused
+test for any bug found that way. Often, there will be many tests failing, but a
+lot of then will have the same root cause.  These tests will be easier to debug
+and will not put undue burden on all other bots and developers.
 
 In conclusion, you'll want to opt for an API test to test the API itself or
 when you need the expressivity, either for the test case itself or for the


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D121900.416415.patch
Type: text/x-patch
Size: 1879 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/lldb-commits/attachments/20220318/6bd20165/attachment.bin>