[Lldb-commits] [lldb] c7fc720 - [lldb] Add more documentation on test variants

[Pavel Labath via lldb-commits]

Author: Pavel Labath
Date: 2022-03-18T08:58:29+01:00
New Revision: c7fc7205bb45ee24193c519e9cd2f9f64bdde4ed

URL: https://github.com/llvm/llvm-project/commit/c7fc7205bb45ee24193c519e9cd2f9f64bdde4ed
DIFF: https://github.com/llvm/llvm-project/commit/c7fc7205bb45ee24193c519e9cd2f9f64bdde4ed.diff

LOG: [lldb] Add more documentation on test variants

This formalizes some of the discussion on D121631.

Differential Revision: https://reviews.llvm.org/D121900

Added: 
    

Modified: 
    lldb/docs/resources/test.rst

Removed: 
    


################################################################################
diff  --git a/lldb/docs/resources/test.rst b/lldb/docs/resources/test.rst
index f76800b88f808..258a8083baca8 100644
--- a/lldb/docs/resources/test.rst
+++ b/lldb/docs/resources/test.rst
@@ -185,10 +185,22 @@ of variants. It's very tempting to add more variants because it's an easy way
 to increase test coverage. It doesn't scale. It's easy to set up, but increases
 the runtime of the tests and has a large ongoing cost.
 
-The key take away is that the 
diff erent variants don't obviate the need for
-focused tests. So relying on it to test say DWARF5 is a really bad idea.
-Instead you should write tests that check the specific DWARF5 feature, and have
-the variant as a nice-to-have.
+The test variants are most useful when developing a larger feature (e.g. support
+for a new DWARF version). The test suite contains a large number of fairly
+generic tests, so running the test suite with the feature enabled is a good way
+to gain confidence that you haven't missed an important aspect. However, this
+genericness makes them poor regression tests. Because it's not clear what a
+specific test covers, a random modification to the test case can make it start
+(or stop) testing a completely 
diff erent part of your feature. And since these
+tests tend to look very similar, it's easy for a simple bug to cause hundreds of
+tests to fail in the same way.
+
+For this reason, we recommend using test variants only while developing a new
+feature. This can often be done by running the test suite with 
diff erent
+arguments -- without any modifications to the code. You can create a focused
+test for any bug found that way. Often, there will be many tests failing, but a
+lot of then will have the same root cause.  These tests will be easier to debug
+and will not put undue burden on all other bots and developers.
 
 In conclusion, you'll want to opt for an API test to test the API itself or
 when you need the expressivity, either for the test case itself or for the