[Lldb-commits] [lldb] r232689 - Comment discouraging writing command based test cases.

[Jim Ingham]

Author: jingham
Date: Wed Mar 18 18:16:58 2015
New Revision: 232689

URL: http://llvm.org/viewvc/llvm-project?rev=232689&view=rev
Log:
Comment discouraging writing command based test cases.

Modified:
    lldb/trunk/test/README-TestSuite

Modified: lldb/trunk/test/README-TestSuite
URL: http://llvm.org/viewvc/llvm-project/lldb/trunk/test/README-TestSuite?rev=232689&r1=232688&r2=232689&view=diff
==============================================================================
--- lldb/trunk/test/README-TestSuite (original)
+++ lldb/trunk/test/README-TestSuite Wed Mar 18 18:16:58 2015
@@ -133,3 +133,27 @@ $ DOTEST_PROFILE=YES DOTEST_SCRIPT_DIR=/
   After that, I used the pstats.py module to browse the statistics:
 
 $ python /System/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/pstats.py my.profile 
+
+o Writing test cases:
+
+ We strongly prefer writing test cases using the SB API's rather than the runCmd & expect.  
+ Unless you are actually testing some feature of the command line, please don't write 
+ command based tests.  For historical reasons there are plenty of examples of tests in the 
+ test suite that use runCmd where they shouldn't, but don't copy them, copy the plenty that 
+ do use the SB API's instead.  
+
+ The reason for this is that our policy is that we will maintain compatibility with the 
+ SB API's.  But we don't make any similar guarantee about the details of command result format.  
+ If your test is using the command line, it is going to have to check against the command result
+ text, and you either end up writing your check  pattern by checking as little as possible so 
+ you won't be exposed to random changes in the text; in which case you can end up missing some 
+ failure, or you test too much and it means irrelevant changes break your tests.
+
+ However, if you use the Python API's it is possible to check all the results you want 
+ to check in a very explicit way, which makes the tests much more robust.
+
+ Even if you are testing that a command-line command does some specific thing, it is still 
+ better in general to use the SB API's to drive to the point where you want to run the test, 
+ then use SBInterpreter::HandleCommand to run the command.  You get the full result text 
+ from the command in the command return object, and all the part where you are driving the 
+ debugger to the point you want to test will be more robust.