[Lldb-commits] [lldb] r232689 - Comment discouraging writing command based test cases.

[Zachary Turner]

+1 and thanks

On Wed, Mar 18, 2015 at 4:16 PM, Jim Ingham <jingham at apple.com> wrote:

> Author: jingham
> Date: Wed Mar 18 18:16:58 2015
> New Revision: 232689
>
> URL: http://llvm.org/viewvc/llvm-project?rev=232689&view=rev
> Log:
> Comment discouraging writing command based test cases.
>
> Modified:
>     lldb/trunk/test/README-TestSuite
>
> Modified: lldb/trunk/test/README-TestSuite
> URL:
> http://llvm.org/viewvc/llvm-project/lldb/trunk/test/README-TestSuite?rev=232689&r1=232688&r2=232689&view=diff
>
> ==============================================================================
> --- lldb/trunk/test/README-TestSuite (original)
> +++ lldb/trunk/test/README-TestSuite Wed Mar 18 18:16:58 2015
> @@ -133,3 +133,27 @@ $ DOTEST_PROFILE=YES DOTEST_SCRIPT_DIR=/
>    After that, I used the pstats.py module to browse the statistics:
>
>  $ python
> /System/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/pstats.py
> my.profile
> +
> +o Writing test cases:
> +
> + We strongly prefer writing test cases using the SB API's rather than the
> runCmd & expect.
> + Unless you are actually testing some feature of the command line, please
> don't write
> + command based tests.  For historical reasons there are plenty of
> examples of tests in the
> + test suite that use runCmd where they shouldn't, but don't copy them,
> copy the plenty that
> + do use the SB API's instead.
> +
> + The reason for this is that our policy is that we will maintain
> compatibility with the
> + SB API's.  But we don't make any similar guarantee about the details of
> command result format.
> + If your test is using the command line, it is going to have to check
> against the command result
> + text, and you either end up writing your check  pattern by checking as
> little as possible so
> + you won't be exposed to random changes in the text; in which case you
> can end up missing some
> + failure, or you test too much and it means irrelevant changes break your
> tests.
> +
> + However, if you use the Python API's it is possible to check all the
> results you want
> + to check in a very explicit way, which makes the tests much more robust.
> +
> + Even if you are testing that a command-line command does some specific
> thing, it is still
> + better in general to use the SB API's to drive to the point where you
> want to run the test,
> + then use SBInterpreter::HandleCommand to run the command.  You get the
> full result text
> + from the command in the command return object, and all the part where
> you are driving the
> + debugger to the point you want to test will be more robust.
>
>
> _______________________________________________
> lldb-commits mailing list
> lldb-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/lldb-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-commits/attachments/20150318/284bc7c1/attachment.html>