[lldb-dev] Writing test cases...

[jingham at apple.com]

I don't mean to pick on Chaoren here, his latest patch just reminded me of this, so I thought I'd repeat it...

We strongly prefer writing test cases using the SB API's rather than the runCmd & expect.  Unless you are actually testing some feature of the command line, please don't write command based tests.  I realize that there are plenty of examples of tests in the test suite that use runCmd where they shouldn't, but don't copy them, copy the plenty that do use the SB API's instead.  

The reason for this is that our policy is that we will maintain compatibility with the SB API's - at least till we decide to do an API-2.0.  But we don't make any similar guarantee about the details of command result format.  If your test is using the command line, it is going to have to check against the command result, and you either end up writing your check pattern by checking as little as possible so you won't be exposed to random changes, in which case you can end up missing some failure, or you test too much and it means irrelevant changes break your tests.

If you use the Python API's it is possible to check all the results you want to check in a very explicit way, which makes the tests much more robust.

Even if you are testing that a command-line command does some specific thing, it is still better in general to use the SB API's to drive to the point where you want to run the test, then use SBInterpreter::HandleCommand to run the command.  You get the full result text from the command in the command return object, and all the part where you are driving the debugger to the point you want to test will be more robust.

BTW, I don't think we've said this anywhere, I just added a note to this effect to the Testsuite README.

Jim