[Lldb-commits] [PATCH] D24988: Improvements to testing blacklist

[Todd Fiala via lldb-commits]

tfiala added a comment.

In https://reviews.llvm.org/D24988#559314, @fjricci wrote:

> For an example of something that couldn't be disabled with the original implementation, consider a test like:
>
> `CreateDuringStepTestCase.test_step_inst`
>
> Disabling by method name (`test_step_inst`) would also disable `CreateDuringInstructionStepTestCase.test_step_inst`.


I see what you're saying there.

The part you're missing is that the Test Case class name itself does not have to be unique, either.  i.e. You *can* have two CreateDuringStepTestCase classes in different files.  Nothing uniquifies at that level.

You can convince yourself of this by doing what I just did:

$ cd packages/Python/lldbsuite/test
$ cp -r driver/batch_mode deleteme

now you have tests/deleteme with contents of batch_mode
=======================================================

$ cd deleteme
$ cp TestBatchMode.py TestBatchMode2.py

Now you have two files with the same exact contents, both of which will run just fine.  i.e. you're still not unique with test case names - they can be the same.
=================================================================================================================================================================

edit Makefile to fix the redirection back to the master makefile rules
======================================================================

run the test suite and tell it to run deleteme dir
==================================================

test/dotest.py --test-subdir deleteme

You'll get results like this:

  $ test/dotest.py --executable `pwd`/build/Debug/lldb --test-subdir deleteme
  Testing: 2 test suites, 8 threads
  2 out of 2 test suites processed - TestBatchMode2.py
  ===================
  Test Result Summary
  ===================
  Test Methods:         18
  Reruns:                0
  Success:              18
  Expected Failure:      0
  Failure:               0
  Error:                 0
  Exceptional Exit:      0
  Unexpected Success:    0
  Skip:                  0
  Timeout:               0
  Expected Timeout:      0

That is why I'm saying you need to include the module name, which comes from the filename, or have it be something like FileBaseName:TestCase.test_method.  I have to do this in the test runner architecture for this very reason.  And you will find at least some test cases that are cut and pasted and therefore have duplicate test case names (at least, they used to exist, and nothing enforces them being different in the runner logic).


https://reviews.llvm.org/D24988