[Lldb-commits] [PATCH] D81697: Add support for batch-testing to the LLDB testsuite.

[Pavel Labath via Phabricator via lldb-commits]

labath added a comment.

it looks like the official python unittest framework has grown support of a subtest <https://docs.python.org/3/library/unittest.html#subtests> feature, which seems to be very similar to what we want here. It may make more sense to update to a newer framework rather than patching the copy that we have. Even if can't update for some reason, it would be good to do something equivalent to that so that it is easier to update in the future.

The part I like about the upstream solution is that it provides scoping for the failures -- so a single failure terminates a "subtest", but still lets the whole test continue. The reason this is important is when you have more complex "assertion" functions consisting of multiple smaller assertions. A typical example would be something like:

  def check_foo(foo):
    assertGreater(len(foo), 2)
    assertEquals(foo[0], 47)
    assertEquals(foo[1], 42)
    ...
  
  def test():
    with self.batchTest():
      check_foo(get_one_foo())
      check_foo(get_another_foo())
      ...

This patch essentially makes all assertions non-fatal, which means that if a first assertion in `check_foo` fails, the function will still continue and spew a bunch of uninteresting errors. With subtests, that won't happen -- the first assertion will abort the rest of the `check_foo` function, but it will let the other call continue (if it's in a different subtest):

  with self.subtest(): check_foo(get_one_foo())
  with self.subtest(): check_foo(get_another_foo())

This can result in a bit more typing, but that can be resolved by adding some extra sugar on top of that.For example, since the typical use case for this will be to run a bunch of independent expression (or frame var) commands, we could make a special version of the `expect_expr` command, which runs the expression in a subtest. (Or that might even be the default.)


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D81697/new/

https://reviews.llvm.org/D81697