[Lldb-commits] [PATCH] D45215: RFC/WIP: Have lit run the lldb test suite

[Pavel Labath via Phabricator via lldb-commits]

labath added a reviewer: zturner.
labath added a comment.

Adding Zachary as he's familiar with lit internals.

I'll try to elaborate more on the approach I had in mind. I would split my approach into a couple of tests.

1. Write a tool which will dump out the list of all tests in the test suite. This could either be a separate python file, or a special flag to dotest.py, whichever is easier. It's implementation would basically be like:

  for each test subdir:
    for each file in subdir:
      if file matches Test*.py:
        test = __import__(file)
        for each class in test:
          for each method in class:
            if method matches "test*":
              print("%s.%s.%s"%(file. class.__name__, method.__name__))

The only tricky part here is the `__import__` line. I think we'll need to do some fiddling with `sys.path` or whatever to get the test files to import correctly. We'll probably need to look at how unittest2 imports them to get it right.

This fiddling is also the reason I think this should be a separate utility. Since the "test discovery" phase happens in the context of the main lit process, which is shared with all other test suites, we should try to avoid messing with it's state too much.

2. have the new test format invoke this utility in its discovery function (getTestsInDirectory)

3. Invoke something like: `['dotest.py'] + common_args + ['-p', TestFile, '-f', testclass + '.' + testmethod`]` in the `execute()` function

The thing to resolve here is to figure out how to translate the result of that command into the test state. Whether we wan't to use just the exit code (this would give us PASS/FAIL states only), do we want to scrape the command output to get the rest of the states (XFAIL, SKIP, ...), or do something else..

I am deliberately bypassing lldb-dotest here, as I think it becomes unnecessary in this setup (so we already remove one layer) -- the common args can be hardcoded by cmake into `lit.site.cfg` which lit will read for us. And since now lit is the one executing the test, you don't have to worry about remembering all the arguments because you can just pass the test name to lit, and it will figure them out for you (we can still print the full dotest line we execute to help debugging).

4. Start ripping out all the multiprocessing/driver logic from dotest.


https://reviews.llvm.org/D45215