[llvm-bugs] [Bug 30670] New: Consider changing lit to always load suites, then resolve command line arguments

[via llvm-bugs]

https://llvm.org/bugs/show_bug.cgi?id=30670

            Bug ID: 30670
           Summary: Consider changing lit to always load suites, then
                    resolve command line arguments
           Product: Test Suite
           Version: trunk
          Hardware: All
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P
         Component: lit
          Assignee: unassignedbugs at nondot.org
          Reporter: modocache at gmail.com
                CC: daniel at zuster.org, llvm-bugs at lists.llvm.org
    Classification: Unclassified

(This task was copied from
https://reviews.llvm.org/diffusion/L/browse/llvm/tags/RELEASE_390/final/utils/lit/TODO.
It was originally written by Daniel Dunbar.)

Currently we expect each input argument to be a path on disk; we do a recursive
search to find the test suite for each item, but then we only do a local search
based at the input path to find tests. Additionally, for any path that matches
a file on disk we explicitly construct a test instance (bypassing the formats
on discovery implementation).

This has a couple problems:

* The test format doesn't have control over the test instances that result from
file paths.

* It isn't possible to specify virtual tests as inputs. For example, it is not
possible to specify an individual subtest to run with the googletest format.

* The test format doesn't have full control over the discovery of tests in
subdirectories.

Instead, we should move to a model whereby first all of the input specifiers
are resolved to test suites, and then the resolution of the input specifier is
delegated to each test suite. This could take a couple forms:

* We could resolve to test suites, then fully load each test suite, then have a
fixed process to map input specifiers to tests in the test suite (presumably
based on path-in-suite derivations). This has the benefit of being consistent
across all test formats, but the downside of requiring loading the entire test
suite.

* We could delegate all of the resolution of specifiers to the test suite. This
would allow formats that anticipate large test suites to manage their own
resolution for better performance. We could provide a default resolution
strategy that was similar to what we do now (start at subpaths for directories,
but allow the test format control over what happens for individual tests).

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20161012/91543304/attachment.html>