[Lldb-commits] [PATCH] D46005: [test] Add a utility for dumping all tests in the dotest suite

[Zachary Turner via Phabricator via lldb-commits]

zturner added a comment.

In https://reviews.llvm.org/D46005#1077013, @zturner wrote:

> In https://reviews.llvm.org/D46005#1077000, @labath wrote:
>
> > In https://reviews.llvm.org/D46005#1076978, @zturner wrote:
> >
> > > Does it print just the names of the .py files, or does it print the actual test classes and methods in the Class.TestName format, evaluate XFAIL and SKIPs, etc?
> >
> >
> > It prints everything: test file (including the relative path in the test suite), test class name and test method name.
> >
> > It does not try to evaluate skips or xfails, but that was never the intention. The idea is to feed this information to lit (as it needs to know about the set of all tests), which can then run them with a bigger (smaller?) granularity. The interpretation of the test results (skip,fail, ...) will still be done by the lit test format, only now it could do it at a test-method level instead of file-level.
>
>
> I thought the intention was going to be parallelize at file-granularity rather than method granularity, since the whole point of grouping tests together into classes is that they can share similar set up and tear down which may be expensive to perform multiple times.  If we're going to parallelize at method-granularity, I would rather have one .py file per method.


Note that there's currently no precedent (that i'm aware of anwyay) in LLVM or any of its subprojects for splitting the running of a single file into multiple parallel jobs.  All of LLVM's other projects parallelize at file granularity, and so it's up to to the person writing tests to ensure that the file granularity matches that with which they want tests to be parallelized at.

That doesn't mean it's not possible (as you've shown here), but it adds some additional complexity and I'm not sure it's worth it.

There are currently two use cases I can come up with for grouping lots of tests together in a single class.

1. Sharing common set up and tear down code that is expensive.

2. Sharing common utility functions that all test cases need to use.

For the first case, we don't *want* to parallelize at a finer granularity because it will cause extra work to be done.

And for the second case, we can solve this by having a file in a directory named `util.py` and writing `from . import util` at the top of each test case file.  Then have the lit config skip files named `util.py` in its discovery.


https://reviews.llvm.org/D46005