[PATCH] D40077: [RFC][LNT][test-suite] - Allow 1 test to report multiple individual test results

[Matthias Braun via llvm-commits]

I was thinking somewhat like this:

- Keep lit test discovery the same (so no changes to worker_run_one_test(), lits progress etc.).
- Instead extend `class Result` in lit to support a notion of a "subtest" or "subresult". So I guess look similar to:
      result = lit.Test.Result(...)
      subresult_test0 = result.addSubResult('subtest0')
      subresult.addMetric('time', 23)
      # ...
      subresult_test1 = result.addSubResult('subtest1')
      subresult.addMetric('time', 40)

And then modify write_test_results() to expand a test with subresults into multiple tests in the json output.

- Matthias

> On Dec 14, 2017, at 11:18 AM, Brian Homerding via Phabricator via llvm-commits <llvm-commits at lists.llvm.org> wrote:
> 
> homerdin added a comment.
> 
> @MatzeB Thanks for your feedback, its good to learn what some other use cases are.  Sorry about the slow response.  I've looked into adding this functionality to the lit tool.  It looks like we would have to have the `worker_run_one_test( )` return a collection of test objects or possible be turned into a generator.  Then provide some logic to add the extra tests to the main thread's test list.  This would have the side effect of producing more results than test discovery had found `PASS: test-suite :: xyz.test (5 of 3)`.
> 
> Do you know if the `--max-tests` option is used and in what context? ( Max number of executables to run or max number of results desired )
> 
> Does this sound like a reasonable change?  Would it be invasive to change the test discovery to have tests say how many results they will return and then make worker threads responsible for slices of the list?
> 
> Thanks again.
> 
> 
> https://reviews.llvm.org/D40077
> 
> 
> 
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits