[PATCH] D97046: [lit] Add "late_tests" test suite config option

[James Henderson via Phabricator via llvm-commits]

jhenderson added a comment.

In D97046#2578524 <https://reviews.llvm.org/D97046#2578524>, @davezarzycki wrote:

> In D97046#2578483 <https://reviews.llvm.org/D97046#2578483>, @jhenderson wrote:
>
>> In D97046#2578480 <https://reviews.llvm.org/D97046#2578480>, @thopre wrote:
>>
>>> In D97046#2578450 <https://reviews.llvm.org/D97046#2578450>, @jhenderson wrote:
>>>
>>>> Could you clarify more what the benefit of running the tests later actually is? Is this intended to be useful in conjunction with something else? On the face of it, I'd expect people to run the full testsuite to completion, and then observe the full results. Ultimately, it doesn't matter whether a test runs first or last in this order for that case.
>>>
>>> You want long running tests to run first and shorter one last as they will use to balance the imbalance from long running test. It's just to maximize the parallelism. If you have some medium to big tests running at the end, the risk is you are waiting for one core to finish while all the others are sitting idle. If you keep lots of small tests at the end, the cores that are idle will pick from it and hopefully reduce the imbalance.
>>
>> I get the idea of running long tests early, but from observation, the vast majority of lit tests in the LLVM test suite are very similar in length, lasting less than a second. As such, it doesn't make any significant difference whether they are run late, or in the middle (as long as the long ones are run early). I would assume the same applies as much for these Swift tests as any other equivalent ones. I could see a case if we're suggesting that this feature is for a downstream test suite that is mostly comprised of long and slow tests (i.e. ones that take several seconds each), so that it's easier to mark the ones that are quick, but I don't think that's the case here?
>
> I don’t think `late_tests` is useful to every lit test suite. That’s why I gave the Swift example in the description. And yes, this is just about maximizing concurrency. In particular, lit runs tests in lexical order so the “compiler_crasher” directories sort before most other tests. I’m not at my computer but my memory is that running the ultra-fast compiler crasher tests last to fill in the otherwise idle cores makes the whole test suite about 5% faster.

So back to my early question on this - is it possible to simply mark the other tests as early tests or are there too many of them to be practical? If it's impractical, that's fine and we can proceed. Also, what's the approximate duration of the slower tests (per test) compared to these fast tests? I'm just trying to get a picture of what is going on, that's all, before I sign off on another feature.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D97046/new/

https://reviews.llvm.org/D97046