[PATCH] D97046: [lit] Add "late_tests" test suite config option

[David Zarzycki via Phabricator via llvm-commits]

davezarzycki added a comment.

It is both impractical and unwise to mark all of the non-compiler-crasher tests in Swift as "early". It's impractical because that's most of the tests. It's unwise because having more than a few tests in the early stage negates the benefit of the early stage itself.

As to test timing, I don't think the specific results matter as much as the patterns that emerge. On a high-end workstation from 2017, it's not hard to be in a scenario where the "total test time" is close to the "slowest individual test time". This is true for LLVM and Swift. I haven't checked clang but I'd expect that to be true too. That's why it matters a lot to run the top five or so slowest tests early.

Now, in the specific case of Swift, we can further speed up the overall testing time by running easily identifiable fast tests "late". This works because there are a lot of "medium slow" tests (far too many to add to the "early_tests" phase) and because `lit` sorts the tests and "compiler_crasher" happens to sort early. For Swift, there are 6000+ fast (and low reward) tests spread across four directories:  compiler_crashers, compiler_crashers_2, compiler_crashers_2_fixed, and compiler_crashers_fixed. So with one simple regular expression, we can get thousands of fast/low-reward tests to run late. This is a small performance win for fast machines and a huge user experience win for slower machines.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D97046/new/

https://reviews.llvm.org/D97046