[PATCH] D97046: [lit] Add "late_tests" test suite config option

[Julian Lettner via Phabricator via llvm-commits]

yln added a comment.

I also want to state that I am a bit concerned about feature creep and the the general usefulness of this effort.

- Does shaving off 5% of test time for one project justify having it in upstream? Think added complexity, maintenance and opportunity cost. It is difficult to decide where to draw the line and I will not block this effort, but I wanted to see where other people stand on this?
- Are there workarounds we have considered? An ugly one would be naming directories to influence order.
- Annotation of fast and slow tests is manual and needs to be maintained.  These things tend to rot over time and the benefit will decrease.

Is the sole goal with early/late tests to optimize execution time or are there any other semantics attached to it?  Dave mentioned that the tests he has in mind are also low reward.  My understanding is that this means that they "rarely fail/signal an issue that's not shown by other test".

For quick feedback, we have `--incremental` (poor name), which runs modified and failed tests first, i.e., which is an approximation of "run high reward tests first".

If we want to optimize overall execution time, maybe we should track test execution times and then run tests in decreasing order the next time around.  This would have the benefit that it doesn't need manual annotation (which may rot over time).  Maybe it should even be the default?  Not to be addressed in this review of course, but I wanted to raise my points.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D97046/new/

https://reviews.llvm.org/D97046