[PATCH] D97046: [lit] Generalize `early_tests`

Wed Mar 3 02:57:28 PST 2021

davezarzycki added a comment.

In D97046#2598124 <https://reviews.llvm.org/D97046#2598124>, @mehdi_amini wrote:

> In D97046#2587436 <https://reviews.llvm.org/D97046#2587436>, @lebedev.ri wrote:
>
>> I feel like this should be handled by lit transparently, like recording the time it took to run each test last time, then somehow deciding on the new test run order based on that,
>
> +1.
>
> This seems like the right thing to do to me. The "early" or phase looks very ad-hoc to me. 
> It also isn't clear to me how it'd would play with a "recording time and replay" solution that would be the desired default for me.

I don't know if you saw my response to @lebedev.ri but I'll try to paraphrase: I agree that recording and then sorting the order would be ideal for people doing incremental builds/testing but it won't help clean builds (bots and paranoid people) unless that sorted list gets committed back to the repository. I'm not looking to propose that anybody do that or that it be the only way to get good testing performance. Some people just want a low maintenance config that they update maybe once or twice a year at most. I.e. the top five or so slowest tests (and maybe in the case of Swift, putting the compiler crasher directories towards the end).

If somebody wants to teach lit how to record a sorted list of tests times and then read that list back in again, that'd be great, but I think that's a great followup patch. Thanks!

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D97046/new/

https://reviews.llvm.org/D97046