[PATCH] D98179: [lit] Sort test start times based on prior test timing data

[David Zarzycki via Phabricator via llvm-commits]

davezarzycki added a comment.

In D98179#2638882 <https://reviews.llvm.org/D98179#2638882>, @mehdi_amini wrote:

>> No, I'm running lit and dumping into a file. sorting is not as easy as piping through sort, as I'm running with -vv (required).
>
> Since tests run in parallel, don't you already have some non-determinism in the -vv output? Isn't it printing as test finish? (just curious, not pushing back on anything here).

Right. Unless `-j1` is passed to lit, then the `lit` output is, strictly speaking, non-deterministic. That being said (and especially on a quiet machine), I do understand why people might feel that the output is consistent enough from run to run to be quasi-deterministic.

>> Anyway, sorting the list on lit side should be 1 line for some that knows where that line goes :)
>
> Yeah I agree that sorting on the lit side for the summary should be fairly easy!

As far as I can tell, the `lit` architecture likes to "pay-as-you-go" with test output, so there isn't a good place to sort the output and I'd wager that a lot of people would be upset if the test output stalled until completion for sorting reasons.

If people feel that deleting the test timing data is onerous (and IMHO, I don't think that's a big ask), then I think the simplest solution is to add a `--ignore-timing-data` option to lit that ignores test timing data and gives people the old behavior (for whatever reason).

But I don't think the *default* lit behavior should be to ignore test failures and ignore test timing data. Both of these are about making the out-of-the-box experience for people more productive. If people have reasons to run tests in specific orders, then let's talk about those needs and why a lexical was convenient.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D98179/new/

https://reviews.llvm.org/D98179