[libcxx-commits] [PATCH] D98179: [lit] Sort test start times based on prior test timing data
David Zarzycki via Phabricator via libcxx-commits
libcxx-commits at lists.llvm.org
Sat Mar 20 03:45:07 PDT 2021
davezarzycki added a comment.
In D98179#2638882 <https://reviews.llvm.org/D98179#2638882>, @mehdi_amini wrote:
>> No, I'm running lit and dumping into a file. sorting is not as easy as piping through sort, as I'm running with -vv (required).
> Since tests run in parallel, don't you already have some non-determinism in the -vv output? Isn't it printing as test finish? (just curious, not pushing back on anything here).
Right. Unless `-j1` is passed to lit, then the `lit` output is, strictly speaking, non-deterministic. That being said (and especially on a quiet machine), I do understand why people might feel that the output is consistent enough from run to run to be quasi-deterministic.
>> Anyway, sorting the list on lit side should be 1 line for some that knows where that line goes :)
> Yeah I agree that sorting on the lit side for the summary should be fairly easy!
As far as I can tell, the `lit` architecture likes to "pay-as-you-go" with test output, so there isn't a good place to sort the output and I'd wager that a lot of people would be upset if the test output stalled until completion for sorting reasons.
If people feel that deleting the test timing data is onerous (and IMHO, I don't think that's a big ask), then I think the simplest solution is to add a `--ignore-timing-data` option to lit that ignores test timing data and gives people the old behavior (for whatever reason).
But I don't think the *default* lit behavior should be to ignore test failures and ignore test timing data. Both of these are about making the out-of-the-box experience for people more productive. If people have reasons to run tests in specific orders, then let's talk about those needs and why a lexical was convenient.
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
More information about the libcxx-commits