[libcxx-commits] [PATCH] D98179: [lit] Sort test start times based on prior test timing data

[Nuno Lopes via Phabricator via libcxx-commits]

nlopes added a comment.

In D98179#2639565 <https://reviews.llvm.org/D98179#2639565>, @davezarzycki wrote:

> In D98179#2639554 <https://reviews.llvm.org/D98179#2639554>, @nlopes wrote:
>
>> In D98179#2639491 <https://reviews.llvm.org/D98179#2639491>, @davezarzycki wrote:
>>
>>> In D98179#2639476 <https://reviews.llvm.org/D98179#2639476>, @nlopes wrote:
>>>
>>>> Why are timeouts important? Our use case is running Alive2 with the test suite. Alive2 is heavy machinery and runs into timeouts. Running the tests in roughly the same order every time is important to avoid timeout tests flipping to failed or vice-versa. Plus slow tests = tests that consume a lot of memory (in our scenario), so we can't bundle slow tests together.
>>>> Adding a `--ignore-timing-data` would be great, yes! But I still feel that sorting the list of failed tests is important for user experience. I diff these all the time.
>>>
>>> That still sounds incredibly brittle. If there is any variety in test machine performance, then any and all attempts at sorting should be futile because the underlying hardware will perturb different timeouts. Is this not your experience? How do you reconcile hardware performance and configuration details (like SMT) with timeout settings?
>>
>> Of course it's brittle :)  Changing from a time-based setting to a ticks-based system is ongoing work, such that resource exhaustion becomes deterministic.
>> Nevertheless, on a same machine, we don't see many test flips. It's quite stable most of the times (just one test flip once in a while).
>
> This seems really beyond the scope and purpose of sorting the tests.
>
> If you don't mind and given that the workaround is trivial (delete the timing data), I'd like to hold off on adding `--ignore-timing-data`. If enough people complain then we can add that option. Is that okay with you?

Sounds good. Thanks for adding support for the summary sort!


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D98179/new/

https://reviews.llvm.org/D98179