[libcxx-commits] [PATCH] D98179: [lit] Sort test start times based on prior test timing data

Sat Mar 20 04:05:07 PDT 2021

davezarzycki added a comment.

In D98179#2638837 <https://reviews.llvm.org/D98179#2638837>, @nlopes wrote:

> In D98179#2638801 <https://reviews.llvm.org/D98179#2638801>, @mehdi_amini wrote:
>
>>> Can we revert to the previous behavior please? The current behavior is not user friendly. Thanks!
>>
>> To clarify: you care about the order in the final summary, not the actual execution order, right? (the goal of this patch is the latter, if it changes the former this is just a side-effect I believe)
>
> I don't love that the default test order was changed. For our use case that isn't good, but I can just delete the `.lit_test_times.txt` file and get the old fixed order. (the reason why having a fixed order is good is because of timeouts. Also, slow tests often consume more memory in our setting, so it's not good to run all the slow tests at the same time).

Can you elaborate more about your usage of timeouts? As far as I know, the feature exists only to detect wildly broken tests. If reordering the tests causes timeouts to fire, then either the timeout is too aggressive or the tests themselves were on the edge of failure anyway and any "unrelated" change could cause them to fail.

As to the resource management challenges, `lit` has an undocumented "parallelism group" feature that would probably help you keep resource usage in check and would be more far appropriate than lexical execution order.

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D98179/new/

https://reviews.llvm.org/D98179