[PATCH] D88807: [lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`.

[Dan Liew via Phabricator via llvm-commits]

delcypher added a comment.

In D88807#2312194 <https://reviews.llvm.org/D88807#2312194>, @yln wrote:

> Flaky tests are the worst! Thanks for working on this :)
>
> I think the principled way here would be to have two separate lit invocations for timeout fail/pass:
>
>   not lit --timeout=1 infinite_hang.py
>   # Will hang for 1 second, then aborted by timeout.
>   
>   lit --timeout=999 short.py
>   # Will take however long it takes to do `short.py`, then pass. 999 just means "very long timeout"
>   # This will be immediate on normal hosts, and whatever time it takes on resource constrained hosts.
>
> Or am I missing something?

You're not missing anything. This is a very good idea and I wish I had thought of it. I'll update the patch to take the approach you've outlined.



================
Comment at: llvm/utils/lit/tests/Inputs/googletest-timeout/DummySubDir/OneTest.py:20-27
+if test_name == 'FirstTest.subTestB':
+    print('I am subTest B, I will hang')
+    while True:
+        pass
 elif test_name == 'FirstTest.subTestC':
     print('I am subTest C, I will hang')
     while True:
----------------
dblaikie wrote:
> If these (subTestB and subTestC) do the same thing - should one of them be removed to avoid redundancy?
> 
> (and/or was the intent to keep the not-slow test, A (& one of only B or C) - if it is sufficiently less flakey?)
Probably but @yln 's comments mean that I might do things very differently.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D88807/new/

https://reviews.llvm.org/D88807