[llvm] r328702 - [lit] Remove a timing senstive part of `shtest-timeout.py`

Dan Liew via llvm-commits llvm-commits at lists.llvm.org
Wed Mar 28 06:55:08 PDT 2018 

Author: delcypher
Date: Wed Mar 28 06:55:08 2018
New Revision: 328702

URL: http://llvm.org/viewvc/llvm-project?rev=328702&view=rev
Log:
[lit] Remove a timing senstive part of `shtest-timeout.py`

The `shtest-timeout.py` test was failing intermittently. It looks like
the issue is that on a resource constrained system lit is unable to run
`quick_then_slow.py` twice and print out the messages the tests expects
within the one second timeout.

The underlying issue is that the test is dependent on the performance of
the host machine is a rather fragile way. This is due to hardcoding
timeout values and having assumptions that the host machine is able to
perform a certain amount of work within the hardcoded timeout values.

We could increase the timeout values but that doesn't really fix the
underlying issue. Instead this patch removes one of fragile assumptions
in the hope that this will be enough to fix the bots.
There are other fragile assumptions in this test (e.g. `quick.py` can be
executed in less than 1 second). If the bots continue to fail we'll have
to revisit this.

rdar://problem/38774530

Removed:
    llvm/trunk/utils/lit/tests/Inputs/shtest-timeout/quick_then_slow.py
Modified:
    llvm/trunk/utils/lit/tests/shtest-timeout.py

Removed: llvm/trunk/utils/lit/tests/Inputs/shtest-timeout/quick_then_slow.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-timeout/quick_then_slow.py?rev=328701&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-timeout/quick_then_slow.py (original)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-timeout/quick_then_slow.py (removed)
@@ -1,24 +0,0 @@
-# RUN: %{python} %s quick
-# RUN: %{python} %s slow
-from __future__ import print_function
-
-import time
-import sys
-
-if len(sys.argv) != 2:
-    print("Wrong number of args")
-    sys.exit(1)
-
-mode =  sys.argv[1]
-
-if mode == 'slow':
-    print("Running in slow mode")
-    sys.stdout.flush() # Make sure the print gets flushed so it appears in lit output.
-    time.sleep(6)
-    sys.exit(0)
-elif mode == 'quick':
-    print("Running in quick mode")
-    sys.exit(0)
-else:
-    print("Unrecognised mode {}".format(mode))
-    sys.exit(1)

Modified: llvm/trunk/utils/lit/tests/shtest-timeout.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/shtest-timeout.py?rev=328702&r1=328701&r2=328702&view=diff
==============================================================================
--- llvm/trunk/utils/lit/tests/shtest-timeout.py (original)
+++ llvm/trunk/utils/lit/tests/shtest-timeout.py Wed Mar 28 06:55:08 2018
@@ -9,7 +9,6 @@
 # Test per test timeout using external shell
 # RUN: not %{lit} \
 # RUN: %{inputs}/shtest-timeout/infinite_loop.py \
-# RUN: %{inputs}/shtest-timeout/quick_then_slow.py \
 # RUN: %{inputs}/shtest-timeout/short.py \
 # RUN: %{inputs}/shtest-timeout/slow.py \
 # RUN: -j 1 -v --debug --timeout 1 --param external=1 > %t.extsh.out 2> %t.extsh.err
@@ -21,7 +20,6 @@
 # Test per test timeout using internal shell
 # RUN: not %{lit} \
 # RUN: %{inputs}/shtest-timeout/infinite_loop.py \
-# RUN: %{inputs}/shtest-timeout/quick_then_slow.py \
 # RUN: %{inputs}/shtest-timeout/short.py \
 # RUN: %{inputs}/shtest-timeout/slow.py \
 # RUN: -j 1 -v --debug --timeout 1 --param external=0 > %t.intsh.out 2> %t.intsh.err
@@ -34,16 +32,6 @@
 # CHECK-INTSH-OUT-NEXT: Running infinite loop
 # CHECK-INTSH-OUT: command reached timeout: True
 
-# CHECK-INTSH-OUT: TIMEOUT: per_test_timeout :: quick_then_slow.py
-# CHECK-INTSH-OUT: Timeout: Reached timeout of 1 seconds
-# CHECK-INTSH-OUT: Command Output
-# CHECK-INTSH-OUT: command output:
-# CHECK-INTSH-OUT-NEXT: Running in quick mode
-# CHECK-INTSH-OUT: command reached timeout: False
-# CHECK-INTSH-OUT: command output:
-# CHECK-INTSH-OUT-NEXT: Running in slow mode
-# CHECK-INTSH-OUT: command reached timeout: True
-
 # CHECK-INTSH-OUT: TIMEOUT: per_test_timeout :: slow.py
 # CHECK-INTSH-OUT: command output:
 # CHECK-INTSH-OUT-NEXT: Running slow program
@@ -54,7 +42,6 @@
 # Test per test timeout set via a config file rather than on the command line
 # RUN: not %{lit} \
 # RUN: %{inputs}/shtest-timeout/infinite_loop.py \
-# RUN: %{inputs}/shtest-timeout/quick_then_slow.py \
 # RUN: %{inputs}/shtest-timeout/short.py \
 # RUN: %{inputs}/shtest-timeout/slow.py \
 # RUN: -j 1 -v --debug --param external=0 \
@@ -69,12 +56,6 @@
 # CHECK-OUT-COMMON: Command {{([0-9]+ )?}}Output
 # CHECK-OUT-COMMON: Running infinite loop
 
-# CHECK-OUT-COMMON: TIMEOUT: per_test_timeout :: quick_then_slow.py
-# CHECK-OUT-COMMON: Timeout: Reached timeout of 1 seconds
-# CHECK-OUT-COMMON: Command {{([0-9]+ )?}}Output
-# CHECK-OUT-COMMON: Running in quick mode
-# CHECK-OUT-COMMON: Running in slow mode
-
 # CHECK-OUT-COMMON: PASS: per_test_timeout :: short.py
 
 # CHECK-OUT-COMMON: TIMEOUT: per_test_timeout :: slow.py
@@ -83,13 +64,12 @@
 # CHECK-OUT-COMMON: Running slow program
 
 # CHECK-OUT-COMMON: Expected Passes{{ *}}: 1
-# CHECK-OUT-COMMON: Individual Timeouts{{ *}}: 3
+# CHECK-OUT-COMMON: Individual Timeouts{{ *}}: 2
 
 # Test per test timeout via a config file and on the command line.
 # The value set on the command line should override the config file.
 # RUN: not %{lit} \
 # RUN: %{inputs}/shtest-timeout/infinite_loop.py \
-# RUN: %{inputs}/shtest-timeout/quick_then_slow.py \
 # RUN: %{inputs}/shtest-timeout/short.py \
 # RUN: %{inputs}/shtest-timeout/slow.py \
 # RUN: -j 1 -v --debug --param external=0 \
@@ -104,12 +84,6 @@
 # CHECK-CMDLINE-OVERRIDE-OUT: Command {{([0-9]+ )?}}Output
 # CHECK-CMDLINE-OVERRIDE-OUT: Running infinite loop
 
-# CHECK-CMDLINE-OVERRIDE-OUT: TIMEOUT: per_test_timeout :: quick_then_slow.py
-# CHECK-CMDLINE-OVERRIDE-OUT: Timeout: Reached timeout of 2 seconds
-# CHECK-CMDLINE-OVERRIDE-OUT: Command {{([0-9]+ )?}}Output
-# CHECK-CMDLINE-OVERRIDE-OUT: Running in quick mode
-# CHECK-CMDLINE-OVERRIDE-OUT: Running in slow mode
-
 # CHECK-CMDLINE-OVERRIDE-OUT: PASS: per_test_timeout :: short.py
 
 # CHECK-CMDLINE-OVERRIDE-OUT: TIMEOUT: per_test_timeout :: slow.py
@@ -118,4 +92,4 @@
 # CHECK-CMDLINE-OVERRIDE-OUT: Running slow program
 
 # CHECK-CMDLINE-OVERRIDE-OUT: Expected Passes{{ *}}: 1
-# CHECK-CMDLINE-OVERRIDE-OUT: Individual Timeouts{{ *}}: 3
+# CHECK-CMDLINE-OVERRIDE-OUT: Individual Timeouts{{ *}}: 2

More information about the llvm-commits mailing list