[PATCH] D118691: Make shtest-format.py more CHECK lines more flexible

[Douglas Yung via Phabricator via llvm-commits]

dyung created this revision.
dyung added reviewers: ddunbar, mgorny, yln, probinson, lebedev.ri.
dyung added a project: LLVM.
Herald added a subscriber: delcypher.
dyung requested review of this revision.

I've noticed that when run on Windows, the lit test shtest-format.py seems to fail quite often on my machine, seemingly randomly. From an examination of the error output, in my case it seemed that it was due to an extra line being printed out which caused the test to fail because it was using CHECK-NEXT. If I change to use CHECK instead of CHECK-NEXT, it should still maintain the intent of the test while making it flexible enough to handle the "random" error that I see often.

For example, here is the output that the test expects:

  Command Output (stderr): 
  -- 
  cat: does-not-exist: No such file or directory 
   
  -- 

However, on my machine, for reasons I haven't been able to figure out and/or fix, the bash.exe that is used to execute the test emits a warning about a missing /tmp directory to stderr which appears first and causes the CHECK-NEXT sequence to fail. The output looks like this:

  Command Output (stderr): 
  -- 
  bash.exe: warning: could not find /tmp, please create! 
  cat: does-not-exist: No such file or directory 
   
  -- 

I strongly suspect that when the test passes on my machine, it is because the bash warning is emitted AFTER the expected line in the stderr stream, so it doesn't fail the test because after matching the cat error, the test only uses CHECK to detect the "--" at the end of the test rather than CHECK-NEXT.

The fix I'm proposing here still checks for the expected output, but doesn't require the lines to immediately follow one-another making it more flexible and should preserve the intent of the test.

The test random failures can also be seen on our public PS4 Windows bot: https://lab.llvm.org/staging/#/builders/204


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D118691

Files:
  llvm/utils/lit/tests/shtest-format.py


Index: llvm/utils/lit/tests/shtest-format.py
===================================================================
--- llvm/utils/lit/tests/shtest-format.py
+++ llvm/utils/lit/tests/shtest-format.py
@@ -16,8 +16,8 @@
 # CHECK-NEXT: line 1: failed test output on stdout
 # CHECK-NEXT: line 2: failed test output on stdout
 # CHECK: Command Output (stderr):
-# CHECK-NEXT: --
-# CHECK-NEXT: cat{{(_64)?(\.exe)?}}: {{cannot open does-not-exist|does-not-exist: No such file or directory}}
+# CHECK: --
+# CHECK: cat{{(_64)?(\.exe)?}}: {{cannot open does-not-exist|does-not-exist: No such file or directory}}
 # CHECK: --
 
 # CHECK: FAIL: shtest-format :: external_shell/fail_with_bad_encoding.txt


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D118691.404870.patch
Type: text/x-patch
Size: 692 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20220201/7c73ab61/attachment.bin>