[llvm-dev] Marking lit::shtest-format.py unsupported on PS4?, Re: buildbot failure in LLVM on llvm-clang-lld-x86_64-scei-ps4-ubuntu-fast

[Justin Bogner via llvm-dev]

Jeremy Morse via llvm-dev <llvm-dev at lists.llvm.org> writes:
> We've given this an eyeball, and reckon there's interference between the
> lit "shtest-format" and "shtest-xunit-output" tests -- the following two
> run commands:
>
>     # RUN: not %{lit} -j 1 -v %{inputs}/shtest-format > %t.out
>     # RUN: not %{lit} -j 1 -v %{inputs}/shtest-format --xunit-xml-output
> %t.xml
>
> appear in the tests respectively, using the
> utils/lit/tests/Inputs/shtest-format directory as the target test suite.
> For external-shell tests in that suite however, lit produces a
> '.script.bat' file in the build directory to be run by the external shell:
>
>     $ pwd
>
> /cygdrive/d/build/utils/lit/tests/Inputs/shtest-format/external_shell/Output
>
>     $ ls
>     fail.txt.script.bat  fail_with_bad_encoding.txt.script.bat
> pass.txt.script.bat
>
> Two concurrent lit invocations running on the same test suite will both
> write to those files, creating racy filesystem behaviour and likely the
> errors that have been seen. I've found at least one shtest-xunit-output
> test run with a "Permission denied" error when writing to the script file
> (on Windows). The race is still present for non-windows systems but
> presumably much rarer.
>
> I'm not very familiar with lit, and presumably making the external-shell
> scripts temporarily named files would fix this, but there may be other
> assumptions lit makes about a test suite not being operated on concurrently.

Thanks for looking into this. You're right, the problem is that lit
writes scripts into its test suite, so the "inputs" directory here is
kind of poorly named. I think the simplest short term fix is to just
merge the check lines from shtest-format and shtest-xunit-output into
one test. I've gone ahead and done that for now in r337718.

Arguably it'd be more correct to copy the test suites from inputs/ into
paths in %t somewhere to properly isolate things, but I think that makes
the tests slower and more complicated without very good reason.