[llvm-dev] RFC: A cross platform way of using shell commands in lit tests

[Justin Bogner via llvm-dev]

Zachary Turner via llvm-dev <llvm-dev at lists.llvm.org> writes:
> I see many tests that are prefixed with "requires: shell".  A quick search
> over the codebase shows 101 tests that contain this directive.  This basically
> means there are 101 tests that say "I don't support running on Windows".  
>
> I would like to get this number to 0.  
>
> Ironically, many of these tests can be made to run on Windows simply by
> removing the requires: shell line.  As in, it doesn't *actually* require a
> shell.  This is because the meaning of requires shell is not intuitive.  
> GnuWin32 is already a required dependency, so any shell command that you are
> familiar with probably exists on Windows already.  For example, consider the
> following test in clang-tidy:

Anecdotally, most of the times I've seen people add "REQUIRES: shell" on
a test they really meant "disable on windows". I think/hope that's
usually temporary while they investigate problems, but it might explain
why so many tests say they require shell when they don't.

> // REQUIRES: shell
> // RUN: mkdir -p %T/compilation-database-test/include
> // RUN: mkdir -p %T/compilation-database-test/a
> // RUN: mkdir -p %T/compilation-database-test/b
> // RUN: echo 'int *AA = 0;' > %T/compilation-database-test/a/a.cpp
> // RUN: echo 'int *AB = 0;' > %T/compilation-database-test/a/b.cpp
> // RUN: echo 'int *BB = 0;' > %T/compilation-database-test/b/b.cpp
> // RUN: echo 'int *BC = 0;' > %T/compilation-database-test/b/c.cpp
> // RUN: echo 'int *HP = 0;' > %T/compilation-database-test/include/header.h
> // RUN: echo '#include "header.h"' > %T/compilation-database-test/b/d.cpp
> // RUN: sed 's|test_dir|%T/compilation-database-test|g' %S/Inputs/
> compilation-database/template.json > %T/compile_commands.json
> // RUN: clang-tidy --checks=-*,modernize-use-nullptr -p %T %T/
> compilation-database-test/b/not-exist -header-filter=.*
> // RUN: clang-tidy --checks=-*,modernize-use-nullptr -p %T %T/
> compilation-database-test/a/a.cpp %T/compilation-database-test/a/b.cpp %T/
> compilation-database-test/b/b.cpp %T/compilation-database-test/b/c.cpp %T/
> compilation-database-test/b/d.cpp -header-filter=.* -fix
> // RUN: FileCheck -input-file=%T/compilation-database-test/a/a.cpp %s
> -check-prefix=CHECK-FIX1
> // RUN: FileCheck -input-file=%T/compilation-database-test/a/b.cpp %s
> -check-prefix=CHECK-FIX2
> // RUN: FileCheck -input-file=%T/compilation-database-test/b/b.cpp %s
> -check-prefix=CHECK-FIX3
> // RUN: FileCheck -input-file=%T/compilation-database-test/b/c.cpp %s
> -check-prefix=CHECK-FIX4
> // RUN: FileCheck -input-file=%T/compilation-database-test/include/header.h %s
> -check-prefix=CHECK-FIX5
>
> This test does not require a shell!  Remove the line (and change the %S and %T
> on the sed line to %/S and %/T) and this test already works on Windows!
>
> I have inspected around 20 tests so far that claim to require shell, and so
> far only about 3 of them are more complicated than removing the line.
>
> "Great, so just remove the lines!  Why are you posting this?"
>
> Well, It would be nice if we could remove the GnuWin32 dependency as well. 
> Lots of tests use grep, sed, cd, ln, or whatever.  Anyone who's ever tried
> downloading and installing GnuWin32 knows what a pain it is.  It installs
> programs called, for example, "link" which interfere with standard windows
> tools, and puts them in your path.  There are other examples of this.  And
> it's also HUGE.  And nobody can ever figure out how to download it correctly. 
> Plus there are some minor differences in the tools behavior.
>
> I'm proposing that we replace shell commands in tests with a tool that we
> create which implements the most common shell commands and options.

To some degree, we already do this. See for example r231017, which adds
'cd' to lit's "internal shell". Adding some more common things here
seems reasonable, as long as they aren't too complicated.

> it doesn't need to be a complete implementation of the commands.  Nobody cares
> about echo -n, or sed -u, for example.  We could keep it simple with the most
> common commands, most of which would be trivially implementable using the llvm
> Support library already.  Complicated commands like awk, for example, could be
> absent.  Chances are you need to simplify your test if you're using something
> like that (presently exactly 1 test in all of LLVM uses it).  What I'm
> imagining is something like this:

>From a quick grep, I'm not even convinced one test in llvm uses awk -
there's a test (X86/hoist-spill) with a comment that has a shell
pipeline including awk, but it isn't in a RUN: line at all :/