[libcxx-dev] Should libc++ .fail.cpp tests use -verify?

[Mikhail Maltsev via libcxx-dev]

On 11/06/2020 21:26, Louis Dionne wrote:

> It should now be more flexible and easier to specify arbitrary executors. You 
> will need to change a few things since executors are now arbitrary scripts 
> instead of subclasses of the Python executor, but that should be fairly 
> straightforward. Can we work together to make sure you're able to migrate to the 
> new format?

Hi Louis,

While working on the migration of our infrastructure to the new format I noticed 
that several '.sh.cpp' tests are failing, e.g. 
libcxx/selftest/newformat/remote-substitutions.sh.cpp and 
std/input.output/iostream.objects/narrow.stream.objects/cin.sh.cpp.

This happens because in our case we are targeting bare-metal platforms and the 
${exec} substitution runs the provided command on a simulator. This obviously 
fails when the command includes Unix shell constructs, such as I/O redirections, 
pipes and shell built-ins (e.g. echo, !).

I propose to modify the set of substitutions as follows:
- add %{exec_sh} meaning "execute a shell command" (e.g. locally or on a remote
   machine). Everything after %{exec_sh} is shell-quoted and passed to the
   corresponding script.
- add %{exec_bin} meaning "execute a target binary". The corresponding script
   accepts the binary and its command-line arguments.
- make %{exec} an alias for '%{exec_sh} %{exec_bin}'

Usage example:
// RUN: %{exec_sh} echo "123" \| %{exec_bin} %t.exe > %t.out

My prototype implementation worked fine for our use cases. Does this approach 
look sensible to you? If it does, I will start working on a patch.

-- 
Regards,
    Mikhail Maltsev