[libcxx-dev] Should libc++ .fail.cpp tests use -verify?

[Louis Dionne via libcxx-dev]

> On Jul 1, 2020, at 14:01, Brian Cain <bcain at codeaurora.org> wrote:
> 
>> -----Original Message-----
>> From: libcxx-dev <libcxx-dev-bounces at lists.llvm.org> On Behalf Of Mikhail
>> Maltsev via libcxx-dev
> ...
>> On 11/06/2020 21:26, Louis Dionne wrote:
> ...
>> This happens because in our case we are targeting bare-metal platforms and
>> the ${exec} substitution runs the provided command on a simulator. This
>> obviously fails when the command includes Unix shell constructs, such as I/O
>> redirections, pipes and shell built-ins (e.g. echo, !).
>> 
>> I propose to modify the set of substitutions as follows:
>> - add %{exec_sh} meaning "execute a shell command" (e.g. locally or on a
>> remote
>>   machine). Everything after %{exec_sh} is shell-quoted and passed to the
>>   corresponding script.
>> - add %{exec_bin} meaning "execute a target binary". The corresponding script
>>   accepts the binary and its command-line arguments.
>> - make %{exec} an alias for '%{exec_sh} %{exec_bin}'
>> 
>> Usage example:
>> // RUN: %{exec_sh} echo "123" \| %{exec_bin} %t.exe > %t.out
>> 
>> My prototype implementation worked fine for our use cases. Does this
>> approach look sensible to you? If it does, I will start working on a patch.

Do you have the full set of failing tests in that environment? I'd like to take a look at whether it's possible to change those tests to avoid using these constructs altogether. Furthermore, I have an incomplete patch (locally) that runs the tests in the internal Lit shell, which would also alleviate some of these issues. But ideally everything that comes after %{exec} would not require Unix shell capabilities.

I'm asking because I'd like to avoid introducing another substitution, if at all possible.

Louis

> 
> It makes sense to me.  Louis' patch for fixing the executors for new-format hasn't yet landed on our downstream repo, so I haven't had a chance to try them yet.  I expect I'll encounter the same issues you are seeing and I look forward to trying out your patch.
> 
> -Brian
>