[libcxx-dev] Should libc++ .fail.cpp tests use -verify?

[Louis Dionne via libcxx-dev]

Hi,

That's only true with the old format -- the new format handles .fail.cpp tests properly (i.e. clang-verify when supported, otherwise not). Support for the old format should be removed and folks should move to the new format, which handles custom executors much better actually. For example, the new format will honour custom executors when performing feature tests to see whether e.g. locales or non-lockfree atomics are supported on the target.

The issue up to now was how to specify a custom executor from the CMake build, but that should be solved since this commit:

commit 96e6cbbf941d0f937b7e823433d4c222967a1817
Author: Louis Dionne <ldionne at apple.com>
Date:   Thu Jun 11 15:41:38 2020 -0400

    [libc++] Allow specifying arbitrary custom executors with the new format

    The integration between CMake and executor selection in the new format
    wasn't very flexible -- only the default executor and SSH executors were
    supported.

    This patch makes it possible to specify arbitrary executors with the new
    format. With the new testing format, a custom executor is just a script
    that gets called with a command-line to execute, and some arguments like
    --env, --codesign_identity and --execdir. As such, the default executor
    is just run.py.

    Remote execution with the SSH executor can be achived by specifying
    LIBCXX_EXECUTOR="<path-to-ssh.py> --host <host>". Similarly, arbitrary
    scripts can be provided.

It should now be more flexible and easier to specify arbitrary executors. You will need to change a few things since executors are now arbitrary scripts instead of subclasses of the Python executor, but that should be fairly straightforward. Can we work together to make sure you're able to migrate to the new format?

Cheers,
Louis

> On Jun 11, 2020, at 08:06, Mikhail Maltsev <Mikhail.Maltsev at arm.com> wrote:
> 
> Hi all,
> 
> After the following commit
> 
> "commit 2fd7d364cd999c743b5bdefa7ee9e5630f3564ad
> Author: Louis Dionne <ldionne at apple.com>
> Date:   Mon Apr 27 12:49:01 2020 -0400
> 
> [libc++] Make the verify-support feature implicit
> 
> Tests that require support for Clang-verify are already marked as such
> explicitly by their extension, which is .verify.cpp. Requiring the use
> of an explicit Lit feature is, after thought, not really helpful.
> 
> This is a change in design: we have been bitten in the past by tests not
> being enabled when we thought they were. However, the issue was mostly
> with file extensions being ignored. The fix for that is not to blindly
> require explicit features all the time, but instead to report all files
> that are in the suite but that don't match any known test format. This
> can be implemented in a follow-up patch."
> 
> All '.fail.cpp' started being run without the -verify command line 
> option. This reduced test coverage: the test suite is now only checking 
> that compilation of the .fail.cpp tests fails, but does not check the 
> locations of the actual errors and warnings.
> 
> It this intentional? Just in case: we are using the 'use_old_format' lit 
> feature because the new format does not support custom executors (e.g. 
> simulators) yet.
> 
> -- 
> Regards,
>    Mikhail Maltsev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/libcxx-dev/attachments/20200611/7eb32f16/attachment.html>