[flang-dev] Porting F18 tests to use LLVM lit

[Stephen Scalpone via flang-dev]

Both swift [1] and clang [2] have built-in capabilities to validate semantic checks.   These compilers, like f18 and GCC, use a special markup that is designed to verify diagnostics and to test semantic analysis.

Here’s a sample line from a  clang test [2]:
  template<> struct __is_floating_point_helper<__float128> {};  // expected-error {{__float128 is not supported on this target}}

From swift [1]:
  private func publicReq() {} // expected-error {{method 'publicReq()' must be declared public because it matches a requirement in public protocol 'PublicProto'}} {{none}}

Both clang and swift use lit.

Here’s the lit line from the clang test [2]:
// RUN: %clang_cc1 -verify -std=c++11 %s

From swift [1]:
// RUN: %target-typecheck-verify-swift -enable-objc-interop -disable-objc-attr-requires-foundation-module -swift-version 5

FileCheck appears to be a very powerful tool; however, using FileCheck markup to verify diagnostics is not as straightforward as the markup used in swift and clang tests. [3]

The f18 semantic tests and the f18 tools used for verifying diagnostics are similar in spirit to clang’s and swift’s. We could adapt the existing f18 test scripts, e.g. test_errors.sh, to work with lit. An alternative form of this proposal is to duplicate the f18 test scripts in python for portability.

- Steve

[1] https://github.com/apple/swift/blob/master/test/Sema/accessibility.swift
[2] https://github.com/llvm/llvm-project/blob/master/clang/test/Sema/128bitfloat.cpp
[3] https://github.com/flang-compiler/f18/pull/934


From: Luke Ireland <Luke.Ireland at arm.com>
Date: Thursday, January 16, 2020 at 5:08 AM
To: Stephen Scalpone <sscalpone at nvidia.com>
Cc: "flang-dev at lists.llvm.org" <flang-dev at lists.llvm.org>
Subject: Re: [flang-dev] Porting F18 tests to use LLVM lit

External email: Use caution opening links or attachments

Hi Stephen,

I don’t see any issue with choosing test names, I can confer if you like but the naming of the existing tests have never come into question before, so I doubt anyone would have any particular preferences. They’re in a separate folder, so make sure the test name isn’t used in the existing tests, then I can’t imagine any problems arising.

And yes, when all tests are successfully ported, ‘test’ will be removed, and ‘test-lit’ will be renamed to ‘test’ taking its place, and the appropriate changes needed to incorporate this into make check-all will be needed as well, but should just be as simple as renaming again.

Thanks,
Luke

Degree Apprentice,
Arm Manchester

From: Stephen Scalpone <sscalpone at nvidia.com>
Date: Wednesday, 15 January 2020 at 23:51
To: Luke Ireland <Luke.Ireland at arm.com>
Cc: flang-dev at lists.llvm.org <flang-dev at lists.llvm.org>
Subject: Re: [flang-dev] Porting F18 tests to use LLVM lit
Hi Luke,

During the transition, when writing a lit test, are there any special guidelines beyond choosing a distinct test name?

After the final step of removing old tests, will you rename the test-lit directory to test?

Thanks for working on this!

 - Steve

On 1/15/20, 10:59 AM, "flang-dev on behalf of Doerfert, Johannes via flang-dev" <flang-dev-bounces at lists.llvm.org on behalf of flang-dev at lists.llvm.org> wrote:

    External email: Use caution opening links or attachments


    Thanks for working on this!

    On 01/15, Luke Ireland via flang-dev wrote:
    > Further to the conversation in llvm-dev<http://lists.llvm.org/pipermail/llvm-dev/2020-January/138282.html>, regarding LLVM testing infrastructure/lit, and the successfully merged patch from Carol<https://github.com/flang-compiler/f18/pull/861/commits/1049aeb81e0195d3fa1050b21e93a3980a0b7245>, we’re continuing development on adding this infrastructure by porting the existing tests to be compatible with lit.
    >
    > We’d like to propose the following approach:
    >
    >   1.  Start off with a new directory, “test-lit” containing lit tests
    >   2.  make check-all will run this and the old test directory together
    >   3.  port over all old tests to this new directory as lit tests
    >   4.  Once all tests are successfully ported, the old directory and test scripts are removed, and “test-lit” becomes “test”.
    >   5.  All new tests will be created to be compatible with lit.
    > During this transition phase, tests can either be written in the old style, and then ported with my script, or preferably, written to work with lit.

    This sounds very reasonable.

    > My PR has three patches:
    >
    >   1.  A script which ports old style tests to new lit tests. At the moment it only handles semantic error tests, i.e. tests run with check_error.sh
    >   2.  The output of the porting script on a subset of the tests for verification. The subset of tests should cover all variations of error tests in my script
    >   3.  All the remaining error tests from semantics ported to lit.

    Did you look at the llvm/utils/update_XXX_test_checks.py scripts? We
    might want to add a scripts there if we can generate, not port, the
    CHECK lines.

    Cheers,
      Johannes

    > My next steps:
    >
    >   1.  Port remaining semantic test types
    >      *   Symbol
    >      *   Generic
    >      *   Modfile
    >   2.  Port decimal tests
    >   3.  Port evaluate tests
    >   4.  Port preprocessing tests (They aren’t in CMakeLists.txt, but are easily portable)
    > Each step will come as a pull request in a similar format:
    >
    >   1.  Script change to handle new test type
    >   2.  Ported tests added
    >      *   Either in 2 commits (verified, remaining – likely for changes with a large number of tests)
    >      *   Or all in 1 commit (small number of tests)
    >   3.  Old style tests removed
    > My initial PR is here<https://github.com/flang-compiler/f18/pull/934>.
    >
    > Thanks,
    > Luke
    >
    > Degree Apprentice,
    > Arm Manchester

    > _______________________________________________
    > flang-dev mailing list
    > flang-dev at lists.llvm.org
    > https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev


    --

    Johannes Doerfert
    Researcher

    Argonne National Laboratory
    Lemont, IL 60439, USA

    jdoerfert at anl.gov



-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/flang-dev/attachments/20200117/d5171846/attachment-0001.html>