[flang-dev] Porting F18 tests to use LLVM lit

Mon Jan 20 07:37:41 PST 2020

Hi Tim

I really like this idea because it allows us to tackle one of the obstacles towards LLVM unification that is non-controversial on our side and then give proper thought to the more controversial choice of how/if to migrate off the existing checking scripts in F18. Luke’s going to change direction a bit to explore this avenue.

He's still working on showing off what FileCheck can do to replace test_errors to further the discussion that started on PR#934. As the above approach would cause those patches to change a lot, Luke will start a new PR for the alternative approach you propose and add all of the reviewers of PR#934 over as reviewers.

Ta
Rich

From: flang-dev <flang-dev-bounces at lists.llvm.org> On Behalf Of Timothy Keith via flang-dev
Sent: 18 January, 2020 17:08
To: Finkel, Hal J. <hfinkel at anl.gov>; Stephen Scalpone <sscalpone at nvidia.com>; Luke Ireland <Luke.Ireland at arm.com>
Cc: flang-dev at lists.llvm.org
Subject: Re: [flang-dev] Porting F18 tests to use LLVM lit

This change should proceed in two steps:
1. Run tests with llvm-lit instead of ctest but don't change how tests are checked.
2. Replace custom checking of results with FileCheck where appropriate.

The first step should be relatively straight-forward. Once this transition is complete the rest can happen incrementally.

The second step is more complicated because there are several different kinds of tests with their own requirements for checking the results. The folding tests might be simplified by using FileCheck. I don't know how .mod file tests and symbol tests can use FileCheck (but my knowledge of FileCheck is limited). There is on-going discussion in PR #934 about how to use FileCheck on the error tests.

Obviously the overriding consideration is making it easy to write high quality tests, run them, and diagnose their failures. Anything we can do to improve that will pay off greatly in the future.

Tim

From: flang-dev <mailto:flang-dev-bounces at lists.llvm.org> on behalf of "Finkel, Hal J. via flang-dev" <mailto:flang-dev at lists.llvm.org>
Reply-To: "Finkel, Hal J." <mailto:hfinkel at anl.gov>
Date: Friday, January 17, 2020 at 4:41 AM
To: Stephen Scalpone <mailto:sscalpone at nvidia.com>, Luke Ireland <mailto:Luke.Ireland at arm.com>
Cc: "mailto:flang-dev at lists.llvm.org" <mailto:flang-dev at lists.llvm.org>
Subject: Re: [flang-dev] Porting F18 tests to use LLVM lit

External email: Use caution opening links or attachments 

On 1/17/20 2:42 AM, Stephen Scalpone via flang-dev wrote:
Both swift [1] and clang [2] have built-in capabilities to validate semantic checks.   These compilers, like f18 and GCC, use a special markup that is designed to verify diagnostics and to test semantic analysis.

Here’s a sample line from a  clang test [2]:
  template<> struct __is_floating_point_helper<__float128> {};  // expected-error {{__float128 is not supported on this target}}

From swift [1]:
  private func publicReq() {} // expected-error {{method 'publicReq()' must be declared public because it matches a requirement in public protocol 'PublicProto'}} {{none}} 

Both clang and swift use lit.

Here’s the lit line from the clang test [2]:
// RUN: %clang_cc1 -verify -std=c++11 %s

From swift [1]:
// RUN: %target-typecheck-verify-swift -enable-objc-interop -disable-objc-attr-requires-foundation-module -swift-version 5

FileCheck appears to be a very powerful tool; however, using FileCheck markup to verify diagnostics is not as straightforward as the markup used in swift and clang tests. [3]

The f18 semantic tests and the f18 tools used for verifying diagnostics are similar in spirit to clang’s and swift’s. We could adapt the existing f18 test scripts, e.g. test_errors.sh, to work with lit. 

I believe that we should use FileCheck where possible.
 -Hal

An alternative form of this proposal is to duplicate the f18 test scripts in python for portability.

- Steve

[1] https://github.com/apple/swift/blob/master/test/Sema/accessibility.swift
[2] https://github.com/llvm/llvm-project/blob/master/clang/test/Sema/128bitfloat.cpp
[3] https://github.com/flang-compiler/f18/pull/934

From: Luke Ireland mailto:Luke.Ireland at arm.com
Date: Thursday, January 16, 2020 at 5:08 AM
To: Stephen Scalpone mailto:sscalpone at nvidia.com
Cc: mailto:flang-dev at lists.llvm.org mailto:flang-dev at lists.llvm.org
Subject: Re: [flang-dev] Porting F18 tests to use LLVM lit

External email: Use caution opening links or attachments 

Hi Stephen,

I don’t see any issue with choosing test names, I can confer if you like but the naming of the existing tests have never come into question before, so I doubt anyone would have any particular preferences. They’re in a separate folder, so make sure the test name isn’t used in the existing tests, then I can’t imagine any problems arising.

And yes, when all tests are successfully ported, ‘test’ will be removed, and ‘test-lit’ will be renamed to ‘test’ taking its place, and the appropriate changes needed to incorporate this into make check-all will be needed as well, but should just be as simple as renaming again.

Thanks,
Luke

Degree Apprentice,
Arm Manchester 

From: Stephen Scalpone mailto:sscalpone at nvidia.com
Date: Wednesday, 15 January 2020 at 23:51
To: Luke Ireland mailto:Luke.Ireland at arm.com
Cc: mailto:flang-dev at lists.llvm.org mailto:flang-dev at lists.llvm.org
Subject: Re: [flang-dev] Porting F18 tests to use LLVM lit
Hi Luke,

During the transition, when writing a lit test, are there any special guidelines beyond choosing a distinct test name?

After the final step of removing old tests, will you rename the test-lit directory to test?

Thanks for working on this!

 - Steve

On 1/15/20, 10:59 AM, "flang-dev on behalf of Doerfert, Johannes via flang-dev" mailto:flang-dev-bounces at lists.llvm.orgonbehalfofflang-dev@lists.llvm.org wrote:

    External email: Use caution opening links or attachments

    Thanks for working on this!

    On 01/15, Luke Ireland via flang-dev wrote:
    > Further to the conversation in llvm-dev<http://lists.llvm.org/pipermail/llvm-dev/2020-January/138282.html>, regarding LLVM testing infrastructure/lit, and the successfully merged patch from Carol<https://github.com/flang-compiler/f18/pull/861/commits/1049aeb81e0195d3fa1050b21e93a3980a0b7245>, we’re continuing development on adding this infrastructure by porting the existing tests to be compatible with lit.
    >
    > We’d like to propose the following approach:
    >
    >   1.  Start off with a new directory, “test-lit” containing lit tests
    >   2.  make check-all will run this and the old test directory together
    >   3.  port over all old tests to this new directory as lit tests
    >   4.  Once all tests are successfully ported, the old directory and test scripts are removed, and “test-lit” becomes “test”.
    >   5.  All new tests will be created to be compatible with lit.
    > During this transition phase, tests can either be written in the old style, and then ported with my script, or preferably, written to work with lit.

    This sounds very reasonable.

    > My PR has three patches:
    >
    >   1.  A script which ports old style tests to new lit tests. At the moment it only handles semantic error tests, i.e. tests run with check_error.sh
    >   2.  The output of the porting script on a subset of the tests for verification. The subset of tests should cover all variations of error tests in my script
    >   3.  All the remaining error tests from semantics ported to lit.

    Did you look at the llvm/utils/update_XXX_test_checks.py scripts? We
    might want to add a scripts there if we can generate, not port, the
    CHECK lines.

    Cheers,
      Johannes

    > My next steps:
    >
    >   1.  Port remaining semantic test types
    >      *   Symbol
    >      *   Generic
    >      *   Modfile
    >   2.  Port decimal tests
    >   3.  Port evaluate tests
    >   4.  Port preprocessing tests (They aren’t in CMakeLists.txt, but are easily portable)
    > Each step will come as a pull request in a similar format:
    >
    >   1.  Script change to handle new test type
    >   2.  Ported tests added
    >      *   Either in 2 commits (verified, remaining – likely for changes with a large number of tests)
    >      *   Or all in 1 commit (small number of tests)
    >   3.  Old style tests removed
    > My initial PR is here<https://github.com/flang-compiler/f18/pull/934>.
    >
    > Thanks,
    > Luke
    >
    > Degree Apprentice,
    > Arm Manchester

    > _______________________________________________
    > flang-dev mailing list
    > mailto:flang-dev at lists.llvm.org
    > https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev

    --

    Johannes Doerfert
    Researcher

    Argonne National Laboratory
    Lemont, IL 60439, USA

    mailto:jdoerfert at anl.gov

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

_______________________________________________
flang-dev mailing list
mailto:flang-dev at lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory