[flang-dev] Porting F18 tests to use LLVM lit
Mehdi AMINI via flang-dev
flang-dev at lists.llvm.org
Fri Jan 17 08:39:54 PST 2020
MLIR is using the same principle: mlir-opt supports a --verify-diagnostic
flag to check for diagnostics.
Here is an example test:
https://github.com/llvm/llvm-project/blob/master/mlir/test/AffineOps/invalid.mlir
--
Mehdi
On Fri, Jan 17, 2020 at 12:42 AM Stephen Scalpone via flang-dev <
flang-dev at lists.llvm.org> wrote:
> Both swift [1] and clang [2] have built-in capabilities to validate
> semantic checks. These compilers, like f18 and GCC, use a special markup
> that is designed to verify diagnostics and to test semantic analysis.
>
>
>
> Here’s a sample line from a clang test [2]:
>
> template<> struct __is_floating_point_helper<__float128> {}; //
> expected-error {{__float128 is not supported on this target}}
>
>
>
> From swift [1]:
>
> private func publicReq() {} // expected-error {{method 'publicReq()'
> must be declared public because it matches a requirement in public protocol
> 'PublicProto'}} {{none}}
>
>
>
> Both clang and swift use lit.
>
>
>
> Here’s the lit line from the clang test [2]:
>
> // RUN: %clang_cc1 -verify -std=c++11 %s
>
>
>
> From swift [1]:
>
> // RUN: %target-typecheck-verify-swift -enable-objc-interop
> -disable-objc-attr-requires-foundation-module -swift-version 5
>
>
>
> FileCheck appears to be a very powerful tool; however, using FileCheck
> markup to verify diagnostics is not as straightforward as the markup used
> in swift and clang tests. [3]
>
>
>
> The f18 semantic tests and the f18 tools used for verifying diagnostics
> are similar in spirit to clang’s and swift’s. We could adapt the existing
> f18 test scripts, e.g. test_errors.sh, to work with lit. An alternative
> form of this proposal is to duplicate the f18 test scripts in python for
> portability.
>
>
>
> - Steve
>
>
>
> [1]
> https://github.com/apple/swift/blob/master/test/Sema/accessibility.swift
>
> [2]
> https://github.com/llvm/llvm-project/blob/master/clang/test/Sema/128bitfloat.cpp
>
> [3] https://github.com/flang-compiler/f18/pull/934
>
>
>
>
>
> *From: *Luke Ireland <Luke.Ireland at arm.com>
> *Date: *Thursday, January 16, 2020 at 5:08 AM
> *To: *Stephen Scalpone <sscalpone at nvidia.com>
> *Cc: *"flang-dev at lists.llvm.org" <flang-dev at lists.llvm.org>
> *Subject: *Re: [flang-dev] Porting F18 tests to use LLVM lit
>
>
>
> *External email: Use caution opening links or attachments*
>
>
>
> Hi Stephen,
>
>
>
> I don’t see any issue with choosing test names, I can confer if you like
> but the naming of the existing tests have never come into question before,
> so I doubt anyone would have any particular preferences. They’re in a
> separate folder, so make sure the test name isn’t used in the existing
> tests, then I can’t imagine any problems arising.
>
>
>
> And yes, when all tests are successfully ported, ‘test’ will be removed,
> and ‘test-lit’ will be renamed to ‘test’ taking its place, and the
> appropriate changes needed to incorporate this into make check-all will be
> needed as well, but should just be as simple as renaming again.
>
>
>
> Thanks,
>
> Luke
>
>
>
> Degree Apprentice,
>
> Arm Manchester
>
>
>
> *From: *Stephen Scalpone <sscalpone at nvidia.com>
> *Date: *Wednesday, 15 January 2020 at 23:51
> *To: *Luke Ireland <Luke.Ireland at arm.com>
> *Cc: *flang-dev at lists.llvm.org <flang-dev at lists.llvm.org>
> *Subject: *Re: [flang-dev] Porting F18 tests to use LLVM lit
>
> Hi Luke,
>
> During the transition, when writing a lit test, are there any special
> guidelines beyond choosing a distinct test name?
>
> After the final step of removing old tests, will you rename the test-lit
> directory to test?
>
> Thanks for working on this!
>
> - Steve
>
> On 1/15/20, 10:59 AM, "flang-dev on behalf of Doerfert, Johannes via
> flang-dev" <flang-dev-bounces at lists.llvm.org on behalf of
> flang-dev at lists.llvm.org> wrote:
>
> External email: Use caution opening links or attachments
>
>
> Thanks for working on this!
>
> On 01/15, Luke Ireland via flang-dev wrote:
> > Further to the conversation in llvm-dev<
> http://lists.llvm.org/pipermail/llvm-dev/2020-January/138282.html>,
> regarding LLVM testing infrastructure/lit, and the successfully merged
> patch from Carol<
> https://github.com/flang-compiler/f18/pull/861/commits/1049aeb81e0195d3fa1050b21e93a3980a0b7245>,
> we’re continuing development on adding this infrastructure by porting the
> existing tests to be compatible with lit.
> >
> > We’d like to propose the following approach:
> >
> > 1. Start off with a new directory, “test-lit” containing lit tests
> > 2. make check-all will run this and the old test directory
> together
> > 3. port over all old tests to this new directory as lit tests
> > 4. Once all tests are successfully ported, the old directory and
> test scripts are removed, and “test-lit” becomes “test”.
> > 5. All new tests will be created to be compatible with lit.
> > During this transition phase, tests can either be written in the old
> style, and then ported with my script, or preferably, written to work with
> lit.
>
> This sounds very reasonable.
>
> > My PR has three patches:
> >
> > 1. A script which ports old style tests to new lit tests. At the
> moment it only handles semantic error tests, i.e. tests run with
> check_error.sh
> > 2. The output of the porting script on a subset of the tests for
> verification. The subset of tests should cover all variations of error
> tests in my script
> > 3. All the remaining error tests from semantics ported to lit.
>
> Did you look at the llvm/utils/update_XXX_test_checks.py scripts? We
> might want to add a scripts there if we can generate, not port, the
> CHECK lines.
>
> Cheers,
> Johannes
>
> > My next steps:
> >
> > 1. Port remaining semantic test types
> > * Symbol
> > * Generic
> > * Modfile
> > 2. Port decimal tests
> > 3. Port evaluate tests
> > 4. Port preprocessing tests (They aren’t in CMakeLists.txt, but
> are easily portable)
> > Each step will come as a pull request in a similar format:
> >
> > 1. Script change to handle new test type
> > 2. Ported tests added
> > * Either in 2 commits (verified, remaining – likely for
> changes with a large number of tests)
> > * Or all in 1 commit (small number of tests)
> > 3. Old style tests removed
> > My initial PR is here<https://github.com/flang-compiler/f18/pull/934
> >.
> >
> > Thanks,
> > Luke
> >
> > Degree Apprentice,
> > Arm Manchester
>
> > _______________________________________________
> > flang-dev mailing list
> > flang-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev
>
>
> --
>
> Johannes Doerfert
> Researcher
>
> Argonne National Laboratory
> Lemont, IL 60439, USA
>
> jdoerfert at anl.gov
>
>
>
>
> -----------------------------------------------------------------------------------
> This email message is for the sole use of the intended recipient(s) and
> may contain
> confidential information. Any unauthorized review, use, disclosure or
> distribution
> is prohibited. If you are not the intended recipient, please contact the
> sender by
> reply email and destroy all copies of the original message.
>
> -----------------------------------------------------------------------------------
> _______________________________________________
> flang-dev mailing list
> flang-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/flang-dev/attachments/20200117/a6646e2e/attachment.html>
More information about the flang-dev
mailing list