<div dir="ltr"><div dir="ltr">MLIR is using the same principle: mlir-opt supports a --verify-diagnostic flag to check for diagnostics.<div>Here is an example test:</div><div><br></div><div><a href="https://github.com/llvm/llvm-project/blob/master/mlir/test/AffineOps/invalid.mlir" style="font-family:-webkit-standard">https://github.com/llvm/llvm-project/blob/master/mlir/test/AffineOps/invalid.mlir</a><br></div><div><br></div><div><br></div><div>-- </div><div>Mehdi</div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 17, 2020 at 12:42 AM Stephen Scalpone via flang-dev <<a href="mailto:flang-dev@lists.llvm.org">flang-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_-3632323114520922622WordSection1">
<p class="MsoNormal">Both swift [1] and clang [2] have built-in capabilities to validate semantic checks. These compilers, like f18 and GCC, use a special markup that is designed to verify diagnostics and to test semantic analysis.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Here’s a sample line from a clang test [2]:<u></u><u></u></p>
<p class="MsoNormal"> template<> struct __is_floating_point_helper<__float128> {}; // expected-error {{__float128 is not supported on this target}}<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">From swift [1]:<u></u><u></u></p>
<p class="MsoNormal"> private func publicReq() {} // expected-error {{method 'publicReq()' must be declared public because it matches a requirement in public protocol 'PublicProto'}} {{none}}
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Both clang and swift use lit.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Here’s the lit line from the clang test [2]:<u></u><u></u></p>
<p class="MsoNormal">// RUN: %clang_cc1 -verify -std=c++11 %s<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">From swift [1]:<u></u><u></u></p>
<p class="MsoNormal">// RUN: %target-typecheck-verify-swift -enable-objc-interop -disable-objc-attr-requires-foundation-module -swift-version 5<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">FileCheck appears to be a very powerful tool; however, using FileCheck markup to verify diagnostics is not as straightforward as the markup used in swift and clang tests. [3]<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">The f18 semantic tests and the f18 tools used for verifying diagnostics are similar in spirit to clang’s and swift’s. We could adapt the existing f18 test scripts, e.g. test_errors.sh, to work with lit. An alternative form of this proposal
is to duplicate the f18 test scripts in python for portability.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">- Steve<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">[1] <a href="https://github.com/apple/swift/blob/master/test/Sema/accessibility.swift" target="_blank">
https://github.com/apple/swift/blob/master/test/Sema/accessibility.swift</a><u></u><u></u></p>
<p class="MsoNormal">[2] <a href="https://github.com/llvm/llvm-project/blob/master/clang/test/Sema/128bitfloat.cpp" target="_blank">
https://github.com/llvm/llvm-project/blob/master/clang/test/Sema/128bitfloat.cpp</a><u></u><u></u></p>
<p class="MsoNormal">[3] <a href="https://github.com/flang-compiler/f18/pull/934" target="_blank">
https://github.com/flang-compiler/f18/pull/934</a><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-style:solid none none;border-top-width:1pt;border-top-color:rgb(181,196,223);padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From: </span></b><span style="font-size:12pt;color:black">Luke Ireland <<a href="mailto:Luke.Ireland@arm.com" target="_blank">Luke.Ireland@arm.com</a>><br>
<b>Date: </b>Thursday, January 16, 2020 at 5:08 AM<br>
<b>To: </b>Stephen Scalpone <<a href="mailto:sscalpone@nvidia.com" target="_blank">sscalpone@nvidia.com</a>><br>
<b>Cc: </b>"<a href="mailto:flang-dev@lists.llvm.org" target="_blank">flang-dev@lists.llvm.org</a>" <<a href="mailto:flang-dev@lists.llvm.org" target="_blank">flang-dev@lists.llvm.org</a>><br>
<b>Subject: </b>Re: [flang-dev] Porting F18 tests to use LLVM lit<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<table border="1" cellpadding="0" style="background-color:rgb(255,235,156);background-position:initial initial;background-repeat:initial initial">
<tbody>
<tr>
<td style="padding:0.75pt">
<p class="MsoNormal"><b><span style="font-size:7.5pt;font-family:Verdana,sans-serif;color:black">External email: Use caution opening links or attachments</span></b><span style="font-size:7.5pt;font-family:Verdana,sans-serif;color:black">
</span><u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Hi Stephen,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I don’t see any issue with choosing test names, I can confer if you like but the naming of the existing tests have never come into question before, so I doubt anyone would have any particular preferences. They’re in a separate folder, so
make sure the test name isn’t used in the existing tests, then I can’t imagine any problems arising.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">And yes, when all tests are successfully ported, ‘test’ will be removed, and ‘test-lit’ will be renamed to ‘test’ taking its place, and the appropriate changes needed to incorporate this into make check-all will be needed as well, but should
just be as simple as renaming again.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="color:black">Thanks,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:black">Luke</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:black"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:black">Degree Apprentice,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:black">Arm Manchester </span><u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div style="border-style:solid none none;border-top-width:1pt;border-top-color:rgb(181,196,223);padding:3pt 0in 0in">
<p class="MsoNormal" style="margin-bottom:12pt"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Stephen Scalpone <<a href="mailto:sscalpone@nvidia.com" target="_blank">sscalpone@nvidia.com</a>><br>
<b>Date: </b>Wednesday, 15 January 2020 at 23:51<br>
<b>To: </b>Luke Ireland <<a href="mailto:Luke.Ireland@arm.com" target="_blank">Luke.Ireland@arm.com</a>><br>
<b>Cc: </b><a href="mailto:flang-dev@lists.llvm.org" target="_blank">flang-dev@lists.llvm.org</a> <<a href="mailto:flang-dev@lists.llvm.org" target="_blank">flang-dev@lists.llvm.org</a>><br>
<b>Subject: </b>Re: [flang-dev] Porting F18 tests to use LLVM lit</span><u></u><u></u></p>
</div>
<p class="MsoNormal">Hi Luke,<br>
<br>
During the transition, when writing a lit test, are there any special guidelines beyond choosing a distinct test name?<br>
<br>
After the final step of removing old tests, will you rename the test-lit directory to test?<br>
<br>
Thanks for working on this!<br>
<br>
- Steve<br>
<br>
On 1/15/20, 10:59 AM, "flang-dev on behalf of Doerfert, Johannes via flang-dev" <<a href="mailto:flang-dev-bounces@lists.llvm.org" target="_blank">flang-dev-bounces@lists.llvm.org</a> on behalf of <a href="mailto:flang-dev@lists.llvm.org" target="_blank">flang-dev@lists.llvm.org</a>> wrote:<br>
<br>
External email: Use caution opening links or attachments<br>
<br>
<br>
Thanks for working on this!<br>
<br>
On 01/15, Luke Ireland via flang-dev wrote:<br>
> Further to the conversation in llvm-dev<<a href="http://lists.llvm.org/pipermail/llvm-dev/2020-January/138282.html" target="_blank">http://lists.llvm.org/pipermail/llvm-dev/2020-January/138282.html</a>>, regarding LLVM testing infrastructure/lit, and the successfully
merged patch from Carol<<a href="https://github.com/flang-compiler/f18/pull/861/commits/1049aeb81e0195d3fa1050b21e93a3980a0b7245" target="_blank">https://github.com/flang-compiler/f18/pull/861/commits/1049aeb81e0195d3fa1050b21e93a3980a0b7245</a>>, we’re continuing development
on adding this infrastructure by porting the existing tests to be compatible with lit.<br>
><br>
> We’d like to propose the following approach:<br>
><br>
> 1. Start off with a new directory, “test-lit” containing lit tests<br>
> 2. make check-all will run this and the old test directory together<br>
> 3. port over all old tests to this new directory as lit tests<br>
> 4. Once all tests are successfully ported, the old directory and test scripts are removed, and “test-lit” becomes “test”.<br>
> 5. All new tests will be created to be compatible with lit.<br>
> During this transition phase, tests can either be written in the old style, and then ported with my script, or preferably, written to work with lit.<br>
<br>
This sounds very reasonable.<br>
<br>
> My PR has three patches:<br>
><br>
> 1. A script which ports old style tests to new lit tests. At the moment it only handles semantic error tests, i.e. tests run with check_error.sh<br>
> 2. The output of the porting script on a subset of the tests for verification. The subset of tests should cover all variations of error tests in my script<br>
> 3. All the remaining error tests from semantics ported to lit.<br>
<br>
Did you look at the llvm/utils/update_XXX_test_checks.py scripts? We<br>
might want to add a scripts there if we can generate, not port, the<br>
CHECK lines.<br>
<br>
Cheers,<br>
Johannes<br>
<br>
> My next steps:<br>
><br>
> 1. Port remaining semantic test types<br>
> * Symbol<br>
> * Generic<br>
> * Modfile<br>
> 2. Port decimal tests<br>
> 3. Port evaluate tests<br>
> 4. Port preprocessing tests (They aren’t in CMakeLists.txt, but are easily portable)<br>
> Each step will come as a pull request in a similar format:<br>
><br>
> 1. Script change to handle new test type<br>
> 2. Ported tests added<br>
> * Either in 2 commits (verified, remaining – likely for changes with a large number of tests)<br>
> * Or all in 1 commit (small number of tests)<br>
> 3. Old style tests removed<br>
> My initial PR is here<<a href="https://github.com/flang-compiler/f18/pull/934" target="_blank">https://github.com/flang-compiler/f18/pull/934</a>>.<br>
><br>
> Thanks,<br>
> Luke<br>
><br>
> Degree Apprentice,<br>
> Arm Manchester<br>
<br>
> _______________________________________________<br>
> flang-dev mailing list<br>
> <a href="mailto:flang-dev@lists.llvm.org" target="_blank">flang-dev@lists.llvm.org</a><br>
> <a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev</a><br>
<br>
<br>
--<br>
<br>
Johannes Doerfert<br>
Researcher<br>
<br>
Argonne National Laboratory<br>
Lemont, IL 60439, USA<br>
<br>
<a href="mailto:jdoerfert@anl.gov" target="_blank">jdoerfert@anl.gov</a><br>
<br>
<br>
<br>
-----------------------------------------------------------------------------------<br>
This email message is for the sole use of the intended recipient(s) and may contain<br>
confidential information. Any unauthorized review, use, disclosure or distribution<br>
is prohibited. If you are not the intended recipient, please contact the sender by<br>
reply email and destroy all copies of the original message.<br>
-----------------------------------------------------------------------------------<u></u><u></u></p>
</div>
</div>
_______________________________________________<br>
flang-dev mailing list<br>
<a href="mailto:flang-dev@lists.llvm.org" target="_blank">flang-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev</a><br>
</blockquote></div>