<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Consolas",serif;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">This change should proceed in two steps:<o:p></o:p></p>
<p class="MsoNormal">1. Run tests with llvm-lit instead of ctest but don't change how tests are checked.<o:p></o:p></p>
<p class="MsoNormal">2. Replace custom checking of results with FileCheck where appropriate.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The first step should be relatively straight-forward. Once this transition is complete the rest can happen incrementally.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The second step is more complicated because there are several different kinds of tests with their own requirements for checking the results. The folding tests might be simplified by using FileCheck. I don't know how .mod file tests and
symbol tests can use FileCheck (but my knowledge of FileCheck is limited). There is on-going discussion in PR #934 about how to use FileCheck on the error tests.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Obviously the overriding consideration is making it easy to write high quality tests, run them, and diagnose their failures. Anything we can do to improve that will pay off greatly in the future.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Tim<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">flang-dev <flang-dev-bounces@lists.llvm.org> on behalf of "Finkel, Hal J. via flang-dev" <flang-dev@lists.llvm.org><br>
<b>Reply-To: </b>"Finkel, Hal J." <hfinkel@anl.gov><br>
<b>Date: </b>Friday, January 17, 2020 at 4:41 AM<br>
<b>To: </b>Stephen Scalpone <sscalpone@nvidia.com>, Luke Ireland <Luke.Ireland@arm.com><br>
<b>Cc: </b>"flang-dev@lists.llvm.org" <flang-dev@lists.llvm.org><br>
<b>Subject: </b>Re: [flang-dev] Porting F18 tests to use LLVM lit<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<table class="MsoNormalTable" border="1" cellpadding="0" style="background:#FFEB9C">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:black">External email: Use caution opening links or attachments</span></b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:black">
</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 1/17/20 2:42 AM, Stephen Scalpone via flang-dev wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Both swift [1] and clang [2] have built-in capabilities to validate semantic checks. These compilers, like f18 and GCC, use a special markup that is designed to verify diagnostics and to test semantic analysis.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Here’s a sample line from a clang test [2]:<o:p></o:p></p>
<p class="MsoNormal"> template<> struct __is_floating_point_helper<__float128> {}; // expected-error {{__float128 is not supported on this target}}<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">From swift [1]:<o:p></o:p></p>
<p class="MsoNormal"> private func publicReq() {} // expected-error {{method 'publicReq()' must be declared public because it matches a requirement in public protocol 'PublicProto'}} {{none}}
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Both clang and swift use lit.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Here’s the lit line from the clang test [2]:<o:p></o:p></p>
<p class="MsoNormal">// RUN: %clang_cc1 -verify -std=c++11 %s<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">From swift [1]:<o:p></o:p></p>
<p class="MsoNormal">// RUN: %target-typecheck-verify-swift -enable-objc-interop -disable-objc-attr-requires-foundation-module -swift-version 5<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">FileCheck appears to be a very powerful tool; however, using FileCheck markup to verify diagnostics is not as straightforward as the markup used in swift and clang tests. [3]<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">The f18 semantic tests and the f18 tools used for verifying diagnostics are similar in spirit to clang’s and swift’s. We could adapt the existing f18 test scripts, e.g. test_errors.sh, to work with lit.
<o:p></o:p></p>
</blockquote>
<p><o:p> </o:p></p>
<p>I believe that we should use FileCheck where possible.<o:p></o:p></p>
<p> -Hal<o:p></o:p></p>
<p><o:p> </o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">An alternative form of this proposal is to duplicate the f18 test scripts in python for portability.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">- Steve<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">[1] <a href="https://github.com/apple/swift/blob/master/test/Sema/accessibility.swift">
https://github.com/apple/swift/blob/master/test/Sema/accessibility.swift</a><o:p></o:p></p>
<p class="MsoNormal">[2] <a href="https://github.com/llvm/llvm-project/blob/master/clang/test/Sema/128bitfloat.cpp">
https://github.com/llvm/llvm-project/blob/master/clang/test/Sema/128bitfloat.cpp</a><o:p></o:p></p>
<p class="MsoNormal">[3] <a href="https://github.com/flang-compiler/f18/pull/934">
https://github.com/flang-compiler/f18/pull/934</a><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Luke Ireland
<a href="mailto:Luke.Ireland@arm.com"><Luke.Ireland@arm.com></a><br>
<b>Date: </b>Thursday, January 16, 2020 at 5:08 AM<br>
<b>To: </b>Stephen Scalpone <a href="mailto:sscalpone@nvidia.com"><sscalpone@nvidia.com></a><br>
<b>Cc: </b><a href="mailto:flang-dev@lists.llvm.org">"flang-dev@lists.llvm.org"</a>
<a href="mailto:flang-dev@lists.llvm.org"><flang-dev@lists.llvm.org></a><br>
<b>Subject: </b>Re: [flang-dev] Porting F18 tests to use LLVM lit</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<table class="MsoNormalTable" border="1" cellpadding="0" style="background:#FFEB9C">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:black">External email: Use caution opening links or attachments</span></b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:black">
</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="color:white"> </span><o:p></o:p></p>
<p class="MsoNormal">Hi Stephen,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I don’t see any issue with choosing test names, I can confer if you like but the naming of the existing tests have never come into question before, so I doubt anyone would have any particular preferences. They’re in a separate folder, so
make sure the test name isn’t used in the existing tests, then I can’t imagine any problems arising.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">And yes, when all tests are successfully ported, ‘test’ will be removed, and ‘test-lit’ will be renamed to ‘test’ taking its place, and the appropriate changes needed to incorporate this into make check-all will be needed as well, but should
just be as simple as renaming again.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="color:black">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">Luke</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">Degree Apprentice,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">Arm Manchester </span><o:p></o:p></p>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Stephen Scalpone <a href="mailto:sscalpone@nvidia.com">
<sscalpone@nvidia.com></a><br>
<b>Date: </b>Wednesday, 15 January 2020 at 23:51<br>
<b>To: </b>Luke Ireland <a href="mailto:Luke.Ireland@arm.com"><Luke.Ireland@arm.com></a><br>
<b>Cc: </b><a href="mailto:flang-dev@lists.llvm.org">flang-dev@lists.llvm.org</a>
<a href="mailto:flang-dev@lists.llvm.org"><flang-dev@lists.llvm.org></a><br>
<b>Subject: </b>Re: [flang-dev] Porting F18 tests to use LLVM lit</span><o:p></o:p></p>
</div>
<p class="MsoNormal">Hi Luke,<br>
<br>
During the transition, when writing a lit test, are there any special guidelines beyond choosing a distinct test name?<br>
<br>
After the final step of removing old tests, will you rename the test-lit directory to test?<br>
<br>
Thanks for working on this!<br>
<br>
- Steve<br>
<br>
On 1/15/20, 10:59 AM, "flang-dev on behalf of Doerfert, Johannes via flang-dev" <a href="mailto:flang-dev-bounces@lists.llvm.orgonbehalfofflang-dev@lists.llvm.org">
<flang-dev-bounces@lists.llvm.org on behalf of flang-dev@lists.llvm.org></a> wrote:<br>
<br>
External email: Use caution opening links or attachments<br>
<br>
<br>
Thanks for working on this!<br>
<br>
On 01/15, Luke Ireland via flang-dev wrote:<br>
> Further to the conversation in llvm-dev<<a href="http://lists.llvm.org/pipermail/llvm-dev/2020-January/138282.html">http://lists.llvm.org/pipermail/llvm-dev/2020-January/138282.html</a>>, regarding LLVM testing infrastructure/lit, and the successfully
merged patch from Carol<<a href="https://github.com/flang-compiler/f18/pull/861/commits/1049aeb81e0195d3fa1050b21e93a3980a0b7245">https://github.com/flang-compiler/f18/pull/861/commits/1049aeb81e0195d3fa1050b21e93a3980a0b7245</a>>, we’re continuing development
on adding this infrastructure by porting the existing tests to be compatible with lit.<br>
><br>
> We’d like to propose the following approach:<br>
><br>
> 1. Start off with a new directory, “test-lit” containing lit tests<br>
> 2. make check-all will run this and the old test directory together<br>
> 3. port over all old tests to this new directory as lit tests<br>
> 4. Once all tests are successfully ported, the old directory and test scripts are removed, and “test-lit” becomes “test”.<br>
> 5. All new tests will be created to be compatible with lit.<br>
> During this transition phase, tests can either be written in the old style, and then ported with my script, or preferably, written to work with lit.<br>
<br>
This sounds very reasonable.<br>
<br>
> My PR has three patches:<br>
><br>
> 1. A script which ports old style tests to new lit tests. At the moment it only handles semantic error tests, i.e. tests run with check_error.sh<br>
> 2. The output of the porting script on a subset of the tests for verification. The subset of tests should cover all variations of error tests in my script<br>
> 3. All the remaining error tests from semantics ported to lit.<br>
<br>
Did you look at the llvm/utils/update_XXX_test_checks.py scripts? We<br>
might want to add a scripts there if we can generate, not port, the<br>
CHECK lines.<br>
<br>
Cheers,<br>
Johannes<br>
<br>
> My next steps:<br>
><br>
> 1. Port remaining semantic test types<br>
> * Symbol<br>
> * Generic<br>
> * Modfile<br>
> 2. Port decimal tests<br>
> 3. Port evaluate tests<br>
> 4. Port preprocessing tests (They aren’t in CMakeLists.txt, but are easily portable)<br>
> Each step will come as a pull request in a similar format:<br>
><br>
> 1. Script change to handle new test type<br>
> 2. Ported tests added<br>
> * Either in 2 commits (verified, remaining – likely for changes with a large number of tests)<br>
> * Or all in 1 commit (small number of tests)<br>
> 3. Old style tests removed<br>
> My initial PR is here<<a href="https://github.com/flang-compiler/f18/pull/934">https://github.com/flang-compiler/f18/pull/934</a>>.<br>
><br>
> Thanks,<br>
> Luke<br>
><br>
> Degree Apprentice,<br>
> Arm Manchester<br>
<br>
> _______________________________________________<br>
> flang-dev mailing list<br>
> <a href="mailto:flang-dev@lists.llvm.org">flang-dev@lists.llvm.org</a><br>
> <a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev">https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev</a><br>
<br>
<br>
--<br>
<br>
Johannes Doerfert<br>
Researcher<br>
<br>
Argonne National Laboratory<br>
Lemont, IL 60439, USA<br>
<br>
<a href="mailto:jdoerfert@anl.gov">jdoerfert@anl.gov</a><br>
<br>
<br>
<br>
-----------------------------------------------------------------------------------<br>
This email message is for the sole use of the intended recipient(s) and may contain<br>
confidential information. Any unauthorized review, use, disclosure or distribution<br>
is prohibited. If you are not the intended recipient, please contact the sender by<br>
reply email and destroy all copies of the original message.<br>
-----------------------------------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>flang-dev mailing list<o:p></o:p></pre>
<pre><a href="mailto:flang-dev@lists.llvm.org">flang-dev@lists.llvm.org</a><o:p></o:p></pre>
<pre><a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev">https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev</a><o:p></o:p></pre>
</blockquote>
<pre>-- <o:p></o:p></pre>
<pre>Hal Finkel<o:p></o:p></pre>
<pre>Lead, Compiler Technology and Programming Languages<o:p></o:p></pre>
<pre>Leadership Computing Facility<o:p></o:p></pre>
<pre>Argonne National Laboratory<o:p></o:p></pre>
</div>
</div>
</body>
</html>