<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">On 1/17/20 2:42 AM, Stephen Scalpone via flang-dev wrote:<br>
</div>
<blockquote type="cite" cite="mid:638E8098-A54F-4F7B-959C-6D8258334687@nvidia.com">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1930499754;
mso-list-type:hybrid;
mso-list-template-ids:-1199000868 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">Both swift [1] and clang [2] have built-in capabilities to validate semantic checks. These compilers, like f18 and GCC, use a special markup that is designed to verify diagnostics and to test semantic analysis.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here’s a sample line from a clang test [2]:<o:p></o:p></p>
<p class="MsoNormal"> template<> struct __is_floating_point_helper<__float128> {}; // expected-error {{__float128 is not supported on this target}}<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">From swift [1]:<o:p></o:p></p>
<p class="MsoNormal"> private func publicReq() {} // expected-error {{method 'publicReq()' must be declared public because it matches a requirement in public protocol 'PublicProto'}} {{none}}
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Both clang and swift use lit.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here’s the lit line from the clang test [2]:<o:p></o:p></p>
<p class="MsoNormal">// RUN: %clang_cc1 -verify -std=c++11 %s<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">From swift [1]:<o:p></o:p></p>
<p class="MsoNormal">// RUN: %target-typecheck-verify-swift -enable-objc-interop -disable-objc-attr-requires-foundation-module -swift-version 5<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">FileCheck appears to be a very powerful tool; however, using FileCheck markup to verify diagnostics is not as straightforward as the markup used in swift and clang tests. [3]<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The f18 semantic tests and the f18 tools used for verifying diagnostics are similar in spirit to clang’s and swift’s. We could adapt the existing f18 test scripts, e.g. test_errors.sh, to work with lit.
</p>
</div>
</blockquote>
<p><br>
</p>
<p>I believe that we should use FileCheck where possible.</p>
<p> -Hal<br>
</p>
<p><br>
</p>
<blockquote type="cite" cite="mid:638E8098-A54F-4F7B-959C-6D8258334687@nvidia.com">
<div class="WordSection1">
<p class="MsoNormal">An alternative form of this proposal is to duplicate the f18 test scripts in python for portability.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">- Steve<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[1] <a href="https://github.com/apple/swift/blob/master/test/Sema/accessibility.swift" moz-do-not-send="true">
https://github.com/apple/swift/blob/master/test/Sema/accessibility.swift</a><o:p></o:p></p>
<p class="MsoNormal">[2] <a href="https://github.com/llvm/llvm-project/blob/master/clang/test/Sema/128bitfloat.cpp" moz-do-not-send="true">
https://github.com/llvm/llvm-project/blob/master/clang/test/Sema/128bitfloat.cpp</a><o:p></o:p></p>
<p class="MsoNormal">[3] <a href="https://github.com/flang-compiler/f18/pull/934" moz-do-not-send="true">
https://github.com/flang-compiler/f18/pull/934</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Luke Ireland
<a class="moz-txt-link-rfc2396E" href="mailto:Luke.Ireland@arm.com"><Luke.Ireland@arm.com></a><br>
<b>Date: </b>Thursday, January 16, 2020 at 5:08 AM<br>
<b>To: </b>Stephen Scalpone <a class="moz-txt-link-rfc2396E" href="mailto:sscalpone@nvidia.com">
<sscalpone@nvidia.com></a><br>
<b>Cc: </b><a class="moz-txt-link-rfc2396E" href="mailto:flang-dev@lists.llvm.org">"flang-dev@lists.llvm.org"</a>
<a class="moz-txt-link-rfc2396E" href="mailto:flang-dev@lists.llvm.org"><flang-dev@lists.llvm.org></a><br>
<b>Subject: </b>Re: [flang-dev] Porting F18 tests to use LLVM lit<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<table class="MsoNormalTable" style="background:#FFEB9C" cellpadding="0" border="1">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:black">External email: Use caution opening links or attachments</span></b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:black">
</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi Stephen,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I don’t see any issue with choosing test names, I can confer if you like but the naming of the existing tests have never come into question before, so I doubt anyone would have any particular preferences. They’re in a separate folder, so
make sure the test name isn’t used in the existing tests, then I can’t imagine any problems arising.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">And yes, when all tests are successfully ported, ‘test’ will be removed, and ‘test-lit’ will be renamed to ‘test’ taking its place, and the appropriate changes needed to incorporate this into make check-all will be needed as well, but should
just be as simple as renaming again.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="color:black">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">Luke</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">Degree Apprentice,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">Arm Manchester </span><o:p></o:p></p>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Stephen Scalpone <a class="moz-txt-link-rfc2396E" href="mailto:sscalpone@nvidia.com">
<sscalpone@nvidia.com></a><br>
<b>Date: </b>Wednesday, 15 January 2020 at 23:51<br>
<b>To: </b>Luke Ireland <a class="moz-txt-link-rfc2396E" href="mailto:Luke.Ireland@arm.com">
<Luke.Ireland@arm.com></a><br>
<b>Cc: </b><a class="moz-txt-link-abbreviated" href="mailto:flang-dev@lists.llvm.org">flang-dev@lists.llvm.org</a>
<a class="moz-txt-link-rfc2396E" href="mailto:flang-dev@lists.llvm.org"><flang-dev@lists.llvm.org></a><br>
<b>Subject: </b>Re: [flang-dev] Porting F18 tests to use LLVM lit</span><o:p></o:p></p>
</div>
<p class="MsoNormal">Hi Luke,<br>
<br>
During the transition, when writing a lit test, are there any special guidelines beyond choosing a distinct test name?<br>
<br>
After the final step of removing old tests, will you rename the test-lit directory to test?<br>
<br>
Thanks for working on this!<br>
<br>
- Steve<br>
<br>
On 1/15/20, 10:59 AM, "flang-dev on behalf of Doerfert, Johannes via flang-dev" <a class="moz-txt-link-rfc2396E" href="mailto:flang-dev-bounces@lists.llvm.orgonbehalfofflang-dev@lists.llvm.org">
<flang-dev-bounces@lists.llvm.org on behalf of flang-dev@lists.llvm.org></a> wrote:<br>
<br>
External email: Use caution opening links or attachments<br>
<br>
<br>
Thanks for working on this!<br>
<br>
On 01/15, Luke Ireland via flang-dev wrote:<br>
> Further to the conversation in llvm-dev<<a href="http://lists.llvm.org/pipermail/llvm-dev/2020-January/138282.html" moz-do-not-send="true">http://lists.llvm.org/pipermail/llvm-dev/2020-January/138282.html</a>>, regarding LLVM testing infrastructure/lit,
and the successfully merged patch from Carol<<a href="https://github.com/flang-compiler/f18/pull/861/commits/1049aeb81e0195d3fa1050b21e93a3980a0b7245" moz-do-not-send="true">https://github.com/flang-compiler/f18/pull/861/commits/1049aeb81e0195d3fa1050b21e93a3980a0b7245</a>>,
we’re continuing development on adding this infrastructure by porting the existing tests to be compatible with lit.<br>
><br>
> We’d like to propose the following approach:<br>
><br>
> 1. Start off with a new directory, “test-lit” containing lit tests<br>
> 2. make check-all will run this and the old test directory together<br>
> 3. port over all old tests to this new directory as lit tests<br>
> 4. Once all tests are successfully ported, the old directory and test scripts are removed, and “test-lit” becomes “test”.<br>
> 5. All new tests will be created to be compatible with lit.<br>
> During this transition phase, tests can either be written in the old style, and then ported with my script, or preferably, written to work with lit.<br>
<br>
This sounds very reasonable.<br>
<br>
> My PR has three patches:<br>
><br>
> 1. A script which ports old style tests to new lit tests. At the moment it only handles semantic error tests, i.e. tests run with check_error.sh<br>
> 2. The output of the porting script on a subset of the tests for verification. The subset of tests should cover all variations of error tests in my script<br>
> 3. All the remaining error tests from semantics ported to lit.<br>
<br>
Did you look at the llvm/utils/update_XXX_test_checks.py scripts? We<br>
might want to add a scripts there if we can generate, not port, the<br>
CHECK lines.<br>
<br>
Cheers,<br>
Johannes<br>
<br>
> My next steps:<br>
><br>
> 1. Port remaining semantic test types<br>
> * Symbol<br>
> * Generic<br>
> * Modfile<br>
> 2. Port decimal tests<br>
> 3. Port evaluate tests<br>
> 4. Port preprocessing tests (They aren’t in CMakeLists.txt, but are easily portable)<br>
> Each step will come as a pull request in a similar format:<br>
><br>
> 1. Script change to handle new test type<br>
> 2. Ported tests added<br>
> * Either in 2 commits (verified, remaining – likely for changes with a large number of tests)<br>
> * Or all in 1 commit (small number of tests)<br>
> 3. Old style tests removed<br>
> My initial PR is here<<a href="https://github.com/flang-compiler/f18/pull/934" moz-do-not-send="true">https://github.com/flang-compiler/f18/pull/934</a>>.<br>
><br>
> Thanks,<br>
> Luke<br>
><br>
> Degree Apprentice,<br>
> Arm Manchester<br>
<br>
> _______________________________________________<br>
> flang-dev mailing list<br>
> <a class="moz-txt-link-abbreviated" href="mailto:flang-dev@lists.llvm.org">
flang-dev@lists.llvm.org</a><br>
> <a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev" moz-do-not-send="true">
https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev</a><br>
<br>
<br>
--<br>
<br>
Johannes Doerfert<br>
Researcher<br>
<br>
Argonne National Laboratory<br>
Lemont, IL 60439, USA<br>
<br>
<a class="moz-txt-link-abbreviated" href="mailto:jdoerfert@anl.gov">jdoerfert@anl.gov</a><br>
<br>
<br>
<br>
-----------------------------------------------------------------------------------<br>
This email message is for the sole use of the intended recipient(s) and may contain<br>
confidential information. Any unauthorized review, use, disclosure or distribution<br>
is prohibited. If you are not the intended recipient, please contact the sender by<br>
reply email and destroy all copies of the original message.<br>
-----------------------------------------------------------------------------------<o:p></o:p></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
flang-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:flang-dev@lists.llvm.org">flang-dev@lists.llvm.org</a>
<a class="moz-txt-link-freetext" href="https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev">https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory</pre>
</body>
</html>