<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
/* Page Definitions */
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="#0563C1" vlink="#954F72" style="text-justify-trim:punctuation">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I would like to discuss the behaviors of openmp `simd` and `ordered simd` directives. I think current Clang may not give expected results as OpenMP 5.0 standard defines.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Let's start one c++ example:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">void func(float *a, float *b, float *c, float *d, int N) {<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #pragma omp simd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> for (int i = 0; i < N; i++) {<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> d[i] = c[i] + 1.0;<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #pragma omp ordered simd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> a[i] = b[i] + 1.0;<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">What is expected according to OpenMP 5.0 standard is like the following:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">void func(float *a, float *b, float *c, float *d, int N) {<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> for (int i = 0; i < N; i += 4) {<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #pragma omp simd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> for (int j = i; j < 4; j++)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> d[i] = c[i] + 1.0; // vectorized<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> for (int j = i; j < 4; j++)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> a[i] = b[i] + 1.0; // not vectorized<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">It seems that current Clang and LLVM do not support it.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Without openmp enabled, clang vectorizes the loop with memcheck as follows:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ clang++ -O3 test.cpp -c -emit-llvm -S && cat test.ll<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %scevgep = getelementptr float, float* %d, i64 %wide.trip.count<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %scevgep22 = getelementptr float, float* %a, i64 %wide.trip.count<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %scevgep25 = getelementptr float, float* %c, i64 %wide.trip.count<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %scevgep28 = getelementptr float, float* %b, i64 %wide.trip.count<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %bound030 = icmp ugt float* %scevgep25, %d<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %bound131 = icmp ugt float* %scevgep, %c<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %found.conflict32 = and i1 %bound030, %bound131<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> ... fadd <4 x float> ...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">With openmp-simd enabled, clang vectorizes the loop without memcheck. This means that only `simd` directive is enabled, while `ordered simd` directive is disabled. The results are expected.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">clang++ -fopenmp-simd -O3 test.cpp -c -emit-llvm -S && cat test.ll<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">With openmp enabled, both `simd` and `ordered simd` directives are enabled. Clang frontend generates the outlined function `captured_stmt(float** %a.addr, i32* %i3, float** %b.addr)` with `AlwaysInline` attribute when
optimization level is more than 0. The generated IR is to vectorize the loop with memcheck as follows:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ clang++ -fopenmp -O3 test.cpp -c -emit-llvm -S && cat test.ll<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %scevgep = getelementptr float, float* %d, i64 %wide.trip.count<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %scevgep29 = getelementptr float, float* %a, i64 %wide.trip.count<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %scevgep32 = getelementptr float, float* %c, i64 %wide.trip.count<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %scevgep35 = getelementptr float, float* %b, i64 %wide.trip.count<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %bound037 = icmp ugt float* %scevgep32, %d<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %bound138 = icmp ugt float* %scevgep, %c<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %found.conflict39 = and i1 %bound037, %bound138<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> ... fadd <4 x float> ...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">But the expected IR should be like the following:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %scevgep29 = getelementptr float, float* %a, i64 %wide.trip.count<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %scevgep35 = getelementptr float, float* %b, i64 %wide.trip.count<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %found.conflict...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> ... fadd <4 x float> ...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I have two questions here:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">1. Does the outlined function `captured_stmt(float** %a.addr, i32* %i3, float** %b.addr)` with `AlwaysInline` attribute cause the memcheck? And how?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2. If my understanding is correct according to the above analysis, should the codegen of `ordered simd` directive be fixed to support the expected behaviors? And should `memcheck` function (emitMemRuntimeChecks) also
support partial check instead of the whole region inside the loop?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Also, for the following test case, both of vectorization of `d[i] = c[i] + 1.0;` and `a[i] = a[i-1] + 1.0;` are disabled.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">void func(float *a, float *b, float *c, float *d, int N) {<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #pragma omp simd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> for (int i = 1; i < N; i++) {<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> d[i] = c[i] + 1.0;<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #pragma omp ordered simd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> a[i] = a[i-1] + 1.0;<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">What is expected is to vectorize the statement `d[i] = c[i] + 1.0;`.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I also test icc and gcc and here are the results:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ icc -v<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">icc version 2021.1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ icc -qopenmp test.cpp -O3 -qopt-report -qopt-report-phase=vec -S && cat test.optrpt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">LOOP BEGIN at test.cpp(3,3)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> remark #15531: Block of statements was serialized due to user request [ test.cpp(5,5) ]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> remark #15301: SIMD LOOP WAS VECTORIZED<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">LOOP END<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ g++ -v<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">gcc version 9.3.0 (GCC)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ g++ test.cpp -fopenmp -fdump-tree-all -fdump-rtl-all -O3 -ftree-vectorize -S && cat test.s<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">fadd s0, s0, s1 // not vectorized<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">fadd s0, s0, s1 // not vectorized<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">// There is `GOMP_SIMD_ORDERED_START` and `GOMP_SIMD_ORDERED_END` before and after the statement of `a[i] = a[i-1] + 1.0` in ifcvt pass, after which they are used in vect pass to break the vectorization.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">For the following test case:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">void func(float *b, float *c, float *d, int N) {<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> float a[N];<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> for (int i = 0; i < N; i++)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> a[i] = 0;<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #pragma omp simd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> for (int i = 1; i < N; i++) {<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> d[i] = c[i] + 1.0;<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #pragma omp ordered simd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> a[i] = a[i-1] + 1.0;<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">The IR generated is as follows:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ clang++ -fopenmp -O3 test.cpp -c -emit-llvm -S<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %scevgep = getelementptr float, float* %d, i64 1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %3 = add nuw nsw i64 %wide.trip.count, 1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %scevgep41 = getelementptr float, float* %d, i64 %3<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %scevgep43 = getelementptr float, float* %c, i64 1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %scevgep45 = getelementptr float, float* %c, i64 %3<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %bound0 = icmp ult float* %scevgep, %scevgep45<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %bound1 = icmp ult float* %scevgep43, %scevgep41<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %found.conflict = and i1 %bound0, %bound1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %induction = fadd <4 x float> %.splat, <float 0.000000e+00, float 1.000000e+00, float 2.000000e+00, float 3.000000e+00><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">The result for the statement of `d[i] = c[i] + 1.0` and `a[i] = a[i-1] + 1.0` are both unexpected. It is safe to vectorize the statement of `a[i] = a[i-1] + 1.0` although it violates the definition of ordered construct
in OpenMP 5.0 standard. But the memcheck of variables `d` and `c` should not be correct as the `simd` directive is there.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">All the best,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Peixin<o:p></o:p></span></p>
</div>
</body>
</html>