<div dir="ltr">Is this system very slow in file write? I think the test tries to dump the profile while the child process still write the file.<div>I have a sleep(1) b/w the end of main process and profile dump. It seems this is no enough.</div><div><br></div><div>Is it possible to get a temporal access to this machine?</div><div>If not, I can reduce the profile size or have a longer wait period before profile dump.</div><div><br></div><div>-Rong</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 17, 2017 at 5:55 AM, Diana Picus <span dir="ltr"><<a href="mailto:diana.picus@linaro.org" target="_blank">diana.picus@linaro.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
This test has been failing every now and then on one of our bots:<br>
<a href="http://lab.llvm.org:8011/waterfall?show=clang-cmake-aarch64-42vma" rel="noreferrer" target="_blank">http://lab.llvm.org:8011/<wbr>waterfall?show=clang-cmake-<wbr>aarch64-42vma</a><br>
(builds 4700, 4695, 4690. 4671 etc)<br>
<br>
Any idea what the problem might be?<br>
<br>
Thanks,<br>
Diana<br>
<br>
On 16 February 2017 at 20:21, Rong Xu via llvm-commits<br>
<<a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a>> wrote:<br>
> Author: xur<br>
> Date: Thu Feb 16 13:21:31 2017<br>
> New Revision: 295364<br>
><br>
> URL: <a href="http://llvm.org/viewvc/llvm-project?rev=295364&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project?rev=295364&view=rev</a><br>
> Log:<br>
> [PGO] Suspend SIGKILL for PR_SET_PDEATHSIG in profile-write<br>
><br>
> Summary:<br>
> We found a nondeterministic behavior when doing online profile merging<br>
> for multi-process applications. The application forks a sub-process and<br>
> sub-process sets to get SIGKILL when the parent process exits,<br>
><br>
> The first process gets the lock, and dumps the profile. The second one<br>
> will mmap the file, do the merge and write out the file. Note that before<br>
> the merged write, we truncate the profile.<br>
><br>
> Depending on the timing, the child process might be terminated<br>
> abnormally when the parent exits first. If this happens:<br>
> (1) before the truncation, we will get the profile for the main process<br>
> (2) after the truncation, and before write-out the profile, we will get<br>
> 0 size profile.<br>
> (3) after the merged write, we get merged profile.<br>
><br>
> This patch temporarily suspend the SIGKILL for PR_SET_PDEATHSIG<br>
> before profile-write and restore it after the write.<br>
><br>
> This patch only applies to Linux system.<br>
><br>
> Reviewers: davidxl<br>
><br>
> Reviewed By: davidxl<br>
><br>
> Subscribers: xur, llvm-commits<br>
><br>
> Differential Revision: <a href="https://reviews.llvm.org/D29954" rel="noreferrer" target="_blank">https://reviews.llvm.org/<wbr>D29954</a><br>
><br>
> Added:<br>
> compiler-rt/trunk/test/<wbr>profile/Linux/prctl.c<br>
> Modified:<br>
> compiler-rt/trunk/lib/profile/<wbr>InstrProfilingFile.c<br>
> compiler-rt/trunk/lib/profile/<wbr>InstrProfilingUtil.c<br>
> compiler-rt/trunk/lib/profile/<wbr>InstrProfilingUtil.h<br>
><br>
> Modified: compiler-rt/trunk/lib/profile/<wbr>InstrProfilingFile.c<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/profile/InstrProfilingFile.c?rev=295364&r1=295363&r2=295364&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/compiler-rt/trunk/lib/<wbr>profile/InstrProfilingFile.c?<wbr>rev=295364&r1=295363&r2=<wbr>295364&view=diff</a><br>
> ==============================<wbr>==============================<wbr>==================<br>
> --- compiler-rt/trunk/lib/profile/<wbr>InstrProfilingFile.c (original)<br>
> +++ compiler-rt/trunk/lib/profile/<wbr>InstrProfilingFile.c Thu Feb 16 13:21:31 2017<br>
> @@ -530,6 +530,7 @@ int __llvm_profile_write_file(<wbr>void) {<br>
> int rc, Length;<br>
> const char *Filename;<br>
> char *FilenameBuf;<br>
> + int PDeathSig = 0;<br>
><br>
> if (lprofProfileDumped()) {<br>
> PROF_NOTE("Profile data not written to file: %s.\n",<br>
> @@ -556,10 +557,18 @@ int __llvm_profile_write_file(<wbr>void) {<br>
> return -1;<br>
> }<br>
><br>
> + // Temporarily suspend getting SIGKILL when the parent exits.<br>
> + PDeathSig = lprofSuspendSigKill();<br>
> +<br>
> /* Write profile data to the file. */<br>
> rc = writeFile(Filename);<br>
> if (rc)<br>
> PROF_ERR("Failed to write file \"%s\": %s\n", Filename, strerror(errno));<br>
> +<br>
> + // Restore SIGKILL.<br>
> + if (PDeathSig == 1)<br>
> + lprofRestoreSigKill();<br>
> +<br>
> return rc;<br>
> }<br>
><br>
><br>
> Modified: compiler-rt/trunk/lib/profile/<wbr>InstrProfilingUtil.c<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/profile/InstrProfilingUtil.c?rev=295364&r1=295363&r2=295364&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/compiler-rt/trunk/lib/<wbr>profile/InstrProfilingUtil.c?<wbr>rev=295364&r1=295363&r2=<wbr>295364&view=diff</a><br>
> ==============================<wbr>==============================<wbr>==================<br>
> --- compiler-rt/trunk/lib/profile/<wbr>InstrProfilingUtil.c (original)<br>
> +++ compiler-rt/trunk/lib/profile/<wbr>InstrProfilingUtil.c Thu Feb 16 13:21:31 2017<br>
> @@ -29,6 +29,11 @@<br>
> #include <stdlib.h><br>
> #include <string.h><br>
><br>
> +#if defined(__linux__)<br>
> +#include <signal.h><br>
> +#include <sys/prctl.h><br>
> +#endif<br>
> +<br>
> COMPILER_RT_VISIBILITY<br>
> void __llvm_profile_recursive_<wbr>mkdir(char *path) {<br>
> int i;<br>
> @@ -219,3 +224,24 @@ COMPILER_RT_VISIBILITY const char *lprof<br>
> #endif<br>
> return Sep;<br>
> }<br>
> +<br>
> +COMPILER_RT_VISIBILITY int lprofSuspendSigKill() {<br>
> +#if defined(__linux__)<br>
> + int PDeachSig = 0;<br>
> + /* Temporarily suspend getting SIGKILL upon exit of the parent process. */<br>
> + if (prctl(PR_GET_PDEATHSIG, &PDeachSig) == 0 && PDeachSig == SIGKILL) {<br>
> + fprintf(stderr, "set\n");<br>
> + prctl(PR_SET_PDEATHSIG, 0);<br>
> + }<br>
> + return (PDeachSig == SIGKILL);<br>
> +#else<br>
> + return 0;<br>
> +#endif<br>
> +}<br>
> +<br>
> +COMPILER_RT_VISIBILITY void lprofRestoreSigKill() {<br>
> +#if defined(__linux__)<br>
> + fprintf(stderr, "restore \n");<br>
> + prctl(PR_SET_PDEATHSIG, SIGKILL);<br>
> +#endif<br>
> +}<br>
><br>
> Modified: compiler-rt/trunk/lib/profile/<wbr>InstrProfilingUtil.h<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/profile/InstrProfilingUtil.h?rev=295364&r1=295363&r2=295364&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/compiler-rt/trunk/lib/<wbr>profile/InstrProfilingUtil.h?<wbr>rev=295364&r1=295363&r2=<wbr>295364&view=diff</a><br>
> ==============================<wbr>==============================<wbr>==================<br>
> --- compiler-rt/trunk/lib/profile/<wbr>InstrProfilingUtil.h (original)<br>
> +++ compiler-rt/trunk/lib/profile/<wbr>InstrProfilingUtil.h Thu Feb 16 13:21:31 2017<br>
> @@ -51,4 +51,12 @@ int lprofGetHostName(char *Name, int Len<br>
> unsigned lprofBoolCmpXchg(void **Ptr, void *OldV, void *NewV);<br>
> void *lprofPtrFetchAdd(void **Mem, long ByteIncr);<br>
><br>
> +/* Temporarily suspend SIGKILL. Return value of 1 means a restore is needed.<br>
> + * Other return values mean no restore is needed.<br>
> + */<br>
> +int lprofSuspendSigKill();<br>
> +<br>
> +/* Restore previously suspended SIGKILL. */<br>
> +void lprofRestoreSigKill();<br>
> +<br>
> #endif /* PROFILE_INSTRPROFILINGUTIL_H */<br>
><br>
> Added: compiler-rt/trunk/test/<wbr>profile/Linux/prctl.c<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/profile/Linux/prctl.c?rev=295364&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/compiler-rt/trunk/<wbr>test/profile/Linux/prctl.c?<wbr>rev=295364&view=auto</a><br>
> ==============================<wbr>==============================<wbr>==================<br>
> --- compiler-rt/trunk/test/<wbr>profile/Linux/prctl.c (added)<br>
> +++ compiler-rt/trunk/test/<wbr>profile/Linux/prctl.c Thu Feb 16 13:21:31 2017<br>
> @@ -0,0 +1,36 @@<br>
> +// RUN: %clang_pgogen -O2 -o %t %s<br>
> +// RUN: rm -rf default_*.profraw<br>
> +// RUN: %run %t && sleep 1<br>
> +// RUN: llvm-profdata show default_*.profraw 2>&1 | FileCheck %s<br>
> +<br>
> +#include <signal.h><br>
> +#include <stdio.h><br>
> +#include <stdlib.h><br>
> +#include <sys/prctl.h><br>
> +#include <unistd.h><br>
> +<br>
> +#define FAKE_COUNT_SZ 2000000<br>
> +/* fake counts to increse the profile size. */<br>
> +unsigned long long __attribute__((section("__<wbr>llvm_prf_cnts")))<br>
> +counts[FAKE_COUNT_SZ];<br>
> +<br>
> +int main(int argc, char **argv) {<br>
> + pid_t pid = fork();<br>
> + if (pid == 0) {<br>
> + int i;<br>
> + int sum = 0;<br>
> + /* child process: sleep 500us and get to runtime before the<br>
> + * main process exits. */<br>
> + prctl(PR_SET_PDEATHSIG, SIGKILL);<br>
> + usleep(500);<br>
> + for (i = 0; i < 5000; ++i)<br>
> + sum += i * i * i;<br>
> + printf("child process (%d): sum=%d\n", getpid(), sum);<br>
> + } else if (pid > 0) {<br>
> + /* parent process: sleep 100us to get into profile runtime first. */<br>
> + usleep(100);<br>
> + }<br>
> + return 0;<br>
> +}<br>
> +<br>
> +// CHECK-NOT: Empty raw profile file<br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> llvm-commits mailing list<br>
> <a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a><br>
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/llvm-commits</a><br>
</blockquote></div><br></div>