<div dir="ltr">Yep, code generation choices based on the iteration order of a hash (or unstable sorted) container is the most common source of non-determinism. Using the reverse iteration mode that Roman mentioned is about the only tool we have & doesn't cover all cases.<br><br>Another possibility is order of evaluation of function arguments (eg: f(g(), h()) - if the evaluation order of 'g' and 'h' effects the code generation, then this differ between GCC and Clang).</div><br><div class="gmail_quote"><div dir="ltr">On Thu, Apr 19, 2018 at 3:22 AM Martin J. O'Riordan via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-IE" link="blue" vlink="purple"><div class="m_3185229991279714432WordSection1"><p class="MsoNormal"><span style="font-family:"Book Antiqua",serif;color:#943634">Hi Michael,<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Book Antiqua",serif;color:#943634"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-family:"Book Antiqua",serif;color:#943634">Last year I had a problem with reproducibility that I detected in the generated assembly for the out-of-tree target I was working on that seemed like non-deterministic code generation. There was nothing incorrect in the alternative code being emitted, but it was making me nervous that it was different at all.<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Book Antiqua",serif;color:#943634"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-family:"Book Antiqua",serif;color:#943634">After some investigation it turned out to be a consequence of iterator ordering. The scheduler was working out the prioritisation of basic blocks and there was no issue if one BB had a different priority to another. However, if two BBs had the same priority, the implicit ordering was based on the hash in the map that was being iterated, and this could vary between platform (Windows vs Linux) and between recompiles on the same platform.<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Book Antiqua",serif;color:#943634"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-family:"Book Antiqua",serif;color:#943634">Just a thought, but perhaps you might have a look at how ordering is handled in your BB or MI iterators.<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Book Antiqua",serif;color:#943634"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-family:"Book Antiqua",serif;color:#943634"> MartinO<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Book Antiqua",serif;color:#943634"><u></u> <u></u></span></p><div><div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm"><p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> llvm-dev [mailto:<a href="mailto:llvm-dev-bounces@lists.llvm.org" target="_blank">llvm-dev-bounces@lists.llvm.org</a>] <b>On Behalf Of </b>Michael Zolotukhin via llvm-dev<br><b>Sent:</b> 19 April 2018 07:15<br><b>To:</b> Steven Wu <<a href="mailto:stevenwu@apple.com" target="_blank">stevenwu@apple.com</a>><br><b>Cc:</b> llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br><b>Subject:</b> Re: [llvm-dev] Need help reproducing a bug<u></u><u></u></span></p></div></div></div></div><div lang="EN-IE" link="blue" vlink="purple"><div class="m_3185229991279714432WordSection1"><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Thanks everyone! What are the best tools/techniques to expose such non-deterministic behavior? My hope is to reproduce it on a smaller test (e.g. use some sanitizer and thus make the compiler *fail* when building the test) - Currently these failures only tell me “there is some bug in your code” without any hints where to look for it.<u></u><u></u></p><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">Michael<u></u><u></u></p><div><p class="MsoNormal"><br><br><u></u><u></u></p><blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"><div><p class="MsoNormal">On Apr 18, 2018, at 9:18 PM, Steven Wu <<a href="mailto:stevenwu@apple.com" target="_blank">stevenwu@apple.com</a>> wrote:<u></u><u></u></p></div><p class="MsoNormal"><u></u> <u></u></p><div><p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br><br style="font-variant-caps:normal;text-align:start;word-spacing:0px"><br></span><u></u><u></u></p><blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"><p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">On Apr 18, 2018, at 9:11 AM, Roman Lebedev via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br><br>On Wed, Apr 18, 2018 at 5:45 PM, Michael Zolotukhin via llvm-dev<br><<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br><br><u></u><u></u></span></p><blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"><p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Hi,<br><br>Recently I committed a change (r330175) that passed all my testing, but<br>failed on several bots. Namely, these are the failed ones:<br><a href="http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu/builds/9803" target="_blank">http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu/builds/9803</a><br><a href="http://lab.llvm.org:8011/builders/clang-with-lto-ubuntu/builds/8173" target="_blank">http://lab.llvm.org:8011/builders/clang-with-lto-ubuntu/builds/8173</a><br><a href="http://lab.llvm.org:8011/builders/lld-x86_64-freebsd/builds/18082" target="_blank">http://lab.llvm.org:8011/builders/lld-x86_64-freebsd/builds/18082</a><u></u><u></u></span></p></blockquote><p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Note what *specifically* failed:<br>* compare-compilers compare stage3 and stage4 compilers failed ( 0 secs )<br>* compare-tablegen-inc-files compare stage3 and stage4 Tablegen inc<br>files failed ( 1 secs )<br><br>I.e. it wasn't tests that failed.<u></u><u></u></span></p></blockquote><p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>Failing that tests means the compiler doesn't produce deterministic output because the stage3 and stage 4 compiler has to be the same.<br><br>Not sure about LLD test.<br><br>Steven<br><br style="font-variant-caps:normal;text-align:start;word-spacing:0px"><br></span><u></u><u></u></p><blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"><p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br><br><u></u><u></u></span></p><blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"><p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">I reverted the change (r330180), but now I’m stuck with how to proceed with<br>it, as I can’t reproduce any of these.<br><br>So far I’ve tried building clang with asan and using this sanitized clang to<br>build clang and lld one more time and run make check - none of these failed<br>on my machine. What else could I try to catch the issue?<br><br>In case you are interested in details and/or want to try to reproduce it,<br>you’ll need to revert r330180 (and thus reapply r330175). The change is<br>about using a new faster SSAUpdater in Jump Threading, more details are<br>available in the phabricator: <a href="https://reviews.llvm.org/D44282" target="_blank">https://reviews.llvm.org/D44282</a>.<br><br>Thanks,<br>Michael<br><br><br><br><br>_______________________________________________<br>LLVM Developers mailing list<br><a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><u></u><u></u></span></p></blockquote><p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">_______________________________________________<br>LLVM Developers mailing list<br><a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><u></u><u></u></span></p></blockquote></div></blockquote></div><p class="MsoNormal"><u></u> <u></u></p></div></div></div>_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div>