<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: arial,helvetica,sans-serif; font-size: 10pt; color: #000000'><br><hr id="zwchr"><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Sean Silva" <chisophugis@gmail.com><br><b>To: </b>"Hal Finkel" <hfinkel@anl.gov><br><b>Cc: </b>reviews+D18055+public+cfa74f4269f8beb5@reviews.llvm.org, "llvm-commits" <llvm-commits@lists.llvm.org>, "Rui Ueyama" <ruiu@google.com><br><b>Sent: </b>Friday, March 11, 2016 2:53:32 AM<br><b>Subject: </b>Re: [PATCH] D18055: ELF: Implement --build-id.<br><br><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 11, 2016 at 12:13 AM, Hal Finkel <span dir="ltr"><<a href="mailto:hfinkel@anl.gov" target="_blank">hfinkel@anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div><div style="font-family: arial,helvetica,sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><br><hr><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Sean Silva via llvm-commits" <<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>><br><b>To: </b>"Rui Ueyama" <<a href="mailto:ruiu@google.com" target="_blank">ruiu@google.com</a>><br><b>Cc: </b><a href="mailto:reviews%2BD18055%2Bpublic%2Bcfa74f4269f8beb5@reviews.llvm.org" target="_blank">reviews+D18055+public+cfa74f4269f8beb5@reviews.llvm.org</a>, "llvm-commits" <<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>><br><b>Sent: </b>Thursday, March 10, 2016 10:59:25 PM<br><b>Subject: </b>Re: [PATCH] D18055: ELF: Implement --build-id.<span><br><br><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 10, 2016 at 7:56 PM, Rui Ueyama <span dir="ltr"><<a href="mailto:ruiu@google.com" target="_blank">ruiu@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr">AES is for encryption, so it's not usable for secure hashing, no?<div class="gmail_extra"><br><div class="gmail_quote"><span>On Thu, Mar 10, 2016 at 6:33 PM, Sean Silva <span dir="ltr"><<a href="mailto:chisophugis@gmail.com" target="_blank">chisophugis@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Thu, Mar 10, 2016 at 2:51 PM, Rui Ueyama <span dir="ltr"><<a href="mailto:ruiu@google.com" target="_blank">ruiu@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr">For LLD, it hashed 75,421,320 bytes. For Scylla, it hashed 207,703,010 bytes. So the throughput of MD5 is 444 MB/s and 480 MB/s, respectively.<div><br></div><div>BLAKE2 claims that it is about 1.6x faster than MD5, but it is probably the lower bound for a secure hash function.</div></div></blockquote></span><div><br>AESNI claims to be able to do 1.3 cycles per byte on an old Nehalem. That is multiple gigabytes per second.</div><div><a href="https://software.intel.com/sites/default/files/m/d/4/1/d/8/10TB24_Breakthrough_AES_Performance_with_Intel_AES_New_Instructions.final.secure.pdf" target="_blank">https://software.intel.com/sites/default/files/m/d/4/1/d/8/10TB24_Breakthrough_AES_Performance_with_Intel_AES_New_Instructions.final.secure.pdf</a><br></div><div><br></div><div>Have you tried using ADT/Hashing.h? That is probably the simplest option.</div></div></div></div></blockquote><div><br></div></span><div>No, I haven't. But if CRC32 is not okay, then it is not okay as well.</div></div></div></div></blockquote><div><br></div><div>I disagree. CRC32 has dramatically weaker characteristics than ADT/Hashing.h. CRC polynomials are constructed to detect specific classes of errors that are characteristic to physical communication channels. e.g. "anything which toggles a single bit is guaranteed to change the CRC value".</div></div></div></div></span></blockquote>One advantage that CRCs might have here over other schemes is that it is possible to efficiently compute them in parallel. I wrote a small library that does this for CRC64 values a few years ago (<a href="http://trac.alcf.anl.gov/projects/hpcrc64" target="_blank">http://trac.alcf.anl.gov/projects/hpcrc64</a>), and while OpenMP is used for the parallelization, which we don't want to use here, it is pretty simple to see how the scheme works.<br></div></div></blockquote><div><br></div><div>Yeah. Even without OpenMP the inner loop of CRC computations typically have high latency but can be fully pipelined, so it is advantageous to do multiple computations simultaneously even on a single core. E.g. Intel's CRC32C instruction has a latency of 3 cycles but is fully pipelined, so to get max performance you have to be processing 3 blocks at a time and periodically merge them with CLMUL. With a table lookup implementation it is even more important because your table lookup latency is load, op, store to L1D so at least 7 cycles.</div><div><br></div><div id="DWT2595">I see that your implementation ticks all the check marks (both multiword (within-a-register parallel) *and* instruction parallel *and* thread parallel). Beautiful! I guess my ramblings above should be considered some extra explanatory comments for your code m(-_-)m</div></div></div></div></blockquote><br>Thanks :-)<br><br> -Hal<br><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div><br></div><div>-- Sean Silva</div><div> </div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div><div style="font-family: arial,helvetica,sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><br> -Hal<br><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><span><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> They are not particularly good as hash values.</div><div>ADT/Hashing.h is designed against much more stringent tests (I believe it is tested against SMHasher).</div><div><br></div><div>The fedora page does not mention any use case that requires cryptographically strong hashing.</div><div><br></div><div>-- Sean Silva</div><div> </div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><div><br></div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><font color="#888888"><div><br></div><div>-- Sean Silva</div></font></span><span><div> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr"><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 10, 2016 at 2:16 PM, Sean Silva <span dir="ltr"><<a href="mailto:chisophugis@gmail.com" target="_blank">chisophugis@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">silvas added a subscriber: silvas.<br>
silvas added a comment.<br>
<br>
One thing that is missing from the data you provided in the OP is the actual amount of data hashed, and in what pattern (lots of small, or a couple large?). How fast does the hashing have to be to be "acceptable"? E.g. ADT/Hashing.h claims 6.5GB/s hashing for large keys on Nehalem (which is pretty old at this point, modern CPU's likely to better). That is 6.5MB/ms. If we need to hash 100MB then the overhead should be about 15ms.<br>
<br>
For LLD, you quoted a number 713.78ms link time. So for 1% overhead we need to spend <7ms. This is enough for 45MB hashed with ADT/Hashing.h (which is similar to LLD text+data size), assuming that we can get max memory bandwidth (like Rafael said, we can do this while we would be copying data otherwise, so I don't see a reason we can't hit the full performance of ADT/Hashing.h).<br>
<br>
<br>
<a href="http://reviews.llvm.org/D18055" rel="noreferrer" target="_blank">http://reviews.llvm.org/D18055</a><br>
<br>
<br>
<br>
</blockquote></div><br></div></div></div></div>
</blockquote></span></div><br></div></div>
</blockquote></span></div><br></div></div>
</blockquote></div><br></div></div>
<br></span><span>_______________________________________________<br>llvm-commits mailing list<br><a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a><br><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br></span></blockquote><span><font color="#888888"><br><br><br>-- <br><div><span></span>Hal Finkel<br>Assistant Computational Scientist<br>Leadership Computing Facility<br>Argonne National Laboratory<span></span><br></div></font></span></div></div></blockquote></div><br></div></div>
</blockquote><br><br><br>-- <br><div><span name="x"></span>Hal Finkel<br>Assistant Computational Scientist<br>Leadership Computing Facility<br>Argonne National Laboratory<span name="x"></span><br></div></div></body></html>