<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br><div><div>On May 6, 2014, at 12:26 AM, Hal Finkel <<a href="mailto:hfinkel@anl.gov">hfinkel@anl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">----- Original Message -----</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">From: "Louis Gerbarg" <<a href="mailto:lgg@apple.com">lgg@apple.com</a>><br>To: "Benjamin Kramer" <<a href="mailto:benny.kra@gmail.com">benny.kra@gmail.com</a>><br>Cc: "llvm-commits" <<a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a>><br>Sent: Monday, May 5, 2014 9:21:32 PM<br>Subject: Re: [llvm] r207940 - LoopUnroll: If we're doing partial unrolling,<span class="Apple-tab-span" style="white-space: pre;"> </span>use the PartialThreshold to limit<br>unrolling.<br><br>I am seeing what appears to be significant regressions on some of the<br>nightly tests from this patch. For example the following benches all<br>show 50-100% slowdowns when I apply the patch:<br><br>SingleSource/Benchmarks/Stanford/Bubblesort<br>SingleSource/Benchmarks/Polybench/linear-algebra/solvers/dynprog/dynprog<br>MultiSource/Benchmarks/FreeBench/neural/neural<br></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">To be clear, on what system is that?</span><br></blockquote><div><br></div>I’ve confirmed the regressions personally on Haswell. Looking at our buildbots it also appears to be occurring on Sandybridge and Penryn (though with some variation, the bubblesort regression is ~100% on Haswell, it is 120% on Penryn).<br><div><br></div><br><blockquote type="cite"><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">Generally speaking, if you use -mllvm -x86-partial-unrolling-threshold=N as you increase the value of N does that improve things for you? The current values for the unrolling were taken from the optimization manual (and I'm guessing is some value like 28 for your system; see X86TTI::getUnrollingPreferences in lib/Target/X86/X86TargetTransformInfo.cpp), but was not actually tuned experimentally. Perhaps this could use some improvement.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"></blockquote><div><br></div><div>No. In fact, the regressed numbers are all stable and appear to occur at any value of -x86-partial-unrolling-threshold. I also see the regressions even without r207940 if I pass -x86-partial-unrolling-threshold. Some quick and dirty numbers from my Haswell system:</div><div><br></div><div>BASELINE:</div><div><div>bash-3.2$ ../good/bin/clang -O3 Bubblesort.c && time ./a.out > /dev/null</div><div><br></div><div>real<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.021s</div><div>user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.011s</div><div>sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.001s</div><div><br></div><div>WITHOUT r207940:</div><div>bash-3.2$ ../good/bin/clang -O3 -mllvm -x86-partial-unrolling-threshold=0 Bubblesort.c && time ./a.out > /dev/null </div><div><br></div><div>real<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.032s</div><div>user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.027s</div><div>sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.001s</div><div>bash-3.2$ ../good/bin/clang -O3 -mllvm -x86-partial-unrolling-threshold=18 Bubblesort.c && time ./a.out > /dev/null </div><div><br></div><div>real<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.030s</div><div>user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.027s</div><div>sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.001s</div><div>bash-3.2$ ../good/bin/clang -O3 -mllvm -x86-partial-unrolling-threshold=28 Bubblesort.c && time ./a.out > /dev/null </div><div><br></div><div>real<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.030s</div><div>user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.027s</div><div>sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.001s</div><div>bash-3.2$ ../good/bin/clang -O3 -mllvm -x86-partial-unrolling-threshold=40 Bubblesort.c && time ./a.out > /dev/null </div><div><br></div><div>real<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.030s</div><div>user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.027s</div><div>sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.001s</div><div>bash-3.2$ ../good/bin/clang -O3 -mllvm -x86-partial-unrolling-threshold=60 Bubblesort.c && time ./a.out > /dev/null </div><div><br></div><div>real<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.030s</div><div>user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.027s</div><div>sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.001s</div><div><br></div><div><div><div>bash-3.2$ ../good/bin/clang -O3 -mllvm -x86-partial-unrolling-threshold=600 Bubblesort.c && time ./a.out > /dev/null </div><div><br></div></div><div>real<span class="Apple-tab-span" style="white-space: pre;"> </span>0m0.030s</div><div>user<span class="Apple-tab-span" style="white-space: pre;"> </span>0m0.027s</div><div>sys<span class="Apple-tab-span" style="white-space: pre;"> </span>0m0.001s</div></div><div><br></div><div>WITH r207940:</div><div>bash-3.2$ ../bad/bin/clang -O3 Bubblesort.c && time ./a.out > /dev/null </div><div><br></div><div>real<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.030s</div><div>user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.027s</div><div>sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.001s..</div><div>bash-3.2$ ../bad/bin/clang -O3 -mllvm -x86-partial-unrolling-threshold=0 Bubblesort.c && time ./a.out > /dev/null </div><div><br></div><div>real<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.030s</div><div>user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.027s</div><div>sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.001s</div><div>bash-3.2$ ../bad/bin/clang -O3 -mllvm -x86-partial-unrolling-threshold=18 Bubblesort.c && time ./a.out > /dev/null </div><div><br></div><div>real<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.030s</div><div>user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.027s</div><div>sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.001s</div><div>bash-3.2$ ../bad/bin/clang -O3 -mllvm -x86-partial-unrolling-threshold=28 Bubblesort.c && time ./a.out > /dev/null </div><div><br></div><div>real<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.030s</div><div>user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.027s</div><div>sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.001s</div><div>bash-3.2$ ../bad/bin/clang -O3 -mllvm -x86-partial-unrolling-threshold=40 Bubblesort.c && time ./a.out > /dev/null </div><div><br></div><div>real<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.030s</div><div>user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.027s</div><div>sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.001s</div><div>bash-3.2$ ../bad/bin/clang -O3 -mllvm -x86-partial-unrolling-threshold=60 Bubblesort.c && time ./a.out > /dev/null </div><div><br></div><div>real<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.031s</div><div>user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.028s</div><div>sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.001s</div><div>bash-3.2$ ../bad/bin/clang -O3 -mllvm -x86-partial-unrolling-threshold=600 Bubblesort.c && time ./a.out > /dev/null </div><div><br></div><div>real<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.030s</div><div>user<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.027s</div><div>sys<span class="Apple-tab-span" style="white-space:pre"> </span>0m0.001s</div><div>bash-3.2$ </div><div><br></div><div>Louis</div></div><br><blockquote type="cite"><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">-Hal</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br>Louis<br><br>On May 5, 2014, at 3:09 AM, Benjamin Kramer <<a href="mailto:benny.kra@gmail.com">benny.kra@gmail.com</a>><br>wrote:<br><br><blockquote type="cite"><br>On 05.05.2014, at 01:18, Nadav Rotem <<a href="mailto:nrotem@apple.com">nrotem@apple.com</a>> wrote:<br><br><blockquote type="cite">Hi Ben,<br><br>Thanks for working on this. Overall it sounds like a good change<br>and unrolling 8 times sounds way too high, even for small loops.<br>Did you get a chance to measure the performance difference of<br>this patch?<br></blockquote><br>I didn't find any significant runtime change in the test suite or<br>when trying some of the synthetic benchmarks that were showing<br>extreme unrolling. Code size is a bit better though.<br><br>I initially observed this behavior when looking into the vectorizer<br>( <a href="http://llvm.org/bugs/show_bug.cgi?id=14985">http://llvm.org/bugs/show_bug.cgi?id=14985</a> ) For the trivial<br>loop in the test case we used to unroll 2x in the loop vectorizer<br>(that's a good thing) and then up to another 8x in the loop<br>unroller, when we're targeting core2 or higher. I asked Hal and he<br>agreed that we were unrolling too much.<br><br>I guess it makes sense to actually use the threshold derived from<br>the processor manuals to drive unrolling instead of assuming that<br>more unrolling is better :)<br><br>- Ben<br><br><blockquote type="cite"><br>Thanks,<br>Nadav<br><br><br>On May 4, 2014, at 12:12 PM, Benjamin Kramer<br><<a href="mailto:benny.kra@googlemail.com">benny.kra@googlemail.com</a>> wrote:<br><br><blockquote type="cite">Author: d0k<br>Date: Sun May 4 14:12:38 2014<br>New Revision: 207940<br><br>URL: <a href="http://llvm.org/viewvc/llvm-project?rev=207940&view=rev">http://llvm.org/viewvc/llvm-project?rev=207940&view=rev</a><br>Log:<br>LoopUnroll: If we're doing partial unrolling, use the<br>PartialThreshold to limit unrolling.<br><br>Otherwise we use the same threshold as for complete unrolling,<br>which is<br>way too high. This made us unroll any loop smaller than 150<br>instructions<br>by 8 times, but only if someone specified -march=core2 or better,<br>which happens to be the default on darwin.<br><br>Modified:<br>llvm/trunk/lib/Transforms/Scalar/LoopUnrollPass.cpp<br>llvm/trunk/test/Transforms/LoopUnroll/X86/partial.ll<br><br>Modified: llvm/trunk/lib/Transforms/Scalar/LoopUnrollPass.cpp<br>URL:<br><a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Scalar/LoopUnrollPass.cpp?rev=207940&r1=207939&r2=207940&view=diff">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/Scalar/LoopUnrollPass.cpp?rev=207940&r1=207939&r2=207940&view=diff</a><br>==============================================================================<br>--- llvm/trunk/lib/Transforms/Scalar/LoopUnrollPass.cpp<br>(original)<br>+++ llvm/trunk/lib/Transforms/Scalar/LoopUnrollPass.cpp Sun May<br>4 14:12:38 2014<br>@@ -238,9 +238,12 @@ bool LoopUnroll::runOnLoop(Loop *L, LPPa<br> return false;<br> }<br> uint64_t Size = (uint64_t)LoopSize*Count;<br>- if (TripCount != 1 && Size > Threshold) {<br>- DEBUG(dbgs() << " Too large to fully unroll with count: "<br><< Count<br>- << " because size: " << Size << ">" << Threshold <<<br>"\n");<br>+ if (TripCount != 1 &&<br>+ (Size > Threshold || (Count != TripCount && Size ><br>PartialThreshold))) {<br>+ if (Size > Threshold)<br>+ DEBUG(dbgs() << " Too large to fully unroll with count:<br>" << Count<br>+ << " because size: " << Size << ">" <<<br>Threshold << "\n");<br>+<br> bool AllowPartial = UserAllowPartial ? CurrentAllowPartial :<br> UP.Partial;<br> if (!AllowPartial && !(Runtime && TripCount == 0)) {<br> DEBUG(dbgs() << " will not try to unroll partially because<br> "<br><br>Modified: llvm/trunk/test/Transforms/LoopUnroll/X86/partial.ll<br>URL:<br>http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/LoopUnroll/X86/partial.ll?rev=207940&r1=207939&r2=207940&view=diff<br>==============================================================================<br>--- llvm/trunk/test/Transforms/LoopUnroll/X86/partial.ll<br>(original)<br>+++ llvm/trunk/test/Transforms/LoopUnroll/X86/partial.ll Sun May<br>4 14:12:38 2014<br>@@ -76,5 +76,52 @@ for.end:<br>ret void<br>}<br><br>+define zeroext i16 @test1(i16* nocapture readonly %arr, i32 %n)<br>#0 {<br>+entry:<br>+ %cmp25 = icmp eq i32 %n, 0<br>+ br i1 %cmp25, label %for.end, label %for.body<br>+<br>+for.body: ; preds =<br>%entry, %for.body<br>+ %indvars.iv = phi i64 [ %indvars.iv.next, %for.body ], [ 0,<br>%entry ]<br>+ %reduction.026 = phi i16 [ %add14, %for.body ], [ 0, %entry ]<br>+ %arrayidx = getelementptr inbounds i16* %arr, i64 %indvars.iv<br>+ %0 = load i16* %arrayidx, align 2<br>+ %add = add i16 %0, %reduction.026<br>+ %sext = mul i64 %indvars.iv, 12884901888<br>+ %idxprom3 = ashr exact i64 %sext, 32<br>+ %arrayidx4 = getelementptr inbounds i16* %arr, i64 %idxprom3<br>+ %1 = load i16* %arrayidx4, align 2<br>+ %add7 = add i16 %add, %1<br>+ %sext28 = mul i64 %indvars.iv, 21474836480<br>+ %idxprom10 = ashr exact i64 %sext28, 32<br>+ %arrayidx11 = getelementptr inbounds i16* %arr, i64 %idxprom10<br>+ %2 = load i16* %arrayidx11, align 2<br>+ %add14 = add i16 %add7, %2<br>+ %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1<br>+ %lftr.wideiv = trunc i64 %indvars.iv.next to i32<br>+ %exitcond = icmp eq i32 %lftr.wideiv, %n<br>+ br i1 %exitcond, label %for.end, label %for.body<br>+<br>+for.end: ; preds =<br>%for.body, %entry<br>+ %reduction.0.lcssa = phi i16 [ 0, %entry ], [ %add14,<br>%for.body ]<br>+ ret i16 %reduction.0.lcssa<br>+<br>+; This loop is too large to be partially unrolled (size=16)<br>+<br>+; CHECK-LABEL: @test1<br>+; CHECK: br<br>+; CHECK: br<br>+; CHECK: br<br>+; CHECK: br<br>+; CHECK-NOT: br<br>+<br>+; CHECK-NOUNRL-LABEL: @test1<br>+; CHECK-NOUNRL: br<br>+; CHECK-NOUNRL: br<br>+; CHECK-NOUNRL: br<br>+; CHECK-NOUNRL: br<br>+; CHECK-NOUNRL-NOT: br<br>+}<br>+<br>attributes #0 = { nounwind uwtable }<br><br><br><br>_______________________________________________<br>llvm-commits mailing list<br>llvm-commits@cs.uiuc.edu<br>http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits<br></blockquote><br></blockquote><br><br>_______________________________________________<br>llvm-commits mailing list<br><a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br>http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits<br></blockquote><br>_______________________________________________<br>llvm-commits mailing list<br><a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br>http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits<br><br></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">--<span class="Apple-converted-space"> </span></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">Hal Finkel</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">Assistant Computational Scientist</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">Leadership Computing Facility</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">Argonne National Laboratory</span></blockquote></div><br></body></html>