<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Let's move this to llvm-dev. I should describe my goals/motivation for the work I have been putting into the llvm-testsuite lately. This is how I see the llvm-test-suite today:</div><div class=""><br class=""></div><div class="">- We provide a familiar cmake build system so people have a known environment to tweak compilation flags.</div><div class="">- Together with the benchmark executable we build a .test file that describes how to invoke the benchmark and can be run by the familiar llvm-lit tool:</div><div class="">- Running a benchmark means executing its executable with a certain set of flags. Some of the SPEC benchmarks even require multiple invocations with different flags.</div><div class="">- There is a set of steps to verify that the benchmark worked correctly. This usually means invoking "diff" or "fpcmp" and comparing the results with a reference file.</div><div class="">- The lit benchmark driver modifies these benchmark descriptions to create a test plan. In the simplest case this means prefixing the executable with "timeit" and collecting the number. But we are adding more features like collecting code size, running the benchmark on a remote device, prefixing different instrumentation tools like the linux "perf" tool, a utility tasks that collects and merge PGO data files after a benchmark run, ...</div><div class=""><br class=""></div><div class="">This allows us to add new instrumentation and metrics in the future without touching the benchmarks itself. It works best for bigger benchmark that run for a while (a few seconds minimum). It works nicely with benchmark suites like SPEC, geekbench, mediabench.... Let's call this "macro benchmarking".</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Having said all that. You make a very good case for what we should call "micro benchmarking". The google benchmarking library does indeed look like a fantastic tool. We should definitely evaluate how we can integrate this into the llvm test-suite, we think of it as a new flavor of benchmarks. We won't be able to redesign SPEC but we surely can find things like TSVC which we could adapt to this. I have no immediate plans to put much more work into the test-suite, but I agree that micro benchmarking would be an exciting addition to our testing strategy. I'd be happy to review patches or talk through possible designs on IRC.</div><div class=""><br class=""></div><div class="">- Matthias</div><div class=""><br class=""><blockquote type="cite" class=""><a href="https://github.com/google/benchmark" class="">https://github.com/google/benchmark</a></blockquote></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div><blockquote type="cite" class=""><div class="">On Mar 23, 2016, at 3:45 PM, Hal Finkel <<a href="mailto:hfinkel@anl.gov" class="">hfinkel@anl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">----- Original Message -----</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">From: "Hal Finkel via llvm-commits" <<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a>><br class="">To: "Matthias Braun" <<a href="mailto:mbraun@apple.com" class="">mbraun@apple.com</a>><br class="">Cc: "nd" <<a href="mailto:nd@arm.com" class="">nd@arm.com</a>>, "llvm-commits" <<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a>><br class="">Sent: Wednesday, March 23, 2016 5:19:37 PM<br class="">Subject: Re: [test-suite] r261857 - [cmake] Add support for arbitrary metrics<br class=""><br class="">----- Original Message -----<br class=""><blockquote type="cite" class="">From: "Matthias Braun" <<a href="mailto:mbraun@apple.com" class="">mbraun@apple.com</a>><br class="">To: "Hal Finkel" <<a href="mailto:hfinkel@anl.gov" class="">hfinkel@anl.gov</a>><br class="">Cc: "James Molloy" <<a href="mailto:James.Molloy@arm.com" class="">James.Molloy@arm.com</a>>, "nd" <<a href="mailto:nd@arm.com" class="">nd@arm.com</a>>,<br class="">"llvm-commits" <<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a>><br class="">Sent: Friday, March 4, 2016 12:36:36 PM<br class="">Subject: Re: [test-suite] r261857 - [cmake] Add support for<br class="">arbitrary metrics<br class=""><br class="">A test can report "internal" metrics now. Though I don't think lnt<br class="">would split those into a notion of sub-tests I think.<br class="">It would be an interesting feature to add. Though if we have the<br class="">choice to modify a benchmark, we should still prefer smaller<br class="">independent ones IMO as that gives a better idea when some of the<br class="">other metrics change (compiletime, codesize, hopefully things like<br class="">memory usage or performance counters in the future).<br class=""></blockquote><br class="">Unless the kernels are large, their code size within the context of a<br class="">complete executable might be hard to track regardless (because by<br class="">the time you add in the static libc startup code, ELF headers, etc.<br class="">any change would be a smaller percentage of the total). Explicitly<br class="">instrumenting the code to mark regions of interest is probably best<br class="">(which is true for timing too), but that seems like a separate<br class="">(although worthwhile) project.<br class=""><br class="">In any case, for TSVC, for example, the single test has 136 kernels;<br class="">which I currently group into 18 binaries. I have a float and double<br class="">version for each, so we have 36 total binaries. What you're<br class="">suggesting would have us produce 272 separate executables, just for<br class="">TSVC. Ideally, I'd like aligned and unaligned variants of each of<br class="">these. I've not done that because I thought that 72 executables<br class="">would be a bit much, but that's 544 executables if I generate one<br class="">per kernel variant.<br class=""><br class="">The LCALS benchmark, which I'd really like to add sometime soon, has<br class="">another ~100 kernels, which is ~200 to do both float and double<br class="">(which we should do).<br class=""><br class="">What do you think is reasonable here?<br class=""><br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Also, we might want to consider updating some of these tests to use Google's benchmark library (</span><a href="https://github.com/google/benchmark" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">https://github.com/google/benchmark</a><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">). Have you looked at this? Aside from giving us a common output format, the real advantage of using a driver library like this is that it lets us dynamically pick the number of loop iterations based on per-iteration timing. This appeals to me because the number of iterations that is reasonable for some embedded device is normally quite different from what is reasonable for a server-class machine. Doing this, however, means that we definitely can't rely on overall executable timing. Thoughts?</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">-Hal</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Thanks again,<br class="">Hal<br class=""><br class=""><blockquote type="cite" class=""><br class="">- Matthias<br class=""><br class=""><blockquote type="cite" class="">On Mar 4, 2016, at 8:22 AM, Hal Finkel <<a href="mailto:hfinkel@anl.gov" class="">hfinkel@anl.gov</a>> wrote:<br class=""><br class="">Hi James,<br class=""><br class="">If I'm reading this correctly, you can have multiple metrics per<br class="">test. Is that correct?<br class=""><br class="">I'd really like to support tests with internal timers (i.e. a<br class="">timer<br class="">per kernel), so that we can have more fine-grained timing without<br class="">splitting executables into multiple parts (e.g. as I had to do<br class="">with TSVC).<br class=""><br class="">Thanks again,<br class="">Hal<br class=""><br class="">----- Original Message -----<br class=""><blockquote type="cite" class="">From: "James Molloy via llvm-commits"<br class=""><<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a>><br class="">To: "Matthias Braun" <<a href="mailto:mbraun@apple.com" class="">mbraun@apple.com</a>><br class="">Cc: "nd" <<a href="mailto:nd@arm.com" class="">nd@arm.com</a>>, "llvm-commits"<br class=""><<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a>><br class="">Sent: Friday, February 26, 2016 3:06:01 AM<br class="">Subject: Re: [test-suite] r261857 - [cmake] Add support for<br class="">arbitrary metrics<br class=""><br class="">Hi Matthias,<br class=""><br class="">Thanks :) I’ve been working internally to move all our testing<br class="">from<br class="">ad-hoc driver scripts to CMake+LIT-based. Currently I have CMake<br class="">drivers for a very popular mobile benchmark (but the pre-release<br class="">version so pushing this upstream might be difficult), and EEMBC<br class="">(automotive, telecom, consumer).<br class=""><br class="">I really want all of these to live upstream, but I have to do a<br class="">bit<br class="">of legal checking before I can push them. In the meantime I’m<br class="">happy<br class="">to add an example to the repositories; alternatively I could<br class="">modify<br class="">the SPEC drivers to also compute SPECrate as a metric?<br class=""><br class="">Cheers,<br class=""><br class="">James<br class=""><br class=""><blockquote type="cite" class="">On 25 Feb 2016, at 21:33, Matthias Braun <<a href="mailto:mbraun@apple.com" class="">mbraun@apple.com</a>><br class="">wrote:<br class=""><br class="">Hi James,<br class=""><br class="">thanks for working on the test-suite. It's nice to see new<br class="">capabilities added to the lit system.<br class=""><br class="">Are you planing to add tests that use this? If not we should<br class="">really<br class="">have at least an example/unit-test type thing in the<br class="">repository.<br class=""><br class="">- Matthias<br class=""><br class=""><blockquote type="cite" class="">On Feb 25, 2016, at 3:06 AM, James Molloy via llvm-commits<br class=""><<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a>> wrote:<br class=""><br class="">Author: jamesm<br class="">Date: Thu Feb 25 05:06:15 2016<br class="">New Revision: 261857<br class=""><br class="">URL: <a href="http://llvm.org/viewvc/llvm-project?rev=261857&view=rev" class="">http://llvm.org/viewvc/llvm-project?rev=261857&view=rev</a><br class="">Log:<br class="">[cmake] Add support for arbitrary metrics<br class=""><br class="">This allows a .test script to specify a command to get a<br class="">metric<br class="">for the test. For example:<br class=""><br class="">METRIC: score: grep "Score:" %o | awk '{print $2}'<br class=""><br class="">Modified:<br class="">test-suite/trunk/cmake/modules/SingleMultiSource.cmake<br class="">test-suite/trunk/litsupport/test.py<br class="">test-suite/trunk/litsupport/testscript.py<br class=""><br class="">Modified:<br class="">test-suite/trunk/cmake/modules/SingleMultiSource.cmake<br class="">URL:<br class=""><a href="http://llvm.org/viewvc/llvm-project/test-suite/trunk/cmake/modules/SingleMultiSource.cmake?rev=261857&r1=261856&r2=261857&view=diff" class="">http://llvm.org/viewvc/llvm-project/test-suite/trunk/cmake/modules/SingleMultiSource.cmake?rev=261857&r1=261856&r2=261857&view=diff</a><br class="">==============================================================================<br class="">--- test-suite/trunk/cmake/modules/SingleMultiSource.cmake<br class="">(original)<br class="">+++ test-suite/trunk/cmake/modules/SingleMultiSource.cmake Thu<br class="">Feb<br class="">25 05:06:15 2016<br class="">@@ -223,3 +223,17 @@ macro(llvm_test_verify)<br class=""> set(TESTSCRIPT "${TESTSCRIPT}VERIFY: ${JOINED_ARGUMENTS}\n")<br class="">endif()<br class="">endmacro()<br class="">+<br class="">+macro(llvm_test_metric)<br class="">+ CMAKE_PARSE_ARGUMENTS(ARGS "" "RUN_TYPE;METRIC" "" ${ARGN})<br class="">+ if(NOT DEFINED TESTSCRIPT)<br class="">+ set(TESTSCRIPT "" PARENT_SCOPE)<br class="">+ endif()<br class="">+ # ARGS_UNPARSED_ARGUMENTS is a semicolon-separated list.<br class="">Change<br class="">it into a<br class="">+ # whitespace-separated string.<br class="">+ string(REPLACE ";" " " JOINED_ARGUMENTS<br class="">"${ARGS_UNPARSED_ARGUMENTS}")<br class="">+ if(NOT DEFINED ARGS_RUN_TYPE OR "${ARGS_RUN_TYPE}" STREQUAL<br class="">"${TEST_SUITE_RUN_TYPE}")<br class="">+ set(TESTSCRIPT "${TESTSCRIPT}METRIC: ${ARGS_METRIC}:<br class="">${JOINED_ARGUMENTS}\n")<br class="">+ endif()<br class="">+endmacro()<br class="">+<br class="">\ No newline at end of file<br class=""><br class="">Modified: test-suite/trunk/litsupport/test.py<br class="">URL:<br class="">http://llvm.org/viewvc/llvm-project/test-suite/trunk/litsupport/test.py?rev=261857&r1=261856&r2=261857&view=diff<br class="">==============================================================================<br class="">--- test-suite/trunk/litsupport/test.py (original)<br class="">+++ test-suite/trunk/litsupport/test.py Thu Feb 25 05:06:15<br class="">2016<br class="">@@ -56,7 +56,7 @@ class TestSuiteTest(FileBasedTest):<br class=""> res = testscript.parse(test.getSourcePath())<br class=""> if litConfig.noExecute:<br class=""> return lit.Test.Result(Test.PASS)<br class="">- runscript, verifyscript = res<br class="">+ runscript, verifyscript, metricscripts = res<br class=""><br class=""> # Apply the usual lit substitutions (%s, %S, %p, %T,<br class=""> ...)<br class=""> tmpDir, tmpBase = getTempPaths(test)<br class="">@@ -65,6 +65,8 @@ class TestSuiteTest(FileBasedTest):<br class=""> substitutions += [('%o', outfile)]<br class=""> runscript = applySubstitutions(runscript, substitutions)<br class=""> verifyscript = applySubstitutions(verifyscript,<br class=""> substitutions)<br class="">+ metricscripts = {k: applySubstitutions(v,<br class="">substitutions)<br class="">+ for k,v in metricscripts.items()}<br class=""> context = TestContext(test, litConfig, runscript,<br class=""> verifyscript, tmpDir,<br class=""> tmpBase)<br class=""><br class="">@@ -80,6 +82,7 @@ class TestSuiteTest(FileBasedTest):<br class=""> output = ""<br class=""> n_runs = 1<br class=""> runtimes = []<br class="">+ metrics = {}<br class=""> for n in range(n_runs):<br class=""> res = runScript(context, runscript)<br class=""> if isinstance(res, lit.Test.Result):<br class="">@@ -94,6 +97,15 @@ class TestSuiteTest(FileBasedTest):<br class=""> output += "\n" + err<br class=""> return lit.Test.Result(Test.FAIL, output)<br class=""><br class="">+ # Execute metric extraction scripts.<br class="">+ for metric, script in metricscripts.items():<br class="">+ res = runScript(context, script)<br class="">+ if isinstance(res, lit.Test.Result):<br class="">+ return res<br class="">+<br class="">+ out, err, exitCode, timeoutInfo = res<br class="">+ metrics.setdefault(metric,<br class="">list()).append(float(out))<br class="">+<br class=""> try:<br class=""> runtime = runsafely.getTime(context)<br class=""> runtimes.append(runtime)<br class="">@@ -128,6 +140,8 @@ class TestSuiteTest(FileBasedTest):<br class=""> result = lit.Test.Result(Test.PASS, output)<br class=""> if len(runtimes) > 0:<br class=""> result.addMetric('exec_time',<br class=""> lit.Test.toMetricValue(runtimes[0]))<br class="">+ for metric, values in metrics.items():<br class="">+ result.addMetric(metric,<br class="">lit.Test.toMetricValue(values[0]))<br class=""> compiletime.collect(context, result)<br class=""><br class=""> return result<br class=""><br class="">Modified: test-suite/trunk/litsupport/testscript.py<br class="">URL:<br class="">http://llvm.org/viewvc/llvm-project/test-suite/trunk/litsupport/testscript.py?rev=261857&r1=261856&r2=261857&view=diff<br class="">==============================================================================<br class="">--- test-suite/trunk/litsupport/testscript.py (original)<br class="">+++ test-suite/trunk/litsupport/testscript.py Thu Feb 25<br class="">05:06:15<br class="">2016<br class="">@@ -22,13 +22,18 @@ def parse(filename):<br class=""> # Collect the test lines from the script.<br class=""> runscript = []<br class=""> verifyscript = []<br class="">- keywords = ['RUN:', 'VERIFY:']<br class="">+ metricscripts = {}<br class="">+ keywords = ['RUN:', 'VERIFY:', 'METRIC:']<br class=""> for line_number, command_type, ln in \<br class=""> parseIntegratedTestScriptCommands(filename,<br class=""> keywords):<br class=""> if command_type == 'RUN':<br class=""> _parseShellCommand(runscript, ln)<br class=""> elif command_type == 'VERIFY':<br class=""> _parseShellCommand(verifyscript, ln)<br class="">+ elif command_type == 'METRIC':<br class="">+ metric, ln = ln.split(':', 1)<br class="">+ metricscript =<br class="">metricscripts.setdefault(metric.strip(), list())<br class="">+ _parseShellCommand(metricscript, ln)<br class=""> else:<br class=""> raise ValueError("unknown script command type: %r" %<br class=""> (<br class=""> command_type,))<br class="">@@ -43,4 +48,4 @@ def parse(filename):<br class=""> raise ValueError("Test has unterminated RUN/VERIFY<br class=""> lines " +<br class=""> "(ending with '\\')")<br class=""><br class="">- return runscript, verifyscript<br class="">+ return runscript, verifyscript, metricscripts<br class=""><br class=""><br class="">_______________________________________________<br class="">llvm-commits mailing list<br class="">llvm-commits@lists.llvm.org<br class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits<br class=""></blockquote></blockquote><br class="">_______________________________________________<br class="">llvm-commits mailing list<br class=""><a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a><br class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits<br class=""></blockquote><br class="">--<br class="">Hal Finkel<br class="">Assistant Computational Scientist<br class="">Leadership Computing Facility<br class="">Argonne National Laboratory<br class=""></blockquote><br class=""></blockquote><br class="">--<br class="">Hal Finkel<br class="">Assistant Computational Scientist<br class="">Leadership Computing Facility<br class="">Argonne National Laboratory<br class="">_______________________________________________<br class="">llvm-commits mailing list<br class=""><a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a><br class=""><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br class=""><br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">--<span class="Apple-converted-space"> </span></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Hal Finkel</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Assistant Computational Scientist</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Leadership Computing Facility</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Argonne National Laboratory</span></div></blockquote></div><br class=""></body></html>