<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi Johannes,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><i>It is not an IR (encoding) in the sense that there are no new<o:p></o:p></i></p>
<p class="MsoNormal" style="margin-left:.5in"><i>instructions, intrinsics, or anything else that is materialized.<o:p></o:p></i></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If the interface provides a set of functions that query information from the underlying “implementation” (via ParallelRegionInfo, ParallelCommunicationInfo) and modify something in the implementation (via ParallelIR/Builder), that sounds
like an IR to me. It may be hiding the implementation details, and different compilers may use different implementations, but the 5 parallel analyses and optimizations don’t care about that: they’re operating through the interface.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><i>I'm still confused about the hooks.<o:p></o:p></i></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Please take a look at the Google Doc. They’re explained clearly there. We’ll be circulating that soon to the list, once we’ve finished up the more detailed parts.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="color:black">—Vikram Adve<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black">// Interim Head, Department of Computer Science<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">// Donald B. Gillies Professor of Computer Science<br>
// University of Illinois at Urbana-Champaign<br>
// Admin Assistant: Amanda Foley - <a href="mailto:ajfoley2@illinois.edu"><span style="color:#0563C1">ajfoley2@illinois.edu</span></a><br>
// Google Hangouts: <a href="mailto:vikram.s.adve@gmail.com"><span style="color:#0563C1">vikram.s.adve@gmail.com</span></a> || Skype: vikramsadve<br>
// Research page: <a href="http://vikram.cs.illinois.edu/"><span style="color:#0563C1">http://vikram.cs.illinois.edu</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Johannes Doerfert <doerfert@cs.uni-saarland.de><br>
<b>Date: </b>Thursday, June 7, 2018 at 6:19 AM<br>
<b>To: </b>"Adve, Vikram Sadanand" <vadve@illinois.edu><br>
<b>Cc: </b>LLVM-Dev <llvm-dev@lists.llvm.org>, Hal Finkel <hfinkel@anl.gov>, TB Schardl <neboat@mit.edu>, "Tian, Xinmin" <xinmin.tian@intel.com>, "llvmpar@lists.cs.illinois.edu" <llvmpar@lists.cs.illinois.edu><br>
<b>Subject: </b>Re: FW: [RFC] Abstract Parallel IR Optimizations<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Hi Vikram, <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Thanks for the quick reply. I put some comments below.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">On 06/07, Adve, Vikram Sadanand wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal" style="margin-left:.5in">Johannes,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I definitely agree that LLVM needs better support for optimizing parallel programs. Comments inline:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> From: Johannes Doerfert <<a href="mailto:jdoerfert@anl.gov">jdoerfert@anl.gov</a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> Date: Wednesday, June 6, 2018 at 10:52 AM<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> To: LLVM-Dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> Cc: Hal Finkel <<a href="mailto:hfinkel@anl.gov">hfinkel@anl.gov</a>>, TB Schardl <<a href="mailto:neboat@mit.edu">neboat@mit.edu</a>>, "Adve, Vikram Sadanand" <<a href="mailto:vadve@illinois.edu">vadve@illinois.edu</a>>,
"Tian, Xinmin" <<a href="mailto:xinmin.tian@intel.com">xinmin.tian@intel.com</a>>, "<a href="mailto:llvmpar@lists.cs.illinois.edu">llvmpar@lists.cs.illinois.edu</a>" <<a href="mailto:llvmpar@lists.cs.illinois.edu">llvmpar@lists.cs.illinois.edu</a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> Subject: [RFC] Abstract Parallel IR Optimizations<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> This is an RFC to add analyses and transformation passes into LLVM to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> optimize programs based on an abstract notion of a parallel region.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> == this is _not_ a proposal to add a new encoding of parallelism ==<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Actually, I suspect that your three “abstract” interfaces below<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">(ParallelRegionInfo, ParallelCommunicationInfo, ParallelIR/Builder)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">*are* a parallel IR definition, and what you call the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">“implementations” essentially encapsulate *target-specific*<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">variations. <o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">It is not an IR (encoding) in the sense that there are no new<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">instructions, intrinsics, or anything else that is materialized. The<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">interface provides a unified view of existing parallel representations<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">and it is as general as possible.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I am unsure what you mean by target-specific here but our<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">implementations are the actual parallel IR representations as we<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">discussed them several times on other occasions.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal" style="margin-left:.5in">IOW, this looks like a perhaps-very-preliminary parallel IR, not some<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">generic abstract interface. I can only suspect this because I don’t<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">have any information about what info or operations these interfaces<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">provide.<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I disagree. Please check out the operations and info for the interface<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">here<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <a href="https://reviews.llvm.org/differential/changeset/?ref=1086909&whitespace=ignore-most">https://reviews.llvm.org/differential/changeset/?ref=1086909&whitespace=ignore-most</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">and for the builder here<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <a href="https://reviews.llvm.org/differential/changeset/?ref=1086915&whitespace=ignore-most">https://reviews.llvm.org/differential/changeset/?ref=1086915&whitespace=ignore-most</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">so we talk about the same thing.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal" style="margin-left:.5in">In fact, I think it *is* a good idea to have a proper parallel IR to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">support the kinds of optimizations you’re describing. I just don’t<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">think you can define a universal parallel IR for all the parallel<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">languages and targets that LLVM may want to support. At a minimum, I<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">think we need to experiment more before committing to one (*if* one<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">proved enough).<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I think you misunderstood our intention here. This patch is in support<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">of such experiments as it allows to apply the optimizations we have and<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">might come up with to different parallel IR representations. Not all<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">optimizations will be applicable to all representations but I don't see<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">a reason why this is a problem. I also don't think this is a universal<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">parallel IR of sorts but it is a way to extract critical information<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">necessary for optimizations out of parallel representations.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal" style="margin-left:.5in">Instead of defining a specific parallel IR or interface, as you’re<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">trying to do, it would be much better to provide IR-independent hooks<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">in LLVM so that different parallel IRs can sit “above” the LLVM<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">representation and support these and other optimizations. Your IR (or<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">IR implementations, whatever you call them) could be layered in this<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">way.<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I do not understand what you mean here. The "implementations" (as I<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">called them so recklessly) lift _some_ information from the actual IR<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">representation to the high level abstract parallel IR interface level<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">for consummation by the optimization passes. Different parallel IRs can<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">come with their own implementation and provide more or less information<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">to enable optimizations.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal" style="margin-left:.5in">The earlier RFC that Hal Finkel and Xinmin Tian circulated were for<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">such a set of hooks. In fact, these hooks also work on regions, but<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">are much more limited and are *not* meant to support parallel passes<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">themselves: those are left to any IR built above them.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">As you know, we (Hal, Xinmin, TB Schardl, George Stelle, Hashim<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Sharif, I, and a few others) are trying to refine that proposal to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">provide examples of multiple IRs the hooks can support, and to make a<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">clearer argument for the soundness and correctness impact of the hooks<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">on LLVM passes. You’ve been invited to our conference calls and to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">edit the Google Doc. It would be valuable if you could add examples<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">of how your IR information could be connected to LLVM via these hooks.<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I'm still confused about the hooks.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal" style="margin-left:.5in">More specific questions below.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> We currently perform poorly when it comes to optimizations for parallel<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> codes. In fact, parallelizing your loops might actually prevent various<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> optimizations that would have been applied otherwise. One solution to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> this problem is to teach the compiler about the semantics of the used<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> parallel representation. While this sounds tedious at first, it turns<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> out that we can perform key optimizations with reasonable implementation<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> effort (and thereby also reasonable maintenance costs). However, we have<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> various parallel representations that are already in use (KMPC,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> GOMP, CILK runtime, ...) or proposed (Tapir, IntelPIR, ...).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> Our proposal seeks to introduce parallelism specific optimizations for<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> multiple representations while minimizing the implementation overhead.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> This is done through an abstract notion of a parallel region which hides<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> the actual representation from the analysis and optimization passes.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">In general, for analysis and optimization passes to work with multiple<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">representations, you’d need to have a pretty rich abstract notion of a<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">parallel region. I suspect that your design is pretty<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">OpenMP-specific. Even within that context, it would have to capture<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">quite a rich set of parallel constructs. Can you describe what<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">exactly is your abstract notion of a parallel region, and how is it<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">different from a concrete representation?<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Good question. The interface implementations will abstract away of the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">representation details that the optimization should not worry about such<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">as:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> - runtime library calls<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> - variable passing via structures<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> - intrinsic calls to encode information<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> - ...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">If you look at our variable privatization pass (which was not included<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">in this commit) it does not need to know if variable capturing is done<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">through a runtime library call (OpenMP), if captured variables are<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">handed to a special intrinsic to mark them (IntelPIR), or if capturing<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">is just a def-use chain crossing the parallel region boundaries (Tapir).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">The pass requires to know the mapping (variable in sequential code -><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">variable in parallel code) as well as the start/end instruction of the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">parallel region. Then it can test for conditions that would allow to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">privatize the variable, pass it by-value instead of by-reference, etc.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Similarly, other passes do not need to know the details of the encoding.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">You can probably see that for barrier elimination but also parallel<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">region expansion is actually only looking at the surrounding code to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">determine if expansion (and region merging) might make sense.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal" style="margin-left:.5in">> In the schemata below, our current five optimizations (described in<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> detail here [0]) are shown on the left, the abstract parallel IR<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> interface is is in the middle, and the representation specific<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> implementations is on the right.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> Optimization (A)nalysis/(T)ransformation Impl.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> ---------------------------------------------------------------------------<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> CodePlacementOpt \ /---> ParallelRegionInfo (A) ---------|-> KMPCImpl (A)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> RegionExpander -\ | | GOMPImpl (A)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> AttributeAnnotator -|-|---> ParallelCommunicationInfo (A) --/ ...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> BarrierElimination -/ |<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> VariablePrivatization / \---> ParallelIR/Builder (T) -----------> KMPCImpl (T)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I wasn’t able to understand parts of this figure. What info do<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">ParallelRegionInfo() and ParallelCommunicationInfo() provide? What<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">operations does ParallelIR/Builder provide?<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">So far not much. ParallelRegionInfo just collects and manages the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">parallel regions. These know about their extend (start/end), provide a<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">way to visit the blocks and instructions and have some helper functions<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">to reason about (potential) barriers. Nothing is specific to OpenMP.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">ParallelCommunicationInfo provides the mapping of variables outside the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">parallel region to the ones inside and tells you about the way they are<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">used (e.g., the pointer is actually a read-only container). Finally, the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">builder is the only way to modify the parallel regions (as we don't know<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">the actual representation in the optimization). Currently it can only<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">add/remove attributes but my local version can create parallel regions,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">build barriers, etc.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">For the current code check the two links from above.<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal" style="margin-left:.5in">> In our setting, a parallel region can be an outlined function called<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> through a runtime library but also a fork-join/attach-reattach region<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> embedded in an otherwise sequential code. The new optimizations will<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> provide parallelism specific optimizations to all of them (if<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> applicable). There are various reasons why we believe this is a<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> worthwhile effort that belongs into the LLVM codebase, including:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Before adding something this broad into the LLVM code base, I think we<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">need to understand a whole slew of things about it, starting with the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">questions above.<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I'm not sure about broad. Do you mean because of the commit size? Most<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">of it is boilerplate and it is a fully working horizontal prototype.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Actually, there are only 1091 lines (all types included) added to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">llvm/lib/.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal" style="margin-left:.5in">> 1. We improve the performance of parallel programs, today.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">There are a number of important parallel languages and both<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">language-specific and target-specific parallel optimizations. Just<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">because these five optimizations improve OpenMP program performance<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">isn’t enough to justify adding a new parallel IR (or interface) to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">LLVM, without knowing whether other languages, targets and<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">optimizations could be correctly and effectively implemented using<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">this approach.<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">This will not prevent us from testing other languages and optimizations,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">actually it should enable us to do so. If it somehow turns out that this<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">abstraction makes it impossible to implement an optimization correctly<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">and effectively and that optimization is very important to us we can<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">still get rid of the abstraction and keep the rest of the code. The<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">other optimization we have will still work even without the abstraction.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">The same holds true if we decide on a parallel IR representation and<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">do not need to support multiple ones anymore.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal" style="margin-left:.5in">> 2) It serves as a meaningful baseline for future discussions on<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> (optimized) parallel representations.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">We can’t just add a new parallel IR design (abstract or concrete) to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">mainline just to serve as a baseline for future discussions.<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Not _just_! Adding something beneficial to mainline that makes sense (at<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">least for me but let's see what others think) does not strike me as a<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">problem. If that would happen, we would finally have a baseline for<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">parallel optimizations instead of comparing unoptimized programs to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">optimized ones encoded in the new IR we came up with.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal" style="margin-left:.5in">> 3) It allows to determine the pros and cons of the different schemes<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> when it comes to actual optimizations and inputs.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Same problem.<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">This is not the same problem. When you design your IR and you have to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">make trade offs you would probably perform experiments and decide.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">However, for that you need a test bed to perform the experiments in, an<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">implementation of both choices, etc. If we combine optimizations we<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">think are meaningful with representations we think are reasonable we can<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">actually learn if and how they impact each other for benchmarks we care<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">about.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal" style="margin-left:.5in">> 4) It helps to identify problems that might arise once we start to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> transform parallel programs but _before_ we commit to a specific<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> representation.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I suspect you *are* committing to a specific representation, although<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">we can’t be sure until you provide more details.<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Please check out the links to the interface codes above.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal" style="margin-left:.5in">> Our prototypes for the OpenMP KMPC library (used by clang) already shows<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> significant speedups for various benchmarks [0]. It also exposed a (to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> me) prior unknown problem between restrict/noalias pointers and<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> (potential) barriers (see Section 3 in [0]).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> We are currently in the process of cleaning the code, extending the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> support for OpenMP constructs and adding a second implementation for a<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> embedded parallel regions. Though, a first horizontal prototype<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> implementation is already available for review [1].<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> Inputs of any kind are welcome and reviewers are needed!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> Cheers,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">> Johannes<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">[0] <a href="http://compilers.cs.uni-saarland.de/people/doerfert/par_opt18.pdf">
http://compilers.cs.uni-saarland.de/people/doerfert/par_opt18.pdf</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">[1] <a href="https://reviews.llvm.org/D47300">
https://reviews.llvm.org/D47300</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">--<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Johannes Doerfert<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">PhD Student / Researcher<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Compiler Design Lab (Professor Hack) / Argonne National Laboratory<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Saarland Informatics Campus, Germany / Lemont, IL 60439, USA<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Building E1.3, Room 4.31<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Tel. +49 (0)681 302-57521 : <a href="mailto:doerfert@cs.uni-saarland.de">
doerfert@cs.uni-saarland.de</a><<a href="mailto:doerfert@cs.uni-saarland.de">mailto:doerfert@cs.uni-saarland.de</a>> /
<a href="mailto:jdoerfert@anl.gov">jdoerfert@anl.gov</a><<a href="mailto:jdoerfert@anl.gov">mailto:jdoerfert@anl.gov</a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Fax. +49 (0)681 302-3065 : <a href="http://www.cdl.uni-saarland.de/people/doerfert">
http://www.cdl.uni-saarland.de/people/doerfert</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> *<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">—Vikram Adve<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">// Interim Head, Department of Computer Science<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">// Donald B. Gillies Professor of Computer Science<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">// University of Illinois at Urbana-Champaign<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">// Admin Assistant: Amanda Foley - <a href="mailto:ajfoley2@illinois.edu">
ajfoley2@illinois.edu</a><<a href="mailto:ajfoley2@illinois.edu">mailto:ajfoley2@illinois.edu</a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">// Google Hangouts: <a href="mailto:vikram.s.adve@gmail.com">
vikram.s.adve@gmail.com</a><<a href="mailto:vikram.s.adve@gmail.com">mailto:vikram.s.adve@gmail.com</a>> || Skype: vikramsadve<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">// Research page: <a href="http://vikram.cs.illinois.edu%3chttp:/vikram.cs.illinois.edu/%3e">
http://vikram.cs.illinois.edu<http://vikram.cs.illinois.edu/></a><o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">-- <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Johannes Doerfert<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">PhD Student / Researcher<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Compiler Design Lab (Professor Hack) / Argonne National Laboratory<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Saarland Informatics Campus, Germany / Lemont, IL 60439, USA<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Building E1.3, Room 4.31<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Tel. +49 (0)681 302-57521 : <a href="mailto:doerfert@cs.uni-saarland.de">
doerfert@cs.uni-saarland.de</a> / <a href="mailto:jdoerfert@anl.gov">jdoerfert@anl.gov</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Fax. +49 (0)681 302-3065 : <a href="http://www.cdl.uni-saarland.de/people/doerfert">
http://www.cdl.uni-saarland.de/people/doerfert</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
</div>
</body>
</html>