<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas;color:black">We are building a neural network compiler using LLVM, see
<a href="https://github.com/Microsoft/ELL">https://github.com/Microsoft/ELL</a>.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas;color:black">We want to put the neural network weights into a bunch of global float arrays because it allows us to more easily leverage<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas;color:black">Flash RAM on small embedded devices. For example, it enables these kinds of scenarios:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas;color:black"><a href="https://lovettchris.github.io/posts/keyword_spotting/">keyword spotting demo</a>.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas;color:black">We are finding some pretty bad compiler performance in some cases. For example,
<a href="https://gist.github.com/lovettchris/91e30bce1d18f16eddaf67306101e4e0">this github gist</a> contains a bitcode file which is a neural network compiled by ELL and it has about 30mb of floating point data, and when we put that through llc it takes 262
seconds to compile (on an Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz), but if we strip out the weights the "code" component of our neural network inference takes only 2 seconds to compile.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas;color:black">We’ve noticed a good improvement in LLVM 8.0 in this area, but we think there’s still a lot more that could be done. For example,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas;color:black">is it possible to dump big arrays of global floating point data into a binary without invoking huge assembly writer overhead?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas;color:black">Perhaps what is happening is the optimizer is trying to optimized away unused floats but we would like to disable that and just<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas;color:black">tell the compiler dump the floats into the object file, don’t bother trying to optimize them….<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Consolas;color:black">Any thoughts?<o:p></o:p></span></p>
</div>
</body>
</html>