<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="PL" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Dear community,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Recently I discovered that llvm gold linker plugin (LLVMgold.so) doesn't add llvm instrinsics symbols to the linker symbol table. I do not claim that something is necessary wrong, just want to share my observations with
the community.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Brief summary<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">If I create a static library with a custom version of ‘exp()’ math function and link it as follows:
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ clang -O0 -ffast-math -flto a.o -L. -Wl,-Bstatic -lexp -Wl,-Bdynamic -lm
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">exp gets resolved from libm instead of my static library (libexp.a) even though the static library appears first on the link command line.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Reproducer<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">1) As a reproducer I built a tiny static library (with the name libexp.a) with exp function defined:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ cat exp.c<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">double exp (double x)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">{<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> return x;<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ clang –c exp.c<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ ar -r libexp.a exp.o<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ nm -S libexp.a | grep exp<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">0000000000000000 0000000000000010 T exp<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2) I have another small piece of code which will use 'exp' from my library (libexp.a):<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ cat a.c<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#include <math.h><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">int main()<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">{<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> return exp(1.0);<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ clang -c -O0 -ffast-math -flto -o a.o a.c<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">After rC318193 we started lowering math libcalls to equivalent LLVM intrinsics:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ llvm-dis a.o -o - | grep llvm.exp<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> %2 = call fast double @llvm.exp.f64(double 1.000000e+00)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">3) I link my library (libexp.a) statically but link libm dynamically (as a fallback option):<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ clang -O0 -ffast-math -flto a.o -L. -Wl,-Bstatic -lexp -Wl,-Bdynamic –lm<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ nm -S a.out | grep exp<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> U exp@@GLIBC_2.2.5<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">So, exp symbol was taken from libm, not from my static library. Notice, that 'llvm.exp.f64' should be lowered into __exp_finite (see
<a href="https://bugs.llvm.org/show_bug.cgi?id=35672#c9">https://bugs.llvm.org/show_bug.cgi?id=35672#c9</a> ).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">To understand why this happened I traced ‘exp’ symbol:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$ clang -O0 -ffast-math -flto a.o -L. -Wl,-Bstatic -lexp -Wl,-Bdynamic -lm -Wl,--trace-symbol=exp
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">/usr/lib64/libm.so: definition of exp<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">/tmp/lto-llvm-8f9b9a.o: reference to exp<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> How linker works in those cases?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">In LTO build linker works in 2 phases:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">1. Runs through all the inputs and identifies which of them should be processed by the plugin. But while doing that linker already starts filling it’s symbol table.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Plugin at that point knows what symbols are defined and what referenced by the input files with bitcode inside (because it knows how to parse it). So, plugin reports that symbols to the linker.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2. After all inputs are read linker signals to the plugin to do it’s magic.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> We run LTO and produce one “fat” relocatable object file and feed it back to the linker.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Linker finishes it’s job as usual.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">More details of this process are described here: https://gcc.gnu.org/wiki/whopr/driver .<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">If we apply the process described above to our case, we will understand what happened:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">1st pass:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> We see a.o. It is claimed by the plugin. But plugin ignored llvm.exp.f64 intrinsic, so ‘exp’ symbol was not reported to the linker as referenced symbol.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> We see static linking of libexp. We are linking static libraries only if someone referenced any symbols defined in it, so we skip libexp (no one referenced ‘exp’).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> We see dynamic linking of libm. We will link to it unconditionally, so linker captures all symbols in libm. It found definition of ‘exp’ and insert it to the linker symbol table.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> We run LTO stage. /tmp/lto-llvm-8f9b9a.o was added to the linker inputs.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2nd pass:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> There is a reference to exp in /tmp/lto-llvm-8f9b9a.o, but we have it’s definition already (in libm).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> We see libexp again, but it’s too late, we already have exp defined in libm, and it won’t be overridden.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I confirmed all the above by adding debug prints in LLVMGold plugin (LLVMgold.so) and built debug version of binutils (added debug prints to gold linker).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Behavior is similar for gold and bfd linkers.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">This problem goes away if I add ‘-u exp’ to the linker command. This option adds undefined symbol ‘exp’ to the linker symbol right at the beginning of the process, so it will be taken in 1st pass from libexp. But of course,
it can't be a solution to this issue.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> How it affects us?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">We have our own runtime library which provides implementation for math functions like exp, log, etc. So, we use it as a replacement for libm.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">We link libm dynamically because not every distribution is supplied with static version of libm.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">It's not so trivial to report symbols that correspond to llvm intrinsics because we don't know for sure to which exact symbols they will be expanded. For example, in rL322087 we started lowering llvm.exp.* to “__exp_finite”
if –ffast-math (or -Ofast) is provided.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Any thoughts on this?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">-Denis Bakhvalov<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Employee of Intel, compiler development team.<o:p></o:p></span></p>
</div>
<p>---------------------------------------------------------------------<br>
<strong style="line-height: 11.25pt;"><span style="font-size: 9pt; color:
#595959;"><span style="font-family: 'Arial Narrow', sans-serif;">Intel
Technology Poland sp. z o.o.<br></span></span></strong><span style="color:
#595959; font-family: 'Arial Narrow', sans-serif; font-size: 9pt; line-height:
11.25pt;">ul. Słowackiego 173 | 80-298 Gdańsk | Sąd Rejonowy Gdańsk
Północ
| VII Wydział Gospodarczy Krajowego Rejestru Sądowego - KRS 101882 | NIP
957-07-52-316 | Kapitał zakładowy 200.000 PLN.</span></p><p>
<span style="font-size:8.0pt;font-family:"Arial
Narrow","sans-serif";
mso-fareast-font-family:"Times New
Roman";mso-bidi-font-family:Arial;
color:#595959;mso-ansi-language:EN-US;mso-fareast-language:EN-US;mso-bidi-language:
AR-SA">Ta wiadomość wraz z załącznikami jest przeznaczona dla określonego
adresata i może zawierać informacje poufne. W razie przypadkowego otrzymania
tej wiadomości, prosimy o powiadomienie nadawcy oraz trwałe jej usunięcie;
jakiekolwiek przeglądanie lub rozpowszechnianie jest zabronione.<br>
This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). If you are not the intended recipient,
please
contact the sender and delete all copies; any review or distribution by others
is strictly prohibited.</span></p><p class="MsoNormal"><o:p></o:p></p>
</body>
</html>