<div dir="ltr"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 3, 2020 at 3:10 PM Sriraman Tallam <<a href="mailto:tmsriram@google.com">tmsriram@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 3, 2020 at 8:30 AM Xiangling Liao <<a href="mailto:xiangxdh@gmail.com" target="_blank">xiangxdh@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">ping.<br><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">---------- Forwarded message ---------<br>From: <b class="gmail_sendername" dir="auto">Xiangling Liao</b> <span dir="auto"><<a href="mailto:xiangxdh@gmail.com" target="_blank">xiangxdh@gmail.com</a>></span><br>Date: Fri, May 29, 2020 at 3:15 PM<br>Subject: [cfe-dev] How to generate Unique Module identifier<br>To: <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>><br>Cc: <<a href="mailto:hubert.reinterpretcast@gmail.com" target="_blank">hubert.reinterpretcast@gmail.com</a>><br></div><br><br><div dir="ltr"><div>Hi All,<br><br>There have been recent discussions about how to generate unique module identifiers which can be embedded in AIX static init function names.<br> <br>On AIX, static init functions are sinit/sterm pairs looking like this:<br><br><span style="font-family:"times new roman",serif"><font size="2"><i>__sinit<priority #>_<unique module identifier><br>__sterm<priority #>_<unique module identifier></i></font></span><br><br>There is one sinit/sterm pair per priority number for each module.<br><br>The AIX linker collects static init functions simply based on their name. So we need to guarantee that each module has its own unique sinit/sterm pairs. To achieve that, we need a unique module identifier which will be used as a part of static init function name as suffix.<br><br></div>Our several thoughts about this so far are as follows:<br><div><br><b>1. `getUniqueModuleId` function to generate unique module identifier</b><br><a href="https://llvm.org/doxygen/ModuleUtils_8cpp_source.html#l00255" target="_blank"><u>https://llvm.org/doxygen/ModuleUtils_8cpp_source.html#l00255 </u></a><br><br><span style="font-family:"times new roman",serif"><i>“Produce unique identifier for a module by taking the MD5 sum of the names of the module's strong external symbols. However, if the module has no strong external symbols (such a module may still have a semantic effect if it performs global initialization), we cannot produce a unique identifier for this module, so we return the empty string.”</i></span><br><br>Issues with this `getUniqueModuleId` function are:<br>(1)Since this function does not take either `Internal linkage` or `WeakOnceODR linkage` global variables, so it is not able to return a string for the following cases:<br>1)<br><i><span style="font-family:"times new roman",serif">class test {<br>public:<br> test();<br> ~test();<br>};<br><br>static test t; //Internal linkage</span></i><br><br>2)<br><span style="font-family:"times new roman",serif"><i>extern "C" int puts(const char *);<br><br>template <typename = void><br>struct A {<br> A() { puts("hello\n"); }<br> ~A() { puts("bye\n"); }<br> static A instance;<br>};<br><br>template <typename T> A<T> A<T>::instance;<br>template A<> A<>::instance; //WeakOnceODR linkage</i></span><br><br>(2) Even if we add our own version `getUniqueModuleId` to care about above linkage types, the biggest issue here is content-based hashing won't work for the identical-content internal linkage case.<br><br><br><b>2. Source filename string as the module</b> <b>identifier</b><br>The `source filename` string is set to the original module identifier, which will be the name of the compiled source file when compiling from source through the clang front end. [<u><i> <a href="https://releases.llvm.org/10.0.0/docs/LangRef.html#source-filename" target="_blank">https://releases.llvm.org/10.0.0/docs/LangRef.html#source-filename</a></i></u> ]</div><div><br></div><div>That means if we have multiple objects compiled with the same command-line source file path, we have same module identifiers. The static init functions are not guaranteed to be unique.<br><br>Also, there's <a href="https://reviews.llvm.org/D73307?vs=on&id=262801&whitespace=ignore-most#change-Qs7eGduOQs42" target="_blank"><u>Unique Names for Functions with Internal Linkage</u></a> patch, whose solution does not guarantee uniqueness either. <br></div></div></div></div></div></blockquote><div><br></div><div>I just have a few thoughts. I worked on the unique names patch for internal linkage functions.</div><div><br></div><div>1) Low probability of collisions: I was only interested in reducing the probability of internal linkage functions getting the same names. In the context of PGO/FDO, this is useful because the profile information can be attributed to the right instance of the function. While the Unique names solution does not guarantee uniqueness, it makes it really small in practice.</div></div></div></blockquote><div>We would need more guaranteed uniqueness than this. It would not be good for packaged static libraries to have symbols that collide by accident with the user program or with other static libraries.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div>2) Name stability: We do not want the symbol names to constantly change either and there should be some amount of stability. This is because we generate profiles with one version of the source and use it to optimize a later version. Name changes across versions could make the profiles for those functions useless. In your case, how important is stability?</div></div></div></blockquote><div>Stability is important for keeping the relative ordering of C++ initialization/destruction for non-locals reasonably the same between builds.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br>3) Using the file system's attributes where possible: Just spitballing here, how about using say inode number in the hash for the symbol with Linux and similar attributes for other file systems. Looks like this could be kept stable and would handle the problem of identical source names.</div></div></div></blockquote><div>I believe we would want stability to extend to having the original source tree moved to another directory and a different source tree placed where the original source tree was in the directory structure.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div> <br></div><div><br></div><div>Thanks</div><div>Sri</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div class="gmail_quote"><div dir="ltr"><div><br><b>3. Using the information around the compilation process itself</b><br>Though using the information around the compilation process itself (PID, timestamp) can give us unique module identifiers, but it could be problematic for reproducibility.<br><br><b>4. source file full path + OutputFile name following -o option</b><br>Another thing hopeful is to use<b> the <i>source file full path plus the OutputFile name following -o option</i></b> as something to hash on or as a suffix for static init functions on AIX.<br><br>We didn’t find any precedent in LLVM to do so so far. And it requires us to pass -o ’s OutputFile name from `FrontendOpts` to `llvm::Module` like we pass each `Input` from `FrontendOpts.Inputs` to `llvm::Module` as SourceFileName.<br><a href="https://llvm.org/doxygen/Module_8cpp_source.html#l00073" target="_blank"><u>https://llvm.org/doxygen/Module_8cpp_source.html#l00073 </u></a><br><br></div><div><br></div><div><br>Any thoughts about what to hash on or encode into the unique ID we need?<br><br>Please let me know if there are any questions as well. Your feedback is appreciated.<br> <br>Regards,<br> <br>Xiangling Liao<br></div></div>
</div></div></div>
</blockquote></div></div>
</blockquote></div></div>