<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jul 22, 2016, at 2:34 PM, Taewook Oh <<a href="mailto:twoh@fb.com" class="">twoh@fb.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);"><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><span style="font-size: 11pt; font-family: Calibri;" class="">Yes, I have the repro, though I can’t publish it externally. It would be great if you can upstream the patch so I can try it. Thank you for your explanation as well!<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><span style="font-size: 11pt; font-family: Calibri;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><span style="font-size: 11pt; font-family: Calibri;" class="">-- Taewook<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><span style="font-size: 11pt; font-family: Calibri;" class=""><o:p class=""> </o:p></span></div><div style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0in 0in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><b class=""><span style="font-family: Calibri;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-family: Calibri;" class=""><<a href="mailto:mehdi.amini@apple.com" class="">mehdi.amini@apple.com</a>> on behalf of Mehdi Amini <<a href="mailto:mehdi.amini@apple.com" class="">mehdi.amini@apple.com</a>><br class=""><b class="">Date:<span class="Apple-converted-space"> </span></b>Friday, July 22, 2016 at 2:16 PM<br class=""><b class="">To:<span class="Apple-converted-space"> </span></b>Taewook Oh <<a href="mailto:twoh@fb.com" class="">twoh@fb.com</a>><br class=""><b class="">Cc:<span class="Apple-converted-space"> </span></b>via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>><br class=""><b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [llvm-dev] [ThinLTO] Using two different IRMovers for the same composite module? (related to PR28180)<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><o:p class=""> </o:p></div></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><o:p class=""> </o:p></div><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">On Jul 22, 2016, at 1:50 PM, Taewook Oh via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" style="color: purple; text-decoration: underline;" class="">llvm-dev@lists.llvm.org</a>> wrote:<o:p class=""></o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><o:p class=""> </o:p></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class=""><span style="font-size: 11pt; font-family: Calibri;" class="">Hello,</span><span style="font-family: Calibri;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class=""><span style="font-size: 11pt; font-family: Calibri;" class=""> </span><span style="font-family: Calibri;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class=""><span style="font-size: 11pt; font-family: Calibri;" class="">While trying ThinLTO, I ran into an assertion failure in IRMover: <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__llvm.org_bugs_show-5Fbug.cgi-3Fid-3D28180&d=CwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=kOsLCgQzH7N8ptZ7diJD9g&m=YPo1XVjbxI5dqJXOdMZRPzHnL3CBAEk6IR74Edlb3Jg&s=C4gDkGqx2eaddJPKkXGtaZqrAn7mTi2w675GAwvY0G8&e=" style="color: purple; text-decoration: underline;" class=""><span style="color: rgb(149, 79, 114);" class="">https://llvm.org/bugs/show_bug.cgi?id=28180</span></a>.</span><span style="font-family: Calibri;" class=""><o:p class=""></o:p></span></div></div></div></blockquote><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">Great, we encountered this bug last month and I have an fix internally but wasn’t sure how to reproduce (I didn’t have any source with internal bug report), so I haven’t upstream the patch yet.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">Do you have the repro?<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><o:p class=""> </o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><br class=""><br class=""><o:p class=""></o:p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class=""><span style="font-size: 11pt; font-family: Calibri;" class="">I found that the assertion failure is happening because IRMover tries to map the metadata that already mapped in the destination module, and it seems that this happens because two different IRMovers are used for the same destination (or composite) module.</span><span style="font-family: Calibri;" class=""><o:p class=""></o:p></span></div></div></div></blockquote><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">It is not clear to me how using two different IRMovers is the issue: as you mentioned, the assertions is encountered when the metadata is already in the map, a new IRMovers would have a new fresh map.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">My debugging of this issue lead me to the new "ODR type uniquing” feature in the context as the culprit. In this mode, when multiple modules are loaded in the context the composite type metadata are uniqued by id. It means that the same composite type (same as same pointer in memory) can be reached from two modules (here source and destination). So the mapper may reach a metadata in the source module and try to map it to the destination module while it is already there (but not in the map).<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">This happens only in ThinLTO and not in LTO because LTO starts with an empty module, so when you move the first module into the "merged module”, the map gets initialized. In ThinLTO the mover starts with the destination module not empty.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">— <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">Mehdi<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><o:p class=""> </o:p></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class=""><span style="font-size: 11pt; font-family: Calibri;" class="">During LTO, an IRMover is created in thinLTOBackendTask function(tools/gold/gold-plugin.cpp). linkInModule function, which is called by thinLTOBackendTask, calls the ‘move’ function of this IRMover. The other IRMover is created when “TheLinker” is created in FunctionImporter::importFunctions (lib/Transforms/IPO/FunctionImport.cpp). thinLTOBackendTask invokes FunctionImporeter::importFunctions as well, with a call chain of thinLTOBackendTask<span class="apple-converted-space"> </span></span><span style="font-size: 11pt; font-family: Wingdings;" class="">à</span><span style="font-size: 11pt; font-family: Calibri;" class="">CodeGen::runAll (tools/gold/gold-plugin.cpp)<span class="apple-converted-space"> </span></span><span style="font-size: 11pt; font-family: Wingdings;" class="">à</span><span class="apple-converted-space"><span style="font-size: 11pt; font-family: Calibri;" class=""> </span></span><span style="font-size: 11pt; font-family: Calibri;" class="">CodeGen::runLTOPasses ((tools/gold/gold-plugin.cpp)</span><span style="font-size: 11pt; font-family: Wingdings;" class="">à</span><span class="apple-converted-space"><span style="font-size: 11pt; font-family: Calibri;" class=""> </span></span><span style="font-size: 11pt; font-family: Calibri;" class="">FunctionImporter::importFunctions.</span><span style="font-family: Calibri;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class=""><span style="font-size: 11pt; font-family: Calibri;" class=""> </span><span style="font-family: Calibri;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class=""><span style="font-size: 11pt; font-family: Calibri;" class="">As these two IRMovers share the same destination module, when the second IRMover tries to map the metadata already mapped by the first IRMover, it eventually results the assertion failure. It seems that IRMover maintains SharedMDs to keep the metadata mapping record across the multiple calls of its move function, but that doesn’t help between two separate IRMovers.</span><span style="font-family: Calibri;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class=""><span style="font-size: 11pt; font-family: Calibri;" class=""> </span><span style="font-family: Calibri;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class=""><span style="font-size: 11pt; font-family: Calibri;" class="">What would be the right fix for this? Please let me know if I misunderstand something.</span><span style="font-family: Calibri;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class=""><span style="font-size: 11pt; font-family: Calibri;" class=""> </span><span style="font-family: Calibri;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class=""><span style="font-size: 11pt; font-family: Calibri;" class="">Thanks,</span><span style="font-family: Calibri;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class=""><span style="font-size: 11pt; font-family: Calibri;" class="">Taewook</span><span style="font-family: Calibri;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class=""><span style="font-size: 11pt; font-family: Calibri;" class=""> </span><span style="font-family: Calibri;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class=""><span style="font-size: 11pt; font-family: Calibri;" class=""> </span><span style="font-family: Calibri;" class=""><o:p class=""></o:p></span></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><span style="font-size: 9pt; font-family: Helvetica; background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">_______________________________________________</span><span style="font-size: 9pt; font-family: Helvetica;" class=""><br class=""><span style="background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">LLVM Developers mailing list</span><br class=""></span><a href="mailto:llvm-dev@lists.llvm.org" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica; color: rgb(149, 79, 114); background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">llvm-dev@lists.llvm.org</span></a><span style="font-size: 9pt; font-family: Helvetica;" class=""><br class=""></span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.llvm.org_cgi-2Dbin_mailman_listinfo_llvm-2Ddev&d=CwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=kOsLCgQzH7N8ptZ7diJD9g&m=YPo1XVjbxI5dqJXOdMZRPzHnL3CBAEk6IR74Edlb3Jg&s=-JVet3M9T-EiSCCak1-U757TIzbzbTT4zxkz4xbvlBs&e=" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica; color: rgb(149, 79, 114); background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</span></a><o:p class=""></o:p></div></div></blockquote></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class=""><o:p class=""> </o:p></div></div></div></div></div></blockquote></div><br class=""></div></div></body></html>