<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Mar 26, 2018, at 6:03 PM, <a href="mailto:katya.romanova@sony.com" class="">katya.romanova@sony.com</a> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">Hi Steven,<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">Look at my replies inline (below your comments).<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">Katya.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div class=""><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">From:</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="Apple-converted-space"> </span><a href="mailto:stevenwu@apple.com" style="color: purple; text-decoration: underline;" class="">stevenwu@apple.com</a><span class="Apple-converted-space"> </span><<a href="mailto:stevenwu@apple.com" style="color: purple; text-decoration: underline;" class="">stevenwu@apple.com</a>><span class="Apple-converted-space"> </span><br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Thursday, March 22, 2018 4:46 PM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span>Romanova, Katya <<a href="mailto:katya.romanova@sony.com" style="color: purple; text-decoration: underline;" class="">katya.romanova@sony.com</a>><br class=""><b class="">Cc:</b><span class="Apple-converted-space"> </span>Teresa Johnson <<a href="mailto:tejohnson@google.com" style="color: purple; text-decoration: underline;" class="">tejohnson@google.com</a>>; Mehdi AMINI <<a href="mailto:joker.eph@gmail.com" style="color: purple; text-decoration: underline;" class="">joker.eph@gmail.com</a>>; Rafael Avila de Espindola <<a href="mailto:rafael.espindola@gmail.com" style="color: purple; text-decoration: underline;" class="">rafael.espindola@gmail.com</a>>; Peter Collingbourne <<a href="mailto:peter@pcc.me.uk" style="color: purple; text-decoration: underline;" class="">peter@pcc.me.uk</a>>; Hans Wennborg <<a href="mailto:hans@chromium.org" style="color: purple; text-decoration: underline;" class="">hans@chromium.org</a>>; Reid Kleckner <<a href="mailto:rnk@google.com" style="color: purple; text-decoration: underline;" class="">rnk@google.com</a>>;<span class="Apple-converted-space"> </span><a href="mailto:bd1976llvm@gmail.com" style="color: purple; text-decoration: underline;" class="">bd1976llvm@gmail.com</a>;<span class="Apple-converted-space"> </span><a href="mailto:llvm-dev@lists.llvm.org" style="color: purple; text-decoration: underline;" class="">llvm-dev@lists.llvm.org</a><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [pre-RFC] Data races in concurrent ThinLTO processes<o:p class=""></o:p></span></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Hi Katya<o:p class=""></o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Thanks for investigating this. Here is my thought inline.<o:p class=""></o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><br class=""><br class=""><o:p class=""></o:p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">On Mar 22, 2018, at 1:32 AM,<span class="Apple-converted-space"> </span><a href="mailto:katya.romanova@sony.com" style="color: purple; text-decoration: underline;" class="">katya.romanova@sony.com</a><span class="Apple-converted-space"> </span>wrote:<o:p class=""></o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Hello,</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">I am sending the following proposal to discuss issues and solutions regarding data races in concurrent ThinLTO processes.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">This caught my attention when we encountered a race condition in ThinLTO with caching.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">While looking into how ThinLTO deals with the problem of cache files reads/writes/deletes I spotted a lot of problems: some of them are related to data races, others - to file/resource sharing between different processes. I wanted to point out these problems and start the discussion about potential solutions. I would like to get your feedback.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Problem #1: In certain common situations, ‘rename’ can fail, causing a data race later on.</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Look at the following code in the function ‘write’ in ThinLTOCodeGenerator.cpp</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">std::error_code EC =</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> sys::fs::createTemporaryFile("Thin", "tmp.o", TempFD, TempFilename);</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">if (EC) {</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> errs() << "Error: " << EC.message() << "\n";</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> report_fatal_error("ThinLTO: Can't get a temporary file");</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">}</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">{</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> raw_fd_ostream OS(TempFD, /* ShouldClose */ true);</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> OS << OutputBuffer.getBuffer();</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">}</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">// Rename to final destination (hopefully race condition won't matter here)</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">EC = sys::fs::rename(TempFilename, EntryPath); </span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Compared to a race-prone direct write of a buffer to a file, this code avoids a data race by first writing a buffer to a temp file and then renaming that temp file to become the final file in the cache. After r315079, when ‘rename’ became more POSIX-compliant, this scheme guarantees atomicity of writing a file to the cache,</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">(except on Wine, where (in some cases) non-atomic ::MoveFileExW is used).</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">However, ‘rename’ might fail and return an error code in several different situations. If this happens, we will fall back to a direct write to the file, which is neither atomic, nor avoids race conditions (causing problem #3).</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">An<span class="apple-converted-space"> </span>example where 'rename' can fail on both Windows and Unix-like systems, causing us to fall back to using non-atomic write, is problem #2.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span lang="EN-GB" style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Solution:</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">All solutions for problem #3 (see below) will take care of problem #1.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div></div></blockquote><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">I don't think there is a race here but it can probably be optimized. I don't see LTOCodeGenerator fall back to non-atomic write in the code. If the rename failed, it will just fatal_error and exit. Maybe a better solution is just leave the temp file and use it as output. The file will be recompiled if needed again (turns into a cache miss).<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><br class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">I think there is a non-atomic write (and race) here. See below. If rename failed, we remove the temp file and write to the cache file directly (OS << OutputBuffer.getBuffer()).<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""> // Rename to final destination (hopefully race condition won't matter here)<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""> EC = sys::fs::rename(TempFilename, EntryPath);<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""> if (EC) {<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""> sys::fs::remove(TempFilename);<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""> raw_fd_ostream OS(EntryPath, EC, sys::fs::F_None);<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""> if (EC)<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""> report_fatal_error(Twine("Failed to open ") + EntryPath +<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""> " to save cached entry\n");<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""> OS << OutputBuffer.getBuffer();<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""> }<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""> }<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">What we can do the following way:<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">Option 1: If the rename fails, we could simply use OutputBuffer (which has the same content as our temp file), bypassing the cache altogether.</span></div></div></div></div></div></blockquote><div><br class=""></div><div>Option 1 is much better for the current model. To be safe, we should avoid non-atomic write (which is directly writing the cache file) to the cache. When we perform such write, the cache is in an invalid state and we have nothing to prevent reading from it.</div><br class=""><blockquote type="cite" class=""><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">Option 2: We could try reading from the existing cache first, and then, if this read fails, we could use OutputBuffer (in this case, we will be closer to the existing code). I don’t understand why do we have to read from the existing cache first (even if we just created this cache). I suspect this was done for a reason. There is a comment below with the explanation, but it doesn’t make it more clear to me. If this reason/explanation makes sense, then we should do Option 2. This is exactly what I previously proposed in Solution #0 for Problem #3. <o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> // Commit to the cache (if enabled)<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> CacheEntry.write(*OutputBuffer);<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> if (SavedObjectsDirectoryPath.empty()) {<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> // We need to generated a memory buffer for the linker.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> if (!CacheEntryPath.empty()) {<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> // Cache is enabled, reload from the cache<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> // We do this to lower memory pressuree: the buffer is on the heap<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> // and releasing it frees memory that can be used for the next input<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> // file. The final binary link will read from the VFS cache<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> // (hopefully!) or from disk if the memory pressure wasn't too high. <- I don’t understand why we are reading from the cache file (via tryLoadingBuffer), though we already have the same content in OutputBuffer?<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> auto ReloadedBufferOrErr = CacheEntry.tryLoadingBuffer(); <-The explanation says this is done to reduce the memory pressure. But it’s not clear to me why this helps to reduce memory pressure, since we still have to keep<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> if (auto EC = ReloadedBufferOrErr.getError()) { <- the content of the cache file in a buffer, pointed to by ReloadedBufferOrErr). <o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> // On error, keeping the preexisting buffer and printing a<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> // diagnostic is more friendly than just crashing.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> errs() << "error: can't reload cached file '" << CacheEntryPath<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> << "': " << EC.message() << "\n";<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> } else {<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> OutputBuffer = std::move(*ReloadedBufferOrErr);<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> }<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; color: rgb(31, 73, 125);" class=""> }</span><span style="font-size: 14pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: "Courier New";" class=""> </span><span style="font-size: 14pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span lang="EN-GB" style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Problem #2 (Use of POSIX-compliant ‘rename’ function is not always suitable for ThinLTO)</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">We just encountered this issue in Sony. For ThinLTO, we use the function ‘rename’, which after r315079 doesn’t support renaming across the different logical volumes on Windows.<span class="apple-converted-space"> </span></span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Let’s say a user has a %TEMP% directory on volume C:\, but he/she passed the caching directory name located on volume D:\, then ‘rename’ fails. Since we don’t support renaming across different volumes after r315079, we then fall back to writing to the cache file directly (which is not atomic) and hit a data race.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">We anticipate that this will be a common situation for our users, many of whom are Windows “power users” and have multiple disks for different purposes.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">I think this problem doesn’t only affect Windows. The same issue might happen on Linux/MacOS, if the user's $TEMP/$TMP directory is located in a different file system than the user-specified caching directory.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Solution #1 (not so good):</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">The patch below solves this problem on Windows, however, we will lose the advantage that ‘rename’ gained in r315079 (i.e., its atomicity), which is not good.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">This patch assumes that the function 'rename' in Windows/Path.inc is expected to work when the source and destination are located of different logical volumes or different file systems. Note that function ‘rename’ is not ThinLTO-specific and might have some other consumers who want this behavior (which was the behavior before r315079). However, it seems that this assumption has changed later to make ‘rename’ more POSIX-compliant. In this case, we should add a comment to the function 'rename' so that its users knew that renaming across different volumes is not supported by design. Or we could have two different functions, one POSIX-compliant, not allowing renames across different volumes, another non-POSIX-compliant.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Before r315079, the function 'rename', (in the case when the destination file isn't opened), boiled down to calling the function 'MoveFileExW' with the MOVEFILE_COPY_ALLOWED flag set, allowing renaming files across the volumes.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">The current implementation of the function ‘rename’ (in 'rename_internal'), essentially calls 'SetFileInformationByHandle'. This function is fast and atomic, so in a sense, it's an improvement over the previously used 'MoveFileExW'. However, 'SetFileInformationByHandle' doesn't work when the source and destination are located on the different volumes.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">My fix just falls back to calling 'MoveFileExW' when 'SetFileInformationByHandle' returns an error 'ERROR_NOT_SAME_DEVICE'.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 32, 96);" class=""> </span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">+ // Wine doesn't support SetFileInformationByHandle in rename_internal.</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">+ // Rename_internal doesn't work accross different disks. In both of these</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">+ // cases fall back to using MoveFileEx.</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> if (EC ==</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">- std::error_code(ERROR_CALL_NOT_IMPLEMENTED, std::system_category())) {</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">- // Wine doesn't support SetFileInformationByHandle in rename_internal.</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">- // Fall back to MoveFileEx.</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">+ std::error_code(ERROR_CALL_NOT_IMPLEMENTED, std::system_category()) ||</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">+ EC == std::error_code(ERROR_NOT_SAME_DEVICE, std::system_category())) {</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> SmallVector<wchar_t, MAX_PATH> WideFrom;</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> if (std::error_code EC2 = realPathFromHandle(FromHandle, WideFrom))</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> return EC2;</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> if (::MoveFileExW(WideFrom.begin(), WideTo.begin(),</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">- MOVEFILE_REPLACE_EXISTING))</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">+ MOVEFILE_COPY_ALLOWED | MOVEFILE_REPLACE_EXISTING))</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> return std::error_code();</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> return mapWindowsError(GetLastError());</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> }</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Note, that when the source and destination are located on the different logical volumes, we will effectively perform a copy, followed by a delete, which are not atomic operations. Since copying to a different volume might be quite time consuming, we also increase the probability that some other process starts to rename to the same destination file.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">So, this fix for ‘rename’ is not good for ThinLTO (we should do something else in this case). ‘rename’ is a generic function and has many different users, but unless renaming across the volumes is needed by someone else, other than ThinLTO, this patch shouldn’t be accepted.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Solution #2 (better)</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Here I only try to fix a ThinLTO issue, not a generic ‘rename’ issue as in solution #1:</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">For ThinLTO+caching we don’t have to create temp files in %TEMP% or %TMP% directories (or $TMP/$TEMP). We can create them in the same location where the cached files are stored or in a ‘temp’ subfolder within this folder. With this approach:</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div style="margin-left: 0.5in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif; text-indent: -0.25in;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">-</span><span style="font-size: 7pt;" class=""> <span class="apple-converted-space"> </span></span><span style="font-size: 18pt; font-family: "Courier New";" class="">If the patch described in Solution #1 gets accepted (for the sake of other consumers), then from ThinLTO’s perspective ‘rename’ will stay atomic.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div style="margin-left: 0.5in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif; text-indent: -0.25in;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">-</span><span style="font-size: 7pt;" class=""> <span class="apple-converted-space"> </span></span><span style="font-size: 18pt; font-family: "Courier New";" class="">If the patch doesn’t get accepted, then rename won’t fail, since we only rename the files within the same directory.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 32, 96);" class=""> </span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 32, 96);" class="">Solution #3</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">All the solutions for problem #3 (see below) will take care of problem #2.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div></div></blockquote><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">I am surprised that we are not doing #2 already. Rename across the volumes is not atomic thus it can cause issues.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><br class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">Yes, we are calling createTemporaryFile, which creates a file in the location pointed by $TEMP, $TMP, etc.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""> sys::fs::createTemporaryFile("Thin", "tmp.o", TempFD, TempFilename);<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">So, it looks like this problem is applicable to Unix-like systems too (assuming that your $TEMP and cache directory are in the different file systems).<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div></div></div></div></blockquote><div><br class=""></div><div>I don't think there is a file system that supports atomic rename between volumes. This is something we can fix easily.</div><br class=""><blockquote type="cite" class=""><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class=""><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 32, 96);" class=""> </span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 32, 96);" class=""> </span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Problem #3 (data race)</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Look at the following code in ThinLTOCodeGenerator.cpp. This code gets executed if the ‘rename’ function failed (e.g., because of the problem #1 or problem #2 described above).</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">EC = sys::fs::rename(TempFilename, EntryPath);</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">if (EC) {</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> sys::fs::remove(TempFilename);</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> raw_fd_ostream OS(EntryPath, EC, sys::fs::F_None);</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> if (EC)</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> report_fatal_error(Twine("Failed to open ") + EntryPath +</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> " to save cached entry\n");</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class=""> OS << OutputBuffer.getBuffer(); <span class="apple-converted-space"> </span></span></b><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: red;" class="">// Two ThinLTO processes can write to the same file here</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: red;" class=""> // causing data race.</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Here, we have a data race problem. We actually stumbled across it while testing one of the games with ThinLTO+caching.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">We need to find a generic solution to solve data race problem, while ensuring ‘atomicity’ of writing to cache files. I’m not a ThinLTO expert and I might not be aware of some ThinLTO constraints. Let me know which problems you see with the two solutions below. Hopefully, it will trigger a further discussion.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span lang="EN-GB" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Solution #0 (it won’t require much modifications):</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div style="margin-left: 0.85in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif; text-indent: -0.6in;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">(a)</span><span style="font-size: 7pt;" class=""> <span class="apple-converted-space"> </span></span><span style="font-size: 18pt; font-family: "Courier New";" class="">If ‘rename’ fails, do not write into the cache directly (we don’t want to have non-atomic writes!).</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div style="margin-left: 0.85in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif; text-indent: -0.6in;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">(b)</span><span style="font-size: 7pt;" class=""> <span class="apple-converted-space"> </span></span><span style="font-size: 18pt; font-family: "Courier New";" class="">If ‘rename’ fails, try to read from the cache. </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div style="margin-left: 0.85in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New"; color: rgb(0, 112, 192);" class="">auto ReloadedBufferOrErr = CacheEntry.tryLoadingBuffer();</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div style="margin-left: 0.85in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif; text-indent: -0.6in;" class=""><b class=""><span lang="EN-GB" style="font-size: 18pt; font-family: "Courier New";" class="">(c)</span></b><b class=""><span lang="EN-GB" style="font-size: 7pt;" class=""> <span class="apple-converted-space"> </span></span></b><span style="font-size: 18pt; font-family: "Courier New";" class="">If reading from the cache fails simply use the file that you just compiled directly (bypassing the cache entirely).</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div style="margin-left: 0.85in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div style="margin-left: 0.85in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">This solution might cause cache misses (causing recompilations), but at least it should prevent data races when two ThinLTO processes write to the same file (which can produce invalid cache!). Correctness is more important than optimization. It’s worth noticing another shortage of this solution: unlike generic solution #1, #2 and #3 described below this particular solution won’t solve the problem #4.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div style="margin-left: 0.85in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span lang="EN-GB" style="font-size: 18pt; font-family: "Courier New";" class=""> </span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div style="margin-left: 0.85in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">BTW, we should do something with WINE support in ‘rename’ (which is non-atomic). I didn’t want to list it as a separate problem here, but it is a problem. I could file a separate bug, if necessary.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div style="margin-left: 0.85in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span lang="EN-GB" style="font-size: 18pt; font-family: "Courier New";" class=""> </span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Solution #1 (naïve generic solution):</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">(a) If the process wants to write to the cache file, it opens it with 'exclusive read/write' access.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">(b) If a process wants to write to the cache file that is ‘exclusively’ opened by some other process, we could assume that the cache file will be successfully created by the first process and simply return from ‘write’ function. Different processes writing to the cache file of the same name, are writing the same content, right?</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">(c) If the process needs to read from the cache file, it might wait a bit while the file is open for 'exclusive write'. If within a certain period the cache file doesn’t get released by a writer or gets removed by a pruner – oh, well, we have a hit miss. After all, using cache is just an acceleration. Correctness is more important.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">(d) If we are concerned about data integrity of a cache file (e.g., a writer started to write to a file and something bad happened in the middle), we could do additional checks (I suspect that this might be already implemented in ThinLTO). A writer could add a unique hash at the end of the cache file or a CRC32 for each section of the file, which could be an indication that the write to this file was successful. A reader checks that this unique hash or a CRC32 checksum matches its expectations.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">I'm sure that there are good reasons why this "naive" solution wasn't implemented in the first place. If this solution is picked by LLVM community, I could write a working prototype for it that we could apply to ThinLTO.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Solution #2 (better generic solution):</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">It looks like cache file sharing between several ThinLTO processes is a classic readers-writers problem.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""><a href="https://en.wikipedia.org/wiki/Readers%E2%80%93writers_problem" style="color: purple; text-decoration: underline;" class=""><span style="color: rgb(149, 79, 114);" class="">https://en.wikipedia.org/wiki/Readers%E2%80%93writers_problem</span></a></span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">We could use read-write locks for global inter-process file sharing.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""><a href="https://en.wikipedia.org/wiki/Readers%E2%80%93writer_lock" style="color: purple; text-decoration: underline;" class=""><span style="color: rgb(149, 79, 114);" class="">https://en.wikipedia.org/wiki/Readers%E2%80%93writer_lock</span></a></span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">ThinLTO's writer (creator/deleter) must acquire a write lock before writing to a file and release a write lock when it's done writing.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">ThinLTO’s user (reader) must acquire a read lock before reading a file and release a read lock when it's done reading. On a Unix-like system, read-write locks are part of POSIX-threads (pthreads). There is an implementation of Slim read-write locks on Windows, but unfortunately for in-process only (i.e., the locks can’t be shared among different processes), so it won’t work for us.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""><a href="https://msdn.microsoft.com/en-us/library/windows/desktop/aa904937(v=vs.85).aspx" style="color: purple; text-decoration: underline;" class=""><span style="color: rgb(149, 79, 114);" class="">https://msdn.microsoft.com/en-us/library/windows/desktop/aa904937(v=vs.85).aspx</span></a></span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">However, on Windows, we could implement read-write locks ourselves, using a combination of a named mutex and a named semaphore (any unique global name could be used for creating a named mutex/semaphore, the simplest one will be the cache file name itself).</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Here is an example of an implementation:</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""><a href="http://www.zachburlingame.com/2011/06/simple-reader-writer-lock-in-c-on-win32/" style="color: purple; text-decoration: underline;" class=""><span style="color: rgb(149, 79, 114);" class="">http://www.zachburlingame.com/2011/06/simple-reader-writer-lock-in-c-on-win32/</span></a></span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">If this solution is picked, I could write a working prototype for it that we could apply to ThinLTO.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Solution #3 (hybrid generic solution)</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Basically, use solution #2 as a base and integrate some optimizations and completeness checking features from solution #1 (1.b and 1.d respectively).</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Problem #4 (cache misses could have been avoided via synchronization).</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">With our current approach we might often race for the cache file resources and as a result have cache misses.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Of course, it's not as dangerous as the data race problem #3, but it definitely lowers the efficiency of caching.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">(a) The reader checks that a file exists, but then the pruner deletes it before the reader had a chance to read it. Cache miss.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">(b) The reader starts reading a file (block by block) and at this time the pruner removes the file. The read might fail. This is OS-specific, but likely a cache miss.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">(c) If the writer will rename a temp file into a file that is currently being read by a reader, I don't expect anything good out of it (the behavior is OS-specific). In the best case scenario, the read will fail, in the worst one the reader will read the wrong content. So, it's a cache miss or a correctness issue.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">(d) This is not a very likely event, but what if two processes are trying to rename to the same file at the same time?</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Solution:</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Solutions #1, #2 and #3 for problem #3 will take care of problem #4.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">(Potential) problem #5:</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">The<span class="apple-converted-space"> </span>implementation of the function ‘rename’ on Windows used by ThinLTO heavily depends on the function CreateFileW or, to be more exact, on its flag FILE_SHARE_DELETE being supported and working correctly on *all* the file systems that can be mounted on Windows. Is the FILE_SHARE_DELETE flag supported on all non-native Windows file systems that we care about? Is it supported on WINE? On FAT32?</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">If the flag FILE_SHARE_DELETE is not supported, the ‘rename’ fails, and we have problem #3.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Solution:</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">All the solutions for problem #3 will take care of problem #5.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Please let me know what do you think about the problems/solutions discussed here. Hopefully this pre-RFC shapes into a RFC soon.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div></div></blockquote><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">I am not really concerned with all the problems, except Problem #2. From my understanding, LTO cache is for incremental build only.<span style="color: rgb(31, 73, 125);" class=""><o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">Yes.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">For every file in the cache, there should be one single writer, otherwise, it is a hash collision and will result in miscompile (or link error most likely). The cached file will only be used when you rebuild the binary. Lots of data races are in situations that are not supported.<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">Are you aware of any other data races when we share LTO cache between two links, except the one that we identified above? Please let me know. I’d rather know all the potential problems/limitation in advance.</span></div></div></div></div></div></div></blockquote><div><br class=""></div><div>The most common issue you are going to hit if you try to share LTO cache is that the file can be removed/updated when the other process is half way reading the file. I can consistently reproduce the issue if you try to build all LLVM/Clang tools with a shared LTO cache.</div><br class=""><blockquote type="cite" class=""><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">For its current design, you cannot shared the LTO cache between two linking. For example, if you are building llc and libLTO from LLVM and they shared many same static archives with common LTO object files, you still need to use two different LTO caches. <o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">But what if we fix this particular problem? We could either do it the way Peter implemented it for the new LTO API or the way we discussed above. I assume that after fixing this, will we be able to share the same cache between 2 links.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">The question is, how exactly we should fix it for C LTO API? I’m aware of only two major C LTO API stakeholders (Apple and Sony), so most likely nobody else cares much about legacy C API. Peter generously proposed to port the non-racy algorithm from new LTO API to the legacy C API. I looked into new LTO API today and it looks that the data race is fixed there, though the code is more heavy C++ and will be harder to read. Alternatively, I could write a simpler patch for legacy C LTO API, doing what we discussed above (i.e. creating temp files in cache directory and if renaming of the temp into a cache file failed, bypass reading from the cache file).<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">What do you think?</span></div></div></div></div></div></div></blockquote><div><br class=""></div><div>I haven't read the C++ API implementation but I am all up to improve C LTO API to allow sharing. If nothing else, we can get faster ThinLTO build time for LLVM.</div><div><br class=""></div><div>Steven</div><br class=""><blockquote type="cite" class=""><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 14pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Steven<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class=""> </span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Thank you!</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 18pt; font-family: "Courier New";" class="">Katya.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></span></div></div></div></blockquote></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div></div></div></blockquote></div><br class=""></body></html>