[PATCH] D27295: Remove existing file in a separate thread asynchronously.

[Rui Ueyama via Phabricator via llvm-commits]

ruiu created this revision.
ruiu added a reviewer: rafael.
ruiu added a subscriber: llvm-commits.

On Linux (and probably on other Unix-like systems), unlink(2) is
noticeably slow. It takes 250 milliseconds to remove a 1 GB file
on ext4 filesystem on my machine, whether the file is on SSD or
on a spinning disk.

To create a new result file, we remove existing file first. So, if
you repeatedly link a 1 GB program in a regular compile-link-debug
cycle, every cycle wastes 250 milliseconds only to remove a file.

Since LLD can link a 1 GB in about 5 seconds, that actually counts.

This patch defines `unlinkAsync` function. The function spawns a
background thread to call unlink. The calling thread returns
almost immediately.


https://reviews.llvm.org/D27295

Files:
  ELF/Writer.cpp


Index: ELF/Writer.cpp
===================================================================
--- ELF/Writer.cpp
+++ ELF/Writer.cpp
@@ -21,8 +21,10 @@
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/Support/FileOutputBuffer.h"
+#include "llvm/Support/FileSystem.h"
 #include "llvm/Support/raw_ostream.h"
 #include <climits>
+#include <thread>
 
 using namespace llvm;
 using namespace llvm::ELF;
@@ -130,10 +132,45 @@
          !Script<ELFT>::X->ignoreInterpSection();
 }
 
+// Removes a given file asynchronously. This is a performance hack,
+// so remove this when operating systems are improved.
+//
+// On Linux (and probably on other Unix-like systems), unlink(2) is a
+// noticeably slow system call. As of 2016, unlink takes 250
+// milliseconds to remove a 1 GB file on ext4 filesystem on my machine.
+//
+// To create a new result file, we first remove existing file. So, if
+// you repeatedly link a 1 GB program in a regular compile-link-debug
+// cycle, every cycle wastes 250 milliseconds only to remove a file.
+// Since LLD can link a 1 GB in about 5 seconds, that actually counts.
+//
+// This function spawns a background thread to call unlink.
+// The calling thread returns almost immediately.
+static void unlinkAsync(StringRef Path) {
+  if (!sys::fs::exists(Path))
+    return;
+
+  // First, rename Path to avoid race condition. We cannot remomve
+  // Path from a different thread becase we are now going to create
+  // Path as a new file. If we do that in a different thread, the
+  // thread can remove the new file.
+  SmallString<128> TempPath;
+  if (auto EC = sys::fs::createTemporaryFile(Path, "temp", TempPath))
+    fatal(EC, "sys::fs::create failed");
+  if (auto EC = sys::fs::rename(Path, TempPath))
+    fatal(EC, "sys::fs::rename failed");
+
+  // Remove TempPath in background.
+  std::thread([=] { ::remove(TempPath.str().str().c_str()); }).detach();
+}
+
 template <class ELFT> void elf::writeResult() { Writer<ELFT>().run(); }
 
 // The main function of the writer.
 template <class ELFT> void Writer<ELFT>::run() {
+  // Remove a result file if it's already exists.
+  unlinkAsync(Config->OutputFile);
+
   // Create linker-synthesized sections such as .got or .plt.
   // Such sections are of type input section.
   createSyntheticSections();


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D27295.79868.patch
Type: text/x-patch
Size: 2321 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20161201/4fffd51a/attachment.bin>