[lld] f42f599 - [LLD][ELF][Windows] Allow LLD to overwrite existing output files that are in use

[Ben Dunbobbin via llvm-commits]

Author: Ben Dunbobbin
Date: 2020-07-03T13:08:56+01:00
New Revision: f42f599d3724cd4a2f470a53551604f722965de6

URL: https://github.com/llvm/llvm-project/commit/f42f599d3724cd4a2f470a53551604f722965de6
DIFF: https://github.com/llvm/llvm-project/commit/f42f599d3724cd4a2f470a53551604f722965de6.diff

LOG: [LLD][ELF][Windows] Allow LLD to overwrite existing output files that are in use

On Windows co-operative programs can be expected to open LLD's
output in FILE_SHARE_DELETE mode. This allows us to delete the
file (by moving it to a temporary filename and then deleting
it) so that we can link another output file that overwrites
the existing file, even if the current file is in use.

A similar strategy is documented here:
https://boostgsoc13.github.io/boost.afio/doc/html/afio/FAQ/deleting_open_files.html

Differential Revision: https://reviews.llvm.org/D82567

Added: 
    lld/test/ELF/link-open-file.test

Modified: 
    lld/Common/Filesystem.cpp

Removed: 
    


################################################################################
diff  --git a/lld/Common/Filesystem.cpp b/lld/Common/Filesystem.cpp
index 21b387ea26f7..206b892f0a69 100644
--- a/lld/Common/Filesystem.cpp
+++ b/lld/Common/Filesystem.cpp
@@ -15,6 +15,7 @@
 #include "llvm/Support/FileOutputBuffer.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Parallel.h"
+#include "llvm/Support/Path.h"
 #if LLVM_ON_UNIX
 #include <unistd.h>
 #endif
@@ -41,6 +42,33 @@ using namespace lld;
 void lld::unlinkAsync(StringRef path) {
 // Removing a file is async on windows.
 #if defined(_WIN32)
+  // On Windows co-operative programs can be expected to open LLD's
+  // output in FILE_SHARE_DELETE mode. This allows us to delete the
+  // file (by moving it to a temporary filename and then deleting
+  // it) so that we can link another output file that overwrites
+  // the existing file, even if the current file is in use.
+  //
+  // This is done on a best effort basis - we do not error if the
+  // operation fails. The consequence is merely that the user
+  // experiences an inconvenient work-flow.
+  //
+  // The code here allows LLD to work on all versions of Windows.
+  // However, at Windows 10 1903 it seems that the behavior of
+  // Windows has changed, so that we could simply delete the output 
+  // file. This code should be simplified once support for older
+  // versions of Windows is dropped.
+  //
+  // Warning: It seems that the WINVER and _WIN32_WINNT preprocessor
+  // defines affect the behavior of the Windows versions of the calls
+  // we are using here. If this code stops working this is worth
+  // bearing in mind.
+  SmallString<128> tmpName;
+  if (!sys::fs::createUniqueFile(path + "%%%%%%%%.tmp", tmpName)) {
+    if (!sys::fs::rename(path, tmpName))
+      path = tmpName;
+    else
+      sys::fs::remove(tmpName);
+  }
   sys::fs::remove(path);
 #else
   if (parallel::strategy.ThreadsRequested == 1 || !sys::fs::exists(path) ||

diff  --git a/lld/test/ELF/link-open-file.test b/lld/test/ELF/link-open-file.test
new file mode 100644
index 000000000000..f1239308324c
--- /dev/null
+++ b/lld/test/ELF/link-open-file.test
@@ -0,0 +1,71 @@
+## On Windows co-operative applications can be expected to open LLD's output
+## with FILE_SHARE_DELETE included in the sharing mode. This allows us to link
+## over the top of an existing file even if it is in use by another application.
+
+# REQUIRES: system-windows, x86
+# RUN: echo '.globl _start; _start:' > %t.s
+# RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-unknown %t.s -o %t.o
+
+## FILE_SHARE_READ   = 1
+## FILE_SHARE_WRITE  = 2
+## FILE_SHARE_DELETE = 4
+
+# RUN:     %python %s %t.o 7
+# RUN: not %python %s %t.o 3 2>&1 | FileCheck %s
+# CHECK: error: failed to write to the output file
+
+import contextlib
+import ctypes
+from ctypes import wintypes as w
+import os
+import shutil
+import subprocess
+import platform
+import sys
+import time
+
+object_file = sys.argv[1]
+share_flags = int(sys.argv[2])
+
+ at contextlib.contextmanager
+def open_with_share_flags(filename, share_flags):
+    GENERIC_READ          = 0x80000000
+    FILE_ATTRIBUTE_NORMAL = 0x80
+    OPEN_EXISTING         = 0x3
+    INVALID_HANDLE_VALUE = w.HANDLE(-1).value
+
+    CreateFileA = ctypes.windll.kernel32.CreateFileA
+    CreateFileA.restype = w.HANDLE
+    h = CreateFileA(filename.encode('mbcs'), GENERIC_READ, share_flags,
+                    None, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, None)
+
+    assert h != INVALID_HANDLE_VALUE, 'Failed to open ' + filename
+    try:
+        yield
+    finally:
+        ctypes.windll.kernel32.CloseHandle(h)
+
+## Ensure we have an empty directory for the output.
+outdir = os.path.basename(__file__) + '.dir'
+if os.path.exists(outdir):
+    shutil.rmtree(outdir)
+os.makedirs(outdir)
+
+## Link on top of an open file.
+elf = os.path.join(outdir, 'output_file.elf')
+open(elf, 'wb').close()
+with open_with_share_flags(elf, share_flags):
+    subprocess.check_call(['ld.lld.exe', object_file, '-o', elf])
+
+## Check the linker wrote the output file.
+with open(elf, 'rb') as f:
+    assert f.read(4) == b'\x7fELF', "linker did not write output file correctly"
+
+## Check no temp files are left around.
+## It might take a while for Windows to remove them, so loop.
+deleted = lambda: len(os.listdir(outdir)) == 1
+for _ in range(10):
+    if not deleted():
+        time.sleep (1)
+
+assert deleted(), "temp file(s) not deleted after grace period"