r365954 - Reland [clang] DirectoryWatcher

Mikael Holmén via cfe-commits cfe-commits at lists.llvm.org
Sun Jul 14 23:45:03 PDT 2019


Hi Jan,

 > +elseif(CMAKE_SYSTEM_NAME MATCHES "Linux")
 > +  check_include_files("sys/inotify.h" HAVE_INOTIFY)
 > +  if(HAVE_INOTIFY)
 > +    list(APPEND DIRECTORY_WATCHER_SOURCES 
linux/DirectoryWatcher-linux.cpp)
 > +    find_package(Threads REQUIRED)
 > +  endif()

I don't think the above is enough, I think the version needs to be 
checked some way as well to make sure we find an inotify.h that really 
defines IN_EXCL_UNLINK.

We have RHEL 6.10 buildbots with glibc 2.12.2 and kernel 2.6.32 where I see

-- Looking for include file sys/inotify.h
-- Looking for include file sys/inotify.h - found

but then get

/repo/uabelho/master/clang/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp:335:48: 
error: 'IN_EXCL_UNLINK' was not declared in this scope
        IN_CREATE | IN_DELETE | IN_DELETE_SELF | IN_EXCL_UNLINK | 
IN_MODIFY |
                                                 ^~~~~~~~~~~~~~

Regards,
Mikael

On 2019-07-12 22:34, Jan Korous via cfe-commits wrote:
> Author: jkorous
> Date: Fri Jul 12 13:34:10 2019
> New Revision: 365954
> 
> URL: http://llvm.org/viewvc/llvm-project?rev=365954&view=rev
> Log:
> Reland [clang] DirectoryWatcher
> 
> This reverts commit f561227d133224d2d6a5a016abe4be051fa75501.
> 
> - DirectoryWatcher
> - Fix the build for platforms that don't have DW implementated.
> - Fix the threading dependencies (thanks to compnerd).
> 
> Added:
>      cfe/trunk/include/clang/DirectoryWatcher/
>      cfe/trunk/include/clang/DirectoryWatcher/DirectoryWatcher.h
>      cfe/trunk/lib/DirectoryWatcher/
>      cfe/trunk/lib/DirectoryWatcher/CMakeLists.txt
>      cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.cpp
>      cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.h
>      cfe/trunk/lib/DirectoryWatcher/default/
>      cfe/trunk/lib/DirectoryWatcher/default/DirectoryWatcher-not-implemented.cpp
>      cfe/trunk/lib/DirectoryWatcher/linux/
>      cfe/trunk/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp
>      cfe/trunk/lib/DirectoryWatcher/mac/
>      cfe/trunk/lib/DirectoryWatcher/mac/DirectoryWatcher-mac.cpp
>      cfe/trunk/unittests/DirectoryWatcher/
>      cfe/trunk/unittests/DirectoryWatcher/CMakeLists.txt
>      cfe/trunk/unittests/DirectoryWatcher/DirectoryWatcherTest.cpp
> Modified:
>      cfe/trunk/lib/CMakeLists.txt
>      cfe/trunk/unittests/CMakeLists.txt
> 
> Added: cfe/trunk/include/clang/DirectoryWatcher/DirectoryWatcher.h
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/DirectoryWatcher/DirectoryWatcher.h?rev=365954&view=auto
> ==============================================================================
> --- cfe/trunk/include/clang/DirectoryWatcher/DirectoryWatcher.h (added)
> +++ cfe/trunk/include/clang/DirectoryWatcher/DirectoryWatcher.h Fri Jul 12 13:34:10 2019
> @@ -0,0 +1,123 @@
> +//===- DirectoryWatcher.h - Listens for directory file changes --*- C++ -*-===//
> +//
> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
> +// See https://llvm.org/LICENSE.txt for license information.
> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
> +//
> +//===----------------------------------------------------------------------===//
> +
> +#ifndef LLVM_CLANG_DIRECTORYWATCHER_DIRECTORYWATCHER_H
> +#define LLVM_CLANG_DIRECTORYWATCHER_DIRECTORYWATCHER_H
> +
> +#include "llvm/ADT/ArrayRef.h"
> +#include "llvm/ADT/StringRef.h"
> +#include <functional>
> +#include <memory>
> +#include <string>
> +
> +namespace clang {
> +/// Provides notifications for file changes in a directory.
> +///
> +/// Invokes client-provided function on every filesystem event in the watched
> +/// directory. Initially the the watched directory is scanned and for every file
> +/// found, an event is synthesized as if the file was added.
> +///
> +/// This is not a general purpose directory monitoring tool - list of
> +/// limitations follows.
> +///
> +/// Only flat directories with no subdirectories are supported. In case
> +/// subdirectories are present the behavior is unspecified - events *might* be
> +/// passed to Receiver on macOS (due to FSEvents being used) while they
> +/// *probably* won't be passed on Linux (due to inotify being used).
> +///
> +/// Known potential inconsistencies
> +/// - For files that are deleted befor the initial scan processed them, clients
> +/// might receive Removed notification without any prior Added notification.
> +/// - Multiple notifications might be produced when a file is added to the
> +/// watched directory during the initial scan. We are choosing the lesser evil
> +/// here as the only known alternative strategy would be to invalidate the
> +/// watcher instance and force user to create a new one whenever filesystem
> +/// event occurs during the initial scan but that would introduce continuous
> +/// restarting failure mode (watched directory is not always "owned" by the same
> +/// process that is consuming it). Since existing clients can handle duplicate
> +/// events well, we decided for simplicity.
> +///
> +/// Notifications are provided only for changes done through local user-space
> +/// filesystem interface. Specifically, it's unspecified if notification would
> +/// be provided in case of a:
> +/// - a file mmap-ed and changed
> +/// - a file changed via remote (NFS) or virtual (/proc) FS access to monitored
> +/// directory
> +/// - another filesystem mounted to the watched directory
> +///
> +/// No support for LLVM VFS.
> +///
> +/// It is unspecified whether notifications for files being deleted are sent in
> +/// case the whole watched directory is sent.
> +///
> +/// Directories containing "too many" files and/or receiving events "too
> +/// frequently" are not supported - if the initial scan can't be finished before
> +/// the watcher instance gets invalidated (see WatcherGotInvalidated) there's no
> +/// good error handling strategy - the only option for client is to destroy the
> +/// watcher, restart watching with new instance and hope it won't repeat.
> +class DirectoryWatcher {
> +public:
> +  struct Event {
> +    enum class EventKind {
> +      Removed,
> +      /// Content of a file was modified.
> +      Modified,
> +      /// The watched directory got deleted.
> +      WatchedDirRemoved,
> +      /// The DirectoryWatcher that originated this event is no longer valid and
> +      /// its behavior is unspecified.
> +      ///
> +      /// The prime case is kernel signalling to OS-specific implementation of
> +      /// DirectoryWatcher some resource limit being hit.
> +      /// *Usually* kernel starts dropping or squashing events together after
> +      /// that and so would DirectoryWatcher. This means that *some* events
> +      /// might still be passed to Receiver but this behavior is unspecified.
> +      ///
> +      /// Another case is after the watched directory itself is deleted.
> +      /// WatcherGotInvalidated will be received at least once during
> +      /// DirectoryWatcher instance lifetime - when handling errors this is done
> +      /// on best effort basis, when an instance is being destroyed then this is
> +      /// guaranteed.
> +      ///
> +      /// The only proper response to this kind of event is to destruct the
> +      /// originating DirectoryWatcher instance and create a new one.
> +      WatcherGotInvalidated
> +    };
> +
> +    EventKind Kind;
> +    /// Filename that this event is related to or an empty string in
> +    /// case this event is related to the watched directory itself.
> +    std::string Filename;
> +
> +    Event(EventKind Kind, llvm::StringRef Filename)
> +        : Kind(Kind), Filename(Filename) {}
> +  };
> +
> +  /// Returns nullptr if \param Path doesn't exist.
> +  /// Returns nullptr if \param Path isn't a directory.
> +  /// Returns nullptr if OS kernel API told us we can't start watching. In such
> +  /// case it's unclear whether just retrying has any chance to succeeed.
> +  static std::unique_ptr<DirectoryWatcher>
> +  create(llvm::StringRef Path,
> +         std::function<void(llvm::ArrayRef<DirectoryWatcher::Event> Events,
> +                            bool IsInitial)>
> +             Receiver,
> +         bool WaitForInitialSync);
> +
> +  virtual ~DirectoryWatcher() = default;
> +  DirectoryWatcher(const DirectoryWatcher &) = delete;
> +  DirectoryWatcher &operator=(const DirectoryWatcher &) = delete;
> +  DirectoryWatcher(DirectoryWatcher &&) = default;
> +
> +protected:
> +  DirectoryWatcher() = default;
> +};
> +
> +} // namespace clang
> +
> +#endif // LLVM_CLANG_DIRECTORYWATCHER_DIRECTORYWATCHER_H
> 
> Modified: cfe/trunk/lib/CMakeLists.txt
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/CMakeLists.txt?rev=365954&r1=365953&r2=365954&view=diff
> ==============================================================================
> --- cfe/trunk/lib/CMakeLists.txt (original)
> +++ cfe/trunk/lib/CMakeLists.txt Fri Jul 12 13:34:10 2019
> @@ -18,6 +18,7 @@ add_subdirectory(Serialization)
>   add_subdirectory(Frontend)
>   add_subdirectory(FrontendTool)
>   add_subdirectory(Tooling)
> +add_subdirectory(DirectoryWatcher)
>   add_subdirectory(Index)
>   if(CLANG_ENABLE_STATIC_ANALYZER)
>     add_subdirectory(StaticAnalyzer)
> 
> Added: cfe/trunk/lib/DirectoryWatcher/CMakeLists.txt
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/CMakeLists.txt?rev=365954&view=auto
> ==============================================================================
> --- cfe/trunk/lib/DirectoryWatcher/CMakeLists.txt (added)
> +++ cfe/trunk/lib/DirectoryWatcher/CMakeLists.txt Fri Jul 12 13:34:10 2019
> @@ -0,0 +1,29 @@
> +include(CheckIncludeFiles)
> +
> +set(LLVM_LINK_COMPONENTS support)
> +
> +set(DIRECTORY_WATCHER_SOURCES DirectoryScanner.cpp)
> +set(DIRECTORY_WATCHER_LINK_LIBS "")
> +
> +if(APPLE)
> +  check_include_files("CoreServices/CoreServices.h" HAVE_CORESERVICES)
> +  if(HAVE_CORESERVICES)
> +    list(APPEND DIRECTORY_WATCHER_SOURCES mac/DirectoryWatcher-mac.cpp)
> +    set(DIRECTORY_WATCHER_LINK_LIBS "-framework CoreServices")
> +  endif()
> +elseif(CMAKE_SYSTEM_NAME MATCHES "Linux")
> +  check_include_files("sys/inotify.h" HAVE_INOTIFY)
> +  if(HAVE_INOTIFY)
> +    list(APPEND DIRECTORY_WATCHER_SOURCES linux/DirectoryWatcher-linux.cpp)
> +    find_package(Threads REQUIRED)
> +  endif()
> +else()
> +  list(APPEND DIRECTORY_WATCHER_SOURCES default/DirectoryWatcher-not-implemented.cpp)
> +endif()
> +
> +add_clang_library(clangDirectoryWatcher
> +  ${DIRECTORY_WATCHER_SOURCES}
> +  )
> +
> +target_link_libraries(clangDirectoryWatcher PUBLIC ${CMAKE_THREAD_LIBS_INIT})
> +target_link_libraries(clangDirectoryWatcher PRIVATE ${DIRECTORY_WATCHER_LINK_LIBS})
> 
> Added: cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.cpp?rev=365954&view=auto
> ==============================================================================
> --- cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.cpp (added)
> +++ cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.cpp Fri Jul 12 13:34:10 2019
> @@ -0,0 +1,54 @@
> +//===- DirectoryScanner.cpp - Utility functions for DirectoryWatcher ------===//
> +//
> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
> +// See https://llvm.org/LICENSE.txt for license information.
> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
> +//
> +//===----------------------------------------------------------------------===//
> +
> +#include "DirectoryScanner.h"
> +
> +#include "llvm/Support/Path.h"
> +
> +namespace clang {
> +
> +using namespace llvm;
> +
> +Optional<sys::fs::file_status> getFileStatus(StringRef Path) {
> +  sys::fs::file_status Status;
> +  std::error_code EC = status(Path, Status);
> +  if (EC)
> +    return None;
> +  return Status;
> +}
> +
> +std::vector<std::string> scanDirectory(StringRef Path) {
> +  using namespace llvm::sys;
> +  std::vector<std::string> Result;
> +
> +  std::error_code EC;
> +  for (auto It = fs::directory_iterator(Path, EC),
> +            End = fs::directory_iterator();
> +       !EC && It != End; It.increment(EC)) {
> +    auto status = getFileStatus(It->path());
> +    if (!status.hasValue())
> +      continue;
> +    Result.emplace_back(sys::path::filename(It->path()));
> +  }
> +
> +  return Result;
> +}
> +
> +std::vector<DirectoryWatcher::Event>
> +getAsFileEvents(const std::vector<std::string> &Scan) {
> +  std::vector<DirectoryWatcher::Event> Events;
> +  Events.reserve(Scan.size());
> +
> +  for (const auto &File : Scan) {
> +    Events.emplace_back(DirectoryWatcher::Event{
> +        DirectoryWatcher::Event::EventKind::Modified, File});
> +  }
> +  return Events;
> +}
> +
> +} // namespace clang
> \ No newline at end of file
> 
> Added: cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.h
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.h?rev=365954&view=auto
> ==============================================================================
> --- cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.h (added)
> +++ cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.h Fri Jul 12 13:34:10 2019
> @@ -0,0 +1,29 @@
> +//===- DirectoryScanner.h - Utility functions for DirectoryWatcher --------===//
> +//
> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
> +// See https://llvm.org/LICENSE.txt for license information.
> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
> +//
> +//===----------------------------------------------------------------------===//
> +
> +#include "clang/DirectoryWatcher/DirectoryWatcher.h"
> +#include "llvm/Support/FileSystem.h"
> +#include <string>
> +#include <vector>
> +
> +namespace clang {
> +
> +/// Gets names (filenames) of items in directory at \p Path.
> +/// \returns empty vector if \p Path is not a directory, doesn't exist or can't
> +/// be read from.
> +std::vector<std::string> scanDirectory(llvm::StringRef Path);
> +
> +/// Create event with EventKind::Added for every element in \p Scan.
> +std::vector<DirectoryWatcher::Event>
> +getAsFileEvents(const std::vector<std::string> &Scan);
> +
> +/// Gets status of file (or directory) at \p Path.
> +/// \returns llvm::None if \p Path doesn't exist or can't get the status.
> +llvm::Optional<llvm::sys::fs::file_status> getFileStatus(llvm::StringRef Path);
> +
> +} // namespace clang
> \ No newline at end of file
> 
> Added: cfe/trunk/lib/DirectoryWatcher/default/DirectoryWatcher-not-implemented.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/default/DirectoryWatcher-not-implemented.cpp?rev=365954&view=auto
> ==============================================================================
> --- cfe/trunk/lib/DirectoryWatcher/default/DirectoryWatcher-not-implemented.cpp (added)
> +++ cfe/trunk/lib/DirectoryWatcher/default/DirectoryWatcher-not-implemented.cpp Fri Jul 12 13:34:10 2019
> @@ -0,0 +1,19 @@
> +//===- DirectoryWatcher-not-implemented.cpp -------------------------------===//
> +//
> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
> +// See https://llvm.org/LICENSE.txt for license information.
> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
> +//
> +//===----------------------------------------------------------------------===//
> +
> +#include "clang/DirectoryWatcher/DirectoryWatcher.h"
> +
> +using namespace llvm;
> +using namespace clang;
> +
> +std::unique_ptr<DirectoryWatcher> clang::DirectoryWatcher::create(
> +    StringRef Path,
> +    std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)> Receiver,
> +    bool WaitForInitialSync) {
> +  return nullptr;
> +}
> \ No newline at end of file
> 
> Added: cfe/trunk/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp?rev=365954&view=auto
> ==============================================================================
> --- cfe/trunk/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp (added)
> +++ cfe/trunk/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp Fri Jul 12 13:34:10 2019
> @@ -0,0 +1,345 @@
> +//===- DirectoryWatcher-linux.cpp - Linux-platform directory watching -----===//
> +//
> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
> +// See https://llvm.org/LICENSE.txt for license information.
> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
> +//
> +//===----------------------------------------------------------------------===//
> +
> +#include "DirectoryScanner.h"
> +#include "clang/DirectoryWatcher/DirectoryWatcher.h"
> +
> +#include "llvm/ADT/STLExtras.h"
> +#include "llvm/ADT/ScopeExit.h"
> +#include "llvm/Support/AlignOf.h"
> +#include "llvm/Support/Errno.h"
> +#include "llvm/Support/Mutex.h"
> +#include "llvm/Support/Path.h"
> +#include <atomic>
> +#include <condition_variable>
> +#include <mutex>
> +#include <queue>
> +#include <string>
> +#include <thread>
> +#include <vector>
> +
> +#include <fcntl.h>
> +#include <sys/epoll.h>
> +#include <sys/inotify.h>
> +#include <unistd.h>
> +
> +namespace {
> +
> +using namespace llvm;
> +using namespace clang;
> +
> +/// Pipe for inter-thread synchronization - for epoll-ing on multiple
> +/// conditions. It is meant for uni-directional 1:1 signalling - specifically:
> +/// no multiple consumers, no data passing. Thread waiting for signal should
> +/// poll the FDRead. Signalling thread should call signal() which writes single
> +/// character to FDRead.
> +struct SemaphorePipe {
> +  // Expects two file-descriptors opened as a pipe in the canonical POSIX
> +  // order: pipefd[0] refers to the read end of the pipe. pipefd[1] refers to
> +  // the write end of the pipe.
> +  SemaphorePipe(int pipefd[2])
> +      : FDRead(pipefd[0]), FDWrite(pipefd[1]), OwnsFDs(true) {}
> +  SemaphorePipe(const SemaphorePipe &) = delete;
> +  void operator=(const SemaphorePipe &) = delete;
> +  SemaphorePipe(SemaphorePipe &&other)
> +      : FDRead(other.FDRead), FDWrite(other.FDWrite),
> +        OwnsFDs(other.OwnsFDs) // Someone could have moved from the other
> +                               // instance before.
> +  {
> +    other.OwnsFDs = false;
> +  };
> +
> +  void signal() {
> +    ssize_t Result = llvm::sys::RetryAfterSignal(-1, write, FDWrite, "A", 1);
> +    assert(Result != -1);
> +  }
> +  ~SemaphorePipe() {
> +    if (OwnsFDs) {
> +      close(FDWrite);
> +      close(FDRead);
> +    }
> +  }
> +  const int FDRead;
> +  const int FDWrite;
> +  bool OwnsFDs;
> +
> +  static llvm::Optional<SemaphorePipe> create() {
> +    int InotifyPollingStopperFDs[2];
> +    if (pipe2(InotifyPollingStopperFDs, O_CLOEXEC) == -1)
> +      return llvm::None;
> +    return SemaphorePipe(InotifyPollingStopperFDs);
> +  }
> +};
> +
> +/// Mutex-protected queue of Events.
> +class EventQueue {
> +  std::mutex Mtx;
> +  std::condition_variable NonEmpty;
> +  std::queue<DirectoryWatcher::Event> Events;
> +
> +public:
> +  void push_back(const DirectoryWatcher::Event::EventKind K,
> +                 StringRef Filename) {
> +    {
> +      std::unique_lock<std::mutex> L(Mtx);
> +      Events.emplace(K, Filename);
> +    }
> +    NonEmpty.notify_one();
> +  }
> +
> +  // Blocks on caller thread and uses codition_variable to wait until there's an
> +  // event to return.
> +  DirectoryWatcher::Event pop_front_blocking() {
> +    std::unique_lock<std::mutex> L(Mtx);
> +    while (true) {
> +      // Since we might have missed all the prior notifications on NonEmpty we
> +      // have to check the queue first (under lock).
> +      if (!Events.empty()) {
> +        DirectoryWatcher::Event Front = Events.front();
> +        Events.pop();
> +        return Front;
> +      }
> +      NonEmpty.wait(L, [this]() { return !Events.empty(); });
> +    }
> +  }
> +};
> +
> +class DirectoryWatcherLinux : public clang::DirectoryWatcher {
> +public:
> +  DirectoryWatcherLinux(
> +      llvm::StringRef WatchedDirPath,
> +      std::function<void(llvm::ArrayRef<Event>, bool)> Receiver,
> +      bool WaitForInitialSync, int InotifyFD, int InotifyWD,
> +      SemaphorePipe &&InotifyPollingStopSignal);
> +
> +  ~DirectoryWatcherLinux() override {
> +    StopWork();
> +    InotifyPollingThread.join();
> +    EventsReceivingThread.join();
> +    inotify_rm_watch(InotifyFD, InotifyWD);
> +    llvm::sys::RetryAfterSignal(-1, close, InotifyFD);
> +  }
> +
> +private:
> +  const std::string WatchedDirPath;
> +  // inotify file descriptor
> +  int InotifyFD = -1;
> +  // inotify watch descriptor
> +  int InotifyWD = -1;
> +
> +  EventQueue Queue;
> +
> +  // Make sure lifetime of Receiver fully contains lifetime of
> +  // EventsReceivingThread.
> +  std::function<void(llvm::ArrayRef<Event>, bool)> Receiver;
> +
> +  // Consumes inotify events and pushes directory watcher events to the Queue.
> +  void InotifyPollingLoop();
> +  std::thread InotifyPollingThread;
> +  // Using pipe so we can epoll two file descriptors at once - inotify and
> +  // stopping condition.
> +  SemaphorePipe InotifyPollingStopSignal;
> +
> +  // Does the initial scan of the directory - directly calling Receiver,
> +  // bypassing the Queue. Both InitialScan and EventReceivingLoop use Receiver
> +  // which isn't necessarily thread-safe.
> +  void InitialScan();
> +
> +  // Processing events from the Queue.
> +  // In case client doesn't want to do the initial scan synchronously
> +  // (WaitForInitialSync=false in ctor) we do the initial scan at the beginning
> +  // of this thread.
> +  std::thread EventsReceivingThread;
> +  // Push event of WatcherGotInvalidated kind to the Queue to stop the loop.
> +  // Both InitialScan and EventReceivingLoop use Receiver which isn't
> +  // necessarily thread-safe.
> +  void EventReceivingLoop();
> +
> +  // Stops all the async work. Reentrant.
> +  void StopWork() {
> +    Queue.push_back(DirectoryWatcher::Event::EventKind::WatcherGotInvalidated,
> +                    "");
> +    InotifyPollingStopSignal.signal();
> +  }
> +};
> +
> +void DirectoryWatcherLinux::InotifyPollingLoop() {
> +  // We want to be able to read ~30 events at once even in the worst case
> +  // (obscenely long filenames).
> +  constexpr size_t EventBufferLength =
> +      30 * (sizeof(struct inotify_event) + NAME_MAX + 1);
> +  // http://man7.org/linux/man-pages/man7/inotify.7.html
> +  // Some systems cannot read integer variables if they are not
> +  // properly aligned. On other systems, incorrect alignment may
> +  // decrease performance. Hence, the buffer used for reading from
> +  // the inotify file descriptor should have the same alignment as
> +  // struct inotify_event.
> +
> +  auto ManagedBuffer =
> +      llvm::make_unique<llvm::AlignedCharArray<alignof(struct inotify_event),
> +                                               EventBufferLength>>();
> +  char *const Buf = ManagedBuffer->buffer;
> +
> +  const int EpollFD = epoll_create1(EPOLL_CLOEXEC);
> +  if (EpollFD == -1) {
> +    StopWork();
> +    return;
> +  }
> +  auto EpollFDGuard = llvm::make_scope_exit([EpollFD]() { close(EpollFD); });
> +
> +  struct epoll_event EventSpec;
> +  EventSpec.events = EPOLLIN;
> +  EventSpec.data.fd = InotifyFD;
> +  if (epoll_ctl(EpollFD, EPOLL_CTL_ADD, InotifyFD, &EventSpec) == -1) {
> +    StopWork();
> +    return;
> +  }
> +
> +  EventSpec.data.fd = InotifyPollingStopSignal.FDRead;
> +  if (epoll_ctl(EpollFD, EPOLL_CTL_ADD, InotifyPollingStopSignal.FDRead,
> +                &EventSpec) == -1) {
> +    StopWork();
> +    return;
> +  }
> +
> +  std::array<struct epoll_event, 2> EpollEventBuffer;
> +
> +  while (true) {
> +    const int EpollWaitResult = llvm::sys::RetryAfterSignal(
> +        -1, epoll_wait, EpollFD, EpollEventBuffer.data(),
> +        EpollEventBuffer.size(), /*timeout=*/-1 /*== infinity*/);
> +    if (EpollWaitResult == -1) {
> +      StopWork();
> +      return;
> +    }
> +
> +    // Multiple epoll_events can be received for a single file descriptor per
> +    // epoll_wait call.
> +    for (const auto &EpollEvent : EpollEventBuffer) {
> +      if (EpollEvent.data.fd == InotifyPollingStopSignal.FDRead) {
> +        StopWork();
> +        return;
> +      }
> +    }
> +
> +    // epoll_wait() always return either error or >0 events. Since there was no
> +    // event for stopping, it must be an inotify event ready for reading.
> +    ssize_t NumRead = llvm::sys::RetryAfterSignal(-1, read, InotifyFD, Buf,
> +                                                  EventBufferLength);
> +    for (char *P = Buf; P < Buf + NumRead;) {
> +      if (P + sizeof(struct inotify_event) > Buf + NumRead) {
> +        StopWork();
> +        llvm_unreachable("an incomplete inotify_event was read");
> +        return;
> +      }
> +
> +      struct inotify_event *Event = reinterpret_cast<struct inotify_event *>(P);
> +      P += sizeof(struct inotify_event) + Event->len;
> +
> +      if (Event->mask & (IN_CREATE | IN_MODIFY | IN_MOVED_TO | IN_DELETE) &&
> +          Event->len <= 0) {
> +        StopWork();
> +        llvm_unreachable("expected a filename from inotify");
> +        return;
> +      }
> +
> +      if (Event->mask & (IN_CREATE | IN_MOVED_TO | IN_MODIFY)) {
> +        Queue.push_back(DirectoryWatcher::Event::EventKind::Modified,
> +                        Event->name);
> +      } else if (Event->mask & (IN_DELETE | IN_MOVED_FROM)) {
> +        Queue.push_back(DirectoryWatcher::Event::EventKind::Removed,
> +                        Event->name);
> +      } else if (Event->mask & (IN_DELETE_SELF | IN_MOVE_SELF)) {
> +        Queue.push_back(DirectoryWatcher::Event::EventKind::WatchedDirRemoved,
> +                        "");
> +        StopWork();
> +        return;
> +      } else if (Event->mask & IN_IGNORED) {
> +        StopWork();
> +        return;
> +      } else {
> +        StopWork();
> +        llvm_unreachable("Unknown event type.");
> +        return;
> +      }
> +    }
> +  }
> +}
> +
> +void DirectoryWatcherLinux::InitialScan() {
> +  this->Receiver(getAsFileEvents(scanDirectory(WatchedDirPath)),
> +                 /*IsInitial=*/true);
> +}
> +
> +void DirectoryWatcherLinux::EventReceivingLoop() {
> +  while (true) {
> +    DirectoryWatcher::Event Event = this->Queue.pop_front_blocking();
> +    this->Receiver(Event, false);
> +    if (Event.Kind ==
> +        DirectoryWatcher::Event::EventKind::WatcherGotInvalidated) {
> +      StopWork();
> +      return;
> +    }
> +  }
> +}
> +
> +DirectoryWatcherLinux::DirectoryWatcherLinux(
> +    StringRef WatchedDirPath,
> +    std::function<void(llvm::ArrayRef<Event>, bool)> Receiver,
> +    bool WaitForInitialSync, int InotifyFD, int InotifyWD,
> +    SemaphorePipe &&InotifyPollingStopSignal)
> +    : WatchedDirPath(WatchedDirPath), InotifyFD(InotifyFD),
> +      InotifyWD(InotifyWD), Receiver(Receiver),
> +      InotifyPollingStopSignal(std::move(InotifyPollingStopSignal)) {
> +
> +  InotifyPollingThread = std::thread([this]() { InotifyPollingLoop(); });
> +  // We have no guarantees about thread safety of the Receiver which is being
> +  // used in both InitialScan and EventReceivingLoop. We shouldn't run these
> +  // only synchronously.
> +  if (WaitForInitialSync) {
> +    InitialScan();
> +    EventsReceivingThread = std::thread([this]() { EventReceivingLoop(); });
> +  } else {
> +    EventsReceivingThread = std::thread([this]() {
> +      // FIXME: We might want to terminate an async initial scan early in case
> +      // of a failure in EventsReceivingThread.
> +      InitialScan();
> +      EventReceivingLoop();
> +    });
> +  }
> +}
> +
> +} // namespace
> +
> +std::unique_ptr<DirectoryWatcher> clang::DirectoryWatcher::create(
> +    StringRef Path,
> +    std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)> Receiver,
> +    bool WaitForInitialSync) {
> +  if (Path.empty())
> +    return nullptr;
> +
> +  const int InotifyFD = inotify_init1(IN_CLOEXEC);
> +  if (InotifyFD == -1)
> +    return nullptr;
> +
> +  const int InotifyWD = inotify_add_watch(
> +      InotifyFD, Path.str().c_str(),
> +      IN_CREATE | IN_DELETE | IN_DELETE_SELF | IN_EXCL_UNLINK | IN_MODIFY |
> +          IN_MOVED_FROM | IN_MOVE_SELF | IN_MOVED_TO | IN_ONLYDIR | IN_IGNORED);
> +  if (InotifyWD == -1)
> +    return nullptr;
> +
> +  auto InotifyPollingStopper = SemaphorePipe::create();
> +
> +  if (!InotifyPollingStopper)
> +    return nullptr;
> +
> +  return llvm::make_unique<DirectoryWatcherLinux>(
> +      Path, Receiver, WaitForInitialSync, InotifyFD, InotifyWD,
> +      std::move(*InotifyPollingStopper));
> +}
> \ No newline at end of file
> 
> Added: cfe/trunk/lib/DirectoryWatcher/mac/DirectoryWatcher-mac.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/mac/DirectoryWatcher-mac.cpp?rev=365954&view=auto
> ==============================================================================
> --- cfe/trunk/lib/DirectoryWatcher/mac/DirectoryWatcher-mac.cpp (added)
> +++ cfe/trunk/lib/DirectoryWatcher/mac/DirectoryWatcher-mac.cpp Fri Jul 12 13:34:10 2019
> @@ -0,0 +1,233 @@
> +//===- DirectoryWatcher-mac.cpp - Mac-platform directory watching ---------===//
> +//
> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
> +// See https://llvm.org/LICENSE.txt for license information.
> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
> +//
> +//===----------------------------------------------------------------------===//
> +
> +#include "DirectoryScanner.h"
> +#include "clang/DirectoryWatcher/DirectoryWatcher.h"
> +
> +#include "llvm/ADT/STLExtras.h"
> +#include "llvm/ADT/StringRef.h"
> +#include "llvm/Support/Path.h"
> +#include <CoreServices/CoreServices.h>
> +
> +using namespace llvm;
> +using namespace clang;
> +
> +static FSEventStreamRef createFSEventStream(
> +    StringRef Path,
> +    std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)>,
> +    dispatch_queue_t);
> +static void stopFSEventStream(FSEventStreamRef);
> +
> +namespace {
> +
> +class DirectoryWatcherMac : public clang::DirectoryWatcher {
> +public:
> +  DirectoryWatcherMac(
> +      FSEventStreamRef EventStream,
> +      std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)>
> +          Receiver,
> +      llvm::StringRef WatchedDirPath)
> +      : EventStream(EventStream), Receiver(Receiver),
> +        WatchedDirPath(WatchedDirPath) {}
> +
> +  ~DirectoryWatcherMac() override {
> +    stopFSEventStream(EventStream);
> +    EventStream = nullptr;
> +    // Now it's safe to use Receiver as the only other concurrent use would have
> +    // been in EventStream processing.
> +    Receiver(DirectoryWatcher::Event(
> +                 DirectoryWatcher::Event::EventKind::WatcherGotInvalidated, ""),
> +             false);
> +  }
> +
> +private:
> +  FSEventStreamRef EventStream;
> +  std::function<void(llvm::ArrayRef<Event>, bool)> Receiver;
> +  const std::string WatchedDirPath;
> +};
> +
> +struct EventStreamContextData {
> +  std::string WatchedPath;
> +  std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)> Receiver;
> +
> +  EventStreamContextData(
> +      std::string &&WatchedPath,
> +      std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)>
> +          Receiver)
> +      : WatchedPath(std::move(WatchedPath)), Receiver(Receiver) {}
> +
> +  // Needed for FSEvents
> +  static void dispose(const void *ctx) {
> +    delete static_cast<const EventStreamContextData *>(ctx);
> +  }
> +};
> +} // namespace
> +
> +constexpr const FSEventStreamEventFlags StreamInvalidatingFlags =
> +    kFSEventStreamEventFlagUserDropped | kFSEventStreamEventFlagKernelDropped |
> +    kFSEventStreamEventFlagMustScanSubDirs;
> +
> +constexpr const FSEventStreamEventFlags ModifyingFileEvents =
> +    kFSEventStreamEventFlagItemCreated | kFSEventStreamEventFlagItemRenamed |
> +    kFSEventStreamEventFlagItemModified;
> +
> +static void eventStreamCallback(ConstFSEventStreamRef Stream,
> +                                void *ClientCallBackInfo, size_t NumEvents,
> +                                void *EventPaths,
> +                                const FSEventStreamEventFlags EventFlags[],
> +                                const FSEventStreamEventId EventIds[]) {
> +  auto *ctx = static_cast<EventStreamContextData *>(ClientCallBackInfo);
> +
> +  std::vector<DirectoryWatcher::Event> Events;
> +  for (size_t i = 0; i < NumEvents; ++i) {
> +    StringRef Path = ((const char **)EventPaths)[i];
> +    const FSEventStreamEventFlags Flags = EventFlags[i];
> +
> +    if (Flags & StreamInvalidatingFlags) {
> +      Events.emplace_back(DirectoryWatcher::Event{
> +          DirectoryWatcher::Event::EventKind::WatcherGotInvalidated, ""});
> +      break;
> +    } else if (!(Flags & kFSEventStreamEventFlagItemIsFile)) {
> +      // Subdirectories aren't supported - if some directory got removed it
> +      // must've been the watched directory itself.
> +      if ((Flags & kFSEventStreamEventFlagItemRemoved) &&
> +          Path == ctx->WatchedPath) {
> +        Events.emplace_back(DirectoryWatcher::Event{
> +            DirectoryWatcher::Event::EventKind::WatchedDirRemoved, ""});
> +        Events.emplace_back(DirectoryWatcher::Event{
> +            DirectoryWatcher::Event::EventKind::WatcherGotInvalidated, ""});
> +        break;
> +      }
> +      // No support for subdirectories - just ignore everything.
> +      continue;
> +    } else if (Flags & kFSEventStreamEventFlagItemRemoved) {
> +      Events.emplace_back(DirectoryWatcher::Event::EventKind::Removed,
> +                          llvm::sys::path::filename(Path));
> +      continue;
> +    } else if (Flags & ModifyingFileEvents) {
> +      if (!getFileStatus(Path).hasValue()) {
> +        Events.emplace_back(DirectoryWatcher::Event::EventKind::Removed,
> +                            llvm::sys::path::filename(Path));
> +      } else {
> +        Events.emplace_back(DirectoryWatcher::Event::EventKind::Modified,
> +                            llvm::sys::path::filename(Path));
> +      }
> +      continue;
> +    }
> +
> +    // default
> +    Events.emplace_back(DirectoryWatcher::Event{
> +        DirectoryWatcher::Event::EventKind::WatcherGotInvalidated, ""});
> +    llvm_unreachable("Unknown FSEvent type.");
> +  }
> +
> +  if (!Events.empty()) {
> +    ctx->Receiver(Events, /*IsInitial=*/false);
> +  }
> +}
> +
> +FSEventStreamRef createFSEventStream(
> +    StringRef Path,
> +    std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)> Receiver,
> +    dispatch_queue_t Queue) {
> +  if (Path.empty())
> +    return nullptr;
> +
> +  CFMutableArrayRef PathsToWatch = [&]() {
> +    CFMutableArrayRef PathsToWatch =
> +        CFArrayCreateMutable(nullptr, 0, &kCFTypeArrayCallBacks);
> +    CFStringRef CfPathStr =
> +        CFStringCreateWithBytes(nullptr, (const UInt8 *)Path.data(),
> +                                Path.size(), kCFStringEncodingUTF8, false);
> +    CFArrayAppendValue(PathsToWatch, CfPathStr);
> +    CFRelease(CfPathStr);
> +    return PathsToWatch;
> +  }();
> +
> +  FSEventStreamContext Context = [&]() {
> +    std::string RealPath;
> +    {
> +      SmallString<128> Storage;
> +      StringRef P = llvm::Twine(Path).toNullTerminatedStringRef(Storage);
> +      char Buffer[PATH_MAX];
> +      if (::realpath(P.begin(), Buffer) != nullptr)
> +        RealPath = Buffer;
> +      else
> +        RealPath = Path;
> +    }
> +
> +    FSEventStreamContext Context;
> +    Context.version = 0;
> +    Context.info = new EventStreamContextData(std::move(RealPath), Receiver);
> +    Context.retain = nullptr;
> +    Context.release = EventStreamContextData::dispose;
> +    Context.copyDescription = nullptr;
> +    return Context;
> +  }();
> +
> +  FSEventStreamRef Result = FSEventStreamCreate(
> +      nullptr, eventStreamCallback, &Context, PathsToWatch,
> +      kFSEventStreamEventIdSinceNow, /* latency in seconds */ 0.0,
> +      kFSEventStreamCreateFlagFileEvents | kFSEventStreamCreateFlagNoDefer);
> +  CFRelease(PathsToWatch);
> +
> +  return Result;
> +}
> +
> +void stopFSEventStream(FSEventStreamRef EventStream) {
> +  if (!EventStream)
> +    return;
> +  FSEventStreamStop(EventStream);
> +  FSEventStreamInvalidate(EventStream);
> +  FSEventStreamRelease(EventStream);
> +}
> +
> +std::unique_ptr<DirectoryWatcher> clang::DirectoryWatcher::create(
> +    StringRef Path,
> +    std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)> Receiver,
> +    bool WaitForInitialSync) {
> +  dispatch_queue_t Queue =
> +      dispatch_queue_create("DirectoryWatcher", DISPATCH_QUEUE_SERIAL);
> +
> +  if (Path.empty())
> +    return nullptr;
> +
> +  auto EventStream = createFSEventStream(Path, Receiver, Queue);
> +  if (!EventStream) {
> +    return nullptr;
> +  }
> +
> +  std::unique_ptr<DirectoryWatcher> Result =
> +      llvm::make_unique<DirectoryWatcherMac>(EventStream, Receiver, Path);
> +
> +  // We need to copy the data so the lifetime is ok after a const copy is made
> +  // for the block.
> +  const std::string CopiedPath = Path;
> +
> +  auto InitWork = ^{
> +    // We need to start watching the directory before we start scanning in order
> +    // to not miss any event. By dispatching this on the same serial Queue as
> +    // the FSEvents will be handled we manage to start watching BEFORE the
> +    // inital scan and handling events ONLY AFTER the scan finishes.
> +    FSEventStreamSetDispatchQueue(EventStream, Queue);
> +    FSEventStreamStart(EventStream);
> +    // We need to decrement the ref count for Queue as initialize() will return
> +    // and FSEvents has incremented it. Since we have to wait for FSEvents to
> +    // take ownership it's the easiest to do it here rather than main thread.
> +    dispatch_release(Queue);
> +    Receiver(getAsFileEvents(scanDirectory(CopiedPath)), /*IsInitial=*/true);
> +  };
> +
> +  if (WaitForInitialSync) {
> +    dispatch_sync(Queue, InitWork);
> +  } else {
> +    dispatch_async(Queue, InitWork);
> +  }
> +
> +  return Result;
> +}
> 
> Modified: cfe/trunk/unittests/CMakeLists.txt
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/unittests/CMakeLists.txt?rev=365954&r1=365953&r2=365954&view=diff
> ==============================================================================
> --- cfe/trunk/unittests/CMakeLists.txt (original)
> +++ cfe/trunk/unittests/CMakeLists.txt Fri Jul 12 13:34:10 2019
> @@ -30,6 +30,7 @@ add_subdirectory(CodeGen)
>   if(NOT WIN32 AND CLANG_TOOL_LIBCLANG_BUILD)
>     add_subdirectory(libclang)
>   endif()
> +add_subdirectory(DirectoryWatcher)
>   add_subdirectory(Rename)
>   add_subdirectory(Index)
>   add_subdirectory(Serialization)
> 
> Added: cfe/trunk/unittests/DirectoryWatcher/CMakeLists.txt
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/unittests/DirectoryWatcher/CMakeLists.txt?rev=365954&view=auto
> ==============================================================================
> --- cfe/trunk/unittests/DirectoryWatcher/CMakeLists.txt (added)
> +++ cfe/trunk/unittests/DirectoryWatcher/CMakeLists.txt Fri Jul 12 13:34:10 2019
> @@ -0,0 +1,17 @@
> +if(APPLE OR CMAKE_SYSTEM_NAME MATCHES "Linux")
> +
> +  set(LLVM_LINK_COMPONENTS
> +    Support
> +    )
> +
> +  add_clang_unittest(DirectoryWatcherTests
> +    DirectoryWatcherTest.cpp
> +    )
> +
> +  target_link_libraries(DirectoryWatcherTests
> +    PRIVATE
> +    clangDirectoryWatcher
> +    clangBasic
> +    )
> +
> +endif()
> \ No newline at end of file
> 
> Added: cfe/trunk/unittests/DirectoryWatcher/DirectoryWatcherTest.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/unittests/DirectoryWatcher/DirectoryWatcherTest.cpp?rev=365954&view=auto
> ==============================================================================
> --- cfe/trunk/unittests/DirectoryWatcher/DirectoryWatcherTest.cpp (added)
> +++ cfe/trunk/unittests/DirectoryWatcher/DirectoryWatcherTest.cpp Fri Jul 12 13:34:10 2019
> @@ -0,0 +1,426 @@
> +//===- unittests/DirectoryWatcher/DirectoryWatcherTest.cpp ----------------===//
> +//
> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
> +// See https://llvm.org/LICENSE.txt for license information.
> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
> +//
> +//===----------------------------------------------------------------------===//
> +
> +#include "clang/DirectoryWatcher/DirectoryWatcher.h"
> +#include "llvm/Support/FileSystem.h"
> +#include "llvm/Support/Mutex.h"
> +#include "llvm/Support/Path.h"
> +#include "llvm/Support/raw_ostream.h"
> +#include "gtest/gtest.h"
> +#include <condition_variable>
> +#include <future>
> +#include <mutex>
> +#include <thread>
> +
> +using namespace llvm;
> +using namespace llvm::sys;
> +using namespace llvm::sys::fs;
> +using namespace clang;
> +
> +namespace clang {
> +static bool operator==(const DirectoryWatcher::Event &lhs,
> +                       const DirectoryWatcher::Event &rhs) {
> +  return lhs.Filename == rhs.Filename &&
> +         static_cast<int>(lhs.Kind) == static_cast<int>(rhs.Kind);
> +}
> +} // namespace clang
> +
> +namespace {
> +
> +struct DirectoryWatcherTestFixture {
> +  std::string TestRootDir;
> +  std::string TestWatchedDir;
> +
> +  DirectoryWatcherTestFixture() {
> +    SmallString<128> pathBuf;
> +    std::error_code UniqDirRes = createUniqueDirectory("dirwatcher", pathBuf);
> +    assert(!UniqDirRes);
> +    TestRootDir = pathBuf.str();
> +    path::append(pathBuf, "watch");
> +    TestWatchedDir = pathBuf.str();
> +    std::error_code CreateDirRes = create_directory(TestWatchedDir, false);
> +    assert(!CreateDirRes);
> +  }
> +
> +  ~DirectoryWatcherTestFixture() { remove_directories(TestRootDir); }
> +
> +  SmallString<128> getPathInWatched(const std::string &testFile) {
> +    SmallString<128> pathBuf;
> +    pathBuf = TestWatchedDir;
> +    path::append(pathBuf, testFile);
> +    return pathBuf;
> +  }
> +
> +  void addFile(const std::string &testFile) {
> +    Expected<file_t> ft = openNativeFileForWrite(getPathInWatched(testFile),
> +                                                 CD_CreateNew, OF_None);
> +    if (ft) {
> +      closeFile(*ft);
> +    } else {
> +      llvm::errs() << llvm::toString(ft.takeError()) << "\n";
> +      llvm::errs() << getPathInWatched(testFile) << "\n";
> +      llvm_unreachable("Couldn't create test file.");
> +    }
> +  }
> +
> +  void deleteFile(const std::string &testFile) {
> +    std::error_code EC =
> +        remove(getPathInWatched(testFile), /*IgnoreNonExisting=*/false);
> +    ASSERT_FALSE(EC);
> +  }
> +};
> +
> +std::string eventKindToString(const DirectoryWatcher::Event::EventKind K) {
> +  switch (K) {
> +  case DirectoryWatcher::Event::EventKind::Removed:
> +    return "Removed";
> +  case DirectoryWatcher::Event::EventKind::Modified:
> +    return "Modified";
> +  case DirectoryWatcher::Event::EventKind::WatchedDirRemoved:
> +    return "WatchedDirRemoved";
> +  case DirectoryWatcher::Event::EventKind::WatcherGotInvalidated:
> +    return "WatcherGotInvalidated";
> +  }
> +  llvm_unreachable("unknown event kind");
> +}
> +
> +struct VerifyingConsumer {
> +  std::vector<DirectoryWatcher::Event> ExpectedInitial;
> +  std::vector<DirectoryWatcher::Event> ExpectedNonInitial;
> +  std::vector<DirectoryWatcher::Event> OptionalNonInitial;
> +  std::vector<DirectoryWatcher::Event> UnexpectedInitial;
> +  std::vector<DirectoryWatcher::Event> UnexpectedNonInitial;
> +  std::mutex Mtx;
> +  std::condition_variable ResultIsReady;
> +
> +  VerifyingConsumer(
> +      const std::vector<DirectoryWatcher::Event> &ExpectedInitial,
> +      const std::vector<DirectoryWatcher::Event> &ExpectedNonInitial,
> +      const std::vector<DirectoryWatcher::Event> &OptionalNonInitial = {})
> +      : ExpectedInitial(ExpectedInitial),
> +        ExpectedNonInitial(ExpectedNonInitial),
> +        OptionalNonInitial(OptionalNonInitial) {}
> +
> +  // This method is used by DirectoryWatcher.
> +  void consume(DirectoryWatcher::Event E, bool IsInitial) {
> +    if (IsInitial)
> +      consumeInitial(E);
> +    else
> +      consumeNonInitial(E);
> +  }
> +
> +  void consumeInitial(DirectoryWatcher::Event E) {
> +    std::unique_lock<std::mutex> L(Mtx);
> +    auto It = std::find(ExpectedInitial.begin(), ExpectedInitial.end(), E);
> +    if (It == ExpectedInitial.end()) {
> +      UnexpectedInitial.push_back(E);
> +    } else {
> +      ExpectedInitial.erase(It);
> +    }
> +    if (result())
> +      ResultIsReady.notify_one();
> +  }
> +
> +  void consumeNonInitial(DirectoryWatcher::Event E) {
> +    std::unique_lock<std::mutex> L(Mtx);
> +    auto It =
> +        std::find(ExpectedNonInitial.begin(), ExpectedNonInitial.end(), E);
> +    if (It == ExpectedNonInitial.end()) {
> +      auto OptIt =
> +          std::find(OptionalNonInitial.begin(), OptionalNonInitial.end(), E);
> +      if (OptIt != OptionalNonInitial.end()) {
> +        OptionalNonInitial.erase(OptIt);
> +      } else {
> +        UnexpectedNonInitial.push_back(E);
> +      }
> +    } else {
> +      ExpectedNonInitial.erase(It);
> +    }
> +    if (result())
> +      ResultIsReady.notify_one();
> +  }
> +
> +  // This method is used by DirectoryWatcher.
> +  void consume(llvm::ArrayRef<DirectoryWatcher::Event> Es, bool IsInitial) {
> +    for (const auto &E : Es)
> +      consume(E, IsInitial);
> +  }
> +
> +  // Not locking - caller has to lock Mtx.
> +  llvm::Optional<bool> result() const {
> +    if (ExpectedInitial.empty() && ExpectedNonInitial.empty() &&
> +        UnexpectedInitial.empty() && UnexpectedNonInitial.empty())
> +      return true;
> +    if (!UnexpectedInitial.empty() || !UnexpectedNonInitial.empty())
> +      return false;
> +    return llvm::None;
> +  }
> +
> +  // This method is used by tests.
> +  // \returns true on success
> +  bool blockUntilResult() {
> +    std::unique_lock<std::mutex> L(Mtx);
> +    while (true) {
> +      if (result())
> +        return *result();
> +
> +      ResultIsReady.wait(L, [this]() { return result().hasValue(); });
> +    }
> +    return false; // Just to make compiler happy.
> +  }
> +
> +  void printUnmetExpectations(llvm::raw_ostream &OS) {
> +    if (!ExpectedInitial.empty()) {
> +      OS << "Expected but not seen initial events: \n";
> +      for (const auto &E : ExpectedInitial) {
> +        OS << eventKindToString(E.Kind) << " " << E.Filename << "\n";
> +      }
> +    }
> +    if (!ExpectedNonInitial.empty()) {
> +      OS << "Expected but not seen non-initial events: \n";
> +      for (const auto &E : ExpectedNonInitial) {
> +        OS << eventKindToString(E.Kind) << " " << E.Filename << "\n";
> +      }
> +    }
> +    if (!UnexpectedInitial.empty()) {
> +      OS << "Unexpected initial events seen: \n";
> +      for (const auto &E : UnexpectedInitial) {
> +        OS << eventKindToString(E.Kind) << " " << E.Filename << "\n";
> +      }
> +    }
> +    if (!UnexpectedNonInitial.empty()) {
> +      OS << "Unexpected non-initial events seen: \n";
> +      for (const auto &E : UnexpectedNonInitial) {
> +        OS << eventKindToString(E.Kind) << " " << E.Filename << "\n";
> +      }
> +    }
> +  }
> +};
> +
> +void checkEventualResultWithTimeout(VerifyingConsumer &TestConsumer) {
> +  std::packaged_task<int(void)> task(
> +      [&TestConsumer]() { return TestConsumer.blockUntilResult(); });
> +  std::future<int> WaitForExpectedStateResult = task.get_future();
> +  std::thread worker(std::move(task));
> +  worker.detach();
> +
> +  EXPECT_TRUE(WaitForExpectedStateResult.wait_for(std::chrono::seconds(3)) ==
> +              std::future_status::ready)
> +      << "The expected result state wasn't reached before the time-out.";
> +  EXPECT_TRUE(TestConsumer.result().hasValue());
> +  if (TestConsumer.result().hasValue()) {
> +    EXPECT_TRUE(*TestConsumer.result());
> +  }
> +  if ((TestConsumer.result().hasValue() && !TestConsumer.result().getValue()) ||
> +      !TestConsumer.result().hasValue())
> +    TestConsumer.printUnmetExpectations(llvm::outs());
> +}
> +
> +} // namespace
> +
> +TEST(DirectoryWatcherTest, InitialScanSync) {
> +  DirectoryWatcherTestFixture fixture;
> +
> +  fixture.addFile("a");
> +  fixture.addFile("b");
> +  fixture.addFile("c");
> +
> +  VerifyingConsumer TestConsumer{
> +      {{DirectoryWatcher::Event::EventKind::Modified, "a"},
> +       {DirectoryWatcher::Event::EventKind::Modified, "b"},
> +       {DirectoryWatcher::Event::EventKind::Modified, "c"}},
> +      {}};
> +
> +  auto DW = DirectoryWatcher::create(
> +      fixture.TestWatchedDir,
> +      [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
> +                      bool IsInitial) {
> +        TestConsumer.consume(Events, IsInitial);
> +      },
> +      /*waitForInitialSync=*/true);
> +
> +  checkEventualResultWithTimeout(TestConsumer);
> +}
> +
> +TEST(DirectoryWatcherTest, InitialScanAsync) {
> +  DirectoryWatcherTestFixture fixture;
> +
> +  fixture.addFile("a");
> +  fixture.addFile("b");
> +  fixture.addFile("c");
> +
> +  VerifyingConsumer TestConsumer{
> +      {{DirectoryWatcher::Event::EventKind::Modified, "a"},
> +       {DirectoryWatcher::Event::EventKind::Modified, "b"},
> +       {DirectoryWatcher::Event::EventKind::Modified, "c"}},
> +      {}};
> +
> +  auto DW = DirectoryWatcher::create(
> +      fixture.TestWatchedDir,
> +      [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
> +                      bool IsInitial) {
> +        TestConsumer.consume(Events, IsInitial);
> +      },
> +      /*waitForInitialSync=*/false);
> +
> +  checkEventualResultWithTimeout(TestConsumer);
> +}
> +
> +TEST(DirectoryWatcherTest, AddFiles) {
> +  DirectoryWatcherTestFixture fixture;
> +
> +  VerifyingConsumer TestConsumer{
> +      {},
> +      {{DirectoryWatcher::Event::EventKind::Modified, "a"},
> +       {DirectoryWatcher::Event::EventKind::Modified, "b"},
> +       {DirectoryWatcher::Event::EventKind::Modified, "c"}}};
> +
> +  auto DW = DirectoryWatcher::create(
> +      fixture.TestWatchedDir,
> +      [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
> +                      bool IsInitial) {
> +        TestConsumer.consume(Events, IsInitial);
> +      },
> +      /*waitForInitialSync=*/true);
> +
> +  fixture.addFile("a");
> +  fixture.addFile("b");
> +  fixture.addFile("c");
> +
> +  checkEventualResultWithTimeout(TestConsumer);
> +}
> +
> +TEST(DirectoryWatcherTest, ModifyFile) {
> +  DirectoryWatcherTestFixture fixture;
> +
> +  fixture.addFile("a");
> +
> +  VerifyingConsumer TestConsumer{
> +      {{DirectoryWatcher::Event::EventKind::Modified, "a"}},
> +      {{DirectoryWatcher::Event::EventKind::Modified, "a"}}};
> +
> +  auto DW = DirectoryWatcher::create(
> +      fixture.TestWatchedDir,
> +      [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
> +                      bool IsInitial) {
> +        TestConsumer.consume(Events, IsInitial);
> +      },
> +      /*waitForInitialSync=*/true);
> +
> +  // modify the file
> +  {
> +    std::error_code error;
> +    llvm::raw_fd_ostream bStream(fixture.getPathInWatched("a"), error,
> +                                 CD_OpenExisting);
> +    assert(!error);
> +    bStream << "foo";
> +  }
> +
> +  checkEventualResultWithTimeout(TestConsumer);
> +}
> +
> +TEST(DirectoryWatcherTest, DeleteFile) {
> +  DirectoryWatcherTestFixture fixture;
> +
> +  fixture.addFile("a");
> +
> +  VerifyingConsumer TestConsumer{
> +      {{DirectoryWatcher::Event::EventKind::Modified, "a"}},
> +      {{DirectoryWatcher::Event::EventKind::Removed, "a"}}};
> +
> +  auto DW = DirectoryWatcher::create(
> +      fixture.TestWatchedDir,
> +      [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
> +                      bool IsInitial) {
> +        TestConsumer.consume(Events, IsInitial);
> +      },
> +      /*waitForInitialSync=*/true);
> +
> +  fixture.deleteFile("a");
> +
> +  checkEventualResultWithTimeout(TestConsumer);
> +}
> +
> +TEST(DirectoryWatcherTest, DeleteWatchedDir) {
> +  DirectoryWatcherTestFixture fixture;
> +
> +  VerifyingConsumer TestConsumer{
> +      {},
> +      {{DirectoryWatcher::Event::EventKind::WatchedDirRemoved, ""},
> +       {DirectoryWatcher::Event::EventKind::WatcherGotInvalidated, ""}}};
> +
> +  auto DW = DirectoryWatcher::create(
> +      fixture.TestWatchedDir,
> +      [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
> +                      bool IsInitial) {
> +        TestConsumer.consume(Events, IsInitial);
> +      },
> +      /*waitForInitialSync=*/true);
> +
> +  remove_directories(fixture.TestWatchedDir);
> +
> +  checkEventualResultWithTimeout(TestConsumer);
> +}
> +
> +TEST(DirectoryWatcherTest, InvalidatedWatcher) {
> +  DirectoryWatcherTestFixture fixture;
> +
> +  VerifyingConsumer TestConsumer{
> +      {}, {{DirectoryWatcher::Event::EventKind::WatcherGotInvalidated, ""}}};
> +
> +  {
> +    auto DW = DirectoryWatcher::create(
> +        fixture.TestWatchedDir,
> +        [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
> +                        bool IsInitial) {
> +          TestConsumer.consume(Events, IsInitial);
> +        },
> +        /*waitForInitialSync=*/true);
> +  } // DW is destructed here.
> +
> +  checkEventualResultWithTimeout(TestConsumer);
> +}
> +
> +TEST(DirectoryWatcherTest, ChangeMetadata) {
> +  DirectoryWatcherTestFixture fixture;
> +  fixture.addFile("a");
> +
> +  VerifyingConsumer TestConsumer{
> +      {{DirectoryWatcher::Event::EventKind::Modified, "a"}},
> +      // We don't expect any notification for file having access file changed.
> +      {},
> +      // Given the timing we are ok with receiving the duplicate event.
> +      {{DirectoryWatcher::Event::EventKind::Modified, "a"}}};
> +
> +  auto DW = DirectoryWatcher::create(
> +      fixture.TestWatchedDir,
> +      [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
> +                      bool IsInitial) {
> +        TestConsumer.consume(Events, IsInitial);
> +      },
> +      /*waitForInitialSync=*/true);
> +
> +  { // Change access and modification time of file a.
> +    Expected<file_t> HopefullyTheFD = llvm::sys::fs::openNativeFileForWrite(
> +        fixture.getPathInWatched("a"), CD_OpenExisting, OF_None);
> +    if (!HopefullyTheFD) {
> +      llvm::outs() << HopefullyTheFD.takeError();
> +    }
> +
> +    const int FD = HopefullyTheFD.get();
> +    const TimePoint<> NewTimePt =
> +        std::chrono::system_clock::now() - std::chrono::minutes(1);
> +
> +    std::error_code setTimeRes =
> +        llvm::sys::fs::setLastAccessAndModificationTime(FD, NewTimePt,
> +                                                        NewTimePt);
> +    assert(!setTimeRes);
> +  }
> +
> +  checkEventualResultWithTimeout(TestConsumer);
> +}
> 
> 
> _______________________________________________
> cfe-commits mailing list
> cfe-commits at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
> 


More information about the cfe-commits mailing list