r365954 - Reland [clang] DirectoryWatcher
Jan Korous via cfe-commits
cfe-commits at lists.llvm.org
Mon Jul 15 11:26:33 PDT 2019
Hi Mikael,
Thanks for the notice.
How about this?
https://reviews.llvm.org/D64764 <https://reviews.llvm.org/D64764>
Jan
> On Jul 14, 2019, at 11:45 PM, Mikael Holmén <mikael.holmen at ericsson.com> wrote:
>
> Hi Jan,
>
>> +elseif(CMAKE_SYSTEM_NAME MATCHES "Linux")
>> + check_include_files("sys/inotify.h" HAVE_INOTIFY)
>> + if(HAVE_INOTIFY)
>> + list(APPEND DIRECTORY_WATCHER_SOURCES
> linux/DirectoryWatcher-linux.cpp)
>> + find_package(Threads REQUIRED)
>> + endif()
>
> I don't think the above is enough, I think the version needs to be
> checked some way as well to make sure we find an inotify.h that really
> defines IN_EXCL_UNLINK.
>
> We have RHEL 6.10 buildbots with glibc 2.12.2 and kernel 2.6.32 where I see
>
> -- Looking for include file sys/inotify.h
> -- Looking for include file sys/inotify.h - found
>
> but then get
>
> /repo/uabelho/master/clang/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp:335:48:
> error: 'IN_EXCL_UNLINK' was not declared in this scope
> IN_CREATE | IN_DELETE | IN_DELETE_SELF | IN_EXCL_UNLINK |
> IN_MODIFY |
> ^~~~~~~~~~~~~~
>
> Regards,
> Mikael
>
> On 2019-07-12 22:34, Jan Korous via cfe-commits wrote:
>> Author: jkorous
>> Date: Fri Jul 12 13:34:10 2019
>> New Revision: 365954
>>
>> URL: http://llvm.org/viewvc/llvm-project?rev=365954&view=rev
>> Log:
>> Reland [clang] DirectoryWatcher
>>
>> This reverts commit f561227d133224d2d6a5a016abe4be051fa75501.
>>
>> - DirectoryWatcher
>> - Fix the build for platforms that don't have DW implementated.
>> - Fix the threading dependencies (thanks to compnerd).
>>
>> Added:
>> cfe/trunk/include/clang/DirectoryWatcher/
>> cfe/trunk/include/clang/DirectoryWatcher/DirectoryWatcher.h
>> cfe/trunk/lib/DirectoryWatcher/
>> cfe/trunk/lib/DirectoryWatcher/CMakeLists.txt
>> cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.cpp
>> cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.h
>> cfe/trunk/lib/DirectoryWatcher/default/
>> cfe/trunk/lib/DirectoryWatcher/default/DirectoryWatcher-not-implemented.cpp
>> cfe/trunk/lib/DirectoryWatcher/linux/
>> cfe/trunk/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp
>> cfe/trunk/lib/DirectoryWatcher/mac/
>> cfe/trunk/lib/DirectoryWatcher/mac/DirectoryWatcher-mac.cpp
>> cfe/trunk/unittests/DirectoryWatcher/
>> cfe/trunk/unittests/DirectoryWatcher/CMakeLists.txt
>> cfe/trunk/unittests/DirectoryWatcher/DirectoryWatcherTest.cpp
>> Modified:
>> cfe/trunk/lib/CMakeLists.txt
>> cfe/trunk/unittests/CMakeLists.txt
>>
>> Added: cfe/trunk/include/clang/DirectoryWatcher/DirectoryWatcher.h
>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/DirectoryWatcher/DirectoryWatcher.h?rev=365954&view=auto
>> ==============================================================================
>> --- cfe/trunk/include/clang/DirectoryWatcher/DirectoryWatcher.h (added)
>> +++ cfe/trunk/include/clang/DirectoryWatcher/DirectoryWatcher.h Fri Jul 12 13:34:10 2019
>> @@ -0,0 +1,123 @@
>> +//===- DirectoryWatcher.h - Listens for directory file changes --*- C++ -*-===//
>> +//
>> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
>> +// See https://llvm.org/LICENSE.txt for license information.
>> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
>> +//
>> +//===----------------------------------------------------------------------===//
>> +
>> +#ifndef LLVM_CLANG_DIRECTORYWATCHER_DIRECTORYWATCHER_H
>> +#define LLVM_CLANG_DIRECTORYWATCHER_DIRECTORYWATCHER_H
>> +
>> +#include "llvm/ADT/ArrayRef.h"
>> +#include "llvm/ADT/StringRef.h"
>> +#include <functional>
>> +#include <memory>
>> +#include <string>
>> +
>> +namespace clang {
>> +/// Provides notifications for file changes in a directory.
>> +///
>> +/// Invokes client-provided function on every filesystem event in the watched
>> +/// directory. Initially the the watched directory is scanned and for every file
>> +/// found, an event is synthesized as if the file was added.
>> +///
>> +/// This is not a general purpose directory monitoring tool - list of
>> +/// limitations follows.
>> +///
>> +/// Only flat directories with no subdirectories are supported. In case
>> +/// subdirectories are present the behavior is unspecified - events *might* be
>> +/// passed to Receiver on macOS (due to FSEvents being used) while they
>> +/// *probably* won't be passed on Linux (due to inotify being used).
>> +///
>> +/// Known potential inconsistencies
>> +/// - For files that are deleted befor the initial scan processed them, clients
>> +/// might receive Removed notification without any prior Added notification.
>> +/// - Multiple notifications might be produced when a file is added to the
>> +/// watched directory during the initial scan. We are choosing the lesser evil
>> +/// here as the only known alternative strategy would be to invalidate the
>> +/// watcher instance and force user to create a new one whenever filesystem
>> +/// event occurs during the initial scan but that would introduce continuous
>> +/// restarting failure mode (watched directory is not always "owned" by the same
>> +/// process that is consuming it). Since existing clients can handle duplicate
>> +/// events well, we decided for simplicity.
>> +///
>> +/// Notifications are provided only for changes done through local user-space
>> +/// filesystem interface. Specifically, it's unspecified if notification would
>> +/// be provided in case of a:
>> +/// - a file mmap-ed and changed
>> +/// - a file changed via remote (NFS) or virtual (/proc) FS access to monitored
>> +/// directory
>> +/// - another filesystem mounted to the watched directory
>> +///
>> +/// No support for LLVM VFS.
>> +///
>> +/// It is unspecified whether notifications for files being deleted are sent in
>> +/// case the whole watched directory is sent.
>> +///
>> +/// Directories containing "too many" files and/or receiving events "too
>> +/// frequently" are not supported - if the initial scan can't be finished before
>> +/// the watcher instance gets invalidated (see WatcherGotInvalidated) there's no
>> +/// good error handling strategy - the only option for client is to destroy the
>> +/// watcher, restart watching with new instance and hope it won't repeat.
>> +class DirectoryWatcher {
>> +public:
>> + struct Event {
>> + enum class EventKind {
>> + Removed,
>> + /// Content of a file was modified.
>> + Modified,
>> + /// The watched directory got deleted.
>> + WatchedDirRemoved,
>> + /// The DirectoryWatcher that originated this event is no longer valid and
>> + /// its behavior is unspecified.
>> + ///
>> + /// The prime case is kernel signalling to OS-specific implementation of
>> + /// DirectoryWatcher some resource limit being hit.
>> + /// *Usually* kernel starts dropping or squashing events together after
>> + /// that and so would DirectoryWatcher. This means that *some* events
>> + /// might still be passed to Receiver but this behavior is unspecified.
>> + ///
>> + /// Another case is after the watched directory itself is deleted.
>> + /// WatcherGotInvalidated will be received at least once during
>> + /// DirectoryWatcher instance lifetime - when handling errors this is done
>> + /// on best effort basis, when an instance is being destroyed then this is
>> + /// guaranteed.
>> + ///
>> + /// The only proper response to this kind of event is to destruct the
>> + /// originating DirectoryWatcher instance and create a new one.
>> + WatcherGotInvalidated
>> + };
>> +
>> + EventKind Kind;
>> + /// Filename that this event is related to or an empty string in
>> + /// case this event is related to the watched directory itself.
>> + std::string Filename;
>> +
>> + Event(EventKind Kind, llvm::StringRef Filename)
>> + : Kind(Kind), Filename(Filename) {}
>> + };
>> +
>> + /// Returns nullptr if \param Path doesn't exist.
>> + /// Returns nullptr if \param Path isn't a directory.
>> + /// Returns nullptr if OS kernel API told us we can't start watching. In such
>> + /// case it's unclear whether just retrying has any chance to succeeed.
>> + static std::unique_ptr<DirectoryWatcher>
>> + create(llvm::StringRef Path,
>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event> Events,
>> + bool IsInitial)>
>> + Receiver,
>> + bool WaitForInitialSync);
>> +
>> + virtual ~DirectoryWatcher() = default;
>> + DirectoryWatcher(const DirectoryWatcher &) = delete;
>> + DirectoryWatcher &operator=(const DirectoryWatcher &) = delete;
>> + DirectoryWatcher(DirectoryWatcher &&) = default;
>> +
>> +protected:
>> + DirectoryWatcher() = default;
>> +};
>> +
>> +} // namespace clang
>> +
>> +#endif // LLVM_CLANG_DIRECTORYWATCHER_DIRECTORYWATCHER_H
>>
>> Modified: cfe/trunk/lib/CMakeLists.txt
>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/CMakeLists.txt?rev=365954&r1=365953&r2=365954&view=diff
>> ==============================================================================
>> --- cfe/trunk/lib/CMakeLists.txt (original)
>> +++ cfe/trunk/lib/CMakeLists.txt Fri Jul 12 13:34:10 2019
>> @@ -18,6 +18,7 @@ add_subdirectory(Serialization)
>> add_subdirectory(Frontend)
>> add_subdirectory(FrontendTool)
>> add_subdirectory(Tooling)
>> +add_subdirectory(DirectoryWatcher)
>> add_subdirectory(Index)
>> if(CLANG_ENABLE_STATIC_ANALYZER)
>> add_subdirectory(StaticAnalyzer)
>>
>> Added: cfe/trunk/lib/DirectoryWatcher/CMakeLists.txt
>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/CMakeLists.txt?rev=365954&view=auto
>> ==============================================================================
>> --- cfe/trunk/lib/DirectoryWatcher/CMakeLists.txt (added)
>> +++ cfe/trunk/lib/DirectoryWatcher/CMakeLists.txt Fri Jul 12 13:34:10 2019
>> @@ -0,0 +1,29 @@
>> +include(CheckIncludeFiles)
>> +
>> +set(LLVM_LINK_COMPONENTS support)
>> +
>> +set(DIRECTORY_WATCHER_SOURCES DirectoryScanner.cpp)
>> +set(DIRECTORY_WATCHER_LINK_LIBS "")
>> +
>> +if(APPLE)
>> + check_include_files("CoreServices/CoreServices.h" HAVE_CORESERVICES)
>> + if(HAVE_CORESERVICES)
>> + list(APPEND DIRECTORY_WATCHER_SOURCES mac/DirectoryWatcher-mac.cpp)
>> + set(DIRECTORY_WATCHER_LINK_LIBS "-framework CoreServices")
>> + endif()
>> +elseif(CMAKE_SYSTEM_NAME MATCHES "Linux")
>> + check_include_files("sys/inotify.h" HAVE_INOTIFY)
>> + if(HAVE_INOTIFY)
>> + list(APPEND DIRECTORY_WATCHER_SOURCES linux/DirectoryWatcher-linux.cpp)
>> + find_package(Threads REQUIRED)
>> + endif()
>> +else()
>> + list(APPEND DIRECTORY_WATCHER_SOURCES default/DirectoryWatcher-not-implemented.cpp)
>> +endif()
>> +
>> +add_clang_library(clangDirectoryWatcher
>> + ${DIRECTORY_WATCHER_SOURCES}
>> + )
>> +
>> +target_link_libraries(clangDirectoryWatcher PUBLIC ${CMAKE_THREAD_LIBS_INIT})
>> +target_link_libraries(clangDirectoryWatcher PRIVATE ${DIRECTORY_WATCHER_LINK_LIBS})
>>
>> Added: cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.cpp
>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.cpp?rev=365954&view=auto
>> ==============================================================================
>> --- cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.cpp (added)
>> +++ cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.cpp Fri Jul 12 13:34:10 2019
>> @@ -0,0 +1,54 @@
>> +//===- DirectoryScanner.cpp - Utility functions for DirectoryWatcher ------===//
>> +//
>> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
>> +// See https://llvm.org/LICENSE.txt for license information.
>> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
>> +//
>> +//===----------------------------------------------------------------------===//
>> +
>> +#include "DirectoryScanner.h"
>> +
>> +#include "llvm/Support/Path.h"
>> +
>> +namespace clang {
>> +
>> +using namespace llvm;
>> +
>> +Optional<sys::fs::file_status> getFileStatus(StringRef Path) {
>> + sys::fs::file_status Status;
>> + std::error_code EC = status(Path, Status);
>> + if (EC)
>> + return None;
>> + return Status;
>> +}
>> +
>> +std::vector<std::string> scanDirectory(StringRef Path) {
>> + using namespace llvm::sys;
>> + std::vector<std::string> Result;
>> +
>> + std::error_code EC;
>> + for (auto It = fs::directory_iterator(Path, EC),
>> + End = fs::directory_iterator();
>> + !EC && It != End; It.increment(EC)) {
>> + auto status = getFileStatus(It->path());
>> + if (!status.hasValue())
>> + continue;
>> + Result.emplace_back(sys::path::filename(It->path()));
>> + }
>> +
>> + return Result;
>> +}
>> +
>> +std::vector<DirectoryWatcher::Event>
>> +getAsFileEvents(const std::vector<std::string> &Scan) {
>> + std::vector<DirectoryWatcher::Event> Events;
>> + Events.reserve(Scan.size());
>> +
>> + for (const auto &File : Scan) {
>> + Events.emplace_back(DirectoryWatcher::Event{
>> + DirectoryWatcher::Event::EventKind::Modified, File});
>> + }
>> + return Events;
>> +}
>> +
>> +} // namespace clang
>> \ No newline at end of file
>>
>> Added: cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.h
>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.h?rev=365954&view=auto
>> ==============================================================================
>> --- cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.h (added)
>> +++ cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.h Fri Jul 12 13:34:10 2019
>> @@ -0,0 +1,29 @@
>> +//===- DirectoryScanner.h - Utility functions for DirectoryWatcher --------===//
>> +//
>> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
>> +// See https://llvm.org/LICENSE.txt for license information.
>> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
>> +//
>> +//===----------------------------------------------------------------------===//
>> +
>> +#include "clang/DirectoryWatcher/DirectoryWatcher.h"
>> +#include "llvm/Support/FileSystem.h"
>> +#include <string>
>> +#include <vector>
>> +
>> +namespace clang {
>> +
>> +/// Gets names (filenames) of items in directory at \p Path.
>> +/// \returns empty vector if \p Path is not a directory, doesn't exist or can't
>> +/// be read from.
>> +std::vector<std::string> scanDirectory(llvm::StringRef Path);
>> +
>> +/// Create event with EventKind::Added for every element in \p Scan.
>> +std::vector<DirectoryWatcher::Event>
>> +getAsFileEvents(const std::vector<std::string> &Scan);
>> +
>> +/// Gets status of file (or directory) at \p Path.
>> +/// \returns llvm::None if \p Path doesn't exist or can't get the status.
>> +llvm::Optional<llvm::sys::fs::file_status> getFileStatus(llvm::StringRef Path);
>> +
>> +} // namespace clang
>> \ No newline at end of file
>>
>> Added: cfe/trunk/lib/DirectoryWatcher/default/DirectoryWatcher-not-implemented.cpp
>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/default/DirectoryWatcher-not-implemented.cpp?rev=365954&view=auto
>> ==============================================================================
>> --- cfe/trunk/lib/DirectoryWatcher/default/DirectoryWatcher-not-implemented.cpp (added)
>> +++ cfe/trunk/lib/DirectoryWatcher/default/DirectoryWatcher-not-implemented.cpp Fri Jul 12 13:34:10 2019
>> @@ -0,0 +1,19 @@
>> +//===- DirectoryWatcher-not-implemented.cpp -------------------------------===//
>> +//
>> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
>> +// See https://llvm.org/LICENSE.txt for license information.
>> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
>> +//
>> +//===----------------------------------------------------------------------===//
>> +
>> +#include "clang/DirectoryWatcher/DirectoryWatcher.h"
>> +
>> +using namespace llvm;
>> +using namespace clang;
>> +
>> +std::unique_ptr<DirectoryWatcher> clang::DirectoryWatcher::create(
>> + StringRef Path,
>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)> Receiver,
>> + bool WaitForInitialSync) {
>> + return nullptr;
>> +}
>> \ No newline at end of file
>>
>> Added: cfe/trunk/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp
>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp?rev=365954&view=auto
>> ==============================================================================
>> --- cfe/trunk/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp (added)
>> +++ cfe/trunk/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp Fri Jul 12 13:34:10 2019
>> @@ -0,0 +1,345 @@
>> +//===- DirectoryWatcher-linux.cpp - Linux-platform directory watching -----===//
>> +//
>> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
>> +// See https://llvm.org/LICENSE.txt for license information.
>> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
>> +//
>> +//===----------------------------------------------------------------------===//
>> +
>> +#include "DirectoryScanner.h"
>> +#include "clang/DirectoryWatcher/DirectoryWatcher.h"
>> +
>> +#include "llvm/ADT/STLExtras.h"
>> +#include "llvm/ADT/ScopeExit.h"
>> +#include "llvm/Support/AlignOf.h"
>> +#include "llvm/Support/Errno.h"
>> +#include "llvm/Support/Mutex.h"
>> +#include "llvm/Support/Path.h"
>> +#include <atomic>
>> +#include <condition_variable>
>> +#include <mutex>
>> +#include <queue>
>> +#include <string>
>> +#include <thread>
>> +#include <vector>
>> +
>> +#include <fcntl.h>
>> +#include <sys/epoll.h>
>> +#include <sys/inotify.h>
>> +#include <unistd.h>
>> +
>> +namespace {
>> +
>> +using namespace llvm;
>> +using namespace clang;
>> +
>> +/// Pipe for inter-thread synchronization - for epoll-ing on multiple
>> +/// conditions. It is meant for uni-directional 1:1 signalling - specifically:
>> +/// no multiple consumers, no data passing. Thread waiting for signal should
>> +/// poll the FDRead. Signalling thread should call signal() which writes single
>> +/// character to FDRead.
>> +struct SemaphorePipe {
>> + // Expects two file-descriptors opened as a pipe in the canonical POSIX
>> + // order: pipefd[0] refers to the read end of the pipe. pipefd[1] refers to
>> + // the write end of the pipe.
>> + SemaphorePipe(int pipefd[2])
>> + : FDRead(pipefd[0]), FDWrite(pipefd[1]), OwnsFDs(true) {}
>> + SemaphorePipe(const SemaphorePipe &) = delete;
>> + void operator=(const SemaphorePipe &) = delete;
>> + SemaphorePipe(SemaphorePipe &&other)
>> + : FDRead(other.FDRead), FDWrite(other.FDWrite),
>> + OwnsFDs(other.OwnsFDs) // Someone could have moved from the other
>> + // instance before.
>> + {
>> + other.OwnsFDs = false;
>> + };
>> +
>> + void signal() {
>> + ssize_t Result = llvm::sys::RetryAfterSignal(-1, write, FDWrite, "A", 1);
>> + assert(Result != -1);
>> + }
>> + ~SemaphorePipe() {
>> + if (OwnsFDs) {
>> + close(FDWrite);
>> + close(FDRead);
>> + }
>> + }
>> + const int FDRead;
>> + const int FDWrite;
>> + bool OwnsFDs;
>> +
>> + static llvm::Optional<SemaphorePipe> create() {
>> + int InotifyPollingStopperFDs[2];
>> + if (pipe2(InotifyPollingStopperFDs, O_CLOEXEC) == -1)
>> + return llvm::None;
>> + return SemaphorePipe(InotifyPollingStopperFDs);
>> + }
>> +};
>> +
>> +/// Mutex-protected queue of Events.
>> +class EventQueue {
>> + std::mutex Mtx;
>> + std::condition_variable NonEmpty;
>> + std::queue<DirectoryWatcher::Event> Events;
>> +
>> +public:
>> + void push_back(const DirectoryWatcher::Event::EventKind K,
>> + StringRef Filename) {
>> + {
>> + std::unique_lock<std::mutex> L(Mtx);
>> + Events.emplace(K, Filename);
>> + }
>> + NonEmpty.notify_one();
>> + }
>> +
>> + // Blocks on caller thread and uses codition_variable to wait until there's an
>> + // event to return.
>> + DirectoryWatcher::Event pop_front_blocking() {
>> + std::unique_lock<std::mutex> L(Mtx);
>> + while (true) {
>> + // Since we might have missed all the prior notifications on NonEmpty we
>> + // have to check the queue first (under lock).
>> + if (!Events.empty()) {
>> + DirectoryWatcher::Event Front = Events.front();
>> + Events.pop();
>> + return Front;
>> + }
>> + NonEmpty.wait(L, [this]() { return !Events.empty(); });
>> + }
>> + }
>> +};
>> +
>> +class DirectoryWatcherLinux : public clang::DirectoryWatcher {
>> +public:
>> + DirectoryWatcherLinux(
>> + llvm::StringRef WatchedDirPath,
>> + std::function<void(llvm::ArrayRef<Event>, bool)> Receiver,
>> + bool WaitForInitialSync, int InotifyFD, int InotifyWD,
>> + SemaphorePipe &&InotifyPollingStopSignal);
>> +
>> + ~DirectoryWatcherLinux() override {
>> + StopWork();
>> + InotifyPollingThread.join();
>> + EventsReceivingThread.join();
>> + inotify_rm_watch(InotifyFD, InotifyWD);
>> + llvm::sys::RetryAfterSignal(-1, close, InotifyFD);
>> + }
>> +
>> +private:
>> + const std::string WatchedDirPath;
>> + // inotify file descriptor
>> + int InotifyFD = -1;
>> + // inotify watch descriptor
>> + int InotifyWD = -1;
>> +
>> + EventQueue Queue;
>> +
>> + // Make sure lifetime of Receiver fully contains lifetime of
>> + // EventsReceivingThread.
>> + std::function<void(llvm::ArrayRef<Event>, bool)> Receiver;
>> +
>> + // Consumes inotify events and pushes directory watcher events to the Queue.
>> + void InotifyPollingLoop();
>> + std::thread InotifyPollingThread;
>> + // Using pipe so we can epoll two file descriptors at once - inotify and
>> + // stopping condition.
>> + SemaphorePipe InotifyPollingStopSignal;
>> +
>> + // Does the initial scan of the directory - directly calling Receiver,
>> + // bypassing the Queue. Both InitialScan and EventReceivingLoop use Receiver
>> + // which isn't necessarily thread-safe.
>> + void InitialScan();
>> +
>> + // Processing events from the Queue.
>> + // In case client doesn't want to do the initial scan synchronously
>> + // (WaitForInitialSync=false in ctor) we do the initial scan at the beginning
>> + // of this thread.
>> + std::thread EventsReceivingThread;
>> + // Push event of WatcherGotInvalidated kind to the Queue to stop the loop.
>> + // Both InitialScan and EventReceivingLoop use Receiver which isn't
>> + // necessarily thread-safe.
>> + void EventReceivingLoop();
>> +
>> + // Stops all the async work. Reentrant.
>> + void StopWork() {
>> + Queue.push_back(DirectoryWatcher::Event::EventKind::WatcherGotInvalidated,
>> + "");
>> + InotifyPollingStopSignal.signal();
>> + }
>> +};
>> +
>> +void DirectoryWatcherLinux::InotifyPollingLoop() {
>> + // We want to be able to read ~30 events at once even in the worst case
>> + // (obscenely long filenames).
>> + constexpr size_t EventBufferLength =
>> + 30 * (sizeof(struct inotify_event) + NAME_MAX + 1);
>> + // http://man7.org/linux/man-pages/man7/inotify.7.html
>> + // Some systems cannot read integer variables if they are not
>> + // properly aligned. On other systems, incorrect alignment may
>> + // decrease performance. Hence, the buffer used for reading from
>> + // the inotify file descriptor should have the same alignment as
>> + // struct inotify_event.
>> +
>> + auto ManagedBuffer =
>> + llvm::make_unique<llvm::AlignedCharArray<alignof(struct inotify_event),
>> + EventBufferLength>>();
>> + char *const Buf = ManagedBuffer->buffer;
>> +
>> + const int EpollFD = epoll_create1(EPOLL_CLOEXEC);
>> + if (EpollFD == -1) {
>> + StopWork();
>> + return;
>> + }
>> + auto EpollFDGuard = llvm::make_scope_exit([EpollFD]() { close(EpollFD); });
>> +
>> + struct epoll_event EventSpec;
>> + EventSpec.events = EPOLLIN;
>> + EventSpec.data.fd = InotifyFD;
>> + if (epoll_ctl(EpollFD, EPOLL_CTL_ADD, InotifyFD, &EventSpec) == -1) {
>> + StopWork();
>> + return;
>> + }
>> +
>> + EventSpec.data.fd = InotifyPollingStopSignal.FDRead;
>> + if (epoll_ctl(EpollFD, EPOLL_CTL_ADD, InotifyPollingStopSignal.FDRead,
>> + &EventSpec) == -1) {
>> + StopWork();
>> + return;
>> + }
>> +
>> + std::array<struct epoll_event, 2> EpollEventBuffer;
>> +
>> + while (true) {
>> + const int EpollWaitResult = llvm::sys::RetryAfterSignal(
>> + -1, epoll_wait, EpollFD, EpollEventBuffer.data(),
>> + EpollEventBuffer.size(), /*timeout=*/-1 /*== infinity*/);
>> + if (EpollWaitResult == -1) {
>> + StopWork();
>> + return;
>> + }
>> +
>> + // Multiple epoll_events can be received for a single file descriptor per
>> + // epoll_wait call.
>> + for (const auto &EpollEvent : EpollEventBuffer) {
>> + if (EpollEvent.data.fd == InotifyPollingStopSignal.FDRead) {
>> + StopWork();
>> + return;
>> + }
>> + }
>> +
>> + // epoll_wait() always return either error or >0 events. Since there was no
>> + // event for stopping, it must be an inotify event ready for reading.
>> + ssize_t NumRead = llvm::sys::RetryAfterSignal(-1, read, InotifyFD, Buf,
>> + EventBufferLength);
>> + for (char *P = Buf; P < Buf + NumRead;) {
>> + if (P + sizeof(struct inotify_event) > Buf + NumRead) {
>> + StopWork();
>> + llvm_unreachable("an incomplete inotify_event was read");
>> + return;
>> + }
>> +
>> + struct inotify_event *Event = reinterpret_cast<struct inotify_event *>(P);
>> + P += sizeof(struct inotify_event) + Event->len;
>> +
>> + if (Event->mask & (IN_CREATE | IN_MODIFY | IN_MOVED_TO | IN_DELETE) &&
>> + Event->len <= 0) {
>> + StopWork();
>> + llvm_unreachable("expected a filename from inotify");
>> + return;
>> + }
>> +
>> + if (Event->mask & (IN_CREATE | IN_MOVED_TO | IN_MODIFY)) {
>> + Queue.push_back(DirectoryWatcher::Event::EventKind::Modified,
>> + Event->name);
>> + } else if (Event->mask & (IN_DELETE | IN_MOVED_FROM)) {
>> + Queue.push_back(DirectoryWatcher::Event::EventKind::Removed,
>> + Event->name);
>> + } else if (Event->mask & (IN_DELETE_SELF | IN_MOVE_SELF)) {
>> + Queue.push_back(DirectoryWatcher::Event::EventKind::WatchedDirRemoved,
>> + "");
>> + StopWork();
>> + return;
>> + } else if (Event->mask & IN_IGNORED) {
>> + StopWork();
>> + return;
>> + } else {
>> + StopWork();
>> + llvm_unreachable("Unknown event type.");
>> + return;
>> + }
>> + }
>> + }
>> +}
>> +
>> +void DirectoryWatcherLinux::InitialScan() {
>> + this->Receiver(getAsFileEvents(scanDirectory(WatchedDirPath)),
>> + /*IsInitial=*/true);
>> +}
>> +
>> +void DirectoryWatcherLinux::EventReceivingLoop() {
>> + while (true) {
>> + DirectoryWatcher::Event Event = this->Queue.pop_front_blocking();
>> + this->Receiver(Event, false);
>> + if (Event.Kind ==
>> + DirectoryWatcher::Event::EventKind::WatcherGotInvalidated) {
>> + StopWork();
>> + return;
>> + }
>> + }
>> +}
>> +
>> +DirectoryWatcherLinux::DirectoryWatcherLinux(
>> + StringRef WatchedDirPath,
>> + std::function<void(llvm::ArrayRef<Event>, bool)> Receiver,
>> + bool WaitForInitialSync, int InotifyFD, int InotifyWD,
>> + SemaphorePipe &&InotifyPollingStopSignal)
>> + : WatchedDirPath(WatchedDirPath), InotifyFD(InotifyFD),
>> + InotifyWD(InotifyWD), Receiver(Receiver),
>> + InotifyPollingStopSignal(std::move(InotifyPollingStopSignal)) {
>> +
>> + InotifyPollingThread = std::thread([this]() { InotifyPollingLoop(); });
>> + // We have no guarantees about thread safety of the Receiver which is being
>> + // used in both InitialScan and EventReceivingLoop. We shouldn't run these
>> + // only synchronously.
>> + if (WaitForInitialSync) {
>> + InitialScan();
>> + EventsReceivingThread = std::thread([this]() { EventReceivingLoop(); });
>> + } else {
>> + EventsReceivingThread = std::thread([this]() {
>> + // FIXME: We might want to terminate an async initial scan early in case
>> + // of a failure in EventsReceivingThread.
>> + InitialScan();
>> + EventReceivingLoop();
>> + });
>> + }
>> +}
>> +
>> +} // namespace
>> +
>> +std::unique_ptr<DirectoryWatcher> clang::DirectoryWatcher::create(
>> + StringRef Path,
>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)> Receiver,
>> + bool WaitForInitialSync) {
>> + if (Path.empty())
>> + return nullptr;
>> +
>> + const int InotifyFD = inotify_init1(IN_CLOEXEC);
>> + if (InotifyFD == -1)
>> + return nullptr;
>> +
>> + const int InotifyWD = inotify_add_watch(
>> + InotifyFD, Path.str().c_str(),
>> + IN_CREATE | IN_DELETE | IN_DELETE_SELF | IN_EXCL_UNLINK | IN_MODIFY |
>> + IN_MOVED_FROM | IN_MOVE_SELF | IN_MOVED_TO | IN_ONLYDIR | IN_IGNORED);
>> + if (InotifyWD == -1)
>> + return nullptr;
>> +
>> + auto InotifyPollingStopper = SemaphorePipe::create();
>> +
>> + if (!InotifyPollingStopper)
>> + return nullptr;
>> +
>> + return llvm::make_unique<DirectoryWatcherLinux>(
>> + Path, Receiver, WaitForInitialSync, InotifyFD, InotifyWD,
>> + std::move(*InotifyPollingStopper));
>> +}
>> \ No newline at end of file
>>
>> Added: cfe/trunk/lib/DirectoryWatcher/mac/DirectoryWatcher-mac.cpp
>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/mac/DirectoryWatcher-mac.cpp?rev=365954&view=auto
>> ==============================================================================
>> --- cfe/trunk/lib/DirectoryWatcher/mac/DirectoryWatcher-mac.cpp (added)
>> +++ cfe/trunk/lib/DirectoryWatcher/mac/DirectoryWatcher-mac.cpp Fri Jul 12 13:34:10 2019
>> @@ -0,0 +1,233 @@
>> +//===- DirectoryWatcher-mac.cpp - Mac-platform directory watching ---------===//
>> +//
>> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
>> +// See https://llvm.org/LICENSE.txt for license information.
>> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
>> +//
>> +//===----------------------------------------------------------------------===//
>> +
>> +#include "DirectoryScanner.h"
>> +#include "clang/DirectoryWatcher/DirectoryWatcher.h"
>> +
>> +#include "llvm/ADT/STLExtras.h"
>> +#include "llvm/ADT/StringRef.h"
>> +#include "llvm/Support/Path.h"
>> +#include <CoreServices/CoreServices.h>
>> +
>> +using namespace llvm;
>> +using namespace clang;
>> +
>> +static FSEventStreamRef createFSEventStream(
>> + StringRef Path,
>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)>,
>> + dispatch_queue_t);
>> +static void stopFSEventStream(FSEventStreamRef);
>> +
>> +namespace {
>> +
>> +class DirectoryWatcherMac : public clang::DirectoryWatcher {
>> +public:
>> + DirectoryWatcherMac(
>> + FSEventStreamRef EventStream,
>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)>
>> + Receiver,
>> + llvm::StringRef WatchedDirPath)
>> + : EventStream(EventStream), Receiver(Receiver),
>> + WatchedDirPath(WatchedDirPath) {}
>> +
>> + ~DirectoryWatcherMac() override {
>> + stopFSEventStream(EventStream);
>> + EventStream = nullptr;
>> + // Now it's safe to use Receiver as the only other concurrent use would have
>> + // been in EventStream processing.
>> + Receiver(DirectoryWatcher::Event(
>> + DirectoryWatcher::Event::EventKind::WatcherGotInvalidated, ""),
>> + false);
>> + }
>> +
>> +private:
>> + FSEventStreamRef EventStream;
>> + std::function<void(llvm::ArrayRef<Event>, bool)> Receiver;
>> + const std::string WatchedDirPath;
>> +};
>> +
>> +struct EventStreamContextData {
>> + std::string WatchedPath;
>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)> Receiver;
>> +
>> + EventStreamContextData(
>> + std::string &&WatchedPath,
>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)>
>> + Receiver)
>> + : WatchedPath(std::move(WatchedPath)), Receiver(Receiver) {}
>> +
>> + // Needed for FSEvents
>> + static void dispose(const void *ctx) {
>> + delete static_cast<const EventStreamContextData *>(ctx);
>> + }
>> +};
>> +} // namespace
>> +
>> +constexpr const FSEventStreamEventFlags StreamInvalidatingFlags =
>> + kFSEventStreamEventFlagUserDropped | kFSEventStreamEventFlagKernelDropped |
>> + kFSEventStreamEventFlagMustScanSubDirs;
>> +
>> +constexpr const FSEventStreamEventFlags ModifyingFileEvents =
>> + kFSEventStreamEventFlagItemCreated | kFSEventStreamEventFlagItemRenamed |
>> + kFSEventStreamEventFlagItemModified;
>> +
>> +static void eventStreamCallback(ConstFSEventStreamRef Stream,
>> + void *ClientCallBackInfo, size_t NumEvents,
>> + void *EventPaths,
>> + const FSEventStreamEventFlags EventFlags[],
>> + const FSEventStreamEventId EventIds[]) {
>> + auto *ctx = static_cast<EventStreamContextData *>(ClientCallBackInfo);
>> +
>> + std::vector<DirectoryWatcher::Event> Events;
>> + for (size_t i = 0; i < NumEvents; ++i) {
>> + StringRef Path = ((const char **)EventPaths)[i];
>> + const FSEventStreamEventFlags Flags = EventFlags[i];
>> +
>> + if (Flags & StreamInvalidatingFlags) {
>> + Events.emplace_back(DirectoryWatcher::Event{
>> + DirectoryWatcher::Event::EventKind::WatcherGotInvalidated, ""});
>> + break;
>> + } else if (!(Flags & kFSEventStreamEventFlagItemIsFile)) {
>> + // Subdirectories aren't supported - if some directory got removed it
>> + // must've been the watched directory itself.
>> + if ((Flags & kFSEventStreamEventFlagItemRemoved) &&
>> + Path == ctx->WatchedPath) {
>> + Events.emplace_back(DirectoryWatcher::Event{
>> + DirectoryWatcher::Event::EventKind::WatchedDirRemoved, ""});
>> + Events.emplace_back(DirectoryWatcher::Event{
>> + DirectoryWatcher::Event::EventKind::WatcherGotInvalidated, ""});
>> + break;
>> + }
>> + // No support for subdirectories - just ignore everything.
>> + continue;
>> + } else if (Flags & kFSEventStreamEventFlagItemRemoved) {
>> + Events.emplace_back(DirectoryWatcher::Event::EventKind::Removed,
>> + llvm::sys::path::filename(Path));
>> + continue;
>> + } else if (Flags & ModifyingFileEvents) {
>> + if (!getFileStatus(Path).hasValue()) {
>> + Events.emplace_back(DirectoryWatcher::Event::EventKind::Removed,
>> + llvm::sys::path::filename(Path));
>> + } else {
>> + Events.emplace_back(DirectoryWatcher::Event::EventKind::Modified,
>> + llvm::sys::path::filename(Path));
>> + }
>> + continue;
>> + }
>> +
>> + // default
>> + Events.emplace_back(DirectoryWatcher::Event{
>> + DirectoryWatcher::Event::EventKind::WatcherGotInvalidated, ""});
>> + llvm_unreachable("Unknown FSEvent type.");
>> + }
>> +
>> + if (!Events.empty()) {
>> + ctx->Receiver(Events, /*IsInitial=*/false);
>> + }
>> +}
>> +
>> +FSEventStreamRef createFSEventStream(
>> + StringRef Path,
>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)> Receiver,
>> + dispatch_queue_t Queue) {
>> + if (Path.empty())
>> + return nullptr;
>> +
>> + CFMutableArrayRef PathsToWatch = [&]() {
>> + CFMutableArrayRef PathsToWatch =
>> + CFArrayCreateMutable(nullptr, 0, &kCFTypeArrayCallBacks);
>> + CFStringRef CfPathStr =
>> + CFStringCreateWithBytes(nullptr, (const UInt8 *)Path.data(),
>> + Path.size(), kCFStringEncodingUTF8, false);
>> + CFArrayAppendValue(PathsToWatch, CfPathStr);
>> + CFRelease(CfPathStr);
>> + return PathsToWatch;
>> + }();
>> +
>> + FSEventStreamContext Context = [&]() {
>> + std::string RealPath;
>> + {
>> + SmallString<128> Storage;
>> + StringRef P = llvm::Twine(Path).toNullTerminatedStringRef(Storage);
>> + char Buffer[PATH_MAX];
>> + if (::realpath(P.begin(), Buffer) != nullptr)
>> + RealPath = Buffer;
>> + else
>> + RealPath = Path;
>> + }
>> +
>> + FSEventStreamContext Context;
>> + Context.version = 0;
>> + Context.info = new EventStreamContextData(std::move(RealPath), Receiver);
>> + Context.retain = nullptr;
>> + Context.release = EventStreamContextData::dispose;
>> + Context.copyDescription = nullptr;
>> + return Context;
>> + }();
>> +
>> + FSEventStreamRef Result = FSEventStreamCreate(
>> + nullptr, eventStreamCallback, &Context, PathsToWatch,
>> + kFSEventStreamEventIdSinceNow, /* latency in seconds */ 0.0,
>> + kFSEventStreamCreateFlagFileEvents | kFSEventStreamCreateFlagNoDefer);
>> + CFRelease(PathsToWatch);
>> +
>> + return Result;
>> +}
>> +
>> +void stopFSEventStream(FSEventStreamRef EventStream) {
>> + if (!EventStream)
>> + return;
>> + FSEventStreamStop(EventStream);
>> + FSEventStreamInvalidate(EventStream);
>> + FSEventStreamRelease(EventStream);
>> +}
>> +
>> +std::unique_ptr<DirectoryWatcher> clang::DirectoryWatcher::create(
>> + StringRef Path,
>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)> Receiver,
>> + bool WaitForInitialSync) {
>> + dispatch_queue_t Queue =
>> + dispatch_queue_create("DirectoryWatcher", DISPATCH_QUEUE_SERIAL);
>> +
>> + if (Path.empty())
>> + return nullptr;
>> +
>> + auto EventStream = createFSEventStream(Path, Receiver, Queue);
>> + if (!EventStream) {
>> + return nullptr;
>> + }
>> +
>> + std::unique_ptr<DirectoryWatcher> Result =
>> + llvm::make_unique<DirectoryWatcherMac>(EventStream, Receiver, Path);
>> +
>> + // We need to copy the data so the lifetime is ok after a const copy is made
>> + // for the block.
>> + const std::string CopiedPath = Path;
>> +
>> + auto InitWork = ^{
>> + // We need to start watching the directory before we start scanning in order
>> + // to not miss any event. By dispatching this on the same serial Queue as
>> + // the FSEvents will be handled we manage to start watching BEFORE the
>> + // inital scan and handling events ONLY AFTER the scan finishes.
>> + FSEventStreamSetDispatchQueue(EventStream, Queue);
>> + FSEventStreamStart(EventStream);
>> + // We need to decrement the ref count for Queue as initialize() will return
>> + // and FSEvents has incremented it. Since we have to wait for FSEvents to
>> + // take ownership it's the easiest to do it here rather than main thread.
>> + dispatch_release(Queue);
>> + Receiver(getAsFileEvents(scanDirectory(CopiedPath)), /*IsInitial=*/true);
>> + };
>> +
>> + if (WaitForInitialSync) {
>> + dispatch_sync(Queue, InitWork);
>> + } else {
>> + dispatch_async(Queue, InitWork);
>> + }
>> +
>> + return Result;
>> +}
>>
>> Modified: cfe/trunk/unittests/CMakeLists.txt
>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/unittests/CMakeLists.txt?rev=365954&r1=365953&r2=365954&view=diff
>> ==============================================================================
>> --- cfe/trunk/unittests/CMakeLists.txt (original)
>> +++ cfe/trunk/unittests/CMakeLists.txt Fri Jul 12 13:34:10 2019
>> @@ -30,6 +30,7 @@ add_subdirectory(CodeGen)
>> if(NOT WIN32 AND CLANG_TOOL_LIBCLANG_BUILD)
>> add_subdirectory(libclang)
>> endif()
>> +add_subdirectory(DirectoryWatcher)
>> add_subdirectory(Rename)
>> add_subdirectory(Index)
>> add_subdirectory(Serialization)
>>
>> Added: cfe/trunk/unittests/DirectoryWatcher/CMakeLists.txt
>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/unittests/DirectoryWatcher/CMakeLists.txt?rev=365954&view=auto
>> ==============================================================================
>> --- cfe/trunk/unittests/DirectoryWatcher/CMakeLists.txt (added)
>> +++ cfe/trunk/unittests/DirectoryWatcher/CMakeLists.txt Fri Jul 12 13:34:10 2019
>> @@ -0,0 +1,17 @@
>> +if(APPLE OR CMAKE_SYSTEM_NAME MATCHES "Linux")
>> +
>> + set(LLVM_LINK_COMPONENTS
>> + Support
>> + )
>> +
>> + add_clang_unittest(DirectoryWatcherTests
>> + DirectoryWatcherTest.cpp
>> + )
>> +
>> + target_link_libraries(DirectoryWatcherTests
>> + PRIVATE
>> + clangDirectoryWatcher
>> + clangBasic
>> + )
>> +
>> +endif()
>> \ No newline at end of file
>>
>> Added: cfe/trunk/unittests/DirectoryWatcher/DirectoryWatcherTest.cpp
>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/unittests/DirectoryWatcher/DirectoryWatcherTest.cpp?rev=365954&view=auto
>> ==============================================================================
>> --- cfe/trunk/unittests/DirectoryWatcher/DirectoryWatcherTest.cpp (added)
>> +++ cfe/trunk/unittests/DirectoryWatcher/DirectoryWatcherTest.cpp Fri Jul 12 13:34:10 2019
>> @@ -0,0 +1,426 @@
>> +//===- unittests/DirectoryWatcher/DirectoryWatcherTest.cpp ----------------===//
>> +//
>> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
>> +// See https://llvm.org/LICENSE.txt for license information.
>> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
>> +//
>> +//===----------------------------------------------------------------------===//
>> +
>> +#include "clang/DirectoryWatcher/DirectoryWatcher.h"
>> +#include "llvm/Support/FileSystem.h"
>> +#include "llvm/Support/Mutex.h"
>> +#include "llvm/Support/Path.h"
>> +#include "llvm/Support/raw_ostream.h"
>> +#include "gtest/gtest.h"
>> +#include <condition_variable>
>> +#include <future>
>> +#include <mutex>
>> +#include <thread>
>> +
>> +using namespace llvm;
>> +using namespace llvm::sys;
>> +using namespace llvm::sys::fs;
>> +using namespace clang;
>> +
>> +namespace clang {
>> +static bool operator==(const DirectoryWatcher::Event &lhs,
>> + const DirectoryWatcher::Event &rhs) {
>> + return lhs.Filename == rhs.Filename &&
>> + static_cast<int>(lhs.Kind) == static_cast<int>(rhs.Kind);
>> +}
>> +} // namespace clang
>> +
>> +namespace {
>> +
>> +struct DirectoryWatcherTestFixture {
>> + std::string TestRootDir;
>> + std::string TestWatchedDir;
>> +
>> + DirectoryWatcherTestFixture() {
>> + SmallString<128> pathBuf;
>> + std::error_code UniqDirRes = createUniqueDirectory("dirwatcher", pathBuf);
>> + assert(!UniqDirRes);
>> + TestRootDir = pathBuf.str();
>> + path::append(pathBuf, "watch");
>> + TestWatchedDir = pathBuf.str();
>> + std::error_code CreateDirRes = create_directory(TestWatchedDir, false);
>> + assert(!CreateDirRes);
>> + }
>> +
>> + ~DirectoryWatcherTestFixture() { remove_directories(TestRootDir); }
>> +
>> + SmallString<128> getPathInWatched(const std::string &testFile) {
>> + SmallString<128> pathBuf;
>> + pathBuf = TestWatchedDir;
>> + path::append(pathBuf, testFile);
>> + return pathBuf;
>> + }
>> +
>> + void addFile(const std::string &testFile) {
>> + Expected<file_t> ft = openNativeFileForWrite(getPathInWatched(testFile),
>> + CD_CreateNew, OF_None);
>> + if (ft) {
>> + closeFile(*ft);
>> + } else {
>> + llvm::errs() << llvm::toString(ft.takeError()) << "\n";
>> + llvm::errs() << getPathInWatched(testFile) << "\n";
>> + llvm_unreachable("Couldn't create test file.");
>> + }
>> + }
>> +
>> + void deleteFile(const std::string &testFile) {
>> + std::error_code EC =
>> + remove(getPathInWatched(testFile), /*IgnoreNonExisting=*/false);
>> + ASSERT_FALSE(EC);
>> + }
>> +};
>> +
>> +std::string eventKindToString(const DirectoryWatcher::Event::EventKind K) {
>> + switch (K) {
>> + case DirectoryWatcher::Event::EventKind::Removed:
>> + return "Removed";
>> + case DirectoryWatcher::Event::EventKind::Modified:
>> + return "Modified";
>> + case DirectoryWatcher::Event::EventKind::WatchedDirRemoved:
>> + return "WatchedDirRemoved";
>> + case DirectoryWatcher::Event::EventKind::WatcherGotInvalidated:
>> + return "WatcherGotInvalidated";
>> + }
>> + llvm_unreachable("unknown event kind");
>> +}
>> +
>> +struct VerifyingConsumer {
>> + std::vector<DirectoryWatcher::Event> ExpectedInitial;
>> + std::vector<DirectoryWatcher::Event> ExpectedNonInitial;
>> + std::vector<DirectoryWatcher::Event> OptionalNonInitial;
>> + std::vector<DirectoryWatcher::Event> UnexpectedInitial;
>> + std::vector<DirectoryWatcher::Event> UnexpectedNonInitial;
>> + std::mutex Mtx;
>> + std::condition_variable ResultIsReady;
>> +
>> + VerifyingConsumer(
>> + const std::vector<DirectoryWatcher::Event> &ExpectedInitial,
>> + const std::vector<DirectoryWatcher::Event> &ExpectedNonInitial,
>> + const std::vector<DirectoryWatcher::Event> &OptionalNonInitial = {})
>> + : ExpectedInitial(ExpectedInitial),
>> + ExpectedNonInitial(ExpectedNonInitial),
>> + OptionalNonInitial(OptionalNonInitial) {}
>> +
>> + // This method is used by DirectoryWatcher.
>> + void consume(DirectoryWatcher::Event E, bool IsInitial) {
>> + if (IsInitial)
>> + consumeInitial(E);
>> + else
>> + consumeNonInitial(E);
>> + }
>> +
>> + void consumeInitial(DirectoryWatcher::Event E) {
>> + std::unique_lock<std::mutex> L(Mtx);
>> + auto It = std::find(ExpectedInitial.begin(), ExpectedInitial.end(), E);
>> + if (It == ExpectedInitial.end()) {
>> + UnexpectedInitial.push_back(E);
>> + } else {
>> + ExpectedInitial.erase(It);
>> + }
>> + if (result())
>> + ResultIsReady.notify_one();
>> + }
>> +
>> + void consumeNonInitial(DirectoryWatcher::Event E) {
>> + std::unique_lock<std::mutex> L(Mtx);
>> + auto It =
>> + std::find(ExpectedNonInitial.begin(), ExpectedNonInitial.end(), E);
>> + if (It == ExpectedNonInitial.end()) {
>> + auto OptIt =
>> + std::find(OptionalNonInitial.begin(), OptionalNonInitial.end(), E);
>> + if (OptIt != OptionalNonInitial.end()) {
>> + OptionalNonInitial.erase(OptIt);
>> + } else {
>> + UnexpectedNonInitial.push_back(E);
>> + }
>> + } else {
>> + ExpectedNonInitial.erase(It);
>> + }
>> + if (result())
>> + ResultIsReady.notify_one();
>> + }
>> +
>> + // This method is used by DirectoryWatcher.
>> + void consume(llvm::ArrayRef<DirectoryWatcher::Event> Es, bool IsInitial) {
>> + for (const auto &E : Es)
>> + consume(E, IsInitial);
>> + }
>> +
>> + // Not locking - caller has to lock Mtx.
>> + llvm::Optional<bool> result() const {
>> + if (ExpectedInitial.empty() && ExpectedNonInitial.empty() &&
>> + UnexpectedInitial.empty() && UnexpectedNonInitial.empty())
>> + return true;
>> + if (!UnexpectedInitial.empty() || !UnexpectedNonInitial.empty())
>> + return false;
>> + return llvm::None;
>> + }
>> +
>> + // This method is used by tests.
>> + // \returns true on success
>> + bool blockUntilResult() {
>> + std::unique_lock<std::mutex> L(Mtx);
>> + while (true) {
>> + if (result())
>> + return *result();
>> +
>> + ResultIsReady.wait(L, [this]() { return result().hasValue(); });
>> + }
>> + return false; // Just to make compiler happy.
>> + }
>> +
>> + void printUnmetExpectations(llvm::raw_ostream &OS) {
>> + if (!ExpectedInitial.empty()) {
>> + OS << "Expected but not seen initial events: \n";
>> + for (const auto &E : ExpectedInitial) {
>> + OS << eventKindToString(E.Kind) << " " << E.Filename << "\n";
>> + }
>> + }
>> + if (!ExpectedNonInitial.empty()) {
>> + OS << "Expected but not seen non-initial events: \n";
>> + for (const auto &E : ExpectedNonInitial) {
>> + OS << eventKindToString(E.Kind) << " " << E.Filename << "\n";
>> + }
>> + }
>> + if (!UnexpectedInitial.empty()) {
>> + OS << "Unexpected initial events seen: \n";
>> + for (const auto &E : UnexpectedInitial) {
>> + OS << eventKindToString(E.Kind) << " " << E.Filename << "\n";
>> + }
>> + }
>> + if (!UnexpectedNonInitial.empty()) {
>> + OS << "Unexpected non-initial events seen: \n";
>> + for (const auto &E : UnexpectedNonInitial) {
>> + OS << eventKindToString(E.Kind) << " " << E.Filename << "\n";
>> + }
>> + }
>> + }
>> +};
>> +
>> +void checkEventualResultWithTimeout(VerifyingConsumer &TestConsumer) {
>> + std::packaged_task<int(void)> task(
>> + [&TestConsumer]() { return TestConsumer.blockUntilResult(); });
>> + std::future<int> WaitForExpectedStateResult = task.get_future();
>> + std::thread worker(std::move(task));
>> + worker.detach();
>> +
>> + EXPECT_TRUE(WaitForExpectedStateResult.wait_for(std::chrono::seconds(3)) ==
>> + std::future_status::ready)
>> + << "The expected result state wasn't reached before the time-out.";
>> + EXPECT_TRUE(TestConsumer.result().hasValue());
>> + if (TestConsumer.result().hasValue()) {
>> + EXPECT_TRUE(*TestConsumer.result());
>> + }
>> + if ((TestConsumer.result().hasValue() && !TestConsumer.result().getValue()) ||
>> + !TestConsumer.result().hasValue())
>> + TestConsumer.printUnmetExpectations(llvm::outs());
>> +}
>> +
>> +} // namespace
>> +
>> +TEST(DirectoryWatcherTest, InitialScanSync) {
>> + DirectoryWatcherTestFixture fixture;
>> +
>> + fixture.addFile("a");
>> + fixture.addFile("b");
>> + fixture.addFile("c");
>> +
>> + VerifyingConsumer TestConsumer{
>> + {{DirectoryWatcher::Event::EventKind::Modified, "a"},
>> + {DirectoryWatcher::Event::EventKind::Modified, "b"},
>> + {DirectoryWatcher::Event::EventKind::Modified, "c"}},
>> + {}};
>> +
>> + auto DW = DirectoryWatcher::create(
>> + fixture.TestWatchedDir,
>> + [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
>> + bool IsInitial) {
>> + TestConsumer.consume(Events, IsInitial);
>> + },
>> + /*waitForInitialSync=*/true);
>> +
>> + checkEventualResultWithTimeout(TestConsumer);
>> +}
>> +
>> +TEST(DirectoryWatcherTest, InitialScanAsync) {
>> + DirectoryWatcherTestFixture fixture;
>> +
>> + fixture.addFile("a");
>> + fixture.addFile("b");
>> + fixture.addFile("c");
>> +
>> + VerifyingConsumer TestConsumer{
>> + {{DirectoryWatcher::Event::EventKind::Modified, "a"},
>> + {DirectoryWatcher::Event::EventKind::Modified, "b"},
>> + {DirectoryWatcher::Event::EventKind::Modified, "c"}},
>> + {}};
>> +
>> + auto DW = DirectoryWatcher::create(
>> + fixture.TestWatchedDir,
>> + [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
>> + bool IsInitial) {
>> + TestConsumer.consume(Events, IsInitial);
>> + },
>> + /*waitForInitialSync=*/false);
>> +
>> + checkEventualResultWithTimeout(TestConsumer);
>> +}
>> +
>> +TEST(DirectoryWatcherTest, AddFiles) {
>> + DirectoryWatcherTestFixture fixture;
>> +
>> + VerifyingConsumer TestConsumer{
>> + {},
>> + {{DirectoryWatcher::Event::EventKind::Modified, "a"},
>> + {DirectoryWatcher::Event::EventKind::Modified, "b"},
>> + {DirectoryWatcher::Event::EventKind::Modified, "c"}}};
>> +
>> + auto DW = DirectoryWatcher::create(
>> + fixture.TestWatchedDir,
>> + [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
>> + bool IsInitial) {
>> + TestConsumer.consume(Events, IsInitial);
>> + },
>> + /*waitForInitialSync=*/true);
>> +
>> + fixture.addFile("a");
>> + fixture.addFile("b");
>> + fixture.addFile("c");
>> +
>> + checkEventualResultWithTimeout(TestConsumer);
>> +}
>> +
>> +TEST(DirectoryWatcherTest, ModifyFile) {
>> + DirectoryWatcherTestFixture fixture;
>> +
>> + fixture.addFile("a");
>> +
>> + VerifyingConsumer TestConsumer{
>> + {{DirectoryWatcher::Event::EventKind::Modified, "a"}},
>> + {{DirectoryWatcher::Event::EventKind::Modified, "a"}}};
>> +
>> + auto DW = DirectoryWatcher::create(
>> + fixture.TestWatchedDir,
>> + [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
>> + bool IsInitial) {
>> + TestConsumer.consume(Events, IsInitial);
>> + },
>> + /*waitForInitialSync=*/true);
>> +
>> + // modify the file
>> + {
>> + std::error_code error;
>> + llvm::raw_fd_ostream bStream(fixture.getPathInWatched("a"), error,
>> + CD_OpenExisting);
>> + assert(!error);
>> + bStream << "foo";
>> + }
>> +
>> + checkEventualResultWithTimeout(TestConsumer);
>> +}
>> +
>> +TEST(DirectoryWatcherTest, DeleteFile) {
>> + DirectoryWatcherTestFixture fixture;
>> +
>> + fixture.addFile("a");
>> +
>> + VerifyingConsumer TestConsumer{
>> + {{DirectoryWatcher::Event::EventKind::Modified, "a"}},
>> + {{DirectoryWatcher::Event::EventKind::Removed, "a"}}};
>> +
>> + auto DW = DirectoryWatcher::create(
>> + fixture.TestWatchedDir,
>> + [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
>> + bool IsInitial) {
>> + TestConsumer.consume(Events, IsInitial);
>> + },
>> + /*waitForInitialSync=*/true);
>> +
>> + fixture.deleteFile("a");
>> +
>> + checkEventualResultWithTimeout(TestConsumer);
>> +}
>> +
>> +TEST(DirectoryWatcherTest, DeleteWatchedDir) {
>> + DirectoryWatcherTestFixture fixture;
>> +
>> + VerifyingConsumer TestConsumer{
>> + {},
>> + {{DirectoryWatcher::Event::EventKind::WatchedDirRemoved, ""},
>> + {DirectoryWatcher::Event::EventKind::WatcherGotInvalidated, ""}}};
>> +
>> + auto DW = DirectoryWatcher::create(
>> + fixture.TestWatchedDir,
>> + [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
>> + bool IsInitial) {
>> + TestConsumer.consume(Events, IsInitial);
>> + },
>> + /*waitForInitialSync=*/true);
>> +
>> + remove_directories(fixture.TestWatchedDir);
>> +
>> + checkEventualResultWithTimeout(TestConsumer);
>> +}
>> +
>> +TEST(DirectoryWatcherTest, InvalidatedWatcher) {
>> + DirectoryWatcherTestFixture fixture;
>> +
>> + VerifyingConsumer TestConsumer{
>> + {}, {{DirectoryWatcher::Event::EventKind::WatcherGotInvalidated, ""}}};
>> +
>> + {
>> + auto DW = DirectoryWatcher::create(
>> + fixture.TestWatchedDir,
>> + [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
>> + bool IsInitial) {
>> + TestConsumer.consume(Events, IsInitial);
>> + },
>> + /*waitForInitialSync=*/true);
>> + } // DW is destructed here.
>> +
>> + checkEventualResultWithTimeout(TestConsumer);
>> +}
>> +
>> +TEST(DirectoryWatcherTest, ChangeMetadata) {
>> + DirectoryWatcherTestFixture fixture;
>> + fixture.addFile("a");
>> +
>> + VerifyingConsumer TestConsumer{
>> + {{DirectoryWatcher::Event::EventKind::Modified, "a"}},
>> + // We don't expect any notification for file having access file changed.
>> + {},
>> + // Given the timing we are ok with receiving the duplicate event.
>> + {{DirectoryWatcher::Event::EventKind::Modified, "a"}}};
>> +
>> + auto DW = DirectoryWatcher::create(
>> + fixture.TestWatchedDir,
>> + [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
>> + bool IsInitial) {
>> + TestConsumer.consume(Events, IsInitial);
>> + },
>> + /*waitForInitialSync=*/true);
>> +
>> + { // Change access and modification time of file a.
>> + Expected<file_t> HopefullyTheFD = llvm::sys::fs::openNativeFileForWrite(
>> + fixture.getPathInWatched("a"), CD_OpenExisting, OF_None);
>> + if (!HopefullyTheFD) {
>> + llvm::outs() << HopefullyTheFD.takeError();
>> + }
>> +
>> + const int FD = HopefullyTheFD.get();
>> + const TimePoint<> NewTimePt =
>> + std::chrono::system_clock::now() - std::chrono::minutes(1);
>> +
>> + std::error_code setTimeRes =
>> + llvm::sys::fs::setLastAccessAndModificationTime(FD, NewTimePt,
>> + NewTimePt);
>> + assert(!setTimeRes);
>> + }
>> +
>> + checkEventualResultWithTimeout(TestConsumer);
>> +}
>>
>>
>> _______________________________________________
>> cfe-commits mailing list
>> cfe-commits at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20190715/644298f0/attachment-0001.html>
More information about the cfe-commits
mailing list