r365954 - Reland [clang] DirectoryWatcher
Mikael Holmén via cfe-commits
cfe-commits at lists.llvm.org
Mon Jul 15 23:29:06 PDT 2019
Hi,
On 2019-07-15 20:26, Jan Korous wrote:
> Hi Mikael,
>
> Thanks for the notice.
>
> How about this?
> https://reviews.llvm.org/D64764
>
Yes that works for me!
Thanks!
Mikael
> Jan
>
>> On Jul 14, 2019, at 11:45 PM, Mikael Holmén
>> <mikael.holmen at ericsson.com <mailto:mikael.holmen at ericsson.com>> wrote:
>>
>> Hi Jan,
>>
>>> +elseif(CMAKE_SYSTEM_NAME MATCHES "Linux")
>>> + check_include_files("sys/inotify.h" HAVE_INOTIFY)
>>> + if(HAVE_INOTIFY)
>>> + list(APPEND DIRECTORY_WATCHER_SOURCES
>> linux/DirectoryWatcher-linux.cpp)
>>> + find_package(Threads REQUIRED)
>>> + endif()
>>
>> I don't think the above is enough, I think the version needs to be
>> checked some way as well to make sure we find an inotify.h that really
>> defines IN_EXCL_UNLINK.
>>
>> We have RHEL 6.10 buildbots with glibc 2.12.2 and kernel 2.6.32 where
>> I see
>>
>> -- Looking for include file sys/inotify.h
>> -- Looking for include file sys/inotify.h - found
>>
>> but then get
>>
>> /repo/uabelho/master/clang/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp:335:48:
>>
>> error: 'IN_EXCL_UNLINK' was not declared in this scope
>> IN_CREATE | IN_DELETE | IN_DELETE_SELF | IN_EXCL_UNLINK |
>> IN_MODIFY |
>> ^~~~~~~~~~~~~~
>>
>> Regards,
>> Mikael
>>
>> On 2019-07-12 22:34, Jan Korous via cfe-commits wrote:
>>> Author: jkorous
>>> Date: Fri Jul 12 13:34:10 2019
>>> New Revision: 365954
>>>
>>> URL: http://llvm.org/viewvc/llvm-project?rev=365954&view=rev
>>> Log:
>>> Reland [clang] DirectoryWatcher
>>>
>>> This reverts commit f561227d133224d2d6a5a016abe4be051fa75501.
>>>
>>> - DirectoryWatcher
>>> - Fix the build for platforms that don't have DW implementated.
>>> - Fix the threading dependencies (thanks to compnerd).
>>>
>>> Added:
>>> cfe/trunk/include/clang/DirectoryWatcher/
>>> cfe/trunk/include/clang/DirectoryWatcher/DirectoryWatcher.h
>>> cfe/trunk/lib/DirectoryWatcher/
>>> cfe/trunk/lib/DirectoryWatcher/CMakeLists.txt
>>> cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.cpp
>>> cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.h
>>> cfe/trunk/lib/DirectoryWatcher/default/
>>> cfe/trunk/lib/DirectoryWatcher/default/DirectoryWatcher-not-implemented.cpp
>>> cfe/trunk/lib/DirectoryWatcher/linux/
>>> cfe/trunk/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp
>>> cfe/trunk/lib/DirectoryWatcher/mac/
>>> cfe/trunk/lib/DirectoryWatcher/mac/DirectoryWatcher-mac.cpp
>>> cfe/trunk/unittests/DirectoryWatcher/
>>> cfe/trunk/unittests/DirectoryWatcher/CMakeLists.txt
>>> cfe/trunk/unittests/DirectoryWatcher/DirectoryWatcherTest.cpp
>>> Modified:
>>> cfe/trunk/lib/CMakeLists.txt
>>> cfe/trunk/unittests/CMakeLists.txt
>>>
>>> Added: cfe/trunk/include/clang/DirectoryWatcher/DirectoryWatcher.h
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/DirectoryWatcher/DirectoryWatcher.h?rev=365954&view=auto
>>> ==============================================================================
>>> --- cfe/trunk/include/clang/DirectoryWatcher/DirectoryWatcher.h (added)
>>> +++ cfe/trunk/include/clang/DirectoryWatcher/DirectoryWatcher.h Fri
>>> Jul 12 13:34:10 2019
>>> @@ -0,0 +1,123 @@
>>> +//===- DirectoryWatcher.h - Listens for directory file changes --*-
>>> C++ -*-===//
>>> +//
>>> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM
>>> Exceptions.
>>> +// See https://llvm.org/LICENSE.txt for license information.
>>> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
>>> +//
>>> +//===----------------------------------------------------------------------===//
>>> +
>>> +#ifndef LLVM_CLANG_DIRECTORYWATCHER_DIRECTORYWATCHER_H
>>> +#define LLVM_CLANG_DIRECTORYWATCHER_DIRECTORYWATCHER_H
>>> +
>>> +#include "llvm/ADT/ArrayRef.h"
>>> +#include "llvm/ADT/StringRef.h"
>>> +#include <functional>
>>> +#include <memory>
>>> +#include <string>
>>> +
>>> +namespace clang {
>>> +/// Provides notifications for file changes in a directory.
>>> +///
>>> +/// Invokes client-provided function on every filesystem event in
>>> the watched
>>> +/// directory. Initially the the watched directory is scanned and
>>> for every file
>>> +/// found, an event is synthesized as if the file was added.
>>> +///
>>> +/// This is not a general purpose directory monitoring tool - list of
>>> +/// limitations follows.
>>> +///
>>> +/// Only flat directories with no subdirectories are supported. In case
>>> +/// subdirectories are present the behavior is unspecified - events
>>> *might* be
>>> +/// passed to Receiver on macOS (due to FSEvents being used) while they
>>> +/// *probably* won't be passed on Linux (due to inotify being used).
>>> +///
>>> +/// Known potential inconsistencies
>>> +/// - For files that are deleted befor the initial scan processed
>>> them, clients
>>> +/// might receive Removed notification without any prior Added
>>> notification.
>>> +/// - Multiple notifications might be produced when a file is added
>>> to the
>>> +/// watched directory during the initial scan. We are choosing the
>>> lesser evil
>>> +/// here as the only known alternative strategy would be to
>>> invalidate the
>>> +/// watcher instance and force user to create a new one whenever
>>> filesystem
>>> +/// event occurs during the initial scan but that would introduce
>>> continuous
>>> +/// restarting failure mode (watched directory is not always "owned"
>>> by the same
>>> +/// process that is consuming it). Since existing clients can handle
>>> duplicate
>>> +/// events well, we decided for simplicity.
>>> +///
>>> +/// Notifications are provided only for changes done through local
>>> user-space
>>> +/// filesystem interface. Specifically, it's unspecified if
>>> notification would
>>> +/// be provided in case of a:
>>> +/// - a file mmap-ed and changed
>>> +/// - a file changed via remote (NFS) or virtual (/proc) FS access
>>> to monitored
>>> +/// directory
>>> +/// - another filesystem mounted to the watched directory
>>> +///
>>> +/// No support for LLVM VFS.
>>> +///
>>> +/// It is unspecified whether notifications for files being deleted
>>> are sent in
>>> +/// case the whole watched directory is sent.
>>> +///
>>> +/// Directories containing "too many" files and/or receiving events "too
>>> +/// frequently" are not supported - if the initial scan can't be
>>> finished before
>>> +/// the watcher instance gets invalidated (see
>>> WatcherGotInvalidated) there's no
>>> +/// good error handling strategy - the only option for client is to
>>> destroy the
>>> +/// watcher, restart watching with new instance and hope it won't
>>> repeat.
>>> +class DirectoryWatcher {
>>> +public:
>>> + struct Event {
>>> + enum class EventKind {
>>> + Removed,
>>> + /// Content of a file was modified.
>>> + Modified,
>>> + /// The watched directory got deleted.
>>> + WatchedDirRemoved,
>>> + /// The DirectoryWatcher that originated this event is no
>>> longer valid and
>>> + /// its behavior is unspecified.
>>> + ///
>>> + /// The prime case is kernel signalling to OS-specific
>>> implementation of
>>> + /// DirectoryWatcher some resource limit being hit.
>>> + /// *Usually* kernel starts dropping or squashing events
>>> together after
>>> + /// that and so would DirectoryWatcher. This means that *some*
>>> events
>>> + /// might still be passed to Receiver but this behavior is
>>> unspecified.
>>> + ///
>>> + /// Another case is after the watched directory itself is deleted.
>>> + /// WatcherGotInvalidated will be received at least once during
>>> + /// DirectoryWatcher instance lifetime - when handling errors
>>> this is done
>>> + /// on best effort basis, when an instance is being destroyed
>>> then this is
>>> + /// guaranteed.
>>> + ///
>>> + /// The only proper response to this kind of event is to
>>> destruct the
>>> + /// originating DirectoryWatcher instance and create a new one.
>>> + WatcherGotInvalidated
>>> + };
>>> +
>>> + EventKind Kind;
>>> + /// Filename that this event is related to or an empty string in
>>> + /// case this event is related to the watched directory itself.
>>> + std::string Filename;
>>> +
>>> + Event(EventKind Kind, llvm::StringRef Filename)
>>> + : Kind(Kind), Filename(Filename) {}
>>> + };
>>> +
>>> + /// Returns nullptr if \param Path doesn't exist.
>>> + /// Returns nullptr if \param Path isn't a directory.
>>> + /// Returns nullptr if OS kernel API told us we can't start
>>> watching. In such
>>> + /// case it's unclear whether just retrying has any chance to
>>> succeeed.
>>> + static std::unique_ptr<DirectoryWatcher>
>>> + create(llvm::StringRef Path,
>>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>
>>> Events,
>>> + bool IsInitial)>
>>> + Receiver,
>>> + bool WaitForInitialSync);
>>> +
>>> + virtual ~DirectoryWatcher() = default;
>>> + DirectoryWatcher(const DirectoryWatcher &) = delete;
>>> + DirectoryWatcher &operator=(const DirectoryWatcher &) = delete;
>>> + DirectoryWatcher(DirectoryWatcher &&) = default;
>>> +
>>> +protected:
>>> + DirectoryWatcher() = default;
>>> +};
>>> +
>>> +} // namespace clang
>>> +
>>> +#endif // LLVM_CLANG_DIRECTORYWATCHER_DIRECTORYWATCHER_H
>>>
>>> Modified: cfe/trunk/lib/CMakeLists.txt
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/CMakeLists.txt?rev=365954&r1=365953&r2=365954&view=diff
>>> ==============================================================================
>>> --- cfe/trunk/lib/CMakeLists.txt (original)
>>> +++ cfe/trunk/lib/CMakeLists.txt Fri Jul 12 13:34:10 2019
>>> @@ -18,6 +18,7 @@ add_subdirectory(Serialization)
>>> add_subdirectory(Frontend)
>>> add_subdirectory(FrontendTool)
>>> add_subdirectory(Tooling)
>>> +add_subdirectory(DirectoryWatcher)
>>> add_subdirectory(Index)
>>> if(CLANG_ENABLE_STATIC_ANALYZER)
>>> add_subdirectory(StaticAnalyzer)
>>>
>>> Added: cfe/trunk/lib/DirectoryWatcher/CMakeLists.txt
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/CMakeLists.txt?rev=365954&view=auto
>>> ==============================================================================
>>> --- cfe/trunk/lib/DirectoryWatcher/CMakeLists.txt (added)
>>> +++ cfe/trunk/lib/DirectoryWatcher/CMakeLists.txt Fri Jul 12 13:34:10
>>> 2019
>>> @@ -0,0 +1,29 @@
>>> +include(CheckIncludeFiles)
>>> +
>>> +set(LLVM_LINK_COMPONENTS support)
>>> +
>>> +set(DIRECTORY_WATCHER_SOURCES DirectoryScanner.cpp)
>>> +set(DIRECTORY_WATCHER_LINK_LIBS "")
>>> +
>>> +if(APPLE)
>>> + check_include_files("CoreServices/CoreServices.h" HAVE_CORESERVICES)
>>> + if(HAVE_CORESERVICES)
>>> + list(APPEND DIRECTORY_WATCHER_SOURCES mac/DirectoryWatcher-mac.cpp)
>>> + set(DIRECTORY_WATCHER_LINK_LIBS "-framework CoreServices")
>>> + endif()
>>> +elseif(CMAKE_SYSTEM_NAME MATCHES "Linux")
>>> + check_include_files("sys/inotify.h" HAVE_INOTIFY)
>>> + if(HAVE_INOTIFY)
>>> + list(APPEND DIRECTORY_WATCHER_SOURCES
>>> linux/DirectoryWatcher-linux.cpp)
>>> + find_package(Threads REQUIRED)
>>> + endif()
>>> +else()
>>> + list(APPEND DIRECTORY_WATCHER_SOURCES
>>> default/DirectoryWatcher-not-implemented.cpp)
>>> +endif()
>>> +
>>> +add_clang_library(clangDirectoryWatcher
>>> + ${DIRECTORY_WATCHER_SOURCES}
>>> + )
>>> +
>>> +target_link_libraries(clangDirectoryWatcher PUBLIC
>>> ${CMAKE_THREAD_LIBS_INIT})
>>> +target_link_libraries(clangDirectoryWatcher PRIVATE
>>> ${DIRECTORY_WATCHER_LINK_LIBS})
>>>
>>> Added: cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.cpp
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.cpp?rev=365954&view=auto
>>> ==============================================================================
>>> --- cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.cpp (added)
>>> +++ cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.cpp Fri Jul 12
>>> 13:34:10 2019
>>> @@ -0,0 +1,54 @@
>>> +//===- DirectoryScanner.cpp - Utility functions for DirectoryWatcher
>>> ------===//
>>> +//
>>> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM
>>> Exceptions.
>>> +// See https://llvm.org/LICENSE.txt for license information.
>>> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
>>> +//
>>> +//===----------------------------------------------------------------------===//
>>> +
>>> +#include "DirectoryScanner.h"
>>> +
>>> +#include "llvm/Support/Path.h"
>>> +
>>> +namespace clang {
>>> +
>>> +using namespace llvm;
>>> +
>>> +Optional<sys::fs::file_status> getFileStatus(StringRef Path) {
>>> + sys::fs::file_status Status;
>>> + std::error_code EC = status(Path, Status);
>>> + if (EC)
>>> + return None;
>>> + return Status;
>>> +}
>>> +
>>> +std::vector<std::string> scanDirectory(StringRef Path) {
>>> + using namespace llvm::sys;
>>> + std::vector<std::string> Result;
>>> +
>>> + std::error_code EC;
>>> + for (auto It = fs::directory_iterator(Path, EC),
>>> + End = fs::directory_iterator();
>>> + !EC && It != End; It.increment(EC)) {
>>> + auto status = getFileStatus(It->path());
>>> + if (!status.hasValue())
>>> + continue;
>>> + Result.emplace_back(sys::path::filename(It->path()));
>>> + }
>>> +
>>> + return Result;
>>> +}
>>> +
>>> +std::vector<DirectoryWatcher::Event>
>>> +getAsFileEvents(const std::vector<std::string> &Scan) {
>>> + std::vector<DirectoryWatcher::Event> Events;
>>> + Events.reserve(Scan.size());
>>> +
>>> + for (const auto &File : Scan) {
>>> + Events.emplace_back(DirectoryWatcher::Event{
>>> + DirectoryWatcher::Event::EventKind::Modified, File});
>>> + }
>>> + return Events;
>>> +}
>>> +
>>> +} // namespace clang
>>> \ No newline at end of file
>>>
>>> Added: cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.h
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.h?rev=365954&view=auto
>>> ==============================================================================
>>> --- cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.h (added)
>>> +++ cfe/trunk/lib/DirectoryWatcher/DirectoryScanner.h Fri Jul 12
>>> 13:34:10 2019
>>> @@ -0,0 +1,29 @@
>>> +//===- DirectoryScanner.h - Utility functions for DirectoryWatcher
>>> --------===//
>>> +//
>>> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM
>>> Exceptions.
>>> +// See https://llvm.org/LICENSE.txt for license information.
>>> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
>>> +//
>>> +//===----------------------------------------------------------------------===//
>>> +
>>> +#include "clang/DirectoryWatcher/DirectoryWatcher.h"
>>> +#include "llvm/Support/FileSystem.h"
>>> +#include <string>
>>> +#include <vector>
>>> +
>>> +namespace clang {
>>> +
>>> +/// Gets names (filenames) of items in directory at \p Path.
>>> +/// \returns empty vector if \p Path is not a directory, doesn't
>>> exist or can't
>>> +/// be read from.
>>> +std::vector<std::string> scanDirectory(llvm::StringRef Path);
>>> +
>>> +/// Create event with EventKind::Added for every element in \p Scan.
>>> +std::vector<DirectoryWatcher::Event>
>>> +getAsFileEvents(const std::vector<std::string> &Scan);
>>> +
>>> +/// Gets status of file (or directory) at \p Path.
>>> +/// \returns llvm::None if \p Path doesn't exist or can't get the
>>> status.
>>> +llvm::Optional<llvm::sys::fs::file_status>
>>> getFileStatus(llvm::StringRef Path);
>>> +
>>> +} // namespace clang
>>> \ No newline at end of file
>>>
>>> Added:
>>> cfe/trunk/lib/DirectoryWatcher/default/DirectoryWatcher-not-implemented.cpp
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/default/DirectoryWatcher-not-implemented.cpp?rev=365954&view=auto
>>> ==============================================================================
>>> ---
>>> cfe/trunk/lib/DirectoryWatcher/default/DirectoryWatcher-not-implemented.cpp
>>> (added)
>>> +++
>>> cfe/trunk/lib/DirectoryWatcher/default/DirectoryWatcher-not-implemented.cpp
>>> Fri Jul 12 13:34:10 2019
>>> @@ -0,0 +1,19 @@
>>> +//===- DirectoryWatcher-not-implemented.cpp
>>> -------------------------------===//
>>> +//
>>> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM
>>> Exceptions.
>>> +// See https://llvm.org/LICENSE.txt for license information.
>>> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
>>> +//
>>> +//===----------------------------------------------------------------------===//
>>> +
>>> +#include "clang/DirectoryWatcher/DirectoryWatcher.h"
>>> +
>>> +using namespace llvm;
>>> +using namespace clang;
>>> +
>>> +std::unique_ptr<DirectoryWatcher> clang::DirectoryWatcher::create(
>>> + StringRef Path,
>>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>,
>>> bool)> Receiver,
>>> + bool WaitForInitialSync) {
>>> + return nullptr;
>>> +}
>>> \ No newline at end of file
>>>
>>> Added: cfe/trunk/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp?rev=365954&view=auto
>>> ==============================================================================
>>> --- cfe/trunk/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp
>>> (added)
>>> +++ cfe/trunk/lib/DirectoryWatcher/linux/DirectoryWatcher-linux.cpp
>>> Fri Jul 12 13:34:10 2019
>>> @@ -0,0 +1,345 @@
>>> +//===- DirectoryWatcher-linux.cpp - Linux-platform directory
>>> watching -----===//
>>> +//
>>> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM
>>> Exceptions.
>>> +// See https://llvm.org/LICENSE.txt for license information.
>>> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
>>> +//
>>> +//===----------------------------------------------------------------------===//
>>> +
>>> +#include "DirectoryScanner.h"
>>> +#include "clang/DirectoryWatcher/DirectoryWatcher.h"
>>> +
>>> +#include "llvm/ADT/STLExtras.h"
>>> +#include "llvm/ADT/ScopeExit.h"
>>> +#include "llvm/Support/AlignOf.h"
>>> +#include "llvm/Support/Errno.h"
>>> +#include "llvm/Support/Mutex.h"
>>> +#include "llvm/Support/Path.h"
>>> +#include <atomic>
>>> +#include <condition_variable>
>>> +#include <mutex>
>>> +#include <queue>
>>> +#include <string>
>>> +#include <thread>
>>> +#include <vector>
>>> +
>>> +#include <fcntl.h>
>>> +#include <sys/epoll.h>
>>> +#include <sys/inotify.h>
>>> +#include <unistd.h>
>>> +
>>> +namespace {
>>> +
>>> +using namespace llvm;
>>> +using namespace clang;
>>> +
>>> +/// Pipe for inter-thread synchronization - for epoll-ing on multiple
>>> +/// conditions. It is meant for uni-directional 1:1 signalling -
>>> specifically:
>>> +/// no multiple consumers, no data passing. Thread waiting for
>>> signal should
>>> +/// poll the FDRead. Signalling thread should call signal() which
>>> writes single
>>> +/// character to FDRead.
>>> +struct SemaphorePipe {
>>> + // Expects two file-descriptors opened as a pipe in the canonical
>>> POSIX
>>> + // order: pipefd[0] refers to the read end of the pipe. pipefd[1]
>>> refers to
>>> + // the write end of the pipe.
>>> + SemaphorePipe(int pipefd[2])
>>> + : FDRead(pipefd[0]), FDWrite(pipefd[1]), OwnsFDs(true) {}
>>> + SemaphorePipe(const SemaphorePipe &) = delete;
>>> + void operator=(const SemaphorePipe &) = delete;
>>> + SemaphorePipe(SemaphorePipe &&other)
>>> + : FDRead(other.FDRead), FDWrite(other.FDWrite),
>>> + OwnsFDs(other.OwnsFDs) // Someone could have moved from the
>>> other
>>> + // instance before.
>>> + {
>>> + other.OwnsFDs = false;
>>> + };
>>> +
>>> + void signal() {
>>> + ssize_t Result = llvm::sys::RetryAfterSignal(-1, write, FDWrite,
>>> "A", 1);
>>> + assert(Result != -1);
>>> + }
>>> + ~SemaphorePipe() {
>>> + if (OwnsFDs) {
>>> + close(FDWrite);
>>> + close(FDRead);
>>> + }
>>> + }
>>> + const int FDRead;
>>> + const int FDWrite;
>>> + bool OwnsFDs;
>>> +
>>> + static llvm::Optional<SemaphorePipe> create() {
>>> + int InotifyPollingStopperFDs[2];
>>> + if (pipe2(InotifyPollingStopperFDs, O_CLOEXEC) == -1)
>>> + return llvm::None;
>>> + return SemaphorePipe(InotifyPollingStopperFDs);
>>> + }
>>> +};
>>> +
>>> +/// Mutex-protected queue of Events.
>>> +class EventQueue {
>>> + std::mutex Mtx;
>>> + std::condition_variable NonEmpty;
>>> + std::queue<DirectoryWatcher::Event> Events;
>>> +
>>> +public:
>>> + void push_back(const DirectoryWatcher::Event::EventKind K,
>>> + StringRef Filename) {
>>> + {
>>> + std::unique_lock<std::mutex> L(Mtx);
>>> + Events.emplace(K, Filename);
>>> + }
>>> + NonEmpty.notify_one();
>>> + }
>>> +
>>> + // Blocks on caller thread and uses codition_variable to wait
>>> until there's an
>>> + // event to return.
>>> + DirectoryWatcher::Event pop_front_blocking() {
>>> + std::unique_lock<std::mutex> L(Mtx);
>>> + while (true) {
>>> + // Since we might have missed all the prior notifications on
>>> NonEmpty we
>>> + // have to check the queue first (under lock).
>>> + if (!Events.empty()) {
>>> + DirectoryWatcher::Event Front = Events.front();
>>> + Events.pop();
>>> + return Front;
>>> + }
>>> + NonEmpty.wait(L, [this]() { return !Events.empty(); });
>>> + }
>>> + }
>>> +};
>>> +
>>> +class DirectoryWatcherLinux : public clang::DirectoryWatcher {
>>> +public:
>>> + DirectoryWatcherLinux(
>>> + llvm::StringRef WatchedDirPath,
>>> + std::function<void(llvm::ArrayRef<Event>, bool)> Receiver,
>>> + bool WaitForInitialSync, int InotifyFD, int InotifyWD,
>>> + SemaphorePipe &&InotifyPollingStopSignal);
>>> +
>>> + ~DirectoryWatcherLinux() override {
>>> + StopWork();
>>> + InotifyPollingThread.join();
>>> + EventsReceivingThread.join();
>>> + inotify_rm_watch(InotifyFD, InotifyWD);
>>> + llvm::sys::RetryAfterSignal(-1, close, InotifyFD);
>>> + }
>>> +
>>> +private:
>>> + const std::string WatchedDirPath;
>>> + // inotify file descriptor
>>> + int InotifyFD = -1;
>>> + // inotify watch descriptor
>>> + int InotifyWD = -1;
>>> +
>>> + EventQueue Queue;
>>> +
>>> + // Make sure lifetime of Receiver fully contains lifetime of
>>> + // EventsReceivingThread.
>>> + std::function<void(llvm::ArrayRef<Event>, bool)> Receiver;
>>> +
>>> + // Consumes inotify events and pushes directory watcher events to
>>> the Queue.
>>> + void InotifyPollingLoop();
>>> + std::thread InotifyPollingThread;
>>> + // Using pipe so we can epoll two file descriptors at once -
>>> inotify and
>>> + // stopping condition.
>>> + SemaphorePipe InotifyPollingStopSignal;
>>> +
>>> + // Does the initial scan of the directory - directly calling Receiver,
>>> + // bypassing the Queue. Both InitialScan and EventReceivingLoop
>>> use Receiver
>>> + // which isn't necessarily thread-safe.
>>> + void InitialScan();
>>> +
>>> + // Processing events from the Queue.
>>> + // In case client doesn't want to do the initial scan synchronously
>>> + // (WaitForInitialSync=false in ctor) we do the initial scan at
>>> the beginning
>>> + // of this thread.
>>> + std::thread EventsReceivingThread;
>>> + // Push event of WatcherGotInvalidated kind to the Queue to stop
>>> the loop.
>>> + // Both InitialScan and EventReceivingLoop use Receiver which isn't
>>> + // necessarily thread-safe.
>>> + void EventReceivingLoop();
>>> +
>>> + // Stops all the async work. Reentrant.
>>> + void StopWork() {
>>> +
>>> Queue.push_back(DirectoryWatcher::Event::EventKind::WatcherGotInvalidated,
>>> + "");
>>> + InotifyPollingStopSignal.signal();
>>> + }
>>> +};
>>> +
>>> +void DirectoryWatcherLinux::InotifyPollingLoop() {
>>> + // We want to be able to read ~30 events at once even in the worst
>>> case
>>> + // (obscenely long filenames).
>>> + constexpr size_t EventBufferLength =
>>> + 30 * (sizeof(struct inotify_event) + NAME_MAX + 1);
>>> + // http://man7.org/linux/man-pages/man7/inotify.7.html
>>> + // Some systems cannot read integer variables if they are not
>>> + // properly aligned. On other systems, incorrect alignment may
>>> + // decrease performance. Hence, the buffer used for reading from
>>> + // the inotify file descriptor should have the same alignment as
>>> + // struct inotify_event.
>>> +
>>> + auto ManagedBuffer =
>>> + llvm::make_unique<llvm::AlignedCharArray<alignof(struct
>>> inotify_event),
>>> + EventBufferLength>>();
>>> + char *const Buf = ManagedBuffer->buffer;
>>> +
>>> + const int EpollFD = epoll_create1(EPOLL_CLOEXEC);
>>> + if (EpollFD == -1) {
>>> + StopWork();
>>> + return;
>>> + }
>>> + auto EpollFDGuard = llvm::make_scope_exit([EpollFD]() {
>>> close(EpollFD); });
>>> +
>>> + struct epoll_event EventSpec;
>>> + EventSpec.events = EPOLLIN;
>>> + EventSpec.data.fd = InotifyFD;
>>> + if (epoll_ctl(EpollFD, EPOLL_CTL_ADD, InotifyFD, &EventSpec) == -1) {
>>> + StopWork();
>>> + return;
>>> + }
>>> +
>>> + EventSpec.data.fd = InotifyPollingStopSignal.FDRead;
>>> + if (epoll_ctl(EpollFD, EPOLL_CTL_ADD, InotifyPollingStopSignal.FDRead,
>>> + &EventSpec) == -1) {
>>> + StopWork();
>>> + return;
>>> + }
>>> +
>>> + std::array<struct epoll_event, 2> EpollEventBuffer;
>>> +
>>> + while (true) {
>>> + const int EpollWaitResult = llvm::sys::RetryAfterSignal(
>>> + -1, epoll_wait, EpollFD, EpollEventBuffer.data(),
>>> + EpollEventBuffer.size(), /*timeout=*/-1 /*== infinity*/);
>>> + if (EpollWaitResult == -1) {
>>> + StopWork();
>>> + return;
>>> + }
>>> +
>>> + // Multiple epoll_events can be received for a single file
>>> descriptor per
>>> + // epoll_wait call.
>>> + for (const auto &EpollEvent : EpollEventBuffer) {
>>> + if (EpollEvent.data.fd == InotifyPollingStopSignal.FDRead) {
>>> + StopWork();
>>> + return;
>>> + }
>>> + }
>>> +
>>> + // epoll_wait() always return either error or >0 events. Since
>>> there was no
>>> + // event for stopping, it must be an inotify event ready for
>>> reading.
>>> + ssize_t NumRead = llvm::sys::RetryAfterSignal(-1, read,
>>> InotifyFD, Buf,
>>> + EventBufferLength);
>>> + for (char *P = Buf; P < Buf + NumRead;) {
>>> + if (P + sizeof(struct inotify_event) > Buf + NumRead) {
>>> + StopWork();
>>> + llvm_unreachable("an incomplete inotify_event was read");
>>> + return;
>>> + }
>>> +
>>> + struct inotify_event *Event = reinterpret_cast<struct
>>> inotify_event *>(P);
>>> + P += sizeof(struct inotify_event) + Event->len;
>>> +
>>> + if (Event->mask & (IN_CREATE | IN_MODIFY | IN_MOVED_TO |
>>> IN_DELETE) &&
>>> + Event->len <= 0) {
>>> + StopWork();
>>> + llvm_unreachable("expected a filename from inotify");
>>> + return;
>>> + }
>>> +
>>> + if (Event->mask & (IN_CREATE | IN_MOVED_TO | IN_MODIFY)) {
>>> + Queue.push_back(DirectoryWatcher::Event::EventKind::Modified,
>>> + Event->name);
>>> + } else if (Event->mask & (IN_DELETE | IN_MOVED_FROM)) {
>>> + Queue.push_back(DirectoryWatcher::Event::EventKind::Removed,
>>> + Event->name);
>>> + } else if (Event->mask & (IN_DELETE_SELF | IN_MOVE_SELF)) {
>>> +
>>> Queue.push_back(DirectoryWatcher::Event::EventKind::WatchedDirRemoved,
>>> + "");
>>> + StopWork();
>>> + return;
>>> + } else if (Event->mask & IN_IGNORED) {
>>> + StopWork();
>>> + return;
>>> + } else {
>>> + StopWork();
>>> + llvm_unreachable("Unknown event type.");
>>> + return;
>>> + }
>>> + }
>>> + }
>>> +}
>>> +
>>> +void DirectoryWatcherLinux::InitialScan() {
>>> + this->Receiver(getAsFileEvents(scanDirectory(WatchedDirPath)),
>>> + /*IsInitial=*/true);
>>> +}
>>> +
>>> +void DirectoryWatcherLinux::EventReceivingLoop() {
>>> + while (true) {
>>> + DirectoryWatcher::Event Event = this->Queue.pop_front_blocking();
>>> + this->Receiver(Event, false);
>>> + if (Event.Kind ==
>>> + DirectoryWatcher::Event::EventKind::WatcherGotInvalidated) {
>>> + StopWork();
>>> + return;
>>> + }
>>> + }
>>> +}
>>> +
>>> +DirectoryWatcherLinux::DirectoryWatcherLinux(
>>> + StringRef WatchedDirPath,
>>> + std::function<void(llvm::ArrayRef<Event>, bool)> Receiver,
>>> + bool WaitForInitialSync, int InotifyFD, int InotifyWD,
>>> + SemaphorePipe &&InotifyPollingStopSignal)
>>> + : WatchedDirPath(WatchedDirPath), InotifyFD(InotifyFD),
>>> + InotifyWD(InotifyWD), Receiver(Receiver),
>>> + InotifyPollingStopSignal(std::move(InotifyPollingStopSignal)) {
>>> +
>>> + InotifyPollingThread = std::thread([this]() {
>>> InotifyPollingLoop(); });
>>> + // We have no guarantees about thread safety of the Receiver which
>>> is being
>>> + // used in both InitialScan and EventReceivingLoop. We shouldn't
>>> run these
>>> + // only synchronously.
>>> + if (WaitForInitialSync) {
>>> + InitialScan();
>>> + EventsReceivingThread = std::thread([this]() {
>>> EventReceivingLoop(); });
>>> + } else {
>>> + EventsReceivingThread = std::thread([this]() {
>>> + // FIXME: We might want to terminate an async initial scan
>>> early in case
>>> + // of a failure in EventsReceivingThread.
>>> + InitialScan();
>>> + EventReceivingLoop();
>>> + });
>>> + }
>>> +}
>>> +
>>> +} // namespace
>>> +
>>> +std::unique_ptr<DirectoryWatcher> clang::DirectoryWatcher::create(
>>> + StringRef Path,
>>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>,
>>> bool)> Receiver,
>>> + bool WaitForInitialSync) {
>>> + if (Path.empty())
>>> + return nullptr;
>>> +
>>> + const int InotifyFD = inotify_init1(IN_CLOEXEC);
>>> + if (InotifyFD == -1)
>>> + return nullptr;
>>> +
>>> + const int InotifyWD = inotify_add_watch(
>>> + InotifyFD, Path.str().c_str(),
>>> + IN_CREATE | IN_DELETE | IN_DELETE_SELF | IN_EXCL_UNLINK |
>>> IN_MODIFY |
>>> + IN_MOVED_FROM | IN_MOVE_SELF | IN_MOVED_TO | IN_ONLYDIR |
>>> IN_IGNORED);
>>> + if (InotifyWD == -1)
>>> + return nullptr;
>>> +
>>> + auto InotifyPollingStopper = SemaphorePipe::create();
>>> +
>>> + if (!InotifyPollingStopper)
>>> + return nullptr;
>>> +
>>> + return llvm::make_unique<DirectoryWatcherLinux>(
>>> + Path, Receiver, WaitForInitialSync, InotifyFD, InotifyWD,
>>> + std::move(*InotifyPollingStopper));
>>> +}
>>> \ No newline at end of file
>>>
>>> Added: cfe/trunk/lib/DirectoryWatcher/mac/DirectoryWatcher-mac.cpp
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/DirectoryWatcher/mac/DirectoryWatcher-mac.cpp?rev=365954&view=auto
>>> ==============================================================================
>>> --- cfe/trunk/lib/DirectoryWatcher/mac/DirectoryWatcher-mac.cpp (added)
>>> +++ cfe/trunk/lib/DirectoryWatcher/mac/DirectoryWatcher-mac.cpp Fri
>>> Jul 12 13:34:10 2019
>>> @@ -0,0 +1,233 @@
>>> +//===- DirectoryWatcher-mac.cpp - Mac-platform directory watching
>>> ---------===//
>>> +//
>>> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM
>>> Exceptions.
>>> +// See https://llvm.org/LICENSE.txt for license information.
>>> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
>>> +//
>>> +//===----------------------------------------------------------------------===//
>>> +
>>> +#include "DirectoryScanner.h"
>>> +#include "clang/DirectoryWatcher/DirectoryWatcher.h"
>>> +
>>> +#include "llvm/ADT/STLExtras.h"
>>> +#include "llvm/ADT/StringRef.h"
>>> +#include "llvm/Support/Path.h"
>>> +#include <CoreServices/CoreServices.h>
>>> +
>>> +using namespace llvm;
>>> +using namespace clang;
>>> +
>>> +static FSEventStreamRef createFSEventStream(
>>> + StringRef Path,
>>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)>,
>>> + dispatch_queue_t);
>>> +static void stopFSEventStream(FSEventStreamRef);
>>> +
>>> +namespace {
>>> +
>>> +class DirectoryWatcherMac : public clang::DirectoryWatcher {
>>> +public:
>>> + DirectoryWatcherMac(
>>> + FSEventStreamRef EventStream,
>>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)>
>>> + Receiver,
>>> + llvm::StringRef WatchedDirPath)
>>> + : EventStream(EventStream), Receiver(Receiver),
>>> + WatchedDirPath(WatchedDirPath) {}
>>> +
>>> + ~DirectoryWatcherMac() override {
>>> + stopFSEventStream(EventStream);
>>> + EventStream = nullptr;
>>> + // Now it's safe to use Receiver as the only other concurrent
>>> use would have
>>> + // been in EventStream processing.
>>> + Receiver(DirectoryWatcher::Event(
>>> +
>>> DirectoryWatcher::Event::EventKind::WatcherGotInvalidated,
>>> ""),
>>> + false);
>>> + }
>>> +
>>> +private:
>>> + FSEventStreamRef EventStream;
>>> + std::function<void(llvm::ArrayRef<Event>, bool)> Receiver;
>>> + const std::string WatchedDirPath;
>>> +};
>>> +
>>> +struct EventStreamContextData {
>>> + std::string WatchedPath;
>>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)>
>>> Receiver;
>>> +
>>> + EventStreamContextData(
>>> + std::string &&WatchedPath,
>>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>, bool)>
>>> + Receiver)
>>> + : WatchedPath(std::move(WatchedPath)), Receiver(Receiver) {}
>>> +
>>> + // Needed for FSEvents
>>> + static void dispose(const void *ctx) {
>>> + delete static_cast<const EventStreamContextData *>(ctx);
>>> + }
>>> +};
>>> +} // namespace
>>> +
>>> +constexpr const FSEventStreamEventFlags StreamInvalidatingFlags =
>>> + kFSEventStreamEventFlagUserDropped |
>>> kFSEventStreamEventFlagKernelDropped |
>>> + kFSEventStreamEventFlagMustScanSubDirs;
>>> +
>>> +constexpr const FSEventStreamEventFlags ModifyingFileEvents =
>>> + kFSEventStreamEventFlagItemCreated |
>>> kFSEventStreamEventFlagItemRenamed |
>>> + kFSEventStreamEventFlagItemModified;
>>> +
>>> +static void eventStreamCallback(ConstFSEventStreamRef Stream,
>>> + void *ClientCallBackInfo, size_t
>>> NumEvents,
>>> + void *EventPaths,
>>> + const FSEventStreamEventFlags
>>> EventFlags[],
>>> + const FSEventStreamEventId EventIds[]) {
>>> + auto *ctx = static_cast<EventStreamContextData *>(ClientCallBackInfo);
>>> +
>>> + std::vector<DirectoryWatcher::Event> Events;
>>> + for (size_t i = 0; i < NumEvents; ++i) {
>>> + StringRef Path = ((const char **)EventPaths)[i];
>>> + const FSEventStreamEventFlags Flags = EventFlags[i];
>>> +
>>> + if (Flags & StreamInvalidatingFlags) {
>>> + Events.emplace_back(DirectoryWatcher::Event{
>>> + DirectoryWatcher::Event::EventKind::WatcherGotInvalidated,
>>> ""});
>>> + break;
>>> + } else if (!(Flags & kFSEventStreamEventFlagItemIsFile)) {
>>> + // Subdirectories aren't supported - if some directory got
>>> removed it
>>> + // must've been the watched directory itself.
>>> + if ((Flags & kFSEventStreamEventFlagItemRemoved) &&
>>> + Path == ctx->WatchedPath) {
>>> + Events.emplace_back(DirectoryWatcher::Event{
>>> + DirectoryWatcher::Event::EventKind::WatchedDirRemoved, ""});
>>> + Events.emplace_back(DirectoryWatcher::Event{
>>> +
>>> DirectoryWatcher::Event::EventKind::WatcherGotInvalidated,
>>> ""});
>>> + break;
>>> + }
>>> + // No support for subdirectories - just ignore everything.
>>> + continue;
>>> + } else if (Flags & kFSEventStreamEventFlagItemRemoved) {
>>> + Events.emplace_back(DirectoryWatcher::Event::EventKind::Removed,
>>> + llvm::sys::path::filename(Path));
>>> + continue;
>>> + } else if (Flags & ModifyingFileEvents) {
>>> + if (!getFileStatus(Path).hasValue()) {
>>> + Events.emplace_back(DirectoryWatcher::Event::EventKind::Removed,
>>> + llvm::sys::path::filename(Path));
>>> + } else {
>>> +
>>> Events.emplace_back(DirectoryWatcher::Event::EventKind::Modified,
>>> + llvm::sys::path::filename(Path));
>>> + }
>>> + continue;
>>> + }
>>> +
>>> + // default
>>> + Events.emplace_back(DirectoryWatcher::Event{
>>> + DirectoryWatcher::Event::EventKind::WatcherGotInvalidated, ""});
>>> + llvm_unreachable("Unknown FSEvent type.");
>>> + }
>>> +
>>> + if (!Events.empty()) {
>>> + ctx->Receiver(Events, /*IsInitial=*/false);
>>> + }
>>> +}
>>> +
>>> +FSEventStreamRef createFSEventStream(
>>> + StringRef Path,
>>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>,
>>> bool)> Receiver,
>>> + dispatch_queue_t Queue) {
>>> + if (Path.empty())
>>> + return nullptr;
>>> +
>>> + CFMutableArrayRef PathsToWatch = [&]() {
>>> + CFMutableArrayRef PathsToWatch =
>>> + CFArrayCreateMutable(nullptr, 0, &kCFTypeArrayCallBacks);
>>> + CFStringRef CfPathStr =
>>> + CFStringCreateWithBytes(nullptr, (const UInt8 *)Path.data(),
>>> + Path.size(), kCFStringEncodingUTF8,
>>> false);
>>> + CFArrayAppendValue(PathsToWatch, CfPathStr);
>>> + CFRelease(CfPathStr);
>>> + return PathsToWatch;
>>> + }();
>>> +
>>> + FSEventStreamContext Context = [&]() {
>>> + std::string RealPath;
>>> + {
>>> + SmallString<128> Storage;
>>> + StringRef P =
>>> llvm::Twine(Path).toNullTerminatedStringRef(Storage);
>>> + char Buffer[PATH_MAX];
>>> + if (::realpath(P.begin(), Buffer) != nullptr)
>>> + RealPath = Buffer;
>>> + else
>>> + RealPath = Path;
>>> + }
>>> +
>>> + FSEventStreamContext Context;
>>> + Context.version = 0;
>>> + Context.info
>>> <https://protect2.fireeye.com/url?k=7f9649a9-231c825d-7f960932-86cd58c48020-a392d417a0cc96b6&q=1&u=http%3A%2F%2Fcontext.info%2F>
>>> = new EventStreamContextData(std::move(RealPath), Receiver);
>>> + Context.retain = nullptr;
>>> + Context.release = EventStreamContextData::dispose;
>>> + Context.copyDescription = nullptr;
>>> + return Context;
>>> + }();
>>> +
>>> + FSEventStreamRef Result = FSEventStreamCreate(
>>> + nullptr, eventStreamCallback, &Context, PathsToWatch,
>>> + kFSEventStreamEventIdSinceNow, /* latency in seconds */ 0.0,
>>> + kFSEventStreamCreateFlagFileEvents |
>>> kFSEventStreamCreateFlagNoDefer);
>>> + CFRelease(PathsToWatch);
>>> +
>>> + return Result;
>>> +}
>>> +
>>> +void stopFSEventStream(FSEventStreamRef EventStream) {
>>> + if (!EventStream)
>>> + return;
>>> + FSEventStreamStop(EventStream);
>>> + FSEventStreamInvalidate(EventStream);
>>> + FSEventStreamRelease(EventStream);
>>> +}
>>> +
>>> +std::unique_ptr<DirectoryWatcher> clang::DirectoryWatcher::create(
>>> + StringRef Path,
>>> + std::function<void(llvm::ArrayRef<DirectoryWatcher::Event>,
>>> bool)> Receiver,
>>> + bool WaitForInitialSync) {
>>> + dispatch_queue_t Queue =
>>> + dispatch_queue_create("DirectoryWatcher", DISPATCH_QUEUE_SERIAL);
>>> +
>>> + if (Path.empty())
>>> + return nullptr;
>>> +
>>> + auto EventStream = createFSEventStream(Path, Receiver, Queue);
>>> + if (!EventStream) {
>>> + return nullptr;
>>> + }
>>> +
>>> + std::unique_ptr<DirectoryWatcher> Result =
>>> + llvm::make_unique<DirectoryWatcherMac>(EventStream, Receiver,
>>> Path);
>>> +
>>> + // We need to copy the data so the lifetime is ok after a const
>>> copy is made
>>> + // for the block.
>>> + const std::string CopiedPath = Path;
>>> +
>>> + auto InitWork = ^{
>>> + // We need to start watching the directory before we start
>>> scanning in order
>>> + // to not miss any event. By dispatching this on the same serial
>>> Queue as
>>> + // the FSEvents will be handled we manage to start watching
>>> BEFORE the
>>> + // inital scan and handling events ONLY AFTER the scan finishes.
>>> + FSEventStreamSetDispatchQueue(EventStream, Queue);
>>> + FSEventStreamStart(EventStream);
>>> + // We need to decrement the ref count for Queue as initialize()
>>> will return
>>> + // and FSEvents has incremented it. Since we have to wait for
>>> FSEvents to
>>> + // take ownership it's the easiest to do it here rather than
>>> main thread.
>>> + dispatch_release(Queue);
>>> + Receiver(getAsFileEvents(scanDirectory(CopiedPath)),
>>> /*IsInitial=*/true);
>>> + };
>>> +
>>> + if (WaitForInitialSync) {
>>> + dispatch_sync(Queue, InitWork);
>>> + } else {
>>> + dispatch_async(Queue, InitWork);
>>> + }
>>> +
>>> + return Result;
>>> +}
>>>
>>> Modified: cfe/trunk/unittests/CMakeLists.txt
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/cfe/trunk/unittests/CMakeLists.txt?rev=365954&r1=365953&r2=365954&view=diff
>>> ==============================================================================
>>> --- cfe/trunk/unittests/CMakeLists.txt (original)
>>> +++ cfe/trunk/unittests/CMakeLists.txt Fri Jul 12 13:34:10 2019
>>> @@ -30,6 +30,7 @@ add_subdirectory(CodeGen)
>>> if(NOT WIN32 AND CLANG_TOOL_LIBCLANG_BUILD)
>>> add_subdirectory(libclang)
>>> endif()
>>> +add_subdirectory(DirectoryWatcher)
>>> add_subdirectory(Rename)
>>> add_subdirectory(Index)
>>> add_subdirectory(Serialization)
>>>
>>> Added: cfe/trunk/unittests/DirectoryWatcher/CMakeLists.txt
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/cfe/trunk/unittests/DirectoryWatcher/CMakeLists.txt?rev=365954&view=auto
>>> ==============================================================================
>>> --- cfe/trunk/unittests/DirectoryWatcher/CMakeLists.txt (added)
>>> +++ cfe/trunk/unittests/DirectoryWatcher/CMakeLists.txt Fri Jul 12
>>> 13:34:10 2019
>>> @@ -0,0 +1,17 @@
>>> +if(APPLE OR CMAKE_SYSTEM_NAME MATCHES "Linux")
>>> +
>>> + set(LLVM_LINK_COMPONENTS
>>> + Support
>>> + )
>>> +
>>> + add_clang_unittest(DirectoryWatcherTests
>>> + DirectoryWatcherTest.cpp
>>> + )
>>> +
>>> + target_link_libraries(DirectoryWatcherTests
>>> + PRIVATE
>>> + clangDirectoryWatcher
>>> + clangBasic
>>> + )
>>> +
>>> +endif()
>>> \ No newline at end of file
>>>
>>> Added: cfe/trunk/unittests/DirectoryWatcher/DirectoryWatcherTest.cpp
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/cfe/trunk/unittests/DirectoryWatcher/DirectoryWatcherTest.cpp?rev=365954&view=auto
>>> ==============================================================================
>>> --- cfe/trunk/unittests/DirectoryWatcher/DirectoryWatcherTest.cpp (added)
>>> +++ cfe/trunk/unittests/DirectoryWatcher/DirectoryWatcherTest.cpp Fri
>>> Jul 12 13:34:10 2019
>>> @@ -0,0 +1,426 @@
>>> +//===- unittests/DirectoryWatcher/DirectoryWatcherTest.cpp
>>> ----------------===//
>>> +//
>>> +// Part of the LLVM Project, under the Apache License v2.0 with LLVM
>>> Exceptions.
>>> +// See https://llvm.org/LICENSE.txt for license information.
>>> +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
>>> +//
>>> +//===----------------------------------------------------------------------===//
>>> +
>>> +#include "clang/DirectoryWatcher/DirectoryWatcher.h"
>>> +#include "llvm/Support/FileSystem.h"
>>> +#include "llvm/Support/Mutex.h"
>>> +#include "llvm/Support/Path.h"
>>> +#include "llvm/Support/raw_ostream.h"
>>> +#include "gtest/gtest.h"
>>> +#include <condition_variable>
>>> +#include <future>
>>> +#include <mutex>
>>> +#include <thread>
>>> +
>>> +using namespace llvm;
>>> +using namespace llvm::sys;
>>> +using namespace llvm::sys::fs;
>>> +using namespace clang;
>>> +
>>> +namespace clang {
>>> +static bool operator==(const DirectoryWatcher::Event &lhs,
>>> + const DirectoryWatcher::Event &rhs) {
>>> + return lhs.Filename == rhs.Filename &&
>>> + static_cast<int>(lhs.Kind) == static_cast<int>(rhs.Kind);
>>> +}
>>> +} // namespace clang
>>> +
>>> +namespace {
>>> +
>>> +struct DirectoryWatcherTestFixture {
>>> + std::string TestRootDir;
>>> + std::string TestWatchedDir;
>>> +
>>> + DirectoryWatcherTestFixture() {
>>> + SmallString<128> pathBuf;
>>> + std::error_code UniqDirRes = createUniqueDirectory("dirwatcher",
>>> pathBuf);
>>> + assert(!UniqDirRes);
>>> + TestRootDir = pathBuf.str();
>>> + path::append(pathBuf, "watch");
>>> + TestWatchedDir = pathBuf.str();
>>> + std::error_code CreateDirRes = create_directory(TestWatchedDir,
>>> false);
>>> + assert(!CreateDirRes);
>>> + }
>>> +
>>> + ~DirectoryWatcherTestFixture() { remove_directories(TestRootDir); }
>>> +
>>> + SmallString<128> getPathInWatched(const std::string &testFile) {
>>> + SmallString<128> pathBuf;
>>> + pathBuf = TestWatchedDir;
>>> + path::append(pathBuf, testFile);
>>> + return pathBuf;
>>> + }
>>> +
>>> + void addFile(const std::string &testFile) {
>>> + Expected<file_t> ft =
>>> openNativeFileForWrite(getPathInWatched(testFile),
>>> + CD_CreateNew, OF_None);
>>> + if (ft) {
>>> + closeFile(*ft);
>>> + } else {
>>> + llvm::errs() << llvm::toString(ft.takeError()) << "\n";
>>> + llvm::errs() << getPathInWatched(testFile) << "\n";
>>> + llvm_unreachable("Couldn't create test file.");
>>> + }
>>> + }
>>> +
>>> + void deleteFile(const std::string &testFile) {
>>> + std::error_code EC =
>>> + remove(getPathInWatched(testFile), /*IgnoreNonExisting=*/false);
>>> + ASSERT_FALSE(EC);
>>> + }
>>> +};
>>> +
>>> +std::string eventKindToString(const
>>> DirectoryWatcher::Event::EventKind K) {
>>> + switch (K) {
>>> + case DirectoryWatcher::Event::EventKind::Removed:
>>> + return "Removed";
>>> + case DirectoryWatcher::Event::EventKind::Modified:
>>> + return "Modified";
>>> + case DirectoryWatcher::Event::EventKind::WatchedDirRemoved:
>>> + return "WatchedDirRemoved";
>>> + case DirectoryWatcher::Event::EventKind::WatcherGotInvalidated:
>>> + return "WatcherGotInvalidated";
>>> + }
>>> + llvm_unreachable("unknown event kind");
>>> +}
>>> +
>>> +struct VerifyingConsumer {
>>> + std::vector<DirectoryWatcher::Event> ExpectedInitial;
>>> + std::vector<DirectoryWatcher::Event> ExpectedNonInitial;
>>> + std::vector<DirectoryWatcher::Event> OptionalNonInitial;
>>> + std::vector<DirectoryWatcher::Event> UnexpectedInitial;
>>> + std::vector<DirectoryWatcher::Event> UnexpectedNonInitial;
>>> + std::mutex Mtx;
>>> + std::condition_variable ResultIsReady;
>>> +
>>> + VerifyingConsumer(
>>> + const std::vector<DirectoryWatcher::Event> &ExpectedInitial,
>>> + const std::vector<DirectoryWatcher::Event> &ExpectedNonInitial,
>>> + const std::vector<DirectoryWatcher::Event> &OptionalNonInitial
>>> = {})
>>> + : ExpectedInitial(ExpectedInitial),
>>> + ExpectedNonInitial(ExpectedNonInitial),
>>> + OptionalNonInitial(OptionalNonInitial) {}
>>> +
>>> + // This method is used by DirectoryWatcher.
>>> + void consume(DirectoryWatcher::Event E, bool IsInitial) {
>>> + if (IsInitial)
>>> + consumeInitial(E);
>>> + else
>>> + consumeNonInitial(E);
>>> + }
>>> +
>>> + void consumeInitial(DirectoryWatcher::Event E) {
>>> + std::unique_lock<std::mutex> L(Mtx);
>>> + auto It = std::find(ExpectedInitial.begin(),
>>> ExpectedInitial.end(), E);
>>> + if (It == ExpectedInitial.end()) {
>>> + UnexpectedInitial.push_back(E);
>>> + } else {
>>> + ExpectedInitial.erase(It);
>>> + }
>>> + if (result())
>>> + ResultIsReady.notify_one();
>>> + }
>>> +
>>> + void consumeNonInitial(DirectoryWatcher::Event E) {
>>> + std::unique_lock<std::mutex> L(Mtx);
>>> + auto It =
>>> + std::find(ExpectedNonInitial.begin(),
>>> ExpectedNonInitial.end(), E);
>>> + if (It == ExpectedNonInitial.end()) {
>>> + auto OptIt =
>>> + std::find(OptionalNonInitial.begin(),
>>> OptionalNonInitial.end(), E);
>>> + if (OptIt != OptionalNonInitial.end()) {
>>> + OptionalNonInitial.erase(OptIt);
>>> + } else {
>>> + UnexpectedNonInitial.push_back(E);
>>> + }
>>> + } else {
>>> + ExpectedNonInitial.erase(It);
>>> + }
>>> + if (result())
>>> + ResultIsReady.notify_one();
>>> + }
>>> +
>>> + // This method is used by DirectoryWatcher.
>>> + void consume(llvm::ArrayRef<DirectoryWatcher::Event> Es, bool
>>> IsInitial) {
>>> + for (const auto &E : Es)
>>> + consume(E, IsInitial);
>>> + }
>>> +
>>> + // Not locking - caller has to lock Mtx.
>>> + llvm::Optional<bool> result() const {
>>> + if (ExpectedInitial.empty() && ExpectedNonInitial.empty() &&
>>> + UnexpectedInitial.empty() && UnexpectedNonInitial.empty())
>>> + return true;
>>> + if (!UnexpectedInitial.empty() || !UnexpectedNonInitial.empty())
>>> + return false;
>>> + return llvm::None;
>>> + }
>>> +
>>> + // This method is used by tests.
>>> + // \returns true on success
>>> + bool blockUntilResult() {
>>> + std::unique_lock<std::mutex> L(Mtx);
>>> + while (true) {
>>> + if (result())
>>> + return *result();
>>> +
>>> + ResultIsReady.wait(L, [this]() { return result().hasValue(); });
>>> + }
>>> + return false; // Just to make compiler happy.
>>> + }
>>> +
>>> + void printUnmetExpectations(llvm::raw_ostream &OS) {
>>> + if (!ExpectedInitial.empty()) {
>>> + OS << "Expected but not seen initial events: \n";
>>> + for (const auto &E : ExpectedInitial) {
>>> + OS << eventKindToString(E.Kind) << " " << E.Filename << "\n";
>>> + }
>>> + }
>>> + if (!ExpectedNonInitial.empty()) {
>>> + OS << "Expected but not seen non-initial events: \n";
>>> + for (const auto &E : ExpectedNonInitial) {
>>> + OS << eventKindToString(E.Kind) << " " << E.Filename << "\n";
>>> + }
>>> + }
>>> + if (!UnexpectedInitial.empty()) {
>>> + OS << "Unexpected initial events seen: \n";
>>> + for (const auto &E : UnexpectedInitial) {
>>> + OS << eventKindToString(E.Kind) << " " << E.Filename << "\n";
>>> + }
>>> + }
>>> + if (!UnexpectedNonInitial.empty()) {
>>> + OS << "Unexpected non-initial events seen: \n";
>>> + for (const auto &E : UnexpectedNonInitial) {
>>> + OS << eventKindToString(E.Kind) << " " << E.Filename << "\n";
>>> + }
>>> + }
>>> + }
>>> +};
>>> +
>>> +void checkEventualResultWithTimeout(VerifyingConsumer &TestConsumer) {
>>> + std::packaged_task<int(void)> task(
>>> + [&TestConsumer]() { return TestConsumer.blockUntilResult(); });
>>> + std::future<int> WaitForExpectedStateResult = task.get_future();
>>> + std::thread worker(std::move(task));
>>> + worker.detach();
>>> +
>>> +
>>> EXPECT_TRUE(WaitForExpectedStateResult.wait_for(std::chrono::seconds(3))
>>> ==
>>> + std::future_status::ready)
>>> + << "The expected result state wasn't reached before the
>>> time-out.";
>>> + EXPECT_TRUE(TestConsumer.result().hasValue());
>>> + if (TestConsumer.result().hasValue()) {
>>> + EXPECT_TRUE(*TestConsumer.result());
>>> + }
>>> + if ((TestConsumer.result().hasValue() &&
>>> !TestConsumer.result().getValue()) ||
>>> + !TestConsumer.result().hasValue())
>>> + TestConsumer.printUnmetExpectations(llvm::outs());
>>> +}
>>> +
>>> +} // namespace
>>> +
>>> +TEST(DirectoryWatcherTest, InitialScanSync) {
>>> + DirectoryWatcherTestFixture fixture;
>>> +
>>> + fixture.addFile("a");
>>> + fixture.addFile("b");
>>> + fixture.addFile("c");
>>> +
>>> + VerifyingConsumer TestConsumer{
>>> + {{DirectoryWatcher::Event::EventKind::Modified, "a"},
>>> + {DirectoryWatcher::Event::EventKind::Modified, "b"},
>>> + {DirectoryWatcher::Event::EventKind::Modified, "c"}},
>>> + {}};
>>> +
>>> + auto DW = DirectoryWatcher::create(
>>> + fixture.TestWatchedDir,
>>> + [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
>>> + bool IsInitial) {
>>> + TestConsumer.consume(Events, IsInitial);
>>> + },
>>> + /*waitForInitialSync=*/true);
>>> +
>>> + checkEventualResultWithTimeout(TestConsumer);
>>> +}
>>> +
>>> +TEST(DirectoryWatcherTest, InitialScanAsync) {
>>> + DirectoryWatcherTestFixture fixture;
>>> +
>>> + fixture.addFile("a");
>>> + fixture.addFile("b");
>>> + fixture.addFile("c");
>>> +
>>> + VerifyingConsumer TestConsumer{
>>> + {{DirectoryWatcher::Event::EventKind::Modified, "a"},
>>> + {DirectoryWatcher::Event::EventKind::Modified, "b"},
>>> + {DirectoryWatcher::Event::EventKind::Modified, "c"}},
>>> + {}};
>>> +
>>> + auto DW = DirectoryWatcher::create(
>>> + fixture.TestWatchedDir,
>>> + [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
>>> + bool IsInitial) {
>>> + TestConsumer.consume(Events, IsInitial);
>>> + },
>>> + /*waitForInitialSync=*/false);
>>> +
>>> + checkEventualResultWithTimeout(TestConsumer);
>>> +}
>>> +
>>> +TEST(DirectoryWatcherTest, AddFiles) {
>>> + DirectoryWatcherTestFixture fixture;
>>> +
>>> + VerifyingConsumer TestConsumer{
>>> + {},
>>> + {{DirectoryWatcher::Event::EventKind::Modified, "a"},
>>> + {DirectoryWatcher::Event::EventKind::Modified, "b"},
>>> + {DirectoryWatcher::Event::EventKind::Modified, "c"}}};
>>> +
>>> + auto DW = DirectoryWatcher::create(
>>> + fixture.TestWatchedDir,
>>> + [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
>>> + bool IsInitial) {
>>> + TestConsumer.consume(Events, IsInitial);
>>> + },
>>> + /*waitForInitialSync=*/true);
>>> +
>>> + fixture.addFile("a");
>>> + fixture.addFile("b");
>>> + fixture.addFile("c");
>>> +
>>> + checkEventualResultWithTimeout(TestConsumer);
>>> +}
>>> +
>>> +TEST(DirectoryWatcherTest, ModifyFile) {
>>> + DirectoryWatcherTestFixture fixture;
>>> +
>>> + fixture.addFile("a");
>>> +
>>> + VerifyingConsumer TestConsumer{
>>> + {{DirectoryWatcher::Event::EventKind::Modified, "a"}},
>>> + {{DirectoryWatcher::Event::EventKind::Modified, "a"}}};
>>> +
>>> + auto DW = DirectoryWatcher::create(
>>> + fixture.TestWatchedDir,
>>> + [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
>>> + bool IsInitial) {
>>> + TestConsumer.consume(Events, IsInitial);
>>> + },
>>> + /*waitForInitialSync=*/true);
>>> +
>>> + // modify the file
>>> + {
>>> + std::error_code error;
>>> + llvm::raw_fd_ostream bStream(fixture.getPathInWatched("a"), error,
>>> + CD_OpenExisting);
>>> + assert(!error);
>>> + bStream << "foo";
>>> + }
>>> +
>>> + checkEventualResultWithTimeout(TestConsumer);
>>> +}
>>> +
>>> +TEST(DirectoryWatcherTest, DeleteFile) {
>>> + DirectoryWatcherTestFixture fixture;
>>> +
>>> + fixture.addFile("a");
>>> +
>>> + VerifyingConsumer TestConsumer{
>>> + {{DirectoryWatcher::Event::EventKind::Modified, "a"}},
>>> + {{DirectoryWatcher::Event::EventKind::Removed, "a"}}};
>>> +
>>> + auto DW = DirectoryWatcher::create(
>>> + fixture.TestWatchedDir,
>>> + [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
>>> + bool IsInitial) {
>>> + TestConsumer.consume(Events, IsInitial);
>>> + },
>>> + /*waitForInitialSync=*/true);
>>> +
>>> + fixture.deleteFile("a");
>>> +
>>> + checkEventualResultWithTimeout(TestConsumer);
>>> +}
>>> +
>>> +TEST(DirectoryWatcherTest, DeleteWatchedDir) {
>>> + DirectoryWatcherTestFixture fixture;
>>> +
>>> + VerifyingConsumer TestConsumer{
>>> + {},
>>> + {{DirectoryWatcher::Event::EventKind::WatchedDirRemoved, ""},
>>> + {DirectoryWatcher::Event::EventKind::WatcherGotInvalidated,
>>> ""}}};
>>> +
>>> + auto DW = DirectoryWatcher::create(
>>> + fixture.TestWatchedDir,
>>> + [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
>>> + bool IsInitial) {
>>> + TestConsumer.consume(Events, IsInitial);
>>> + },
>>> + /*waitForInitialSync=*/true);
>>> +
>>> + remove_directories(fixture.TestWatchedDir);
>>> +
>>> + checkEventualResultWithTimeout(TestConsumer);
>>> +}
>>> +
>>> +TEST(DirectoryWatcherTest, InvalidatedWatcher) {
>>> + DirectoryWatcherTestFixture fixture;
>>> +
>>> + VerifyingConsumer TestConsumer{
>>> + {},
>>> {{DirectoryWatcher::Event::EventKind::WatcherGotInvalidated, ""}}};
>>> +
>>> + {
>>> + auto DW = DirectoryWatcher::create(
>>> + fixture.TestWatchedDir,
>>> + [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
>>> + bool IsInitial) {
>>> + TestConsumer.consume(Events, IsInitial);
>>> + },
>>> + /*waitForInitialSync=*/true);
>>> + } // DW is destructed here.
>>> +
>>> + checkEventualResultWithTimeout(TestConsumer);
>>> +}
>>> +
>>> +TEST(DirectoryWatcherTest, ChangeMetadata) {
>>> + DirectoryWatcherTestFixture fixture;
>>> + fixture.addFile("a");
>>> +
>>> + VerifyingConsumer TestConsumer{
>>> + {{DirectoryWatcher::Event::EventKind::Modified, "a"}},
>>> + // We don't expect any notification for file having access
>>> file changed.
>>> + {},
>>> + // Given the timing we are ok with receiving the duplicate event.
>>> + {{DirectoryWatcher::Event::EventKind::Modified, "a"}}};
>>> +
>>> + auto DW = DirectoryWatcher::create(
>>> + fixture.TestWatchedDir,
>>> + [&TestConsumer](llvm::ArrayRef<DirectoryWatcher::Event> Events,
>>> + bool IsInitial) {
>>> + TestConsumer.consume(Events, IsInitial);
>>> + },
>>> + /*waitForInitialSync=*/true);
>>> +
>>> + { // Change access and modification time of file a.
>>> + Expected<file_t> HopefullyTheFD =
>>> llvm::sys::fs::openNativeFileForWrite(
>>> + fixture.getPathInWatched("a"), CD_OpenExisting, OF_None);
>>> + if (!HopefullyTheFD) {
>>> + llvm::outs() << HopefullyTheFD.takeError();
>>> + }
>>> +
>>> + const int FD = HopefullyTheFD.get();
>>> + const TimePoint<> NewTimePt =
>>> + std::chrono::system_clock::now() - std::chrono::minutes(1);
>>> +
>>> + std::error_code setTimeRes =
>>> + llvm::sys::fs::setLastAccessAndModificationTime(FD, NewTimePt,
>>> + NewTimePt);
>>> + assert(!setTimeRes);
>>> + }
>>> +
>>> + checkEventualResultWithTimeout(TestConsumer);
>>> +}
>>>
>>>
>>> _______________________________________________
>>> cfe-commits mailing list
>>> cfe-commits at lists.llvm.org <mailto:cfe-commits at lists.llvm.org>
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
>>>
>
More information about the cfe-commits
mailing list