[cfe-dev] RFC: A virtual file system for clang

[Maurizio Vitale]

What are the pros of OverlayFileSystem inside clang compared to unionFS (or
aufs) of a RO directory (possibly maintained by an external caching daemon)
with the state of the repository and a RW directory with the changes the
user is working on?

Thanks,

  Maurizio


On Thu, Feb 6, 2014 at 6:04 PM, Ben Langmuir <blangmuir at apple.com> wrote:

> Hi all,
>
> I’ve been hacking on a virtual file system for clang and this seemed like
> the right time to start getting some feedback.  Briefly, the idea is to
> interpose a virtual file system layer between llvm::sys::fs and Clang’s
> FileManager that allows us to mix virtual files/links/etc. with the ‘real’
> file system in a general way.
>
>
> Motivation
>
> The use case that I have in mind is to allow a build system to provide a
> file/directory layout to clang without having to construct it “for real” on
> disk.  For example, I am building a project containing two modules, and
> module A imports module B. It would be useful if we could bundle up the
> headers and module.map file for module B from wherever they may exist in
> the source directories and provide clang with a notion of the file layout
> of B _as it will be installed_.   Right now, I know of two existing ways to
> accomplish this:
>
> 1) Copy the files into a fake installation during build.  This is
> unsatisfying, as it requires tracking and copying files every time they are
> changed.  And diagnostics, debug info, etc. do not refer back to the
> original source file.
>
> 2) Header maps provide this functionality for header files.  However,
> header maps work from within the header search logic, which does not extend
> well to other kinds of files.  They are also insufficient for bundling
> modules, as clang needs to see the framework for the module laid out as
> described in the module map.
>
>
> Description
>
> The idea is to abstract the view of the file system using an
> AbstractFileSystem class that mimics the llvm::sys::fs interface:
>
> class AbstractFileSystem {
> public:
>   class Status { … };
>   // openFileForRead
>   // status, and maybe ‘stat'
>   // recursive iteration
>   // getBuffer
>   // getBufferForOpenFile
>   // recursive directory iteration
> };
>
> that can be implemented by any concrete file system that we want. Clients
> that want to lookup files/directories (notably the FileManager) can operate
> on an AbstractFileSystem object. One leaky part of this interface is that
> clients that need to care whether they are working with a ‘real path’ will
> need to explicitly ask for it.  For example, debug information and
> diagnostics should ask for the real path.  I suggest putting that
> information into the AbstractFIleSystem::Status object.
>
> Some non-goals (at least for a first iteration):
> 1) File system modification operations (create_directory, rename, etc.).
>  Clients will continue to use the real file system for these operations,
> and we don’t intend to detect any conflicts this might create.
> 2) Completely virtual file buffers that do not exist on disk.
>
> One implementation of the AbstractFileSystem interface would be a wrapper
> over the ‘real’ file system, which would just defer to llvm::sys::fs.
>
> class RealFileSystem : public AbstractFileSystem { … };
>
> And to provide a unified view of the file system, we can create an overlay
> file system, similar to [1].
>
> class OverlayFileSystem : public AbstractFileSystem { … };
>
> To support a build system providing clang with a virtual file layout, we
> could add an option to clang that accepts a file describing the layout of a
> virtual file system.  In a first iteration, this could be a simple json
> file describing the mapping from virtual paths to real paths, and a
> corresponding class VFSFromJSONFile : public AbstractFileSystem.  Later we
> can evolve a more efficient binary format for this.  In addition we should
> provide functions in libclang to produce these files.
>
>
> I would appreciate any feedback you might have,
>
> Ben
>
> [1]
> https://git.kernel.org/cgit/linux/kernel/git/mszeredi/vfs.git/plain/Documentation/filesystems/overlayfs.txt?h=overlayfs.current
>
> Also, thanks to everyone who has already given me feedback thus far.
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20140522/fb9a396b/attachment.html>