[cfe-dev] RFC: A virtual file system for clang

[Manuel Klimek]

On Thu, May 22, 2014 at 10:42 PM, Maurizio Vitale <mrz.vtl at gmail.com> wrote:

> What are the pros of OverlayFileSystem inside clang compared to unionFS
> (or aufs) of a RO directory (possibly maintained by an external caching
> daemon) with the state of the repository and a RW directory with the
> changes the user is working on?
>

I'm not sure I understand what you're getting at completely. Can you
elaborate or provide links to what you propose?


>
> Thanks,
>
>   Maurizio
>
>
> On Thu, Feb 6, 2014 at 6:04 PM, Ben Langmuir <blangmuir at apple.com> wrote:
>
>> Hi all,
>>
>> I’ve been hacking on a virtual file system for clang and this seemed like
>> the right time to start getting some feedback.  Briefly, the idea is to
>> interpose a virtual file system layer between llvm::sys::fs and Clang’s
>> FileManager that allows us to mix virtual files/links/etc. with the ‘real’
>> file system in a general way.
>>
>>
>> Motivation
>>
>> The use case that I have in mind is to allow a build system to provide a
>> file/directory layout to clang without having to construct it “for real” on
>> disk.  For example, I am building a project containing two modules, and
>> module A imports module B. It would be useful if we could bundle up the
>> headers and module.map file for module B from wherever they may exist in
>> the source directories and provide clang with a notion of the file layout
>> of B _as it will be installed_.   Right now, I know of two existing ways to
>> accomplish this:
>>
>> 1) Copy the files into a fake installation during build.  This is
>> unsatisfying, as it requires tracking and copying files every time they are
>> changed.  And diagnostics, debug info, etc. do not refer back to the
>> original source file.
>>
>> 2) Header maps provide this functionality for header files.  However,
>> header maps work from within the header search logic, which does not extend
>> well to other kinds of files.  They are also insufficient for bundling
>> modules, as clang needs to see the framework for the module laid out as
>> described in the module map.
>>
>>
>> Description
>>
>> The idea is to abstract the view of the file system using an
>> AbstractFileSystem class that mimics the llvm::sys::fs interface:
>>
>> class AbstractFileSystem {
>> public:
>>   class Status { … };
>>   // openFileForRead
>>   // status, and maybe ‘stat'
>>   // recursive iteration
>>   // getBuffer
>>   // getBufferForOpenFile
>>   // recursive directory iteration
>> };
>>
>> that can be implemented by any concrete file system that we want. Clients
>> that want to lookup files/directories (notably the FileManager) can operate
>> on an AbstractFileSystem object. One leaky part of this interface is that
>> clients that need to care whether they are working with a ‘real path’ will
>> need to explicitly ask for it.  For example, debug information and
>> diagnostics should ask for the real path.  I suggest putting that
>> information into the AbstractFIleSystem::Status object.
>>
>> Some non-goals (at least for a first iteration):
>> 1) File system modification operations (create_directory, rename, etc.).
>>  Clients will continue to use the real file system for these operations,
>> and we don’t intend to detect any conflicts this might create.
>> 2) Completely virtual file buffers that do not exist on disk.
>>
>> One implementation of the AbstractFileSystem interface would be a wrapper
>> over the ‘real’ file system, which would just defer to llvm::sys::fs.
>>
>> class RealFileSystem : public AbstractFileSystem { … };
>>
>> And to provide a unified view of the file system, we can create an
>> overlay file system, similar to [1].
>>
>> class OverlayFileSystem : public AbstractFileSystem { … };
>>
>> To support a build system providing clang with a virtual file layout, we
>> could add an option to clang that accepts a file describing the layout of a
>> virtual file system.  In a first iteration, this could be a simple json
>> file describing the mapping from virtual paths to real paths, and a
>> corresponding class VFSFromJSONFile : public AbstractFileSystem.  Later we
>> can evolve a more efficient binary format for this.  In addition we should
>> provide functions in libclang to produce these files.
>>
>>
>> I would appreciate any feedback you might have,
>>
>> Ben
>>
>> [1]
>> https://git.kernel.org/cgit/linux/kernel/git/mszeredi/vfs.git/plain/Documentation/filesystems/overlayfs.txt?h=overlayfs.current
>>
>> Also, thanks to everyone who has already given me feedback thus far.
>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>
>>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20140523/27c4a5c1/attachment.html>