[cfe-dev] RFC: A virtual file system for clang

[Maurizio Vitale]

I was not really proposing anything, just trying to understand the benefit
of a virtual file system inside clang as opposed to an unmodified compiler
+ something like
http://mike-bland.com/2012/10/01/tools.html#tools-blaze-forge-srcfs-objfs,
that you're probably familiar with :-)


On Fri, May 23, 2014 at 3:31 AM, Manuel Klimek <klimek at google.com> wrote:

> On Thu, May 22, 2014 at 10:42 PM, Maurizio Vitale <mrz.vtl at gmail.com>wrote:
>
>> What are the pros of OverlayFileSystem inside clang compared to unionFS
>> (or aufs) of a RO directory (possibly maintained by an external caching
>> daemon) with the state of the repository and a RW directory with the
>> changes the user is working on?
>>
>
> I'm not sure I understand what you're getting at completely. Can you
> elaborate or provide links to what you propose?
>
>
>>
>> Thanks,
>>
>>   Maurizio
>>
>>
>> On Thu, Feb 6, 2014 at 6:04 PM, Ben Langmuir <blangmuir at apple.com> wrote:
>>
>>> Hi all,
>>>
>>> I’ve been hacking on a virtual file system for clang and this seemed
>>> like the right time to start getting some feedback.  Briefly, the idea is
>>> to interpose a virtual file system layer between llvm::sys::fs and Clang’s
>>> FileManager that allows us to mix virtual files/links/etc. with the ‘real’
>>> file system in a general way.
>>>
>>>
>>> Motivation
>>>
>>> The use case that I have in mind is to allow a build system to provide a
>>> file/directory layout to clang without having to construct it “for real” on
>>> disk.  For example, I am building a project containing two modules, and
>>> module A imports module B. It would be useful if we could bundle up the
>>> headers and module.map file for module B from wherever they may exist in
>>> the source directories and provide clang with a notion of the file layout
>>> of B _as it will be installed_.   Right now, I know of two existing ways to
>>> accomplish this:
>>>
>>> 1) Copy the files into a fake installation during build.  This is
>>> unsatisfying, as it requires tracking and copying files every time they are
>>> changed.  And diagnostics, debug info, etc. do not refer back to the
>>> original source file.
>>>
>>> 2) Header maps provide this functionality for header files.  However,
>>> header maps work from within the header search logic, which does not extend
>>> well to other kinds of files.  They are also insufficient for bundling
>>> modules, as clang needs to see the framework for the module laid out as
>>> described in the module map.
>>>
>>>
>>> Description
>>>
>>> The idea is to abstract the view of the file system using an
>>> AbstractFileSystem class that mimics the llvm::sys::fs interface:
>>>
>>> class AbstractFileSystem {
>>> public:
>>>   class Status { … };
>>>   // openFileForRead
>>>   // status, and maybe ‘stat'
>>>   // recursive iteration
>>>   // getBuffer
>>>   // getBufferForOpenFile
>>>   // recursive directory iteration
>>> };
>>>
>>> that can be implemented by any concrete file system that we want.
>>> Clients that want to lookup files/directories (notably the FileManager) can
>>> operate on an AbstractFileSystem object. One leaky part of this interface
>>> is that clients that need to care whether they are working with a ‘real
>>> path’ will need to explicitly ask for it.  For example, debug information
>>> and diagnostics should ask for the real path.  I suggest putting that
>>> information into the AbstractFIleSystem::Status object.
>>>
>>> Some non-goals (at least for a first iteration):
>>> 1) File system modification operations (create_directory, rename, etc.).
>>>  Clients will continue to use the real file system for these operations,
>>> and we don’t intend to detect any conflicts this might create.
>>> 2) Completely virtual file buffers that do not exist on disk.
>>>
>>> One implementation of the AbstractFileSystem interface would be a
>>> wrapper over the ‘real’ file system, which would just defer to
>>> llvm::sys::fs.
>>>
>>> class RealFileSystem : public AbstractFileSystem { … };
>>>
>>> And to provide a unified view of the file system, we can create an
>>> overlay file system, similar to [1].
>>>
>>> class OverlayFileSystem : public AbstractFileSystem { … };
>>>
>>> To support a build system providing clang with a virtual file layout, we
>>> could add an option to clang that accepts a file describing the layout of a
>>> virtual file system.  In a first iteration, this could be a simple json
>>> file describing the mapping from virtual paths to real paths, and a
>>> corresponding class VFSFromJSONFile : public AbstractFileSystem.  Later we
>>> can evolve a more efficient binary format for this.  In addition we should
>>> provide functions in libclang to produce these files.
>>>
>>>
>>> I would appreciate any feedback you might have,
>>>
>>> Ben
>>>
>>> [1]
>>> https://git.kernel.org/cgit/linux/kernel/git/mszeredi/vfs.git/plain/Documentation/filesystems/overlayfs.txt?h=overlayfs.current
>>>
>>> Also, thanks to everyone who has already given me feedback thus far.
>>>
>>> _______________________________________________
>>> cfe-dev mailing list
>>> cfe-dev at cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>>
>>>
>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20140523/0b0b3b6e/attachment.html>