[cfe-dev] RFC: A virtual file system for clang

Fri May 23 06:42:57 PDT 2014

On Fri, May 23, 2014 at 3:37 PM, Maurizio Vitale <mrz.vtl at gmail.com> wrote:

> I was not really proposing anything, just trying to understand the benefit
> of a virtual file system inside clang as opposed to an unmodified compiler
> + something like
> http://mike-bland.com/2012/10/01/tools.html#tools-blaze-forge-srcfs-objfs,
> that you're probably familiar with :-)
>

That's easy - if you want to run clang inside something that doesn't have
fuse, say, an IDE, or a MapReduce ;), and you want to run clang in parallel
on many files, it's much simpler to not have to set up something on disk
(even if ram-disk), and set up potentially huge directory structures.

And on the other end, some things are just not possible to do well with a
fuse based solution - for example intercepting all file reads in order to
store all files involved in the compilation, so you can store it and later
replay it exactly the same way it was (for added benefit, you can do
cross-platform record/replay runs).

Cheers,
/Manuel

>
>
> On Fri, May 23, 2014 at 3:31 AM, Manuel Klimek <klimek at google.com> wrote:
>
>> On Thu, May 22, 2014 at 10:42 PM, Maurizio Vitale <mrz.vtl at gmail.com>wrote:
>>
>>> What are the pros of OverlayFileSystem inside clang compared to unionFS
>>> (or aufs) of a RO directory (possibly maintained by an external caching
>>> daemon) with the state of the repository and a RW directory with the
>>> changes the user is working on?
>>>
>>
>> I'm not sure I understand what you're getting at completely. Can you
>> elaborate or provide links to what you propose?
>>
>>
>>>
>>> Thanks,
>>>
>>>   Maurizio
>>>
>>>
>>> On Thu, Feb 6, 2014 at 6:04 PM, Ben Langmuir <blangmuir at apple.com>wrote:
>>>
>>>> Hi all,
>>>>
>>>> I’ve been hacking on a virtual file system for clang and this seemed
>>>> like the right time to start getting some feedback.  Briefly, the idea is
>>>> to interpose a virtual file system layer between llvm::sys::fs and Clang’s
>>>> FileManager that allows us to mix virtual files/links/etc. with the ‘real’
>>>> file system in a general way.
>>>>
>>>>
>>>> Motivation
>>>>
>>>> The use case that I have in mind is to allow a build system to provide
>>>> a file/directory layout to clang without having to construct it “for real”
>>>> on disk.  For example, I am building a project containing two modules, and
>>>> module A imports module B. It would be useful if we could bundle up the
>>>> headers and module.map file for module B from wherever they may exist in
>>>> the source directories and provide clang with a notion of the file layout
>>>> of B _as it will be installed_.   Right now, I know of two existing ways to
>>>> accomplish this:
>>>>
>>>> 1) Copy the files into a fake installation during build.  This is
>>>> unsatisfying, as it requires tracking and copying files every time they are
>>>> changed.  And diagnostics, debug info, etc. do not refer back to the
>>>> original source file.
>>>>
>>>> 2) Header maps provide this functionality for header files.  However,
>>>> header maps work from within the header search logic, which does not extend
>>>> well to other kinds of files.  They are also insufficient for bundling
>>>> modules, as clang needs to see the framework for the module laid out as
>>>> described in the module map.
>>>>
>>>>
>>>> Description
>>>>
>>>> The idea is to abstract the view of the file system using an
>>>> AbstractFileSystem class that mimics the llvm::sys::fs interface:
>>>>
>>>> class AbstractFileSystem {
>>>> public:
>>>>   class Status { … };
>>>>   // openFileForRead
>>>>   // status, and maybe ‘stat'
>>>>   // recursive iteration
>>>>   // getBuffer
>>>>   // getBufferForOpenFile
>>>>   // recursive directory iteration
>>>> };
>>>>
>>>> that can be implemented by any concrete file system that we want.
>>>> Clients that want to lookup files/directories (notably the FileManager) can
>>>> operate on an AbstractFileSystem object. One leaky part of this interface
>>>> is that clients that need to care whether they are working with a ‘real
>>>> path’ will need to explicitly ask for it.  For example, debug information
>>>> and diagnostics should ask for the real path.  I suggest putting that
>>>> information into the AbstractFIleSystem::Status object.
>>>>
>>>> Some non-goals (at least for a first iteration):
>>>> 1) File system modification operations (create_directory, rename,
>>>> etc.).  Clients will continue to use the real file system for these
>>>> operations, and we don’t intend to detect any conflicts this might create.
>>>> 2) Completely virtual file buffers that do not exist on disk.
>>>>
>>>> One implementation of the AbstractFileSystem interface would be a
>>>> wrapper over the ‘real’ file system, which would just defer to
>>>> llvm::sys::fs.
>>>>
>>>> class RealFileSystem : public AbstractFileSystem { … };
>>>>
>>>> And to provide a unified view of the file system, we can create an
>>>> overlay file system, similar to [1].
>>>>
>>>> class OverlayFileSystem : public AbstractFileSystem { … };
>>>>
>>>> To support a build system providing clang with a virtual file layout,
>>>> we could add an option to clang that accepts a file describing the layout
>>>> of a virtual file system.  In a first iteration, this could be a simple
>>>> json file describing the mapping from virtual paths to real paths, and a
>>>> corresponding class VFSFromJSONFile : public AbstractFileSystem.  Later we
>>>> can evolve a more efficient binary format for this.  In addition we should
>>>> provide functions in libclang to produce these files.
>>>>
>>>>
>>>> I would appreciate any feedback you might have,
>>>>
>>>> Ben
>>>>
>>>> [1]
>>>> https://git.kernel.org/cgit/linux/kernel/git/mszeredi/vfs.git/plain/Documentation/filesystems/overlayfs.txt?h=overlayfs.current
>>>>
>>>> Also, thanks to everyone who has already given me feedback thus far.
>>>>
>>>> _______________________________________________
>>>> cfe-dev mailing list
>>>> cfe-dev at cs.uiuc.edu
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> cfe-dev mailing list
>>> cfe-dev at cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20140523/a08a8846/attachment.html>