[cfe-dev] RFC: A virtual file system for clang

Fri Feb 7 11:03:10 PST 2014

On Fri, Feb 7, 2014 at 6:25 PM, Ben Langmuir <blangmuir at apple.com> wrote:

> Hi Manuel,
>
> On Feb 7, 2014, at 1:01 AM, Manuel Klimek <klimek at google.com> wrote:
>
> Hi Ben,
>
> ah, the good old discussion :) To get some more context in, see
> http://lists.cs.uiuc.edu/pipermail/llvmdev/2011-November/045657.html (the
> discussion goes into December, which the web interface doesn't seem to be
> able to cope with). More comments inline.
>
>
> Looks like I have some reading to do :)
>
>
> On Fri, Feb 7, 2014 at 12:04 AM, Ben Langmuir <blangmuir at apple.com> wrote:
>
>> Hi all,
>>
>> I’ve been hacking on a virtual file system for clang and this seemed like
>> the right time to start getting some feedback.  Briefly, the idea is to
>> interpose a virtual file system layer between llvm::sys::fs and Clang’s
>> FileManager that allows us to mix virtual files/links/etc. with the ‘real’
>> file system in a general way.
>>
>>
>> Motivation
>>
>> The use case that I have in mind is to allow a build system to provide a
>> file/directory layout to clang without having to construct it “for real” on
>> disk.  For example, I am building a project containing two modules, and
>> module A imports module B. It would be useful if we could bundle up the
>> headers and module.map file for module B from wherever they may exist in
>> the source directories and provide clang with a notion of the file layout
>> of B _as it will be installed_.   Right now, I know of two existing ways to
>> accomplish this:
>>
>> 1) Copy the files into a fake installation during build.  This is
>> unsatisfying, as it requires tracking and copying files every time they are
>> changed.  And diagnostics, debug info, etc. do not refer back to the
>> original source file.
>>
>> 2) Header maps provide this functionality for header files.  However,
>> header maps work from within the header search logic, which does not extend
>> well to other kinds of files.  They are also insufficient for bundling
>> modules, as clang needs to see the framework for the module laid out as
>> described in the module map.
>>
>
> By "Header map", do you mean the RemappedFileBuffers in
> PreprocessorOptions? This seems to actually be enough for us to map files
> even in the presence of modules (at least in C++, I don't know how
> different Framework handling in Obj-C is here).
>
>
> I mean HeaderMap in the HeaderSearchOptions, which is somewhat similar but
> is considered only during header search, not at the file system layer.
>
> The RemappedFileBuffers is closer to the infrastructure we would need to
> map files, but it doesn’t allow recursive directory iteration, which is
> another requirement for handling modules.  We could always teach
> FileManager to support more operations, but increasing the duplication
> between FileManager and sys::fs seems like a bad idea.
>

I'm fully with you here, just wanted to make sure all related issues are
clear...

> Description
>>
>> The idea is to abstract the view of the file system using an
>> AbstractFileSystem class that mimics the llvm::sys::fs interface:
>>
>> class AbstractFileSystem {
>> public:
>>   class Status { … };
>>   // openFileForRead
>>   // status, and maybe ‘stat'
>>   // recursive iteration
>>   // getBuffer
>>   // getBufferForOpenFile
>>   // recursive directory iteration
>> };
>>
>> that can be implemented by any concrete file system that we want. Clients
>> that want to lookup files/directories (notably the FileManager) can operate
>> on an AbstractFileSystem object. One leaky part of this interface is that
>> clients that need to care whether they are working with a ‘real path’ will
>> need to explicitly ask for it.  For example, debug information and
>> diagnostics should ask for the real path.  I suggest putting that
>> information into the AbstractFIleSystem::Status object.
>>
>> Some non-goals (at least for a first iteration):
>> 1) File system modification operations (create_directory, rename, etc.).
>>  Clients will continue to use the real file system for these operations,
>> and we don’t intend to detect any conflicts this might create.
>> 2) Completely virtual file buffers that do not exist on disk.
>>
>
> I'd vote for making that an explicit goal; two reasons:
> 1. I don't think it'll make a first iteration harder to implement
> 2. saying that we'll do things like that later will almost certainly make
> it super-hard to do later
>
> For us, the ability to have virtual file buffers that do not exist on disk
> is one of the core requirements we have for all our tools; I think in a
> more and more network-based world this will also become more necessary in
> general in the future.
>
>
> How do you imagine changing clients that currently expect to be able to
> get a file descriptor?  Do you remove that concept and provide only
> higher-level APIs, like “getBuffer” and “getRawOstream”, or create some
> opaque file descriptor that can be returned from openFileForReading and
> openFileForWriting?  The latter seems like it doesn’t need to be built in
> from the start, since we can continue to have the usual file descriptor
> APIs, and update clients later when we change what a file descriptor is.
>  That’s what I was imagining, but you may have a better idea.  Also, even
> if adding fully virtual files doesn’t make a first iteration harder to
> implement, what about testing it?
>

Well, that's an interesting question :) So, do you want a virtual file
system that we can plug below the file manager pretty much "as-is", and it
just works? In that case, I'd guess we need to do the latter. I'm also not
sure which clients we might be able to convert later.

If you don't want to provide a vfs below the file manager, where do you
want to use it? Currently, all Tooling/ stuff relies on being able to use
the file overlaying logic to inject into PPOptions / SourceManager /
FileManager to allow (nearly) fully file system independent replays of
compilations. Would you propose to break this behavior as part of the
transition?

> One implementation of the AbstractFileSystem interface would be a wrapper
>> over the ‘real’ file system, which would just defer to llvm::sys::fs.
>>
>> class RealFileSystem : public AbstractFileSystem { … };
>>
>> And to provide a unified view of the file system, we can create an
>> overlay file system, similar to [1].
>>
>> class OverlayFileSystem : public AbstractFileSystem { … };
>>
>> To support a build system providing clang with a virtual file layout, we
>> could add an option to clang that accepts a file describing the layout of a
>> virtual file system.  In a first iteration, this could be a simple json
>> file describing the mapping from virtual paths to real paths, and a
>> corresponding class VFSFromJSONFile : public AbstractFileSystem.  Later we
>> can evolve a more efficient binary format for this.  In addition we should
>> provide functions in libclang to produce these files.
>>
>
> The rest sounds generally good.
>
> One concern I have that has not been brought up is the old problem of
> making file operations relative to a directory entry that is taken once at
> the start of an operation (imagine starting a compilation from a symlinked
> directory, and somebody changing the link). I think this will probably not
> a goal for phase 1, but would be nice if we could keep it in the back of
> our heads ;)
>
>
> Can you expand on this? Dumb question: why can’t we just ask the OS for a
> canonical path and work from that?
>

The main problem is less whether the path is canonical than that an
unrelated happenstance in the file system might lead to inconsistencies in
a compile step.

Imagine a source tree in src-head and one in src-branch.
$ ln -s src-head src
$ cd src
$ clang file.cc &
$ cd ..
$ rm src
$ ln -s src-branch src

Now if the clang process takes a while to run and resolves files it opens
via "absolute" paths rather than relative to the current working dir file
entry, it can see inconsistent state (for example, get a header from
src-branch instead of from src-head).

Cheers,
/Manuel

> Thanks,
>
> Ben
>
>
>
> Cheers,
> /Manuel
>
>
>>
>>
>> I would appreciate any feedback you might have,
>>
>> Ben
>>
>> [1]
>> https://git.kernel.org/cgit/linux/kernel/git/mszeredi/vfs.git/plain/Documentation/filesystems/overlayfs.txt?h=overlayfs.current
>>
>> Also, thanks to everyone who has already given me feedback thus far.
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20140207/113f0456/attachment.html>