[cfe-dev] [PATCH] [Driver] Add -ivfsoverlay-lib option to load VFS from shared library

[Andrew Savonichev via cfe-dev]

Hi Volodymyr,

> Can you preprocess files with code-for-external-tool as a separate
> step during the build? And then the compiler treats #include as usual.

The problem is that the external tool cannot pre-process all files in
advance, because it doesn't know what files will be #included. And I
cannot run the standard C preprocessor before the external tool, because
'raw' files are not suitable for it.

So an ideal solution would be if the tool and the standard C
preprocessor run one after another on each file.

Anyway, my use case is not the best example of how this option can be
used. I do not encourage anybody to write another preprocessor for C :)

> My main concern is that this feature seems to be pretty heavy-weight
> and I want to make sure there are enough use cases and they are
> compelling enough to maintain this feature in the future.

On the other hand, we already have the -ivfsoverlay option to remap
files using YAML config, which is using VFS too. It is more specific and
cannot be easily extended. With -ivfsoverlay-lib this functionality can
be implemented as an external library.

So, this feature is obviously not something that will be used a lot, but
it may be useful for tooling. After all, there are many use cases for
FUSE, so some of them may be applicable for clang as well.

> For example, if you write preprocessed files to the disk, you’ll get
> reasonable debugger support for free. While with pure in-memory VFS,
> it is more challenging.

Agree.

-- 
Andrew

Volodymyr Sapsai writes:

> Can you preprocess files with code-for-external-tool as a separate step during the build? And then the compiler treats #include as usual.
>
> My main concern is that this feature seems to be pretty heavy-weight and I want to make sure there are enough use cases and they are compelling enough to maintain this feature in the future. I might sound too conservative and too negativistic but that shouldn’t discourage you.
>
> VFS is more portable between different operating systems but not between different tools. For example, if you write preprocessed files to the disk, you’ll get reasonable debugger support for free. While with pure in-memory VFS, it is more challenging.
>
> Thanks,
> Volodymyr
>
>> On Jun 4, 2018, at 03:40, Andrew Savonichev <andrew.savonichev at intel.com> wrote:
>> 
>> Hi Volodymyr,
>> 
>> I'm trying to setup a pre-processing of input files: input source file
>> contains a region, which is not a valid C code, but external tool can
>> consume it and replace it with meaningful code. This replacement may
>> have standard C preprocessor directives (e.g. #include), so the tool
>> should run before C preprocessor and follow #include directives,
>> preprocessing #included files as well.
>> 
>> With VFS this is pretty easy to do: you just implement a
>> FileSystem::openFileForRead function which returns a file after
>> pre-processing, and Clang calls this function whenever it needs a file.
>> 
>> In general, VFS is similar to Filesystem in Userspace (FUSE): it allows
>> you to read files from memory instead of a real file system, but it is
>> more portable than FUSE.
>> 
>>> Hello Andrew,
>>> 
>>> That is an interesting idea with providing VFS as a shared
>>> library. Can you please provide more details about what you are trying
>>> to accomplish with custom VFS? It is hard to judge about merits of the
>>> approach without knowing the intended usage.
>>> 
>>> Thanks,
>>> Volodymyr
>>> 
>>>> On May 28, 2018, at 04:23, Andrew Savonichev via cfe-dev <cfe-dev at lists.llvm.org> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> Recently I was trying to hijack into clang's file management to do a
>>>> custom pre-processing of input files, and found that VirtualFileSystem
>>>> (VFS) is pretty flexible and can be extended to support any weird
>>>> use-cases related to files: preprocessing, in-memory files, code
>>>> generation, etc.
>>>> 
>>>> Unfortunately, it is not exposed from clang driver, and you have to
>>>> patch clang to add your own custom VFS. Thus I added an option to clang
>>>> cc1 driver that loads a said shared library, gets a VFS and pushes it on
>>>> top of real file system.
>>>> 
>>>> The patch is on the review: https://reviews.llvm.org/D47190
>>>> 
>>>> Any commets/suggestions are welcome.
>>>> 
>>>> --
>>>> Andrew

--------------------------------------------------------------------
Joint Stock Company Intel A/O
Registered legal address: Krylatsky Hills Business Park,
17 Krylatskaya Str., Bldg 4, Moscow 121614,
Russian Federation

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.