[llvm-dev] RFC: Add an llvm::vfs::OutputManager to allow Clang to virtualize compiler outputs

[Duncan P. N. Exon Smith via llvm-dev]

TL;DR: Let's virtualize compiler outputs in Clang. These patches would get us started:
- https://reviews.llvm.org/D95501 <https://reviews.llvm.org/D95501> Add llvm::vfs::OutputManager
- https://reviews.llvm.org/D95502 <https://reviews.llvm.org/D95502> Initial adoption of llvm::vfs::OutputManager in Clang.

Questions for the reader
- Should we virtualize compiler outputs in Clang? (Hint: yes.)
	- Does this support belong in LLVM? (I think it does, so that non-Clang tools can easily reuse it.)
	- Is `llvm::vfs::` a reasonable namespace? (If not, suggestions? I think `llvm::` itself is too broad.)
- Do you have a use case that this won't address well?
	- Should that be fixed in the initial patch, or could this be evolved in-tree to address that?
- Any other major concerns / suggestions?
- If you think the above patches should be split up for initial review / commit, how?

(Other feedback welcome too!)

Longer version
There are a number of use cases for capturing compiler outputs, which I'm hoping this proposal is a step toward addressing.

- Tooling wants to capture outputs directly, without going through the filesystem.
	- Sometimes, tertiary outputs can be ignored, or still need to be written to disk.
- clang-scan-deps is using a form of stripped down "implicit" modules to determine which modules need to be built explicitly. It doesn't really want to be using the on-disk module cache—in-memory would be better.
- clang's ModuleManager manually maintains an in-memory modules cache for implicit modules. This involves copying the PCM outputs into memory. It'd be better for these modules to be file-backed, instead of copies of the stream.

The patch has a bunch of details written / tested (https://reviews.llvm.org/D95501 <https://reviews.llvm.org/D95501>). Here are the high-level structures in the design:

- OutputManager—a shared manager for creating outputs without knowing about the storage.
- OutputConfig—configuration set on the OutputManager that can be (partially) overridden for specific outputs.
- Output—opaque object with a raw_pwrite_stream, output path, and `erase`/`close` functionality. Internally, it has a linked list of output destinations.
- OutputBackend—abstract interface installed in an OutputManager to create the "guts" of an output. While an OutputManager only has one installed, these can be layered / forked / mirrored.
- OutputDestination—abstract interface paired with an OutputBackend, whose lifetime is managed by an Output.
- ContentBuffer—actual content to allow efficient use of data by multiple backends. For example, the installed backend is a mirror between an on-disk and in-memory backend, the in-memory backend will either get the content moved directly into an llvm::MemoryBuffer, or a just-written mmap'ed file.

The patch includes a few backends:

- NullOutputBackend, for ignoring all backends.
- OnDiskOutputBackend, for writing to disk (the default), initially based on the logic in `clang::CompilerInstance`.
- InMemoryOutputBackend, for writing to an `InMemoryFileSystem`.
- MirroringOutputBackend, for writing to multiple backends. OutputDestination's API is designed around supporting this.
- FilteringOutputBackend, for filtering which outputs get written to the underlying backend.

Why doesn't this inherit from llvm::vfs::FileSystem?
Separating the input and output abstractions seems a bit cleaner. It's easy enough to join them, when useful: e.g., write to an `InMemoryFileSystem` (via an `InMemoryOutputBackend`) and install this same FS in the `FileManager` (maybe indirectly via an `OverlayFileSystem`).

Other work in the area
See also: https://reviews.llvm.org/D78058 <https://reviews.llvm.org/D78058> (thanks to Marc Rasi for posting that patch, and to Sam McCall for some feedback on an earlier version of this proposal).

Thanks for reading!
Duncan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210126/f608bed8/attachment.html>