[llvm-dev] RFC: APIs for bitcode files containing multiple modules

[Peter Collingbourne via llvm-dev]

On Tue, Oct 25, 2016 at 8:36 PM, Mehdi Amini <mehdi.amini at apple.com> wrote:

>
> On Oct 25, 2016, at 6:28 PM, Peter Collingbourne <peter at pcc.me.uk> wrote:
>
> Hi all,
>
> As mentioned in my recent RFC entitled "RFC: a more detailed design for
> ThinLTO + vcall CFI" I would like to introduce the ability for bitcode
> files to contain multiple modules. In https://reviews.llvm.org/D24786 I
> took a step towards that by proposing a change to the module format so that
> the block info block is stored at the top level. The next step is to think
> about what the API would look like for reading and writing multiple modules.
>
> Here's what I have in mind. To create a multi-module bitcode file, you
> would create a BitcodeWriter object and add modules to it:
>
> BitcodeWriter W(OS);
> W.addModule(M1);
> W.addModule(M2);
> W.write();
>
>
> That requires the two modules to lives longer than the bitcode write, the
> API could be:
>
> BitcodeWriter W(OS);
> W.writeModule(M1);
> // delete M1
> // ...
> // create M2
> W.writeModule(M2);
>
> (Maybe you had this in mind, but the API naming didn’t reflect it so I’m
> not sure).
>

In the API I prototyped, I took the maximum BitsRequiredForTypeIndices
value from all the modules, and used it to produce the abbreviations for
the top level block info block (without this I was seeing "Unexpected
abbrev ordering!" errors in the bitcode writer as a result of emitting the
"same" abbreviation multiple times). That would have required us to keep
the modules around until the call to write(). However, let me revisit this,
because it does not seem necessary (i.e. we can just continue to emit block
info blocks within the module block except with different abbreviation
numbers for each module).

> Reading a multi-module bitcode file would be supported with a
> BitcodeReader class. Each of the functional reader APIs in ReaderWriter.h
> would have a member function on BitcodeReader. We would also have a next()
> member function which would move to the next module in the file. For
> example:
>
> BitcodeReader R(MBRef);
> Expected<bool> B = R.hasGlobalValueSummary();
> std::unique_ptr<Module> M1 = R.getLazyModule(Ctx); // lazily load the
> first module
> R.next();
> std::unique_ptr<Module> M2 = R.parseBitcodeFile(Ctx); // eagerly load the
> second module
>
>
>
> That makes the API quite stateful, you may have good implementation reason
> for this, but they’re not clear to me.
> I rather see the bitcode reader as a random access container, iterating
> over modules.
>

Random access seems reasonable to me as well. I will see how feasible that
is.

Thanks,
-- 
-- 
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20161026/a5465bf1/attachment.html>