[PATCH] D31364: LTO: Reduce memory consumption by creating an in-memory symbol table for InputFiles. NFCI.

[Peter Collingbourne via llvm-commits]

On Tue, Mar 28, 2017 at 7:10 AM, Rafael Espíndola <
rafael.espindola at gmail.com> wrote:

> > ================
> > Comment at: llvm/include/llvm/Object/IRSymtab.h:31
> > +
> > +typedef support::ulittle32_t Word;
> > +
> > ----------------
> > rafael wrote:
> >> why do you use this for Symbol? If we are storing something by value it
> is better to just use a uint32_t.
> > If we eventually want to write these data structures out to files I
> guess we will need to specify an endianness. The plan I have in mind is:
> > 1. define data structures that can be potentially written to files, but
> use them only as an in-memory format to start with (that is what this patch
> does)
> > 2. potentially improve them in-tree (e.g. by sharing the string table
> between the module and the symbol table)
> > 3. flip the switch and start writing them to files
>
> The two step solution is very reasonable. I see that you currently
> write to memory and read back, so it should be easy to write and read
> from disk, and yes, we have to specify the endianness for a disk
> format. But I think the endianness should be taken care of at the
> read/write functions. This is similar to ELF:



* The on disk format has an specific endianness.
> * A pointer to a mmap area should use something like support::ulittle32_t*.
> * But it should be read once producing a uint32_t.
>

I agree that the pointers to the mmap area should use something low level
that exposes the endianness, and that is what I intend the structs that I
am introducing in this patch to be used for. Although the reader code could
unpack the data structures manually out of a support::ulittle32_t* pointer,
I think the structs help with readability.

Essentially the data structures I am adding are the equivalent of the
definitions in llvm/Object/ELFTypes.h for ELF.

On top of that, I think it seems reasonable to build a more convenient API
that reads data once and uses types like uint32_t, and it could even be
part of this patch. I'll start working on that.

>From the work on lld I think a reasonable rule of the thumb is then
> that the endianness tagged types like support::ulittle32_t should only
> be used in pointers.
>

I guess you're referring to for example how we read fields once from an
Elf_Sym and copy them into a SymbolBody. If so, that makes perfect sense to
me.

Thanks,
-- 
-- 
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20170328/a8a6f1ae/attachment.html>