[LLVMdev] RFC: ThinLTO Impementation Plan

[Eric Christopher]

On Sun, May 17, 2015 at 9:15 PM Xinliang David Li <xinliangli at gmail.com>
wrote:

> Eric, I don't think anybody would disagree to define APIs to hide details
> of container format, which can be taken for granted.
>

Good. The outline I gave should be the direction that you and Teresa are
going with this then? The proposal has been light on the details of some of
this and formats are occasionally important - a good abstract interface
will hide some of the bikeshedding that's likely to go on with an API.



>
> Such APIs are mostly internally used in thinLTO implementation. Having
> 'stable' APIs is also not the right goal, at least initially because it
> serves no real purpose. We expect them to to evolve and get refined in many
> iterations of testing (with very large apps).  Someday when there is a need
> to make the interfaces public (to be used by other clients), we can start
> to talk about stable APIs.
>
>
You seem to have misunderstood my mail. I apologize if I wasn't clear in
some way. I have no desire for an actual stable API - this would take the
place of a C API for LLVM. I'm proposing that we define an abstract enough
class API so that we can make progress along some lines without becoming
bogged down by container format details. The API can, and should be usable
by other clients, but I think you'll agree that those clients can be as
simple as a dumping tool in order to test that the container emitters are
working properly and hide enough of the details that alternate containers
can be proposed and worked on while hiding the precise implementation. It
will also go a long way to providing a self-documenting interface into the
information that the thin-lto work needs.

-eric



> thanks,
>
> David
>
> On Sun, May 17, 2015 at 2:29 PM, Eric Christopher <echristo at gmail.com>
> wrote:
>
>> Hi Teresa, David,
>>
>> I've got a proposed solution to the object/bitcode discussion below,
>> please take a look - it's a bit light on details because I'm designing an
>> API without quite enough information, but based on our discussions I think
>> this is the best way forward.
>>
>> On Fri, May 15, 2015 at 9:47 AM Xinliang David Li <xinliangli at gmail.com>
>> wrote:
>>
>>> On Fri, May 15, 2015 at 5:11 AM, Dave Bozier <seifsta at gmail.com> wrote:
>>>
>>>> > Are you sure about the additional I/O? With native symtab, existing
>>>> tools just need to read those, while plugin based approach needs to read
>>>> bit code section to feedback symbols to the tool.
>>>>
>>>> The additional I/O will be quite big if you are going to emit the full
>>>> symbol table. Looking at some of our real world links the symbol table and
>>>> string tables of all the inputs seen by the linker add up to about 50 -
>>>> 100mb.
>>>>
>>> (resent as the previous message got bounced)
>>>
>>> There is no need for emitting the full symtab. I checked the overhead
>>> with a huge internal C++ source. The overhead of symtab + str table
>>> compared with byte code with debug is about 3%.
>>>
>>>
>> (We should take this up later, not sure this is an apples to apples
>> comparison if you're including debug info, but it's orthogonal to this
>> email)
>>
>>
>>> More importantly, there is plan to use the symtab also for thinLTO
>>> indexing purpose, which makes the space usage completely 'unwasted'. That
>>> gets into the details which will follow when the patches are in (with
>>> design docs).
>>>
>>
>> Well, let's come up with a design that makes sense before we put patches
>> in :)
>>
>> That said, I have what's probably a compromise, but also what I think
>> might be a good API design for this and avoids the question of container
>> format completely.
>>
>> I think what we're going to want is to solve this (as with about half of
>> all problems in CS) with another layer of indirection. Let's get an API
>> proposed that'll be a way to get the data in an abstract way outside of
>> whatever container format we happen to be speaking about and then vending
>> that data through to the thinlto functionality, or any process that wants
>> to grab information across all of the inputs. Then, underneath that, we can
>> use the existing object/module layer to crack open the container and read
>> the information we want.
>>
>> This way we'll be able to abstract away any issues and still offer what's
>> likely going to be a stable enough API so that we can experiment with the
>> underlying formats without too much worrying. Not a Stable(tm) format, this
>> is still too early, but stable enough.
>>
>> So we'll want a simple wrapper class with a few functions to encompass
>> what we want (pointers to functions, symbol information, etc) with ties to
>> the Object library (for native object files) and IR (for Module based).
>> Make sense? I think it'd make sense to have a single instance wrap a single
>> input for ease of iterating over them, but I'm definitely open to other
>> ideas.
>>
>> Thoughts?
>>
>> -eric
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150518/eb42410c/attachment.html>