[cfe-dev] [RFC] clang-ifso: A new clang-based tool for generating interface libraries.

Puyan Lotfi via cfe-dev cfe-dev at lists.llvm.org
Mon Apr 8 16:57:54 PDT 2019 

So, we are actually now thinking of redoing this patch but using a similar
idea.

Petr, would llvm-elfabi be capable of merging multiple of these tbe files?

I'm looking at implementing something in the Driver that basically
generates a tbe file for each .o file. The benefit would be that you could
pretty easily just let loose your standard clang build invocation already
in your build system with a couple added flags, and that's all you'd need
to do to generate the ifsos assuming llvm-elfabi could consume all of the
tbe's and link them together etc.

I think tbe could be a point of collaboration on this. Thoughts?

PL


On Fri, Apr 5, 2019 at 2:35 PM Puyan Lotfi <puyan.lotfi.llvm at gmail.com>
wrote:

> A few of us (Alex, Saleem and I) have discussed ideas for ELF generation.
> It sounds like aside from yaml2obj, lld, and MC's ELFObjectWriter there is
> also all of the ELF emission done by llvm-objcopy.
> Alex tells me he's actually been intending to extract a lot of the ELF
> emission code from llvm-objcopy (llvm/tools/llvm-objcopy/ELF/Object.cpp/h)
> into a library somewhere in lib/Object.
> We think it's probably best to do this refactoring, then eventually have
> llvm-elfabi, clang-ifso, yaml2obj, etc all do their ELF writing through it
> in some capacity (either going through one of the text formats, or
> directly).
>
> As far as clang-ifso goes, I think it should be a standalone tool that is
> capable of generating multiple ifso formats (yaml, tbe, raw elf, etc).
> At some point soon I will probably put up a phabricator patch that only
> includes the yaml emission along with some tests; I think that could work
> for now.
>
> Thoughts?
>
> PL
>
> On Fri, Apr 5, 2019 at 9:47 AM Puyan Lotfi <puyan.lotfi.llvm at gmail.com>
> wrote:
>
>> CC'ing George and Alex:
>>
>> I was thinking along these lines, that it would be nice to have some kind
>> of common code base for things like yaml2obj as well. I spent some time at
>> first looking at and trying to decouple yaml2elf from the yaml parts until
>> I realized that it'd be a lot better to just generate the yaml in memory
>> and pass it on to yaml2elf.
>>
>> Is llvm-elfabi planning to interop with yaml2obj any? As far as I can
>> tell, currently in the monorepo the only places that are creating ELF .dyn*
>> sections are in yaml2obj and lld. I agree it'd be nice to figure out some
>> kind of code reuse plan.
>>
>> PL
>>
>>
>>
>> On Thu, Apr 4, 2019 at 6:21 PM Petr Hosek <phosek at chromium.org> wrote:
>>
>>> llvm-elfabi is going to generate ELF binaries directly as well (we
>>> originally considered extending lld to support tbe files directly but we
>>> later decided not to do that to avoid introducing the additional
>>> complexity). It'd be great if clang-ifso could emit tbe files and use the
>>> same implementation for generating ELF binaries to avoid duplication
>>> between the two tools.
>>>
>>> On Thu, Apr 4, 2019 at 5:42 PM Puyan Lotfi <puyan.lotfi.llvm at gmail.com>
>>> wrote:
>>>
>>>> Shoaib, Petr:
>>>>
>>>> I don’t yet know the full details of elf-tapi, but the two ways I
>>>> believe these differs are:
>>>>
>>>> 1) clang-ifso is a clang based tool that scans from the source level
>>>> and uses visibility attributes to put together a set of symbols to expose
>>>> in the ifso file.
>>>>
>>>> 2) clang-ifso primarily aims to generate the ELF binary directly, but
>>>> also can generate yaml that yaml2obj can generate the ifso from. Currently
>>>> the aim is to avoid introducing linker changes.
>>>>
>>>> It’s possible clang-ifso could also support generating said text stubs
>>>> (tbe’s) too.
>>>>
>>>> PL
>>>>
>>>>
>>>> On Thu, Apr 4, 2019 at 5:23 PM Petr Hosek <phosek at chromium.org> wrote:
>>>>
>>>>> To expand on Shoaib's answer, llvm-elfabi
>>>>> <https://github.com/llvm/llvm-project/tree/master/llvm/tools/llvm-elfabi> is
>>>>> a very similar tool that we've been working on for the past few months in
>>>>> upstream. The current implementation is still incomplete, but once
>>>>> D56569 <https://reviews.llvm.org/D56569>, D55839
>>>>> <https://reviews.llvm.org/D55839> and D55864
>>>>> <https://reviews.llvm.org/D55864> land (hopefully within the next few
>>>>> weeks), it should be possible to use llvm-elfabi to produce text stubs
>>>>> (called .tbe files) and generate ELF interface libraries out of those.
>>>>>
>>>>> On Thu, Apr 4, 2019 at 5:18 PM Shoaib Meenai via cfe-dev <
>>>>> cfe-dev at lists.llvm.org> wrote:
>>>>>
>>>>>> How does this compare to the ELF TAPI implementation in LLVM (
>>>>>> http://lists.llvm.org/pipermail/llvm-dev/2018-September/126472.html
>>>>>> and https://reviews.llvm.org/D53051)?
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From: *cfe-dev <cfe-dev-bounces at lists.llvm.org> on behalf of
>>>>>> cfe-dev <cfe-dev at lists.llvm.org>
>>>>>> *Reply-To: *Puyan Lotfi <puyan.lotfi.llvm at gmail.com>
>>>>>> *Date: *Thursday, April 4, 2019 at 4:52 PM
>>>>>> *To: *cfe-dev <cfe-dev at lists.llvm.org>
>>>>>> *Cc: *Puyan Lotfi <puyan at puyan.org>
>>>>>> *Subject: *[cfe-dev] [RFC] clang-ifso: A new clang-based tool for
>>>>>> generating interface libraries.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>>
>>>>>>
>>>>>> The following is our proposal for clang-ifso, we hope to get some
>>>>>> valuable feedback from the community on this and we hope this work helps
>>>>>> SDK authors in the future:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On platforms such as Darwin and Windows, there exist library
>>>>>> interface files (TAPI tbd files, Windows Import Library files) that appear
>>>>>> to the compile-time linker as just another library file (i.e. dylib, dll,
>>>>>> etc) but are in-fact only empty listings of symbols to interface functions
>>>>>> that were intended for exposure by the library writer (i.e. their `.text`
>>>>>> sections are stripped, but they have the API symbols needed for linking).
>>>>>> These library interfaces can be used to both limit access to internals of a
>>>>>> library at static compile time, and can be used to speedup link time in the
>>>>>> case of linking with extremely large dynamic libraries. Aside from
>>>>>> providing more controlled API exposure and reduced memory usage in linking,
>>>>>> there is also the benefit of having a much smaller distribution size for
>>>>>> development SDKs (in the case where you’ve got an SDK with applications
>>>>>> that are built and linked on a PC but then deployed to run on a totally
>>>>>> different device). Finally, these interface libraries can in many cases be
>>>>>> used to break up build dependencies as well if they can be cached or
>>>>>> generated quickly in some way prior to building all the different libraries
>>>>>> in a build.
>>>>>>
>>>>>>
>>>>>>
>>>>>> clang-ifso is a tool that intends to bring the concept of interface
>>>>>> libraries to ELF shared objects. We call it ifso as a shorthand for
>>>>>> InterFace-Shared-Object: as in, we intend to support ELF by producing a .so
>>>>>> that looks just like a regular .so file to the linker but has most of the
>>>>>> .text and other contents dropped and has only the intended API interface
>>>>>> symbols populated. That means a .so file generated by clang-ifso contains
>>>>>> only a stripped .text section plus the `.dynsym` and `.dynstr` sections
>>>>>> along with the global symbols populated. Currently it is in a prototype
>>>>>> state and is capable of generating object-yaml or ELF, and is implemented
>>>>>> using the ClangTool interface.
>>>>>>
>>>>>>
>>>>>>
>>>>>> As mentioned at the beginning, there is prior art in this area with
>>>>>> things like TBD files on Darwin as well as Microsoft’s import libraries. As
>>>>>> an aside, there is evidence that ELF versions of ifsos are also deployed in
>>>>>> production in some settings, so this is not completely new with ELF either
>>>>>> but the effort to make all of the work upstream is. One other way that
>>>>>> clang-ifso does differ from a lot of the prior work is that it uses the
>>>>>> clang parser to glean what visible NamedDecls are present in the library
>>>>>> headers.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Because clang-ifso operates from the clang level we can and do enable
>>>>>> the programmer to use visibility attributes to expose or hide whatever
>>>>>> interface libraries they want in their actual code rather than relying on a
>>>>>> separate file to strip symbols as a post-build step. And since all
>>>>>> compile-time linking will be done with the ifso rather than the .so, all
>>>>>> clients of the library will need to completely rely on whatever was exposed
>>>>>> through visibility attributes.
>>>>>>
>>>>>>
>>>>>>
>>>>>> We are eager to to hear feedback and ideas in this space. Code for
>>>>>> clang-ifso is available at:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> https://github.com/plotfi/llvm-project/tree/master/clang/tools/clang-ifso
>>>>>> .
>>>>>>
>>>>>>
>>>>>>
>>>>>> PL
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>>> cfe-dev mailing list
>>>>>> cfe-dev at lists.llvm.org
>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20190408/748769f5/attachment.html>

More information about the cfe-dev mailing list