[cfe-dev] [RFC] clang-ifso: A new clang-based tool for generating interface libraries.

Fri Apr 5 09:47:43 PDT 2019

CC'ing George and Alex:

I was thinking along these lines, that it would be nice to have some kind
of common code base for things like yaml2obj as well. I spent some time at
first looking at and trying to decouple yaml2elf from the yaml parts until
I realized that it'd be a lot better to just generate the yaml in memory
and pass it on to yaml2elf.

Is llvm-elfabi planning to interop with yaml2obj any? As far as I can tell,
currently in the monorepo the only places that are creating ELF .dyn*
sections are in yaml2obj and lld. I agree it'd be nice to figure out some
kind of code reuse plan.

PL

On Thu, Apr 4, 2019 at 6:21 PM Petr Hosek <phosek at chromium.org> wrote:

> llvm-elfabi is going to generate ELF binaries directly as well (we
> originally considered extending lld to support tbe files directly but we
> later decided not to do that to avoid introducing the additional
> complexity). It'd be great if clang-ifso could emit tbe files and use the
> same implementation for generating ELF binaries to avoid duplication
> between the two tools.
>
> On Thu, Apr 4, 2019 at 5:42 PM Puyan Lotfi <puyan.lotfi.llvm at gmail.com>
> wrote:
>
>> Shoaib, Petr:
>>
>> I don’t yet know the full details of elf-tapi, but the two ways I believe
>> these differs are:
>>
>> 1) clang-ifso is a clang based tool that scans from the source level and
>> uses visibility attributes to put together a set of symbols to expose in
>> the ifso file.
>>
>> 2) clang-ifso primarily aims to generate the ELF binary directly, but
>> also can generate yaml that yaml2obj can generate the ifso from. Currently
>> the aim is to avoid introducing linker changes.
>>
>> It’s possible clang-ifso could also support generating said text stubs
>> (tbe’s) too.
>>
>> PL
>>
>>
>> On Thu, Apr 4, 2019 at 5:23 PM Petr Hosek <phosek at chromium.org> wrote:
>>
>>> To expand on Shoaib's answer, llvm-elfabi
>>> <https://github.com/llvm/llvm-project/tree/master/llvm/tools/llvm-elfabi> is
>>> a very similar tool that we've been working on for the past few months in
>>> upstream. The current implementation is still incomplete, but once
>>> D56569 <https://reviews.llvm.org/D56569>, D55839
>>> <https://reviews.llvm.org/D55839> and D55864
>>> <https://reviews.llvm.org/D55864> land (hopefully within the next few
>>> weeks), it should be possible to use llvm-elfabi to produce text stubs
>>> (called .tbe files) and generate ELF interface libraries out of those.
>>>
>>> On Thu, Apr 4, 2019 at 5:18 PM Shoaib Meenai via cfe-dev <
>>> cfe-dev at lists.llvm.org> wrote:
>>>
>>>> How does this compare to the ELF TAPI implementation in LLVM (
>>>> http://lists.llvm.org/pipermail/llvm-dev/2018-September/126472.html
>>>> and https://reviews.llvm.org/D53051)?
>>>>
>>>>
>>>>
>>>> *From: *cfe-dev <cfe-dev-bounces at lists.llvm.org> on behalf of cfe-dev <
>>>> cfe-dev at lists.llvm.org>
>>>> *Reply-To: *Puyan Lotfi <puyan.lotfi.llvm at gmail.com>
>>>> *Date: *Thursday, April 4, 2019 at 4:52 PM
>>>> *To: *cfe-dev <cfe-dev at lists.llvm.org>
>>>> *Cc: *Puyan Lotfi <puyan at puyan.org>
>>>> *Subject: *[cfe-dev] [RFC] clang-ifso: A new clang-based tool for
>>>> generating interface libraries.
>>>>
>>>>
>>>>
>>>> Hi All,
>>>>
>>>>
>>>>
>>>> The following is our proposal for clang-ifso, we hope to get some
>>>> valuable feedback from the community on this and we hope this work helps
>>>> SDK authors in the future:
>>>>
>>>>
>>>>
>>>> On platforms such as Darwin and Windows, there exist library interface
>>>> files (TAPI tbd files, Windows Import Library files) that appear to the
>>>> compile-time linker as just another library file (i.e. dylib, dll, etc) but
>>>> are in-fact only empty listings of symbols to interface functions that were
>>>> intended for exposure by the library writer (i.e. their `.text` sections
>>>> are stripped, but they have the API symbols needed for linking). These
>>>> library interfaces can be used to both limit access to internals of a
>>>> library at static compile time, and can be used to speedup link time in the
>>>> case of linking with extremely large dynamic libraries. Aside from
>>>> providing more controlled API exposure and reduced memory usage in linking,
>>>> there is also the benefit of having a much smaller distribution size for
>>>> development SDKs (in the case where you’ve got an SDK with applications
>>>> that are built and linked on a PC but then deployed to run on a totally
>>>> different device). Finally, these interface libraries can in many cases be
>>>> used to break up build dependencies as well if they can be cached or
>>>> generated quickly in some way prior to building all the different libraries
>>>> in a build.
>>>>
>>>>
>>>>
>>>> clang-ifso is a tool that intends to bring the concept of interface
>>>> libraries to ELF shared objects. We call it ifso as a shorthand for
>>>> InterFace-Shared-Object: as in, we intend to support ELF by producing a .so
>>>> that looks just like a regular .so file to the linker but has most of the
>>>> .text and other contents dropped and has only the intended API interface
>>>> symbols populated. That means a .so file generated by clang-ifso contains
>>>> only a stripped .text section plus the `.dynsym` and `.dynstr` sections
>>>> along with the global symbols populated. Currently it is in a prototype
>>>> state and is capable of generating object-yaml or ELF, and is implemented
>>>> using the ClangTool interface.
>>>>
>>>>
>>>>
>>>> As mentioned at the beginning, there is prior art in this area with
>>>> things like TBD files on Darwin as well as Microsoft’s import libraries. As
>>>> an aside, there is evidence that ELF versions of ifsos are also deployed in
>>>> production in some settings, so this is not completely new with ELF either
>>>> but the effort to make all of the work upstream is. One other way that
>>>> clang-ifso does differ from a lot of the prior work is that it uses the
>>>> clang parser to glean what visible NamedDecls are present in the library
>>>> headers.
>>>>
>>>>
>>>>
>>>> Because clang-ifso operates from the clang level we can and do enable
>>>> the programmer to use visibility attributes to expose or hide whatever
>>>> interface libraries they want in their actual code rather than relying on a
>>>> separate file to strip symbols as a post-build step. And since all
>>>> compile-time linking will be done with the ifso rather than the .so, all
>>>> clients of the library will need to completely rely on whatever was exposed
>>>> through visibility attributes.
>>>>
>>>>
>>>>
>>>> We are eager to to hear feedback and ideas in this space. Code for
>>>> clang-ifso is available at:
>>>>
>>>>
>>>>
>>>>
>>>> https://github.com/plotfi/llvm-project/tree/master/clang/tools/clang-ifso
>>>> .
>>>>
>>>>
>>>>
>>>> PL
>>>>
>>>>
>>>>
>>>>
>>>>
>>> _______________________________________________
>>>> cfe-dev mailing list
>>>> cfe-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20190405/1fb73088/attachment.html>