[llvm-dev] [RFC] Open sourcing and contributing TAPI back to the LLVM community

[Juergen Ributzka via llvm-dev]

On Thu, Sep 7, 2017 at 6:52 PM, Davide Italiano <davide at freebsd.org> wrote:

> On Thu, Sep 7, 2017 at 5:01 PM, Juergen Ributzka via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> > Hi @ll,
> >
> > Over the past years I have been looking into how to reduce the size of
> the
> > SDK that ships with Xcode and how to improve build times for the overall
> OS
> > inside Apple. The result is a tool called TAPI, which is used at Apple
> for
> > all things related to text-based dynamic library files (.tbd).
> >
> > What are text-based dynamic library files?
> > Text-based dynamic library files (TBDs) are a textual representation of
> the
> > information in a dynamic library / shared library that is required by the
> > static linker - basically a symbol list of the exported symbols.
> >
> > Apple’s SDKs originally used Mach-O Dynamic Library Stubs. Mach-O Dynamic
> > Library Stubs are dynamic library files, but with all the text and data
> > stripped out. TBD files were introduced to replaced Mach-O Dynamic
> Library
> > Stub files in the SDK to further reduce its overall size.
> >
> > Over time the TAPI tool has grown and is used now in a variety of ways.
> >
> > Dynamic Library Stubbing:
> > As mentioned above, TAPI is used to read the content of dynamic library /
> > shared library and generates a textual representation that can be used by
> > the static linker. The current implementation reads MachO files, but it
> > could be extended to also provide the same functionality for other object
> > file formats.
> >
> > Framework / Dynamic Library Verification:
> > The symbols that are exported from a dynamic library should ideally
> match,
> > or at least contain, all the API that is specified in the associated
> header
> > files. TAPI performs this verification by parsing the header files with
> > CLANG and compare the findings to the exported symbols from the library.
> >
> > InstallAPI:
> > InstallAPI is a new build phase that generates the TBD file from header
> > files only. This allows a dependency of the library to build concurrently
> > even before the library has been built itself. This can be used to
> increase
> > parallelism in the build or larger projects or operating systems.
> >
> > Misc:
> > - display and operate on TBD files
> > - automatically generate API tests from header files
> > - libtapi, which is used by the linker (ld64) to parse the TBD files
> >
>
> I'm interested in whether you plan to have this integrated in lld as well.
> As far as I understand, this is going to be the de-facto way of
> shipping for Mach-O binaries (at least, the ones released by Apple).
> Please correct me if I'm wrong.
>

Yes, this is already the de-facto way of shipping Mach-O files in the SDK.
That means self-hosting LLD against the SDK is currently not possible. The
system itself is obviously still shipping full Mach-O files in /System, so
you should be still able to self-host against those file.

My plan is to integrate support for TBD files into all LLVM tools where it
makes sense (including LLD). This is why I wanted to start to put the basic
support into LLVM first, so it can be used by other tools and libraries.


> I tried to self-host lld on El Capitan and it fails because lld
> doesn't really know about TBD files.
> This, unfortunately, makes the linker not really usable for modern Mac
> OS releases.
>
> >
> > The functionality of the tool is currently limited to Mach-O object
> files,
> > but that is not a technical limitation. In making the tool open source I
> > hope others will be able to take advantage of it too and extend its
> > functionality to other object file formats.
> >
> >
> > I initially developed the project as a CLANG project, but that was mostly
> > for practical reasons (out-of-tree development, separate repo, etc). For
> the
> > curious ones I pushed the repo to github (https://github.com/ributzka/
> tapi).
> >
> > I imagine, for example, that the reading/writing of TBD files is
> something
> > that would fit better into the LLVM sources, which makes it available to
> > other libraries and tools (e.g. LLVMObject, llvm-nm, lld, ...).
> >
> > I created a small patch that integrates it with llvm-nm and LLVMObject.
> This
> > patch is not complete and I will split it up into smaller patches for
> > review. I am providing it as a reference to get the discussion started.
> >
> > Please let me know what you think and bikeshed away :)
> >
> > Thanks
> >
> > Cheers,
> > Juergen
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
>
> --
> Davide
>
> "There are no solved problems; there are only problems that are more
> or less solved" -- Henri Poincare
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170908/ffdd50a3/attachment-0001.html>