[llvm-dev] An update on the Program Repository Project

[Ying Yi via llvm-dev]

Hi all,



We have previously talked about the idea of a “program repository” (prepo)
[1,2], the implementation in LLVM [3] and usage of the prepo [4-5]. This
project has now reached the stage where we could build a simple C/C++
program using the repo compiler and linker, so we’d like to hear the
thoughts of the LLVM community. We would be especially interested in
hearing from anyone interested in collaborating with us.



Below are some new components developed since last year’s LLVM conference.



Main New Components

===================



1.      rld - program repository linker

A high-level overview of the internals of rld is shown in [6]. rld is at
the stage where it can be used to link C++ code. However, it currently
lacks static archive support meaning that doing so is somewhat clumsy.



2.      Standard libraries

Standard libraries need to be recompiled to a program repository. We have
ported musl-libc [7], libunwind [8], compiler-rt [9], libc++abi [10], and
libc++ [11].



3.      Docker image [12]

The docker image has both the prepo-aware toolchain and the collection of
standard libraries mentioned above pre-installed, as well as tools to
explore the repository contents.



Other New Components

====================



-        llvm-prepo-prepo [13]: The modified LLVM.

Some new tools have been added in the repository:

·        repo-fragments [14] is a utility for dumping the names and digests
from a compilation [15] using its ticket file [16].

·        repo-ticket-dump [14] is a simple tool to extract the compilation
digest from a ticket file. We can then use this in combination with
pstore-dump to examine the contents of that compilation record.

·        repo-create-ticket is intended to be a companion to
repo-ticket-dump. Where repo-ticket-dump will tell you the compilation
digest associated with an existing ticket file, repo-create-ticket will
create a new ticket file given an existing compilation digest.



-        pstore [17]: this is the storage engine for the project and
there’s currently a hard dependency on it. We envisage providing an
abstraction layer to decouple the compiler and allow the database to be
replaced.



pstore-export and pstore-import enable import and export of a pstore
database to and from a JSON file. Using the repo-create-ticket tool, the
pstore-import tool, and the pstore-export tool, it is possible to construct
a repository with specific characteristics for use in testing tools such as
repo2obj or rld.



Timing Linkers

============



-        timey [18]

This is the rld benchmark utility. timey shows the results of running the
two linkers (for example, rld and lld) with varying amounts of input. The
inputs are created by the rld-gen [19] tool which creates program
repositories where particular characteristics are present. For timing
results, please refer to [20].



Sample Code

===========



-        hello-rld [21]

hello-rld shows the repository compiler and linker building various
versions of “hello world” using direct system calls, the standard C
library, and the standard C++ library.



There is some documentation, background, and data to support some of the
claims we have made here on the llvm-project-prepo wiki [22].



Previous Talks/Poster

=================



[1] Describing the concept ("Demo of a repository for statically compiled
programs", Technical Talk at 2016 LLVM Developers' Meeting).
https://www.youtube.com/watch?v=-pL94rqyQ6c

[2] An earlier update post to the mailing list:
https://lists.llvm.org/pipermail/llvm-dev/2018-November/127899.html

[3] Implementation in LLVM ("Targeting a statically compiled program
repository with LLVM", Lightning Talk at 2019 EuroLLVM Developers’
Meeting).
https://www.youtube.com/watch?v=mlQyEBDnDJE

[4] Usage of llvm-prepo-prepo ("LLVM Build Times using a Program
Repository", Poster at 2019 LLVM Developers' Meeting).
http://llvm.org/devmtg/2019-10/talk-abstracts.html#post2

[5] Usage of llvm-prepo-prepo ("Compile Faster with the Program Repository
and ccache", Lightning Talk at 2020 LLVM Developers’ Meeting).
https://www.youtube.com/watch?v=9bQEtJHkDTw&t=11s



Links

====

[6]
https://github.com/SNSystems/llvm-project-prepo/wiki/%5Brld%5D-Linker-Structure

[7] https://github.com/SNSystems/musl-prepo

[8] https://clang.llvm.org/docs/Toolchain.html#unwind-library

[9] https://compiler-rt.llvm.org/

[10] https://libcxxabi.llvm.org/

[11] https://libcxx.llvm.org/

[12] https://hub.docker.com/r/sndevelopment/llvm-prepo

[13] https://github.com/SNSystems/llvm-project-prepo

[14]
https://github.com/SNSystems/llvm-project-prepo/wiki/Exploring-a-Program-Repository

[15]
https://github.com/SNSystems/llvm-project-prepo/wiki/Glossary#compilation

[16] https://github.com/SNSystems/llvm-project-prepo/wiki/Glossary#ticket

[17] https://github.com/SNSystems/pstore

[18] https://github.com/SNSystems/timey

[19]
https://github.com/SNSystems/llvm-project-prepo/tree/master/rld/tools/gen

[20] https://github.com/SNSystems/timey/blob/master/results/README.md

[21] https://github.com/SNSystems/hello-rld

[22] https://github.com/SNSystems/llvm-project-prepo/wiki

Thanks,

-- 
Ying Yi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20211115/dd304396/attachment.html>