[llvm-dev] A libc in LLVM

Tue Jun 25 19:23:08 PDT 2019

I agree with your point 1.

With regards to point 2, I think there's a difference between Fuchsia,
which Google controls (where every check-in is authored by a Fuchsia eng
and reviewed by another Fuchsia eng), and LLVM, which Google doesn't
control.  There's also a difference between Google in general, and the
Fuchsia project, which I'd summarize as simply: Google is not a
monoculture.  Case in point: Jake, who works at Google, immediately
countered Siva's suggestion that "Support for more architectures" is not
something Google intends to invest in, by pointing out his need for AArch64
support. I work for Google too, and I personally need RISC-V support.
(Separately, I'm sorry to hear about your experience with Fuchsia's musl
fork... though I've not worked on Fuchsia and have no knowledge of that
situation and therefore won't say anything more about it.)

With regards to point 3, I agree with your points, in particular, I agree
that it's important for there to be a variety of libc implementations.  But
it seems to me that while gnu has both gcc and glibc, gcc doesn't require
the use of glibc, and I would anticipate that clang would never require
llvmlibc.  I would anticipate that a user would continue to have their
choice of compiler, their choice of STL implementation, their choice of
libc implementation.  To the extent that there would be a "library
implementation that's first-class for use with LLVM", I think there already
is: glibc.  But it would be better if there were two first-class
implementations.

On Tue, Jun 25, 2019 at 2:34 PM Rich Felker via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> On Tue, Jun 25, 2019 at 03:24:04AM +0000, Siva Chandra via llvm-dev wrote:
> > Hello LLVM Developers,
> >
> > Within Google, we have a growing range of needs that existing libc
> > implementations don't quite address. This is pushing us to start
> > working on a new libc implementation.
> >
> > Informal conversations with others within the LLVM community has
> > told us that a libc in LLVM is actually a broader need, and we are
> > increasingly consolidating our toolchains around LLVM. Hence, we
> > wanted to see if the LLVM project would be interested in us
> > developing this upstream as part of the project.
> >
> > To be very clear: we don't expect our needs to exactly match
> > everyone else's -- part of our impetus is to simplify things
> > wherever we can, and that may not quite match what others want in a
> > libc. That said, we do believe that the effort will still be
> > directly beneficial and usable for the broader LLVM community, and
> > may serve as a starting point for others in the community to flesh
> > out an increasingly complete set of libc functionality.
> >
> > We are still in the early stages, but we do have some high-level
> > goals and guiding principles of the initial scope we are interested
> > in pursuing:
> >
> > The project should mesh with the "as a library" philosophy of the
> > LLVM project: even though "the C Standard Library" is nominally "a
> > library," most implementations are, in practice, quite monolithic.
> >
> > The libc should support static non-PIE and static-PIE linking. This
> > means, providing the CRT (the C runtime) and a PIE loader for static
> > non-PIE and static-PIE linked executables.
> >
> > If there is a specification, we should follow it. The scope that we
> > need includes most of the C Standard Library; POSIX additions; and
> > some necessary, system-specific extensions. This does not mean we
> > should (or can) follow the entire specification -- there will be
> > some parts which simply aren't worth implementing, and some parts
> > which cannot be safely used in modern coding practice.
> >
> > Vendor extensions must be considered very carefully, and only
> > admitted when necessary. Similar to Clang and libc++, it does seem
> > inevitable that we will need to provide some level of compatibility
> > with other vendors' extensions.
> >
> > The project should be an exemplar of developing with LLVM tooling.
> > Two examples are fuzz testing from the start, and
> > sanitizer-supported testing.
> >
> > There are also few areas which we do not intend to invest in at this
> point:
> >
> > Implement dynamic loading and linking support.
> > Support for more architectures (we'll start with just x86-64 for
> simplicity).
> >
> > For these areas, the community is of course free to contribute. Our
> > hope is that, preserving the "as a library" design philosophy will
> > make such extensions easy, and allow retaining the simplicity when
> > these features aren't needed.
> >
> > We intend to build the new libc in a gradual manner. To begin with,
> > the new libc will be a layer sitting between the application and the
> > system libc. Eventually, when the implementation is sufficiently
> > complete, it will be able to replace the system libc at least for
> > some use cases and contexts.
> >
> > So, what do you think about incorporating this new libc under the
> > LLVM project?
>
> Since I have a little experience in this area, I'd like to chime in on
> it. :-) TL;DR I think it's a reall, REALLY bad idea.
>
> First, writing and maintaining a correct, compatible, high-quality
> libc is a monumental task. The amount of code needed is not all that
> large, but the subtleties of how it behaves and the difficulties of
> implementing various interfaces that have no capacity to fail or
> report failure, and the astronomical "compatibility surface" of
> interfacing with all C and C++ software ever written as well as a
> large amount of software written in other languages whose runtimes
> "pass through" the behavior of libc to the applications they host, all
> contribute to the scale of work, and of knowledge/expertise, involved
> in making something of even decent quality. (As an aside, note that I
> love to see hobby libc projects even if they have major problems, but
> that's totally different from proposing something that lots of people
> will end up stuck using.)
>
> Second, corporate development teams are uniquely qualified to utterly
> botch a libc, yet still push it into widespread use, and the cost is
> painful compatibility hacks in all applications. Apple did this with
> their fork of BSD libc code. Google has done it once already with
> their fork of musl in Fuchsia -- a project which I contributed
> significant amounts of free labor to in terms of tracking down folks
> for license clarification their lawyers wanted, only to have them
> never bother to ask me why technical things were done they way they
> were before making random useless and broken changes in their fork. A
> corporate-led project does not have to answer to the community, and
> will leave whatever bugs they introduce in place for the sake of
> bug-compatibility with their own software rather than fixing them.
>
> Third, there is tremendous value in non-monoculture of libc
> implementations, or implementations of any important library
> interfaces or language runtimes. Likewise there's tremendous value in
> non-monoculture of tooling (compilers, linkers, etc.). Avoiding
> monoculture preserves the motivation for consensus-based standards
> processes rather than single-party control (see also: Chrome and what
> it's done to the web) and the motivation for people writing software
> to write to the standards rather than to a particular implementation.
> A big part of making that possible is clear delineation of roles
> between parts of the toolchain and runtime, with well-defined
> interface boundaries. Some folks have told me that I should press LLVM
> to make musl the "LLVM libc" instead of whatever Google wants to do,
> but that misses the point: there *shouldn't be* a "LLVM libc", or any
> one library implementation that's "first class" for use with LLVM
> while others are only "second class".
>
> So, in summary:
>
> Point 1 is why making a libc for real-world use is not to be taken
> lightly.
>
> Point 2 is why, if it is done, it shouldn't be a Google project.
>
> Point 3 is why there should not be an "LLVM libc".
>
> Hope this is all helpful.
>
> Regards,
>
> Rich
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190625/698143ef/attachment-0001.html>