[cfe-dev] [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

[Eric Fiselier via cfe-dev]

On Sat, Mar 25, 2017 at 9:18 AM, Ben Craig <ben.craig at ni.com> wrote:

> Abstract:
> I would like to add Windows x86 and x64 kernel support to libc++.  My
> initial
> compiler would be MSVC 2015 Update 3, though MSVC 2017 shouldn't be
> difficult to
> add after the fact.
>
> I would like to know if this is a port that the libc++ maintainers are
> willing
> to accept.
>

Before responding in depth I need to ask this question: Why not use
Clang/C2 or simply Clang+LLVM?
I'm assuming they don't support targeting the kernel?

There seem to be three separate changes going on here, and each of the
changes
will benefits and challenges. For that reason I think it's best to consider
them
separately.

(1) Porting to MSVC
(2) Supporting the Windows Kernel C library.
(3) Implementation a Freestanding configuration of libc++
(4) Implementing the Kernel-space compatible configuration described in the
original email.

In general I'm happy to accept specific or niche libc++ changes as long as
they
aren't detrimental to the overall libc++ quality and implementation
complexity.
Changes to better support (1) or (3) would be beneficial to the larger
libc++ audience and I would b
happy to upstream them. However I have large concerns regarding the changes
required for (2) and (4)
as I suspect they won't satisfy the above requirement.

For example What if the Windows Kernel C library is incomplete, or
significantly different from
existing implementations, and supporting it requires re-implementing the
missing parts within libc++?
These portions would be virtually untestable outside of the Windows Kernel
environment and would
quickly become unmaintained. Having such code in Libc++ could quickly
become a detriment.

My main concern with (4) is the limited feature set that has been proposed.
Specifically
how it limits the libc++ internals to the same feature set and the changes
that would be
needed to support and maintain it.

First Libc++ cannot reasonably limit itself to the proposed language and
library feature set since
it must be free to use "restricted" features within the implementation of
non-restricted ones whenever
it is beneficial to the implementation. The burden of maintaining a working
restricted feature set could
not fall on upstream maintainers.

Second, the changes required to either guard restricted features using
#ifdef or remove restricted features
by re-structuring the headers would be vast and would require constant
maintenance. Frankly I don't
see how libc++ or its existing users could benefit from upstreaming these
changes (For example adding
#ifdef guards for every usage of `operator new`).

I think it would be much better to get as much of libc++ compiling as
possible, even if it depends on restricted
features, and then finding another mechanism to restrict the features
end-users are allowed to use (Such as clang-tidy).
This would eliminate the need to restructure headers or spam out internal
#ifdef guards.



> This means that string, vector, and the non-array containers won't be
> ported.
> Any class or function that requires throwing exceptions to meet their
> standards
> required behavior will be omitted.  That rules out a lot of classes that
> allocate memory.


There are often ways to safely use STL containers w/o exceptions
(Especially when
custom allocators are provided).


Avoiding allocations allows us to sidestep one other large issue.  In the
> kernel, not all memory is equal.  There are several memory pools to choose
> from,
> but the two most common memory pools are the pageable pool and the
> non-pageable
> pool.  There is no clear correct answer for which pool a global operator
> new
> should use, so we simply won't require an allocating new to be present for
> our
> implementation.  Placement new shall remain though.


Containers don't use new/delete directly but instead go through the
specified allocator,
allowing containers to change the allocation behavior on a per-object
basis. Not
supporting containers because of global operator new's behavior seems
misguided.


My employer has significant experience using C++ in the kernel.  We have
> been
> using a modified version of STLPort for quite some time and learned a lot
> about
> C++ library usage in the kernel, often the hard way.  The big, obvious
> lesson
> is that a lot of the STL is either difficult, or impossible to work with
> when
> exceptions are off and std::terminate is undesired.  There's nothing wrong
> with
> sorting in the kernel though.


> Challenges:
> * Header partitioning.
>

Libc++ prefers larger monolithic headers over many well-partitioned
headers. The idea is that hitting the filesystem
multiple times is slower than processing the single include file. Any
proposed changes should keep this in mind.


>     * I don't have an exact list of what functions and classes I will be
>       keeping.  I do know that some of the headers I want to bring along
> have
>       parts that I won't be keeping.  For instance, many of the algorithms
>       allocate a temporary buffer in order to meet performance
> requirements.
>

I don't think partitioning <algorithm> into smaller headers would be
beneficial
to existing libc++ users (If that's what you're suggesting).



>     * I'll also need to figure out how to not drag along unwanted header
>       dependencies.
>

I don't see how libc++ could upstream changes since it requires every header
it currently includes (modulo bugs).

* Testing.
>     * Installing code so that it can be run in the kernel takes several
> seconds
>       for each binary.

    * There is no facility in the Windows kernel for running a program
> starting
>       at "main" in the context of the kernel.
>

I suspect this will be the largest hurdle to get the test-suite running. My
only
idea for handling this would be to write a script to rename main() to some
unique function name and creating a separate test-driver that calls the
re-named main.

This could also allow us to combine multiple tests into a single
executable, avoiding
the cost of installing every test manually.




>     * The 32-bit Windows kernel requires a default calling convention of
>       stdcall, but warns if "main" is not cdecl.
>     * A common failure mode for libc++ tests is to crash or assert.  That
>       will "blue-screen" a machine.
>

I could envision a fix which replaces `<assert.h>` when compiling the
tests, but that
would be a hack. Maybe the Windows Kernel C library provides a mechanism
for replacing
the assert handler?

 .

> * MSVC compiler feature set
>     * No #include_next.  I'll need to use computed include paths, like an
>       STL using caveman.
>

Using hard-coded kernel paths is not a change I see being upstream-able.
However
we might be able to convince MSVC to implement #include_next if we can
provide
strong enough rational for why we need it.



>     * Limited expression SFINAE.  Some areas in the code are
> straightforward
>       to fix, and others are not.
>

I'm violently against working around MSVC template bugs. Every time I've
seen
it done it makes the implementation worse.

Another point is that libc++ doesn't have an <atomic> implementation for
MSVC
so adding MSVC support would required adding one.



> * C-runtime
>     * The Windows kernel has its own implementation of the C-runtime.  I
> don't
>       know all the details on it.  I suspect (but don't know) that it is
>       derived from Dinkumware, but I know that it is not the same C-runtime
>       as used in user mode.
>

I'm very curious to know the changes required to support the kernel C
runtime.
Hopefully it's similar enough to other supported C libraries that very few
changes
are needed.

I hope my response has been helpful, and it hasn't scared you away from
using libc++.
These are my initial reactions and are in no way absolute. Please let me
know if I can
clarify anything or if I got anything wrong.

/Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20170327/0e6ceaf2/attachment.html>