[cfe-dev] [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

[Billy O'Neal (VC LIBS) via cfe-dev]

> There are often ways to safely use STL containers w/o exceptions (Especially when custom allocators are provided).

Yes, but those ways generally involve going to terminate() on OOM. That's not OK in a driver.

From: Eric Fiselier<mailto:eric at efcs.ca>
Sent: Monday, March 27, 2017 4:37 PM
To: Ben Craig<mailto:ben.craig at ni.com>
Cc: via cfe-dev<mailto:cfe-dev at lists.llvm.org>; mclow.lists at gmail.com<mailto:mclow.lists at gmail.com>; Billy O'Neal (VC LIBS)<mailto:bion at microsoft.com>; Stephan T. Lavavej<mailto:stl at exchange.microsoft.com>; Casey at Carter.net<mailto:Casey at carter.net>
Subject: Re: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support





On Sat, Mar 25, 2017 at 9:18 AM, Ben Craig <ben.craig at ni.com<mailto:ben.craig at ni.com>> wrote:
Abstract:
I would like to add Windows x86 and x64 kernel support to libc++.  My initial
compiler would be MSVC 2015 Update 3, though MSVC 2017 shouldn't be difficult to
add after the fact.

I would like to know if this is a port that the libc++ maintainers are willing
to accept.

Before responding in depth I need to ask this question: Why not use Clang/C2 or simply Clang+LLVM?
I'm assuming they don't support targeting the kernel?

There seem to be three separate changes going on here, and each of the changes
will benefits and challenges. For that reason I think it's best to consider them
separately.

(1) Porting to MSVC
(2) Supporting the Windows Kernel C library.
(3) Implementation a Freestanding configuration of libc++
(4) Implementing the Kernel-space compatible configuration described in the original email.

In general I'm happy to accept specific or niche libc++ changes as long as they
aren't detrimental to the overall libc++ quality and implementation complexity.
Changes to better support (1) or (3) would be beneficial to the larger libc++ audience and I would b
happy to upstream them. However I have large concerns regarding the changes required for (2) and (4)
as I suspect they won't satisfy the above requirement.

For example What if the Windows Kernel C library is incomplete, or significantly different from
existing implementations, and supporting it requires re-implementing the missing parts within libc++?
These portions would be virtually untestable outside of the Windows Kernel environment and would
quickly become unmaintained. Having such code in Libc++ could quickly become a detriment.

My main concern with (4) is the limited feature set that has been proposed. Specifically
how it limits the libc++ internals to the same feature set and the changes that would be
needed to support and maintain it.

First Libc++ cannot reasonably limit itself to the proposed language and library feature set since
it must be free to use "restricted" features within the implementation of non-restricted ones whenever
it is beneficial to the implementation. The burden of maintaining a working restricted feature set could
not fall on upstream maintainers.

Second, the changes required to either guard restricted features using #ifdef or remove restricted features
by re-structuring the headers would be vast and would require constant maintenance. Frankly I don't
see how libc++ or its existing users could benefit from upstreaming these changes (For example adding
#ifdef guards for every usage of `operator new`).

I think it would be much better to get as much of libc++ compiling as possible, even if it depends on restricted
features, and then finding another mechanism to restrict the features end-users are allowed to use (Such as clang-tidy).
This would eliminate the need to restructure headers or spam out internal #ifdef guards.


This means that string, vector, and the non-array containers won't be ported.
Any class or function that requires throwing exceptions to meet their standards
required behavior will be omitted.  That rules out a lot of classes that
allocate memory.

There are often ways to safely use STL containers w/o exceptions (Especially when
custom allocators are provided).


Avoiding allocations allows us to sidestep one other large issue.  In the
kernel, not all memory is equal.  There are several memory pools to choose from,
but the two most common memory pools are the pageable pool and the non-pageable
pool.  There is no clear correct answer for which pool a global operator new
should use, so we simply won't require an allocating new to be present for our
implementation.  Placement new shall remain though.

Containers don't use new/delete directly but instead go through the specified allocator,
allowing containers to change the allocation behavior on a per-object basis. Not
supporting containers because of global operator new's behavior seems misguided.


My employer has significant experience using C++ in the kernel.  We have been
using a modified version of STLPort for quite some time and learned a lot about
C++ library usage in the kernel, often the hard way.  The big, obvious lesson
is that a lot of the STL is either difficult, or impossible to work with when
exceptions are off and std::terminate is undesired.  There's nothing wrong with
sorting in the kernel though.

Challenges:
* Header partitioning.

Libc++ prefers larger monolithic headers over many well-partitioned headers. The idea is that hitting the filesystem
multiple times is slower than processing the single include file. Any proposed changes should keep this in mind.

    * I don't have an exact list of what functions and classes I will be
      keeping.  I do know that some of the headers I want to bring along have
      parts that I won't be keeping.  For instance, many of the algorithms
      allocate a temporary buffer in order to meet performance requirements.

I don't think partitioning <algorithm> into smaller headers would be beneficial
to existing libc++ users (If that's what you're suggesting).


    * I'll also need to figure out how to not drag along unwanted header
      dependencies.

I don't see how libc++ could upstream changes since it requires every header
it currently includes (modulo bugs).

* Testing.
    * Installing code so that it can be run in the kernel takes several seconds
      for each binary.
    * There is no facility in the Windows kernel for running a program starting
      at "main" in the context of the kernel.

I suspect this will be the largest hurdle to get the test-suite running. My only
idea for handling this would be to write a script to rename main() to some
unique function name and creating a separate test-driver that calls the re-named main.

This could also allow us to combine multiple tests into a single executable, avoiding
the cost of installing every test manually.



    * The 32-bit Windows kernel requires a default calling convention of
      stdcall, but warns if "main" is not cdecl.
    * A common failure mode for libc++ tests is to crash or assert.  That
      will "blue-screen" a machine.

I could envision a fix which replaces `<assert.h>` when compiling the tests, but that
would be a hack. Maybe the Windows Kernel C library provides a mechanism for replacing
the assert handler?

 .
* MSVC compiler feature set
    * No #include_next.  I'll need to use computed include paths, like an
      STL using caveman.

Using hard-coded kernel paths is not a change I see being upstream-able. However
we might be able to convince MSVC to implement #include_next if we can provide
strong enough rational for why we need it.


    * Limited expression SFINAE.  Some areas in the code are straightforward
      to fix, and others are not.

I'm violently against working around MSVC template bugs. Every time I've seen
it done it makes the implementation worse.

Another point is that libc++ doesn't have an <atomic> implementation for MSVC
so adding MSVC support would required adding one.


* C-runtime
    * The Windows kernel has its own implementation of the C-runtime.  I don't
      know all the details on it.  I suspect (but don't know) that it is
      derived from Dinkumware, but I know that it is not the same C-runtime
      as used in user mode.

I'm very curious to know the changes required to support the kernel C runtime.
Hopefully it's similar enough to other supported C libraries that very few changes
are needed.

I hope my response has been helpful, and it hasn't scared you away from using libc++.
These are my initial reactions and are in no way absolute. Please let me know if I can
clarify anything or if I got anything wrong.

/Eric


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20170327/dc02d4e5/attachment.html>