[cfe-dev] [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

Mon Mar 27 10:11:13 PDT 2017

You'll probably find VC 2017 to be significantly friendlier to libc++ than 2015 Update 3 was, given that so many compiler features and fixes were implemented for C++17 features, and for making libc++'s tests happier with VC's STL and FE. I would strongly recommend ignoring 2015 Update 3 and starting directly with VC 2017 (they are bin-compatible and built from the same branch of source code).

STL

-----Original Message-----
From: Ben Craig [mailto:ben.craig at ni.com] 
Sent: Saturday, March 25, 2017 8:18 AM
To: via cfe-dev <cfe-dev at lists.llvm.org>; mclow.lists at gmail.com; eric at efcs.ca; Billy O'Neal (VC LIBS) <bion at microsoft.com>; Stephan T. Lavavej <stl at exchange.microsoft.com>; Casey at Carter.net
Subject: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

Abstract:
I would like to add Windows x86 and x64 kernel support to libc++.  My initial compiler would be MSVC 2015 Update 3, though MSVC 2017 shouldn't be difficult to add after the fact.

I would like to know if this is a port that the libc++ maintainers are willing to accept.

Motivation:
C++ is intended to be useful for systems programming, and the kernel is 
C++ as
"systems" as it gets.  There are a fair number of language features that lean heavily on the presence of the library.  Initializer lists require library support.  R-value references require std::move, std::forward, and others, or require users to duplicate their functionality.  Variadic templates are extremely cumbersome to work with if std::tuple is not available.  std::atomic provides a much better interface than compiler intrinsics for working with the
C++ memory model.

A subset of the C++ language can be used (with caution) in the Windows kernel, but the library remains out of reach.  The C++ library that ships with MSVC won't compile at all in the kernel, even for headers that ostensibly make sense, like <atomic>.

Since the vendor's C++ library doesn't work in the kernel, might as well try and make libc++ work there instead.

There's no reason that the Windows ARM kernel couldn't be addressed as well.
However, my employer has no significant interest in that environment, and I have no access to devices to test with for that environment.  I would be glad to help review code to support the Windows ARM kernel though.

Strategy:
I will not be porting the entirety of libc++.  I will have exceptions and RTTI turned off.  I plan on porting the freestanding headers [17.5.1.3,2 compliance], minus <exception>, and <typeinfo>.  I also plan on porting the portions of <algorithm>, <array>, <tuple>, and <utility> that can naturally be expressed in an exception-free and RTTI-free environment.  I can be convinced that other headers should also be included.

This means that string, vector, and the non-array containers won't be ported.
Any class or function that requires throwing exceptions to meet their standards required behavior will be omitted.  That rules out a lot of classes that allocate memory.

Avoiding allocations allows us to sidestep one other large issue.  In the kernel, not all memory is equal.  There are several memory pools to choose from, but the two most common memory pools are the pageable pool and the non-pageable pool.  There is no clear correct answer for which pool a global operator new should use, so we simply won't require an allocating new to be present for our implementation.  Placement new shall remain though.

I don't believe I will need to port libc++abi, libunwind, and compiler-rt.

In some ways, this is a good first step for getting MSVC support.  The functionality that is most coupled to POSIX-like operating systems is all in the "hosted" portion of the library.

Prior Work:
Lawrence Crowl and Alberto Ganesh Barbati proposed this paper back in the C++11 time frame with mixed results:
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.open-std.org%2Fjtc1%2Fsc22%2Fwg21%2Fdocs%2Fpapers%2F2011%2Fn3256.html&data=02%7C01%7Cstl%40exchange.microsoft.com%7C832c31de248041e34e6108d4739227c1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636260518944916138&sdata=wRs3XX8FFwVLqZBaIttvL%2FRouxaE31o9WjqaTOEQ5Ic%3D&reserved=0

My employer has significant experience using C++ in the kernel.  We have been using a modified version of STLPort for quite some time and learned a lot about
C++ library usage in the kernel, often the hard way.  The big, obvious 
C++ lesson
is that a lot of the STL is either difficult, or impossible to work with when exceptions are off and std::terminate is undesired.  There's nothing wrong with sorting in the kernel though.

Challenges:
* Header partitioning.
    * I don't have an exact list of what functions and classes I will be 
      keeping.  I do know that some of the headers I want to bring along have
      parts that I won't be keeping.  For instance, many of the algorithms
      allocate a temporary buffer in order to meet performance requirements.
    * I'll also need to figure out how to not drag along unwanted header
      dependencies.
* I don't know if anyone has ever attempted to get cmake to target a Windows
  kernel environment.  I don't think I will be trying to do this either.  I
  think I will be using a script similar in spirit to buildit.sh.
* Testing.
    * New 'requires' code will be needed to only attempt to run the tests that
      I care about.  I plan on doing this by white listing headers, and having
      the test runner examine the test to see if any unsupported headers are in
      the test.
    * Installing code so that it can be run in the kernel takes several seconds
      for each binary.
    * There is no facility in the Windows kernel for running a program starting
      at "main" in the context of the kernel.
    * The 32-bit Windows kernel requires a default calling convention of
      stdcall, but warns if "main" is not cdecl.
    * A common failure mode for libc++ tests is to crash or assert.  That
      will "blue-screen" a machine.
    * The existing lit test infrastructure assumes flags similar to gcc and
      llvm.  These assumptions are scattered throughout the code, and not in a
      centralized location.
* MSVC compiler feature set
    * No #include_next.  I'll need to use computed include paths, like an
      STL using caveman.
    * Limited expression SFINAE.  Some areas in the code are straightforward
      to fix, and others are not.
* C-runtime
    * The Windows kernel has its own implementation of the C-runtime.  I don't
      know all the details on it.  I suspect (but don't know) that it is
      derived from Dinkumware, but I know that it is not the same C-runtime
      as used in user mode.

I have a very early prototype that is poorly tested, and won't work at all for existing supporting platforms.  Basically, I took the quick and dirty approach of commenting out / deleting code that got in the way of the Windows kernel.  It can still be somewhat instructive in the kinds of things I expect to need to do (eventually) to the libc++ code base.  The prototype can be found here: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fben-craig%2Flibcxx%2Ftree%2Fntddk&data=02%7C01%7Cstl%40exchange.microsoft.com%7C832c31de248041e34e6108d4739227c1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636260518944916138&sdata=uX9xJkK0U3yecRp36pY2ECE2zHnsPZo72ZZgF89m7YY%3D&reserved=0 .

Thanks,
Ben Craig