[PATCH] D121761: [draft] runtimes doc wip

Wed Mar 16 05:58:50 PDT 2022

thakis added a comment.

Thanks for the feedback! As background, I started this doc to try and understand this build config better. (What goals does it serve? Why does it have basically 0 adoption 3 years in? etc)

Then I got distracted and did something else, but aeubanks asked me to upload what I had so far so he could read it. So it's pretty incomplete and not fully researched. But I'm glad I uploaded it, since I got all these valuable comments :)

I'll update it in a bit.

================
Comment at: llvm/docs/RuntimeBuild.rst:24
+1. Where LLVM's build system puts the runtime libraries it builds
+2. Where clang looks for runtime libraries
+
----------------
mstorsjo wrote:
> AFAIK, 2. here isn't true. Clang doesn't get the value of `LLVM_ENABLE_PER_TARGET_RUNTIME_DIR` hardcoded in anywhere (and depending on the `--target` option, there might be different expectations for where to find the runtimes).
> 
> Additionally, Clang doesn't consistently check for runtimes in in the per-target runtime directory, but code for doing that is enabled in a handful of targets (some of the ones in clang/lib/Driver/ToolChains), but not all of them; the code for doing it is scattered around the subclasses. The ones that support it look for both kinds of layouts and add the directories if they are found - so afaik, Clang is pretty much agnostic.
True clang always looks in the place where `LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON` puts the libraries first, and then in the place where `LVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF` puts them second.

Will update.

================
Comment at: llvm/docs/RuntimeBuild.rst:52
+* ``lib/clang/14.0.0/lib/aarch64-unknown-fuchsia/libclang_rt.builtins.a``
+
+Where clang looks for runtime libraries
----------------
mstorsjo wrote:
> These examples cover the case of compiler-rt runtimes, but e.g. libc++ usually is installed in `<toolchain>/lib` right? And with the per-target runtimes, it's installed in `<toolchain>/lib/<triple>`.
> 
> It also affects how libc++'s `__config_site` is installed; most libc++ headers are installed in a target-independent directory, while `__config_site` is kept in a per-target directory.
Yes, this is incomplete. Will fix.

================
Comment at: llvm/docs/RuntimeBuild.rst:81
+  discover the macOS version of the machine that built Xcode's ``clang`` binary
+  this way!)
+
----------------
mstorsjo wrote:
> Hmm, I'm pretty sure the version number is picked up from the host OS. (I get the same version number, matching my version of macOS, when I run the clang binary from a bunch of differing old and new Xcode versions.)
Will check and update, thanks.

================
Comment at: llvm/docs/RuntimeBuild.rst:87
+* "``cpu``" is something like ``x86_64`` (64-bit intel), ``i386`` (32-bit
+  intel), ``armv7a`` (32-bit arm v7 application profile), ``arm64`` (64-bit arm
+  on darwin), ``aarch64`` (64-bit arm on non-darwin). It determines which CPU
----------------
mstorsjo wrote:
> Just as a tangentially related sidenote; `arm64` on darwin is a tricky case, as LLVM internally normalizes it to `aarch64` even on that OS - or at least it used to. `clang -arch arm64 -dumpmachine` did print `aarch64-apple-darwin...` still in Xcode 12; in Xcode 13 it keeps it as `arm64-apple-darwin...` though.
As a tangent to the tangent, (32-bit) arm is also weird in that (e.g.) `--target=arm-linux-androideabi` leads to a cc1 `-triple armv4t-unknown-linux-android`, but `--target=arm-linux-androideabi -march=armv7a` leads to a cc1 `-triple armv7-unknown-linux-android` – i.e. march flags affect the cc1 level triple. You can get the same cc triple by passing `--armv7a-linux-androideabi` without `-march` flag.

On the flipside, `--target=i386-linux` and `--target=i386-unknown-linux` result in the same cc1 triple (`i386-unknown-linux`).

I don't fully understand the motivation behind `LLVM_ENABLE_PER_TARGET_RUNTIME_DIR` yet. I thought it's so it's possible to build compiler-rt for different target configs, but it imho it currently doesn't work well because:

1. It uses the canonicalized driver-level triple.
  * This makes it impossible to build a toolchain that just works for users, since users can use a `--target` flag that canonicalizes to the same triple that you've built libraries for, but since clang looks in the non-canonicalized location you have to guess all possible --target spellings users might use and include symlinks or what
  * likewise, you have to guess if users will pick arm level via -march or via triple (or have symlinks for both)
  * before D115049, you had to have a dedicated symlink for every single android abi level
  * In the other direction, it makes it impossible to have different compiler-rt prebuilts for (say) avx512 on and off, since that's not in the triple

2. It's not really clear (to me) what the goal even is

================
Comment at: llvm/docs/RuntimeBuild.rst:100
+  targeting Windows, ``x86_64-pc-windows-msvc`` picks the Microsoft-compatible
+  ABI, while ``x86_64-pc-windows-mingw`` picks the gcc-compatible ABI. As
+  another example, when targeting Android, this is set to ``android`` (or,
----------------
mstorsjo wrote:
> In the normalized form, mingw (any OS string starting with `mingw32` iirc) corresponds to `windows-gnu`, not `windows-mingw`.
Thanks, wil fix.

================
Comment at: llvm/docs/RuntimeBuild.rst:141
+and others can be built either by including them in ``LLVM_ENABLE_PROJECTS``
+or in ``LLVM_ENABLE_RUNTIMES`` ("runtimes build").
+
----------------
mstorsjo wrote:
> FWIW, at least for libc++, building it with `LLVM_ENABLE_PROJECTS` is discouraged/deprecated these days. I think both options are still meant to work for e.g. compiler-rt though. (One usecase I've heard about there, is for other languages, who build LLVM and compiler-rt using the existing host C compiler, but don't actually build Clang at all.)
> 
> 
> Here it could also be useful to mention building by pointing cmake at `llvm-project/runtimes`, which is a standalone-ish build, where you manually configure cmake to use your desired (cross)compiler. Similar to pointing cmake directly at e.g. `llvm-project/libc++` before, but allowing processing more than one runtime in one cmake invocation.
Thanks, will add.

Side note: It's unfortunate that cmake doesn't have a way to achieve this (yet?) without building the runtimes in a sub-ninja-invocation. I tried `LLVM_ENABLE_PROJECTS` the other day, and one issue I ran into while debugging something was that `ninja -v` (at the outer level) didn't print verbose commands for the runtimes build. The GN build conceptually only supports a "runtimes build" (in that compiler-rt etc are always built with just-built clang), but it can do that in single build graph.

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D121761/new/

https://reviews.llvm.org/D121761