[llvm-dev] Linking the FreeBSD base system with lld

[Ed Maste via llvm-dev]

I've been occasionally trying to build the FreeBSD base system (on
amd64 / x86_64) using lld in order to see how things are progressing.
With a few hacks in the FreeBSD build I can now link the base system
userland.

The missing functionality I had to work around is listed below. Some
of these should probably be addressed in our FreeBSD build
infrastructure, while others should be handled in lld. I'm listing
this here only as a snapshot of where lld stands today for our use.

1. -r / --relocatable (PR 23024)
There are four or five uses of -r in the FreeBSD base system, and at
least some of them cannot be handled by just using ar instead.

2. -E / --export-dynamic
This is used by our GDB build.

3. -Y [path]
Used by the 32-bit compat libraries. GNU ld documents this as "add
path to the default library search path." I don't understand exactly
what this option does, versus -L.

4. unrecognized reloc 22 (R_X86_64_GOTTPOFF)
Encountered in one of the libc tests.

5. --no-undefined vs _GLOBAL_OFFSET_TABLE_
A number of libraries are linked with --no-undefined. These fail wtih
an error of the form:
undefined symbol: _GLOBAL_OFFSET_TABLE_ in /<path>tmp/usr/lib/crtbeginS.o

6. emulation type "elf_i386_fbsd"
This is used by the 32-bit library build, which fails with "Unknown
emulation: elf_i386_fbsd"

7. static linking fails because _end is undefined (PR 25528)

8. Library search paths
The FreeBSD build infrastructure builds the toolchain twice: once for
the build host to build the target FreeBSD libraries and binaries, and
once for installation as the toolchain installed in the target. The
host toolchain is built with a set of baked-in library search paths
(.../tmp/lib and .../tmp/usr/lib).

I have a WIP patch to always apply --sysroot during the FreeBSD build,
but with this added it seems we do not get $SYSROOT/lib and
$SYSROOT/usr/lib as default search paths in lld. We also lack a
convenient way to debug this today in lld; with GNU ld the --verbose
option dumps a built-in linker script which includes the paths in use,
but lld does not have something equivalent.

Also, the kernel link fails because FreeBSD's kernel linker script is
too complicated for the current parser.


For reference, here are the workarounds I applied to build FreeBSD
userland with lld:

* Remove -Wl,--no-undefined from those library makefiles where it is set
* Disconnect lib/libpam/statc_modules from the build
* Disable the addition of -static in bsd.lib.mk
* Explicitly add the temporary build paths as -L options in various LDFLAGS
* Set the following options in src.conf:
WITHOUT_BOOT=yes
WITHOUT_RESCUE=yes
WITHOUT_LIB32=yes
WITHOUT_GDB=yes
WITHOUT_TESTS=yes
MODULES_OVERRIDE=