[cfe-dev] Configuration files

[James Y Knight via cfe-dev]

On Thu, Mar 16, 2017 at 9:25 AM, Serge Pavlov <sepavloff at gmail.com> wrote:

> 2017-03-16 9:46 GMT+07:00 James Y Knight <jyknight at google.com>:
>
>>
>> I'd really like to at least have a *design* for how this can eventually
>> incorporate target-specific options before moving forward with adding a
>> --config option, even if the initial commit won't implement the full design.
>>
>> I don't believe hand-wavy "maybe we'll add syntax that looks kinda like a
>> comment so older versions will ignore it" is good enough there.
>>
>> I'd like to again keep in mind the use-case I mentioned a while ago.
>> Approximately every linux distro configures GCC to set their default target
>> cpu levels. E.g., Debian seems to set the following:
>> - On x86, the default CPU should be i686.
>> - But on x86-64, the default CPU isn't changed (aka it's left as
>> "x86-64").
>> - For ppc32/ppc64, the default CPU should be power7 but tune for power8.
>> - For ppc64le, the default CPU should be power8.
>> - On ARM (hf), armv7-a should be the default cpu, vfpv3-d16 the default
>> fpu, and it should default to thumb mode.
>> etc...
>>
>> Note that those defaults are different on different releases of the
>> distro.
>>
>> The way you do this with GCC is via options passed to the configure
>> script: --with-arch-32= --with-arch-64= --with-fpu= --with-mode= etc. which
>> turn into values used in the target-specific OPTION_DEFAULT_SPECS macro.
>> Since GCC only builds for one target at a time (or two, if you count 32/64
>> separately), and you're expected to need to build a new gcc any time you
>> want to cross-compile, that's sufficient.
>>
>> Clang is intrinsically a cross-compiler, so gcc's solution isn't good
>> enough for clang (nor is clang's current behavior enough). So, what's the
>> plan to actually solve this?
>>
>
> Initial versions of this proposal defined two kinds of config files:
> - named, which should be explicitly specified by a user by option --config
> or be encoded into executable name as `armv7l-clang`.
> - default, which is loaded always much like `.bashrc` or any similar file.
> These two kinds of config file shared implementation but addressed
> different use cases, which made confusion during discussion. To facilitate
> review process the support of default config files was removed from the
> proposal. The issues you mention should mostly be solved by the default
> config files.
>

OK -- focusing on cross-compilation for a minute. Clang and LLD have the
potential to make cross-compilation *MUCH* simpler than it's traditionally
been, and I'd like to be careful to think about how to take the best
advantage of the fact that we're getting really close to having a single
toolchain which is able to easily build programs for any target
architecture/environment.


Here's the scenario I'm using in my mind:

Let's say I've got a super-fast RISC-V machine running OpenBSD, and want to
cross-build from that to all of the various supported Debian architectures.
I want to use my existing OpenBSD-supplied clang+lld toolchain, because
that toolchain can natively cross-compile to everything, if given
appropriate command-line arguments. So, I don't need to make a special
build of the compiler or linker. Woohoo!

I have copied over the root partition of the target Debian system into a
directory on this machine, so I have headers and libraries to link against.

(I also need the crtbegin/end and libgcc or compiler-rt. For this thread,
let's assume I copied those over as well, even though they're typically
"part of" the compiler toolchain suite. A separate discussion should be had
about what to do about making THAT part of the cross-compile story easier
-- it's almost the only tricky part remaining!)

Now, given that scenario --

How do I specify that I want to cross-build to that debian system?

I need to know the proper configuration, and I need to know how I'm
supposed to invoke clang to use the configuration.

At the moment, you might run something like:
 "clang --target x86_64-linux-gnu --sysroot=/path/to/debian-installation
--other-super-important-cpu/abi-flags"

With your proposed patch, I would create a config file,
"x86_64-debian.cfg", containing the above flags. I could put it in
/etc/something or ~/.something, and run:
$ clang -config x86_64-debian

Alternatively, I could put it in ~/bin/x86_64-debian.cfg and
$ ln -s /usr/bin/clang ~/bin/x86_64-debian-clang
$ x86_64-debian-clang

If the default config files were implemented in clang, driver would search
> binary directory for default configuration description. If compiler is
> named as `armv7l-clang`, driver first tries file `armv7l.cfg` then
> `clang.cfg`. If config file is found, options listed there are put into the
> set of driver arguments before any option specified in command line.
>


> With this feature a distribution or SDK can supply set of config files as
> a part of clang package and tunes compiler appropriately.
>


> 1. There needs to a way to be able to configure the defaults for all
>> supported architectures of the platform (either in a single config, or in
>> multiple that clang knows how to select).
>>
>>
> Each supported target can have separate config file.
>

But, despite the target being x86_64, some software builds 32-bit x86 code,
and will run "x86_64-debian-clang -m32" to do so. In that case, where does
clang get the appropriate default x86 cpu from? (Note again that GCC
compiles-in separate defaults for the 32 and 64-bit architecture variants
to handle this).

If it's the case that each supported target has its own config file, clang
will somehow need to know how to choose the "i386-debian.cfg" file when
invoked as "x86_64-debian-clang -m32"....or something like that.

2. Those platform defaults should, somehow, avoid interfering with the use
>> of clang to cross-compile TO a different platform, and be easy to use FROM
>> a different host platform. (find default config via sysroot, maybe?)
>>
>>
> Default config in sysroot could be included by default clang config,
> however driver must know where the sysroot is. We could support a set of
> macros, for instance expand `$TARGET` in config file to target name as
>  `armv7l-clang`. This topic is not elaborated yet.
>

So -- I should've separated out the two points in my comment above.

2.1 (TO different): What happens if you do something like
"x86_64-debian-clang -target armv7a-apple-darwin". Is that an error? Or, if
not, what does it do? Does it still use the x86_64-debian-clang config
file? Or, does it completely ignore the config file?

This problem isn't so major for explicitly-specified configs ("Don't do
that!"), but becomes a larger issue if there's a default config file.

This is why I was pondering if -target perhaps should not be the
recommended way for end-users to select an architecture. The "-target" flag
could be made to mean "ignore the platform configuration, just use this
target with clang defaults". And, the "-arch" flag could mean: "within the
current configuration, select this named architecture configuration".

2.2 (FROM different): Default config files could be searched for inside the
specified sysroot. That is, "clang --sysroot=/wherever/" would find the
proper configs for the platform automatically, if they've been distributed
in the sysroot. The canonical way to cross-build with clang would be to use
your existing compiler, and specify two additional arguments: --sysroot and
-target (or -arch?).

3. How do we recommend users select a target?
>> Do we need to look for the proper defaults when the user specifies
>> "--target $TARGET" to clang?
>> Or maybe we favor of the "$TARGET-clang" symlink method?
>> Or maybe "--target" is a low-level feature not recommended for end-users,
>> and we should steer people to using something like "-arch", to select a
>> named architecture within the current platform configuration, like apple
>> does with darwin-based platforms now?
>>
>>
> To specify a target looks like a more flexible solution. "$TARGET-clang"
> symlink method was already implemented in early versions of
> https://reviews.llvm.org/D24933. It is possible also to extend treatment
> of `--target` so that it acted similar to `--config`.
>

I note that Apple has hardcoded behavior in clang for their targets, that
lets it do something sensible when specifying the sysroot:

That is, you can do:
$ clang -isysroot /path/to/iPhoneOS9.3.sdk -arch arm64
[Note, the "i" in the "isysroot" in that commandline there is correct, but
a presumably-historical bogosity. It actually finds libraries in the
specified "i"sysroot, too.]

Given the above, it:
1. Sets the target platform to ios9.3 (*-apple-ios9.3.0), based on matching
the pathname specified by isysroot.
2. Sets the target to aarch64 (printed arm64-apple-watchos2.2.0), based on
mapping the -arch name back to a triple.
3. Also sets the default CPU to "cyclone". (-target-cpu cyclone), based on
hardcoded behavior for "-arch arm64".

That seems a pretty good example of what I'm talking about, except it's
been implemented with driver hacks instead of a generic config file support.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20170320/37de9a3e/attachment.html>