[cfe-dev] Configuration files

[Serge Pavlov via cfe-dev]

2017-03-20 12:48 GMT+07:00 James Y Knight <jyknight at google.com>:

> On Thu, Mar 16, 2017 at 9:25 AM, Serge Pavlov <sepavloff at gmail.com> wrote:
>
>> 2017-03-16 9:46 GMT+07:00 James Y Knight <jyknight at google.com>:
>>
>>>
>>> I'd really like to at least have a *design* for how this can eventually
>>> incorporate target-specific options before moving forward with adding a
>>> --config option, even if the initial commit won't implement the full design.
>>>
>>> I don't believe hand-wavy "maybe we'll add syntax that looks kinda like
>>> a comment so older versions will ignore it" is good enough there.
>>>
>>> I'd like to again keep in mind the use-case I mentioned a while ago.
>>> Approximately every linux distro configures GCC to set their default target
>>> cpu levels. E.g., Debian seems to set the following:
>>> - On x86, the default CPU should be i686.
>>> - But on x86-64, the default CPU isn't changed (aka it's left as
>>> "x86-64").
>>> - For ppc32/ppc64, the default CPU should be power7 but tune for power8.
>>> - For ppc64le, the default CPU should be power8.
>>> - On ARM (hf), armv7-a should be the default cpu, vfpv3-d16 the default
>>> fpu, and it should default to thumb mode.
>>> etc...
>>>
>>> Note that those defaults are different on different releases of the
>>> distro.
>>>
>>> The way you do this with GCC is via options passed to the configure
>>> script: --with-arch-32= --with-arch-64= --with-fpu= --with-mode= etc. which
>>> turn into values used in the target-specific OPTION_DEFAULT_SPECS macro.
>>> Since GCC only builds for one target at a time (or two, if you count 32/64
>>> separately), and you're expected to need to build a new gcc any time you
>>> want to cross-compile, that's sufficient.
>>>
>>> Clang is intrinsically a cross-compiler, so gcc's solution isn't good
>>> enough for clang (nor is clang's current behavior enough). So, what's the
>>> plan to actually solve this?
>>>
>>
>> Initial versions of this proposal defined two kinds of config files:
>> - named, which should be explicitly specified by a user by option
>> --config or be encoded into executable name as `armv7l-clang`.
>> - default, which is loaded always much like `.bashrc` or any similar file.
>> These two kinds of config file shared implementation but addressed
>> different use cases, which made confusion during discussion. To facilitate
>> review process the support of default config files was removed from the
>> proposal. The issues you mention should mostly be solved by the default
>> config files.
>>
>
> OK -- focusing on cross-compilation for a minute. Clang and LLD have the
> potential to make cross-compilation *MUCH* simpler than it's traditionally
> been, and I'd like to be careful to think about how to take the best
> advantage of the fact that we're getting really close to having a single
> toolchain which is able to easily build programs for any target
> architecture/environment.
>
> Here's the scenario I'm using in my mind:
>
> Let's say I've got a super-fast RISC-V machine running OpenBSD, and want
> to cross-build from that to all of the various supported Debian
> architectures. I want to use my existing OpenBSD-supplied clang+lld
> toolchain, because that toolchain can natively cross-compile to everything,
> if given appropriate command-line arguments. So, I don't need to make a
> special build of the compiler or linker. Woohoo!
>
> I have copied over the root partition of the target Debian system into a
> directory on this machine, so I have headers and libraries to link against.
>
> (I also need the crtbegin/end and libgcc or compiler-rt. For this thread,
> let's assume I copied those over as well, even though they're typically
> "part of" the compiler toolchain suite. A separate discussion should be had
> about what to do about making THAT part of the cross-compile story easier
> -- it's almost the only tricky part remaining!)
>
> Now, given that scenario --
>
> How do I specify that I want to cross-build to that debian system?
>
> I need to know the proper configuration, and I need to know how I'm
> supposed to invoke clang to use the configuration.
>
> At the moment, you might run something like:
>  "clang --target x86_64-linux-gnu --sysroot=/path/to/debian-installation
> --other-super-important-cpu/abi-flags"
>
> With your proposed patch, I would create a config file,
> "x86_64-debian.cfg", containing the above flags. I could put it in
> /etc/something or ~/.something, and run:
> $ clang -config x86_64-debian
>
> Alternatively, I could put it in ~/bin/x86_64-debian.cfg and
> $ ln -s /usr/bin/clang ~/bin/x86_64-debian-clang
> $ x86_64-debian-clang
>
>
In general you are right I think.


> If the default config files were implemented in clang, driver would search
>> binary directory for default configuration description. If compiler is
>> named as `armv7l-clang`, driver first tries file `armv7l.cfg` then
>> `clang.cfg`. If config file is found, options listed there are put into the
>> set of driver arguments before any option specified in command line.
>>
>
>
>> With this feature a distribution or SDK can supply set of config files as
>> a part of clang package and tunes compiler appropriately.
>>
>
>
>> 1. There needs to a way to be able to configure the defaults for all
>>> supported architectures of the platform (either in a single config, or in
>>> multiple that clang knows how to select).
>>>
>>>
>> Each supported target can have separate config file.
>>
>
> But, despite the target being x86_64, some software builds 32-bit x86
> code, and will run "x86_64-debian-clang -m32" to do so. In that case, where
> does clang get the appropriate default x86 cpu from? (Note again that GCC
> compiles-in separate defaults for the 32 and 64-bit architecture variants
> to handle this).
>
>
This is a real problem. In fact -m32 convert the specified target
`x86_64-debian-clang` into `i686-debian-clang`. We cannot rely on the
specified target as effective target can be different. This is the root of
evil.

Simple solution as config file cannot solve this problem. We could:
- extend syntax of config file by allowing some sort of conditional
directives, or
- add additional logic (target specific) that reloads config file if
effective target changes. For instance, invocation of `x86_64-debian-clang
-m32` first loads `x86_64-debian.cfg` which contains -target=..., which
sets defaults for 64-bit target, then command line is scanned to calculate
effective target, the option `-m32` changes the effective target to
`i686-debian-clang`. Driver then removes all options read from config file
and puts defaults from `i686-debian.cfg`. Only then the command line
options are processed by regular logic. It sounds complex, it fact it can
be easily implemented.

More complex solution such as spec file does not suffer this problem, it
can check presence of options like `m32` and conditionally choose
appropriate defaults.


> If it's the case that each supported target has its own config file, clang
> will somehow need to know how to choose the "i386-debian.cfg" file when
> invoked as "x86_64-debian-clang -m32"....or something like that.
>
> 2. Those platform defaults should, somehow, avoid interfering with the use
>>> of clang to cross-compile TO a different platform, and be easy to use FROM
>>> a different host platform. (find default config via sysroot, maybe?)
>>>
>>>
>> Default config in sysroot could be included by default clang config,
>> however driver must know where the sysroot is. We could support a set of
>> macros, for instance expand `$TARGET` in config file to target name as
>>  `armv7l-clang`. This topic is not elaborated yet.
>>
>
> So -- I should've separated out the two points in my comment above.
>
> 2.1 (TO different): What happens if you do something like
> "x86_64-debian-clang -target armv7a-apple-darwin". Is that an error? Or, if
> not, what does it do? Does it still use the x86_64-debian-clang config
> file? Or, does it completely ignore the config file?
>
>
I think multiple options `-target` must produce error. We now can call
clang in such way: `clang -target x86_64-linux -target armv7-apple-ios6.0.0
abc.c` and clang successfully executes. Is there any use case for such
target override?


> This problem isn't so major for explicitly-specified configs ("Don't do
> that!"), but becomes a larger issue if there's a default config file.
>
> This is why I was pondering if -target perhaps should not be the
> recommended way for end-users to select an architecture. The "-target" flag
> could be made to mean "ignore the platform configuration, just use this
> target with clang defaults". And, the "-arch" flag could mean: "within the
> current configuration, select this named architecture configuration".
>
>
It would mean that clang must somehow calculate other things as ABI.


> 2.2 (FROM different): Default config files could be searched for inside
> the specified sysroot. That is, "clang --sysroot=/wherever/" would find the
> proper configs for the platform automatically, if they've been distributed
> in the sysroot. The canonical way to cross-build with clang would be to use
> your existing compiler, and specify two additional arguments: --sysroot and
> -target (or -arch?).
>
> 3. How do we recommend users select a target?
>>> Do we need to look for the proper defaults when the user specifies
>>> "--target $TARGET" to clang?
>>> Or maybe we favor of the "$TARGET-clang" symlink method?
>>> Or maybe "--target" is a low-level feature not recommended for
>>> end-users, and we should steer people to using something like "-arch", to
>>> select a named architecture within the current platform configuration, like
>>> apple does with darwin-based platforms now?
>>>
>>>
>> To specify a target looks like a more flexible solution. "$TARGET-clang"
>> symlink method was already implemented in early versions of
>> https://reviews.llvm.org/D24933. It is possible also to extend treatment
>> of `--target` so that it acted similar to `--config`.
>>
>
> I note that Apple has hardcoded behavior in clang for their targets, that
> lets it do something sensible when specifying the sysroot:
>
> That is, you can do:
> $ clang -isysroot /path/to/iPhoneOS9.3.sdk -arch arm64
> [Note, the "i" in the "isysroot" in that commandline there is correct, but
> a presumably-historical bogosity. It actually finds libraries in the
> specified "i"sysroot, too.]
>
> Given the above, it:
> 1. Sets the target platform to ios9.3 (*-apple-ios9.3.0), based on
> matching the pathname specified by isysroot.
> 2. Sets the target to aarch64 (printed arm64-apple-watchos2.2.0), based on
> mapping the -arch name back to a triple.
> 3. Also sets the default CPU to "cyclone". (-target-cpu cyclone), based on
> hardcoded behavior for "-arch arm64".
>
> That seems a pretty good example of what I'm talking about, except it's
> been implemented with driver hacks instead of a generic config file support.
>

That would be one more way to specify config file, in addition to option
`--config` and symlink `x86_64-debian-clang`. Does such way have advantage
over these ways?
$ clang --config /path/to/iPhoneOS9.3.sdk.cfg
or
$ clang --config iPhoneOS9.3.sdk.cfg

Sysroot directories could contain config files that set defaults
appropriate for this SDK. For instance, directory /path/to/iPhoneOS9.3.sdk
contains file platform.cfg. Config file loaded by clang could contain
command `@/path/to/iPhoneOS9.3.sdk/platform.cfg`. That could be used to
separate per-SDK defaults from other defaults.

I don't object to such use case, we just need to find balance between
solution complexity, user convenience and maintainability.

Thanks,
--Serge
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20170320/20d46bc9/attachment.html>