[cfe-dev] Configuration files

Tue Nov 8 23:59:51 PST 2016

The proposal has been evolved due to feedback. Below is a summary of
updated version and feedback from various sources.

**Problems**

The two main problems this feature addresses are:

1. Ability to modify default compiler settings.

As an example, the warning `-Wundefined-var-template` can be helpful for
people doing module-enabled builds   (
https://llvm.org/bugs/show_bug.cgi?id=24425), but it makes headache for
others, who even come with ideas to turn this warning off by default (
http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20160808/167354.html
).

To cope with this and similar problem clang need a mechanism that
effectively change default settings hardcoded in compiler sources.

2. Support of cross builds.

Doing cross builds requires to specify lots of options that set target,
backend options, include and library directories etc. These options could
be placed into config files and then be specified with single option or
even without such. This feature could make cross compilation with clang
more convenient.

**Solution**

Configuration file groups related options together and allow to specify
them in more simple and less error prone way than just listing them
somewhere in build scripts. According to the two problems to solve, there
are two use patterns of configuration files:

- Default config file is applied without need to specify anything in
compiler invocation. It can be read from predefined location or be
specified by environmental variable. It allows to override compiler
defaults.
- Named config file is specified explicitly by a user, either by command
line option `--config`, or by using special executable file names, such as
`armv7l-clang`. Such configuration may be thought as a "macro" that
designates an option set and is expanded when compiler is called.

Only one config file is used, if it is specified explicitly, default config
file is not searched. Config file may contain comments (# style),
references to other config files using syntax `@file`.

Config file are searched in the directory where clang executable resides.
User may specify any config file using option `--config` followed by full
path of the file. Alternatively the path to config file may be specified in
environmental variable CLANGCFG.
**Existing experience**

Configuration file is not a new concept, it has been used in various form
in many compilers including clang.

1. Intel compiler supports default configuration files (
https://software.intel.com/en-us/node/522780), the file is searched for in
the directory where executable resides or can be specified by setting
environmental variable. The feature is pretty convenient in practice.
2. Clang allows to specify target by using special executable names, like
`armv7l-clang`. Only target name can be specified by this way, which is not
sufficient to tune compiler for a new target. Obvious question arise - why
not extend this mechanism for other target specific options too.
3. Clang allows modifying options using environmental variable
`CCC_OVERRIDE_OPTIONS`. This is undocumented way and is looks more like a
hack for internal use. Its existence however indicates that such
functionality is useful.
4. Cross compilation environment ELLCC (http://ellcc.org), based on clang,
supports advanced config files in YAML format. It has proven to be very
handy to support a wide variety of targets.
5. GCC has support of spec files (
https://gcc.gnu.org/onlinedocs/gcc/Spec-Files.html), which can be taken
from default location or be specified by command line option `--spec`. Spec
files are more powerful tool than just config files as they not only set
options but also program driver logic. Using custom spec files allows GCC
to be tuned to particular target without need to specify lots of options in
command line.
6. Visual Studio supports custom project templates (
https://msdn.microsoft.com/en-us/library/xkh1wxd8.aspx), which may specify
any compiler flags as default setting. Although it can be considered as
feature of build system rather than compiler, it allows to override default
compiler settings.
7. Keil C compilers allow to specify compiler options in environmental
variables, ARMCOMPILER6_CLANGOPT (
http://www.keil.com/support/man/docs/armclang_intro/armclang_intro_chr1374139991387.htm)
or ARMCC5_CCOPT (
http://infocenter.arm.com/help/topic/com.arm.doc.dui0592e/pge1399635613963.html)
depending on compiler version.

**Concerns**

Somewhat random set of concerns extracted from discussions with answers to
them.

1. With default config files the options specified for clang in command
line do not represent definite state anymore, because some options may come
from the config file. A user that prepares bug report can forget to specify
content of the used default config file, it can make difficult to reproduce
failure conditions.
Answer: Actual set of compiler options is provided in error dump files, in
output of `clang -v` and `clang -###`, these invocations also report used
config file. If a project wants to disable some warnings by default, add it
to the Makefile.
2. It is not a business of compiler to adjust options. What flags to pass
to the compiler should be configured in a project's build system.
Answer: A project may include dozens of components each come with own build
systems, which are tuned differently. Retargeting such build system can be
time consuming and error prone. On the other hand, setting compiler and
target specific options is not a business of a component, ideally it may be
unaware of using in cross compilation environment. Also, not all component
build systems are flexible enough. As a result some users do not want
modify their build files, they would prefer customized compiler.
Config files brings more power and convenience to the compiler.
3. Clang already has `@file` for bundling up compiler flags.
Answer: Config file make this feature a bit more convenient by providing a
way to choose proper option set and making files containing options more
user friendly by allowing comments and nested `@file`. Also `@file` cannot
solve the problem of overriding compiler defaults.
4. The idea of having the compiler search for config files in random places
on the filesystem seems scary.
Answer: Default config files are searched only in the directory where
compiler executable file resides. If compiler is installed from package,
this file is placed in system directories and should not be changed by a
simple user. It is assumed that such file is a part of SDK or some package
prepared by experienced person, it is not changed frequently. The same
applies to target-specific config files. A user may specify full path to
config file, in this case the file may be anywhere but this explicit action.
Most of elaborated environments allows search for libraries, packages,
classes etc in some predefined places. Running simple command `gcc file.c`
make compiler search set of various places, and the set may change
depending on version and OS variant.
5. There is already a lot of magic in the driver to figure out how to
invoke the tools, this feature makes driver more complex.
Answer: Orchestrating tools made by clang driver is already a complex task
and hardly it can be made simple. But it can be made more structured,
config file may be a step in this direction making existing target
selection in clang more powerful and useful.
6. This is too primitive solution:
- A good solution must be able to deduce target from arguments (-m32).
- We should also probably automatically silence unused options from config
files.  For example, if my cross-compile toolchain passes some extra linker
arguments with -Wl,-foo, then I don’t want every single line in my build to
complain that -Wl is unused in conjunction with -c.
- It should provide default mapping from targets to sysroots and associated
paths; we currently hardcode a lot of this into the compiler itself.
- Any workable config-file scheme *must* interact with the various target
specification command-line arguments, and allow for setting options
dependent upon the actually-selected target.
- Maybe it'd be better to create a regular, .ini/.conf-style configuration
file.
Answer: This is a simple solution, that solves a good deal of problems. It
does not require big changes in documentation and substantial maintenance
efforts. It can be extended in future by allowing more complex constructs
in config files.

Thanks,
--Serge

2016-10-05 0:29 GMT+07:00 Reid Kleckner <rnk at google.com>:

> My main concern with configuration files is that they should interact with
> regular command line flags in a predictable and maintainable way. Your
> approach of extending response files seems like it's going in the right
> direction.
>
> I'm not excited about adding something like ELLC's YAML format to Clang (
> http://ellcc.org/viewvc/svn/ellcc/trunk/libecc/config/
> ppc64el-linux?view=markup) because then we will have to document and
> maintain the semantics of things like multiple configuration files, in what
> order they override each other, and how they interact with flags. Our
> command line flag logic is already too complicated. We have one public
> interface, and it's flags, and I'm hesitant to add a new one.
>
> On Mon, Sep 19, 2016 at 1:06 AM, Serge Pavlov via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
>> Hi all,
>>
>> I would like to propose implementing configuration files in clang. The
>> idea is to read some set of options prior to the options specified in a
>> command line in every tool invocation. These options are taken from a file
>> which is searched either in predefined place or its location is specified
>> by environmental variable.
>>
>> Few words about particular problem this facility can solve. Clang issues
>> many warnings, some refer to potential errors, some merely attract
>> attention to code that can cause problems in some circumstances but
>> otherwise is pretty innocent. For example, warning -Wundefined-var-template
>> is issued for template usages, which nor cannot be instantiated implicitly
>> neither are mentioned in explicit instantiation declarations. Such usage is
>> not an error or bad style, but the warning may be helpful for a user in a
>> complex case, such as described in PR24425. For other users this warning
>> may be annoying and they would prefer to turn it off by default (see
>> discussion in http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-
>> 20160808/167354.html). Different categories of users like to have
>> different set of warnings enabled by default.
>>
>> If configuration files were supported by clang, a user can put
>> -Wno-undefined-var-template to such file and this would have the same
>> effect as if the unwanted warning were turned off by default. No changes to
>> build scripts would be required.
>>
>> Configuration files are supported by the Intel Compiler (
>> https://software.intel.com/en-us/node/522780) and similar behavior is
>> proposed to be implemented by clang.
>>
>> By default the configuration file is searched in the same directory where
>> tool executable resides. In the case of clang the file would be
>> 'clang.cfg', for other tool named 'foo'- 'foo.cfg'. User can specify any
>> location for this file if he sets up environmental variable 'CLANGCFG' (or
>> 'FOOCFG'). The variable should contain path to configuration file. If the
>> configuration file does not exist, it is silently ignored. Content of
>> configuration file is treated as a part of command line of each tool
>> invocation preceding any arguments specified in command line.
>>
>> Does implementation of this facility makes sense?
>>
>> Thanks,
>> --Serge
>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20161109/97ae1eb2/attachment.html>