[llvm-commits] lit configuration for test-suite and some minor test changes for MSVC.

[Michael Spencer]

Attached is a lit configuration for running the single source tests in
the test suite. It has been tested on Linux and Windows with MSVC,
clang, and gcc.

Usage:
To use this simply call lit.py (llvm-lit does not currently know how
to run these tests) and pass it a directory under
test-suite/SingleSource. The config looks for available compilers in
PATH and runs each one it finds on each source file that also has a
.reference_output.

Options:
The following --param's are supported.
* no-default-flags={True,False} - Defaults to False. If True no
default flags are used on any compiler. This only includes flags set
in the lit.cfg, not those set in lit.local.cfg.
* <compiler>_[<lang>_]flags - Defaults to "". A space separated quoted
string of arguments. The exact semantics are explained below.

Flags are formatted as [<compiler>_][<lang>_]flags.
They are loaded first from lit.cfg, then lit.site.cfg, then the
lit.local.cfg's, and finally from the command line using --param. Flags
specified in lit config files follow Python list rules for replacement,
appending, and removing. Flags specified in --param are replaced if the first
char is '=', appended if it is '+=', and removed if it is '-='.

The available flags are:
* <compiler>_flags are used for all languages on <compiler>.
* <compiler>_<lang>_flags are used for <lang> in {c, cxx, objc, objcxx} on
  <compiler>.
* gcccompatible_[<lang>_]flags are used for all gcc compatible compilers. The
  lang option is the same as above.

flags are added/removed in the following order AFTER each has been loaded and
merged:
* gcccompatible_flags - if the compiler supports them
* <compiler>_flags
* gcccompatible_<lang>_flags - if the compiler supports them
* <compiler>_<lang>_flags

Currently supported compilers are:
* MSVC - Microsoft Visual C/C++. Which ever version is on the PATH is used.
  This can be changed by running the vcvars.bat script of the on you want to
  use and then run lit from that environment.
* clang - The LLVM C Language Family Compiler.
* gcc - The GNU Compiler Collection.

lit.local.cfg:
The compilation of each source file is configurable with the following
variables in each directory's lit.local.cfg (they default to whatever
value the parent config had).

* config.(all the flags above) - These follow the Python list rules. -
ex. config.gcccombatible_flags -= ['-fomit-frame-pointer'] # Removes
-fomit-frame-pointer from flags.
* config.unsupported['<compiler>']. - This is either a list of strings
of unsupported files (without extension), or the value True if none
are supported.
* config.compilers - A dictionary mapping compiler names to compilers.
This may be used to add a compiler for a specific directory.
* config.hashed - If this is True, the output of the program is hashed
using md5 before it is compared to the reference output.

TODO:
There are a few issues with this config, many of which stem from lit
infrastructure problems that Daniel and I have been discussing.
However, there are others unrelated to that:
* There is currently no way to specify exactly which executables to
use to compile, link, JIT, and such. This needs to be set in a
lit.site.cfg and have an associated param.
* With the current setup there are two different "compilers" for
native executables (which actually generates two executables. One with
integrated as and one without) and bitcode to be run with lli. What I
would like is to separate compilers from what is being generated. A
major use of this would be to directly compare the same compiler with
a different set of flags.
* There's currently no way to generate statistics other than
PASS/FAIL. I would like to be able to be able to get the time
measurement granularity down to {compile, [JIT,] run}.

I'm sure there are other issues, but I think it's far enough along for
others to test out.

I have also attached another patch that contains (for some tests) the
minimal changes required to get them to compile with MSVC and clang on
Windows. Most of them are header file issues, but some are due to
MSVC's ignorance of any recent C standard. The rest are differences in
extensions.

I thought about just adding headers with the same name as the ones
Windows doesn't have have including them on Windows, but I decided
against because then the tests would no longer be stand alone. Running
them without lit would require explicitly adding some crazy include
path on the command line that is not stable across systems.

- Michael Spencer
-------------- next part --------------
A non-text attachment was scrubbed...
Name: msvc-compat.patch
Type: application/octet-stream
Size: 8442 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20100926/251288be/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lit-test-suite-runner.patch
Type: application/octet-stream
Size: 28391 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20100926/251288be/attachment-0001.obj>