[cfe-dev] RFC: Clang test runner changes

[Daniel Dunbar]

Hi all,

I thought I would give a little roadmap for where I am taking clang's
test runner. You can consider it an RFC and give feedback if you like!
:)

--

Problem:
We want a portable way to run clang's tests. The immediate motivation
is to make it easier to test clang on Windows, but there are a lot of
related things I would like to improve:
 1. Low interaction with the build system, so that test results are
consistent between build systems and platforms.

 2. Easy to use, it should be straightforward to run tests and
understand failures.

 3. Good reporting, for example to improve the logs in buildbot.

 4. Long-term, I hope to kill off DejaGNU in LLVM. I could make a lot
of claims about why, but the real reason is just because I want it to
run faster (i.e., be multithreaded). :)

--

My plan, not necessarily in order:
 - Rename it.

 - Add a configuration file, which specifies things like what variable
substitutions should occur (this is part of #4 above). This is a
static configuration file, not a build system configure-step product.

 - Kill TestRunner.sh, users should call the test tool directly. The
syntax will be the same, but the test tool will locate the
configuration file by walking up the path hierarchy. This will make it
easy to test a single component, e.g. CodeGen or Sema. It would be
nice to fix things at the same time so that running the tool from the
command line used the same substitutions as "make check" (or the
matching equivalent for your build environment), instead of relying on
the PATH variable. This requires some interaction with the build
system to work.

 - Implement the execution of RUN lines internally to the tool. The
format will probably become a bit more strict, instead of "sh" format.
The runner can then start to validate the lines, for example to ensure
that only certain tools are called (clang-cc, count, not, FileCheck,
etc.). The runner can then also implement certain features internally
(like grep).

 - With the latter step, the runner can now be more helpful in
pointing out what test step failed, showing the contents of related
temp files, etc.

 - Daniel