[cfe-dev] RFC: Clang test runner changes

[Chris Lattner]

On Jul 27, 2009, at 3:32 PM, Daniel Dunbar wrote:
> Hi all,
>
> I thought I would give a little roadmap for where I am taking clang's
> test runner. You can consider it an RFC and give feedback if you like!
> :)

Sounds awesome to me!

-Chris

>
> --
>
> Problem:
> We want a portable way to run clang's tests. The immediate motivation
> is to make it easier to test clang on Windows, but there are a lot of
> related things I would like to improve:
> 1. Low interaction with the build system, so that test results are
> consistent between build systems and platforms.
>
> 2. Easy to use, it should be straightforward to run tests and
> understand failures.
>
> 3. Good reporting, for example to improve the logs in buildbot.
>
> 4. Long-term, I hope to kill off DejaGNU in LLVM. I could make a lot
> of claims about why, but the real reason is just because I want it to
> run faster (i.e., be multithreaded). :)
>
> --
>
> My plan, not necessarily in order:
> - Rename it.
>
> - Add a configuration file, which specifies things like what variable
> substitutions should occur (this is part of #4 above). This is a
> static configuration file, not a build system configure-step product.
>
> - Kill TestRunner.sh, users should call the test tool directly. The
> syntax will be the same, but the test tool will locate the
> configuration file by walking up the path hierarchy. This will make it
> easy to test a single component, e.g. CodeGen or Sema. It would be
> nice to fix things at the same time so that running the tool from the
> command line used the same substitutions as "make check" (or the
> matching equivalent for your build environment), instead of relying on
> the PATH variable. This requires some interaction with the build
> system to work.
>
> - Implement the execution of RUN lines internally to the tool. The
> format will probably become a bit more strict, instead of "sh" format.
> The runner can then start to validate the lines, for example to ensure
> that only certain tools are called (clang-cc, count, not, FileCheck,
> etc.). The runner can then also implement certain features internally
> (like grep).
>
> - With the latter step, the runner can now be more helpful in
> pointing out what test step failed, showing the contents of related
> temp files, etc.
>
> - Daniel
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev