[llvm-dev] [lit] RFC: Per test timeout

[Dan Liew via llvm-dev]

Hi,

A feature I've wanted in lit for a while is a having a timeout per
test. Attached
are patches that implement this idea.

I'm e-mailing llvm-dev rather than llvm-commits
because I want to gather more feedback on my initial implementation and
hopefully some answers to some unresolved issues with my implementation.

Currently in lit you can set a global timeout for
all of the tests but not for each individual test.

The attached patches add

* Support for a new ResultCode called TIMEOUT
* A new command line option --max-individual-test-time
* Support for running external and internal ShTests with a per test timeout
* Support for running GTests with a per test timeout

I wanted to get some initial feedback on the implementation.

* If a timeout is requested the Python psutil module is required.
  This module does not ship with Python but is availble via pip
  (or on Linux your distribution's package manager). How do people feel
  about this? I don't like adding extra dependencies but this module
  makes it really easy to kill a process and all its children recursively
  in a platform neutral way. Note that if a per test timeout is not requested
  then the psutil module is not imported and lit acts just like it did
before my patches.

* If the platform running lit doesn't have psutil installed and a
timeout is requested
  an exception will be thrown. Should we provide a more friendly error message?
  If so where should this go in lit's code?

* I've not tested these patches on OSX or Windows. Is anyone on those
platforms willing
  to give them a try?

* The behaviour of --max-individual-test-time is a little at odds with
the behaviour of --max-time. --max-time will mark unexecuted tests as
UNRESOLVED whereas --max-individual-test-time will mark a test that
ran out of time as TIMEOUT. Is this okay?

* @Chris Matthews. Does the code that emits xunit xml files need to
change in anyway?

* @Daniel Dunbar. If these changes (in some form) end up being
committed would you be happy
  to push a new release of lit to PyPy?

-- 
Dan Liew
PhD Student - Imperial College London
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-lit-Partially-implement-support-of-per-test-timeout-.patch
Type: text/x-patch
Size: 11265 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20151114/56bf5111/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-lit-Implement-per-test-timeout-when-using-a-ShTest-a.patch
Type: text/x-patch
Size: 7601 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20151114/56bf5111/attachment-0001.bin>