[PATCH] Teach CMake build system to run lit's own test suite

[Jonathan Roelofs via llvm-commits]

On 12/28/15 8:12 PM, Dan Liew wrote:
> Hi Jonathan,
>
>
>> Make them faster? Timeout tests don't have to be slow. Just set them up as
>> infinite loops that get killed by a very short timeout. If the timeout is
>> working, the test runs quick. If the timeout is not working, then the test
>> runs slowly (or never finishes), indicating that the test is broken.
>
> There are a few issues with making them run faster
>
> - Currently the timeout is an integer (seconds). If you want a shorter
> timeout than 1 second the implementation needs to be changed to use
> floats. If the timeout is made too short the tests may also become
> unreliable.
> - Some of the tests are more sophisticated than just running a tool
> that loops infinitely. For example the "quick_then_slow.py" has two
> ``RUN:`` lines and checks that when the timeout occurred it is during
> the second ``RUN:`` line and that the first invocation of the ``RUN:``
> line managed to finished.
> - The tests (e.g. ``shtest-timeout.py``) are run sequentially so that
> the output checked by FileCheck is in a predictable order.
> - The shell tests actually run multiple times. Once for the internal
> shell, once for the external shell and once to check that the timeout
> can be overriden correctly.

Another possibility is to stick them under: `REQUIRES: long_tests` which 
a few but not all of the buildbots will check.

>
>> One more issue:
>>
>> The lit testsuite writes temporaries to the src dir, which is inappropriate
>> for out-of-source builds. I implemented this in
>> http://reviews.llvm.org/D14722, and promptly forgot about it.
>
> Oh. I didn't know about that patch.
>
> I've implemented a new version of my patch (attached) that fixes the
> out of source build issue. My patch is similar to your implementation
> but there are some differences in how I've handled performing the copy
> of the tests from the source directory to the build directory.

LGTM with two nits:

No need for the "In source build" side of the conditional... in source 
builds are forbidden by the top-level CMakeLists.txt

Add this so that IDE-flavored builds don't get cluttered:

     set_target_properties(check-lit PROPERTIES FOLDER "Tests")


Cheers,

Jon

>
> Thanks,
> Dan.
>

-- 
Jon Roelofs
jonathan at codesourcery.com
CodeSourcery / Mentor Embedded