[PATCH] lit: Incremental test scheduling

[Alp Toker]

Incremental testing is useful when you're writing code and need to
answer these questions fast:

  * Did my recent change fix tests that were previously failing?
  * Are the tests I just modified passing?


A standard check-clang run takes around 40 seconds on my system, and
needs to be run repeatedly while working on a new feature or bug fix.
Most of that is time spent waiting.

With the attached patch, lit instead runs failing and recently modified
tests first. This way, I'm likely to get an answer to both questions in
under one second, letting me get straight back to fixing the issue.

*Workflow**
*
After fixing the code and running lit again, if there are still failures
at the start I'll usually hit Ctrl-C, try to fix the issue and repeat
until all failures are resolved.

The feature isn't just for local testing -- on our internal clang build
server, LLVM test failures or fixes often get detected within 10 seconds
of a commit thanks to ccache/cmake/ninja together with this patch.

*Performance

*The additional ordering overhead and cache updates come at a cost of
about 2 seconds on top of a 40 second run.

So enabling the feature depends on your workflow and whether you're able
to act upon the early results.

For example, the Buildbot version used on llvm.org doesn't send failure
notifications until the test run is complete so the feature would only
be marginally useful there until Buildbot gets early notification support.
*
Implementation notes*

lit shell tests and compiled unit tests are both supported. At the
moment, the cache is implemented by scanning and modifying test source
mtimes directly. This hasn't caused trouble but the technique could be
refined in future to store test failures in a separate cache file if needed.

*Status**
*
Not yet tested on Windows. Let me know if you find this useful!

Alp.

-- 
http://www.nuanti.com
the browser experts

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20131025/3f13a776/attachment.html>
-------------- next part --------------
diff --git a/utils/lit/lit/Test.py b/utils/lit/lit/Test.py
index b4988f5..e51bf12 100644
--- a/utils/lit/lit/Test.py
+++ b/utils/lit/lit/Test.py
@@ -128,10 +128,11 @@ class TestSuite:
 class Test:
     """Test - Information on a single test instance."""
 
-    def __init__(self, suite, path_in_suite, config):
+    def __init__(self, suite, path_in_suite, config, file_path = None):
         self.suite = suite
         self.path_in_suite = path_in_suite
         self.config = config
+        self.file_path = file_path
         # A list of conditions under which this test is expected to fail. These
         # can optionally be provided by test format handlers, and will be
         # honored when the test result is supplied.
@@ -157,6 +158,11 @@ class Test:
     def getFullName(self):
         return self.suite.config.name + ' :: ' + '/'.join(self.path_in_suite)
 
+    def getFilePath(self):
+        if self.file_path:
+            return self.file_path
+        return self.getSourcePath()
+
     def getSourcePath(self):
         return self.suite.getSourcePath(self.path_in_suite)
 
diff --git a/utils/lit/lit/formats/googletest.py b/utils/lit/lit/formats/googletest.py
index b77e184..3d14b729 100644
--- a/utils/lit/lit/formats/googletest.py
+++ b/utils/lit/lit/formats/googletest.py
@@ -66,7 +66,7 @@ class GoogleTest(TestFormat):
         # Discover the tests in this executable.
         for testname in self.getGTestTests(execpath, litConfig, localConfig):
             testPath = path_in_suite + (basename, testname)
-            yield lit.Test.Test(testSuite, testPath, localConfig)
+            yield lit.Test.Test(testSuite, testPath, localConfig, file_path=execpath)
 
     def getTestsInDirectory(self, testSuite, path_in_suite,
                             litConfig, localConfig):
diff --git a/utils/lit/lit/main.py b/utils/lit/lit/main.py
index 6f672a0..3554ad3 100755
--- a/utils/lit/lit/main.py
+++ b/utils/lit/lit/main.py
@@ -34,6 +34,10 @@ class TestingProgressDisplay(object):
 
     def update(self, test):
         self.completed += 1
+
+        if self.opts.incremental:
+            update_incremental_cache(test)
+
         if self.progressBar:
             self.progressBar.update(float(self.completed)/self.numTests,
                                     test.getFullName())
@@ -108,11 +112,32 @@ def write_test_results(run, lit_config, testing_time, output_path):
     finally:
         f.close()
 
+def update_incremental_cache(test):
+    if not test.result.code.isFailure:
+      return
+    fname = test.getFilePath()
+    os.utime(fname, None)
+
+def sort_by_incremental_cache(run, litConfig):
+  def sortIndex(test):
+    fname = test.getFilePath()
+    try:
+      index = -os.path.getmtime(fname)
+    except os.error as e:
+      if litConfig.debug:
+        litConfig.note(e)
+      index = 0
+    return index
+  run.tests.sort(key = lambda t: sortIndex(t) )
+
 def main(builtinParameters = {}):
     # Use processes by default on Unix platforms.
     isWindows = platform.system() == 'Windows'
     useProcessesIsDefault = not isWindows
 
+    # Enable incremental testing by default if possible.
+    incrementalIsDefault = hasattr(os, 'utime')
+
     global options
     from optparse import OptionParser, OptionGroup
     parser = OptionParser("usage: %prog [options] {file-or-path}")
@@ -179,6 +204,9 @@ def main(builtinParameters = {}):
     group.add_option("", "--shuffle", dest="shuffle",
                      help="Run tests in random order",
                      action="store_true", default=False)
+    group.add_option("", "--incremental", dest="incremental",
+                     help="Run failing and recently modified tests first",
+                     action="store_true", default=incrementalIsDefault)
     group.add_option("", "--filter", dest="filter", metavar="REGEX",
                      help=("Only run tests with paths matching the given "
                            "regular expression"),
@@ -292,6 +320,8 @@ def main(builtinParameters = {}):
     # Then select the order.
     if opts.shuffle:
         random.shuffle(run.tests)
+    elif opts.incremental:
+        sort_by_incremental_cache(run, litConfig)
     else:
         run.tests.sort(key = lambda t: t.getFullName())