[PATCH] D30380: Teach lit to expand glob expressions

[Zachary Turner via Phabricator via llvm-commits]

zturner created this revision.
Herald added a reviewer: modocache.

This is a proposed solution to https://reviews.llvm.org/D30371.  libfuzzer and compiler-rt in general often need to pass lists of files as inputs to the tools, frequently involving wildcard expressions, and this is not easy to do portably given the difference in environment between Windows and non-Windows platforms.  Usually it involves some hacky call to `xargs`, but then when you mix `find` into the equation you run into trouble with Windows tools.

This patch adds a new lit substitution, `%[[expr]]` which will be treated as a glob expression and replaced in the input.  Note that all glob expressions are expanded **before** doing any other substitutions, and because of this you can use traditional substitutions as part of a glob expression, which allows you to root your paths `%S`, `%T`, etc.

A simple test is added to demonstrate the use of the syntax.


https://reviews.llvm.org/D30380

Files:
  llvm/docs/CommandGuide/lit.rst
  llvm/test/Other/Inputs/glob-input
  llvm/test/Other/lit-globbing.ll
  llvm/utils/lit/lit/TestRunner.py


Index: llvm/utils/lit/lit/TestRunner.py
===================================================================
--- llvm/utils/lit/lit/TestRunner.py
+++ llvm/utils/lit/lit/TestRunner.py
@@ -1,4 +1,5 @@
 from __future__ import absolute_import
+import glob
 import os, signal, subprocess, sys
 import re
 import platform
@@ -723,16 +724,31 @@
                 ])
     return substitutions
 
-def applySubstitutions(script, substitutions):
+def applySubstitutions(script, substitutions, tmpDir):
     """Apply substitutions to the script.  Allow full regular expression syntax.
     Replace each matching occurrence of regular expression pattern a with
     substitution b in line ln."""
-    def processLine(ln):
+    def doDirectSubstitutions(s):
         # Apply substitutions
         for a,b in substitutions:
             if kIsWindows:
                 b = b.replace("\\","\\\\")
-            ln = re.sub(a, b, ln)
+            s = re.sub(a, b, s)
+        return s
+
+    def processLine(ln):
+        while True:
+            # First expand any glob expressions.
+            m = re.search(r"%\[\[(.*?)]]", ln)
+            if m is None:
+                break
+            glob_pattern = doDirectSubstitutions(m.groups()[0])
+            if kIsWindows:
+                glob_pattern = glob_pattern.replace("\\", "\\\\")
+            result = [os.path.normpath(x) for x in glob.glob(glob_pattern)]
+            expanded = " ".join(result)
+            ln = ln.replace(m.group(), expanded)
+        ln = doDirectSubstitutions(ln)
 
         # Strip the trailing newline and any extra whitespace.
         return ln.strip()
@@ -1038,7 +1054,7 @@
     substitutions = list(extra_substitutions)
     substitutions += getDefaultSubstitutions(test, tmpDir, tmpBase,
                                              normalize_slashes=useExternalSh)
-    script = applySubstitutions(script, substitutions)
+    script = applySubstitutions(script, substitutions, tmpDir)
 
     # Re-run failed tests up to test_retry_attempts times.
     attempts = 1
Index: llvm/test/Other/lit-globbing.ll
===================================================================
--- /dev/null
+++ llvm/test/Other/lit-globbing.ll
@@ -0,0 +1,13 @@
+RUN: echo No%[[Foo*]]Match | FileCheck -check-prefix=NOMATCH %s
+RUN: echo Test1 > %T/Test1.txt
+RUN: echo Test2 > %T/Test2.txt
+RUN: echo Test12 > %T/Test12.txt
+RUN: echo Left%[[%T/Test?.txt]]Right | FileCheck -check-prefix=MATCH %s
+RUN: echo Left%[[%S\Inputs\glob-input]]Right | FileCheck -check-prefix=MATCH2 %s
+
+NOMATCH: NoMatch
+
+MATCH-NOT: Test12.txt
+MATCH: Left{{.*}}Test1.txt{{.*}}Test2.txtRight
+
+MATCH2: Left{{.*}}glob-inputRight
\ No newline at end of file
Index: llvm/docs/CommandGuide/lit.rst
===================================================================
--- llvm/docs/CommandGuide/lit.rst
+++ llvm/docs/CommandGuide/lit.rst
@@ -388,6 +388,7 @@
  %S         source dir (directory of the file currently being run)
  %p         same as %S
  %{pathsep} path separator
+ %[[glob]]  A list of all files matching the glob expression, separated by spaces.
  %t         temporary file name unique to the test
  %T         temporary directory unique to the test
  %%         %


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D30380.89799.patch
Type: text/x-patch
Size: 3222 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20170225/8539ed3b/attachment.bin>