[llvm-commits] [llvm] r81190 - in /llvm/trunk: docs/CommandGuide/lit.pod utils/lit/ utils/lit/LitConfig.py utils/lit/LitFormats.py utils/lit/ProgressBar.py utils/lit/ShCommands.py utils/lit/ShTest.py utils/lit/ShUtil.py utils/lit/TODO utils/lit/TclTest.py utils/lit/TclUtil.py utils/lit/Test.py utils/lit/TestRunner.py utils/lit/TestingConfig.py utils/lit/Util.py utils/lit/lit.py
Daniel Dunbar
daniel at zuster.org
Mon Sep 7 22:31:18 PDT 2009
Author: ddunbar
Date: Tue Sep 8 00:31:18 2009
New Revision: 81190
URL: http://llvm.org/viewvc/llvm-project?rev=81190&view=rev
Log:
Add 'lit' testing tool.
- make install && man $(llvm-config --prefix)/share/man/man1/lit.1 for more
information.
Added:
llvm/trunk/docs/CommandGuide/lit.pod
llvm/trunk/utils/lit/
llvm/trunk/utils/lit/LitConfig.py
llvm/trunk/utils/lit/LitFormats.py
llvm/trunk/utils/lit/ProgressBar.py
llvm/trunk/utils/lit/ShCommands.py
llvm/trunk/utils/lit/ShTest.py
llvm/trunk/utils/lit/ShUtil.py
llvm/trunk/utils/lit/TODO
llvm/trunk/utils/lit/TclTest.py
llvm/trunk/utils/lit/TclUtil.py
llvm/trunk/utils/lit/Test.py
llvm/trunk/utils/lit/TestRunner.py
llvm/trunk/utils/lit/TestingConfig.py
llvm/trunk/utils/lit/Util.py
llvm/trunk/utils/lit/lit.py (with props)
Added: llvm/trunk/docs/CommandGuide/lit.pod
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/CommandGuide/lit.pod?rev=81190&view=auto
==============================================================================
--- llvm/trunk/docs/CommandGuide/lit.pod (added)
+++ llvm/trunk/docs/CommandGuide/lit.pod Tue Sep 8 00:31:18 2009
@@ -0,0 +1,222 @@
+=pod
+
+=head1 NAME
+
+lit - LLVM Integrated Tester
+
+=head1 SYNOPSIS
+
+B<lit> [I<options>] [I<tests>]
+
+=head1 DESCRIPTION
+
+B<lit> is a portable tool for executing LLVM and Clang style test suites,
+summarizing their results, and providing indication of failures. B<lit> is
+designed to be a lightweight testing tool with as simple a user interface as
+possible.
+
+B<lit> should be run with one or more I<tests> to run specified on the command
+line. Tests can be either individual test files or directories to search for
+tests (see L<"TEST DISCOVERY">).
+
+Each specified test will be executed (potentially in parallel) and once all
+tests have been run B<lit> will print summary information on the number of tests
+which passed or failed (see L<"TEST STATUS RESULTS">). The B<lit> program will
+execute with a non-zero exit code if any tests fail.
+
+By default B<lit> will use a succinct progress display and will only print
+summary information for test failures. See L<"OUTPUT OPTIONS"> for options
+controlling the B<lit> progress display and output.
+
+B<lit> also includes a number of options for controlling how tests are exected
+(specific features may depend on the particular test format). See L<"EXECUTION
+OPTIONS"> for more information.
+
+Finally, B<lit> also supports additional options for only running a subset of
+the options specified on the command line, see L<"SELECTION OPTIONS"> for
+more information.
+
+=head1 GENERAL OPTIONS
+
+=over
+
+=item B<-h>, B<--help>
+
+Show the B<lit> help message.
+
+=item B<-j> I<N>, B<--threads>=I<N>
+
+Run I<N> tests in parallel. By default, this is automatically chose to match the
+number of detected available CPUs.
+
+=back
+
+=head1 OUTPUT OPTIONS
+
+=over
+
+=item B<-q>, B<--quiet>
+
+Suppress any output except for test failures.
+
+=item B<-s>, B<--succinct>
+
+Show less output, for example don't show information on tests that pass.
+
+=item B<-v>, B<--verbose>
+
+Show more information on test failures, for example the entire test output
+instead of just the test result.
+
+=item B<--no-progress-bar>
+
+Do not use curses based progress bar.
+
+=back
+
+=head1 EXECUTION OPTIONS
+
+=over
+
+=item B<--path>=I<PATH>
+
+Specify an addition I<PATH> to use when searching for executables in tests.
+
+=item B<--vg>
+
+Run individual tests under valgrind (using the memcheck tool). The
+I<--error-exitcode> argument for valgrind is used so that valgrind failures will
+cause the program to exit with a non-zero status.
+
+=item B<--vg-arg>=I<ARG>
+
+When I<--vg> is used, specify an additional argument to pass to valgrind itself.
+
+=item B<--time-tests>
+
+Track the wall time individual tests take to execute and includes the results in
+the summary output. This is useful for determining which tests in a test suite
+take the most time to execute. Note that this option is most useful with I<-j
+1>.
+
+=back
+
+=head1 SELECTION OPTIONS
+
+=over
+
+=item B<--max-tests>=I<N>
+
+Run at most I<N> tests and then terminate.
+
+=item B<--max-time>=I<N>
+
+Spend at most I<N> seconds (approximately) running tests and then terminate.
+
+=item B<--shuffle>
+
+Run the tests in a random order.
+
+=back
+
+=head1 ADDITIONAL OPTIONS
+
+=over
+
+=item B<--debug>
+
+Run B<lit> in debug mode, for debugging configuration issues and B<lit> itself.
+
+=item B<--show-suites>
+
+List the discovered test suites as part of the standard output.
+
+=item B<--no-tcl-as-sh>
+
+Run Tcl scripts internally (instead of converting to shell scripts).
+
+=back
+
+=head1 EXIT STATUS
+
+B<lit> will exit with an exit code of 1 if there are any FAIL or XPASS
+results. Otherwise, it will exit with the status 0. Other exit codes used for
+non-test related failures (for example a user error or an internal program
+error).
+
+=head1 TEST DISCOVERY
+
+The inputs passed to B<lit> can be either individual tests, or entire
+directories or hierarchies of tests to run. When B<lit> starts up, the first
+thing it does is convert the inputs into a complete list of tests to run as part
+of I<test discovery>.
+
+In the B<lit> model, every test must exist inside some I<test suite>. B<lit>
+resolves the inputs specified on the command line to test suites by searching
+upwards from the input path until it finds a I<lit.cfg> or I<lit.site.cfg>
+file. These files serve as both a marker of test suites and as configuration
+files which B<lit> loads in order to understand how to find and run the tests
+inside the test suite.
+
+Once B<lit> has mapped the inputs into test suites it traverses the list of
+inputs adding tests for individual files and recursively searching for tests in
+directories.
+
+This behavior makes it easy to specify a subset of tests to run, while still
+allowing the test suite configuration to control exactly how tests are
+interpreted. In addition, B<lit> always identifies tests by the test suite they
+are in, and their relative path inside the test suite. For appropriately
+configured projects, this allows B<lit> to provide convenient and flexible
+support for out-of-tree builds.
+
+=head1 TEST STATUS RESULTS
+
+Each test ultimately produces one of the following six results:
+
+=over
+
+=item B<PASS>
+
+The test succeeded.
+
+=item B<XFAIL>
+
+The test failed, but that is expected. This is used for test formats which allow
+specifying that a test does not currently work, but wish to leave it in the test
+suite.
+
+=item B<XPASS>
+
+The test succeeded, but it was expected to fail. This is used for tests which
+were specified as expected to fail, but are now succeeding (generally because
+the feautre they test was broken and has been fixed).
+
+=item B<FAIL>
+
+The test failed.
+
+=item B<UNRESOLVED>
+
+The test result could not be determined. For example, this occurs when the test
+could not be run, the test itself is invalid, or the test was interrupted.
+
+=item B<UNSUPPORTED>
+
+The test is not supported in this environment. This is used by test formats
+which can report unsupported tests.
+
+=back
+
+Depending on the test format tests may produce additional information about
+their status (generally only for failures). See the L<Output|"LIT OUTPUT">
+section for more information.
+
+=head1 SEE ALSO
+
+L<valgrind(1)>
+
+=head1 AUTHOR
+
+Written by Daniel Dunbar and maintained by the LLVM Team (L<http://llvm.org>).
+
+=cut
Added: llvm/trunk/utils/lit/LitConfig.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/LitConfig.py?rev=81190&view=auto
==============================================================================
--- llvm/trunk/utils/lit/LitConfig.py (added)
+++ llvm/trunk/utils/lit/LitConfig.py Tue Sep 8 00:31:18 2009
@@ -0,0 +1,71 @@
+class LitConfig:
+ """LitConfig - Configuration data for a 'lit' test runner instance, shared
+ across all tests.
+
+ The LitConfig object is also used to communicate with client configuration
+ files, it is always passed in as the global variable 'lit' so that
+ configuration files can access common functionality and internal components
+ easily.
+ """
+
+ # Provide access to built-in formats.
+ import LitFormats as formats
+
+ # Provide access to built-in utility functions.
+ import Util as util
+
+ def __init__(self, progname, path, quiet,
+ useValgrind, valgrindArgs,
+ useTclAsSh,
+ noExecute, debug, isWindows):
+ # The name of the test runner.
+ self.progname = progname
+ # The items to add to the PATH environment variable.
+ self.path = list(map(str, path))
+ self.quiet = bool(quiet)
+ self.useValgrind = bool(useValgrind)
+ self.valgrindArgs = list(valgrindArgs)
+ self.useTclAsSh = bool(useTclAsSh)
+ self.noExecute = noExecute
+ self.debug = debug
+ self.isWindows = bool(isWindows)
+
+ self.numErrors = 0
+ self.numWarnings = 0
+
+ def load_config(self, config, path):
+ """load_config(config, path) - Load a config object from an alternate
+ path."""
+ from TestingConfig import TestingConfig
+ return TestingConfig.frompath(path, config.parent, self,
+ mustExist = True,
+ config = config)
+
+ def _write_message(self, kind, message):
+ import inspect, os, sys
+
+ # Get the file/line where this message was generated.
+ f = inspect.currentframe()
+ # Step out of _write_message, and then out of wrapper.
+ f = f.f_back.f_back
+ file,line,_,_,_ = inspect.getframeinfo(f)
+ location = '%s:%d' % (os.path.basename(file), line)
+
+ print >>sys.stderr, '%s: %s: %s: %s' % (self.progname, location,
+ kind, message)
+
+ def note(self, message):
+ self._write_message('note', message)
+
+ def warning(self, message):
+ self._write_message('warning', message)
+ self.numWarnings += 1
+
+ def error(self, message):
+ self._write_message('error', message)
+ self.numErrors += 1
+
+ def fatal(self, message):
+ import sys
+ self._write_message('fatal', message)
+ sys.exit(2)
Added: llvm/trunk/utils/lit/LitFormats.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/LitFormats.py?rev=81190&view=auto
==============================================================================
--- llvm/trunk/utils/lit/LitFormats.py (added)
+++ llvm/trunk/utils/lit/LitFormats.py Tue Sep 8 00:31:18 2009
@@ -0,0 +1,2 @@
+from ShTest import ShTest
+from TclTest import TclTest
Added: llvm/trunk/utils/lit/ProgressBar.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/ProgressBar.py?rev=81190&view=auto
==============================================================================
--- llvm/trunk/utils/lit/ProgressBar.py (added)
+++ llvm/trunk/utils/lit/ProgressBar.py Tue Sep 8 00:31:18 2009
@@ -0,0 +1,267 @@
+#!/usr/bin/env python
+
+# Source: http://code.activestate.com/recipes/475116/, with
+# modifications by Daniel Dunbar.
+
+import sys, re, time
+
+class TerminalController:
+ """
+ A class that can be used to portably generate formatted output to
+ a terminal.
+
+ `TerminalController` defines a set of instance variables whose
+ values are initialized to the control sequence necessary to
+ perform a given action. These can be simply included in normal
+ output to the terminal:
+
+ >>> term = TerminalController()
+ >>> print 'This is '+term.GREEN+'green'+term.NORMAL
+
+ Alternatively, the `render()` method can used, which replaces
+ '${action}' with the string required to perform 'action':
+
+ >>> term = TerminalController()
+ >>> print term.render('This is ${GREEN}green${NORMAL}')
+
+ If the terminal doesn't support a given action, then the value of
+ the corresponding instance variable will be set to ''. As a
+ result, the above code will still work on terminals that do not
+ support color, except that their output will not be colored.
+ Also, this means that you can test whether the terminal supports a
+ given action by simply testing the truth value of the
+ corresponding instance variable:
+
+ >>> term = TerminalController()
+ >>> if term.CLEAR_SCREEN:
+ ... print 'This terminal supports clearning the screen.'
+
+ Finally, if the width and height of the terminal are known, then
+ they will be stored in the `COLS` and `LINES` attributes.
+ """
+ # Cursor movement:
+ BOL = '' #: Move the cursor to the beginning of the line
+ UP = '' #: Move the cursor up one line
+ DOWN = '' #: Move the cursor down one line
+ LEFT = '' #: Move the cursor left one char
+ RIGHT = '' #: Move the cursor right one char
+
+ # Deletion:
+ CLEAR_SCREEN = '' #: Clear the screen and move to home position
+ CLEAR_EOL = '' #: Clear to the end of the line.
+ CLEAR_BOL = '' #: Clear to the beginning of the line.
+ CLEAR_EOS = '' #: Clear to the end of the screen
+
+ # Output modes:
+ BOLD = '' #: Turn on bold mode
+ BLINK = '' #: Turn on blink mode
+ DIM = '' #: Turn on half-bright mode
+ REVERSE = '' #: Turn on reverse-video mode
+ NORMAL = '' #: Turn off all modes
+
+ # Cursor display:
+ HIDE_CURSOR = '' #: Make the cursor invisible
+ SHOW_CURSOR = '' #: Make the cursor visible
+
+ # Terminal size:
+ COLS = None #: Width of the terminal (None for unknown)
+ LINES = None #: Height of the terminal (None for unknown)
+
+ # Foreground colors:
+ BLACK = BLUE = GREEN = CYAN = RED = MAGENTA = YELLOW = WHITE = ''
+
+ # Background colors:
+ BG_BLACK = BG_BLUE = BG_GREEN = BG_CYAN = ''
+ BG_RED = BG_MAGENTA = BG_YELLOW = BG_WHITE = ''
+
+ _STRING_CAPABILITIES = """
+ BOL=cr UP=cuu1 DOWN=cud1 LEFT=cub1 RIGHT=cuf1
+ CLEAR_SCREEN=clear CLEAR_EOL=el CLEAR_BOL=el1 CLEAR_EOS=ed BOLD=bold
+ BLINK=blink DIM=dim REVERSE=rev UNDERLINE=smul NORMAL=sgr0
+ HIDE_CURSOR=cinvis SHOW_CURSOR=cnorm""".split()
+ _COLORS = """BLACK BLUE GREEN CYAN RED MAGENTA YELLOW WHITE""".split()
+ _ANSICOLORS = "BLACK RED GREEN YELLOW BLUE MAGENTA CYAN WHITE".split()
+
+ def __init__(self, term_stream=sys.stdout):
+ """
+ Create a `TerminalController` and initialize its attributes
+ with appropriate values for the current terminal.
+ `term_stream` is the stream that will be used for terminal
+ output; if this stream is not a tty, then the terminal is
+ assumed to be a dumb terminal (i.e., have no capabilities).
+ """
+ # Curses isn't available on all platforms
+ try: import curses
+ except: return
+
+ # If the stream isn't a tty, then assume it has no capabilities.
+ if not term_stream.isatty(): return
+
+ # Check the terminal type. If we fail, then assume that the
+ # terminal has no capabilities.
+ try: curses.setupterm()
+ except: return
+
+ # Look up numeric capabilities.
+ self.COLS = curses.tigetnum('cols')
+ self.LINES = curses.tigetnum('lines')
+
+ # Look up string capabilities.
+ for capability in self._STRING_CAPABILITIES:
+ (attrib, cap_name) = capability.split('=')
+ setattr(self, attrib, self._tigetstr(cap_name) or '')
+
+ # Colors
+ set_fg = self._tigetstr('setf')
+ if set_fg:
+ for i,color in zip(range(len(self._COLORS)), self._COLORS):
+ setattr(self, color, curses.tparm(set_fg, i) or '')
+ set_fg_ansi = self._tigetstr('setaf')
+ if set_fg_ansi:
+ for i,color in zip(range(len(self._ANSICOLORS)), self._ANSICOLORS):
+ setattr(self, color, curses.tparm(set_fg_ansi, i) or '')
+ set_bg = self._tigetstr('setb')
+ if set_bg:
+ for i,color in zip(range(len(self._COLORS)), self._COLORS):
+ setattr(self, 'BG_'+color, curses.tparm(set_bg, i) or '')
+ set_bg_ansi = self._tigetstr('setab')
+ if set_bg_ansi:
+ for i,color in zip(range(len(self._ANSICOLORS)), self._ANSICOLORS):
+ setattr(self, 'BG_'+color, curses.tparm(set_bg_ansi, i) or '')
+
+ def _tigetstr(self, cap_name):
+ # String capabilities can include "delays" of the form "$<2>".
+ # For any modern terminal, we should be able to just ignore
+ # these, so strip them out.
+ import curses
+ cap = curses.tigetstr(cap_name) or ''
+ return re.sub(r'\$<\d+>[/*]?', '', cap)
+
+ def render(self, template):
+ """
+ Replace each $-substitutions in the given template string with
+ the corresponding terminal control string (if it's defined) or
+ '' (if it's not).
+ """
+ return re.sub(r'\$\$|\${\w+}', self._render_sub, template)
+
+ def _render_sub(self, match):
+ s = match.group()
+ if s == '$$': return s
+ else: return getattr(self, s[2:-1])
+
+#######################################################################
+# Example use case: progress bar
+#######################################################################
+
+class SimpleProgressBar:
+ """
+ A simple progress bar which doesn't need any terminal support.
+
+ This prints out a progress bar like:
+ 'Header: 0 .. 10.. 20.. ...'
+ """
+
+ def __init__(self, header):
+ self.header = header
+ self.atIndex = None
+
+ def update(self, percent, message):
+ if self.atIndex is None:
+ sys.stdout.write(self.header)
+ self.atIndex = 0
+
+ next = int(percent*50)
+ if next == self.atIndex:
+ return
+
+ for i in range(self.atIndex, next):
+ idx = i % 5
+ if idx == 0:
+ sys.stdout.write('%-2d' % (i*2))
+ elif idx == 1:
+ pass # Skip second char
+ elif idx < 4:
+ sys.stdout.write('.')
+ else:
+ sys.stdout.write(' ')
+ sys.stdout.flush()
+ self.atIndex = next
+
+ def clear(self):
+ if self.atIndex is not None:
+ sys.stdout.write('\n')
+ sys.stdout.flush()
+ self.atIndex = None
+
+class ProgressBar:
+ """
+ A 3-line progress bar, which looks like::
+
+ Header
+ 20% [===========----------------------------------]
+ progress message
+
+ The progress bar is colored, if the terminal supports color
+ output; and adjusts to the width of the terminal.
+ """
+ BAR = '%s${GREEN}[${BOLD}%s%s${NORMAL}${GREEN}]${NORMAL}%s\n'
+ HEADER = '${BOLD}${CYAN}%s${NORMAL}\n\n'
+
+ def __init__(self, term, header, useETA=True):
+ self.term = term
+ if not (self.term.CLEAR_EOL and self.term.UP and self.term.BOL):
+ raise ValueError("Terminal isn't capable enough -- you "
+ "should use a simpler progress dispaly.")
+ self.width = self.term.COLS or 75
+ self.bar = term.render(self.BAR)
+ self.header = self.term.render(self.HEADER % header.center(self.width))
+ self.cleared = 1 #: true if we haven't drawn the bar yet.
+ self.useETA = useETA
+ if self.useETA:
+ self.startTime = time.time()
+ self.update(0, '')
+
+ def update(self, percent, message):
+ if self.cleared:
+ sys.stdout.write(self.header)
+ self.cleared = 0
+ prefix = '%3d%% ' % (percent*100,)
+ suffix = ''
+ if self.useETA:
+ elapsed = time.time() - self.startTime
+ if percent > .0001 and elapsed > 1:
+ total = elapsed / percent
+ eta = int(total - elapsed)
+ h = eta//3600.
+ m = (eta//60) % 60
+ s = eta % 60
+ suffix = ' ETA: %02d:%02d:%02d'%(h,m,s)
+ barWidth = self.width - len(prefix) - len(suffix) - 2
+ n = int(barWidth*percent)
+ if len(message) < self.width:
+ message = message + ' '*(self.width - len(message))
+ else:
+ message = '... ' + message[-(self.width-4):]
+ sys.stdout.write(
+ self.term.BOL + self.term.UP + self.term.CLEAR_EOL +
+ (self.bar % (prefix, '='*n, '-'*(barWidth-n), suffix)) +
+ self.term.CLEAR_EOL + message)
+
+ def clear(self):
+ if not self.cleared:
+ sys.stdout.write(self.term.BOL + self.term.CLEAR_EOL +
+ self.term.UP + self.term.CLEAR_EOL +
+ self.term.UP + self.term.CLEAR_EOL)
+ self.cleared = 1
+
+def test():
+ import time
+ tc = TerminalController()
+ p = ProgressBar(tc, 'Tests')
+ for i in range(101):
+ p.update(i/100., str(i))
+ time.sleep(.3)
+
+if __name__=='__main__':
+ test()
Added: llvm/trunk/utils/lit/ShCommands.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/ShCommands.py?rev=81190&view=auto
==============================================================================
--- llvm/trunk/utils/lit/ShCommands.py (added)
+++ llvm/trunk/utils/lit/ShCommands.py Tue Sep 8 00:31:18 2009
@@ -0,0 +1,86 @@
+import ShUtil
+
+class Command:
+ def __init__(self, args, redirects):
+ self.args = list(args)
+ self.redirects = list(redirects)
+
+ def __repr__(self):
+ return 'Command(%r, %r)' % (self.args, self.redirects)
+
+ def __cmp__(self, other):
+ if not isinstance(other, Command):
+ return -1
+
+ return cmp((self.args, self.redirects),
+ (other.args, other.redirects))
+
+ def toShell(self, file):
+ for arg in self.args:
+ if "'" not in arg:
+ quoted = "'%s'" % arg
+ elif '"' not in arg and '$' not in arg:
+ quoted = '"%s"' % arg
+ else:
+ raise NotImplementedError,'Unable to quote %r' % arg
+ print >>file, quoted,
+
+ # For debugging / validation.
+ dequoted = list(ShUtil.ShLexer(quoted).lex())
+ if dequoted != [arg]:
+ raise NotImplementedError,'Unable to quote %r' % arg
+
+ for r in self.redirects:
+ if len(r[0]) == 1:
+ print >>file, "%s '%s'" % (r[0][0], r[1]),
+ else:
+ print >>file, "%s%s '%s'" % (r[0][1], r[0][0], r[1]),
+
+class Pipeline:
+ def __init__(self, commands, negate=False, pipe_err=False):
+ self.commands = commands
+ self.negate = negate
+ self.pipe_err = pipe_err
+
+ def __repr__(self):
+ return 'Pipeline(%r, %r, %r)' % (self.commands, self.negate,
+ self.pipe_err)
+
+ def __cmp__(self, other):
+ if not isinstance(other, Pipeline):
+ return -1
+
+ return cmp((self.commands, self.negate, self.pipe_err),
+ (other.commands, other.negate, self.pipe_err))
+
+ def toShell(self, file, pipefail=False):
+ if pipefail != self.pipe_err:
+ raise ValueError,'Inconsistent "pipefail" attribute!'
+ if self.negate:
+ print >>file, '!',
+ for cmd in self.commands:
+ cmd.toShell(file)
+ if cmd is not self.commands[-1]:
+ print >>file, '|\n ',
+
+class Seq:
+ def __init__(self, lhs, op, rhs):
+ assert op in (';', '&', '||', '&&')
+ self.op = op
+ self.lhs = lhs
+ self.rhs = rhs
+
+ def __repr__(self):
+ return 'Seq(%r, %r, %r)' % (self.lhs, self.op, self.rhs)
+
+ def __cmp__(self, other):
+ if not isinstance(other, Seq):
+ return -1
+
+ return cmp((self.lhs, self.op, self.rhs),
+ (other.lhs, other.op, other.rhs))
+
+ def toShell(self, file, pipefail=False):
+ self.lhs.toShell(file, pipefail)
+ print >>file, ' %s\n' % self.op
+ self.rhs.toShell(file, pipefail)
Added: llvm/trunk/utils/lit/ShTest.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/ShTest.py?rev=81190&view=auto
==============================================================================
--- llvm/trunk/utils/lit/ShTest.py (added)
+++ llvm/trunk/utils/lit/ShTest.py Tue Sep 8 00:31:18 2009
@@ -0,0 +1,12 @@
+import TestRunner
+
+class ShTest:
+ def __init__(self, execute_external = False, require_and_and = False):
+ self.execute_external = execute_external
+ self.require_and_and = require_and_and
+
+ def execute(self, test, litConfig):
+ return TestRunner.executeShTest(test, litConfig,
+ self.execute_external,
+ self.require_and_and)
+
Added: llvm/trunk/utils/lit/ShUtil.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/ShUtil.py?rev=81190&view=auto
==============================================================================
--- llvm/trunk/utils/lit/ShUtil.py (added)
+++ llvm/trunk/utils/lit/ShUtil.py Tue Sep 8 00:31:18 2009
@@ -0,0 +1,346 @@
+import itertools
+
+import Util
+from ShCommands import Command, Pipeline, Seq
+
+class ShLexer:
+ def __init__(self, data, win32Escapes = False):
+ self.data = data
+ self.pos = 0
+ self.end = len(data)
+ self.win32Escapes = win32Escapes
+
+ def eat(self):
+ c = self.data[self.pos]
+ self.pos += 1
+ return c
+
+ def look(self):
+ return self.data[self.pos]
+
+ def maybe_eat(self, c):
+ """
+ maybe_eat(c) - Consume the character c if it is the next character,
+ returning True if a character was consumed. """
+ if self.data[self.pos] == c:
+ self.pos += 1
+ return True
+ return False
+
+ def lex_arg_fast(self, c):
+ # Get the leading whitespace free section.
+ chunk = self.data[self.pos - 1:].split(None, 1)[0]
+
+ # If it has special characters, the fast path failed.
+ if ('|' in chunk or '&' in chunk or
+ '<' in chunk or '>' in chunk or
+ "'" in chunk or '"' in chunk or
+ '\\' in chunk):
+ return None
+
+ self.pos = self.pos - 1 + len(chunk)
+ return chunk
+
+ def lex_arg_slow(self, c):
+ if c in "'\"":
+ str = self.lex_arg_quoted(c)
+ else:
+ str = c
+ while self.pos != self.end:
+ c = self.look()
+ if c.isspace() or c in "|&":
+ break
+ elif c in '><':
+ # This is an annoying case; we treat '2>' as a single token so
+ # we don't have to track whitespace tokens.
+
+ # If the parse string isn't an integer, do the usual thing.
+ if not str.isdigit():
+ break
+
+ # Otherwise, lex the operator and convert to a redirection
+ # token.
+ num = int(str)
+ tok = self.lex_one_token()
+ assert isinstance(tok, tuple) and len(tok) == 1
+ return (tok[0], num)
+ elif c == '"':
+ self.eat()
+ str += self.lex_arg_quoted('"')
+ elif not self.win32Escapes and c == '\\':
+ # Outside of a string, '\\' escapes everything.
+ self.eat()
+ if self.pos == self.end:
+ Util.warning("escape at end of quoted argument in: %r" %
+ self.data)
+ return str
+ str += self.eat()
+ else:
+ str += self.eat()
+ return str
+
+ def lex_arg_quoted(self, delim):
+ str = ''
+ while self.pos != self.end:
+ c = self.eat()
+ if c == delim:
+ return str
+ elif c == '\\' and delim == '"':
+ # Inside a '"' quoted string, '\\' only escapes the quote
+ # character and backslash, otherwise it is preserved.
+ if self.pos == self.end:
+ Util.warning("escape at end of quoted argument in: %r" %
+ self.data)
+ return str
+ c = self.eat()
+ if c == '"': #
+ str += '"'
+ elif c == '\\':
+ str += '\\'
+ else:
+ str += '\\' + c
+ else:
+ str += c
+ Util.warning("missing quote character in %r" % self.data)
+ return str
+
+ def lex_arg_checked(self, c):
+ pos = self.pos
+ res = self.lex_arg_fast(c)
+ end = self.pos
+
+ self.pos = pos
+ reference = self.lex_arg_slow(c)
+ if res is not None:
+ if res != reference:
+ raise ValueError,"Fast path failure: %r != %r" % (res, reference)
+ if self.pos != end:
+ raise ValueError,"Fast path failure: %r != %r" % (self.pos, end)
+ return reference
+
+ def lex_arg(self, c):
+ return self.lex_arg_fast(c) or self.lex_arg_slow(c)
+
+ def lex_one_token(self):
+ """
+ lex_one_token - Lex a single 'sh' token. """
+
+ c = self.eat()
+ if c in ';!':
+ return (c,)
+ if c == '|':
+ if self.maybe_eat('|'):
+ return ('||',)
+ return (c,)
+ if c == '&':
+ if self.maybe_eat('&'):
+ return ('&&',)
+ if self.maybe_eat('>'):
+ return ('&>',)
+ return (c,)
+ if c == '>':
+ if self.maybe_eat('&'):
+ return ('>&',)
+ if self.maybe_eat('>'):
+ return ('>>',)
+ return (c,)
+ if c == '<':
+ if self.maybe_eat('&'):
+ return ('<&',)
+ if self.maybe_eat('>'):
+ return ('<<',)
+ return (c,)
+
+ return self.lex_arg(c)
+
+ def lex(self):
+ while self.pos != self.end:
+ if self.look().isspace():
+ self.eat()
+ else:
+ yield self.lex_one_token()
+
+###
+
+class ShParser:
+ def __init__(self, data, win32Escapes = False):
+ self.data = data
+ self.tokens = ShLexer(data, win32Escapes = win32Escapes).lex()
+
+ def lex(self):
+ try:
+ return self.tokens.next()
+ except StopIteration:
+ return None
+
+ def look(self):
+ next = self.lex()
+ if next is not None:
+ self.tokens = itertools.chain([next], self.tokens)
+ return next
+
+ def parse_command(self):
+ tok = self.lex()
+ if not tok:
+ raise ValueError,"empty command!"
+ if isinstance(tok, tuple):
+ raise ValueError,"syntax error near unexpected token %r" % tok[0]
+
+ args = [tok]
+ redirects = []
+ while 1:
+ tok = self.look()
+
+ # EOF?
+ if tok is None:
+ break
+
+ # If this is an argument, just add it to the current command.
+ if isinstance(tok, str):
+ args.append(self.lex())
+ continue
+
+ # Otherwise see if it is a terminator.
+ assert isinstance(tok, tuple)
+ if tok[0] in ('|',';','&','||','&&'):
+ break
+
+ # Otherwise it must be a redirection.
+ op = self.lex()
+ arg = self.lex()
+ if not arg:
+ raise ValueError,"syntax error near token %r" % op[0]
+ redirects.append((op, arg))
+
+ return Command(args, redirects)
+
+ def parse_pipeline(self):
+ negate = False
+ if self.look() == ('!',):
+ self.lex()
+ negate = True
+
+ commands = [self.parse_command()]
+ while self.look() == ('|',):
+ self.lex()
+ commands.append(self.parse_command())
+ return Pipeline(commands, negate)
+
+ def parse(self):
+ lhs = self.parse_pipeline()
+
+ while self.look():
+ operator = self.lex()
+ assert isinstance(operator, tuple) and len(operator) == 1
+
+ if not self.look():
+ raise ValueError, "missing argument to operator %r" % operator[0]
+
+ # FIXME: Operator precedence!!
+ lhs = Seq(lhs, operator[0], self.parse_pipeline())
+
+ return lhs
+
+###
+
+import unittest
+
+class TestShLexer(unittest.TestCase):
+ def lex(self, str, *args, **kwargs):
+ return list(ShLexer(str, *args, **kwargs).lex())
+
+ def test_basic(self):
+ self.assertEqual(self.lex('a|b>c&d<e'),
+ ['a', ('|',), 'b', ('>',), 'c', ('&',), 'd',
+ ('<',), 'e'])
+
+ def test_redirection_tokens(self):
+ self.assertEqual(self.lex('a2>c'),
+ ['a2', ('>',), 'c'])
+ self.assertEqual(self.lex('a 2>c'),
+ ['a', ('>',2), 'c'])
+
+ def test_quoting(self):
+ self.assertEqual(self.lex(""" 'a' """),
+ ['a'])
+ self.assertEqual(self.lex(""" "hello\\"world" """),
+ ['hello"world'])
+ self.assertEqual(self.lex(""" "hello\\'world" """),
+ ["hello\\'world"])
+ self.assertEqual(self.lex(""" "hello\\\\world" """),
+ ["hello\\world"])
+ self.assertEqual(self.lex(""" he"llo wo"rld """),
+ ["hello world"])
+ self.assertEqual(self.lex(""" a\\ b a\\\\b """),
+ ["a b", "a\\b"])
+ self.assertEqual(self.lex(""" "" "" """),
+ ["", ""])
+ self.assertEqual(self.lex(""" a\\ b """, win32Escapes = True),
+ ['a\\', 'b'])
+
+class TestShParse(unittest.TestCase):
+ def parse(self, str):
+ return ShParser(str).parse()
+
+ def test_basic(self):
+ self.assertEqual(self.parse('echo hello'),
+ Pipeline([Command(['echo', 'hello'], [])], False))
+ self.assertEqual(self.parse('echo ""'),
+ Pipeline([Command(['echo', ''], [])], False))
+
+ def test_redirection(self):
+ self.assertEqual(self.parse('echo hello > c'),
+ Pipeline([Command(['echo', 'hello'],
+ [((('>'),), 'c')])], False))
+ self.assertEqual(self.parse('echo hello > c >> d'),
+ Pipeline([Command(['echo', 'hello'], [(('>',), 'c'),
+ (('>>',), 'd')])], False))
+ self.assertEqual(self.parse('a 2>&1'),
+ Pipeline([Command(['a'], [(('>&',2), '1')])], False))
+
+ def test_pipeline(self):
+ self.assertEqual(self.parse('a | b'),
+ Pipeline([Command(['a'], []),
+ Command(['b'], [])],
+ False))
+
+ self.assertEqual(self.parse('a | b | c'),
+ Pipeline([Command(['a'], []),
+ Command(['b'], []),
+ Command(['c'], [])],
+ False))
+
+ self.assertEqual(self.parse('! a'),
+ Pipeline([Command(['a'], [])],
+ True))
+
+ def test_list(self):
+ self.assertEqual(self.parse('a ; b'),
+ Seq(Pipeline([Command(['a'], [])], False),
+ ';',
+ Pipeline([Command(['b'], [])], False)))
+
+ self.assertEqual(self.parse('a & b'),
+ Seq(Pipeline([Command(['a'], [])], False),
+ '&',
+ Pipeline([Command(['b'], [])], False)))
+
+ self.assertEqual(self.parse('a && b'),
+ Seq(Pipeline([Command(['a'], [])], False),
+ '&&',
+ Pipeline([Command(['b'], [])], False)))
+
+ self.assertEqual(self.parse('a || b'),
+ Seq(Pipeline([Command(['a'], [])], False),
+ '||',
+ Pipeline([Command(['b'], [])], False)))
+
+ self.assertEqual(self.parse('a && b || c'),
+ Seq(Seq(Pipeline([Command(['a'], [])], False),
+ '&&',
+ Pipeline([Command(['b'], [])], False)),
+ '||',
+ Pipeline([Command(['c'], [])], False)))
+
+if __name__ == '__main__':
+ unittest.main()
Added: llvm/trunk/utils/lit/TODO
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/TODO?rev=81190&view=auto
==============================================================================
--- llvm/trunk/utils/lit/TODO (added)
+++ llvm/trunk/utils/lit/TODO Tue Sep 8 00:31:18 2009
@@ -0,0 +1,19 @@
+ - Move temp directory name into local test config.
+
+ - Add --show-unsupported, don't show by default?
+
+ - Finish documentation.
+
+ - Optionally use multiprocessing.
+
+ - Support llvmc and ocaml tests.
+
+ - Support valgrind in all configs, and LLVM style valgrind.
+
+ - Provide test suite config for running unit tests.
+
+ - Support a timeout / ulimit.
+
+ - Support "disabling" tests? The advantage of making this distinct from XFAIL
+ is it makes it more obvious that it is a temporary measure (and lit can put
+ in a separate category).
Added: llvm/trunk/utils/lit/TclTest.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/TclTest.py?rev=81190&view=auto
==============================================================================
--- llvm/trunk/utils/lit/TclTest.py (added)
+++ llvm/trunk/utils/lit/TclTest.py Tue Sep 8 00:31:18 2009
@@ -0,0 +1,7 @@
+import TestRunner
+
+class TclTest:
+ def execute(self, test, litConfig):
+ return TestRunner.executeTclTest(test, litConfig)
+
+
Added: llvm/trunk/utils/lit/TclUtil.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/TclUtil.py?rev=81190&view=auto
==============================================================================
--- llvm/trunk/utils/lit/TclUtil.py (added)
+++ llvm/trunk/utils/lit/TclUtil.py Tue Sep 8 00:31:18 2009
@@ -0,0 +1,322 @@
+import itertools
+
+from ShCommands import Command, Pipeline
+
+def tcl_preprocess(data):
+ # Tcl has a preprocessing step to replace escaped newlines.
+ i = data.find('\\\n')
+ if i == -1:
+ return data
+
+ # Replace '\\\n' and subsequent whitespace by a single space.
+ n = len(data)
+ str = data[:i]
+ i += 2
+ while i < n and data[i] in ' \t':
+ i += 1
+ return str + ' ' + data[i:]
+
+class TclLexer:
+ """TclLexer - Lex a string into "words", following the Tcl syntax."""
+
+ def __init__(self, data):
+ self.data = tcl_preprocess(data)
+ self.pos = 0
+ self.end = len(self.data)
+
+ def at_end(self):
+ return self.pos == self.end
+
+ def eat(self):
+ c = self.data[self.pos]
+ self.pos += 1
+ return c
+
+ def look(self):
+ return self.data[self.pos]
+
+ def maybe_eat(self, c):
+ """
+ maybe_eat(c) - Consume the character c if it is the next character,
+ returning True if a character was consumed. """
+ if self.data[self.pos] == c:
+ self.pos += 1
+ return True
+ return False
+
+ def escape(self, c):
+ if c == 'a':
+ return '\x07'
+ elif c == 'b':
+ return '\x08'
+ elif c == 'f':
+ return '\x0c'
+ elif c == 'n':
+ return '\n'
+ elif c == 'r':
+ return '\r'
+ elif c == 't':
+ return '\t'
+ elif c == 'v':
+ return '\x0b'
+ elif c in 'uxo':
+ raise ValueError,'Invalid quoted character %r' % c
+ else:
+ return c
+
+ def lex_braced(self):
+ # Lex until whitespace or end of string, the opening brace has already
+ # been consumed.
+
+ str = ''
+ while 1:
+ if self.at_end():
+ raise ValueError,"Unterminated '{' quoted word"
+
+ c = self.eat()
+ if c == '}':
+ break
+ elif c == '{':
+ str += '{' + self.lex_braced() + '}'
+ elif c == '\\' and self.look() in '{}':
+ str += self.eat()
+ else:
+ str += c
+
+ return str
+
+ def lex_quoted(self):
+ str = ''
+
+ while 1:
+ if self.at_end():
+ raise ValueError,"Unterminated '\"' quoted word"
+
+ c = self.eat()
+ if c == '"':
+ break
+ elif c == '\\':
+ if self.at_end():
+ raise ValueError,'Missing quoted character'
+
+ str += self.escape(self.eat())
+ else:
+ str += c
+
+ return str
+
+ def lex_unquoted(self, process_all=False):
+ # Lex until whitespace or end of string.
+ str = ''
+ while not self.at_end():
+ if not process_all:
+ if self.look().isspace() or self.look() == ';':
+ break
+
+ c = self.eat()
+ if c == '\\':
+ if self.at_end():
+ raise ValueError,'Missing quoted character'
+
+ str += self.escape(self.eat())
+ elif c == '[':
+ raise NotImplementedError, ('Command substitution is '
+ 'not supported')
+ elif c == '$' and not self.at_end() and (self.look().isalpha() or
+ self.look() == '{'):
+ raise NotImplementedError, ('Variable substitution is '
+ 'not supported')
+ else:
+ str += c
+
+ return str
+
+ def lex_one_token(self):
+ if self.maybe_eat('"'):
+ return self.lex_quoted()
+ elif self.maybe_eat('{'):
+ # Check for argument substitution.
+ if not self.maybe_eat('*'):
+ return self.lex_braced()
+
+ if not self.maybe_eat('}'):
+ return '*' + self.lex_braced()
+
+ if self.at_end() or self.look().isspace():
+ return '*'
+
+ raise NotImplementedError, "Argument substitution is unsupported"
+ else:
+ return self.lex_unquoted()
+
+ def lex(self):
+ while not self.at_end():
+ c = self.look()
+ if c in ' \t':
+ self.eat()
+ elif c in ';\n':
+ self.eat()
+ yield (';',)
+ else:
+ yield self.lex_one_token()
+
+class TclExecCommand:
+ kRedirectPrefixes1 = ('<', '>')
+ kRedirectPrefixes2 = ('<@', '<<', '2>', '>&', '>>', '>@')
+ kRedirectPrefixes3 = ('2>@', '2>>', '>>&', '>&@')
+ kRedirectPrefixes4 = ('2>@1',)
+
+ def __init__(self, args):
+ self.args = iter(args)
+
+ def lex(self):
+ try:
+ return self.args.next()
+ except StopIteration:
+ return None
+
+ def look(self):
+ next = self.lex()
+ if next is not None:
+ self.args = itertools.chain([next], self.args)
+ return next
+
+ def parse_redirect(self, tok, length):
+ if len(tok) == length:
+ arg = self.lex()
+ if next is None:
+ raise ValueError,'Missing argument to %r redirection' % tok
+ else:
+ tok,arg = tok[:length],tok[length:]
+
+ if tok[0] == '2':
+ op = (tok[1:],2)
+ else:
+ op = (tok,)
+ return (op, arg)
+
+ def parse_pipeline(self):
+ if self.look() is None:
+ raise ValueError,"Expected at least one argument to exec"
+
+ commands = [Command([],[])]
+ while 1:
+ arg = self.lex()
+ if arg is None:
+ break
+ elif arg == '|':
+ commands.append(Command([],[]))
+ elif arg == '|&':
+ # Write this as a redirect of stderr; it must come first because
+ # stdout may have already been redirected.
+ commands[-1].redirects.insert(0, (('>&',2),'1'))
+ commands.append(Command([],[]))
+ elif arg[:4] in TclExecCommand.kRedirectPrefixes4:
+ commands[-1].redirects.append(self.parse_redirect(arg, 4))
+ elif arg[:3] in TclExecCommand.kRedirectPrefixes3:
+ commands[-1].redirects.append(self.parse_redirect(arg, 3))
+ elif arg[:2] in TclExecCommand.kRedirectPrefixes2:
+ commands[-1].redirects.append(self.parse_redirect(arg, 2))
+ elif arg[:1] in TclExecCommand.kRedirectPrefixes1:
+ commands[-1].redirects.append(self.parse_redirect(arg, 1))
+ else:
+ commands[-1].args.append(arg)
+
+ return Pipeline(commands, False, pipe_err=True)
+
+ def parse(self):
+ ignoreStderr = False
+ keepNewline = False
+
+ # Parse arguments.
+ while 1:
+ next = self.look()
+ if not isinstance(next, str) or next[0] != '-':
+ break
+
+ if next == '--':
+ self.lex()
+ break
+ elif next == '-ignorestderr':
+ ignoreStderr = True
+ elif next == '-keepnewline':
+ keepNewline = True
+ else:
+ raise ValueError,"Invalid exec argument %r" % next
+
+ return (ignoreStderr, keepNewline, self.parse_pipeline())
+
+###
+
+import unittest
+
+class TestTclLexer(unittest.TestCase):
+ def lex(self, str, *args, **kwargs):
+ return list(TclLexer(str, *args, **kwargs).lex())
+
+ def test_preprocess(self):
+ self.assertEqual(tcl_preprocess('a b'), 'a b')
+ self.assertEqual(tcl_preprocess('a\\\nb c'), 'a b c')
+
+ def test_unquoted(self):
+ self.assertEqual(self.lex('a b c'),
+ ['a', 'b', 'c'])
+ self.assertEqual(self.lex(r'a\nb\tc\ '),
+ ['a\nb\tc '])
+ self.assertEqual(self.lex(r'a \\\$b c $\\'),
+ ['a', r'\$b', 'c', '$\\'])
+
+ def test_braced(self):
+ self.assertEqual(self.lex('a {b c} {}'),
+ ['a', 'b c', ''])
+ self.assertEqual(self.lex(r'a {b {c\n}}'),
+ ['a', 'b {c\\n}'])
+ self.assertEqual(self.lex(r'a {b\{}'),
+ ['a', 'b{'])
+ self.assertEqual(self.lex(r'{*}'), ['*'])
+ self.assertEqual(self.lex(r'{*} a'), ['*', 'a'])
+ self.assertEqual(self.lex(r'{*} a'), ['*', 'a'])
+ self.assertEqual(self.lex('{a\\\n b}'),
+ ['a b'])
+
+ def test_quoted(self):
+ self.assertEqual(self.lex('a "b c"'),
+ ['a', 'b c'])
+
+ def test_terminators(self):
+ self.assertEqual(self.lex('a\nb'),
+ ['a', (';',), 'b'])
+ self.assertEqual(self.lex('a;b'),
+ ['a', (';',), 'b'])
+ self.assertEqual(self.lex('a ; b'),
+ ['a', (';',), 'b'])
+
+class TestTclExecCommand(unittest.TestCase):
+ def parse(self, str):
+ return TclExecCommand(list(TclLexer(str).lex())).parse()
+
+ def test_basic(self):
+ self.assertEqual(self.parse('echo hello'),
+ (False, False,
+ Pipeline([Command(['echo', 'hello'], [])],
+ False, True)))
+ self.assertEqual(self.parse('echo hello | grep hello'),
+ (False, False,
+ Pipeline([Command(['echo', 'hello'], []),
+ Command(['grep', 'hello'], [])],
+ False, True)))
+
+ def test_redirect(self):
+ self.assertEqual(self.parse('echo hello > a >b >>c 2> d |& e'),
+ (False, False,
+ Pipeline([Command(['echo', 'hello'],
+ [(('>',),'a'),
+ (('>',),'b'),
+ (('>>',),'c'),
+ (('>',2),'d'),
+ (('>&',2),'1')]),
+ Command(['e'], [])],
+ False, True)))
+
+if __name__ == '__main__':
+ unittest.main()
Added: llvm/trunk/utils/lit/Test.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/Test.py?rev=81190&view=auto
==============================================================================
--- llvm/trunk/utils/lit/Test.py (added)
+++ llvm/trunk/utils/lit/Test.py Tue Sep 8 00:31:18 2009
@@ -0,0 +1,71 @@
+import os
+
+# Test results.
+
+class TestResult:
+ def __init__(self, name, isFailure):
+ self.name = name
+ self.isFailure = isFailure
+
+PASS = TestResult('PASS', False)
+XFAIL = TestResult('XFAIL', False)
+FAIL = TestResult('FAIL', True)
+XPASS = TestResult('XPASS', True)
+UNRESOLVED = TestResult('UNRESOLVED', True)
+UNSUPPORTED = TestResult('UNSUPPORTED', False)
+
+# Test classes.
+
+class TestFormat:
+ """TestFormat - Test information provider."""
+
+ def __init__(self, name):
+ self.name = name
+
+class TestSuite:
+ """TestSuite - Information on a group of tests.
+
+ A test suite groups together a set of logically related tests.
+ """
+
+ def __init__(self, name, source_root, exec_root, config):
+ self.name = name
+ self.source_root = source_root
+ self.exec_root = exec_root
+ # The test suite configuration.
+ self.config = config
+
+ def getSourcePath(self, components):
+ return os.path.join(self.source_root, *components)
+
+ def getExecPath(self, components):
+ return os.path.join(self.exec_root, *components)
+
+class Test:
+ """Test - Information on a single test instance."""
+
+ def __init__(self, suite, path_in_suite, config):
+ self.suite = suite
+ self.path_in_suite = path_in_suite
+ self.config = config
+ # The test result code, once complete.
+ self.result = None
+ # Any additional output from the test, once complete.
+ self.output = None
+ # The wall time to execute this test, if timing and once complete.
+ self.elapsed = None
+
+ def setResult(self, result, output, elapsed):
+ assert self.result is None, "Test result already set!"
+ self.result = result
+ self.output = output
+ self.elapsed = elapsed
+
+ def getFullName(self):
+ return self.suite.config.name + '::' + '/'.join(self.path_in_suite)
+
+ def getSourcePath(self):
+ return self.suite.getSourcePath(self.path_in_suite)
+
+ def getExecPath(self):
+ return self.suite.getExecPath(self.path_in_suite)
Added: llvm/trunk/utils/lit/TestRunner.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/TestRunner.py?rev=81190&view=auto
==============================================================================
--- llvm/trunk/utils/lit/TestRunner.py (added)
+++ llvm/trunk/utils/lit/TestRunner.py Tue Sep 8 00:31:18 2009
@@ -0,0 +1,460 @@
+import os, signal, subprocess, sys
+import StringIO
+
+import ShUtil
+import Test
+import Util
+
+def executeCommand(command, cwd=None, env=None):
+ p = subprocess.Popen(command, cwd=cwd,
+ stdin=subprocess.PIPE,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ env=env)
+ out,err = p.communicate()
+ exitCode = p.wait()
+
+ # Detect Ctrl-C in subprocess.
+ if exitCode == -signal.SIGINT:
+ raise KeyboardInterrupt
+
+ return out, err, exitCode
+
+def executeShCmd(cmd, cfg, cwd, results):
+ if isinstance(cmd, ShUtil.Seq):
+ if cmd.op == ';':
+ res = executeShCmd(cmd.lhs, cfg, cwd, results)
+ return executeShCmd(cmd.rhs, cfg, cwd, results)
+
+ if cmd.op == '&':
+ raise NotImplementedError,"unsupported test command: '&'"
+
+ if cmd.op == '||':
+ res = executeShCmd(cmd.lhs, cfg, cwd, results)
+ if res != 0:
+ res = executeShCmd(cmd.rhs, cfg, cwd, results)
+ return res
+ if cmd.op == '&&':
+ res = executeShCmd(cmd.lhs, cfg, cwd, results)
+ if res is None:
+ return res
+
+ if res == 0:
+ res = executeShCmd(cmd.rhs, cfg, cwd, results)
+ return res
+
+ raise ValueError,'Unknown shell command: %r' % cmd.op
+
+ assert isinstance(cmd, ShUtil.Pipeline)
+ procs = []
+ input = subprocess.PIPE
+ for j in cmd.commands:
+ redirects = [(0,), (1,), (2,)]
+ for r in j.redirects:
+ if r[0] == ('>',2):
+ redirects[2] = [r[1], 'w', None]
+ elif r[0] == ('>&',2) and r[1] in '012':
+ redirects[2] = redirects[int(r[1])]
+ elif r[0] == ('>&',) or r[0] == ('&>',):
+ redirects[1] = redirects[2] = [r[1], 'w', None]
+ elif r[0] == ('>',):
+ redirects[1] = [r[1], 'w', None]
+ elif r[0] == ('<',):
+ redirects[0] = [r[1], 'r', None]
+ else:
+ raise NotImplementedError,"Unsupported redirect: %r" % (r,)
+
+ final_redirects = []
+ for index,r in enumerate(redirects):
+ if r == (0,):
+ result = input
+ elif r == (1,):
+ if index == 0:
+ raise NotImplementedError,"Unsupported redirect for stdin"
+ elif index == 1:
+ result = subprocess.PIPE
+ else:
+ result = subprocess.STDOUT
+ elif r == (2,):
+ if index != 2:
+ raise NotImplementedError,"Unsupported redirect on stdout"
+ result = subprocess.PIPE
+ else:
+ if r[2] is None:
+ r[2] = open(r[0], r[1])
+ result = r[2]
+ final_redirects.append(result)
+
+ stdin, stdout, stderr = final_redirects
+
+ # If stderr wants to come from stdout, but stdout isn't a pipe, then put
+ # stderr on a pipe and treat it as stdout.
+ if (stderr == subprocess.STDOUT and stdout != subprocess.PIPE):
+ stderr = subprocess.PIPE
+ stderrIsStdout = True
+ else:
+ stderrIsStdout = False
+ procs.append(subprocess.Popen(j.args, cwd=cwd,
+ stdin = stdin,
+ stdout = stdout,
+ stderr = stderr,
+ env = cfg.environment,
+ close_fds = True))
+
+ # Immediately close stdin for any process taking stdin from us.
+ if stdin == subprocess.PIPE:
+ procs[-1].stdin.close()
+ procs[-1].stdin = None
+
+ # Update the current stdin source.
+ if stdout == subprocess.PIPE:
+ input = procs[-1].stdout
+ elif stderrIsStdout:
+ input = procs[-1].stderr
+ else:
+ input = subprocess.PIPE
+
+ # FIXME: There is a potential for deadlock here, when we have a pipe and
+ # some process other than the last one ends up blocked on stderr.
+ procData = [None] * len(procs)
+ procData[-1] = procs[-1].communicate()
+ for i in range(len(procs) - 1):
+ if procs[i].stdout is not None:
+ out = procs[i].stdout.read()
+ else:
+ out = ''
+ if procs[i].stderr is not None:
+ err = procs[i].stderr.read()
+ else:
+ err = ''
+ procData[i] = (out,err)
+
+ exitCode = None
+ for i,(out,err) in enumerate(procData):
+ res = procs[i].wait()
+ # Detect Ctrl-C in subprocess.
+ if res == -signal.SIGINT:
+ raise KeyboardInterrupt
+
+ results.append((cmd.commands[i], out, err, res))
+ if cmd.pipe_err:
+ # Python treats the exit code as a signed char.
+ if res < 0:
+ exitCode = min(exitCode, res)
+ else:
+ exitCode = max(exitCode, res)
+ else:
+ exitCode = res
+
+ if cmd.negate:
+ exitCode = not exitCode
+
+ return exitCode
+
+def executeScriptInternal(test, litConfig, tmpBase, commands, cwd):
+ ln = ' &&\n'.join(commands)
+ try:
+ cmd = ShUtil.ShParser(ln, litConfig.isWindows).parse()
+ except:
+ return (Test.FAIL, "shell parser error on: %r" % ln)
+
+ results = []
+ exitCode = executeShCmd(cmd, test.config, cwd, results)
+
+ out = err = ''
+ for i,(cmd, cmd_out,cmd_err,res) in enumerate(results):
+ out += 'Command %d: %s\n' % (i, ' '.join('"%s"' % s for s in cmd.args))
+ out += 'Command %d Result: %r\n' % (i, res)
+ out += 'Command %d Output:\n%s\n\n' % (i, cmd_out)
+ out += 'Command %d Stderr:\n%s\n\n' % (i, cmd_err)
+
+ return out, err, exitCode
+
+def executeTclScriptInternal(test, litConfig, tmpBase, commands, cwd):
+ import TclUtil
+ cmds = []
+ for ln in commands:
+ # Given the unfortunate way LLVM's test are written, the line gets
+ # backslash substitution done twice.
+ ln = TclUtil.TclLexer(ln).lex_unquoted(process_all = True)
+
+ try:
+ tokens = list(TclUtil.TclLexer(ln).lex())
+ except:
+ return (Test.FAIL, "Tcl lexer error on: %r" % ln)
+
+ # Validate there are no control tokens.
+ for t in tokens:
+ if not isinstance(t, str):
+ return (Test.FAIL,
+ "Invalid test line: %r containing %r" % (ln, t))
+
+ try:
+ cmds.append(TclUtil.TclExecCommand(tokens).parse_pipeline())
+ except:
+ return (TestStatus.Fail, "Tcl 'exec' parse error on: %r" % ln)
+
+ cmd = cmds[0]
+ for c in cmds[1:]:
+ cmd = ShUtil.Seq(cmd, '&&', c)
+
+ if litConfig.useTclAsSh:
+ script = tmpBase + '.script'
+
+ # Write script file
+ f = open(script,'w')
+ print >>f, 'set -o pipefail'
+ cmd.toShell(f, pipefail = True)
+ f.close()
+
+ if 0:
+ print >>sys.stdout, cmd
+ print >>sys.stdout, open(script).read()
+ print >>sys.stdout
+ return '', '', 0
+
+ command = ['/bin/sh', script]
+ out,err,exitCode = executeCommand(command, cwd=cwd,
+ env=test.config.environment)
+
+ # Tcl commands fail on standard error output.
+ if err:
+ exitCode = 1
+ out = 'Command has output on stderr!\n\n' + out
+
+ return out,err,exitCode
+ else:
+ results = []
+ exitCode = executeShCmd(cmd, test.config, cwd, results)
+
+ out = err = ''
+
+ # Tcl commands fail on standard error output.
+ if [True for _,_,err,res in results if err]:
+ exitCode = 1
+ out += 'Command has output on stderr!\n\n'
+
+ for i,(cmd, cmd_out, cmd_err, res) in enumerate(results):
+ out += 'Command %d: %s\n' % (i, ' '.join('"%s"' % s for s in cmd.args))
+ out += 'Command %d Result: %r\n' % (i, res)
+ out += 'Command %d Output:\n%s\n\n' % (i, cmd_out)
+ out += 'Command %d Stderr:\n%s\n\n' % (i, cmd_err)
+
+ return out, err, exitCode
+
+def executeScript(test, litConfig, tmpBase, commands, cwd):
+ script = tmpBase + '.script'
+ if litConfig.isWindows:
+ script += '.bat'
+
+ # Write script file
+ f = open(script,'w')
+ if litConfig.isWindows:
+ f.write('\nif %ERRORLEVEL% NEQ 0 EXIT\n'.join(commands))
+ else:
+ f.write(' &&\n'.join(commands))
+ f.write('\n')
+ f.close()
+
+ if litConfig.isWindows:
+ command = ['cmd','/c', script]
+ else:
+ command = ['/bin/sh', script]
+ if litConfig.useValgrind:
+ # FIXME: Running valgrind on sh is overkill. We probably could just
+ # run on clang with no real loss.
+ valgrindArgs = ['valgrind', '-q',
+ '--tool=memcheck', '--trace-children=yes',
+ '--error-exitcode=123']
+ valgrindArgs.extend(litConfig.valgrindArgs)
+
+ command = valgrindArgs + command
+
+ return executeCommand(command, cwd=cwd, env=test.config.environment)
+
+def parseIntegratedTestScript(test, xfailHasColon, requireAndAnd):
+ """parseIntegratedTestScript - Scan an LLVM/Clang style integrated test
+ script and extract the lines to 'RUN' as well as 'XFAIL' and 'XTARGET'
+ information. The RUN lines also will have variable substitution performed.
+ """
+
+ # Get the temporary location, this is always relative to the test suite
+ # root, not test source root.
+ #
+ # FIXME: This should not be here?
+ sourcepath = test.getSourcePath()
+ execpath = test.getExecPath()
+ execdir,execbase = os.path.split(execpath)
+ tmpBase = os.path.join(execdir, 'Output', execbase)
+
+ # We use #_MARKER_# to hide %% while we do the other substitutions.
+ substitutions = [('%%', '#_MARKER_#')]
+ substitutions.extend(test.config.substitutions)
+ substitutions.extend([('%s', sourcepath),
+ ('%S', os.path.dirname(sourcepath)),
+ ('%p', os.path.dirname(sourcepath)),
+ ('%t', tmpBase + '.tmp'),
+ ('#_MARKER_#', '%')])
+
+ # Collect the test lines from the script.
+ script = []
+ xfails = []
+ xtargets = []
+ for ln in open(sourcepath):
+ if 'RUN:' in ln:
+ # Isolate the command to run.
+ index = ln.index('RUN:')
+ ln = ln[index+4:]
+
+ # Trim trailing whitespace.
+ ln = ln.rstrip()
+
+ # Collapse lines with trailing '\\'.
+ if script and script[-1][-1] == '\\':
+ script[-1] = script[-1][:-1] + ln
+ else:
+ script.append(ln)
+ elif xfailHasColon and 'XFAIL:' in ln:
+ items = ln[ln.index('XFAIL:') + 6:].split(',')
+ xfails.extend([s.strip() for s in items])
+ elif not xfailHasColon and 'XFAIL' in ln:
+ items = ln[ln.index('XFAIL') + 5:].split(',')
+ xfails.extend([s.strip() for s in items])
+ elif 'XTARGET:' in ln:
+ items = ln[ln.index('XTARGET:') + 8:].split(',')
+ xtargets.extend([s.strip() for s in items])
+ elif 'END.' in ln:
+ # Check for END. lines.
+ if ln[ln.index('END.'):].strip() == 'END.':
+ break
+
+ # Apply substitutions to the script.
+ def processLine(ln):
+ # Apply substitutions
+ for a,b in substitutions:
+ ln = ln.replace(a,b)
+
+ # Strip the trailing newline and any extra whitespace.
+ return ln.strip()
+ script = map(processLine, script)
+
+ # Verify the script contains a run line.
+ if not script:
+ return (Test.UNRESOLVED, "Test has no run line!")
+
+ if script[-1][-1] == '\\':
+ return (Test.UNRESOLVED, "Test has unterminated run lines (with '\\')")
+
+ # Validate interior lines for '&&', a lovely historical artifact.
+ if requireAndAnd:
+ for i in range(len(script) - 1):
+ ln = script[i]
+
+ if not ln.endswith('&&'):
+ return (Test.FAIL,
+ ("MISSING \'&&\': %s\n" +
+ "FOLLOWED BY : %s\n") % (ln, script[i + 1]))
+
+ # Strip off '&&'
+ script[i] = ln[:-2]
+
+ return script,xfails,xtargets,tmpBase,execdir
+
+def formatTestOutput(status, out, err, exitCode, script):
+ output = StringIO.StringIO()
+ print >>output, "Script:"
+ print >>output, "--"
+ print >>output, '\n'.join(script)
+ print >>output, "--"
+ print >>output, "Exit Code: %r" % exitCode
+ print >>output, "Command Output (stdout):"
+ print >>output, "--"
+ output.write(out)
+ print >>output, "--"
+ print >>output, "Command Output (stderr):"
+ print >>output, "--"
+ output.write(err)
+ print >>output, "--"
+ return (status, output.getvalue())
+
+def executeTclTest(test, litConfig):
+ if test.config.unsupported:
+ return (Test.UNSUPPORTED, 'Test is unsupported')
+
+ res = parseIntegratedTestScript(test, True, False)
+ if len(res) == 2:
+ return res
+
+ script, xfails, xtargets, tmpBase, execdir = res
+
+ if litConfig.noExecute:
+ return (Test.PASS, '')
+
+ # Create the output directory if it does not already exist.
+ Util.mkdir_p(os.path.dirname(tmpBase))
+
+ res = executeTclScriptInternal(test, litConfig, tmpBase, script, execdir)
+ if len(res) == 2:
+ return res
+
+ isXFail = False
+ for item in xfails:
+ if item == '*' or item in test.suite.config.target_triple:
+ isXFail = True
+ break
+
+ # If this is XFAIL, see if it is expected to pass on this target.
+ if isXFail:
+ for item in xtargets:
+ if item == '*' or item in test.suite.config.target_triple:
+ isXFail = False
+ break
+
+ out,err,exitCode = res
+ if isXFail:
+ ok = exitCode != 0
+ status = (Test.XPASS, Test.XFAIL)[ok]
+ else:
+ ok = exitCode == 0
+ status = (Test.FAIL, Test.PASS)[ok]
+
+ if ok:
+ return (status,'')
+
+ return formatTestOutput(status, out, err, exitCode, script)
+
+def executeShTest(test, litConfig, useExternalSh, requireAndAnd):
+ if test.config.unsupported:
+ return (Test.UNSUPPORTED, 'Test is unsupported')
+
+ res = parseIntegratedTestScript(test, False, requireAndAnd)
+ if len(res) == 2:
+ return res
+
+ script, xfails, xtargets, tmpBase, execdir = res
+
+ if litConfig.noExecute:
+ return (Test.PASS, '')
+
+ # Create the output directory if it does not already exist.
+ Util.mkdir_p(os.path.dirname(tmpBase))
+
+ if useExternalSh:
+ res = executeScript(test, litConfig, tmpBase, script, execdir)
+ else:
+ res = executeScriptInternal(test, litConfig, tmpBase, script, execdir)
+ if len(res) == 2:
+ return res
+
+ out,err,exitCode = res
+ if xfails:
+ ok = exitCode != 0
+ status = (Test.XPASS, Test.XFAIL)[ok]
+ else:
+ ok = exitCode == 0
+ status = (Test.FAIL, Test.PASS)[ok]
+
+ if ok:
+ return (status,'')
+
+ return formatTestOutput(status, out, err, exitCode, script)
Added: llvm/trunk/utils/lit/TestingConfig.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/TestingConfig.py?rev=81190&view=auto
==============================================================================
--- llvm/trunk/utils/lit/TestingConfig.py (added)
+++ llvm/trunk/utils/lit/TestingConfig.py Tue Sep 8 00:31:18 2009
@@ -0,0 +1,95 @@
+import os
+
+class TestingConfig:
+ """"
+ TestingConfig - Information on the tests inside a suite.
+ """
+
+ @staticmethod
+ def frompath(path, parent, litConfig, mustExist, config = None):
+ if config is None:
+ # Set the environment based on the command line arguments.
+ environment = {
+ 'PATH' : os.pathsep.join(litConfig.path +
+ [os.environ.get('PATH','')]),
+ 'SYSTEMROOT' : os.environ.get('SYSTEMROOT',''),
+ }
+
+ config = TestingConfig(parent,
+ name = '<unnamed>',
+ suffixes = set(),
+ test_format = None,
+ environment = environment,
+ substitutions = [],
+ unsupported = False,
+ on_clone = None,
+ test_exec_root = None,
+ test_source_root = None,
+ excludes = [])
+
+ if os.path.exists(path):
+ # FIXME: Improve detection and error reporting of errors in the
+ # config file.
+ f = open(path)
+ cfg_globals = dict(globals())
+ cfg_globals['config'] = config
+ cfg_globals['lit'] = litConfig
+ cfg_globals['__file__'] = path
+ try:
+ exec f in cfg_globals
+ except SystemExit,status:
+ # We allow normal system exit inside a config file to just
+ # return control without error.
+ if status.args:
+ raise
+ f.close()
+ elif mustExist:
+ litConfig.fatal('unable to load config from %r ' % path)
+
+ config.finish(litConfig)
+ return config
+
+ def __init__(self, parent, name, suffixes, test_format,
+ environment, substitutions, unsupported, on_clone,
+ test_exec_root, test_source_root, excludes):
+ self.parent = parent
+ self.name = str(name)
+ self.suffixes = set(suffixes)
+ self.test_format = test_format
+ self.environment = dict(environment)
+ self.substitutions = list(substitutions)
+ self.unsupported = unsupported
+ self.on_clone = on_clone
+ self.test_exec_root = test_exec_root
+ self.test_source_root = test_source_root
+ self.excludes = set(excludes)
+
+ def clone(self, path):
+ # FIXME: Chain implementations?
+ #
+ # FIXME: Allow extra parameters?
+ cfg = TestingConfig(self, self.name, self.suffixes, self.test_format,
+ self.environment, self.substitutions,
+ self.unsupported, self.on_clone,
+ self.test_exec_root, self.test_source_root,
+ self.excludes)
+ if cfg.on_clone:
+ cfg.on_clone(self, cfg, path)
+ return cfg
+
+ def finish(self, litConfig):
+ """finish() - Finish this config object, after loading is complete."""
+
+ self.name = str(self.name)
+ self.suffixes = set(self.suffixes)
+ self.environment = dict(self.environment)
+ self.substitutions = list(self.substitutions)
+ if self.test_exec_root is not None:
+ # FIXME: This should really only be suite in test suite config
+ # files. Should we distinguish them?
+ self.test_exec_root = str(self.test_exec_root)
+ if self.test_source_root is not None:
+ # FIXME: This should really only be suite in test suite config
+ # files. Should we distinguish them?
+ self.test_source_root = str(self.test_source_root)
+ self.excludes = set(self.excludes)
Added: llvm/trunk/utils/lit/Util.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/Util.py?rev=81190&view=auto
==============================================================================
--- llvm/trunk/utils/lit/Util.py (added)
+++ llvm/trunk/utils/lit/Util.py Tue Sep 8 00:31:18 2009
@@ -0,0 +1,124 @@
+import os, sys
+
+def detectCPUs():
+ """
+ Detects the number of CPUs on a system. Cribbed from pp.
+ """
+ # Linux, Unix and MacOS:
+ if hasattr(os, "sysconf"):
+ if os.sysconf_names.has_key("SC_NPROCESSORS_ONLN"):
+ # Linux & Unix:
+ ncpus = os.sysconf("SC_NPROCESSORS_ONLN")
+ if isinstance(ncpus, int) and ncpus > 0:
+ return ncpus
+ else: # OSX:
+ return int(os.popen2("sysctl -n hw.ncpu")[1].read())
+ # Windows:
+ if os.environ.has_key("NUMBER_OF_PROCESSORS"):
+ ncpus = int(os.environ["NUMBER_OF_PROCESSORS"]);
+ if ncpus > 0:
+ return ncpus
+ return 1 # Default
+
+def mkdir_p(path):
+ """mkdir_p(path) - Make the "path" directory, if it does not exist; this
+ will also make directories for any missing parent directories."""
+ import errno
+
+ if not path or os.path.exists(path):
+ return
+
+ parent = os.path.dirname(path)
+ if parent != path:
+ mkdir_p(parent)
+
+ try:
+ os.mkdir(path)
+ except OSError,e:
+ # Ignore EEXIST, which may occur during a race condition.
+ if e.errno != errno.EEXIST:
+ raise
+
+def capture(args):
+ import subprocess
+ """capture(command) - Run the given command (or argv list) in a shell and
+ return the standard output."""
+ p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ out,_ = p.communicate()
+ return out
+
+def which(command, paths = None):
+ """which(command, [paths]) - Look up the given command in the paths string
+ (or the PATH environment variable, if unspecified)."""
+
+ if paths is None:
+ paths = os.environ.get('PATH','')
+
+ # Check for absolute match first.
+ if os.path.exists(command):
+ return command
+
+ # Would be nice if Python had a lib function for this.
+ if not paths:
+ paths = os.defpath
+
+ # Get suffixes to search.
+ pathext = os.environ.get('PATHEXT', '').split(os.pathsep)
+
+ # Search the paths...
+ for path in paths.split(os.pathsep):
+ for ext in pathext:
+ p = os.path.join(path, command + ext)
+ if os.path.exists(p):
+ return p
+
+ return None
+
+def printHistogram(items, title = 'Items'):
+ import itertools, math
+
+ items.sort(key = lambda (_,v): v)
+
+ maxValue = max([v for _,v in items])
+
+ # Select first "nice" bar height that produces more than 10 bars.
+ power = int(math.ceil(math.log(maxValue, 10)))
+ for inc in itertools.cycle((5, 2, 2.5, 1)):
+ barH = inc * 10**power
+ N = int(math.ceil(maxValue / barH))
+ if N > 10:
+ break
+ elif inc == 1:
+ power -= 1
+
+ histo = [set() for i in range(N)]
+ for name,v in items:
+ bin = min(int(N * v/maxValue), N-1)
+ histo[bin].add(name)
+
+ barW = 40
+ hr = '-' * (barW + 34)
+ print '\nSlowest %s:' % title
+ print hr
+ for name,value in items[-20:]:
+ print '%.2fs: %s' % (value, name)
+ print '\n%s Times:' % title
+ print hr
+ pDigits = int(math.ceil(math.log(maxValue, 10)))
+ pfDigits = max(0, 3-pDigits)
+ if pfDigits:
+ pDigits += pfDigits + 1
+ cDigits = int(math.ceil(math.log(len(items), 10)))
+ print "[%s] :: [%s] :: [%s]" % ('Range'.center((pDigits+1)*2 + 3),
+ 'Percentage'.center(barW),
+ 'Count'.center(cDigits*2 + 1))
+ print hr
+ for i,row in enumerate(histo):
+ pct = float(len(row)) / len(items)
+ w = int(barW * pct)
+ print "[%*.*fs,%*.*fs)" % (pDigits, pfDigits, i*barH,
+ pDigits, pfDigits, (i+1)*barH),
+ print ":: [%s%s] :: [%*d/%*d]" % ('*'*w, ' '*(barW-w),
+ cDigits, len(row),
+ cDigits, len(items))
+
Added: llvm/trunk/utils/lit/lit.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/lit.py?rev=81190&view=auto
==============================================================================
--- llvm/trunk/utils/lit/lit.py (added)
+++ llvm/trunk/utils/lit/lit.py Tue Sep 8 00:31:18 2009
@@ -0,0 +1,519 @@
+#!/usr/bin/env python
+
+"""
+lit - LLVM Integrated Tester.
+
+See lit.pod for more information.
+"""
+
+import math, os, platform, random, re, sys, time, threading, traceback
+
+import ProgressBar
+import TestRunner
+import Util
+
+from TestingConfig import TestingConfig
+import LitConfig
+import Test
+
+# FIXME: Rename to 'config.lit', 'site.lit', and 'local.lit' ?
+kConfigName = 'lit.cfg'
+kSiteConfigName = 'lit.site.cfg'
+kLocalConfigName = 'lit.local.cfg'
+
+class TestingProgressDisplay:
+ def __init__(self, opts, numTests, progressBar=None):
+ self.opts = opts
+ self.numTests = numTests
+ self.current = None
+ self.lock = threading.Lock()
+ self.progressBar = progressBar
+ self.completed = 0
+
+ def update(self, test):
+ # Avoid locking overhead in quiet mode
+ if self.opts.quiet and not test.result.isFailure:
+ return
+
+ # Output lock.
+ self.lock.acquire()
+ try:
+ self.handleUpdate(test)
+ finally:
+ self.lock.release()
+
+ def finish(self):
+ if self.progressBar:
+ self.progressBar.clear()
+ elif self.opts.quiet:
+ pass
+ elif self.opts.succinct:
+ sys.stdout.write('\n')
+
+ def handleUpdate(self, test):
+ self.completed += 1
+ if self.progressBar:
+ self.progressBar.update(float(self.completed)/self.numTests,
+ test.getFullName())
+
+ if self.opts.succinct and not test.result.isFailure:
+ return
+
+ if self.progressBar:
+ self.progressBar.clear()
+
+ print '%s: %s (%d of %d)' % (test.result.name, test.getFullName(),
+ self.completed, self.numTests)
+
+ if test.result.isFailure and self.opts.showOutput:
+ print "%s TEST '%s' FAILED %s" % ('*'*20, test.getFullName(),
+ '*'*20)
+ print test.output
+ print "*" * 20
+
+ sys.stdout.flush()
+
+class TestProvider:
+ def __init__(self, tests, maxTime):
+ self.maxTime = maxTime
+ self.iter = iter(tests)
+ self.lock = threading.Lock()
+ self.startTime = time.time()
+
+ def get(self):
+ # Check if we have run out of time.
+ if self.maxTime is not None:
+ if time.time() - self.startTime > self.maxTime:
+ return None
+
+ # Otherwise take the next test.
+ self.lock.acquire()
+ try:
+ item = self.iter.next()
+ except StopIteration:
+ item = None
+ self.lock.release()
+ return item
+
+class Tester(threading.Thread):
+ def __init__(self, litConfig, provider, display):
+ threading.Thread.__init__(self)
+ self.litConfig = litConfig
+ self.provider = provider
+ self.display = display
+
+ def run(self):
+ while 1:
+ item = self.provider.get()
+ if item is None:
+ break
+ self.runTest(item)
+
+ def runTest(self, test):
+ result = None
+ startTime = time.time()
+ try:
+ result, output = test.config.test_format.execute(test,
+ self.litConfig)
+ except KeyboardInterrupt:
+ # This is a sad hack. Unfortunately subprocess goes
+ # bonkers with ctrl-c and we start forking merrily.
+ print '\nCtrl-C detected, goodbye.'
+ os.kill(0,9)
+ except:
+ if self.litConfig.debug:
+ raise
+ result = Test.UNRESOLVED
+ output = 'Exception during script execution:\n'
+ output += traceback.format_exc()
+ output += '\n'
+ elapsed = time.time() - startTime
+
+ test.setResult(result, output, elapsed)
+ self.display.update(test)
+
+def dirContainsTestSuite(path):
+ cfgpath = os.path.join(path, kSiteConfigName)
+ if os.path.exists(cfgpath):
+ return cfgpath
+ cfgpath = os.path.join(path, kConfigName)
+ if os.path.exists(cfgpath):
+ return cfgpath
+
+def getTestSuite(item, litConfig, cache):
+ """getTestSuite(item, litConfig, cache) -> (suite, relative_path)
+
+ Find the test suite containing @arg item.
+
+ @retval (None, ...) - Indicates no test suite contains @arg item.
+ @retval (suite, relative_path) - The suite that @arg item is in, and its
+ relative path inside that suite.
+ """
+ def search1(path):
+ # Check for a site config or a lit config.
+ cfgpath = dirContainsTestSuite(path)
+
+ # If we didn't find a config file, keep looking.
+ if not cfgpath:
+ parent,base = os.path.split(path)
+ if parent == item:
+ return (None, ())
+
+ ts, relative = search(parent)
+ return (ts, relative + (base,))
+
+ # We found a config file, load it.
+ if litConfig.debug:
+ litConfig.note('loading suite config %r' % cfgpath)
+
+ cfg = TestingConfig.frompath(cfgpath, None, litConfig, mustExist = True)
+ source_root = os.path.realpath(cfg.test_source_root or path)
+ exec_root = os.path.realpath(cfg.test_exec_root or path)
+ return Test.TestSuite(cfg.name, source_root, exec_root, cfg), ()
+
+ def search(path):
+ # Check for an already instantiated test suite.
+ res = cache.get(path)
+ if res is None:
+ cache[path] = res = search1(path)
+ return res
+
+ # Canonicalize the path.
+ item = os.path.realpath(item)
+
+ # Skip files and virtual components.
+ components = []
+ while not os.path.isdir(item):
+ parent,base = os.path.split(item)
+ if parent == item:
+ return (None, ())
+ components.append(base)
+ item = parent
+ components.reverse()
+
+ ts, relative = search(item)
+ return ts, tuple(relative + tuple(components))
+
+def getLocalConfig(ts, path_in_suite, litConfig, cache):
+ def search1(path_in_suite):
+ # Get the parent config.
+ if not path_in_suite:
+ parent = ts.config
+ else:
+ parent = search(path_in_suite[:-1])
+
+ # Load the local configuration.
+ source_path = ts.getSourcePath(path_in_suite)
+ cfgpath = os.path.join(source_path, kLocalConfigName)
+ if litConfig.debug:
+ litConfig.note('loading local config %r' % cfgpath)
+ return TestingConfig.frompath(cfgpath, parent, litConfig,
+ mustExist = False,
+ config = parent.clone(cfgpath))
+
+ def search(path_in_suite):
+ key = (ts, path_in_suite)
+ res = cache.get(key)
+ if res is None:
+ cache[key] = res = search1(path_in_suite)
+ return res
+
+ return search(path_in_suite)
+
+def getTests(path, litConfig, testSuiteCache, localConfigCache):
+ # Find the test suite for this input and its relative path.
+ ts,path_in_suite = getTestSuite(path, litConfig, testSuiteCache)
+ if ts is None:
+ litConfig.warning('unable to find test suite for %r' % path)
+ return ()
+
+ if litConfig.debug:
+ litConfig.note('resolved input %r to %r::%r' % (path, ts.name,
+ path_in_suite))
+
+ return getTestsInSuite(ts, path_in_suite, litConfig,
+ testSuiteCache, localConfigCache)
+
+def getTestsInSuite(ts, path_in_suite, litConfig,
+ testSuiteCache, localConfigCache):
+ # Check that the source path exists (errors here are reported by the
+ # caller).
+ source_path = ts.getSourcePath(path_in_suite)
+ if not os.path.exists(source_path):
+ return
+
+ # Check if the user named a test directly.
+ if not os.path.isdir(source_path):
+ lc = getLocalConfig(ts, path_in_suite[:-1], litConfig, localConfigCache)
+ yield Test.Test(ts, path_in_suite, lc)
+ return
+
+ # Otherwise we have a directory to search for tests, start by getting the
+ # local configuration.
+ lc = getLocalConfig(ts, path_in_suite, litConfig, localConfigCache)
+ for filename in os.listdir(source_path):
+ # FIXME: This doesn't belong here?
+ if filename == 'Output' or filename in lc.excludes:
+ continue
+
+ filepath = os.path.join(source_path, filename)
+ if os.path.isdir(filepath):
+ # If this directory contains a test suite, reload it.
+ if dirContainsTestSuite(filepath):
+ for res in getTests(filepath, litConfig,
+ testSuiteCache, localConfigCache):
+ yield res
+ else:
+ # Otherwise, continue loading from inside this test suite.
+ for res in getTestsInSuite(ts, path_in_suite + (filename,),
+ litConfig, testSuiteCache,
+ localConfigCache):
+ yield res
+ else:
+ # Otherwise add tests for matching suffixes.
+ base,ext = os.path.splitext(filename)
+ if ext in lc.suffixes:
+ yield Test.Test(ts, path_in_suite + (filename,), lc)
+
+def runTests(numThreads, litConfig, provider, display):
+ # If only using one testing thread, don't use threads at all; this lets us
+ # profile, among other things.
+ if numThreads == 1:
+ t = Tester(litConfig, provider, display)
+ t.run()
+ return
+
+ # Otherwise spin up the testing threads and wait for them to finish.
+ testers = [Tester(litConfig, provider, display)
+ for i in range(numThreads)]
+ for t in testers:
+ t.start()
+ try:
+ for t in testers:
+ t.join()
+ except KeyboardInterrupt:
+ sys.exit(2)
+
+def main():
+ global options
+ from optparse import OptionParser, OptionGroup
+ parser = OptionParser("usage: %prog [options] {file-or-path}")
+
+ parser.add_option("-j", "--threads", dest="numThreads", metavar="N",
+ help="Number of testing threads",
+ type=int, action="store", default=None)
+
+ group = OptionGroup(parser, "Output Format")
+ # FIXME: I find these names very confusing, although I like the
+ # functionality.
+ group.add_option("-q", "--quiet", dest="quiet",
+ help="Suppress no error output",
+ action="store_true", default=False)
+ group.add_option("-s", "--succinct", dest="succinct",
+ help="Reduce amount of output",
+ action="store_true", default=False)
+ group.add_option("-v", "--verbose", dest="showOutput",
+ help="Show all test output",
+ action="store_true", default=False)
+ group.add_option("", "--no-progress-bar", dest="useProgressBar",
+ help="Do not use curses based progress bar",
+ action="store_false", default=True)
+ parser.add_option_group(group)
+
+ group = OptionGroup(parser, "Test Execution")
+ group.add_option("", "--path", dest="path",
+ help="Additional paths to add to testing environment",
+ action="append", type=str, default=[])
+ group.add_option("", "--vg", dest="useValgrind",
+ help="Run tests under valgrind",
+ action="store_true", default=False)
+ group.add_option("", "--vg-arg", dest="valgrindArgs", metavar="ARG",
+ help="Specify an extra argument for valgrind",
+ type=str, action="append", default=[])
+ group.add_option("", "--time-tests", dest="timeTests",
+ help="Track elapsed wall time for each test",
+ action="store_true", default=False)
+ group.add_option("", "--no-execute", dest="noExecute",
+ help="Don't execute any tests (assume PASS)",
+ action="store_true", default=False)
+ parser.add_option_group(group)
+
+ group = OptionGroup(parser, "Test Selection")
+ group.add_option("", "--max-tests", dest="maxTests", metavar="N",
+ help="Maximum number of tests to run",
+ action="store", type=int, default=None)
+ group.add_option("", "--max-time", dest="maxTime", metavar="N",
+ help="Maximum time to spend testing (in seconds)",
+ action="store", type=float, default=None)
+ group.add_option("", "--shuffle", dest="shuffle",
+ help="Run tests in random order",
+ action="store_true", default=False)
+ parser.add_option_group(group)
+
+ group = OptionGroup(parser, "Debug and Experimental Options")
+ group.add_option("", "--debug", dest="debug",
+ help="Enable debugging (for 'lit' development)",
+ action="store_true", default=False)
+ group.add_option("", "--show-suites", dest="showSuites",
+ help="Show discovered test suites",
+ action="store_true", default=False)
+ group.add_option("", "--no-tcl-as-sh", dest="useTclAsSh",
+ help="Don't run Tcl scripts using 'sh'",
+ action="store_false", default=True)
+ parser.add_option_group(group)
+
+ (opts, args) = parser.parse_args()
+
+ if not args:
+ parser.error('No inputs specified')
+
+ if opts.numThreads is None:
+ opts.numThreads = Util.detectCPUs()
+
+ inputs = args
+
+ # Create the global config object.
+ litConfig = LitConfig.LitConfig(progname = os.path.basename(sys.argv[0]),
+ path = opts.path,
+ quiet = opts.quiet,
+ useValgrind = opts.useValgrind,
+ valgrindArgs = opts.valgrindArgs,
+ useTclAsSh = opts.useTclAsSh,
+ noExecute = opts.noExecute,
+ debug = opts.debug,
+ isWindows = (platform.system()=='Windows'))
+
+ # Load the tests from the inputs.
+ tests = []
+ testSuiteCache = {}
+ localConfigCache = {}
+ for input in inputs:
+ prev = len(tests)
+ tests.extend(getTests(input, litConfig,
+ testSuiteCache, localConfigCache))
+ if prev == len(tests):
+ litConfig.warning('input %r contained no tests' % input)
+
+ # If there were any errors during test discovery, exit now.
+ if litConfig.numErrors:
+ print >>sys.stderr, '%d errors, exiting.' % litConfig.numErrors
+ sys.exit(2)
+
+ if opts.showSuites:
+ suitesAndTests = dict([(ts,[])
+ for ts,_ in testSuiteCache.values()])
+ for t in tests:
+ suitesAndTests[t.suite].append(t)
+
+ print '-- Test Suites --'
+ suitesAndTests = suitesAndTests.items()
+ suitesAndTests.sort(key = lambda (ts,_): ts.name)
+ for ts,tests in suitesAndTests:
+ print ' %s - %d tests' %(ts.name, len(tests))
+ print ' Source Root: %s' % ts.source_root
+ print ' Exec Root : %s' % ts.exec_root
+
+ # Select and order the tests.
+ numTotalTests = len(tests)
+ if opts.shuffle:
+ random.shuffle(tests)
+ else:
+ tests.sort(key = lambda t: t.getFullName())
+ if opts.maxTests is not None:
+ tests = tests[:opts.maxTests]
+
+ extra = ''
+ if len(tests) != numTotalTests:
+ extra = ' of %d' % numTotalTests
+ header = '-- Testing: %d%s tests, %d threads --'%(len(tests),extra,
+ opts.numThreads)
+
+ progressBar = None
+ if not opts.quiet:
+ if opts.succinct and opts.useProgressBar:
+ try:
+ tc = ProgressBar.TerminalController()
+ progressBar = ProgressBar.ProgressBar(tc, header)
+ except ValueError:
+ print header
+ progressBar = ProgressBar.SimpleProgressBar('Testing: ')
+ else:
+ print header
+
+ # Don't create more threads than tests.
+ opts.numThreads = min(len(tests), opts.numThreads)
+
+ startTime = time.time()
+ display = TestingProgressDisplay(opts, len(tests), progressBar)
+ provider = TestProvider(tests, opts.maxTime)
+ runTests(opts.numThreads, litConfig, provider, display)
+ display.finish()
+
+ if not opts.quiet:
+ print 'Testing Time: %.2fs'%(time.time() - startTime)
+
+ # Update results for any tests which weren't run.
+ for t in tests:
+ if t.result is None:
+ t.setResult(Test.UNRESOLVED, '', 0.0)
+
+ # List test results organized by kind.
+ hasFailures = False
+ byCode = {}
+ for t in tests:
+ if t.result not in byCode:
+ byCode[t.result] = []
+ byCode[t.result].append(t)
+ if t.result.isFailure:
+ hasFailures = True
+
+ # FIXME: Show unresolved and (optionally) unsupported tests.
+ for title,code in (('Unexpected Passing Tests', Test.XPASS),
+ ('Failing Tests', Test.FAIL)):
+ elts = byCode.get(code)
+ if not elts:
+ continue
+ print '*'*20
+ print '%s (%d):' % (title, len(elts))
+ for t in elts:
+ print ' %s' % t.getFullName()
+ print
+
+ if opts.timeTests:
+ byTime = list(tests)
+ byTime.sort(key = lambda t: t.elapsed)
+ if byTime:
+ Util.printHistogram([(t.getFullName(), t.elapsed) for t in byTime],
+ title='Tests')
+
+ for name,code in (('Expected Passes ', Test.PASS),
+ ('Expected Failures ', Test.XFAIL),
+ ('Unsupported Tests ', Test.UNSUPPORTED),
+ ('Unresolved Tests ', Test.UNRESOLVED),
+ ('Unexpected Passes ', Test.XPASS),
+ ('Unexpected Failures', Test.FAIL),):
+ if opts.quiet and not code.isFailure:
+ continue
+ N = len(byCode.get(code,[]))
+ if N:
+ print ' %s: %d' % (name,N)
+
+ # If we encountered any additional errors, exit abnormally.
+ if litConfig.numErrors:
+ print >>sys.stderr, '\n%d error(s), exiting.' % litConfig.numErrors
+ sys.exit(2)
+
+ # Warn about warnings.
+ if litConfig.numWarnings:
+ print >>sys.stderr, '\n%d warning(s) in tests.' % litConfig.numWarnings
+
+ if hasFailures:
+ sys.exit(1)
+ sys.exit(0)
+
+if __name__=='__main__':
+ # Bump the GIL check interval, its more important to get any one thread to a
+ # blocking operation (hopefully exec) than to try and unblock other threads.
+ import sys
+ sys.setcheckinterval(1000)
+ main()
Propchange: llvm/trunk/utils/lit/lit.py
------------------------------------------------------------------------------
svn:executable = *
More information about the llvm-commits
mailing list