[llvm-commits] [zorg] r101740 - in /zorg/trunk/lnt: docs/contents.rst docs/tests.rst docs/tools.rst lnt/lnttool/__init__.py lnt/tests/ lnt/tests/__init__.py lnt/tests/builtintest.py lnt/tests/nt.py lnt/util/ServerUtil.py
Daniel Dunbar
daniel at zuster.org
Sun Apr 18 12:18:45 PDT 2010
Author: ddunbar
Date: Sun Apr 18 14:18:45 2010
New Revision: 101740
URL: http://llvm.org/viewvc/llvm-project?rev=101740&view=rev
Log:
LNT: Add 'lnt showtests' and 'lnt runtest' for running built-in test producers.
Added:
zorg/trunk/lnt/docs/tests.rst
zorg/trunk/lnt/lnt/tests/
zorg/trunk/lnt/lnt/tests/__init__.py
zorg/trunk/lnt/lnt/tests/builtintest.py
zorg/trunk/lnt/lnt/tests/nt.py
Modified:
zorg/trunk/lnt/docs/contents.rst
zorg/trunk/lnt/docs/tools.rst
zorg/trunk/lnt/lnt/lnttool/__init__.py
zorg/trunk/lnt/lnt/util/ServerUtil.py
Modified: zorg/trunk/lnt/docs/contents.rst
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/docs/contents.rst?rev=101740&r1=101739&r2=101740&view=diff
==============================================================================
--- zorg/trunk/lnt/docs/contents.rst (original)
+++ zorg/trunk/lnt/docs/contents.rst Sun Apr 18 14:18:45 2010
@@ -10,6 +10,8 @@
tools
+ tests
+
changes
todo
Added: zorg/trunk/lnt/docs/tests.rst
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/docs/tests.rst?rev=101740&view=auto
==============================================================================
--- zorg/trunk/lnt/docs/tests.rst (added)
+++ zorg/trunk/lnt/docs/tests.rst Sun Apr 18 14:18:45 2010
@@ -0,0 +1,62 @@
+.. _tests:
+
+Test Producers
+==============
+
+On the client-side, LNT comes with a number of built-in test data producers.
+This section focuses on the LLVM test-suite (aka nightly test) generator, since
+it is the primary test run using the LNT infrastructure, but note that LNT also
+includes tests for other interesting pieces of data, for example Clang
+compile-time performance.
+
+LNT also makes it easy to add new test data producers and includes examples of
+custom data importers (e.g., to import buildbot build information into) and
+dynamic test data generators (e.g., abusing the infrastructure to plot graphs,
+for example).
+
+Running a Local Server
+----------------------
+
+It is useful to set up a local LNT server to view the results of tests, either
+for personal use or to preview results before submitting them to a public
+server. To set up a one-off server for testing::
+
+ # Create a new installation in /tmp/FOO.
+ $ lnt create /tmp/FOO
+ created LNT configuration in '/tmp/FOO'
+ ...
+
+ # Run a local LNT server.
+ $ lnt runserver /tmp/FOO &> /tmp/FOO/runserver.log &
+ [2] 69694
+
+ # Watch the server log.
+ $ tail -f /tmp/runserver.log
+ * Running on http://localhost:8000/
+ ...
+
+Running Tests
+-------------
+
+The built-in tests are designed to be run via the ``lnt`` tool. The
+following tools for working with built-in tests are available:
+
+ ``lnt showtests``
+ List the available tests. Tests are defined with an extensible
+ architecture. FIXME: Point at docs on how to add a new test.
+
+ ``lnt runtest [<run options>] <test name> ... test arguments ...``
+ Run the named test. The run tool itself accepts a number of options which
+ are common to all tests. The most common option is ``--submit=<url>`` which
+ specifies the server to submit the results to after testing is complete. See
+ ``lnt runtest --help`` for more information on the available options.
+
+ The remainder of the options are passed to the test tool itself. The options
+ are specific to the test, but well behaved tests should respond to ``lnt
+ runtest <test name> --help``. The following section provides specific
+ documentation on the built-in tests.
+
+Built-in Tests
+--------------
+
+None yet.
Modified: zorg/trunk/lnt/docs/tools.rst
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/docs/tools.rst?rev=101740&r1=101739&r2=101740&view=diff
==============================================================================
--- zorg/trunk/lnt/docs/tools.rst (original)
+++ zorg/trunk/lnt/docs/tools.rst Sun Apr 18 14:18:45 2010
@@ -33,6 +33,13 @@
commit the data. When testing, you should verify that the server returns an
acceptable response before committing runs.
+ ``lnt showtests``
+ List available built-in tests. See the :ref:`tests` documentation for more
+ details on this tool.
+
+ ``lnt runtest [<run options>] <test name> ... test arguments ...``
+ Run a built-in test. See the :ref:`tests` documentation for more
+ details on this tool.
Server-Side Tools
-----------------
Modified: zorg/trunk/lnt/lnt/lnttool/__init__.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/lnttool/__init__.py?rev=101740&r1=101739&r2=101740&view=diff
==============================================================================
--- zorg/trunk/lnt/lnt/lnttool/__init__.py (original)
+++ zorg/trunk/lnt/lnt/lnttool/__init__.py Sun Apr 18 14:18:45 2010
@@ -95,6 +95,59 @@
data = formats.read_any(input, '<auto>')
PerfDB.importDataFromDict(db, data)
+def action_runtest(name, args):
+ from optparse import OptionParser, OptionGroup
+ parser = OptionParser("%%prog %s test-name [options]" % name)
+ parser.disable_interspersed_args()
+ parser.add_option("", "--submit", dest="submit_url", metavar="URL",
+ help=("autosubmit the test result to the given server "
+ "[%default]"),
+ type=str, default=None)
+ parser.add_option("", "--commit", dest="commit",
+ help=("whether the autosubmit result should be committed "
+ "[%default]"),
+ type=int, default=True)
+
+ (opts, args) = parser.parse_args(args)
+ if len(args) < 1:
+ parser.error("incorrect number of argments")
+
+ test_name,args = args[0],args[1:]
+
+ import lnt.tests
+ try:
+ test_instance = lnt.tests.get_test_instance(test_name)
+ except KeyError:
+ parser.error('invalid test name %r' % test_name)
+
+ report = test_instance.run_test('%s %s' % (name, test_name), args)
+
+ if opts.submit_url is not None:
+ if report is None:
+ raise SystemExit,"error: report generation failed"
+
+ from lnt.util import ServerUtil
+ io = StringIO.StringIO(report.render(indent=None))
+ ServerUtil.submitFile(opts.submit_url, io, True)
+
+def action_showtests(name, args):
+ """show the available built-in tests."""
+
+ from optparse import OptionParser, OptionGroup
+ parser = OptionParser("%%prog %s" % name)
+ (opts, args) = parser.parse_args(args)
+ if len(args) != 0:
+ parser.error("incorrect number of argments")
+
+ import lnt.tests
+
+ print 'Available tests:'
+ test_names = lnt.tests.get_test_names()
+ max_name = max(map(len, test_names))
+ for name in test_names:
+ print ' %-*s - %s' % (max_name, name,
+ lnt.tests.get_test_description(name))
+
def action_submit(name, args):
"""submit a test report to the server."""
Added: zorg/trunk/lnt/lnt/tests/__init__.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/tests/__init__.py?rev=101740&view=auto
==============================================================================
--- zorg/trunk/lnt/lnt/tests/__init__.py (added)
+++ zorg/trunk/lnt/lnt/tests/__init__.py Sun Apr 18 14:18:45 2010
@@ -0,0 +1,39 @@
+"""
+Access to built-in tests.
+"""
+
+# FIXME: There are better ways to do this, no doubt. We also would like this to
+# be extensible outside of the installation. Lookup how 'nose' handles this.
+
+known_tests = set(['nt'])
+
+def get_test_names():
+ """get_test_names() -> list
+
+ Return the list of known built-in test names.
+ """
+
+ return known_tests
+
+def get_test_instance(name):
+ """get_test_instance(name) -> lnt.test.BuiltinTest
+
+ Return an instance of the named test.
+ """
+
+ if name not in known_tests:
+ raise KeyError,name
+
+ module = getattr(__import__('lnt.tests.%s' % name, level=0).tests,
+ name)
+ return module.create_instance()
+
+def get_test_description(name):
+ """get_test_description(name) -> str
+
+ Return the description of the given test.
+ """
+
+ return get_test_instance(name).describe()
+
+__all__ = ['get_test_names', 'get_test_instance', 'get_test_description']
Added: zorg/trunk/lnt/lnt/tests/builtintest.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/tests/builtintest.py?rev=101740&view=auto
==============================================================================
--- zorg/trunk/lnt/lnt/tests/builtintest.py (added)
+++ zorg/trunk/lnt/lnt/tests/builtintest.py Sun Apr 18 14:18:45 2010
@@ -0,0 +1,21 @@
+"""
+Base class for builtin-in tests.
+"""
+
+class BuiltinTest(object):
+ def __init__(self):
+ pass
+
+ def describe(self):
+ """"describe() -> str
+
+ Return a short description of the test.
+ """
+
+ def run_test(self, name, args):
+ """run_test(name, args) -> lnt.testing.Report
+
+ Execute the test (accessed via name, for use in the usage message) with
+ the given command line args.
+ """
+ abstract
Added: zorg/trunk/lnt/lnt/tests/nt.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/tests/nt.py?rev=101740&view=auto
==============================================================================
--- zorg/trunk/lnt/lnt/tests/nt.py (added)
+++ zorg/trunk/lnt/lnt/tests/nt.py Sun Apr 18 14:18:45 2010
@@ -0,0 +1,13 @@
+import builtintest
+
+class NTTest(builtintest.BuiltinTest):
+ def describe(self):
+ return 'LLVM test-suite compile and execution tests'
+
+ def run_test(self, name, args):
+ raise NotImplementedError
+
+def create_instance():
+ return NTTest()
+
+__all__ = ['create_instance']
Modified: zorg/trunk/lnt/lnt/util/ServerUtil.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/util/ServerUtil.py?rev=101740&r1=101739&r2=101740&view=diff
==============================================================================
--- zorg/trunk/lnt/lnt/util/ServerUtil.py (original)
+++ zorg/trunk/lnt/lnt/util/ServerUtil.py Sun Apr 18 14:18:45 2010
@@ -6,15 +6,18 @@
import urllib
import urllib2
+def submitFile(url, file, commit):
+ values = { 'input_data' : file.read(),
+ 'commit' : ("0","1")[not not commit] }
+
+ data = urllib.urlencode(values)
+ response = urllib2.urlopen(urllib2.Request(url, data))
+ the_page = response.read()
+
+ print the_page
+
def submitFiles(url, files, commit):
for file in files:
f = open(file, 'rb')
- values = { 'input_data' : f.read(),
- 'commit' : ("0","1")[not not commit] }
+ submitFile(url, f, commit)
f.close()
-
- data = urllib.urlencode(values)
- response = urllib2.urlopen(urllib2.Request(url, data))
- the_page = response.read()
-
- print the_page
More information about the llvm-commits
mailing list