[test-suite] r297570 - Improve documentation for lit benchmark runner
Matthias Braun via llvm-commits
llvm-commits at lists.llvm.org
Sat Mar 11 15:54:00 PST 2017
Author: matze
Date: Sat Mar 11 17:54:00 2017
New Revision: 297570
URL: http://llvm.org/viewvc/llvm-project?rev=297570&view=rev
Log:
Improve documentation for lit benchmark runner
Added:
test-suite/trunk/litsupport/README.md
Modified:
test-suite/trunk/litsupport/codesize.py
test-suite/trunk/litsupport/compiletime.py
test-suite/trunk/litsupport/hash.py
test-suite/trunk/litsupport/perf.py
test-suite/trunk/litsupport/profilegen.py
test-suite/trunk/litsupport/remote.py
test-suite/trunk/litsupport/run.py
test-suite/trunk/litsupport/run_under.py
test-suite/trunk/litsupport/shellcommand.py
test-suite/trunk/litsupport/test.py
test-suite/trunk/litsupport/testfile.py
Added: test-suite/trunk/litsupport/README.md
URL: http://llvm.org/viewvc/llvm-project/test-suite/trunk/litsupport/README.md?rev=297570&view=auto
==============================================================================
--- test-suite/trunk/litsupport/README.md (added)
+++ test-suite/trunk/litsupport/README.md Sat Mar 11 17:54:00 2017
@@ -0,0 +1,45 @@
+Introduction
+============
+
+This is the benchmark runner for llvm test-suite. It is only available when the
+test-suite was built with cmake. It runs benchmarks, check the results and
+collect various metrics such as runtime, compiletime or code size.
+
+The runner is implemented as a custom test format for the llvm-lit tool.
+
+.test Files
+===========
+
+.test files specify how to run a benchmark and check the results.
+
+Each line in a .test file may specify a PREPARE:, RUN: or VERIFY: command as a
+shell command. Each command kind can be specified multiple times to execute
+multiple commands. A benchmark run will first execute all prepare commands,
+then all run commands and finally all verify commands. Metrics like runtime or
+profile data is collected during the RUN: commands.
+
+The lit runner accepts the most common posix shell commands like assigning
+environment variables, redirecting input and outputs and changing the directory
+before starting a benchmark (see shellcommand.py for details).
+
+Example:
+
+ RUN: ./mybenchmark --size 500 --verbose > run0.txt
+ VERIFY: diff reference_results0.txt run0.txt
+ RUN: ./mybenchmark --size 300 --seed 5555 --verbose > run1.txt
+ VERIFY: diff reference_results1.txt run1.txt
+
+Usage
+=====
+
+Running cmake on the test-suite creates a `lit.site.cfg` and .test files for each bechmark. llvm-lit can then be used as usual. Some examples:
+
+ # Run all benchmark in the current directory an subdirecories one at a time.
+ $ llvm-lit -j1 .
+
+ # Run a single benchmark and show executed commands and collected metrics.
+ $ llvm-lit -a SingleSource/Benchmarks/Misc/pi.test
+
+ # Run benchmarks with reduced lit output but save collected metrics to json
+ # file. This format is used by LNT or viewable by test-suite/utils/compare.py.
+ $ llvm-lit . -o result.json -s
Modified: test-suite/trunk/litsupport/codesize.py
URL: http://llvm.org/viewvc/llvm-project/test-suite/trunk/litsupport/codesize.py?rev=297570&r1=297569&r2=297570&view=diff
==============================================================================
--- test-suite/trunk/litsupport/codesize.py (original)
+++ test-suite/trunk/litsupport/codesize.py Sat Mar 11 17:54:00 2017
@@ -1,3 +1,4 @@
+"""Test module to collect code size metrics of the benchmark executable."""
from litsupport import testplan
import lit.Test
import logging
Modified: test-suite/trunk/litsupport/compiletime.py
URL: http://llvm.org/viewvc/llvm-project/test-suite/trunk/litsupport/compiletime.py?rev=297570&r1=297569&r2=297570&view=diff
==============================================================================
--- test-suite/trunk/litsupport/compiletime.py (original)
+++ test-suite/trunk/litsupport/compiletime.py Sat Mar 11 17:54:00 2017
@@ -1,3 +1,5 @@
+"""Test module to collect compile time metrics. This just finds and summarizes
+the *.time files generated by the build."""
import lit.Test
from litsupport import timeit
import os
Modified: test-suite/trunk/litsupport/hash.py
URL: http://llvm.org/viewvc/llvm-project/test-suite/trunk/litsupport/hash.py?rev=297570&r1=297569&r2=297570&view=diff
==============================================================================
--- test-suite/trunk/litsupport/hash.py (original)
+++ test-suite/trunk/litsupport/hash.py Sat Mar 11 17:54:00 2017
@@ -1,3 +1,4 @@
+"""Test module to collect test executable hashsum."""
from lit.Test import toMetricValue
from litsupport import shellcommand
from litsupport import testplan
Modified: test-suite/trunk/litsupport/perf.py
URL: http://llvm.org/viewvc/llvm-project/test-suite/trunk/litsupport/perf.py?rev=297570&r1=297569&r2=297570&view=diff
==============================================================================
--- test-suite/trunk/litsupport/perf.py (original)
+++ test-suite/trunk/litsupport/perf.py Sat Mar 11 17:54:00 2017
@@ -1,3 +1,5 @@
+"""Test Module to perform an extra execution of the benchmark in the linux
+perf tool."""
from litsupport import shellcommand
from litsupport import testplan
from litsupport import run_under
Modified: test-suite/trunk/litsupport/profilegen.py
URL: http://llvm.org/viewvc/llvm-project/test-suite/trunk/litsupport/profilegen.py?rev=297570&r1=297569&r2=297570&view=diff
==============================================================================
--- test-suite/trunk/litsupport/profilegen.py (original)
+++ test-suite/trunk/litsupport/profilegen.py Sat Mar 11 17:54:00 2017
@@ -1,3 +1,4 @@
+"""Test module that runs llvm-profdata merge after executing the benchmark."""
from litsupport import shellcommand
from litsupport import testplan
Modified: test-suite/trunk/litsupport/remote.py
URL: http://llvm.org/viewvc/llvm-project/test-suite/trunk/litsupport/remote.py?rev=297570&r1=297569&r2=297570&view=diff
==============================================================================
--- test-suite/trunk/litsupport/remote.py (original)
+++ test-suite/trunk/litsupport/remote.py Sat Mar 11 17:54:00 2017
@@ -1,3 +1,6 @@
+"""Test module to execute a benchmark through ssh on a remote device.
+This assumes all relevant directories and files are present on the remote
+device (typically shared by NFS)."""
from litsupport import testplan
import logging
Modified: test-suite/trunk/litsupport/run.py
URL: http://llvm.org/viewvc/llvm-project/test-suite/trunk/litsupport/run.py?rev=297570&r1=297569&r2=297570&view=diff
==============================================================================
--- test-suite/trunk/litsupport/run.py (original)
+++ test-suite/trunk/litsupport/run.py Sat Mar 11 17:54:00 2017
@@ -1,6 +1,9 @@
+"""Test module to just run the benchmark. Without this module the benchmark is
+not executed. This may be interesting when just collecting compile time and
+code size."""
+
+
def mutatePlan(context, plan):
- """The most basic test module: Execute the RUN:, VERIFY: and METRIC:
- scripts"""
plan.preparescript = context.parsed_preparescript
plan.runscript = context.parsed_runscript
plan.verifyscript = context.parsed_verifyscript
Modified: test-suite/trunk/litsupport/run_under.py
URL: http://llvm.org/viewvc/llvm-project/test-suite/trunk/litsupport/run_under.py?rev=297570&r1=297569&r2=297570&view=diff
==============================================================================
--- test-suite/trunk/litsupport/run_under.py (original)
+++ test-suite/trunk/litsupport/run_under.py Sat Mar 11 17:54:00 2017
@@ -1,3 +1,5 @@
+"""Test module to run benchmarks in a wrapper application. This is typically
+used to prefix the benchmark command with simulator commands."""
from litsupport import shellcommand
from litsupport import testplan
Modified: test-suite/trunk/litsupport/shellcommand.py
URL: http://llvm.org/viewvc/llvm-project/test-suite/trunk/litsupport/shellcommand.py?rev=297570&r1=297569&r2=297570&view=diff
==============================================================================
--- test-suite/trunk/litsupport/shellcommand.py (original)
+++ test-suite/trunk/litsupport/shellcommand.py Sat Mar 11 17:54:00 2017
@@ -1,3 +1,4 @@
+"""Utilities for analyzing and manipulating shell commands."""
import shlex
import logging
import re
@@ -8,10 +9,10 @@ except:
class ShellCommand:
- '''This class represents a parsed shell command (a subset, we do not support
- arbitrary shell commands yet). The parsed form allows direct access to
- information like the executable name, working directory or the file
- to which stdin/stdout/stderr is redirected.'''
+ """This class represents a parsed shell command (a subset of posix shell
+ commands; see parse()). The parsed form allows direct access to information
+ like the executable name, working directory or the file to which
+ stdin/stdout/stderr is redirected."""
def __init__(self, executable=None, arguments=None):
self.stdin = None
self.stdout = None
@@ -24,6 +25,7 @@ class ShellCommand:
self.envvars = {}
def toCommandline(self):
+ """Transforms ShellCommand object to a posix shell commandline."""
result = ""
if self.workdir is not None:
@@ -45,6 +47,9 @@ class ShellCommand:
return result
def wrap(self, new_executable, args):
+ """Adds a prefix to the exeutable. Example:
+ Prefixing `SOMVAR=42 cd mydir && mycmd -v > /dev/null` with `lldb --`
+ becomes `SOMEVAR=42 cd mydir && lldb -- mycmd -v > /dev/null`."""
self.arguments = args + [self.executable] + self.arguments
self.executable = new_executable
@@ -60,6 +65,11 @@ unhandled_tokens = set([';;', '<<', '>>'
def parse(commandline):
+ """Parses a posix shell commandline to a ShellCommand object. This supports
+ typical commandline with environment variables, input/output redirection
+ and switching directories upfront. It does not support full posix shell
+ and will throw an exception if the commandline uses unsupported features.
+ """
previous_commands = []
result = ShellCommand()
tokens = shlex.split(commandline)
@@ -112,12 +122,18 @@ def parse(commandline):
# Some executables are just used to cleanup/prepare for a test run, ignore them
-# here.
+# here. This is from a time when there was no concept of a prepare script,
+# it should not be necessary anymore for new test files.
_ignore_executables = set(['cd', 'rm', 'cp'])
def getMainExecutable(context):
- """Collect md5sum of tested executable"""
+ """Returns the main executable of the current run script. This skips over
+ some commands typically used to setup a benchmark (see _ignore_executables)
+ and returns the first executable found in the run script and prints a
+ warning if more than one executable is found."""
+
+ # Executable name already determined in a previous run?
if hasattr(context, 'executable'):
return context.executable
Modified: test-suite/trunk/litsupport/test.py
URL: http://llvm.org/viewvc/llvm-project/test-suite/trunk/litsupport/test.py?rev=297570&r1=297569&r2=297570&view=diff
==============================================================================
--- test-suite/trunk/litsupport/test.py (original)
+++ test-suite/trunk/litsupport/test.py Sat Mar 11 17:54:00 2017
@@ -1,3 +1,7 @@
+"""
+Main integration for llvm-lit: This defines a lit test format.
+Also contains logic to load benchmark modules.
+"""
import importlib
import lit
import lit.util
Modified: test-suite/trunk/litsupport/testfile.py
URL: http://llvm.org/viewvc/llvm-project/test-suite/trunk/litsupport/testfile.py?rev=297570&r1=297569&r2=297570&view=diff
==============================================================================
--- test-suite/trunk/litsupport/testfile.py (original)
+++ test-suite/trunk/litsupport/testfile.py Sat Mar 11 17:54:00 2017
@@ -1,6 +1,4 @@
-"""
-Parse a .test file
-"""
+"""Parser for .test files"""
from lit.TestRunner import parseIntegratedTestScriptCommands, \
getDefaultSubstitutions, applySubstitutions
from litsupport import shellcommand
More information about the llvm-commits
mailing list