[llvm-commits] [zorg] r99164 - in /zorg/trunk/lnt: lnt/formats/ lnt/import/ lnt/lnttool/ tests/Formats/ tests/Formats/Inputs/
Daniel Dunbar
daniel at zuster.org
Mon Mar 22 00:18:29 PDT 2010
Author: ddunbar
Date: Mon Mar 22 02:18:29 2010
New Revision: 99164
URL: http://llvm.org/viewvc/llvm-project?rev=99164&view=rev
Log:
LNT: Reimplement format handling.
- Transparently convert between formats.
- Change NTAuxSubmit to submit the data directly; this lets it be free-standing which is useful for automating submission scripts (one can just 'svn export' it). The server can handle the conversion.
- Add support for JSON files.
Added:
zorg/trunk/lnt/lnt/formats/
zorg/trunk/lnt/lnt/formats/AppleOpenSSLReader.py
- copied, changed from r99163, zorg/trunk/lnt/lnt/import/AppleOpenSSLReader.py
zorg/trunk/lnt/lnt/formats/JSONFormat.py
zorg/trunk/lnt/lnt/formats/NightlytestReader.py
- copied, changed from r99163, zorg/trunk/lnt/lnt/import/NightlytestReader.py
zorg/trunk/lnt/lnt/formats/PlistFormat.py
zorg/trunk/lnt/lnt/formats/__init__.py
zorg/trunk/lnt/lnt/lnttool/convert.py
zorg/trunk/lnt/tests/Formats/
zorg/trunk/lnt/tests/Formats/Inputs/
zorg/trunk/lnt/tests/Formats/Inputs/test.json
zorg/trunk/lnt/tests/Formats/Inputs/test.nightlytest
zorg/trunk/lnt/tests/Formats/Inputs/test.plist
zorg/trunk/lnt/tests/Formats/json.py
zorg/trunk/lnt/tests/Formats/nightlytest.py
zorg/trunk/lnt/tests/Formats/plist.py
Removed:
zorg/trunk/lnt/lnt/import/AppleOpenSSLReader.py
zorg/trunk/lnt/lnt/import/ImportXCBTimes
zorg/trunk/lnt/lnt/import/NightlytestReader.py
Modified:
zorg/trunk/lnt/lnt/import/ImportData
zorg/trunk/lnt/lnt/import/NTAuxSubmit
zorg/trunk/lnt/lnt/lnttool/__init__.py
Copied: zorg/trunk/lnt/lnt/formats/AppleOpenSSLReader.py (from r99163, zorg/trunk/lnt/lnt/import/AppleOpenSSLReader.py)
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/formats/AppleOpenSSLReader.py?p2=zorg/trunk/lnt/lnt/formats/AppleOpenSSLReader.py&p1=zorg/trunk/lnt/lnt/import/AppleOpenSSLReader.py&r1=99163&r2=99164&rev=99164&view=diff
==============================================================================
--- zorg/trunk/lnt/lnt/import/AppleOpenSSLReader.py (original)
+++ zorg/trunk/lnt/lnt/formats/AppleOpenSSLReader.py Mon Mar 22 02:18:29 2010
@@ -1,3 +1,7 @@
+"""
+Converter for a custom format with the output of OpenSSL test runs.
+"""
+
import os
def parseOpenSSLFile(path):
@@ -23,7 +27,17 @@
return data
-def loadData(path):
+def _matches_format(path_or_file):
+ # If this is a file, we definitely can't load it.
+ if not isinstance(path_or_file,str):
+ return False
+
+ # Assume an input matches this format if any of the key files exists.
+ return (os.path.exists(os.path.join(path_or_file, 'svn-revision')) or
+ os.path.exists(os.path.join(path_or_file, 'start.timestamp')) or
+ os.path.exists(os.path.join(path_or_file, 'finished.timestamp')))
+
+def _load_data(path):
# Look for svn-revision and timestamps.
llvmRevision = ''
@@ -92,23 +106,6 @@
'Tests' : tests,
'Group Info' : groupInfo }
-def main():
- import plistlib
- import sys
-
- global opts
- from optparse import OptionParser
- parser = OptionParser("usage: %prog raw-data-path output")
- opts,args = parser.parse_args()
-
- if len(args) != 2:
- parser.error("incorrect number of argments")
-
- file,output = args
-
- data = loadData(file)
-
- plistlib.writePlist(data, output)
-
-if __name__=='__main__':
- main()
+format = { 'name' : 'apple_openssl',
+ 'predicate' : _matches_format,
+ 'read' : _load_data }
Added: zorg/trunk/lnt/lnt/formats/JSONFormat.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/formats/JSONFormat.py?rev=99164&view=auto
==============================================================================
--- zorg/trunk/lnt/lnt/formats/JSONFormat.py (added)
+++ zorg/trunk/lnt/lnt/formats/JSONFormat.py Mon Mar 22 02:18:29 2010
@@ -0,0 +1,22 @@
+import json
+
+def _matches_format(path_or_file):
+ if isinstance(path_or_file, str):
+ path_or_file = open(path_or_file)
+
+ try:
+ json.load(path_or_file)
+ return True
+ except:
+ return False
+
+def _load_format(path_or_file):
+ if isinstance(path_or_file, str):
+ path_or_file = open(path_or_file)
+
+ return json.load(path_or_file)
+
+format = { 'name' : 'json',
+ 'predicate' : _matches_format,
+ 'read' : _load_format,
+ 'write' : json.dump }
Copied: zorg/trunk/lnt/lnt/formats/NightlytestReader.py (from r99163, zorg/trunk/lnt/lnt/import/NightlytestReader.py)
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/formats/NightlytestReader.py?p2=zorg/trunk/lnt/lnt/formats/NightlytestReader.py&p1=zorg/trunk/lnt/lnt/import/NightlytestReader.py&r1=99163&r2=99164&rev=99164&view=diff
==============================================================================
--- zorg/trunk/lnt/lnt/import/NightlytestReader.py (original)
+++ zorg/trunk/lnt/lnt/formats/NightlytestReader.py Mon Mar 22 02:18:29 2010
@@ -1,17 +1,26 @@
-#!/usr/bin/env python
-
"""
Data converter from the llvm/utils/NewNightlyTest.pl report file format
(*-sentdata.txt) to the LNT plist format.
"""
-# FIXME: Refactor data conversion code.
-
import re
kDataKeyStart = re.compile('(.*) =>(.*)')
-def loadSentData(path):
+def _matches_format(path_or_file):
+ if isinstance(path_or_file, str):
+ path_or_file = open(path_or_file)
+
+ # Assume this is in nightlytes format if the first line matches the
+ # key-value format.
+ for ln in path_or_file:
+ m = kDataKeyStart.match(ln)
+ if m:
+ return True
+ return False
+
+
+def _load_data(path_or_file):
def parseDGResults(text):
results = {}
if 'Dejagnu skipped by user choice' in text:
@@ -22,13 +31,17 @@
results[result].append(value)
return results
+ if isinstance(path_or_file, str):
+ path_or_file = open(path_or_file)
+
basename = 'nightlytest'
# Guess the format (server side or client side) based on the first
# character.
- isServerSide = (open(path).read(1) == '\'')
+ f = path_or_file
+ isServerSide = (f.read(1) == '\'')
+ f.seek(0)
- f = open(path)
data = {}
current = None
@@ -224,26 +237,6 @@
'Tests' : tests,
'Group Info' : groupInfo }
-def convertNTData(inputPath, outputPath):
- """convertNTData - Convert a nightlytest "sentdata.txt" file into a zorg
- plist file."""
- import plistlib
-
- data = loadSentData(inputPath)
- plistlib.writePlist(data, outputPath)
-
-def main():
- global opts
- from optparse import OptionParser
- parser = OptionParser("usage: %prog file output")
- opts,args = parser.parse_args()
-
- if len(args) != 2:
- parser.error("incorrect number of argments")
-
- file,output = args
-
- convertNTData(file, output)
-
-if __name__=='__main__':
- main()
+format = { 'name' : 'nightlytest',
+ 'predicate' : _matches_format,
+ 'read' : _load_data }
Added: zorg/trunk/lnt/lnt/formats/PlistFormat.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/formats/PlistFormat.py?rev=99164&view=auto
==============================================================================
--- zorg/trunk/lnt/lnt/formats/PlistFormat.py (added)
+++ zorg/trunk/lnt/lnt/formats/PlistFormat.py Mon Mar 22 02:18:29 2010
@@ -0,0 +1,13 @@
+import plistlib
+
+def _matches_format(path_or_file):
+ try:
+ plistlib.readPlist(path_or_file)
+ return True
+ except:
+ return False
+
+format = { 'name' : 'plist',
+ 'predicate' : _matches_format,
+ 'read' : plistlib.readPlist,
+ 'write' : plistlib.writePlist }
Added: zorg/trunk/lnt/lnt/formats/__init__.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/formats/__init__.py?rev=99164&view=auto
==============================================================================
--- zorg/trunk/lnt/lnt/formats/__init__.py (added)
+++ zorg/trunk/lnt/lnt/formats/__init__.py Mon Mar 22 02:18:29 2010
@@ -0,0 +1,67 @@
+"""
+Utilities for converting to LNT's test format.
+"""
+
+from AppleOpenSSLReader import format as apple_openssl
+from NightlytestReader import format as nightlytest
+from PlistFormat import format as plist
+from JSONFormat import format as json
+
+# FIXME: Lazy loading would be nice.
+formats = [plist, json, nightlytest, apple_openssl]
+formats_by_name = dict((f['name'], f) for f in formats)
+format_names = formats_by_name.keys()
+
+def get_format(name):
+ return formats_by_name.get(name)
+
+def guess_format(path_or_file):
+ """guess_format(path_or_file) -> [format]
+
+ Guess which format should be used to load the given file and return it, if
+ found.
+ """
+
+ # Check that files are seekable.
+ is_file = False
+ if not isinstance(path_or_file, str):
+ is_file = True
+ path_or_file.seek(0)
+
+ matches = None
+ for f in formats:
+ # Check if the path matches this format, ignoring exceptions.
+ try:
+ try:
+ if not f['predicate'](path_or_file):
+ continue
+ except:
+ continue
+ finally:
+ if is_file:
+ # Reset seek.
+ path_or_file.seek(0)
+
+ # Reject anything which matches multiple formats.
+ if matches:
+ return None
+
+ matches = f
+
+ return matches
+
+def read_any(path_or_file, format_name):
+ # Figure out the input format.
+ if format_name == '<auto>':
+ f = guess_format(path_or_file)
+ if f is None:
+ if isinstance(path_or_file, str):
+ raise SystemExit("unable to guess input format for %r" % input)
+ else:
+ raise SystemExit("unable to guess input format for file")
+ else:
+ f = get_format(format_name)
+ if f is None or not f.get('read'):
+ raise SystemExit("unknown input format: %r" % inFormat)
+
+ return f['read'](path_or_file)
Removed: zorg/trunk/lnt/lnt/import/AppleOpenSSLReader.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/import/AppleOpenSSLReader.py?rev=99163&view=auto
==============================================================================
--- zorg/trunk/lnt/lnt/import/AppleOpenSSLReader.py (original)
+++ zorg/trunk/lnt/lnt/import/AppleOpenSSLReader.py (removed)
@@ -1,114 +0,0 @@
-import os
-
-def parseOpenSSLFile(path):
- data = open(path).read()
- lines = list(open(path))
- lnfields = [ln.strip().split(':') for ln in lines]
- assert(lnfields[0][0] == '+H')
- header = lnfields[0]
- blockSizes = map(int, header[1:])
-
- # Cipher -> [(Block Size,Value)*]
- data = {}
- for fields in lnfields[1:]:
- # Ignore other fields
- if fields[0] != '+F':
- continue
-
- name = fields[2]
- countsPerBlock = fields[3:]
- assert len(countsPerBlock) == len(blockSizes)
- data[name] = [(b,float(c))
- for b,c in zip(blockSizes,countsPerBlock)]
-
- return data
-
-def loadData(path):
- # Look for svn-revision and timestamps.
-
- llvmRevision = ''
- startTime = endTime = ''
-
- f = os.path.join(path, 'svn-revision')
- if os.path.exists(f):
- svnRevisionData = open(f).read()
- assert(svnRevisionData[0] == 'r')
- llvmRevision = int(svnRevisionData[1:])
-
- f = os.path.join(path, 'start.timestamp')
- if os.path.exists(f):
- startTime = open(f).read().strip()
-
- f = os.path.join(path, 'finished.timestamp')
- if os.path.exists(f):
- endTime = open(f).read().strip()
-
- # Look for sub directories
- openSSLData = []
- for file in os.listdir(path):
- p = os.path.join(path, file)
- if os.path.isdir(p):
- # Look for Tests/Apple.OpenSSL.64/speed.txt
- p = os.path.join(p, 'Tests/Apple.OpenSSL.64/speed.txt')
- if os.path.exists(p):
- openSSLData.append((file, parseOpenSSLFile(p)))
-
- basename = 'apple_openssl'
-
- machine = { 'Name' : 'dgohman.apple.com',
- 'Info' : { } }
-
- run = { 'Start Time' : startTime,
- 'End Time' : endTime,
- 'Info' : { 'llvm-revision' : llvmRevision,
- 'tag' : 'apple_openssl' } }
-
- tests = []
- groupInfo = []
-
- for dirName,dirData in openSSLData:
- # Demangle compiler & flags
- if dirName.startswith('gcc'):
- compiler = 'gcc'
- elif dirName.startswith('llvm-gcc'):
- compiler = 'llvm-gcc'
- else:
- raise ValueError,compiler
- assert dirName[len(compiler)] == '-'
- flags = dirName[len(compiler)+1:]
-
- for cipher,values in dirData.items():
- testName = basename + '.' + cipher + '.ips'
- for block,value in values:
- parameters = { 'blockSize' : block,
- 'compiler' : compiler,
- 'compiler_flags' : flags }
- tests.append( { 'Name' : testName,
- 'Info' : parameters,
- 'Data' : [value] } )
-
- return { 'Machine' : machine,
- 'Run' : run,
- 'Tests' : tests,
- 'Group Info' : groupInfo }
-
-def main():
- import plistlib
- import sys
-
- global opts
- from optparse import OptionParser
- parser = OptionParser("usage: %prog raw-data-path output")
- opts,args = parser.parse_args()
-
- if len(args) != 2:
- parser.error("incorrect number of argments")
-
- file,output = args
-
- data = loadData(file)
-
- plistlib.writePlist(data, output)
-
-if __name__=='__main__':
- main()
Modified: zorg/trunk/lnt/lnt/import/ImportData
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/import/ImportData?rev=99164&r1=99163&r2=99164&view=diff
==============================================================================
--- zorg/trunk/lnt/lnt/import/ImportData (original)
+++ zorg/trunk/lnt/lnt/import/ImportData Mon Mar 22 02:18:29 2010
@@ -11,10 +11,8 @@
import sys
import time
-from lnt import viewer
+from lnt import formats, viewer
from lnt.viewer import Util, PerfDB
-import NightlytestReader
-import AppleOpenSSLReader
def main():
global opts
@@ -31,8 +29,8 @@
parser.add_option("", "--email-to", dest="emailReportTo", type=str,
default=None)
parser.add_option("", "--format", dest="format",
- choices=('plist','nightlytest','apple_openssl'),
- default='plist')
+ choices=formats.format_names + ['<auto>'],
+ default='<auto>')
parser.add_option("", "--commit", dest="commit", type=int,
default=True)
parser.add_option("", "--show-sql", dest="showSQL", action="store_true",
@@ -62,13 +60,9 @@
print 'TOTAL IMPORT TIME: %.2fs' % (time.time() - startTime,)
def importFiles(db, files):
- importer = { 'plist' : plistlib.readPlist,
- 'nightlytest' : NightlytestReader.loadSentData,
- 'apple_openssl' : AppleOpenSSLReader.loadData }[opts.format]
-
def consumer(file):
try:
- return importer(file)
+ return formats.read_any(file, opts.format)
except KeyboardInterrupt:
raise
except:
Removed: zorg/trunk/lnt/lnt/import/ImportXCBTimes
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/import/ImportXCBTimes?rev=99163&view=auto
==============================================================================
--- zorg/trunk/lnt/lnt/import/ImportXCBTimes (original)
+++ zorg/trunk/lnt/lnt/import/ImportXCBTimes (removed)
@@ -1,106 +0,0 @@
-#!/usr/bin/env python
-
-import os
-import time
-import sys
-
-from lnt import viewer
-from lnt.viewer import Util, PerfDB
-
-def main():
- global opts
- from optparse import OptionParser
- parser = OptionParser("usage: %prog file dbpath")
- parser.add_option("", "--test-prefix",
- action="store", dest="testPrefix", default=None)
- opts,args = parser.parse_args()
-
- if len(args) != 2:
- parser.error("incorrect number of argments")
- if not opts.testPrefix:
- parser.error("must specify test prefix")
-
- dbpath,file = args
-
- globals = {}
- exec open(file) in globals, globals
-
- db = PerfDB.PerfDB(dbpath)
-
- numMachines = db.getNumMachines()
- numTests = db.getNumTests()
- numSamples = db.getNumSamples()
-
- # Hardcode some things that aren't in the file
- machine = PerfDB.Machine(-1,
- name = 'lordcrumb.apple.com',
- arch = 'Intel i386',
- os = 'SnowLeopard',
- hwconfig = '<unknown>',
- compiler = '<unknown>')
- machine = db.getOrCreateMachine(machine)
- print "MACHINE: %r" % machine
-
- # Treat as a single "run"; our DB format has no way to lock
- # individual samples together. I recall now that this was the
- # motivation in treating a sample as a group of numbers, not just
- # one.
-
- # Rough guess.
- mtime = os.stat(file).st_mtime
- timestamp = time.strftime('%Y-%m-%dT%H:%M:%Sz', time.localtime(mtime))
-
- # FIXME: Need to extract revision. :(
- run = db.createRun(machine, PerfDB.Run(-1, -1, timestamp=timestamp, svnRevision=None))
- print "RUN: %r" % run
-
- ####
-
- runs = globals.get('runs')
- for keys,data in runs:
- # Mangle a test name
- testName = '%s:threads=%s:pch=%s:mode=%s' % (opts.testPrefix,
- keys.get('threads'),
- int(keys.get('pch') == 'pch'),
- keys.get('script'))
- compiler = keys.get('cc')
- compiler = compiler.replace('clang_driver','clang')
- compiler = compiler.replace('_','/')
- compiler = compiler.replace('xcc','ccc')
- compilerOpts = '-O0,-g'
-
- userTest = db.getOrCreateTest(PerfDB.Test(-1,
- name = testName,
- subtest = 'user',
- kindID = None,
- groupID = None,
- compiler = compiler,
- compilerOpts = compilerOpts))
- sysTest = db.getOrCreateTest(PerfDB.Test(-1,
- name = testName,
- subtest = 'sys',
- kindID = None,
- groupID = None,
- compiler = compiler,
- compilerOpts = compilerOpts))
- wallTest = db.getOrCreateTest(PerfDB.Test(-1,
- name = testName,
- subtest = 'wall',
- kindID = None,
- groupID = None,
- compiler = compiler,
- compilerOpts = compilerOpts))
- assert data['version'] == 0
- for (mem,user,sys,wall) in data['samples']:
- db.addSample(userTest, run, PerfDB.Sample(-1, -1, -1, '', user))
- db.addSample(sysTest, run, PerfDB.Sample(-1, -1, -1, '', sys))
- db.addSample(wallTest, run, PerfDB.Sample(-1, -1, -1, '', wall))
-
- db.commit()
-
- print "ADDED: %d machines, %d tests, and %d samples." % (db.getNumMachines() - numMachines,
- db.getNumTests() - numTests,
- db.getNumSamples() - numSamples)
-
-if __name__ == '__main__':
- main()
Modified: zorg/trunk/lnt/lnt/import/NTAuxSubmit
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/import/NTAuxSubmit?rev=99164&r1=99163&r2=99164&view=diff
==============================================================================
--- zorg/trunk/lnt/lnt/import/NTAuxSubmit (original)
+++ zorg/trunk/lnt/lnt/import/NTAuxSubmit Mon Mar 22 02:18:29 2010
@@ -4,15 +4,17 @@
Command line tool for submitting to a LNT server.
This script is explicitly designed to work with llvm/utils/NewNightlyTest.pl's
--submit-aux argument. It simply converts the data and submits it to the llvm.org
-nightly test server.
-"""
+-submit-aux argument. It simply uploads the data to the server, and assumes the
+server will sort out any input conversion issues.
-# FIXME: Roll into lnttool and kill.
+This script should be totally free-standing; it is designed so users can svn
+export it directly from the LLVM repository as part of their submission scripts.
+"""
+import plistlib
import sys
-import NightlytestReader
-import ServerUtil
+import urllib
+import urllib2
def main():
global opts
@@ -20,7 +22,6 @@
parser = OptionParser("usage: %prog {nightlytest sentdata.txt}*")
parser.add_option("", "--commit", dest="commit", type=int,
default=True)
- parser.add_option("", "--no-convert", dest="noConvert")
# FIXME: It would be nice to support an easy mechanism for localized
# instances of the LNT infrastructure to default to the correct server for
@@ -36,17 +37,20 @@
for inputFile in args:
print '%s: note: submitting %s' % (sys.argv[0], inputFile)
- if opts.noConvert:
- plistPath = inputFile
- else:
- # Convert to the zorg format.
- #
- # FIXME: Avoid temp file.
- plistPath = "/tmp/t.plist"
- NightlytestReader.convertNTData(inputFile, plistPath)
+ # Encode the form data.
+ f = open(inputFile, 'rb')
+ values = { 'input_data' : f.read(),
+ 'commit' : ("0","1")[not not commit] }
+ f.close()
+ data = urllib.urlencode(values)
# Send it off.
- ServerUtil.submitFiles(opts.serverUrl, [plistPath], opts.commit)
+ response = urllib2.urlopen(urllib2.Request(url, data))
+ the_page = response.read()
+
+ # FIXME: Parse results and return proper error code.
+ print the_page
+
if __name__ == '__main__':
main()
Removed: zorg/trunk/lnt/lnt/import/NightlytestReader.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/import/NightlytestReader.py?rev=99163&view=auto
==============================================================================
--- zorg/trunk/lnt/lnt/import/NightlytestReader.py (original)
+++ zorg/trunk/lnt/lnt/import/NightlytestReader.py (removed)
@@ -1,249 +0,0 @@
-#!/usr/bin/env python
-
-"""
-Data converter from the llvm/utils/NewNightlyTest.pl report file format
-(*-sentdata.txt) to the LNT plist format.
-"""
-
-# FIXME: Refactor data conversion code.
-
-import re
-
-kDataKeyStart = re.compile('(.*) =>(.*)')
-
-def loadSentData(path):
- def parseDGResults(text):
- results = {}
- if 'Dejagnu skipped by user choice' in text:
- return results
- for ln in text.strip().split('\n'):
- result,value = ln.split(':',1)
- results[result] = results.get(result,[])
- results[result].append(value)
- return results
-
- basename = 'nightlytest'
-
- # Guess the format (server side or client side) based on the first
- # character.
- isServerSide = (open(path).read(1) == '\'')
-
- f = open(path)
- data = {}
-
- current = None
- inData = False
- for ln in f:
- if inData:
- if ln == 'EOD\n':
- inData = False
- else:
- data[current] += ln
- continue
-
- m = kDataKeyStart.match(ln)
- if m:
- current,value = m.groups()
- if isServerSide:
- assert current[0] == current[-1] == "'"
- current = current[1:-1]
- assert value[0] == value[1] == ' '
- value = value[2:]
- if value == '<<EOD':
- value = ''
- inData = True
- else:
- assert value[0] == value[-2] == '"'
- assert value[-1] == ','
- value = value[1:-2]
- data[current] = value
- elif isServerSide:
- assert ln == ',\n'
- else:
- assert current is not None
- data[current] += ln
-
- # Things we are ignoring for now
- data.pop('a_file_sizes')
- data.pop('all_tests')
- data.pop('build_data')
- data.pop('cvs_dir_count')
- data.pop('cvs_file_count')
- data.pop('cvsaddedfiles')
- data.pop('cvsmodifiedfiles')
- data.pop('cvsremovedfiles')
- data.pop('cvsusercommitlist')
- data.pop('cvsuserupdatelist')
- data.pop('dejagnutests_log')
- data.pop('expfail_tests')
- data.pop('lines_of_code')
- data.pop('llcbeta_options')
- data.pop('new_tests')
- data.pop('o_file_sizes')
- data.pop('passing_tests')
- data.pop('removed_tests')
- data.pop('target_triple')
- data.pop('unexpfail_tests')
- data.pop('warnings')
- data.pop('warnings_added')
- data.pop('warnings_removed')
-
- starttime = data.pop('starttime').strip()
- endtime = data.pop('endtime').strip()
-
- nickname = data.pop('nickname').strip()
- machine_data = data.pop('machine_data').strip()
- buildstatus = data.pop('buildstatus').strip()
- configtime_user = data.pop('configtime_cpu')
- configtime_wall = data.pop('configtime_wall')
- checkouttime_user = data.pop('cvscheckouttime_cpu')
- checkouttime_wall = data.pop('cvscheckouttime_wall')
- dgtime_user = data.pop('dejagnutime_cpu')
- dgtime_wall = data.pop('dejagnutime_wall')
- buildtime_wall = float(data.pop('buildtime_wall').strip())
- buildtime_user = float(data.pop('buildtime_cpu').strip())
- gcc_version = data.pop('gcc_version')
- dejagnutests_results = data.pop('dejagnutests_results')
- multisource = data.pop('multisource_programstable')
- singlesource = data.pop('singlesource_programstable')
- externals = data.pop('externalsource_programstable')
-
- assert not data.keys()
-
- machine = { 'Name' : nickname,
- 'Info' : { 'gcc_version' : gcc_version } }
- for ln in machine_data.split('\n'):
- ln = ln.strip()
- if not ln:
- continue
- assert ':' in ln
- key,value = ln.split(':',1)
- machine['Info'][key] = value
-
- # We definitely don't want these in the machine data.
- if 'time' in machine['Info']:
- machine['Info'].pop('time')
- if 'date' in machine['Info']:
- machine['Info'].pop('date')
-
- run = { 'Start Time' : starttime,
- 'End Time' : endtime,
- 'Info' : { 'tag' : 'nightlytest' } }
-
- tests = []
-
- groupInfo = []
-
- # llvm-test doesn't provide this
- infoData = {}
-
- # Summary test information
- tests.append( { 'Name' : basename + '.Summary.configtime.wall',
- 'Info' : infoData,
- 'Data' : [configtime_wall] } )
- tests.append( { 'Name' : basename + '.Summary.configtime.user',
- 'Info' : infoData,
- 'Data' : [configtime_user] } )
- tests.append( { 'Name' : basename + '.Summary.checkouttime.wall',
- 'Info' : infoData,
- 'Data' : [checkouttime_wall] } )
- tests.append( { 'Name' : basename + '.Summary.checkouttime.user',
- 'Info' : infoData,
- 'Data' : [checkouttime_user] } )
- tests.append( { 'Name' : basename + '.Summary.buildtime.wall',
- 'Info' : infoData,
- 'Data' : [buildtime_wall] } )
- tests.append( { 'Name' : basename + '.Summary.buildtime.user',
- 'Info' : infoData,
- 'Data' : [buildtime_user] } )
- tests.append( { 'Name' : basename + '.Summary.dgtime.wall',
- 'Info' : infoData,
- 'Data' : [dgtime_wall] } )
- tests.append( { 'Name' : basename + '.Summary.dgtime.user',
- 'Info' : infoData,
- 'Data' : [dgtime_user] } )
- tests.append( { 'Name' : basename + '.Summary.buildstatus',
- 'Info' : infoData,
- 'Data' : [buildstatus == 'OK'] } )
-
- # DejaGNU Info
- results = parseDGResults(dejagnutests_results)
- for name in ('PASS', 'FAIL', 'XPASS', 'XFAIL'):
- tests.append( { 'Name' : basename + '.DejaGNU.' + name,
- 'Info' : infoData,
- 'Data' : [len(results.get(name,[]))] } )
-
- # llvm-test results
- groupInfo.append( { 'Name' : basename,
- 'Primary' : 1 } )
- for groupname,data in (('SingleSource', singlesource),
- ('MultiSource', multisource),
- ('Externals', externals)):
- groupInfo.append( { 'Name' : basename + '.' + groupname,
- 'Primary' : 1 } )
- lines = data.split('\n')
- header = lines[0].strip().split(',')
- for ln in lines[1:]:
- ln = ln.strip()
- if not ln:
- continue
- entry = dict([(k,v.strip())
- for k,v in zip(header, ln.split(','))])
- testname = basename + '.%s/%s' % (groupname,
- entry['Program'].replace('.','_'))
- groupInfo.append( { 'Name' : testname,
- 'Primary' : 1 } )
-
- for name,key,tname in (('gcc.compile', 'GCCAS', 'time'),
- ('bc.compile', 'Bytecode', 'size'),
- ('llc.compile', 'LLC compile', 'time'),
- ('llc-beta.compile', 'LLC-BETA compile', 'time'),
- ('jit.compile', 'JIT codegen', 'time'),
- ('gcc.exec', 'GCC', 'time'),
- ('cbe.exec', 'CBE', 'time'),
- ('llc.exec', 'LLC', 'time'),
- ('llc-beta.exec', 'LLC-BETA', 'time'),
- ('jit.exec', 'JIT', 'time'),
- ):
- time = entry[key]
- if time == '*':
- tests.append( { 'Name' : testname + '.%s.success' % name,
- 'Info' : infoData,
- 'Data' : [0] } )
- else:
- tests.append( { 'Name' : testname + '.%s.success' % name,
- 'Info' : infoData,
- 'Data' : [1] } )
- tests.append( { 'Name' : testname + '.%s.%s' % (name, tname),
- 'Info' : infoData,
- 'Data' : [float(time)] } )
- pass
-
- return { 'Machine' : machine,
- 'Run' : run,
- 'Tests' : tests,
- 'Group Info' : groupInfo }
-
-def convertNTData(inputPath, outputPath):
- """convertNTData - Convert a nightlytest "sentdata.txt" file into a zorg
- plist file."""
- import plistlib
-
- data = loadSentData(inputPath)
- plistlib.writePlist(data, outputPath)
-
-def main():
- global opts
- from optparse import OptionParser
- parser = OptionParser("usage: %prog file output")
- opts,args = parser.parse_args()
-
- if len(args) != 2:
- parser.error("incorrect number of argments")
-
- file,output = args
-
- convertNTData(file, output)
-
-if __name__=='__main__':
- main()
Modified: zorg/trunk/lnt/lnt/lnttool/__init__.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/lnttool/__init__.py?rev=99164&r1=99163&r2=99164&view=diff
==============================================================================
--- zorg/trunk/lnt/lnt/lnttool/__init__.py (original)
+++ zorg/trunk/lnt/lnt/lnttool/__init__.py Mon Mar 22 02:18:29 2010
@@ -45,6 +45,7 @@
False, None, 1, opts.threaded, opts.processes)
from create import action_create
+from convert import action_convert
commands = dict((name[7:], f) for name,f in locals().items()
if name.startswith('action_'))
Added: zorg/trunk/lnt/lnt/lnttool/convert.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/lnttool/convert.py?rev=99164&view=auto
==============================================================================
--- zorg/trunk/lnt/lnt/lnttool/convert.py (added)
+++ zorg/trunk/lnt/lnt/lnttool/convert.py Mon Mar 22 02:18:29 2010
@@ -0,0 +1,63 @@
+import os
+import sys
+
+def convert_data(input, output, inFormat, outFormat):
+ from lnt import formats
+
+ out = formats.get_format(outFormat)
+ if out is None or not out.get('write'):
+ raise SystemExit("unknown output format: %r" % outFormat)
+
+ data = formats.read_any(input, inFormat)
+
+ out['write'](data, output)
+ output.flush()
+
+def action_convert(name, args):
+ """convert between input formats"""
+
+ from optparse import OptionParser, OptionGroup
+ from lnt import formats
+ global parser
+ parser = OptionParser("%%prog %s [options] [<input>, [<output>]]" % name)
+ parser.add_option("", "--from", dest="inputFormat", metavar="NAME",
+ help="input format name [%default]", default='<auto>',
+ choices=formats.format_names + ['<auto>'])
+ parser.add_option("", "--to", dest="outputFormat", metavar="NAME",
+ help="output format name [%default]", default='plist',
+ choices=formats.format_names + ['<auto>'])
+ (opts, args) = parser.parse_args(args)
+
+ input = output = '-'
+ if len(args) == 0:
+ pass
+ elif len(args) == 1:
+ input, = args
+ elif len(args) == 2:
+ input,output = args
+ else:
+ parser.error("invalid number of arguments")
+
+ if input == '-':
+ # Guarantee that we can seek.
+ import StringIO
+ data = sys.stdin.read()
+ inf = StringIO.StringIO(data)
+ else:
+ inf = input
+
+ if output == '-':
+ outf = sys.stdout
+ else:
+ outf = open(output, 'wb')
+
+ try:
+ try:
+ convert_data(inf, outf, opts.inputFormat, opts.outputFormat)
+ finally:
+ if outf != sys.stdout:
+ outf.close()
+ except:
+ if outf != sys.stdout:
+ os.remove(output)
+ raise
Added: zorg/trunk/lnt/tests/Formats/Inputs/test.json
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/tests/Formats/Inputs/test.json?rev=99164&view=auto
==============================================================================
--- zorg/trunk/lnt/tests/Formats/Inputs/test.json (added)
+++ zorg/trunk/lnt/tests/Formats/Inputs/test.json Mon Mar 22 02:18:29 2010
@@ -0,0 +1 @@
+{ "a" : 1 }
Added: zorg/trunk/lnt/tests/Formats/Inputs/test.nightlytest
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/tests/Formats/Inputs/test.nightlytest?rev=99164&view=auto
==============================================================================
--- zorg/trunk/lnt/tests/Formats/Inputs/test.nightlytest (added)
+++ zorg/trunk/lnt/tests/Formats/Inputs/test.nightlytest Mon Mar 22 02:18:29 2010
@@ -0,0 +1,58 @@
+cvsaddedfiles =>
+cvsuserupdatelist =>
+all_tests => TEST-PASS: compile /MultiSource/Applications/aha/aha
+TEST-PASS: compile /MultiSource/Applications/aha/aha
+
+llcbeta_options =>
+
+cvsusercommitlist =>
+warnings_added =>
+new_tests =>
+starttime => 2010-03-20 05:23:59
+
+dejagnutests_log =>
+buildstatus => OK
+multisource_programstable => Program,GCCAS,Bytecode,LLC compile,LLC-BETA compile,JIT codegen,GCC,CBE,LLC,LLC-BETA,JIT,GCC/CBE,GCC/LLC,GCC/LLC-BETA,LLC/LLC-BETA
+Applications/aha/aha,0.0326,9696,0.0448,*,*, 2.7800,*, 2.1700,*,*,n/a,1.28,n/a,n/a
+
+warnings =>
+dejagnutime_cpu => 0.0
+cvscheckouttime_wall => 22.86
+cvsremovedfiles =>
+cvscheckouttime_cpu => 0
+unexpfail_tests =>
+gcc_version => i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5659)
+buildtime_wall => 229.26
+build_data =>
+
+buildtime_cpu => 1458.92
+removed_tests =>
+dejagnutime_wall => 0.0
+o_file_sizes =>
+configtime_cpu => 20.04
+target_triple => x86_64-apple-darwin10
+warnings_removed =>
+cvs_file_count => 0
+machine_data => uname: Darwin lordcrumb.apple.com 10.2.0 Darwin Kernel Version 10.2.0: Tue Nov 3 10:37:10 PST 2009; root:xnu-1486.2.11~1/RELEASE_I386 i386
+hardware: i386
+os: Darwin 10.2.0
+name: lordcrumb.apple.com
+date: 2010-03-20
+time: 06:41:16
+
+cvsmodifiedfiles =>
+a_file_sizes =>
+externalsource_programstable =>
+
+dejagnutests_results => Dejagnu skipped by user choice.
+configtime_wall => 21.38
+endtime => 2010-03-20 06:41:16
+
+nickname => lc-01-clang-i386-apple-darwin10
+singlesource_programstable =>
+
+passing_tests =>
+expfail_tests =>
+lines_of_code =>
+cvs_dir_count => 0
+
Added: zorg/trunk/lnt/tests/Formats/Inputs/test.plist
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/tests/Formats/Inputs/test.plist?rev=99164&view=auto
==============================================================================
--- zorg/trunk/lnt/tests/Formats/Inputs/test.plist (added)
+++ zorg/trunk/lnt/tests/Formats/Inputs/test.plist Mon Mar 22 02:18:29 2010
@@ -0,0 +1,8 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+ <key>a</key>
+ <integer>1</integer>
+</dict>
+</plist>
Added: zorg/trunk/lnt/tests/Formats/json.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/tests/Formats/json.py?rev=99164&view=auto
==============================================================================
--- zorg/trunk/lnt/tests/Formats/json.py (added)
+++ zorg/trunk/lnt/tests/Formats/json.py Mon Mar 22 02:18:29 2010
@@ -0,0 +1,4 @@
+# RUN: lnt convert --to=json %S/Inputs/test.json | FileCheck %s
+# RUN: lnt convert --to=json < %S/Inputs/test.json | FileCheck %s
+
+# CHECK: {"a": 1}
Added: zorg/trunk/lnt/tests/Formats/nightlytest.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/tests/Formats/nightlytest.py?rev=99164&view=auto
==============================================================================
--- zorg/trunk/lnt/tests/Formats/nightlytest.py (added)
+++ zorg/trunk/lnt/tests/Formats/nightlytest.py Mon Mar 22 02:18:29 2010
@@ -0,0 +1,6 @@
+# RUN: lnt convert --to=json %S/Inputs/test.nightlytest %t
+# RUN: FileCheck %s < %t
+
+# We are just checking the conversion, not validating the format.
+# CHECK: "Machine":
+# CHECK: "Tests": [
Added: zorg/trunk/lnt/tests/Formats/plist.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/tests/Formats/plist.py?rev=99164&view=auto
==============================================================================
--- zorg/trunk/lnt/tests/Formats/plist.py (added)
+++ zorg/trunk/lnt/tests/Formats/plist.py Mon Mar 22 02:18:29 2010
@@ -0,0 +1,4 @@
+# RUN: lnt convert --to=json %S/Inputs/test.plist | FileCheck %s
+# RUN: lnt convert --to=json < %S/Inputs/test.plist | FileCheck %s
+
+# CHECK: {"a": 1}
More information about the llvm-commits
mailing list