[llvm-commits] [zorg] r99107 - in /zorg/trunk: ./ lnt/ lnt/db/ lnt/import/ lnt/test/ lnt/test/DB/ lnt/test/Misc/ lnt/test/Web/ lnt/viewer/ lnt/viewer/js/ lnt/viewer/resources/ lnt/viewer/zview/
Daniel Dunbar
daniel at zuster.org
Sat Mar 20 18:00:06 PDT 2010
Author: ddunbar
Date: Sat Mar 20 20:00:06 2010
New Revision: 99107
URL: http://llvm.org/viewvc/llvm-project?rev=99107&view=rev
Log:
Add zorg/lnt, the latest incarnation of the LLVM nightly test infrastructure.
- See zorg/lnt/README.txt for some more information on what the new infrastructure looks like.
- This is a mass import from the software which runs on smooshlab; it isn't entirely functional yet, and needs a good bit of cleaning and reorg.
Added:
zorg/trunk/lnt/
zorg/trunk/lnt/README.txt
zorg/trunk/lnt/db/
zorg/trunk/lnt/db/CreateTables.sql
zorg/trunk/lnt/db/UpdateTables.sql
zorg/trunk/lnt/db/sqlite-to-mysql.sh (with props)
zorg/trunk/lnt/import/
zorg/trunk/lnt/import/AppleOpenSSLReader.py
zorg/trunk/lnt/import/ImportData (with props)
zorg/trunk/lnt/import/ImportXCBTimes (with props)
zorg/trunk/lnt/import/NTAuxSubmit (with props)
zorg/trunk/lnt/import/NTEmailReport.py (with props)
zorg/trunk/lnt/import/NightlytestReader.py (with props)
zorg/trunk/lnt/import/ServerUtil.py
zorg/trunk/lnt/import/SubmitData (with props)
zorg/trunk/lnt/test/
zorg/trunk/lnt/test/DB/
zorg/trunk/lnt/test/DB/Create.py
zorg/trunk/lnt/test/DB/Import.py
zorg/trunk/lnt/test/Misc/
zorg/trunk/lnt/test/Misc/SubmitAndEmail.py
zorg/trunk/lnt/test/Web/
zorg/trunk/lnt/test/Web/NightlytestMachinesRoot.py
zorg/trunk/lnt/test/Web/NightlytestRoot.py
zorg/trunk/lnt/test/Web/NightlytestRunRoot.py
zorg/trunk/lnt/test/Web/RootPage.py
zorg/trunk/lnt/test/lit.cfg
zorg/trunk/lnt/viewer/
zorg/trunk/lnt/viewer/Config.py
zorg/trunk/lnt/viewer/NTStyleBrowser.ptl
zorg/trunk/lnt/viewer/NTUtil.py
zorg/trunk/lnt/viewer/PerfDB.py
zorg/trunk/lnt/viewer/Util.py
zorg/trunk/lnt/viewer/__init__.py
zorg/trunk/lnt/viewer/js/
zorg/trunk/lnt/viewer/js/View2D.js
zorg/trunk/lnt/viewer/js/View2DTest.html
zorg/trunk/lnt/viewer/machines.ptl
zorg/trunk/lnt/viewer/nightlytest.ptl
zorg/trunk/lnt/viewer/publisher.py
zorg/trunk/lnt/viewer/resources/
zorg/trunk/lnt/viewer/resources/form.css
zorg/trunk/lnt/viewer/resources/popup.js
zorg/trunk/lnt/viewer/resources/style.css
zorg/trunk/lnt/viewer/root.ptl
zorg/trunk/lnt/viewer/runs.ptl
zorg/trunk/lnt/viewer/tests.ptl
zorg/trunk/lnt/viewer/wsgi_restart.py
zorg/trunk/lnt/viewer/zorg.cfg.sample
zorg/trunk/lnt/viewer/zorg.cgi (with props)
zorg/trunk/lnt/viewer/zorg.wsgi (with props)
zorg/trunk/lnt/viewer/zview/
zorg/trunk/lnt/viewer/zview/__init__.py
zorg/trunk/lnt/viewer/zview/zviewui.ptl
Modified:
zorg/trunk/README.txt
Modified: zorg/trunk/README.txt
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/README.txt?rev=99107&r1=99106&r2=99107&view=diff
==============================================================================
--- zorg/trunk/README.txt (original)
+++ zorg/trunk/README.txt Sat Mar 20 20:00:06 2010
@@ -23,3 +23,4 @@
$ROOT/buildbot/ - Buildbot configurations.
$ROOT/zorg/ - The root zorg Python module.
$ROOT/zorg/buildbot/ - Reusable components for buildbot configurations.
+ $ROOT/lnt/ - The LLVM "nightly test" infrastructure.
Added: zorg/trunk/lnt/README.txt
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/README.txt?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/README.txt (added)
+++ zorg/trunk/lnt/README.txt Sat Mar 20 20:00:06 2010
@@ -0,0 +1,38 @@
+LLVM "Nightly Test" Infrastructure
+==================================
+
+This directory and its subdirectories contain the LLVM nightly test
+infrastructure. This is technically version "3.0" of the LLVM nightly test
+architecture.
+
+LNT is written in Python and implements a (old-school) Quixote web-app,
+available by CGI and WSGI, and utilities for submitting data via LLVM's
+NewNightlyTest.pl in conjunction with LLVM's test-suite repository.
+
+The infrastructure has the following layout:
+
+ lnt/db - Database schema, utilities, and examples of the LNT plist format.
+
+ lnt/import - Utilities for converting to the LNT plist format for test data,
+ and for submitting plist to the server.
+
+ lnt/test - Tests for the infrastructure; they currently assume they are running
+ on a system with a live instance available at
+ 'http://localhost/zorg/'.
+
+ lnt/viewer - The LNT web-app itself.
+
+
+Installation Instructions
+-------------------------
+
+External Dependencies: SQLAlchemy, Quixote, mod_wsgi, SQLite,
+ MySQL (optional), urllib2_file
+
+Internal Dependencies: MooTools
+
+These are the steps to get a working LNT installation:
+
+ 1. Figure it out yourself, write installation instructions, add to README.txt.
+
+ 2. M-x revert-buffer, goto 1.
Added: zorg/trunk/lnt/db/CreateTables.sql
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/db/CreateTables.sql?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/db/CreateTables.sql (added)
+++ zorg/trunk/lnt/db/CreateTables.sql Sat Mar 20 20:00:06 2010
@@ -0,0 +1,80 @@
+-- ===- CreateTables.sql - Create SQL Performance DB Tables -----------------===
+--
+-- The LLVM Compiler Infrastructure
+--
+-- This file is distributed under the University of Illinois Open Source
+-- License. See LICENSE.TXT for details.
+--
+-- ===-----------------------------------------------------------------------===
+
+-- Machine Table
+-- Main machine list.
+CREATE TABLE Machine
+ (ID INTEGER PRIMARY KEY,
+ Name VARCHAR(512),
+ Number INTEGER);
+
+CREATE INDEX [Machine_IDX1] ON Machine(ID);
+CREATE INDEX [Machine_IDX2] ON Machine(Name);
+
+-- Machine Info Table
+-- Arbitrary information associated with a machine.
+CREATE TABLE MachineInfo
+ (ID INTEGER PRIMARY KEY,
+ Machine INTEGER,
+ `Key` TEXT,
+ Value TEXT,
+ FOREIGN KEY(Machine) REFERENCES Machine(ID));
+
+-- Run Table
+-- A specific run of a test on a machine.
+CREATE TABLE Run
+ (ID INTEGER PRIMARY KEY,
+ MachineID INTEGER,
+ StartTime DATETIME,
+ EndTime DATETIME,
+ FOREIGN KEY(MachineID) REFERENCES Machine(ID));
+
+CREATE INDEX [Run_IDX1] ON Run(ID);
+
+-- Run Info Table
+-- Arbitrary information about a run.
+CREATE TABLE RunInfo
+ (ID INTEGER PRIMARY KEY,
+ Run INTEGER,
+ `Key` TEXT,
+ Value TEXT,
+ FOREIGN KEY(Run) REFERENCES Run(ID));
+
+-- Test Table
+-- Tests are made up of several samples.
+CREATE TABLE Test
+ (ID INTEGER PRIMARY KEY,
+ Name VARCHAR(512),
+ Number INTEGER);
+
+CREATE INDEX [Test_IDX1] ON Test(ID);
+CREATE INDEX [Test_IDX2] ON Test(Name);
+
+-- Run Info Table
+-- Arbitrary information about a run.
+CREATE TABLE TestInfo
+ (ID INTEGER PRIMARY KEY,
+ Test INTEGER,
+ `Key` TEXT,
+ Value TEXT,
+ FOREIGN KEY(Test) REFERENCES Test(ID));
+
+-- Sample Table
+-- One data point for a particular test.
+CREATE TABLE Sample
+ (ID INTEGER PRIMARY KEY,
+ RunID INTEGER,
+ TestID INTEGER,
+ Value REAL,
+ FOREIGN KEY(RunID) REFERENCES Run(ID),
+ FOREIGN KEY(TestID) REFERENCES Test(ID));
+
+CREATE INDEX [Sample_IDX1] ON Sample(RunID);
+CREATE INDEX [Sample_IDX2] ON Sample(TestID);
+CREATE INDEX [Sample_IDX3] ON Sample(TestID, RunID);
Added: zorg/trunk/lnt/db/UpdateTables.sql
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/db/UpdateTables.sql?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/db/UpdateTables.sql (added)
+++ zorg/trunk/lnt/db/UpdateTables.sql Sat Mar 20 20:00:06 2010
@@ -0,0 +1,53 @@
+PRAGMA default_cache_size = 2000000;
+
+ALTER TABLE MachineInfo RENAME TO MachineInfoOld;
+CREATE TABLE MachineInfo
+ (ID INTEGER PRIMARY KEY,
+ Machine INTEGER,
+ Key TEXT,
+ Value TEXT,
+ FOREIGN KEY(Machine) REFERENCES Machine(ID));
+INSERT INTO MachineInfo (Machine,Key,Value)
+ SELECT Machine,Key,Value FROM MachineInfoOld;
+DROP TABLE MachineInfoOld;
+
+ALTER TABLE RunInfo RENAME TO RunInfoOld;
+CREATE TABLE RunInfo
+ (ID INTEGER PRIMARY KEY,
+ Run INTEGER,
+ Key TEXT,
+ Value TEXT,
+ FOREIGN KEY(Run) REFERENCES Run(ID));
+INSERT INTO RunInfo (Run,Key,Value)
+ SELECT Run,Key,Value FROM RunInfoOld;
+DROP TABLE RunInfoOld;
+
+ALTER TABLE TestInfo RENAME TO TestInfoOld;
+CREATE TABLE TestInfo
+ (ID INTEGER PRIMARY KEY,
+ Test INTEGER,
+ Key TEXT,
+ Value TEXT,
+ FOREIGN KEY(Test) REFERENCES Test(ID));
+INSERT INTO TestInfo (Test,Key,Value)
+ SELECT Test,Key,Value FROM TestInfoOld;
+DROP TABLE TestInfoOld;
+
+ALTER TABLE Sample RENAME TO SampleOld;
+CREATE TABLE Sample
+ (ID INTEGER PRIMARY KEY,
+ RunID INTEGER,
+ TestID INTEGER,
+ Key TEXT,
+ Value REAL,
+ FOREIGN KEY(RunID) REFERENCES Run(ID),
+ FOREIGN KEY(TestID) REFERENCES Test(ID));
+DROP INDEX [Sample_IDX1];
+DROP INDEX [Sample_IDX2];
+BEGIN TRANSACTION;
+INSERT INTO Sample (RunID,TestID,Key,Value)
+ SELECT RunID,TestID,Key,Value FROM SampleOld;
+COMMIT TRANSACTION;
+BEGIN TRANSACTION; CREATE INDEX [Sample_IDX1] ON Sample(RunID); COMMIT TRANSACTION;
+BEGIN TRANSACTION; CREATE INDEX [Sample_IDX2] ON Sample(TestID); COMMIT TRANSACTION;
+DROP TABLE SampleOld;
Added: zorg/trunk/lnt/db/sqlite-to-mysql.sh
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/db/sqlite-to-mysql.sh?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/db/sqlite-to-mysql.sh (added)
+++ zorg/trunk/lnt/db/sqlite-to-mysql.sh Sat Mar 20 20:00:06 2010
@@ -0,0 +1,23 @@
+#!/bin/sh
+
+set -eu
+
+if [ $# != 2 ]; then
+ echo "Usage: $0 <sqlite3-database> <output file>"
+
+ echo " Dumps the sqlite3 database to the output file "
+ echo " in SQL syntax that MySQL can understand."
+
+ exit 1
+fi
+
+in=$1
+out=$2
+
+sqlite3 $in .dump | \
+ sed -e 's#CREATE INDEX.*##g' \
+ -e 's#ANALYZE sqlite_master.*##g' \
+ -e 's#INSERT INTO "sqlite_stat1" VALUES.*##g' \
+ -e 's# Key \([ ]*\)TEXT# `Key`\1TEXT#g' \
+ -e 's#BEGIN TRANSACTION#START TRANSACTION#g' \
+ -e 's#INSERT INTO "\(.*\)"#INSERT INTO \1#g' > $out
Propchange: zorg/trunk/lnt/db/sqlite-to-mysql.sh
------------------------------------------------------------------------------
svn:executable = *
Added: zorg/trunk/lnt/import/AppleOpenSSLReader.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/import/AppleOpenSSLReader.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/import/AppleOpenSSLReader.py (added)
+++ zorg/trunk/lnt/import/AppleOpenSSLReader.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1,114 @@
+import os
+
+def parseOpenSSLFile(path):
+ data = open(path).read()
+ lines = list(open(path))
+ lnfields = [ln.strip().split(':') for ln in lines]
+ assert(lnfields[0][0] == '+H')
+ header = lnfields[0]
+ blockSizes = map(int, header[1:])
+
+ # Cipher -> [(Block Size,Value)*]
+ data = {}
+ for fields in lnfields[1:]:
+ # Ignore other fields
+ if fields[0] != '+F':
+ continue
+
+ name = fields[2]
+ countsPerBlock = fields[3:]
+ assert len(countsPerBlock) == len(blockSizes)
+ data[name] = [(b,float(c))
+ for b,c in zip(blockSizes,countsPerBlock)]
+
+ return data
+
+def loadData(path):
+ # Look for svn-revision and timestamps.
+
+ llvmRevision = ''
+ startTime = endTime = ''
+
+ f = os.path.join(path, 'svn-revision')
+ if os.path.exists(f):
+ svnRevisionData = open(f).read()
+ assert(svnRevisionData[0] == 'r')
+ llvmRevision = int(svnRevisionData[1:])
+
+ f = os.path.join(path, 'start.timestamp')
+ if os.path.exists(f):
+ startTime = open(f).read().strip()
+
+ f = os.path.join(path, 'finished.timestamp')
+ if os.path.exists(f):
+ endTime = open(f).read().strip()
+
+ # Look for sub directories
+ openSSLData = []
+ for file in os.listdir(path):
+ p = os.path.join(path, file)
+ if os.path.isdir(p):
+ # Look for Tests/Apple.OpenSSL.64/speed.txt
+ p = os.path.join(p, 'Tests/Apple.OpenSSL.64/speed.txt')
+ if os.path.exists(p):
+ openSSLData.append((file, parseOpenSSLFile(p)))
+
+ basename = 'apple_openssl'
+
+ machine = { 'Name' : 'dgohman.apple.com',
+ 'Info' : { } }
+
+ run = { 'Start Time' : startTime,
+ 'End Time' : endTime,
+ 'Info' : { 'llvm-revision' : llvmRevision,
+ 'tag' : 'apple_openssl' } }
+
+ tests = []
+ groupInfo = []
+
+ for dirName,dirData in openSSLData:
+ # Demangle compiler & flags
+ if dirName.startswith('gcc'):
+ compiler = 'gcc'
+ elif dirName.startswith('llvm-gcc'):
+ compiler = 'llvm-gcc'
+ else:
+ raise ValueError,compiler
+ assert dirName[len(compiler)] == '-'
+ flags = dirName[len(compiler)+1:]
+
+ for cipher,values in dirData.items():
+ testName = basename + '.' + cipher + '.ips'
+ for block,value in values:
+ parameters = { 'blockSize' : block,
+ 'compiler' : compiler,
+ 'compiler_flags' : flags }
+ tests.append( { 'Name' : testName,
+ 'Info' : parameters,
+ 'Data' : [value] } )
+
+ return { 'Machine' : machine,
+ 'Run' : run,
+ 'Tests' : tests,
+ 'Group Info' : groupInfo }
+
+def main():
+ import plistlib
+ import sys
+
+ global opts
+ from optparse import OptionParser
+ parser = OptionParser("usage: %prog raw-data-path output")
+ opts,args = parser.parse_args()
+
+ if len(args) != 2:
+ parser.error("incorrect number of argments")
+
+ file,output = args
+
+ data = loadData(file)
+
+ plistlib.writePlist(data, output)
+
+if __name__=='__main__':
+ main()
Added: zorg/trunk/lnt/import/ImportData
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/import/ImportData?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/import/ImportData (added)
+++ zorg/trunk/lnt/import/ImportData Sat Mar 20 20:00:06 2010
@@ -0,0 +1,113 @@
+#!/usr/bin/env python
+
+import os
+import plistlib
+import sys
+import time
+sys.path.append(os.path.join(os.path.dirname(__file__),'../'))
+
+import viewer
+from viewer import Util
+from viewer import PerfDB
+import NightlytestReader
+import AppleOpenSSLReader
+
+def main():
+ global opts
+ from optparse import OptionParser
+ parser = OptionParser("usage: %prog dbpath files+")
+ parser.add_option("", "--email-on-import", dest="emailOnImport", type=int,
+ default=False)
+ parser.add_option("", "--email-base-url", dest="emailReportURL", type=str,
+ default=None)
+ parser.add_option("", "--email-host", dest="emailReportHost", type=str,
+ default=None)
+ parser.add_option("", "--email-from", dest="emailReportFrom", type=str,
+ default=None)
+ parser.add_option("", "--email-to", dest="emailReportTo", type=str,
+ default=None)
+ parser.add_option("", "--format", dest="format",
+ choices=('plist','nightlytest','apple_openssl'),
+ default='plist')
+ parser.add_option("", "--commit", dest="commit", type=int,
+ default=True)
+ parser.add_option("", "--show-sql", dest="showSQL", action="store_true",
+ default=False)
+ parser.add_option("", "--show-sample-count", dest="showSampleCount",
+ action="store_true", default=False)
+ opts,args = parser.parse_args()
+
+ if len(args) < 2:
+ parser.error("incorrect number of argments")
+
+ dbpath = args[0]
+
+ startTime = time.time()
+ db = PerfDB.PerfDB(dbpath, echo=opts.showSQL)
+ importFiles(db, args[1:])
+ if opts.commit:
+ db.commit()
+ else:
+ db.rollback()
+ print 'TOTAL IMPORT TIME: %.2fs' % (time.time() - startTime,)
+
+def importFiles(db, files):
+ importer = { 'plist' : plistlib.readPlist,
+ 'nightlytest' : NightlytestReader.loadSentData,
+ 'apple_openssl' : AppleOpenSSLReader.loadData }[opts.format]
+
+ def consumer(file):
+ try:
+ return importer(file)
+ except KeyboardInterrupt:
+ raise
+ except:
+ print 'ERROR: %r: import failed' % file
+ import traceback
+ traceback.print_exc()
+ return None
+
+ numMachines = db.getNumMachines()
+ numRuns = db.getNumRuns()
+ numTests = db.getNumTests()
+
+ # If the database gets fragmented, count(*) in SQLite can get really slow!?!
+ if opts.showSampleCount:
+ numSamples = db.getNumSamples()
+
+ for file in files:
+ print 'IMPORT: %s' % file
+ startTime = time.time()
+ data = consumer(file)
+ print ' LOAD TIME: %.2fs' % (time.time() - startTime,)
+ if data is None:
+ continue
+
+ startTime = time.time()
+ success,(machine,run) = PerfDB.importDataFromDict(db, data)
+ print ' IMPORT TIME: %.2fs' % (time.time() - startTime,)
+ if not success:
+ print " IGNORING DUPLICATE RUN"
+ print " MACHINE: %d" % (run.machine_id, )
+ print " START : %s" % (run.start_time, )
+ print " END : %s" % (run.end_time, )
+ for ri in run.info.values():
+ print " INFO : %r = %r" % (ri.key, ri.value)
+ continue
+ else:
+ if opts.emailOnImport:
+ import NTEmailReport
+ NTEmailReport.emailReport(db, run,
+ opts.emailReportURL,
+ opts.emailReportHost,
+ opts.emailReportFrom,
+ opts.emailReportTo)
+
+ print "ADDED: %d machines" % (db.getNumMachines() - numMachines,)
+ print "ADDED: %d runs" % (db.getNumRuns() - numRuns,)
+ print "ADDED: %d tests" % (db.getNumTests() - numTests,)
+ if opts.showSampleCount:
+ print "ADDED: %d samples" % (db.getNumSamples() - numSamples)
+
+if __name__ == '__main__':
+ main()
Propchange: zorg/trunk/lnt/import/ImportData
------------------------------------------------------------------------------
svn:executable = *
Added: zorg/trunk/lnt/import/ImportXCBTimes
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/import/ImportXCBTimes?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/import/ImportXCBTimes (added)
+++ zorg/trunk/lnt/import/ImportXCBTimes Sat Mar 20 20:00:06 2010
@@ -0,0 +1,107 @@
+#!/usr/bin/env python
+
+import os
+import time
+import sys
+sys.path.append(os.path.join(os.path.dirname(__file__),'../'))
+import viewer
+from viewer import Util
+from viewer import PerfDB
+
+def main():
+ global opts
+ from optparse import OptionParser
+ parser = OptionParser("usage: %prog file dbpath")
+ parser.add_option("", "--test-prefix",
+ action="store", dest="testPrefix", default=None)
+ opts,args = parser.parse_args()
+
+ if len(args) != 2:
+ parser.error("incorrect number of argments")
+ if not opts.testPrefix:
+ parser.error("must specify test prefix")
+
+ dbpath,file = args
+
+ globals = {}
+ exec open(file) in globals, globals
+
+ db = PerfDB.PerfDB(dbpath)
+
+ numMachines = db.getNumMachines()
+ numTests = db.getNumTests()
+ numSamples = db.getNumSamples()
+
+ # Hardcode some things that aren't in the file
+ machine = PerfDB.Machine(-1,
+ name = 'lordcrumb.apple.com',
+ arch = 'Intel i386',
+ os = 'SnowLeopard',
+ hwconfig = '<unknown>',
+ compiler = '<unknown>')
+ machine = db.getOrCreateMachine(machine)
+ print "MACHINE: %r" % machine
+
+ # Treat as a single "run"; our DB format has no way to lock
+ # individual samples together. I recall now that this was the
+ # motivation in treating a sample as a group of numbers, not just
+ # one.
+
+ # Rough guess.
+ mtime = os.stat(file).st_mtime
+ timestamp = time.strftime('%Y-%m-%dT%H:%M:%Sz', time.localtime(mtime))
+
+ # FIXME: Need to extract revision. :(
+ run = db.createRun(machine, PerfDB.Run(-1, -1, timestamp=timestamp, svnRevision=None))
+ print "RUN: %r" % run
+
+ ####
+
+ runs = globals.get('runs')
+ for keys,data in runs:
+ # Mangle a test name
+ testName = '%s:threads=%s:pch=%s:mode=%s' % (opts.testPrefix,
+ keys.get('threads'),
+ int(keys.get('pch') == 'pch'),
+ keys.get('script'))
+ compiler = keys.get('cc')
+ compiler = compiler.replace('clang_driver','clang')
+ compiler = compiler.replace('_','/')
+ compiler = compiler.replace('xcc','ccc')
+ compilerOpts = '-O0,-g'
+
+ userTest = db.getOrCreateTest(PerfDB.Test(-1,
+ name = testName,
+ subtest = 'user',
+ kindID = None,
+ groupID = None,
+ compiler = compiler,
+ compilerOpts = compilerOpts))
+ sysTest = db.getOrCreateTest(PerfDB.Test(-1,
+ name = testName,
+ subtest = 'sys',
+ kindID = None,
+ groupID = None,
+ compiler = compiler,
+ compilerOpts = compilerOpts))
+ wallTest = db.getOrCreateTest(PerfDB.Test(-1,
+ name = testName,
+ subtest = 'wall',
+ kindID = None,
+ groupID = None,
+ compiler = compiler,
+ compilerOpts = compilerOpts))
+ assert data['version'] == 0
+ for (mem,user,sys,wall) in data['samples']:
+ db.addSample(userTest, run, PerfDB.Sample(-1, -1, -1, '', user))
+ db.addSample(sysTest, run, PerfDB.Sample(-1, -1, -1, '', sys))
+ db.addSample(wallTest, run, PerfDB.Sample(-1, -1, -1, '', wall))
+
+ db.commit()
+
+ print "ADDED: %d machines, %d tests, and %d samples." % (db.getNumMachines() - numMachines,
+ db.getNumTests() - numTests,
+ db.getNumSamples() - numSamples)
+
+if __name__ == '__main__':
+ main()
Propchange: zorg/trunk/lnt/import/ImportXCBTimes
------------------------------------------------------------------------------
svn:executable = *
Added: zorg/trunk/lnt/import/NTAuxSubmit
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/import/NTAuxSubmit?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/import/NTAuxSubmit (added)
+++ zorg/trunk/lnt/import/NTAuxSubmit Sat Mar 20 20:00:06 2010
@@ -0,0 +1,42 @@
+#!/usr/bin/env python
+
+import sys
+import NightlytestReader
+import ServerUtil
+
+def main():
+ global opts
+ from optparse import OptionParser
+ parser = OptionParser("usage: %prog {nightlytest sentdata.txt}*")
+ parser.add_option("", "--commit", dest="commit", type=int,
+ default=True)
+ parser.add_option("", "--no-convert", dest="noConvert")
+
+ # FIXME: It would be nice to support an easy mechanism for localized
+ # instances of the LNT infrastructure to default to the correct server for
+ # their installation.
+ parser.add_option("", "--server", dest="serverUrl", type=str,
+ default="http://llvm.org/perf/db_nt_internal/submitRun")
+
+ opts,args = parser.parse_args()
+
+ if not args:
+ parser.error("no input files")
+
+ for inputFile in args:
+ print '%s: note: submitting %s' % (sys.argv[0], inputFile)
+
+ if opts.noConvert:
+ plistPath = inputFile
+ else:
+ # Convert to the zorg format.
+ #
+ # FIXME: Avoid temp file.
+ plistPath = "/tmp/t.plist"
+ NightlytestReader.convertNTData(inputFile, plistPath)
+
+ # Send it off.
+ ServerUtil.submitFiles(opts.serverUrl, [plistPath], opts.commit)
+
+if __name__ == '__main__':
+ main()
Propchange: zorg/trunk/lnt/import/NTAuxSubmit
------------------------------------------------------------------------------
svn:executable = *
Added: zorg/trunk/lnt/import/NTEmailReport.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/import/NTEmailReport.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/import/NTEmailReport.py (added)
+++ zorg/trunk/lnt/import/NTEmailReport.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1,214 @@
+#!/usr/bin/python
+
+import os
+import smtplib
+import sys
+sys.path.append(os.path.join(os.path.dirname(__file__),'../'))
+
+import StringIO
+import viewer
+from viewer import PerfDB
+from viewer.NTUtil import *
+
+def main():
+ global opts
+ from optparse import OptionParser
+ parser = OptionParser("usage: %prog database run-id baseurl sendmail-host from to")
+ opts,args = parser.parse_args()
+
+ if len(args) != 6:
+ parser.error("incorrect number of argments")
+
+ dbpath,runID,baseurl,host,from_,to = args
+
+ db = PerfDB.PerfDB(dbpath)
+ run = db.getRun(int(runID))
+
+ emailReport(db, run, baseurl, host, from_, to)
+
+def emailReport(db, run, baseurl, host, from_, to):
+ import email.mime.text
+
+ subject, report = getReport(db, run, baseurl)
+
+ msg = email.mime.text.MIMEText(report)
+ msg['Subject'] = subject
+ msg['From'] = from_
+ msg['To'] = to
+
+ s = smtplib.SMTP(host)
+ s.sendmail(from_, [to], msg.as_string())
+ s.quit()
+
+def findPreceedingRun(query, run):
+ """findPreceedingRun - Find the most recent run in query which
+ preceeds run."""
+ best = None
+ for r in query:
+ # Restrict to nightlytest runs.
+ if 'tag' in r.info and r.info['tag'].value != 'nightlytest':
+ continue
+
+ # Select most recent run prior to the one we are reporting on.
+ if (r.start_time < run.start_time and
+ (best is None or r.start_time > best.start_time)):
+ best = r
+ return best
+
+def getReport(db, run, baseurl):
+ report = StringIO.StringIO()
+
+ machine = run.machine
+ compareTo = None
+
+ # Find comparison run.
+ # FIXME: Share this code with similar stuff in the viewer.
+ # FIXME: Scalability.
+ compareCrossesMachine = False
+ compareTo = findPreceedingRun(db.runs(machine=machine), run)
+
+ # If we didn't find a comparison run against this machine, look
+ # for a comparison run against the same machine name, and warn the
+ # user we are crosses machines.
+ if compareTo is None:
+ compareCrossesMachine = True
+ q = db.session.query(PerfDB.Run).join(PerfDB.Machine)
+ q = q.filter_by(name=machine.name)
+ compareTo = findPreceedingRun(q, run)
+
+ summary = RunSummary()
+ summary.addRun(db, run)
+ if compareTo:
+ summary.addRun(db, compareTo)
+
+ def getTestValue(run, testname, keyname):
+ fullname = 'nightlytest.' + testname + '.' + keyname
+ t = summary.testMap.get(str(fullname))
+ if t is None:
+ return None
+ samples = summary.getRunSamples(run).get(t.id)
+ if not samples:
+ return None
+ return samples[0]
+ def getTestSuccess(run, testname, keyname):
+ res = getTestValue(run, testname, keyname + '.success')
+ if res is None:
+ return res
+ return not not res
+
+ newPasses = Util.multidict()
+ newFailures = Util.multidict()
+ addedTests = Util.multidict()
+ removedTests = Util.multidict()
+ allTests = set()
+ allFailures = set()
+ allFailuresByKey = Util.multidict()
+ for keyname,title in kTSKeys.items():
+ for testname in summary.testNames:
+ curResult = getTestSuccess(run, testname, keyname)
+ prevResult = getTestSuccess(compareTo, testname, keyname)
+
+ if curResult is not None:
+ allTests.add((testname,keyname))
+ if curResult is False:
+ allFailures.add((testname,keyname))
+ allFailuresByKey[title] = testname
+
+ # Count as new pass if it passed, and previous result was failure.
+ if curResult and prevResult == False:
+ newPasses[testname] = title
+
+ # Count as new failure if it failed, and previous result was not
+ # failure.
+ if curResult == False and prevResult != False:
+ newFailures[testname] = title
+
+ if curResult is not None and prevResult is None:
+ addedTests[testname] = title
+ if curResult is None and prevResult is not None:
+ removedTests[testname] = title
+
+ changes = Util.multidict()
+ for i,(name,key) in enumerate(kComparisonKinds):
+ if not key:
+ # FIXME: File Size
+ continue
+
+ for testname in summary.testNames:
+ curValue = getTestValue(run, testname, key)
+ prevValue = getTestValue(compareTo, testname, key)
+
+ # Skip missing tests.
+ if curValue is None or prevValue is None:
+ continue
+
+ pct = Util.safediv(curValue, prevValue)
+ if pct is None:
+ continue
+ pctDelta = pct - 1.
+ if abs(pctDelta) < .05:
+ continue
+ if min(prevValue, curValue) <= .2:
+ continue
+
+ changes[name] = (testname, curValue, prevValue, pctDelta)
+
+ if baseurl[-1] == '/':
+ baseurl = baseurl[:-1]
+ print >>report, """%s/%d/""" % (baseurl, run.id)
+ print >>report, """Name: %s""" % (machine.info['name'].value,)
+ print >>report, """Nickname: %s:%d""" % (machine.name, machine.number)
+ print >>report
+ print >>report, """Run: %d, Start Time: %s, End Time: %s""" % (run.id, run.start_time, run.end_time)
+ if compareTo:
+ print >>report, """Comparing To: %d, Start Time: %s, End Time: %s""" % (compareTo.id, compareTo.start_time, compareTo.end_time)
+ if compareCrossesMachine:
+ print >>report, """*** WARNING ***:""",
+ print >>report, """comparison is against a different machine""",
+ print >>report, """(%s:%d)""" % (compareTo.machine.name,
+ compareTo.machine.number)
+ else:
+ print >>report, """Comparing To: (none)"""
+ print >>report
+
+ print >>report, """--- Changes Summary ---"""
+ for title,elts in (('New Test Passes', newPasses),
+ ('New Test Failures', newFailures),
+ ('Added Tests', addedTests),
+ ('Removed Tests', removedTests)):
+ print >>report, """%s: %d""" % (title,
+ sum([len(values)
+ for key,values in elts.items()]))
+ numSignificantChanges = sum([len(changelist)
+ for name,changelist in changes.items()])
+ print >>report, """Significant Changes: %d""" % (numSignificantChanges,)
+ print >>report
+ print >>report, """--- Tests Summary ---"""
+ print >>report, """Total Tests: %d""" % (len(allTests),)
+ print >>report, """Total Test Failures: %d""" % (len(allFailures),)
+ print >>report
+ print >>report, """Total Test Failures By Type:"""
+ for name,items in Util.sorted(allFailuresByKey.items()):
+ print >>report, """ %s: %d""" % (name, len(set(items)))
+
+ print >>report
+ print >>report, """--- Changes Detail ---"""
+ for title,elts in (('New Test Passes', newPasses),
+ ('New Test Failures', newFailures),
+ ('Added Tests', addedTests),
+ ('Removed Tests', removedTests)):
+ print >>report, """%s:""" % (title,)
+ print >>report, "".join("%s [%s]\n" % (key, ", ".join(values))
+ for key,values in Util.sorted(elts.items()))
+ print >>report, """Significant Changes in Test Results:"""
+ for name,changelist in changes.items():
+ print >>report, """%s:""" % name
+ for name,curValue,prevValue,delta in Util.sorted(changelist):
+ print >>report, """ %s: %.2f%% (%.4f => %.4f)""" % (name, delta*100, prevValue, curValue)
+
+ # FIXME: Where is the old mailer getting the arch from?
+ subject = """%s nightly tester results""" % machine.name
+ return subject,report.getvalue()
+
+if __name__ == '__main__':
+ main()
Propchange: zorg/trunk/lnt/import/NTEmailReport.py
------------------------------------------------------------------------------
svn:executable = *
Added: zorg/trunk/lnt/import/NightlytestReader.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/import/NightlytestReader.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/import/NightlytestReader.py (added)
+++ zorg/trunk/lnt/import/NightlytestReader.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1,242 @@
+#!/usr/bin/env python
+
+import re
+
+kDataKeyStart = re.compile('(.*) =>(.*)')
+
+def loadSentData(path):
+ def parseDGResults(text):
+ results = {}
+ if 'Dejagnu skipped by user choice' in text:
+ return results
+ for ln in text.strip().split('\n'):
+ result,value = ln.split(':',1)
+ results[result] = results.get(result,[])
+ results[result].append(value)
+ return results
+
+ basename = 'nightlytest'
+
+ # Guess the format (server side or client side) based on the first
+ # character.
+ isServerSide = (open(path).read(1) == '\'')
+
+ f = open(path)
+ data = {}
+
+ current = None
+ inData = False
+ for ln in f:
+ if inData:
+ if ln == 'EOD\n':
+ inData = False
+ else:
+ data[current] += ln
+ continue
+
+ m = kDataKeyStart.match(ln)
+ if m:
+ current,value = m.groups()
+ if isServerSide:
+ assert current[0] == current[-1] == "'"
+ current = current[1:-1]
+ assert value[0] == value[1] == ' '
+ value = value[2:]
+ if value == '<<EOD':
+ value = ''
+ inData = True
+ else:
+ assert value[0] == value[-2] == '"'
+ assert value[-1] == ','
+ value = value[1:-2]
+ data[current] = value
+ elif isServerSide:
+ assert ln == ',\n'
+ else:
+ assert current is not None
+ data[current] += ln
+
+ # Things we are ignoring for now
+ data.pop('a_file_sizes')
+ data.pop('all_tests')
+ data.pop('build_data')
+ data.pop('cvs_dir_count')
+ data.pop('cvs_file_count')
+ data.pop('cvsaddedfiles')
+ data.pop('cvsmodifiedfiles')
+ data.pop('cvsremovedfiles')
+ data.pop('cvsusercommitlist')
+ data.pop('cvsuserupdatelist')
+ data.pop('dejagnutests_log')
+ data.pop('expfail_tests')
+ data.pop('lines_of_code')
+ data.pop('llcbeta_options')
+ data.pop('new_tests')
+ data.pop('o_file_sizes')
+ data.pop('passing_tests')
+ data.pop('removed_tests')
+ data.pop('target_triple')
+ data.pop('unexpfail_tests')
+ data.pop('warnings')
+ data.pop('warnings_added')
+ data.pop('warnings_removed')
+
+ starttime = data.pop('starttime').strip()
+ endtime = data.pop('endtime').strip()
+
+ nickname = data.pop('nickname').strip()
+ machine_data = data.pop('machine_data').strip()
+ buildstatus = data.pop('buildstatus').strip()
+ configtime_user = data.pop('configtime_cpu')
+ configtime_wall = data.pop('configtime_wall')
+ checkouttime_user = data.pop('cvscheckouttime_cpu')
+ checkouttime_wall = data.pop('cvscheckouttime_wall')
+ dgtime_user = data.pop('dejagnutime_cpu')
+ dgtime_wall = data.pop('dejagnutime_wall')
+ buildtime_wall = float(data.pop('buildtime_wall').strip())
+ buildtime_user = float(data.pop('buildtime_cpu').strip())
+ gcc_version = data.pop('gcc_version')
+ dejagnutests_results = data.pop('dejagnutests_results')
+ multisource = data.pop('multisource_programstable')
+ singlesource = data.pop('singlesource_programstable')
+ externals = data.pop('externalsource_programstable')
+
+ assert not data.keys()
+
+ machine = { 'Name' : nickname,
+ 'Info' : { 'gcc_version' : gcc_version } }
+ for ln in machine_data.split('\n'):
+ ln = ln.strip()
+ if not ln:
+ continue
+ assert ':' in ln
+ key,value = ln.split(':',1)
+ machine['Info'][key] = value
+
+ # We definitely don't want these in the machine data.
+ if 'time' in machine['Info']:
+ machine['Info'].pop('time')
+ if 'date' in machine['Info']:
+ machine['Info'].pop('date')
+
+ run = { 'Start Time' : starttime,
+ 'End Time' : endtime,
+ 'Info' : { 'tag' : 'nightlytest' } }
+
+ tests = []
+
+ groupInfo = []
+
+ # llvm-test doesn't provide this
+ infoData = {}
+
+ # Summary test information
+ tests.append( { 'Name' : basename + '.Summary.configtime.wall',
+ 'Info' : infoData,
+ 'Data' : [configtime_wall] } )
+ tests.append( { 'Name' : basename + '.Summary.configtime.user',
+ 'Info' : infoData,
+ 'Data' : [configtime_user] } )
+ tests.append( { 'Name' : basename + '.Summary.checkouttime.wall',
+ 'Info' : infoData,
+ 'Data' : [checkouttime_wall] } )
+ tests.append( { 'Name' : basename + '.Summary.checkouttime.user',
+ 'Info' : infoData,
+ 'Data' : [checkouttime_user] } )
+ tests.append( { 'Name' : basename + '.Summary.buildtime.wall',
+ 'Info' : infoData,
+ 'Data' : [buildtime_wall] } )
+ tests.append( { 'Name' : basename + '.Summary.buildtime.user',
+ 'Info' : infoData,
+ 'Data' : [buildtime_user] } )
+ tests.append( { 'Name' : basename + '.Summary.dgtime.wall',
+ 'Info' : infoData,
+ 'Data' : [dgtime_wall] } )
+ tests.append( { 'Name' : basename + '.Summary.dgtime.user',
+ 'Info' : infoData,
+ 'Data' : [dgtime_user] } )
+ tests.append( { 'Name' : basename + '.Summary.buildstatus',
+ 'Info' : infoData,
+ 'Data' : [buildstatus == 'OK'] } )
+
+ # DejaGNU Info
+ results = parseDGResults(dejagnutests_results)
+ for name in ('PASS', 'FAIL', 'XPASS', 'XFAIL'):
+ tests.append( { 'Name' : basename + '.DejaGNU.' + name,
+ 'Info' : infoData,
+ 'Data' : [len(results.get(name,[]))] } )
+
+ # llvm-test results
+ groupInfo.append( { 'Name' : basename,
+ 'Primary' : 1 } )
+ for groupname,data in (('SingleSource', singlesource),
+ ('MultiSource', multisource),
+ ('Externals', externals)):
+ groupInfo.append( { 'Name' : basename + '.' + groupname,
+ 'Primary' : 1 } )
+ lines = data.split('\n')
+ header = lines[0].strip().split(',')
+ for ln in lines[1:]:
+ ln = ln.strip()
+ if not ln:
+ continue
+ entry = dict([(k,v.strip())
+ for k,v in zip(header, ln.split(','))])
+ testname = basename + '.%s/%s' % (groupname,
+ entry['Program'].replace('.','_'))
+ groupInfo.append( { 'Name' : testname,
+ 'Primary' : 1 } )
+
+ for name,key,tname in (('gcc.compile', 'GCCAS', 'time'),
+ ('bc.compile', 'Bytecode', 'size'),
+ ('llc.compile', 'LLC compile', 'time'),
+ ('llc-beta.compile', 'LLC-BETA compile', 'time'),
+ ('jit.compile', 'JIT codegen', 'time'),
+ ('gcc.exec', 'GCC', 'time'),
+ ('cbe.exec', 'CBE', 'time'),
+ ('llc.exec', 'LLC', 'time'),
+ ('llc-beta.exec', 'LLC-BETA', 'time'),
+ ('jit.exec', 'JIT', 'time'),
+ ):
+ time = entry[key]
+ if time == '*':
+ tests.append( { 'Name' : testname + '.%s.success' % name,
+ 'Info' : infoData,
+ 'Data' : [0] } )
+ else:
+ tests.append( { 'Name' : testname + '.%s.success' % name,
+ 'Info' : infoData,
+ 'Data' : [1] } )
+ tests.append( { 'Name' : testname + '.%s.%s' % (name, tname),
+ 'Info' : infoData,
+ 'Data' : [float(time)] } )
+ pass
+
+ return { 'Machine' : machine,
+ 'Run' : run,
+ 'Tests' : tests,
+ 'Group Info' : groupInfo }
+
+def convertNTData(inputPath, outputPath):
+ """convertNTData - Convert a nightlytest "sentdata.txt" file into a zorg
+ plist file."""
+ import plistlib
+
+ data = loadSentData(inputPath)
+ plistlib.writePlist(data, outputPath)
+
+def main():
+ global opts
+ from optparse import OptionParser
+ parser = OptionParser("usage: %prog file output")
+ opts,args = parser.parse_args()
+
+ if len(args) != 2:
+ parser.error("incorrect number of argments")
+
+ file,output = args
+
+ convertNTData(file, output)
+
+if __name__=='__main__':
+ main()
Propchange: zorg/trunk/lnt/import/NightlytestReader.py
------------------------------------------------------------------------------
svn:executable = *
Added: zorg/trunk/lnt/import/ServerUtil.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/import/ServerUtil.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/import/ServerUtil.py (added)
+++ zorg/trunk/lnt/import/ServerUtil.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1,14 @@
+import plistlib
+import urllib
+import urllib2
+import urllib2_file
+
+def submitFiles(url, files, commit):
+ for file in files:
+ data = { 'file' : open(file),
+ 'commit' : ("0","1")[not not commit] }
+
+ response = urllib2.urlopen(url, data)
+ the_page = response.read()
+
+ print the_page
Added: zorg/trunk/lnt/import/SubmitData
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/import/SubmitData?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/import/SubmitData (added)
+++ zorg/trunk/lnt/import/SubmitData Sat Mar 20 20:00:06 2010
@@ -0,0 +1,19 @@
+#!/usr/bin/python
+
+import ServerUtil
+
+def main():
+ global opts
+ from optparse import OptionParser
+ parser = OptionParser("usage: %prog serverUrl files+")
+ parser.add_option("", "--commit", dest="commit", type=int,
+ default=False)
+ opts,args = parser.parse_args()
+
+ if len(args) < 2:
+ parser.error("incorrect number of argments")
+
+ ServerUtil.submitFiles(args[0], args[1:], opts.commit)
+
+if __name__ == '__main__':
+ main()
Propchange: zorg/trunk/lnt/import/SubmitData
------------------------------------------------------------------------------
svn:executable = *
Added: zorg/trunk/lnt/test/DB/Create.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/test/DB/Create.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/test/DB/Create.py (added)
+++ zorg/trunk/lnt/test/DB/Create.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1,73 @@
+# RUN: rm -f %t.db
+# RUN: sqlite3 %t.db ".read %src_root/db/CreateTables.sql"
+# RUN: python %s %t.db
+
+import sys
+from viewer.PerfDB import PerfDB, Run
+
+# Check creation.
+
+db = PerfDB(sys.argv[1])
+
+assert db.getNumMachines() == 0
+assert db.getNumRuns() == 0
+assert db.getNumTests() == 0
+
+m,created = db.getOrCreateMachine("machine-0", [('m_key','m_value')])
+assert created
+
+r,created = db.getOrCreateRun(m, '2000-01-02 03:04:05', '2006-07-08 09:10:11',
+ [('r_key','r_value')])
+
+assert created
+t,created = db.getOrCreateTest("test-0", [('t_key','t_value')])
+assert created
+
+s = db.addSample(r, t, 1.0)
+
+print m
+print r
+print t
+
+db.commit()
+
+# Check uniquing.
+
+db2 = PerfDB(sys.argv[1])
+assert [m.id] == [i.id for i in db2.machines()]
+assert [r.id] == [i.id for i in db2.runs().all()]
+assert [t.id] == [i.id for i in db2.tests().all()]
+assert [s.id] == [i.id for i in db2.samples().all()]
+
+m2,created = db2.getOrCreateMachine("machine-0", [('m_key','m_value')])
+assert m.id == m2.id and not created
+
+r2,created = db2.getOrCreateRun(m, '2000-01-02 03:04:05', '2006-07-08 09:10:11',
+ [('r_key','r_value')])
+assert r.id == r2.id and not created
+
+t2,created = db2.getOrCreateTest("test-0", [('t_key','t_value')])
+assert t.id == t2.id and not created
+
+s2 = db2.addSample(r2, t2, 2.0)
+assert s.id != s2.id
+
+assert r.id == s.run.id == s2.run.id
+assert t.id == s.test.id == s2.test.id
+
+db2.commit()
+
+# Check load.
+
+db3 = PerfDB(sys.argv[1])
+m3 = db3.machines().one()
+r3 = db3.runs().one()
+t3 = db3.tests().one()
+s3a,s3b = db3.samples().all()
+print m3,r3,t3,s3a,s3b
+
+assert m.id == m3.id
+assert r.id == r3.id
+assert t.id == t3.id
+assert s.id == s3a.id
+assert s2.id == s3b.id
Added: zorg/trunk/lnt/test/DB/Import.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/test/DB/Import.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/test/DB/Import.py (added)
+++ zorg/trunk/lnt/test/DB/Import.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1,67 @@
+# RUN: rm -f %t.db
+# RUN: sqlite3 %t.db ".read %src_root/db/CreateTables.sql"
+
+# RUN: %src_root/import/ImportData --show-sample-count \
+# RUN: %t.db %S/Inputs/sample-a-small.plist |\
+# RUN: FileCheck -check-prefix=IMPORT-A-1 %s
+
+# IMPORT-A-1: ADDED: 1 machines
+# IMPORT-A-1: ADDED: 1 runs
+# IMPORT-A-1: ADDED: 90 tests
+# IMPORT-A-1: ADDED: 90 samples
+
+# RUN: %src_root/import/ImportData --show-sample-count \
+# RUN: %t.db %S/Inputs/sample-b-small.plist |\
+# RUN: FileCheck -check-prefix=IMPORT-B %s
+
+# IMPORT-B: ADDED: 0 machines
+# IMPORT-B: ADDED: 1 runs
+# IMPORT-B: ADDED: 0 tests
+# IMPORT-B: ADDED: 90 samples
+
+# RUN: %src_root/import/ImportData --show-sample-count \
+# RUN: %t.db %S/Inputs/sample-a-small.plist |\
+# RUN: FileCheck -check-prefix=IMPORT-A-2 %s
+
+# IMPORT-A-2: IGNORING DUPLICATE RUN
+# IMPORT-A-2: ADDED: 0 machines
+# IMPORT-A-2: ADDED: 0 runs
+# IMPORT-A-2: ADDED: 0 tests
+# IMPORT-A-2: ADDED: 0 samples
+
+# RUN: python %s %t.db
+
+import datetime, sys
+from viewer.PerfDB import PerfDB, Run, Test
+
+db = PerfDB(sys.argv[1])
+
+m = db.machines().one()
+assert m.id == 1
+assert m.name == 'smoosh-01.apple.com'
+
+info = dict((i.key,i.value) for i in m.info.values())
+assert 'os' in info
+assert info['os'] == ' Darwin 10.2.0'
+
+runs = db.runs().all()
+assert len(runs) == 2
+rA,rB = runs
+assert rA.machine == m
+assert rB.machine == m
+assert rA.start_time == datetime.datetime(2009, 11, 17, 2, 12, 25)
+assert rA.end_time == datetime.datetime(2009, 11, 17, 3, 44, 48)
+assert rA.info['tag'].key == 'tag'
+assert rA.info['tag'].value == 'nightlytest'
+
+t = db.tests().order_by(Test.name)[20]
+assert t.name == 'nightlytest.SingleSource/Benchmarks/BenchmarkGame/fannkuch.llc.compile.success'
+assert t.info.values() == []
+
+samples = db.samples(test=t).all()
+assert len(samples) == 2
+sA,sB = samples
+assert sA.run == rA
+assert sB.run == rB
+assert sA.value == 1.0
+assert sB.value == 1.0
Added: zorg/trunk/lnt/test/Misc/SubmitAndEmail.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/test/Misc/SubmitAndEmail.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/test/Misc/SubmitAndEmail.py (added)
+++ zorg/trunk/lnt/test/Misc/SubmitAndEmail.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1,19 @@
+# RUN: rm -f %t.db
+# RUN: sqlite3 %t.db ".read %src_root/db/CreateTables.sql"
+
+# FIXME: Find a way to test email works, without being annoying.
+# RUN: %src_root/import/ImportData \
+# RUN: --show-sample-count \
+# RUN: --commit=0 \
+# RUN: --email-on-import=1 --email-host=relay.example.com \
+# RUN: --email-from=example at example.com --email-to=example at example.com \
+# RUN: --email-base-url=ZORG_TEST %t.db %S/Inputs/sample-a-small.plist > %t
+# RUN: FileCheck %s < %t
+
+# CHECK: ADDED: 1 machines
+# CHECK: ADDED: 1 runs
+# CHECK: ADDED: 90 tests
+# CHECK: ADDED: 90 samples
+
+
+
Added: zorg/trunk/lnt/test/Web/NightlytestMachinesRoot.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/test/Web/NightlytestMachinesRoot.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/test/Web/NightlytestMachinesRoot.py (added)
+++ zorg/trunk/lnt/test/Web/NightlytestMachinesRoot.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1,4 @@
+# RUN: curl -s http://localhost/zorg/nightlytest/machines/1/ | FileCheck %s
+# CHECK: <h1>LLVM Nightly Test Results</h1>
+# CHECK: Render Time:
+
Added: zorg/trunk/lnt/test/Web/NightlytestRoot.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/test/Web/NightlytestRoot.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/test/Web/NightlytestRoot.py (added)
+++ zorg/trunk/lnt/test/Web/NightlytestRoot.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1,3 @@
+# RUN: curl -s http://localhost/zorg/nightlytest/ | FileCheck %s
+# CHECK: <h2>LLVM Nightly Test</h2>
+# CHECK: Render Time:
Added: zorg/trunk/lnt/test/Web/NightlytestRunRoot.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/test/Web/NightlytestRunRoot.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/test/Web/NightlytestRunRoot.py (added)
+++ zorg/trunk/lnt/test/Web/NightlytestRunRoot.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1,10 @@
+# RUN: curl -s http://localhost/zorg/nightlytest/1/ | FileCheck --check-prefix=BRIEF %s
+# BRIEF: <h1>LLVM Nightly Test Results</h1>
+# BRIEF: See Full Test Results
+# BRIEF: Render Time:
+
+# RUN: curl -s http://localhost/zorg/nightlytest/1/?full=1 | FileCheck --check-prefix=FULL %s
+# FULL: <h1>LLVM Nightly Test Results</h1>
+# FULL: See Brief Test Results
+# FULL: Render Time:
+
Added: zorg/trunk/lnt/test/Web/RootPage.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/test/Web/RootPage.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/test/Web/RootPage.py (added)
+++ zorg/trunk/lnt/test/Web/RootPage.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1,2 @@
+# RUN: curl -s http://localhost/zorg/ | FileCheck %s
+# CHECK: <h2>LLVM Testing DB</h2>
Added: zorg/trunk/lnt/test/lit.cfg
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/test/lit.cfg?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/test/lit.cfg (added)
+++ zorg/trunk/lnt/test/lit.cfg Sat Mar 20 20:00:06 2010
@@ -0,0 +1,30 @@
+# -*- Python -*-
+
+import os
+import platform
+
+# Configuration file for the 'lit' test runner.
+
+# name: The name of this test suite.
+config.name = 'Zorg'
+
+# testFormat: The test format to use to interpret tests.
+#
+# For now we require '&&' between commands, until they get globally killed and
+# the test runner updated.
+execute_external = platform.system() != 'Windows'
+config.test_format = lit.formats.ShTest(execute_external)
+
+# suffixes: A list of file extensions to treat as test files.
+config.suffixes = ['.py']
+
+# test_source_root: The root path where tests are located.
+config.test_source_root = os.path.dirname(__file__)
+config.test_exec_root = config.test_source_root
+
+config.target_triple = None
+
+src_root = os.path.join(config.test_source_root, '..')
+config.environment['PYTHONPATH'] = src_root
+
+config.substitutions.append(('%src_root', src_root))
Added: zorg/trunk/lnt/viewer/Config.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/Config.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/Config.py (added)
+++ zorg/trunk/lnt/viewer/Config.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1,60 @@
+import os
+
+class DBInfo:
+ @staticmethod
+ def fromData(baseDir, dict):
+ dbPath = dict.get('path')
+ if not dbPath.startswith('mysql://'):
+ dbPath = os.path.join(baseDir, dbPath)
+ return DBInfo(dbPath,
+ bool(dict.get('showNightlytest')),
+ bool(dict.get('showGeneral')))
+
+ def __init__(self, path, showNightlytest, showGeneral):
+ self.path = path
+ self.showNightlytest = showNightlytest
+ self.showGeneral = showGeneral
+
+class Config:
+ @staticmethod
+ def fromData(path, data):
+ # Paths are resolved relative to the absolute real path of the
+ # config file.
+ baseDir = os.path.dirname(os.path.abspath(path))
+
+ ntEmailer = data.get('nt_emailer')
+ if ntEmailer:
+ ntEmailEnabled = bool(ntEmailer.get('enabled'))
+ ntEmailHost = str(ntEmailer.get('host'))
+ ntEmailFrom = str(ntEmailer.get('from'))
+
+ # The email to field can either be a string, or a list of tuples of
+ # the form [(accept-regexp-pattern, to-address)].
+ item = ntEmailer.get('to')
+ if isinstance(item, str):
+ ntEmailTo = item
+ else:
+ ntEmailTo = [(str(a),str(b))
+ for a,b in item]
+ else:
+ ntEmailEnabled = False
+ ntEmailHost = ntEmailFrom = ntEmailTo = ""
+
+ return Config(os.path.join(baseDir, data['zorg']),
+ data['zorgURL'],
+ dict([(k,DBInfo.fromData(baseDir, v))
+ for k,v in data['databases'].items()]),
+ ntEmailEnabled, ntEmailHost, ntEmailFrom, ntEmailTo)
+
+ def __init__(self, zorgDir, zorgURL, databases,
+ ntEmailEnabled, ntEmailHost, ntEmailFrom, ntEmailTo):
+ self.zorgDir = zorgDir
+ self.zorgURL = zorgURL
+ self.tempDir = os.path.join(zorgDir, 'viewer', 'resources', 'graphs')
+ while self.zorgURL.endswith('/'):
+ self.zorgURL = zorgURL[:-1]
+ self.databases = databases
+ self.ntEmailEnabled = ntEmailEnabled
+ self.ntEmailHost = ntEmailHost
+ self.ntEmailFrom = ntEmailFrom
+ self.ntEmailTo = ntEmailTo
Added: zorg/trunk/lnt/viewer/NTStyleBrowser.ptl
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/NTStyleBrowser.ptl?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/NTStyleBrowser.ptl (added)
+++ zorg/trunk/lnt/viewer/NTStyleBrowser.ptl Sat Mar 20 20:00:06 2010
@@ -0,0 +1,404 @@
+# -*- python -*-
+
+import re
+
+import quixote
+from quixote.directory import Directory
+from quixote.errors import TraversalError
+
+import Util
+from NTUtil import *
+
+from PerfDB import Machine, Run
+
+class TestRunUI(Directory):
+ def __init__(self, root, idstr):
+ self.root = root
+ try:
+ self.id = int(idstr)
+ except ValueError, exc:
+ raise TraversalError(str(exc))
+
+ def getActiveRun(self, db):
+ # Check for overrides
+ request = quixote.get_request()
+ id = self.id
+ run_id = request.form.get('run', '')
+ if run_id:
+ id = int(run_id)
+ return db.getRun(id)
+
+ def getInfo(self, db):
+ request = quixote.get_request()
+
+ compareToID = request.form.get('compare', '')
+ compareTo = None
+ if compareToID:
+ try:
+ compareTo = db.getRun(int(compareToID))
+ except:
+ pass
+
+ run = self.getActiveRun(db)
+
+ # Find previous runs, ordered by time.
+ runs = db.runs(run.machine).order_by(Run.start_time.desc()).all()
+
+ # Find previous run to compare to
+ if compareTo is None:
+ for r in runs:
+ # FIXME: Compare revisions, not times.
+ if r != run and r.start_time < run.start_time:
+ compareTo = r
+ break
+
+ return run, runs, compareTo
+
+ def getRunSummary(self, db, run, compareTo, form=None):
+ testPredicate = None
+ infoPredicates = []
+ if form:
+ testPattern = form['testPattern']
+ if testPattern and testPattern.strip():
+ testPattern = re.compile(testPattern)
+ testPredicate = lambda t: testPattern.search(t.name)
+ parameters = self.getParameters()
+ for i,(title,name) in enumerate(self.getParameters()):
+ pattern = form['parmPattern.%d' % i]
+ if pattern and pattern.strip():
+ pattern = re.compile(pattern)
+ infoPredicates.append((name,
+ lambda t,k,v,p=pattern: p.search(v)))
+
+ # Compare the summary information
+ summary = RunSummary()
+ summary.addRun(db, run, testPredicate, infoPredicates)
+ if compareTo:
+ summary.addRun(db, compareTo, testPredicate, infoPredicates)
+ return summary
+
+ def _q_index [html] (self):
+ self.root.getHeader(self.getTitle(), "../..",
+ addPopupJS=True, addFormCSS=True)
+
+ request = quixote.get_request()
+ full = request.form.get('full', '')
+ allResults = not not full
+
+ # Get the filtering form.
+ form = quixote.form.Form(method=str("get"))
+ form.add(quixote.form.StringWidget, "testPattern",
+ title="Test Pattern")
+ for i,(title,name) in enumerate(self.getParameters()):
+ form.add(quixote.form.StringWidget, "parmPattern.%d" %i,
+ title="Parameter Pattern: %s" % title)
+ form.add_submit("submit", "Update")
+ Util.addOtherFormValues(form)
+
+ # Get a DB connection
+ db = self.root.getDB()
+ run,runs,compareTo = self.getInfo(db)
+ machine = run.machine
+ summary = self.getRunSummary(db, run, compareTo, form)
+
+ """
+ <center>
+ <h1>%s</h1>
+ <table>
+ <tr>
+ <td align=right>Machine:</td>
+ <td>%s:%d</td>
+ </tr>
+ <tr>
+ <td align=right>Run:</td>
+ <td>%s</td>
+ </tr>
+ """ % (self.getHeaderTitle(), machine.name, machine.number, run.start_time)
+ if compareTo:
+ """
+ <tr>
+ <td align=right>Compare To:</td>
+ <td>%s</td>
+ </tr>
+ """ % (compareTo.start_time,)
+ """
+ </table>
+ </center>
+ <p>
+ """
+
+ # Hide by default unless filled in
+ hidden = True
+ for w in form.get_all_widgets():
+ if isinstance(w, (quixote.form.HiddenWidget,)):
+ continue
+ value = w.value
+ if value:
+ hidden = False
+ break
+ key = 'filteringOptions'
+ """
+ <a href="javascript://" onclick="toggleLayer('%s')"; id="%s_">(%s)
+ Show Filtering Options</a>
+ <div id="%s" style="display: %s;" class="hideable">
+ """ % (key, key, ("+","-")[hidden], key, ("","none")[hidden])
+ form.render()
+ """
+ </div>
+ <p>
+ """
+
+ """
+ <table width="100%%" border=1>
+ <tr>
+ <td valign="top" width="200">
+ <a href="..">Homepage</a>
+ <h4>Machine:</h4>
+ <a href="../machines/%d/">%s:%d</a>
+ <h4>Runs:</h4>
+ <ul>
+ """ % (machine.id, machine.name, machine.number)
+
+ # Show a small number of neighboring runs.
+ runIndex = runs.index(run)
+ for r in runs[max(0,runIndex-3):runIndex+6]:
+ if r == run:
+ """ <li> <h3><a href="../%d/">%s</a></h3> """ % (r.id, r.start_time)
+ else:
+ """ <li> <a href="../%d/">%s</a> """ % (r.id, r.start_time)
+
+ # Full list of runs in a drop down.
+ #
+ # FIXME: Make this link to the proper run.
+ """
+ <p>
+ <form method="GET" action=".">
+ <input type="hidden" name="full" value="%s">
+ <select name="run">
+ """ % (full,)
+ for r in runs:
+ """\
+ <option value="%d"%s>%s""" % (r.id, ('', ' selected')[r == run], r.start_time)
+
+ """
+ </select>
+ <input type="submit" value="Jump to Run">
+ </form>
+ """
+ # Set comparison run.
+ """
+ <form method="GET" action=".">
+ <input type="hidden" name="full" value="%s">
+ <select name="compare">
+ """ % (full,)
+ for r in runs:
+ """\
+ <option value="%d"%s>%s</option>""" % (r.id, ('', ' selected')[r == compareTo], r.start_time)
+
+ """
+ </select>
+ <input type="submit" value="Compare to Run">
+ </form>
+ """
+
+ """
+ </ul>
+ </td>
+ <td valign="top">
+ <table border=1>
+ <tr>
+ <td> <b>Nickname</b> </td>
+ <td> %s </td>
+ </tr>
+ """ % (machine.name,)
+ for mi in machine.info.values():
+ """
+ <tr>
+ <td> <b>%s</b> </td>
+ <td>%s</td>
+ </tr>
+ """ % (mi.key, mi.value)
+ """
+ <tr>
+ <td> <b>Machine ID</b> </td>
+ <td> %d </td>
+ </tr>
+ </table>
+ """ % (machine.id,)
+
+ if allResults:
+ """<h4><a href="?full=">See Brief Test Results</a></h4>"""
+ else:
+ """<h4><a href="?full=1">See Full Test Results</a></h4>"""
+
+ self.renderCommonContents(db, run, compareTo, summary)
+ if not allResults:
+ self.renderBriefContents(db, run, compareTo, summary)
+
+ """
+ </td>
+ </tr>
+ </table>
+ """
+
+ if allResults:
+ self.renderFullContents(db, run, compareTo, summary)
+
+ self.root.getFooter()
+
+ ###
+
+ def getTags(self):
+ abstract
+
+ def getTitle(self):
+ abstract
+
+ def getHeaderTitle(self):
+ abstract
+
+ def getParameters(self):
+ abstract
+
+ def renderFullContents(self, db, run, compareTo, summary):
+ abstract
+
+ def renderBriefContents(self, db, run, compareTo, summary):
+ abstract
+
+ def renderCommonContents(self, db, run, compareTo, summary):
+ abstract
+
+class MachinesDirectory(Directory):
+ _q_exports = [""]
+
+ def __init__(self, parent):
+ Directory.__init__(self)
+ self.parent = parent
+
+ def _q_index [plain] (self):
+ """
+ machine access
+ """
+
+ def _q_lookup(self, component):
+ return self.parent.getTestMachineUI(component)
+
+class ProgramsDirectory(Directory):
+ _q_exports = [""]
+
+ def __init__(self, parent):
+ Directory.__init__(self)
+ self.parent = parent
+
+ def _q_index [plain] (self):
+ """
+ program access
+ """
+
+ def _q_lookup(self, component):
+ return self.parent.getProgramUI(component)
+
+class RecentMachineDirectory(Directory):
+ def __init__(self, root):
+ Directory.__init__(self)
+ self.root = root
+
+ def getTitle(self):
+ abstract
+
+ def getHeaderTitle(self):
+ abstract
+
+ def getTags(self):
+ abstract
+
+ def _q_index [plain] (self):
+ self.root.getHeader(self.getTitle(), "..")
+
+ # Get a DB connection
+ db = self.root.getDB()
+
+ # Find recent runs.
+ """
+ <center>
+ <h2>%s</h2>
+ </center>
+ """ % (self.getHeaderTitle(),)
+
+ """
+ <table width="100%%">
+ <tr>
+ <td valign="top" width="50%">
+ <center>
+ <h3>Test Machines</h3>
+ <table class="sortable" border=1>
+ <thead>
+ <tr>
+ <th>Latest Submission</th>
+ <th>Machine</th>
+ <th>Results</th>
+ </tr>
+ </thead>
+ """
+
+ # Show the most recent entry for each machine.
+ q = db.session.query(Machine.name).distinct().order_by(Machine.name)
+ for name, in q:
+ # Get the most recent run for this machine name.
+ q = db.session.query(Run).join(Machine).filter(Machine.name == name)
+ r = q.order_by(Run.start_time.desc()).first()
+ """
+ <tr>
+ <td>%s</td>
+ <td align=left><a href="machines/%d/">%s:%d</a></td>
+ <td><a href="%d/">View Results</a></td>
+ </tr>
+ """ % (r.start_time, r.machine.id, r.machine.name, r.machine.number, r.id)
+
+ """
+ </table>
+ </center>
+ </td>
+ <td valign="top">
+ <center>
+ <h3>Recent Submissions</h3>
+ <table class="sortable" border=1>
+ <thead>
+ <tr>
+ <th>Start Time</th>
+ <th>End Time</th>
+ <th>Machine</th>
+ <th>Results</th>
+ </tr>
+ </thead>
+ """
+
+ # Show the 20 most recent submissions, ordered by time.
+ for r in db.session.query(Run).order_by(Run.start_time.desc())[:20]:
+ m = r.machine
+ """
+ <tr>
+ <td>%s</td>
+ <td>%s</td>
+ <td align=left><a href="machines/%d/">%s:%d</a></td>
+ <td><a href="%d/">View Results</a></td>
+ </tr>
+ """ % (r.start_time, r.end_time, m.id, m.name, m.number, r.id)
+
+ """
+ </table>
+ </center>
+ </td>
+ </tr>
+ </table>
+ """
+
+ self.root.getFooter()
+
+ def _q_lookup(self, component):
+ if component == 'machines':
+ return MachinesDirectory(self)
+ if component == 'programs':
+ return ProgramsDirectory(self)
+ return self.getTestRunUI(component)
Added: zorg/trunk/lnt/viewer/NTUtil.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/NTUtil.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/NTUtil.py (added)
+++ zorg/trunk/lnt/viewer/NTUtil.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1,113 @@
+import Util
+from PerfDB import Run, Sample, Test
+
+kPrefix = 'nightlytest'
+
+# FIXME: We shouldn't need this.
+kSentinelKeyName = 'bc.compile.success'
+
+kComparisonKinds = [('File Size',None),
+ ('CBE','cbe.exec.time'),
+ ('LLC','llc.exec.time'),
+ ('JIT','jit.exec.time'),
+ ('GCCAS','gcc.compile.time'),
+ ('Bitcode','bc.compile.size'),
+ ('LLC compile','llc.compile.time'),
+ ('LLC-BETA compile','llc-beta.compile.time'),
+ ('JIT codegen','jit.compile.time'),
+ ('LLC-BETA','llc-beta.exec.time')]
+
+kTSKeys = { 'gcc.compile' : 'GCCAS',
+ 'bc.compile' : 'Bitcode',
+ 'llc.compile' : 'LLC compile',
+ 'llc-beta.compile' : 'LLC_BETA compile',
+ 'jit.compile' : 'JIT codegen',
+ 'cbe.exec' : 'CBE',
+ 'llc.exec' : 'LLC',
+ 'llc-beta.exec' : 'LLC-BETA',
+ 'jit.exec' : 'JIT' }
+
+# This isn't very fast, compute a summary if querying the same run
+# repeatedly.
+def getTestValueInRun(db, r, t, default=None, coerce=None):
+ for value, in db.session.query(Sample.value).\
+ filter(Sample.test == t).\
+ filter(Sample.run == r):
+ if coerce is not None:
+ return coerce(value)
+ return value
+ return default
+
+def getTestNameValueInRun(db, r, testname, default=None, coerce=None):
+ for value, in db.session.query(Sample.value).join(Test).\
+ filter(Test.name == testname).\
+ filter(Sample.run == r):
+ if coerce is not None:
+ return coerce(value)
+ return value
+ return default
+
+class RunSummary:
+ def __init__(self):
+ # The union of test names seen.
+ self.testNames = set()
+ # Map of test ids to test instances.
+ self.testIds = {}
+ # Map of test names to test instances
+ self.testMap = {}
+ # Map of run to multimap of test ID to sample list.
+ self.runSamples = {}
+
+ # FIXME: Should we worry about the info parameters on a
+ # nightlytest test?
+
+ def testMatchesPredicates(self, db, t, testPredicate, infoPredicates):
+ if testPredicate:
+ if not testPredicate(t):
+ return False
+ if infoPredicates:
+ info = dict((i.key,i.value) for i in t.info.values())
+ for key,predicate in infoPredicates:
+ value = info.get(key)
+ if not predicate(t, key, value):
+ return False
+ return True
+
+ def addRun(self, db, run, testPredicate=None, infoPredicates=None):
+ sampleMap = self.runSamples.get(run.id)
+ if sampleMap is None:
+ sampleMap = self.runSamples[run.id] = Util.multidict()
+
+ q = db.session.query(Sample.value,Test).join(Test)
+ q = q.filter(Sample.run == run)
+ for s_value,t in q:
+ if not self.testMatchesPredicates(db, t, testPredicate, infoPredicates):
+ continue
+
+ sampleMap[t.id] = s_value
+ self.testMap[t.name] = t
+ self.testIds[t.id] = t
+
+ # Filter out summary things in name lists by only looking
+ # for things which have a .success entry.
+ if t.name.endswith('.success'):
+ self.testNames.add(t.name.split('.', 3)[1])
+
+ def getRunSamples(self, run):
+ if run is None:
+ return {}
+ return self.runSamples.get(run.id, {})
+
+ def getTestValueByName(self, run, testName, default, coerce=None):
+ t = self.testMap.get(testName)
+ if t is None:
+ return default
+ sampleMap = self.runSamples.get(run.id, {})
+ samples = sampleMap.get(t.id)
+ if sampleMap is None or samples is None:
+ return default
+ # FIXME: Multiple samples?
+ if coerce:
+ return coerce(samples[0].value)
+ else:
+ return samples[0].value
Added: zorg/trunk/lnt/viewer/PerfDB.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/PerfDB.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/PerfDB.py (added)
+++ zorg/trunk/lnt/viewer/PerfDB.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1,429 @@
+#!/usr/bin/python
+
+###
+# SQLAlchemy database layer
+
+import sqlalchemy
+import sqlalchemy.ext.declarative
+import sqlalchemy.orm
+from sqlalchemy import *
+from sqlalchemy.orm import relation, backref
+from sqlalchemy.orm.collections import attribute_mapped_collection
+
+Base = sqlalchemy.ext.declarative.declarative_base()
+class Machine(Base):
+ __tablename__ = 'Machine'
+
+ id = Column("ID", Integer, primary_key=True)
+ name = Column("Name", String(256))
+ number = Column("Number", Integer)
+
+ info = relation('MachineInfo',
+ collection_class=attribute_mapped_collection('key'),
+ backref=backref('machine'))
+
+ def __init__(self, name, number):
+ self.name = name
+ self.number = number
+
+ def __repr__(self):
+ return '%s%r' % (self.__class__.__name__, (self.name, self.number))
+
+class MachineInfo(Base):
+ __tablename__ = 'MachineInfo'
+
+ id = Column("ID", Integer, primary_key=True)
+ machine_id = Column("Machine", Integer, ForeignKey('Machine.ID'))
+ key = Column("Key", String(256))
+ value = Column("Value", String(4096))
+
+ def __init__(self, machine, key, value):
+ self.machine = machine
+ self.key = key
+ self.value = value
+
+ def __repr__(self):
+ return '%s%r' % (self.__class__.__name__,
+ (self.machine, self.key, self.value))
+
+class Run(Base):
+ __tablename__ = 'Run'
+
+ id = Column("ID", Integer, primary_key=True)
+ machine_id = Column("MachineID", Integer, ForeignKey('Machine.ID'))
+ start_time = Column("StartTime", DateTime)
+ end_time = Column("EndTime", DateTime)
+
+ machine = relation(Machine)
+
+ info = relation('RunInfo',
+ collection_class=attribute_mapped_collection('key'),
+ backref=backref('run'))
+
+ def __init__(self, machine, start_time, end_time):
+ self.machine = machine
+ self.start_time = start_time
+ self.end_time = end_time
+
+ def __repr__(self):
+ return '%s%r' % (self.__class__.__name__,
+ (self.machine, self.start_time, self.end_time))
+
+class RunInfo(Base):
+ __tablename__ = 'RunInfo'
+
+ id = Column("ID", Integer, primary_key=True)
+ run_id = Column("Run", Integer, ForeignKey('Run.ID'))
+ key = Column("Key", String(256))
+ value = Column("Value", String(4096))
+
+ def __init__(self, run, key, value):
+ self.run = run
+ self.key = key
+ self.value = value
+
+ def __repr__(self):
+ return '%s%r' % (self.__class__.__name__,
+ (self.run, self.key, self.value))
+
+class Test(Base):
+ __tablename__ = 'Test'
+
+ id = Column("ID", Integer, primary_key=True)
+ name = Column("Name", String(512))
+
+ info = relation('TestInfo',
+ collection_class=attribute_mapped_collection('key'),
+ backref=backref('test'))
+
+ def __init__(self, name):
+ self.name = name
+
+ def __repr__(self):
+ return '%s%r' % (self.__class__.__name__,
+ (self.name,))
+
+class TestInfo(Base):
+ __tablename__ = 'TestInfo'
+
+ id = Column("ID", Integer, primary_key=True)
+ test_id = Column("Test", Integer, ForeignKey('Test.ID'))
+ key = Column("Key", String(256))
+ value = Column("Value", String(4096))
+
+ def __init__(self, test, key, value):
+ self.test = test
+ self.key = key
+ self.value = value
+
+ def __repr__(self):
+ return '%s%r' % (self.__class__.__name__,
+ (self.test, self.key, self.value))
+
+class Sample(Base):
+ __tablename__ = 'Sample'
+
+ id = Column("ID", Integer, primary_key=True)
+ run_id = Column("RunID", Integer, ForeignKey('Run.ID'))
+ test_id = Column("TestID", Integer, ForeignKey('Test.ID'))
+ value = Column("Value", Float)
+
+ run = relation(Run)
+ test = relation(Test)
+
+ def __init__(self, run, test, value):
+ self.run = run
+ self.test = test
+ self.value = value
+
+ def __repr__(self):
+ return '%s%r' % (self.__class__.__name__,
+ (self.run, self.test, self.value))
+
+###
+# PerfDB wrapper, to avoid direct SA dependency when possible.
+
+def info_eq(a, b):
+ a = list(a)
+ b = list(b)
+ a.sort()
+ b.sort()
+ return a == b
+
+class PerfDB:
+ def __init__(self, path, echo=False):
+ if not path.startswith('mysql://'):
+ path = 'sqlite:///' + path
+ self.engine = sqlalchemy.create_engine(path, echo=echo)
+ Session = sqlalchemy.orm.sessionmaker(self.engine)
+ self.session = Session()
+
+ def machines(self, name=None):
+ q = self.session.query(Machine)
+ if name:
+ q = q.filter_by(name=name)
+ return q
+
+ def tests(self, name=None):
+ q = self.session.query(Test)
+ if name:
+ q = q.filter_by(name=name)
+ return q
+
+ def runs(self, machine=None):
+ q = self.session.query(Run)
+ if machine:
+ q = q.filter_by(machine=machine)
+ return q
+
+ def samples(self, run=None, test=None):
+ q = self.session.query(Sample)
+ if run:
+ q = q.filter_by(run_id=run.id)
+ if test:
+ q = q.filter_by(test_id=test.id)
+ return q
+
+ def getNumMachines(self):
+ return self.machines().count()
+
+ def getNumRuns(self):
+ return self.runs().count()
+
+ def getNumTests(self):
+ return self.tests().count()
+
+ def getNumSamples(self):
+ return self.samples().count()
+
+ def getMachine(self, id):
+ return self.session.query(Machine).filter_by(id=id).one()
+
+ def getRun(self, id):
+ return self.session.query(Run).filter_by(id=id).one()
+
+ def getTest(self, id):
+ return self.session.query(Test).filter_by(id=id).one()
+
+ def getOrCreateMachine(self, name, info):
+ # FIXME: Not really the right way...
+ number = 1
+ for m in self.machines(name=name):
+ if info_eq([(i.key, i.value) for i in m.info.values()], info):
+ return m,False
+ number += 1
+
+ # Make a new record
+ m = Machine(name, number)
+ m.info = dict((k,MachineInfo(m,k,v)) for k,v in info)
+ self.session.add(m)
+ return m,True
+
+ def getOrCreateTest(self, name, info):
+ # FIXME: Not really the right way...
+ for t in self.tests(name):
+ if info_eq([(i.key, i.value) for i in t.info.values()], info):
+ return t,False
+
+ t = Test(name)
+ t.info = dict((k,TestInfo(t,k,v)) for k,v in info)
+ self.session.add(t)
+ return t,True
+
+ def getOrCreateRun(self, machine, start_time, end_time, info):
+ from datetime import datetime
+ start_time = datetime.strptime(start_time,
+ "%Y-%m-%d %H:%M:%S")
+ end_time = datetime.strptime(end_time,
+ "%Y-%m-%d %H:%M:%S")
+
+ # FIXME: Not really the right way...
+ for r in self.session.query(Run).filter_by(machine=machine):
+ # FIXME: Execute this filter in SQL, but resolve the
+ # normalization issue w.r.t. SQLAlchemy first. I think we
+ # may be running afoul of SQLite not normalizing the
+ # datetime. If I don't do this then sqlalchemy issues a
+ # query in the format YYYY-MM-DD HH:MM:SS.ssss which
+ # doesn't work.
+ if r.start_time != start_time or r.end_time != end_time:
+ continue
+ if info_eq([(i.key, i.value) for i in r.info.values()], info):
+ return r,False
+
+ # Make a new record
+ r = Run(machine, start_time, end_time)
+ r.info = dict((k,RunInfo(r,k,v)) for k,v in info)
+ self.session.add(r)
+ return r,True
+
+ def addSample(self, run, test, value):
+ s = Sample(run, test, value)
+ self.session.add(s)
+ return s
+
+ def addSamples(self, samples):
+ """addSamples([(run_id, test_id, value), ...]) -> None
+
+ Batch insert a list of samples."""
+
+ # Flush to keep session consistent.
+ self.session.flush()
+
+ for run_id,test_id,value in samples:
+ q = Sample.__table__.insert().values(RunID = run_id,
+ TestID = test_id,
+ Value = value)
+ self.session.execute(q)
+
+ def commit(self):
+ self.session.commit()
+
+ def rollback(self):
+ self.session.rollback()
+
+def importDataFromDict(db, data):
+ # FIXME: Validate data
+ machineData = data['Machine']
+ runData = data['Run']
+ testsData = data['Tests']
+
+ # Get the machine
+ # FIXME: Validate machine
+ machine,_ = db.getOrCreateMachine(machineData['Name'],
+ machineData['Info'].items())
+
+ # Accept 'Time' as an alias for 'Start Time'
+ if 'Start Time' not in runData and 'Time' in runData:
+ import time
+ t = time.strptime(runData['Time'],
+ "%a, %d %b %Y %H:%M:%S -0700 (PDT)")
+ runData['Start Time'] = time.strftime('%Y-%m-%d %H:%M', t)
+
+ # Create the run.
+ run,inserted = db.getOrCreateRun(machine,
+ runData.get('Start Time',''),
+ runData.get('End Time',''),
+ runData.get('Info',{}).items())
+ if not inserted:
+ return False,(machine,run)
+
+ # Batch load the set of tests instead of repeatedly querying to unique.
+ #
+ # FIXME: Add explicit config object.
+ test_info = {}
+ for id,k,v in db.session.query(TestInfo.test_id, TestInfo.key,
+ TestInfo.value):
+ test_info[id] = (str(k),str(v))
+
+ testMap = {}
+ for test_id,test_name in db.session.query(Test.id, Test.name):
+ info = test_info.get(test_id,[])
+ info.sort()
+ testMap[(str(test_name),tuple(info))] = test_id
+
+ # Create the tests up front, so we can resolve IDs.
+ test_ids = []
+ late_ids = []
+ for i,testData in enumerate(testsData):
+ name = str(testData['Name'])
+ info = [(str(k),str(v)) for k,v in testData['Info'].items()]
+ info.sort()
+ test_id = testMap.get((name,tuple(info)))
+ if test_id is None:
+ test,created = db.getOrCreateTest(testData['Name'],testData['Info'])
+ assert created
+ late_ids.append((i,test))
+ test_ids.append(test_id)
+
+ # Flush now to resolve test and run ids.
+ #
+ # FIXME: Surely there is a cleaner way to handle this?
+ db.session.flush()
+
+ if late_ids:
+ for i,t in late_ids:
+ test_ids[i] = t.id
+
+ db.addSamples([(run.id, test_id, value)
+ for test_id,testData in zip(test_ids, testsData)
+ for value in testData['Data']])
+
+ return True,(machine,run)
+
+def test_sa_db(dbpath):
+ if not dbpath.startswith('mysql://'):
+ dbpath = 'sqlite:///' + dbpath
+ engine = sqlalchemy.create_engine(dbpath)
+
+ Session = sqlalchemy.orm.sessionmaker(engine)
+ Session.configure(bind=engine)
+
+ session = Session()
+
+ m = session.query(Machine).first()
+ print m
+ print m.info
+
+ r = session.query(Run).first()
+ print r
+ print r.info
+
+ t = session.query(Test)[20]
+ print t
+ print t.info
+
+ s = session.query(Sample)[20]
+ print s
+
+ import time
+ start = time.time()
+ print
+ q = session.query(Sample)
+ q = q.filter(Sample.run_id == 994)
+ print
+ res = session.execute(q)
+ print res
+ N = 0
+ for row in res:
+ if N == 1:
+ print row
+ N += 1
+ print N, time.time() - start
+ print
+
+ start = time.time()
+ N = 0
+ for row in q:
+ if N == 1:
+ print row
+ N += 1
+ print N, time.time() - start
+
+def main():
+ global opts
+ from optparse import OptionParser
+ parser = OptionParser("usage: %prog dbpath")
+ opts,args = parser.parse_args()
+
+ if len(args) != 1:
+ parser.error("incorrect number of argments")
+
+ dbpath, = args
+
+ # Test the SQLAlchemy layer.
+ test_sa_db(dbpath)
+
+ # Test the PerfDB wrapper.
+ db = PerfDB(dbpath)
+
+ print "Opened %r" % dbpath
+
+ for m in db.machines():
+ print m
+ for r in db.runs(m):
+ print ' run - id:%r, start:%r,'\
+ ' # samples: %d.' % (r.id, r.start_time,
+ db.samples(run=r).count())
+
+if __name__ == '__main__':
+ main()
Added: zorg/trunk/lnt/viewer/Util.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/Util.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/Util.py (added)
+++ zorg/trunk/lnt/viewer/Util.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1,213 @@
+import colorsys
+import math
+
+def detectCPUs():
+ """
+ Detects the number of CPUs on a system. Cribbed from pp.
+ """
+ import os
+ # Linux, Unix and MacOS:
+ if hasattr(os, "sysconf"):
+ if os.sysconf_names.has_key("SC_NPROCESSORS_ONLN"):
+ # Linux & Unix:
+ ncpus = os.sysconf("SC_NPROCESSORS_ONLN")
+ if isinstance(ncpus, int) and ncpus > 0:
+ return ncpus
+ else: # OSX:
+ return int(os.popen2("sysctl -n hw.ncpu")[1].read())
+ # Windows:
+ if os.environ.has_key("NUMBER_OF_PROCESSORS"):
+ ncpus = int(os.environ["NUMBER_OF_PROCESSORS"]);
+ if ncpus > 0:
+ return ncpus
+ return 1 # Default
+
+def safediv(a, b, default=None):
+ try:
+ return a/b
+ except ZeroDivisionError:
+ return default
+
+def makeDarkColor(h):
+ h = h%1.
+ s = 0.95
+ v = 0.8
+ return colorsys.hsv_to_rgb(h,0.9+s*.1,v)
+
+def makeMediumColor(h):
+ h = h%1.
+ s = .68
+ v = 0.92
+ return colorsys.hsv_to_rgb(h,s,v)
+
+def makeLightColor(h):
+ h = h%1.
+ s = (0.5,0.4)[h>0.5 and h<0.8]
+ v = 1.0
+ return colorsys.hsv_to_rgb(h,s,v)
+
+def makeBetterColor(h):
+ h = math.cos(h*math.pi*.5)
+ s = .8 + ((math.cos(h * math.pi*.5) + 1)*.5) * .2
+ v = .88
+ return colorsys.hsv_to_rgb(h,s,v)
+
+class multidict:
+ def __init__(self, elts=()):
+ self.data = {}
+ for key,value in elts:
+ self[key] = value
+
+ def __getitem__(self, item):
+ return self.data[item]
+ def __setitem__(self, key, value):
+ if key in self.data:
+ self.data[key].append(value)
+ else:
+ self.data[key] = [value]
+ def items(self):
+ return self.data.items()
+ def values(self):
+ return self.data.values()
+ def keys(self):
+ return self.data.keys()
+ def __len__(self):
+ return len(self.data)
+ def get(self, key, default=None):
+ return self.data.get(key, default)
+
+def any_true(list, predicate):
+ for i in list:
+ if predicate(i):
+ return True
+ return False
+
+def any_false(list, predicate):
+ return any_true(list, lambda x: not predicate(x))
+
+def all_true(list, predicate):
+ return not any_false(list, predicate)
+
+def all_false(list, predicate):
+ return not any_true(list, predicate)
+
+def geometric_mean(l):
+ iPow = 1./len(l)
+ return reduce(lambda a,b: a*b, [v**iPow for v in l])
+
+def mean(l):
+ return sum(l) / len(l)
+
+def median(l):
+ l = list(l)
+ l.sort()
+ N = len(l)
+ return (l[(N - 1)//2] +
+ l[(N + 0)//2]) * .5
+
+def prependLines(prependStr, str):
+ return ('\n'+prependStr).join(str.splitlines())
+
+def pprint(object, useRepr=True):
+ def recur(ob):
+ return pprint(ob, useRepr)
+ def wrapString(prefix, string, suffix):
+ return '%s%s%s' % (prefix,
+ prependLines(' ' * len(prefix),
+ string),
+ suffix)
+ def pprintArgs(name, args):
+ return wrapString(name + '(', ',\n'.join(map(recur,args)), ')')
+
+ if isinstance(object, tuple):
+ return wrapString('(', ',\n'.join(map(recur,object)),
+ [')',',)'][len(object) == 1])
+ elif isinstance(object, list):
+ return wrapString('[', ',\n'.join(map(recur,object)), ']')
+ elif isinstance(object, set):
+ return pprintArgs('set', list(object))
+ elif isinstance(object, dict):
+ elts = []
+ for k,v in object.items():
+ kr = recur(k)
+ vr = recur(v)
+ elts.append('%s : %s' % (kr,
+ prependLines(' ' * (3 + len(kr.splitlines()[-1])),
+ vr)))
+ return wrapString('{', ',\n'.join(elts), '}')
+ else:
+ if useRepr:
+ return repr(object)
+ return str(object)
+
+def prefixAndPPrint(prefix, object, useRepr=True):
+ return prefix + prependLines(' '*len(prefix), pprint(object, useRepr))
+
+def clamp(v, minVal, maxVal):
+ return min(max(v, minVal), maxVal)
+
+def lerp(a,b,t):
+ t_ = 1. - t
+ return tuple([av*t_ + bv*t for av,bv in zip(a,b)])
+
+class PctCell:
+ # Color levels
+ kNeutralColor = (1,1,1)
+ kNegativeColor = (0,1,0)
+ kPositiveColor = (1,0,0)
+ # Invalid color
+ kNANColor = (.86,.86,.86)
+ kInvalidColor = (0,0,1)
+
+ def __init__(self, value, reverse=False, precision=2, delta=False):
+ if delta and isinstance(value, float):
+ value -= 1
+ self.value = value
+ self.reverse = reverse
+ self.precision = precision
+
+ def getColor(self):
+ v = self.value
+ if not isinstance(v, float):
+ return self.kNANColor
+
+ # Clamp value.
+ v = clamp(v, -1, 1)
+
+ if self.reverse:
+ v = -v
+ if v < 0:
+ c = self.kNegativeColor
+ else:
+ c = self.kPositiveColor
+ t = abs(v)
+
+ # Smooth mapping to put first 20% of change into 50% of range, although
+ # really we should compensate for luma.
+ t = math.sin((t ** .477) * math.pi * .5)
+ return lerp(self.kNeutralColor, c, t)
+
+ def getValue(self):
+ if not isinstance(self.value, float):
+ return self.value
+ return '%.*f%%' % (self.precision, self.value*100)
+
+ def render(self):
+ import quixote.html
+ r,g,b = [clamp(int(v*255), 0, 255)
+ for v in self.getColor()]
+ res = '<td bgcolor="#%02x%02x%02x">%s</td>' % (r,g,b, self.getValue())
+ return quixote.html.htmltext(res)
+
+
+def addOtherFormValues(form):
+ import quixote
+ request = quixote.get_request()
+ for name,value in request.form.items():
+ if form.get_widget(name) is None:
+ form.add(quixote.form.HiddenWidget, name, value=value)
+
+def sorted(l, *args, **kwargs):
+ l = list(l)
+ l.sort(*args, **kwargs)
+ return l
Added: zorg/trunk/lnt/viewer/__init__.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/__init__.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/__init__.py (added)
+++ zorg/trunk/lnt/viewer/__init__.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1 @@
+__all__ = []
Added: zorg/trunk/lnt/viewer/js/View2D.js
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/js/View2D.js?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/js/View2D.js (added)
+++ zorg/trunk/lnt/viewer/js/View2D.js Sat Mar 20 20:00:06 2010
@@ -0,0 +1,842 @@
+//===-- View2D.js - HTML5 Canvas Based 2D View/Graph Widget ---------------===//
+//
+// The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements a generic 2D view widget and a 2D graph widget on top of
+// it, using the HTML5 Canvas. It currently supports Firefox and Safari
+// (Chromium should work, but is untested).
+//
+// See the Graph2D implementation for details of how to extend the View2D
+// object.
+//
+// FIXME: Currently, this uses MooTools extensions, but I would like to rewrite
+// it in pure JS for more portability (e.g., use in Buildbot).
+//
+//===----------------------------------------------------------------------===//
+
+function lerp(a, b, t) {
+ return a * (1.0 - t) + b * t;
+}
+
+function clamp(a, lower, upper) {
+ return Math.max(lower, Math.min(upper, a));
+}
+
+function vec2_neg (a) { return [-a[0] , -a[1] ]; };
+function vec2_add (a, b) { return [a[0]+b[0], a[1]+b[1]]; };
+function vec2_addN(a, b) { return [a[0]+b , a[1]+b ]; };
+function vec2_sub (a, b) { return [a[0]-b[0], a[1]-b[1]]; };
+function vec2_subN(a, b) { return [a[0]-b , a[1]-b ]; };
+function vec2_mul (a, b) { return [a[0]*b[0], a[1]*b[1]]; };
+function vec2_mulN(a, b) { return [a[0]*b , a[1]*b ]; };
+function vec2_div (a, b) { return [a[0]/b[0], a[1]/b[1]]; };
+function vec2_divN(a, b) { return [a[0]/b , a[1]/b ]; };
+
+function vec3_neg (a) { return [-a[0] , -a[1] , -a[2] ]; };
+function vec3_add (a, b) { return [a[0]+b[0], a[1]+b[1], a[2]+b[2]]; };
+function vec3_addN(a, b) { return [a[0]+b , a[1]+b, a[2]+b ]; };
+function vec3_sub (a, b) { return [a[0]-b[0], a[1]-b[1], a[2]-b[2]]; };
+function vec3_subN(a, b) { return [a[0]-b , a[1]-b, a[2]-b ]; };
+function vec3_mul (a, b) { return [a[0]*b[0], a[1]*b[1], a[2]*b[2]]; };
+function vec3_mulN(a, b) { return [a[0]*b , a[1]*b, a[2]*b ]; };
+function vec3_div (a, b) { return [a[0]/b[0], a[1]/b[1], a[2]/b[2]]; };
+function vec3_divN(a, b) { return [a[0]/b , a[1]/b, a[2]/b ]; };
+
+function vec2_lerp(a, b, t) {
+ return [lerp(a[0], b[0], t), lerp(a[1], b[1], t)];
+}
+function vec2_mag(a) {
+ return a[0] * a[0] + a[1] * a[1];
+}
+function vec2_len(a) {
+ return Math.sqrt(vec2_mag(a));
+}
+
+function vec2_floor(a) {
+ return [Math.floor(a[0]), Math.floor(a[1])];
+}
+function vec2_ceil(a) {
+ return [Math.ceil(a[0]), Math.ceil(a[1])];
+}
+
+function vec3_floor(a) {
+ return [Math.floor(a[0]), Math.floor(a[1]), Math.floor(a[2])];
+}
+function vec3_ceil(a) {
+ return [Math.ceil(a[0]), Math.ceil(a[1]), Math.ceil(a[2])];
+}
+
+function vec2_log(a) {
+ return [Math.log(a[0]), Math.log(a[1])];
+}
+function vec2_pow(a, b) {
+ return [Math.pow(a[0], b[0]), Math.pow(a[1], b[1])];
+}
+function vec2_Npow(a, b) {
+ return [Math.pow(a, b[0]), Math.pow(a, b[1])];
+}
+function vec2_powN(a, b) {
+ return [Math.pow(a[0], b), Math.pow(a[1], b)];
+}
+
+function vec2_min(a, b) {
+ return [Math.min(a[0], b[0]), Math.min(a[1], b[1])];
+}
+function vec2_max(a, b) {
+ return [Math.max(a[0], b[0]), Math.max(a[1], b[1])];
+}
+function vec2_clamp(a, lower, upper) {
+ return [clamp(a[0], lower[0], upper[0]),
+ clamp(a[1], lower[1], upper[1])];
+}
+function vec2_clampN(a, lower, upper) {
+ return [clamp(a[0], lower, upper),
+ clamp(a[1], lower, upper)];
+}
+
+function vec3_min(a, b) {
+ return [Math.min(a[0], b[0]), Math.min(a[1], b[1]), Math.min(a[2], b[2])];
+}
+function vec3_max(a, b) {
+ return [Math.max(a[0], b[0]), Math.max(a[1], b[1]), Math.max(a[2], b[2])];
+}
+function vec3_clamp(a, lower, upper) {
+ return [clamp(a[0], lower[0], upper[0]),
+ clamp(a[1], lower[1], upper[1]),
+ clamp(a[2], lower[2], upper[2])];
+}
+function vec3_clampN(a, lower, upper) {
+ return [clamp(a[0], lower, upper),
+ clamp(a[1], lower, upper),
+ clamp(a[2], lower, upper)];
+}
+
+function vec2_cswap(a, swap) {
+ if (swap)
+ return [a[1],a[0]];
+ return a;
+}
+
+function col3_to_rgb(col) {
+ var norm = vec3_floor(vec3_clampN(vec3_mulN(col, 255), 0, 255));
+ return "rgb(" + norm[0] + "," + norm[1] + "," + norm[2] + ")";
+}
+
+function col4_to_rgba(col) {
+ var norm = vec3_floor(vec3_clampN(vec3_mulN(col, 255), 0, 255));
+ return "rgb(" + norm[0] + "," + norm[1] + "," + norm[2] + "," + col[3] + ")";
+}
+
+var ViewData = new Class ({
+ initialize: function(location, scale) {
+ if (!location)
+ location = [0, 0];
+ if (!scale)
+ scale = [1, 1];
+
+ this.location = location;
+ this.scale = scale;
+ },
+
+ copy: function() {
+ return new ViewData(this.location, this.scale);
+ },
+});
+
+var ViewAction = new Class ({
+ initialize: function(mode, v2d, start) {
+ this.mode = mode;
+ this.start = start;
+ this.vd = v2d.viewData.copy();
+ },
+
+ update: function(v2d, co) {
+ if (this.mode == 'p') {
+ var delta = vec2_sub(v2d.convertClientToNDC(co, this.vd),
+ v2d.convertClientToNDC(this.start, this.vd))
+ v2d.viewData.location = vec2_add(this.vd.location, delta);
+ } else {
+ var delta = vec2_sub(v2d.convertClientToNDC(co, this.vd),
+ v2d.convertClientToNDC(this.start, this.vd))
+ v2d.viewData.scale = vec2_Npow(Math.E,
+ vec2_addN(vec2_log(this.vd.scale),
+ delta[1]))
+ v2d.viewData.location = vec2_mul(this.vd.location,
+ vec2_div(v2d.viewData.scale,
+ this.vd.scale))
+ }
+
+ v2d.refresh();
+ },
+
+ complete: function(v2d, co) {
+ this.update(v2d, co);
+ },
+
+ abort: function(v2d) {
+ v2d.viewData = this.vd;
+ },
+});
+
+var View2D = new Class ({
+ initialize: function(canvasname) {
+ this.canvasname = canvasname
+ this.viewData = new ViewData();
+ this.size = [1, 1];
+ this.aspect = 1;
+ this.registered = false;
+
+ this.viewAction = null;
+
+ this.useWidgets = true;
+ this.previewPosition = [5, 5];
+ this.previewSize = [60, 60];
+
+ this.clearColor = [1, 1, 1];
+
+ // Bound once registered.
+ this.canvas = null;
+ },
+
+ registerEvents: function(canvas) {
+ if (this.registered)
+ return;
+
+ this.registered = true;
+
+ this.canvas = canvas;
+
+ // FIXME: Why do I have to do this?
+ var obj = this;
+
+ canvas.addEvent('mousedown', function(event) { obj.onMouseDown(event); })
+ canvas.addEvent('mousemove', function(event) { obj.onMouseMove(event); })
+ canvas.addEvent('mouseup', function(event) { obj.onMouseUp(event); })
+ canvas.addEvent('mousewheel', function(event) { obj.onMouseWheel(event); })
+
+ // FIXME: Capturing!
+ },
+
+ onMouseDown: function(event) {
+ pos = [event.client.x - this.canvas.offsetLeft,
+ this.size[1] - 1 - (event.client.y - this.canvas.offsetTop)];
+
+ if (this.viewAction != null)
+ this.viewAction.abort(this);
+
+ if (event.shift)
+ this.viewAction = new ViewAction('p', this, pos);
+ else if (event.alt || event.meta)
+ this.viewAction = new ViewAction('z', this, pos);
+ event.stop();
+ },
+ onMouseMove: function(event) {
+ pos = [event.client.x - this.canvas.offsetLeft,
+ this.size[1] - 1 - (event.client.y - this.canvas.offsetTop)];
+
+ if (this.viewAction != null)
+ this.viewAction.update(this, pos);
+ event.stop();
+ },
+ onMouseUp: function(event) {
+ pos = [event.client.x - this.canvas.offsetLeft,
+ this.size[1] - 1 - (event.client.y - this.canvas.offsetTop)];
+
+ if (this.viewAction != null)
+ this.viewAction.complete(this, pos);
+ this.viewAction = null;
+ event.stop();
+ },
+ onMouseWheel: function(event) {
+ if (this.viewAction == null) {
+ var factor = event.wheel;
+ if (event.shift)
+ factor *= .1;
+ var zoom = 1.0 + .03 * factor;
+ this.viewData.location = vec2_mulN(this.viewData.location, zoom);
+ this.viewData.scale = vec2_mulN(this.viewData.scale, zoom);
+ this.refresh();
+ }
+ event.stop();
+ },
+
+ setViewData: function(vd) {
+ // FIXME: Check equality and avoid refresh.
+ this.viewData = vd;
+ this.refresh();
+ },
+
+ refresh: function() {
+ // FIXME: Event loop?
+ this.draw();
+ },
+
+ // Coordinate conversion.
+
+ getAspectScale: function() {
+ if (this.aspect > 1) {
+ return [1.0 / this.aspect, 1.0];
+ } else {
+ return [1.0, this.aspect];
+ }
+ },
+
+ getPixelSize: function() {
+ return vec2_sub(this.convertClientToWorld([1,1]),
+ this.convertClientToWorld([0,0]));
+ },
+
+ convertClientToNDC: function(pt, vd) {
+ if (vd == null)
+ vd = this.viewData
+ return [pt[0] / this.size[0] * 2 - 1,
+ pt[1] / this.size[1] * 2 - 1];
+ },
+
+ convertClientToWorld: function(pt, vd) {
+ if (vd == null)
+ vd = this.viewData
+ pt = this.convertClientToNDC(pt, vd)
+ pt = vec2_sub(pt, vd.location);
+ pt = vec2_div(pt, vec2_mul(vd.scale, this.getAspectScale()));
+ return pt;
+ },
+
+ convertWorldToPreview: function(pt, pos, size) {
+ var asp_scale = this.getAspectScale();
+ pt = vec2_mul(pt, asp_scale);
+ pt = vec2_addN(pt, 1);
+ pt = vec2_mulN(pt, .5);
+ pt = vec2_mul(pt, size);
+ pt = vec2_add(pt, pos);
+ return pt;
+ },
+
+ setViewMatrix: function(ctx) {
+ ctx.scale(this.size[0], this.size[1]);
+ ctx.scale(.5, .5);
+ ctx.translate(1, 1);
+ ctx.translate(this.viewData.location[0], this.viewData.location[1]);
+ var scale = vec2_mul(this.viewData.scale, this.getAspectScale());
+ ctx.scale(scale[0], scale[1]);
+ },
+
+ setPreviewMatrix: function(ctx, pos, size) {
+ ctx.translate(pos[0], pos[1]);
+ ctx.scale(size[0], size[1]);
+ ctx.scale(.5, .5);
+ ctx.translate(1, 1);
+ var scale = this.getAspectScale();
+ ctx.scale(scale[0], scale[1]);
+ },
+
+ setWindowMatrix: function(ctx) {
+ ctx.translate(.5, .5);
+ ctx.translate(0, this.size[1]);
+ ctx.scale(1, -1);
+ },
+
+ draw: function() {
+ var canvas = document.getElementById(this.canvasname);
+ var ctx = canvas.getContext("2d");
+
+ this.registerEvents(canvas);
+
+ if (canvas.width != this.size[0] || canvas.height != this.size[1]) {
+ this.size = [canvas.width, canvas.height];
+ this.aspect = canvas.width / canvas.height;
+ this.previewPosition[0] = this.size[0] - this.previewSize[0] - 5;
+ this.on_size_change();
+ }
+
+ this.on_draw_start();
+
+ ctx.save();
+
+ // Clear and draw the view content.
+ ctx.save();
+ this.setWindowMatrix(ctx);
+
+ ctx.clearRect(0, 0, this.size[0], this.size[1]);
+ ctx.fillStyle = col3_to_rgb(this.clearColor);
+ ctx.fillRect(0, 0, this.size[0], this.size[1]);
+
+ this.setViewMatrix(ctx);
+ this.on_draw(canvas, ctx);
+ ctx.restore();
+
+ if (this.useWidgets)
+ this.drawPreview(canvas, ctx)
+
+ ctx.restore();
+ },
+
+ drawPreview: function(canvas, ctx) {
+ // Setup the preview context.
+ this.setWindowMatrix(ctx);
+
+ // Draw the preview area outline.
+ ctx.fillStyle = "rgba(128,128,128,.5)";
+ ctx.fillRect(this.previewPosition[0]-1, this.previewPosition[1]-1,
+ this.previewSize[0]+2, this.previewSize[1]+2);
+ ctx.lineWidth = 1;
+ ctx.strokeStyle = "rgb(0,0,0)";
+ ctx.strokeRect(this.previewPosition[0]-1, this.previewPosition[1]-1,
+ this.previewSize[0]+2, this.previewSize[1]+2);
+
+ // Compute the aspect corrected preview area.
+ var pv_size = [this.previewSize[0], this.previewSize[1]];
+ if (this.aspect > 1) {
+ pv_size[1] /= this.aspect;
+ } else {
+ pv_size[0] *= this.aspect;
+ }
+ var pv_pos = vec2_add(this.previewPosition,
+ vec2_mulN(vec2_sub(this.previewSize, pv_size), .5));
+
+ // Draw the preview, making sure to clip to the proper area.
+ ctx.save();
+ ctx.beginPath();
+ ctx.rect(pv_pos[0], pv_pos[1], pv_size[0], pv_size[1]);
+ ctx.clip();
+ ctx.closePath();
+
+ this.setPreviewMatrix(ctx, pv_pos, pv_size);
+ this.on_draw_preview(canvas, ctx);
+ ctx.restore();
+
+ // Draw the current view overlay.
+ //
+ // FIXME: Find a replacement for stippling.
+ ll = this.convertClientToWorld([0, 0])
+ ur = this.convertClientToWorld(this.size);
+
+ // Convert to pixel coordinates instead of drawing in content
+ // perspective.
+ ll = vec2_floor(this.convertWorldToPreview(ll, pv_pos, pv_size))
+ ur = vec2_ceil(this.convertWorldToPreview(ur, pv_pos, pv_size))
+ ll = vec2_clamp(ll, this.previewPosition,
+ vec2_add(this.previewPosition, this.previewSize))
+ ur = vec2_clamp(ur, this.previewPosition,
+ vec2_add(this.previewPosition, this.previewSize))
+
+ ctx.strokeStyle = "rgba(128,128,128,255)";
+ ctx.lineWidth = 1;
+ ctx.strokeRect(ll[0], ll[1], ur[0] - ll[0], ur[1] - ll[1]);
+ },
+
+ on_size_change: function() {},
+ on_draw_start: function() {},
+ on_draw: function(canvas, ctx) {},
+ on_draw_preview: function(canvas, ctx) {},
+});
+
+var View2DTest = new Class ({
+ Extends: View2D,
+
+ on_draw: function(canvas, ctx) {
+ ctx.fillStyle = "rgb(255,255,255)";
+ ctx.fillRect(-1000, -1000, 2000, 20000);
+
+ ctx.lineWidth = .01;
+ ctx.strokeTyle = "rgb(0,200,0)";
+ ctx.strokeRect(-1, -1, 2, 2);
+
+ ctx.fillStyle = "rgb(200,0,0)";
+ ctx.fillRect(-.8, -.8, 1, 1);
+
+ ctx.fillStyle = "rgb(0,0,200)";
+ ctx.beginPath();
+ ctx.arc(0, 0, .5, 0, 2 * Math.PI, false);
+ ctx.fill();
+ ctx.closePath();
+ },
+
+ on_draw_preview: function(canvas, ctx) {
+ ctx.fillStyle = "rgba(255,255,255,.4)";
+ ctx.fillRect(-1000, -1000, 2000, 20000);
+
+ ctx.lineWidth = .01;
+ ctx.strokeTyle = "rgba(0,200,0,.4)";
+ ctx.strokeRect(-1, -1, 2, 2);
+
+ ctx.fillStyle = "rgba(200,0,0,.4)";
+ ctx.fillRect(-.8, -.8, 1, 1);
+
+ ctx.fillStyle = "rgba(0,0,200,.4)";
+ ctx.beginPath();
+ ctx.arc(0, 0, .5, 0, 2 * Math.PI, false);
+ ctx.fill();
+ ctx.closePath();
+ },
+});
+
+var Graph2D_GraphInfo = new Class ({
+ initialize: function() {
+ this.xAxisH = 0;
+ this.yAxisW = 0;
+ this.ll = [0, 0];
+ this.ur = [1, 1];
+ },
+
+ toNDC: function(pt) {
+ return [2 * (pt[0] - this.ll[0]) / (this.ur[0] - this.ll[0]) - 1,
+ 2 * (pt[1] - this.ll[1]) / (this.ur[1] - this.ll[1]) - 1];
+ },
+
+ fromNDC: function(pt) {
+ return [this.ll[0] + (this.ur[0] - this.ll[0]) * (pt[0] + 1) * .5,
+ this.ll[1] + (this.ur[1] - this.ll[1]) * (pt[1] + 1) * .5];
+ },
+});
+
+var Graph2D_PlotStyle = new Class ({
+ initialize: function() {},
+
+ plot: function(graph, ctx, data) {},
+});
+
+var Graph2D_LinePlotStyle = new Class ({
+ Extends: Graph2D_PlotStyle,
+
+ initialize: function(width, color) {
+ if (!width)
+ width = 1;
+ if (!color)
+ color = [0,0,0];
+
+ this.parent();
+ this.width = width;
+ this.color = color;
+ },
+
+ plot: function(graph, ctx, data) {
+ if (data.length === 0)
+ return;
+
+ ctx.beginPath();
+ var co = graph.graphInfo.toNDC(data[0]);
+ ctx.moveTo(co[0], co[1]);
+ for (var i = 1, e = data.length; i != e; ++i) {
+ var co = graph.graphInfo.toNDC(data[i]);
+ ctx.lineTo(co[0], co[1]);
+ }
+ ctx.lineWidth = this.width * (graph.getPixelSize()[0] + graph.getPixelSize()[1]) * .5;
+ ctx.strokeStyle = col3_to_rgb(this.color);
+ ctx.stroke();
+ },
+});
+
+var Graph2D_Axis = new Class ({
+ // Static Methods
+ formats: {
+ normal: function(value, iDigits, fDigits) {
+ // FIXME: iDigits?
+ return value.toFixed(fDigits);
+ },
+ day: function(value, iDigits, fDigits) {
+ var date = new Date(value * 1000.);
+ var res = date.getUTCFullYear();
+ res += "-" + (date.getUTCMonth() + 1);
+ res += "-" + (date.getUTCDate() + 1);
+ return res;
+ },
+ },
+
+ initialize: function(dir, format) {
+ if (!format)
+ format = this.formats.normal;
+
+ this.dir = dir;
+ this.format = format;
+ },
+
+ draw: function(graph, ctx, ll, ur, mainUR) {
+ var dir = this.dir, ndir = 1 - this.dir;
+ var vMin = ll[dir];
+ var vMax = ur[dir];
+ var near = ll[ndir];
+ var far = ur[ndir];
+ var border = mainUR[ndir];
+
+ var line_base = (graph.getPixelSize()[0] + graph.getPixelSize()[1]) * .5;
+ ctx.lineWidth = 2 * line_base;
+ ctx.strokeStyle = "rgb(0,0,0)";
+
+ ctx.beginPath();
+ var co = vec2_cswap([vMin, far], dir);
+ co = graph.graphInfo.toNDC(co);
+ ctx.moveTo(co[0], co[1]);
+ var co = vec2_cswap([vMax, far], dir);
+ co = graph.graphInfo.toNDC(co);
+ ctx.lineTo(co[0], co[1]);
+ ctx.stroke();
+
+ var delta = vMax - vMin;
+ var steps = Math.floor(Math.log(delta) / Math.log(10));
+ if (delta / Math.pow(10, steps) >= 5.0) {
+ var size = .5;
+ } else if (delta / Math.pow(10, steps) >= 2.5) {
+ var size = .25;
+ } else {
+ var size = .1;
+ }
+ size *= Math.pow(10, steps);
+
+ if (steps <= 0) {
+ var iDigits = 0, fDigits = 1 + Math.abs(steps);
+ } else {
+ var iDigits = steps, fDigits = 0;
+ }
+
+ var start = Math.ceil(vMin / size);
+ var end = Math.ceil(vMax / size);
+
+ // FIXME: Draw in window coordinates to make crisper.
+
+ // FIXME: Draw grid in layers to avoid ugly overlaps.
+
+ for (var i = start; i != end; ++i) {
+ if (i == 0) {
+ ctx.lineWidth = 3 * line_base;
+ var p = .5;
+ } else if (!(i & 1)) {
+ ctx.lineWidth = 2 * line_base;
+ var p = .5;
+ } else {
+ ctx.lineWidth = 1 * line_base;
+ var p = .75;
+ }
+
+ ctx.beginPath();
+ var co = vec2_cswap([i * size, lerp(near, far, p)], dir);
+ co = graph.graphInfo.toNDC(co);
+ ctx.moveTo(co[0], co[1]);
+ var co = vec2_cswap([i * size, far], dir);
+ co = graph.graphInfo.toNDC(co);
+ ctx.lineTo(co[0], co[1]);
+ ctx.stroke();
+ }
+
+ for (var alt = 0; alt < 2; ++alt) {
+ if (alt)
+ ctx.strokeStyle = "rgba(190,190,190,.5)";
+ else
+ ctx.strokeStyle = "rgba(128,128,128,.5)";
+ ctx.lineWidth = 1 * line_base;
+ ctx.beginPath();
+ for (var i = start; i != end; ++i) {
+ if (i == 0)
+ continue;
+ if ((i & 1) == alt) {
+ var co = vec2_cswap([i * size, far], dir);
+ co = graph.graphInfo.toNDC(co);
+ ctx.moveTo(co[0], co[1]);
+ var co = vec2_cswap([i * size, border], dir);
+ co = graph.graphInfo.toNDC(co);
+ ctx.lineTo(co[0], co[1]);
+ }
+ }
+ ctx.stroke();
+
+ if (start <= 0 && 0 < end) {
+ ctx.beginPath();
+ var co = vec2_cswap([0, far], dir);
+ co = graph.graphInfo.toNDC(co);
+ ctx.moveTo(co[0], co[1]);
+ var co = vec2_cswap([0, border], dir);
+ co = graph.graphInfo.toNDC(co);
+ ctx.lineTo(co[0], co[1]);
+ ctx.strokeStyle = "rgba(64,64,64,.5)";
+ ctx.lineWidth = 3 * line_base;
+ ctx.stroke();
+ }
+ }
+
+ // FIXME: Draw this in screen coordinates, and stop being stupid. Also,
+ // figure out font height?
+ if (this.dir == 1) {
+ var offset = [-.5, -.25];
+ } else {
+ var offset = [-.5, 1.1];
+ }
+ ctx.fillStyle = "rgb(0,0,0)";
+ var pxl = graph.getPixelSize();
+ for (var i = start; i != end; ++i) {
+ if ((i & 1) == 0) {
+ var label = this.format(i * size, iDigits, fDigits);
+ ctx.save();
+ var co = vec2_cswap([i*size, lerp(near, far, .5)], dir);
+ co = graph.graphInfo.toNDC(co);
+ ctx.translate(co[0], co[1]);
+ ctx.scale(pxl[0], -pxl[1]);
+ // FIXME: Abstract.
+ var bb_w = label.length * 5;
+ if (ctx.measureText != null)
+ bb_w = ctx.measureText(label).width;
+ var bb_h = 12;
+ // FIXME: Abstract? Or ignore.
+ if (ctx.fillText != null) {
+ ctx.fillText(label, bb_w*offset[0], bb_h*offset[1]);
+ } else if (ctx.mozDrawText != null) {
+ ctx.translate(bb_w*offset[0], bb_h*offset[1]);
+ ctx.mozDrawText(label);
+ }
+ ctx.restore();
+ }
+ }
+ },
+});
+
+var Graph2D = new Class ({
+ Extends: View2D,
+
+ initialize: function(canvasname) {
+ this.parent(canvasname);
+
+ this.useWidgets = false;
+ this.plots = [];
+ this.graphInfo = null;
+ this.xAxis = new Graph2D_Axis(0);
+ this.yAxis = new Graph2D_Axis(1);
+ this.debugText = null;
+
+ this.clearColor = [.8, .8, .8];
+ },
+
+ //
+
+ graphChanged: function() {
+ this.graphInfo = null;
+ // FIXME: Need event loop.
+ this.refresh();
+ },
+
+ layoutGraph: function() {
+ var gi = new Graph2D_GraphInfo();
+
+ gi.xAxisH = 40;
+ gi.yAxisW = 40;
+
+ var min = null, max = null;
+ for (var i = 0, e = this.plots.length; i != e; ++i) {
+ var data = this.plots[i][0];
+ for (var i2 = 0, e2 = data.length; i2 != e2; ++i2) {
+ if (min == null)
+ min = data[i2];
+ else
+ min = vec2_min(min, data[i2]);
+
+ if (max == null)
+ max = data[i2];
+ else
+ max = vec2_max(max, data[i2]);
+ }
+ }
+
+ if (min === null)
+ min = [0, 0];
+ if (max === null)
+ max = [0, 0];
+ if (Math.abs(max[0] - min[0]) < .001)
+ max[0] += 1;
+ if (Math.abs(max[1] - min[1]) < .001)
+ max[1] += 1;
+
+ // Set graph transform to the [min,max] rect to the content area with
+ // some padding.
+ //
+ // FIXME: Add real mat3 and implement this properly.
+ var pad = 5;
+ var vd = new ViewData();
+ var ll_target = this.convertClientToWorld([gi.yAxisW + pad, gi.xAxisH + pad], vd);
+ var ur_target = this.convertClientToWorld(vec2_subN(this.size, pad), vd);
+ var target_size = vec2_sub(ur_target, ll_target);
+ var target_center = vec2_add(ll_target, vec2_mulN(target_size, .5));
+
+ var center = vec2_mulN(vec2_add(min, max), .5);
+ var size = vec2_sub(max, min);
+
+ var scale = vec2_mulN(target_size, .5);
+ size = vec2_div(size, scale);
+ center = vec2_sub(center, vec2_mulN(vec2_mul(target_center, size), .5));
+
+ gi.ll = vec2_sub(center, vec2_mulN(size, .5));
+ gi.ur = vec2_add(center, vec2_mulN(size, .5));
+
+ return gi;
+ },
+
+ //
+
+ convertClientToGraph: function(pt) {
+ return this.graphInfo.fromNDC(this.convertClientToWorld(pt));
+ },
+
+ //
+
+ on_size_change: function() {
+ this.graphInfo = null;
+ },
+
+ on_draw_start: function() {
+ if (!this.graphInfo)
+ this.graphInfo = this.layoutGraph();
+ },
+
+ on_draw: function(canvas, ctx) {
+ var gi = this.graphInfo;
+ var w = this.size[0], h = this.size[1];
+
+ this.xAxis.draw(this, ctx,
+ this.convertClientToGraph([gi.yAxisW, 0]),
+ this.convertClientToGraph([w, gi.xAxisH]),
+ this.convertClientToGraph([w, h]))
+ this.yAxis.draw(this, ctx,
+ this.convertClientToGraph([0, gi.xAxisH]),
+ this.convertClientToGraph([gi.yAxisW, h]),
+ this.convertClientToGraph([w, h]))
+
+ if (this.debugText != null) {
+ ctx.save();
+ ctx.setTransform(1, 0, 0, 1, 0, 0);
+ ctx.fillText(this.debugText, this.size[0]/2 + 10, this.size[1]/2 + 10);
+ ctx.restore();
+ }
+
+ // Draw the contents.
+ ctx.save();
+ ctx.beginPath();
+ var content_ll = this.convertClientToWorld([gi.yAxisW, gi.xAxisH]);
+ var content_ur = this.convertClientToWorld(this.size);
+ ctx.rect(content_ll[0], content_ll[1],
+ content_ur[0]-content_ll[0], content_ur[1]-content_ll[1]);
+ ctx.clip();
+
+ for (var i = 0, e = this.plots.length; i != e; ++i) {
+ var data = this.plots[i][0];
+ var style = this.plots[i][1];
+ style.plot(this, ctx, data);
+ }
+ ctx.restore();
+ },
+
+ // Client API.
+
+ clearPlots: function() {
+ this.plots = [];
+ this.graphChanged();
+ },
+
+ addPlot: function(data, style) {
+ if (!style)
+ style = new Graph2D_LinePlotStyle(1);
+ this.plots.push( [data, style] );
+ this.graphChanged();
+ },
+});
Added: zorg/trunk/lnt/viewer/js/View2DTest.html
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/js/View2DTest.html?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/js/View2DTest.html (added)
+++ zorg/trunk/lnt/viewer/js/View2DTest.html Sat Mar 20 20:00:06 2010
@@ -0,0 +1,57 @@
+<html>
+ <head>
+ <title>View2D Test</title>
+ <script src="../resources/mootools-1.2.4-core-nc.js"></script>
+ <script src="View2D.js"></script>
+ <script type="text/javascript">
+var viewA, viewB, graphA;
+
+function init() {
+ viewA = new View2DTest("view2d");
+ viewA.viewData.location = [-.2, -.15];
+ viewA.viewData.scale = [2, 2];
+ viewA.draw();
+
+ viewB = new View2DTest("view2d_2");
+ viewB.viewData.location = [-.2, -.15];
+ viewB.viewData.scale = [.5, .5];
+ viewB.draw();
+
+ var pts_0 = [ [0, 0], [1, 1.3], [2, 2.2], [3, 1.3], [4, 4],
+ [5,3], [6, 4], [7, 4.4]];
+ pts_0 = pts_0.map(function(item, index, array) {
+ return vec2_mulN(item, .1);
+ });
+ var pts_1 = pts_0.map(function(item, index, array) {
+ return [item[0], .2 + item[1] * (1 - item[0] / 2)];
+ });
+ var pts_2 = pts_0.map(function(item, index, array) {
+ return [item[0], 1 - item[1]];
+ });
+ graphA = new Graph2D("graph2d");
+if (1) {
+ graphA.addPlot(pts_0, new Graph2D_LinePlotStyle(2));
+ graphA.addPlot(pts_1);
+ graphA.addPlot(pts_2, new Graph2D_LinePlotStyle(null, [1,0,0]));
+} else {
+ graphA.addPlot([[1248617856.000000,1.398600],[1248960469.000000,1.241700],[1249396047.000000,1.214300],[1249488792.000000,1.211600],[1249564554.000000,1.239500],[1249732514.000000,1.239200],[1249819044.000000,1.236300],[1249910746.000000,1.234200],[1249996733.000000,1.236000],[1250083258.000000,1.252600],[1250255237.000000,1.260400],[1250341501.000000,1.253500],[1250427966.000000,1.262600],[1250514329.000000,1.261600],[1250600211.000000,1.249100],[1250686801.000000,1.250700],[1250773065.000000,1.255400],[1250859519.000000,1.246300],[1250946756.000000,1.245900],[1251032239.000000,1.238200],[1251118837.000000,1.238000],[1251205034.000000,1.241900],[1251292148.000000,1.236500],[1251379354.000000,1.230200],[1251466226.000000,1.228700],[1251552218.000000,1.233600],[1251638633.000000,1.222400],[1251725230.000000,1.222100],[1251811608.000000,1.239100],[1251898565.000000,1.230200],[1251984695.000000,1.241700],[1252070572.000000,1.205800],[1252157268.000000,1.211100],[1252243914.
000000,1.210300],[1252330714.000000,1.206000],[1252416809.000000,1.210800],[1252590190.000000,1.216800],[1252675459.000000,1.214400],[1252762486.000000,1.220700],[1252886652.000000,1.221700],[1252935938.000000,1.215200],[1253021864.000000,1.213800],[1253108596.000000,1.213800],[1253194864.000000,1.231900],[1253281344.000000,1.232000],[1253366664.000000,1.238400],[1253453169.000000,1.232500],[1253539839.000000,1.230100],[1253627357.000000,1.232600],[1253713381.000000,1.235700],[1253799754.000000,1.232400],[1253886021.000000,1.241900],[1253972269.000000,1.229700],[1254058753.000000,1.229100],[1254145361.000000,1.284000],[1254231276.000000,1.295200],[1254317581.000000,1.299800],[1254405166.000000,1.340600],[1254490117.000000,1.305300],[1254576631.000000,1.304100],[1254662979.000000,1.303700],[1254749498.000000,1.306800],[1254835808.000000,1.306200],[1254922190.000000,1.270100],[1255008713.000000,1.270900],[1255094924.000000,1.267300],[1255181629.000000,1.265100],[1255267916.000
000,1.283300],[1255354566.000000,1.285500],[1255440921.000000,1.287100],[1255526237.000000,1.269800],[1255613420.000000,1.318400],[1255700706.000000,1.301600],[1255786760.000000,1.305000],[1255873293.000000,1.305000],[1255959875.000000,1.438800],[1256046280.000000,1.437400],[1256132389.000000,1.439100],[1256218592.000000,1.461200],[1256304662.000000,1.464100],[1256391812.000000,1.462300],[1256478151.000000,1.469600],[1256564495.000000,1.472300],[1256823554.000000,1.474000],[1256909311.000000,1.477200],[1256996077.000000,1.459000],[1257082516.000000,1.426100],[1257258543.000000,1.433700],[1257344527.000000,1.452600],[1257431075.000000,1.440700],[1257516720.000000,1.443300],[1257602894.000000,1.503300],[1257863543.000000,1.550200],[1257948938.000000,1.532500],[1258035827.000000,1.534900],[1258122304.000000,1.535600],[1258207973.000000,1.546700],[1258294248.000000,1.534700],[1258382337.000000,1.544000],[1258551986.000000,1.509100],[1258640333.000000,1.554300],[1258728912.000000
,1.541100],[1258815149.000000,1.602700],[1258901720.000000,1.685900]]);
+graphA.xAxis.format = graphA.xAxis.formats.day;
+}
+ graphA.draw();
+}
+ </script>
+ </head>
+ <body onload="init()" bgcolor="#AAAAAA">
+ <canvas id="graph2d" width="600" height="400"></canvas>
+ <br>
+ <table>
+ <tr>
+ <td><canvas id="view2d" width="300" height="200"></canvas></td>
+ <td><canvas id="view2d_2" width="200" height="350"></canvas></td>
+ </tr>
+ </table>
+ Shift-Left Mouse: Pan<br>
+ Alt/Meta-Left Mouse: Zoom<br>
+ Wheel: Zoom (<i>Shift Slows</i>)<br>
+ </body>
+</html>
Added: zorg/trunk/lnt/viewer/machines.ptl
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/machines.ptl?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/machines.ptl (added)
+++ zorg/trunk/lnt/viewer/machines.ptl Sat Mar 20 20:00:06 2010
@@ -0,0 +1,90 @@
+# -*- python -*-
+
+import sys
+from quixote import get_response, redirect
+from quixote.directory import Directory
+from quixote.errors import TraversalError
+
+class MachineUI(Directory):
+ _q_exports = [""]
+
+ def __init__(self, root, idstr):
+ self.root = root
+ try:
+ self.id = int(idstr)
+ except ValueError, exc:
+ raise TraversalError(str(exc))
+
+
+ def _q_index [html] (self):
+ # Get a DB conntection
+ db = self.root.getDB()
+
+ m = db.getMachine(self.id)
+
+ self.root.getHeader("zorg:machine:%d" % self.id, '../..')
+
+ """
+ <h2>Machine: %s:%d</h2>
+ """ % (m.name, m.number)
+
+
+ # Machine Info
+ """
+ <table border=1 cellborder=1>
+ <tr>
+ <th>Key</th>
+ <th>Value</th>
+ </tr>
+ </thead>
+ """
+ for mi in m.info.values():
+ """
+ <tr>
+ <td>%s</td>
+ <td>%s</td>
+ </tr>""" % (mi.key, mi.value)
+ """
+ </table>
+ """
+
+ # List runs
+ """
+ <h3>Associated Runs</h3>
+ <table class="sortable" border=1 cellborder=1>
+ <thead>
+ <tr>
+ <th>Run ID</th>
+ <th>Start Time</th>
+ <th>End Time</th>
+ </tr>
+ </thead>
+ """
+ for r in db.runs(machine=m):
+ """
+ <tr>
+ <td><a href="../../runs/%d/">%d</a></td>
+ <td>%s</td>
+ <td>%s</td>
+ </tr>
+ """ % (r.id, r.id, r.start_time, r.end_time)
+ """
+ </table>
+ """
+
+ self.root.getFooter()
+
+class MachinesDirectory(Directory):
+ _q_exports = [""]
+
+ def __init__(self, root):
+ Directory.__init__(self)
+ self.root = root
+
+ def _q_index [plain] (self):
+ """
+ machine access
+ """
+
+ def _q_lookup(self, component):
+ return MachineUI(self.root, component)
Added: zorg/trunk/lnt/viewer/nightlytest.ptl
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/nightlytest.ptl?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/nightlytest.ptl (added)
+++ zorg/trunk/lnt/viewer/nightlytest.ptl Sat Mar 20 20:00:06 2010
@@ -0,0 +1,800 @@
+# -*- python -*-
+
+import sys
+import time
+
+import quixote
+from quixote.directory import Directory
+from quixote.errors import TraversalError
+
+import Util, NTStyleBrowser
+from Util import safediv
+from NTUtil import *
+
+from PerfDB import Machine, Run
+
+class NightlyTestRunUI(NTStyleBrowser.TestRunUI):
+ _q_exports = ["", "graphSingle"]
+
+ def __init__(self, *args, **kwargs):
+ NTStyleBrowser.TestRunUI.__init__(self, *args, **kwargs)
+ self.popupDepth = 0
+
+ def getTags(self):
+ return (None, 'nightlytest')
+
+ def getTitle(self):
+ return 'LLVM Nightly Test Results'
+
+ def getHeaderTitle(self):
+ return 'LLVM Nightly Test Results'
+
+ def getParameters(self):
+ return ()
+
+ def renderFullContents [html] (self, db, run, compareTo, summary):
+ """<p>"""
+ self.getAllResults(db, run, compareTo, summary)
+
+ def renderBriefContents [html] (self, db, run, compareTo, summary):
+ if compareTo:
+ self.getComparisonPopups(db, run, compareTo, summary)
+
+ def renderCommonContents [html] (self, db, run, compareTo, summary):
+ # Test suite failure information.
+
+ failures = self.getFailuresForRun(db, run, summary)
+
+ if compareTo is not None:
+ prevFailures = self.getFailuresForRun(db, compareTo, summary)
+ newFailures = [(name, list(set(items) -
+ set(prevFailures.get(name,[]))))
+ for name,items in failures.items()]
+ newFailures = dict([(name, items)
+ for name,items in newFailures
+ if items])
+
+ if newFailures:
+ newFailures = sorted(newFailures.items())
+ self.renderTestSuiteFailures('newTSFailures',
+ '<b>New Test Suite Failures</b>',
+ newFailures)
+
+ failures = sorted(failures.items())
+ self.renderTestSuiteFailures('tsFailures',
+ 'Test Suite Failures', failures)
+
+ def renderTestSuiteFailures [html] (self, id, title, failures):
+ numFailures = sum([len(items) for _,items in failures])
+ title = '%d %s (All)' % (numFailures, title)
+
+ self.renderPopupBegin(id, title, True)
+ self.renderPopupBegin(id+'.all', 'All', True)
+ for name,items in failures:
+ """
+ %s <font color="grey">[%s]</font><br>
+ """ % (name, ', '.join(items))
+ self.renderPopupEnd()
+
+ # Also show failure by type.
+ byType = Util.multidict([(item,name)
+ for name,items in failures
+ for item in items])
+ byType = sorted(byType.items())
+
+ for i,(item,names) in enumerate(byType):
+ title = '%d %s' % (len(names), item)
+ self.renderPopupBegin(id+'.%d' % i, title, True)
+ for name in names:
+ """
+ %s<br>""" % name
+ self.renderPopupEnd()
+
+ self.renderPopupEnd()
+
+ def renderPopupBegin [html] (self, id, title, hidden):
+ self.popupDepth += 1
+ """\
+ <p>
+ <a href="javascript://" onclick="toggleLayer('%s')"; id="%s_">(%s) %s</a>
+ <div id="%s" style="display: %s;" class="hideable_%d">
+ """ % (id, id, ("+","-")[hidden], title, id, ("","none")[hidden],
+ self.popupDepth)
+ def renderPopupEnd [html] (self):
+ """
+ </div>"""
+ self.popupDepth -= 1
+
+ def getFailuresForRun(self, db, run, summary):
+ failures = Util.multidict()
+ for keyname,title in kTSKeys.items():
+ for testname in summary.testNames:
+ fullname = 'nightlytest.' + testname + '.' + keyname + '.success'
+ t = summary.testMap.get(str(fullname))
+ if t is None:
+ continue
+ samples = summary.getRunSamples(run).get(t.id)
+ if not samples or samples[0]:
+ continue
+ failures[testname] = title
+ return failures
+
+ def getAllResults [html] (self, db, run, compareTo, summary):
+ columns = [('GCCAS', 'gcc.compile.time', None, ()),
+ ('Bitcode','bc.compile.size', None, ()),
+ ('LLC<br>compile','llc.compile.time', None, ('bc.compile.size',)),
+ ('LLC-BETA<br>compile','llc-beta.compile.time', None, ('bc.compile.size',)),
+ ('JIT<br>codegen','jit.compile.time', None, ('bc.compile.size',)),
+ ('GCC','gcc.exec.time', None, ('gcc.compile.time',)),
+ ('CBE','cbe.exec.time', None, ('bc.compile.size',)),
+ ('LLC','llc.exec.time', None, ('llc.compile.time',)),
+ ('LLC-BETA','llc-beta.exec.time', None, ('llc-beta.compile.time',)),
+ ('JIT','jit.exec.time', None, ('jit.compile.time',)),
+ ('GCC/CBE','gcc.exec.time','cbe.exec.time', ()),
+ ('GCC/LLC','gcc.exec.time','llc.exec.time', ()),
+ ('GCC/LLC-BETA','gcc.exec.time','llc-beta.exec.time',()),
+ ('LLC/LLC-BETA','llc.exec.time','llc-beta.exec.time',())]
+
+ # Add interface to hiding columns by test or column type.
+ keyIndices = Util.multidict()
+ ratioIndices = []
+ pctIndices = []
+
+ idx = 1
+ for info in columns:
+ isCmp = info[2] is not None
+ key = str(info[1]).split(str('.'))[0]
+ if not isCmp:
+ keyIndices[key] = idx
+ keyIndices[key] = idx + 1
+ pctIndices.append(idx + 1)
+ idx += 2
+ else:
+ keyIndices[key] = idx
+ ratioIndices.append(idx)
+ idx += 1
+ """
+ <form>
+ <table border="1">
+ <thead>
+ <tr>
+ <th>Column Visibility</th>
+ <th>GCC</th>
+ <th>LLC</th>
+ <th>CBE</th>
+ <th>JIT</th>
+ <th>LLC-BETA</th>
+ <th>Percentages</th>
+ <th>Ratios</th>
+ </tr>
+ </thead>
+ <tr>
+ <td>Enabled</td>
+ <td><input type='checkbox' onClick='javascript:show_hide_column("programs", [%s]);' checked></td>
+ <td><input type='checkbox' onClick='javascript:show_hide_column("programs", [%s]);' checked></td>
+ <td><input type='checkbox' onClick='javascript:show_hide_column("programs", [%s]);' checked></td>
+ <td><input type='checkbox' onClick='javascript:show_hide_column("programs", [%s]);' checked></td>
+ <td><input type='checkbox' onClick='javascript:show_hide_column("programs", [%s]);' checked></td>
+ <td><input type='checkbox' onClick='javascript:show_hide_column("programs", [%s]);' checked></td>
+ <td><input type='checkbox' onClick='javascript:show_hide_column("programs", [%s]);' checked></td>
+ </tr>
+ </table>
+ </form>
+ """ % (', '.join(map(str, keyIndices['gcc'])),
+ ', '.join(map(str, keyIndices['llc'])),
+ ', '.join(map(str, keyIndices['cbe'])),
+ ', '.join(map(str, keyIndices['jit'])),
+ ', '.join(map(str, keyIndices['llc-beta'])),
+ ', '.join(map(str, pctIndices)),
+ ', '.join(map(str, ratioIndices)))
+
+ # The main table.
+ """
+ <table class="sortable" border="1" cellspacing="0" cellpadding="0" id="programs">
+ <thead>
+ <tr>
+ <th>Program</th>
+ """
+ for name,key,cmp,dependsOn in columns:
+ if cmp is None:
+ """
+ <th>%s</th>
+ <th>%%<br>change<br>in<br>%s</th>
+ """ % (name, name)
+ else:
+ """<th>%s</th>""" % name
+ """
+ </tr>
+ </thead>
+ """
+
+ runSamples = summary.getRunSamples(run)
+ prevSamples = summary.getRunSamples(compareTo)
+ testNames = list(summary.testNames)
+ testNames.sort(key = lambda x: x.lower())
+ for testName in testNames:
+ # FIXME: We need some id for the "program". The dotted name system
+ # solves this...
+ fullname = str('nightlytest.' + testName + '.' +
+ 'gcc.compile.success')
+ t = summary.testMap.get(fullname)
+ assert t
+ """
+ <tr>
+ <td><a href="../programs/%s/">%s</a></td>
+ """ % (t.id, testName,)
+
+ for name,key,cmp,dependsOn in columns:
+ if cmp is None:
+ fullname = str('nightlytest.' + testName + '.' + key)
+ t = summary.testMap.get(fullname)
+ if t is None:
+ current = prev = None
+ else:
+ current = runSamples.get(t.id)
+ prev = prevSamples.get(t.id)
+ if current:
+ value = current[0]
+ if key.endswith('size'):
+ """<td>%d</td>""" % int(value)
+ else:
+ """<td>%.4f</td>""" % value
+ if prev:
+ pct = safediv(value, prev[0],
+ '<center><font size=-2>nan</font></center>')
+ else:
+ pct = 'N/A'
+ else:
+ # Only mark failure if nothing we depend on failed.
+ failed = True
+ for d in dependsOn:
+ t = summary.testMap.get(str('nightlytest.' + testName + '.' + d))
+ if not t or not runSamples.get(t.id):
+ failed = False
+ break
+ if failed:
+ """<td bgcolor="#FF0000">*</td>"""
+ else:
+ """<td>N/A</td>"""
+ pct = 'N/A'
+ Util.PctCell(pct, delta=True).render()
+ else:
+ tNum = summary.testMap.get(str('nightlytest.' + testName + '.' + key))
+ tDen = summary.testMap.get(str('nightlytest.' + testName + '.' + cmp))
+ if tNum is None or tDen is None:
+ num = den = None
+ else:
+ num = runSamples.get(tNum.id)
+ den = runSamples.get(tDen.id)
+ if num and den:
+ pct = safediv(num[0], den[0])
+ if pct is None:
+ """<td>N/A</td>"""
+ else:
+ """<td>%.2f</td>""" % (pct,)
+ else:
+ """<td>N/A</td>"""
+ """
+ </tr>
+ """
+ """
+ </table>
+ """
+
+ def getComparisonPopups [html] (self, db, run, compareTo, summary):
+ runSamples = summary.getRunSamples(run)
+ prevSamples = summary.getRunSamples(compareTo)
+
+ for i,(name,key) in enumerate(kComparisonKinds):
+ if not key:
+ # FIXME: File Size
+ deltas = []
+ else:
+ deltas = []
+ for testName in summary.testNames:
+ fullname = str('nightlytest.' + testName + '.' + key)
+ t = summary.testMap.get(fullname)
+ if not t:
+ continue
+ current = runSamples.get(t.id)
+ prev = prevSamples.get(t.id)
+ if not current or not prev:
+ continue
+ current = current[0]
+ prev = prev[0]
+ pct = safediv(current, prev)
+ if pct is None:
+ continue
+ pctDelta = pct - 1.
+ if abs(pctDelta) < .05:
+ continue
+ if min(prev,current) <= .2:
+ continue
+ deltas.append( (t.id, testName, current, prev, pctDelta) )
+
+ hidden = len(deltas) == 0
+ """
+ <p>
+ <a href="javascript://" onclick="toggleLayer('%s')"; id="%s_">(%s) %d %s Significant Changes</a>
+ <div id="%s" style="display: %s;" class="hideable">
+ """ % (name, name, ("+","-")[hidden], len(deltas), name, name, ("","none")[hidden])
+ if deltas:
+ # Redirect or something so we don't have to specify
+ # run here; that is silly.
+ """
+ <form method="GET" action="graphSingle">
+ <input type="hidden" name="run" value="%d">
+ <input type="hidden" name="kind" value="%d">
+ <form method="GET" action="graphSingle">
+ <table class="sortable" border=1>
+ <thead>
+ <tr>
+ <th class="sorttable_nosort"><input type="checkbox" id="checkAll.%d"></th>
+ <th>Program</th>
+ <th>%% Change</th>
+ <th>Previous Value</th>
+ <th>Current Value</th>
+ </tr>
+ </thead>
+ """ % (run.id, i, i,)
+ for id, name, current, prev, pctDelta in deltas:
+ """
+ <tr>
+ <td><input type="checkbox" name="cb.%d" id="cb_group.%d"></td>
+ <td><a href="../programs/%s/">%s</a></td>
+ %s
+ <td>%s</td>
+ <td>%s</td>
+ </tr>
+ """ % (id, i, id, name,
+ Util.PctCell(pctDelta).render(), prev, current)
+ """
+ </table>
+ <input type="submit" value="Compare Values">
+ </form>
+ """
+
+ """
+ </div>
+ """
+
+ def graphSingle [html] (self):
+ request = quixote.get_request()
+ full = request.form.get('full', '')
+ allResults = not not full
+
+ # Get a DB connection
+ db = self.root.getDB()
+
+ run = self.getActiveRun(db)
+ runs = db.runs(run.machine).order_by(Run.start_time.desc()).all()
+ machine = run.machine
+
+ request = quixote.get_request()
+ kindStr = request.form.get('kind')
+ kind = None
+ try:
+ kind = kComparisonKinds[int(kindStr)]
+ except:
+ pass
+ tests = []
+ for name,value in request.form.items():
+ if name.startswith(str('cb.')):
+ testIDStr = name[3:]
+ try:
+ testID = int(str(testIDStr))
+ tests.append(db.getTest(testID))
+ except:
+ pass
+
+ # Collect samples by test and machine, then bin into runs.
+ samplesByTest = {}
+ for t in tests:
+ samples = samplesByTest[t.id] = samplesByTest.get(t.id,{})
+
+ q = db.session.query(Sample.run_id,
+ Sample.value).join(Run)
+ q = q.filter(Run.machine_id == machine.id)
+ q = q.filter(Sample.test_id == t.id)
+ for s_run_id,s_value in q:
+ samples[s_run_id] = s_value
+
+ legend = []
+ plots = ""
+ for i,test in enumerate(tests):
+ data = []
+ for run in runs:
+ value = samplesByTest.get(test.id,{}).get(run.id)
+ if value is not None:
+ timeval = time.mktime(run.start_time.timetuple())
+ data.append((timeval, value))
+ data.sort()
+
+ col = list(Util.makeDarkColor(float(i) / len(tests)))
+ pts = ','.join(['[%f,%f]' % (t,v) for t,v in data])
+ style = "new Graph2D_LinePlotStyle(1, %r)" % col
+ plots += " graph.addPlot([%s], %s);\n" % (pts,style)
+
+ legend.append((test.name.split(str('.'),1)[1], col))
+ graph_init = """\
+ function init() {
+ graph = new Graph2D("graph");
+ graph.clearColor = [1, 1, 1];
+ %s
+ graph.xAxis.format = graph.xAxis.formats.day;
+ graph.draw();
+ }
+ """ % (plots,)
+
+ self.root.getHeader("Nightly Test Results", "../..",
+ addPopupJS=True, addGraphJS=True,
+ addJSScript=graph_init,
+ onload='init()')
+
+ """
+ <center>
+ <h2>LLVM Nightly Test Results</h2>
+ </center>
+ """
+
+ # Graph2D based graph.
+ """
+ <h3>Graph</h3>
+ <table>
+ <tr>
+ <td rowspan=2 valign="top">
+ <canvas id="graph" width="600" height="400"></canvas>
+ </td>
+ <td valign="top">
+ <table cellspacing=4 border=1>
+ <tr><th colspan=2>Test</th></tr>
+ """
+ for name,col in legend:
+ """<tr><td bgcolor="%02x%02x%02x"> </td><td>%s</td></tr>""" % (
+ 255*col[0], 255*col[1], 255*col[2], name)
+ """
+ </table>
+ </td></tr>
+ <tr><td align="right" valign="bottom">
+ <font size="-2">
+ Shift-Left Mouse: Pan<br>
+ Alt/Meta-Left Mouse: Zoom<br>
+ Wheel: Zoom (<i>Shift Slows</i>)<br>
+ </font>
+ </td></tr>
+ </table>
+ """
+
+ """<h3>Values</h3>
+ <a href="javascript://" onclick="toggleLayer('graph_values');"
+ id="graph_values_">(-) Graph Values</a>
+ <div id="graph_values" style="display: none;" class="hideable">
+ <table class="sortable" border=1>
+ <thead>
+ <tr>
+ <th>Run</th>
+ <th>Timestamp</th>
+ """
+ for t in tests:
+ """
+ <th>%s</th>
+ """ % (t.name,)
+ """
+ </thead>
+ """
+
+ for run in runs:
+ """
+ <tr>
+ <td>%d</td>
+ <td>%s</td>""" % (run.id, run.start_time)
+ for t in tests:
+ value = samplesByTest.get(t.id,{}).get(run.id, 'N/A')
+ """
+ <td>%s</td>""" % value
+ """
+ </tr>"""
+ """
+ </table>
+ </div>
+ """
+
+ self.root.getFooter()
+
+class NightlyTestMachineUI(Directory):
+ _q_exports = [""]
+
+ def __init__(self, root, idstr):
+ self.root = root
+ try:
+ self.id = int(idstr)
+ except ValueError, exc:
+ raise TraversalError(str(exc))
+
+ def _q_index [html] (self):
+ self.root.getHeader("Nightly Test Results", "../../..",
+ addPopupJS=True)
+
+ # Get a DB connection
+ db = self.root.getDB()
+
+ machine = db.getMachine(self.id)
+
+ # Find all runs on this machine.
+ runs = db.runs(machine).order_by(Run.start_time.desc()).all()
+
+ """
+ <center>
+ <h1>LLVM Nightly Test Results</h1>
+ <table>
+ <tr>
+ <td align=right>Machine:</td>
+ <td>%s:%d</td>
+ </tr>
+ </table>
+ </center>
+ <p>
+ """ % (machine.name, machine.number)
+
+ # FIXME: List previous machines with the same nickname?
+ """
+ <table width="100%%" border=1>
+ <tr>
+ <td valign="top" width="200">
+ <a href="../..">Homepage</a>
+ <h4>Relatives:</h4>
+ <ul>
+ """
+ # List all machines with this name.
+ for m in db.machines(name=machine.name):
+ """<li><a href="../%d">%s:%d</a></li>""" % (m.id, m.name, m.number)
+ """
+ </ul>
+ <h4>Runs:</h4>
+ <ul>
+ """
+
+ # Show the most recent 10 runs.
+ for r in runs[:10]:
+ """ <li> <a href="../../%d/">%s</a> """ % (r.id, r.start_time)
+
+ # Full list of runs in a drop down.
+ #
+ # FIXME: Link to run correctly.
+ """
+ <p>
+ <form method="GET" action="../../1/">
+ <select name="run">
+ """
+ for r in runs:
+ """\
+ <option value="%d">%s""" % (r.id, r.start_time)
+
+ """
+ </select>
+ <input type="submit" value="Go to Run">
+ </form>
+ """
+
+ """
+ </ul>
+ </td>
+ <td valign="top">
+ <table border=1>
+ <tr>
+ <td> <b>Nickname</b> </td>
+ <td> %s </td>
+ </tr>
+ """ % (machine.name,)
+ for mi in machine.info.values():
+ """
+ <tr>
+ <td> <b>%s</b> </td>
+ <td>%s</td>
+ </tr>
+ """ % (mi.key, mi.value)
+ """
+ <tr>
+ <td> <b>Machine ID</b> </td>
+ <td> %d </td>
+ </tr>
+ </table>
+ <p>
+ """ % (machine.id,)
+
+ # List associated runs.
+ """
+ <table class="sortable" border=1>
+ <thead>
+ <tr>
+ <th>Start Time</th>
+ <th>End Time</th>
+ <th> </th>
+ </thead>
+ """
+ for r in runs:
+ """
+ <tr>
+ <td>%s</td>
+ <td>%s</td>
+ <td><a href="../../%d">View Results</a></td>
+ </tr>""" % (r.start_time, r.end_time, r.id)
+ """
+ </table>
+ """
+
+ """
+ </td>
+ </tr>
+ </table>
+ """
+
+ self.root.getFooter()
+
+class NightlyTestProgramUI(Directory):
+ _q_exports = [""]
+
+ def __init__(self, root, testIDStr):
+ self.root = root
+ try:
+ self.testID = int(testIDStr)
+ except ValueError, exc:
+ raise TraversalError(str(exc))
+
+ def _q_index [html] (self):
+ self.root.getHeader("Nightly Test Results", "../../..",
+ addPopupJS=True)
+
+ # Get a DB connection.
+ db = self.root.getDB()
+
+ # Get the test we use to derive the name.
+ t = db.getTest(id = self.testID)
+ programName = t.name.split(str('.'), 3)[1]
+
+ # Collect runs within the last 48 hours of the most recent report.
+ import datetime
+ runs = []
+ most_recent, = db.session.query(Run.start_time).\
+ order_by(Run.start_time.desc()).first()
+ cutoff = most_recent - datetime.timedelta(days=2)
+ runs = db.session.query(Run).\
+ filter(Run.start_time >= cutoff).\
+ order_by(Run.start_time.desc()).all()
+
+ """
+ <center>
+ <h1>LLVM Nightly Test Results</h1>
+ <table>
+ <tr>
+ <td align=right>Program:</td>
+ <td>%s</td>
+ </tr>
+ </table>
+ </center>
+ <p>
+ """ % (programName,)
+
+ self.getAllResults(db, programName, runs)
+
+ self.root.getFooter()
+
+ def getAllResults [html] (self, db, testName, runs):
+ columns = [('GCCAS', 'gcc.compile.time', ()),
+ ('Bitcode','bc.compile.size', ()),
+ ('LLC<br>compile','llc.compile.time', ('bc.compile.size',)),
+ ('LLC-BETA<br>compile','llc-beta.compile.time', ('bc.compile.size',)),
+ ('JIT<br>codegen','jit.compile.time', ('bc.compile.size',)),
+ ('GCC','gcc.exec.time', ('gcc.compile.time',)),
+ ('CBE','cbe.exec.time', ('bc.compile.size',)),
+ ('LLC','llc.exec.time', ('llc.compile.time',)),
+ ('LLC-BETA','llc-beta.exec.time', ('llc-beta.compile.time',)),
+ ('JIT','jit.exec.time', ('jit.compile.time',))]
+
+ # Add interface to hiding columns by test or column type.
+ keyIndices = Util.multidict()
+ for idx,info in enumerate(columns):
+ key = str(info[1]).split(str('.'))[0]
+ keyIndices[key] = idx + 2
+ """
+ <form>
+ <table border="1">
+ <thead>
+ <tr>
+ <th>Column Visibility</th>
+ <th>GCC</th>
+ <th>LLC</th>
+ <th>CBE</th>
+ <th>JIT</th>
+ <th>LLC-BETA</th>
+ </tr>
+ </thead>
+ <tr>
+ <td>Enabled</td>
+ <td><input type='checkbox' onClick='javascript:show_hide_column("programs", [%s]);' checked></td>
+ <td><input type='checkbox' onClick='javascript:show_hide_column("programs", [%s]);' checked></td>
+ <td><input type='checkbox' onClick='javascript:show_hide_column("programs", [%s]);' checked></td>
+ <td><input type='checkbox' onClick='javascript:show_hide_column("programs", [%s]);' checked></td>
+ <td><input type='checkbox' onClick='javascript:show_hide_column("programs", [%s]);' checked></td>
+ </tr>
+ </table>
+ </form>
+ """ % (', '.join(map(str, keyIndices['gcc'])),
+ ', '.join(map(str, keyIndices['llc'])),
+ ', '.join(map(str, keyIndices['cbe'])),
+ ', '.join(map(str, keyIndices['jit'])),
+ ', '.join(map(str, keyIndices['llc-beta'])))
+
+ # The main table.
+ """
+ <table class="sortable" border="1" cellspacing="0" cellpadding="0" id="programs">
+ <thead>
+ <tr>
+ <th>Machine</th>
+ <th>Run Start</th>
+ """
+ for name,key,dependsOn in columns:
+ """<th>%s</th>""" % (name, )
+ """
+ </tr>
+ </thead>
+ """
+
+ for run in runs:
+ """
+ <tr>
+ <td><a href="../../machines/%d">%s:%d</a></td>
+ <td><a href="../../%d">%s</a></td>
+ """ % (run.machine.id, run.machine.name, run.machine.number,
+ run.id, run.start_time)
+
+ for name,key,dependsOn in columns:
+ fullname = str('nightlytest.' + testName + '.' + key)
+ # FIXME: Make fast.
+ current = getTestNameValueInRun(db, run, fullname)
+ if current is not None:
+ if key.endswith('size'):
+ """<td>%d</td>""" % int(current)
+ else:
+ """<td>%.4f</td>""" % current
+ else:
+ # Only mark failure if nothing we depend on failed.
+ failed = True
+ for d in dependsOn:
+ # FIXME: Make fast.
+ t = getTestNameValueInRun(db, run,
+ str('nightlytest.' + testName + '.' + d))
+ if t is None:
+ failed = False
+ break
+ if failed:
+ """<td bgcolor="#FF0000">*</td>"""
+ else:
+ """<td>N/A</td>"""
+ """
+ </tr>
+ """
+ """
+ </table>
+ """
+
+class NightlyTestDirectory(NTStyleBrowser.RecentMachineDirectory):
+ _q_exports = [""]
+
+ def getTags(self):
+ return (None, 'nightlytest')
+
+ def getTitle(self):
+ return 'Nightly Test'
+
+ def getHeaderTitle(self):
+ return 'LLVM Nightly Test'
+
+ def getTestRunUI(self, component):
+ return NightlyTestRunUI(self.root, component)
+
+ def getTestMachineUI(self, component):
+ return NightlyTestMachineUI(self.root, component)
+
+ def getProgramUI(self, component):
+ return NightlyTestProgramUI(self.root, component)
Added: zorg/trunk/lnt/viewer/publisher.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/publisher.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/publisher.py (added)
+++ zorg/trunk/lnt/viewer/publisher.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1,48 @@
+import time
+from quixote.publish import Publisher
+
+# FIXME: This is a bit of a hack.
+class ExtPublisher(Publisher):
+ def __init__(self, *args, **kwargs):
+ Publisher.__init__(self, *args, **kwargs)
+ self.create_time = time.time()
+
+ def process_request(self, request):
+ request.start_time = time.time()
+ return Publisher.process_request(self, request)
+
+class ThreadedPublisher(ExtPublisher):
+ is_thread_safe = True
+
+ def __init__ (self, root_namespace, *args, **kwargs):
+ ExtPublisher.__init__(self, root_namespace, *args, **kwargs)
+ self._request_dict = {}
+
+ def _set_request(self, request):
+ import thread
+ self._request_dict[thread.get_ident()] = request
+
+ def _clear_request(self):
+ import thread
+ try:
+ del self._request_dict[thread.get_ident()]
+ except KeyError:
+ pass
+
+ def get_request(self):
+ import thread
+ return self._request_dict.get(thread.get_ident())
+
+def create_publisher(configPath, configData, threaded=False):
+ import Config
+ config = Config.Config.fromData(configPath, configData)
+
+ from quixote import enable_ptl
+ enable_ptl()
+
+ from root import RootDirectory
+ if threaded:
+ publisher_class = ThreadedPublisher
+ else:
+ publisher_class = ExtPublisher
+ return publisher_class(RootDirectory(config), display_exceptions='plain')
Added: zorg/trunk/lnt/viewer/resources/form.css
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/resources/form.css?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/resources/form.css (added)
+++ zorg/trunk/lnt/viewer/resources/form.css Sat Mar 20 20:00:06 2010
@@ -0,0 +1,75 @@
+/* Derived from Quixote's BASIC_FORM_CSS */
+
+form.quixote div.title {
+ font-weight: bold;
+}
+
+form.quixote br.submit,
+form.quixote br.widget,
+br.quixoteform {
+ clear: left;
+}
+
+form.quixote div.submit br.widget {
+ display: none;
+}
+
+form.quixote div.widget {
+ float: left;
+ padding: 4px;
+ padding-right: 1em;
+ margin-bottom: 6px;
+}
+
+/* pretty forms (attribute selector hides from broken browsers (e.g. IE) */
+form.quixote[action] {
+ float: left;
+}
+
+form.quixote[action] > div.widget {
+ float: none;
+}
+
+form.quixote[action] > br.widget {
+ display: none;
+}
+
+form.quixote div.widget div.widget {
+ padding: 0;
+ margin-bottom: 0;
+}
+
+form.quixote div.SubmitWidget {
+ float: left
+}
+
+form.quixote div.content {
+ margin-left: 0.6em; /* indent content */
+}
+
+form.quixote div.content div.content {
+ margin-left: 0; /* indent content only for top-level widgets */
+}
+
+form.quixote div.error {
+ color: #c00;
+ font-size: small;
+ margin-top: .1em;
+}
+
+form.quixote div.hint {
+ font-size: small;
+ font-style: italic;
+ margin-top: .1em;
+}
+
+form.quixote div.errornotice {
+ color: #c00;
+ padding: 0.5em;
+ margin: 0.5em;
+}
+
+form.quixote div.FormTokenWidget,
+form.quixote.div.HiddenWidget {
+ display: none;
+}
Added: zorg/trunk/lnt/viewer/resources/popup.js
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/resources/popup.js?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/resources/popup.js (added)
+++ zorg/trunk/lnt/viewer/resources/popup.js Sat Mar 20 20:00:06 2010
@@ -0,0 +1,155 @@
+function ShowPop(id)
+{
+ if (document.getElementById)
+ {
+ document.getElementById(id).style.visibility = " visible";
+ }
+ else if (document.all)
+ {
+ document.all[id].style.visibility = " visible";
+ }
+ else if (document.layers)
+ {
+ document.layers[id].style.visibility = " visible";
+ }
+}
+
+
+
+
+
+
+function HidePop(id)
+{
+ if (document.getElementById)
+ {
+ document.getElementById(id).style.visibility = " hidden";
+ }
+ /*else if (document.all)
+ {
+ document.all[id].style.visibility = " hidden";
+ }
+ else if (document.layers)
+ {
+ document.layers[id].style.visibility = " hidden";
+ }*/
+}
+
+
+
+function TogglePop(id)
+{
+ if (document.getElementById)
+ {
+ if(document.getElementById(id).style.visibility == "visible"){
+ document.getElementById(id).style.visibility = "hidden";
+ }
+ else{
+ document.getElementById(id).style.visibility = "visible";
+ }
+ }
+ else if (document.all)
+ {
+ if(document.all[id].style.visibility == "visible"){
+ document.all[id].style.visibility = "hidden";
+ }
+ else{
+ document.all[id].style.visibility = "visible";
+ }
+ }
+ else if (document.layers)
+ {
+ if(document.layers[id].style.visibility == "visible"){
+ document.layers[id].style.visibility = "hidden";
+ }
+ else{
+ document.layers[id].style.visibility = "visible";
+ }
+ }
+}
+
+
+function toggleLayer(whichLayer)
+{
+ if (document.getElementById)
+ {
+ // this is the way the standards work
+ var style2 = document.getElementById(whichLayer).style;
+ style2.display = style2.display? "":"none";
+ var link = document.getElementById(whichLayer+"_").innerHTML;
+ if(link.indexOf("(+)") >= 0){
+ document.getElementById(whichLayer+"_").innerHTML="(-)"+link.substring(3,link.length);
+ }
+ else{
+ document.getElementById(whichLayer+"_").innerHTML="(+)"+link.substring(3,link.length);
+ }
+ }//end if
+ else if (document.all)
+ {
+ // this is the way old msie versions work
+ var style2 = document.all[whichLayer].style;
+ style2.display = style2.display? "":"none";
+ var link = document.all[wwhichLayer+"_"].innerHTML;
+ if(link.indexOf("(+)") >= 0){
+ document.all[whichLayer+"_"].innerHTML="(-)"+link.substring(3,link.length);
+ }
+ else{
+ document.all[whichLayer+"_"].innerHTML="(+)"+link.substring(3,link.length);
+ }
+ }
+ else if (document.layers)
+ {
+ // this is the way nn4 works
+ var style2 = document.layers[whichLayer].style;
+ style2.display = style2.display? "":"none";
+ var link = document.layers[whichLayer+"_"].innerHTML;
+ if(link.indexOf("(+)") >= 0){
+ document.layers[whichLayer+"_"].innerHTML="(-)"+link.substring(3,link.length);
+ }
+ else{
+ document.layers[whichLayer+"_"].innerHTML="(+)"+link.substring(3,link.length);
+ }
+ }
+}//end function
+
+var checkflag="false";
+function check(field) {
+ if (checkflag == "false") {
+ for (i = 0; i < field.length; i++) {
+ field[i].checked = true;
+ }
+ checkflag = "true";
+ return "Uncheck all";
+ }
+ else {
+ for (i = 0; i < field.length; i++) {
+ if(field[i].type == 'checkbox'){
+ field[i].checked = false;
+ }
+ }
+ checkflag = "false";
+ return "Check all";
+ }
+}
+
+function show_hide_column(tableName, columns) {
+ // Let's be clear hear, I have no idea how to write portable
+ // JavaScript. This works in Safari, yo.
+ var event = window.event;
+ var cb = event.target;
+
+ var style = cb.checked ? "table-cell" : "none";
+
+ var tbl = document.getElementById(tableName);
+ var rows = tbl.getElementsByTagName('tr');
+
+ for (var row = 0; row < rows.length; ++row) {
+ var cells = rows[row].getElementsByTagName('td');
+
+ if (cells.length == 0)
+ cells = rows[row].getElementsByTagName('th');
+
+ for (var i = 0; i < columns.length; ++i)
+ cells[columns[i]].style.display = style;
+ }
+}
Added: zorg/trunk/lnt/viewer/resources/style.css
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/resources/style.css?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/resources/style.css (added)
+++ zorg/trunk/lnt/viewer/resources/style.css Sat Mar 20 20:00:06 2010
@@ -0,0 +1,79 @@
+.zorg_navheader {
+ background-color: #cccccc;
+}
+
+body {
+ color:#000000;
+ background-color:#ffffff
+}
+
+body {
+ font-family: Helvetica, sans-serif;
+ font-size:9pt
+}
+
+h1 {
+ font-size: 14pt;
+}
+
+h2 {
+ font-size: 12pt;
+}
+
+table {
+ font-size:9pt
+}
+
+table {
+ border-spacing: 0px;
+ border: 1px solid black
+}
+
+th, table thead {
+ background-color:#eee;
+ color:#666666;
+ font-weight: bold;
+ cursor: default;
+ text-align:center;
+ font-weight: bold;
+ font-family: Verdana;
+}
+
+.W {
+ font-size:0px
+}
+
+th, td {
+ padding:5px;
+ padding-left:8px;
+}
+
+tbody.scrollContent {
+ overflow:auto
+}
+
+.hideable {
+ border-width:thin;
+ border-color:background;
+ border-style:solid;
+ background: #F8F8FF;
+ padding:8px;
+}
+
+/* Nested popups */
+
+.hideable_1 {
+ border-width:thin;
+ border-color:background;
+ border-style:solid;
+ background: #F8F8FF;
+ padding:8px;
+}
+
+.hideable_2 {
+ border-width:thin;
+ border-color:background;
+ border-style:solid;
+ background: #E8E8E8;
+ padding:8px;
+}
Added: zorg/trunk/lnt/viewer/root.ptl
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/root.ptl?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/root.ptl (added)
+++ zorg/trunk/lnt/viewer/root.ptl Sat Mar 20 20:00:06 2010
@@ -0,0 +1,432 @@
+# -*- python -*-
+
+import os
+import re
+import time
+
+import quixote
+import quixote.form
+import quixote.form.css
+from quixote import get_response
+from quixote.directory import Directory, Resolving
+from quixote.util import StaticDirectory
+
+import PerfDB, Util
+from PerfDB import Machine, Run
+
+class RootDirectory(Resolving, Directory):
+ _q_exports = ["", "resources", "js", "machines", "runs", "tests",
+ "browse", "submitRun", "nightlytest", "zview",
+
+ # Redirections,
+ "select_db",
+
+ ("favicon.ico", "favicon_ico")]
+
+ def __init__(self, config, dbName='default', dbInfo=None, pathToRoot="./"):
+ self.config = config
+ self.dbName = dbName
+ self.dbInfo = dbInfo
+ if self.dbInfo is None:
+ self.dbInfo = config.databases[dbName]
+ self.pathToRoot = pathToRoot
+ self.db_log = None
+
+ def getDB(self):
+ db = PerfDB.PerfDB(self.dbInfo.path)
+
+ # Enable SQL logging with db_log.
+ #
+ # FIXME: Conditionalize on an is_production variable.
+ request = quixote.get_request()
+ if self.db_log is None and request.form.get('db_log'):
+ import logging, StringIO
+ self.db_log = StringIO.StringIO()
+ logger = logging.getLogger("sqlalchemy")
+ logger.addHandler(logging.StreamHandler(self.db_log))
+ db.db2.engine.echo = True
+
+ return db
+
+ def getHeader [html] (self, title, pathToRoot,
+ addSorttableJS=True,
+ addFormCSS=False,
+ addPopupJS=False,
+ addGraphJS=False,
+ addJSScript=None,
+ onload=None):
+ pathToRoot = os.path.join(self.pathToRoot,
+ pathToRoot)
+
+ """
+ <html>
+ <head>
+ """
+ if addSorttableJS:
+ """
+ <script src="%s/resources/sorttable.js"></script>
+ """ % (pathToRoot,)
+ if addPopupJS:
+ """
+ <script src="%s/resources/popup.js"></script>
+ """ % (pathToRoot,)
+ if addGraphJS:
+ """
+ <script src="%s/resources/mootools-1.2.4-core-nc.js"></script>
+ <script src="%s/js/View2D.js"></script>
+ """ % (pathToRoot,pathToRoot)
+ if addJSScript:
+ """\
+<script type="text/javascript">
+%s
+</script>
+""" % (addJSScript,)
+
+ """
+ <link rel="stylesheet" href="%s/resources/style.css" type="text/css" />
+ """ % (pathToRoot,)
+ if addFormCSS:
+ """
+ <link rel="stylesheet" href="%s/resources/form.css" type="text/css" />
+ """ % (pathToRoot,)
+ """
+ <link rel="icon" type="image/png" href="%s/favicon.ico">
+ <title>%s</title>
+ </head>
+ """ % (pathToRoot, title)
+
+ """\
+ <body"""
+ if onload:
+ """ onload="%s">""" % (onload,)
+ else:
+ """>"""
+
+ # Database selection
+ """\
+ <div class="zorg_navheader">
+ <form method="get" action="%s/select_db">
+ <table style="padding:0.1em;" width="100%%">
+ <tr>
+ <td>
+ <strong>
+ [%s]
+ </strong>
+ </td>
+ <td style="text-align:right;">
+ <strong>Database:</strong>
+ <select name="db" onchange="submit()">
+ """ % (pathToRoot, self.dbName)
+ for name in self.config.databases.keys():
+ """\
+ <option %s>%s</option>
+ """ % (('', 'selected')[name == self.dbName],
+ name)
+ """\
+ </select>
+ <input type="submit" value="Go" />
+ </td>
+ </tr>
+ </table>
+ </form>
+ </div>
+ """
+
+ def getFooter [html] (self):
+ if self.db_log:
+ """<hr><h3>SQL Log</h3><pre>%s</pre>""" % self.db_log.getvalue()
+
+ current = time.time()
+ """
+ <hr>
+ Server Started: %s<br>
+ Generated: %s<br>
+ Render Time: %.2fs<br>
+ </body>
+ </html>
+ """ % (time.strftime(str('%Y-%m-%dT%H:%M:%Sz'),
+ time.localtime(quixote.get_publisher().create_time)),
+ time.strftime(str('%Y-%m-%dT%H:%M:%Sz'),
+ time.localtime(current)),
+ current - quixote.get_request().start_time)
+
+ def _q_index [html] (self):
+ self.getHeader("zorg", ".")
+
+ """
+ <h2>LLVM Testing DB</h2>
+ """
+
+ # Features
+
+ if self.dbInfo.showNightlytest:
+ """
+ <h3>Nightly Test Results</h3>
+ <a href="nightlytest/">Nightly Test</a>
+ """
+
+ if self.dbInfo.showGeneral:
+ """
+ <hr>
+
+ <h3>General Database Access</h3>
+ <p><a href="browse">Browse DB</a>
+ <p><a href="submitRun">Submit Run</a>
+ """
+
+ self.getFooter()
+
+ def browse [html] (self):
+ self.getHeader("zorg", ".", addSorttableJS=False)
+
+ # Get a DB connection
+ db = self.getDB()
+
+ """
+ <h2>LLVM Testing DB</h2>
+ """
+
+ # List machines
+ """
+ <h3>Machines</h3>
+ <table class="sortable" border=1 cellborder=1>
+ <thead>
+ <tr>
+ <th>Name</th>
+ </tr>
+ </thead>
+ """
+ for m in db.machines():
+ """
+ <tr>
+ <td><a href="machines/%d/">%s:%d</a></td>
+ </tr>
+ """ % (m.id, m.name, m.number)
+ """
+ </table>
+ """
+
+ # List runs
+ """
+ <h3>Run List</h3>
+ <table class="sortable" border=1 cellborder=1>
+ <thead>
+ <tr>
+ <th>ID</th>
+ <th>Machine</th>
+ <th>Start Time</th>
+ <th>End Time</th>
+ </tr>
+ </thead>
+ """
+ for r,m in db.session.query(Run,Machine).join(Machine):
+ """
+ <tr>
+ <td><a href="runs/%d/">%d</a></td>
+ <td><a href="machines/%d/">%s:%d</a></td>
+ <td>%s</td>
+ <td>%s</td>
+ </tr>
+ """ % (r.id, r.id,
+ r.machine_id, m.name, m.number,
+ r.start_time, r.end_time)
+ """
+ </table>
+ """
+
+
+ # List tests
+ """
+ <h3>Test List</h3>
+ <table class="sortable" border=1 cellborder=1>
+ <thead>
+ <tr>
+ <th>ID</th>
+ <th>Test</th>
+ </tr>
+ </thead>
+ """
+ for t in db.tests():
+ """
+ <tr>
+ <td><a href="tests/%d/">%d</a></td>
+ <td>%s</td>
+ </tr>
+ """ % (t.id, t.id, t.name)
+ """
+ </table>
+ """
+
+ self.getFooter()
+
+ def submitRun(self):
+ form = quixote.form.Form(enctype="multipart/form-data")
+ form.add(quixote.form.FileWidget, "file",
+ title="Input File (plist)", required=True)
+ form.add(quixote.form.SingleSelectWidget, "commit",
+ title="Commit", value="0",
+ options=["0", "1"], required=True)
+ form.add_submit("submit", "Submit")
+
+ def render [html] ():
+ self.getHeader("Submit Run", ".", addFormCSS=1)
+ """
+ <h1>Submit Runs</h1>
+ """
+ form.render()
+ self.getFooter()
+
+ def process():
+ import plistlib
+ import tempfile
+ fileWidget = form.get_widget('file')
+ value = fileWidget.parse()
+
+ data = value.fp.read()
+ value.fp.close()
+ prefix = time.strftime("data-%Y-%m-%d_%H-%M-%S")
+ fd,path = tempfile.mkstemp(prefix=prefix,
+ suffix='.plist',
+ dir=self.config.tempDir)
+ os.write(fd, data)
+ os.close(fd)
+
+ # Find the email address for this machine's results.
+ toAddress = None
+ if isinstance(self.config.ntEmailTo, str):
+ toAddress = self.config.ntEmailTo
+ else:
+ # Find the machine name.
+ #
+ # FIXME: This is really stupid, we shouldn't load the plist
+ # twice just for this.
+ import plistlib
+ data = plistlib.readPlist(path)
+ machineName = data.get('Machine',{}).get('Name')
+
+ for pattern,addr in self.config.ntEmailTo:
+ if re.match(pattern, machineName):
+ toAddress = addr
+ break
+ else:
+ return 1,"","error: unable to match machine name for test results email address!"
+
+ # Execute ImportData to actually do the import.
+ #
+ # FIXME: This is both broken and annoying. We want to do this
+ # internally to fix the FIXME above and keep it more readable, we
+ # want to serialize imports to keep SQLite happy and avoiding
+ # dropping submissions.
+ import subprocess
+ p = subprocess.Popen([os.path.join(self.config.zorgDir,
+ "import/ImportData"),
+ "--commit=%s" % form['commit'],
+ "--email-on-import=%s" % int(self.config.ntEmailEnabled),
+ "--email-base-url=%s/db_%s/nightlytest/" % (self.config.zorgURL,
+ self.dbName),
+ "--email-host=%s" % self.config.ntEmailHost,
+ "--email-from=%s" % self.config.ntEmailFrom,
+ "--email-to=%s" % toAddress,
+ self.dbInfo.path,
+ path],
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE)
+ stdout,stderr = p.communicate(None)
+ res = p.wait()
+ stdout += "\nMAILING RESULTS TO: %r\n" % toAddress
+ return res,stdout,stderr
+
+ def result [plain] ():
+ res,stdout,stderr = process()
+ if True:
+ """\
+STATUS: %s
+
+OUTPUT:
+%s
+ERRORS:
+%s""" % (res, stdout, stderr)
+ else:
+ self.getHeader("Data Received", "../..")
+ """
+ <h3>Result</h3>
+ %s
+ <h3>Output</h3>
+ <pre>\n%s</pre>
+ <h3>Errors</h3>
+ <pre>\n%s</pre>
+ </body>
+ """ % (res, stdout, stderr)
+ self.getFooter()
+
+ if not form.is_submitted() or form.has_errors():
+ return render()
+ return result()
+
+ def favicon_ico(self):
+ response = get_response()
+ response.set_content_type("image/x-icon")
+ response.set_expires(days=1)
+ return FAVICON
+
+ def _q_resolve(self, component):
+ if component == 'machines':
+ import machines
+ return machines.MachinesDirectory(self)
+ if component == 'runs':
+ import runs
+ return runs.RunsDirectory(self)
+ if component == 'tests':
+ import tests
+ return tests.TestsDirectory(self)
+ if component == 'nightlytest':
+ import nightlytest
+ return nightlytest.NightlyTestDirectory(self)
+ if component == 'zview':
+ from zview import zviewui
+ return zviewui.ZViewUI(self)
+
+ def _q_lookup(self, component):
+ if component.startswith('db_'):
+ dbName = component[3:]
+ dbInfo = self.config.databases.get(dbName)
+ if dbInfo:
+ return RootDirectory(self.config, dbName, dbInfo, "../")
+
+ def select_db(self):
+ request = quixote.get_request()
+ dbName = request.form.get('db')
+ return quixote.redirect("db_%s/" % (dbName,))
+
+ resources = StaticDirectory(os.path.join(os.path.dirname(__file__),
+ 'resources'),
+ list_directory=True)
+ js = StaticDirectory(os.path.join(os.path.dirname(__file__), 'js'),
+ list_directory=True)
+
+FAVICON = """\
+AAABAAEAEBAAAAAAAABoBQAAFgAAACgAAAAQAAAAIAAAAAEACAAAAAAAAAEAAAAAAAAAAAAAAAEA
+AAAAAAD///8AAAD/ALOz/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgEAAAAAAAAAAAAAAAAAAgECAQIAAAAAAAAAAAAAAAEA
+AAIBAQIAAAACAQIAAAECAAAAAAIBAQICAQIBAgECAAAAAAAAAAIBAQIAAAECAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=""".decode('base64')
Added: zorg/trunk/lnt/viewer/runs.ptl
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/runs.ptl?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/runs.ptl (added)
+++ zorg/trunk/lnt/viewer/runs.ptl Sat Mar 20 20:00:06 2010
@@ -0,0 +1,84 @@
+# -*- python -*-
+
+import sys
+from quixote import get_response, redirect
+from quixote.directory import Directory
+from quixote.errors import TraversalError
+
+class RunUI(Directory):
+ _q_exports = [""]
+
+ def __init__(self, root, idstr):
+ self.root = root
+ try:
+ self.id = int(idstr)
+ except ValueError, exc:
+ raise TraversalError(str(exc))
+
+ def _q_index [html] (self):
+ # Get a DB conntection
+ db = self.root.getDB()
+
+ r = db.getRun(self.id)
+ m = db.getMachine(r.machine_id)
+
+ self.root.getHeader("zorg:run:%d" % self.id, '../..')
+
+ """
+ <body>
+ <h2>Run: %d</h2>
+ """ % (r.id)
+
+ """
+ <table border=1 cellborder=1>
+ <tr>
+ <th>Machine</th>
+ <th>Start Time</th>
+ <th>End Time</th>
+ </tr>
+ </thead>
+ <tr>
+ <td><a href="../../machines/%d/">%s:%d</a></td>
+ <td>%s</td>
+ <td>%s</td>
+ </tr>
+ </table>
+ """ % (r.machine_id, m.name, m.number, r.start_time, r.end_time)
+
+
+ # Run Info
+ """
+ <h3>Run Info</h3>
+ <table border=1 cellborder=1>
+ <tr>
+ <th>Key</th>
+ <th>Value</th>
+ </tr>
+ </thead>
+ """
+ for mi in r.info.values():
+ """
+ <tr>
+ <td>%s</td>
+ <td>%s</td>
+ </tr>""" % (mi.key, mi.value)
+ """
+ </table>
+ """
+
+ self.root.getFooter()
+
+class RunsDirectory(Directory):
+ _q_exports = [""]
+
+ def __init__(self, root):
+ Directory.__init__(self)
+ self.root = root
+
+ def _q_index [plain] (self):
+ """
+ run access
+ """
+
+ def _q_lookup(self, component):
+ return RunUI(self.root, component)
Added: zorg/trunk/lnt/viewer/tests.ptl
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/tests.ptl?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/tests.ptl (added)
+++ zorg/trunk/lnt/viewer/tests.ptl Sat Mar 20 20:00:06 2010
@@ -0,0 +1,88 @@
+# -*- python -*-
+
+import sys
+from quixote import get_response, redirect
+from quixote.directory import Directory
+from quixote.errors import TraversalError
+
+class TestUI(Directory):
+ _q_exports = [""]
+
+ def __init__(self, root, idstr):
+ self.root = root
+ try:
+ self.id = int(idstr)
+ except ValueError, exc:
+ raise TraversalError(str(exc))
+
+ def _q_index [html] (self):
+ # Get a DB conntection
+ db = self.root.getDB()
+
+ t = db.getTest(self.id)
+
+ self.root.getHeader("zorg:test:%d" % self.id, '../..')
+
+ """
+ <body>
+ <h2>Test: %s</h2>
+ """ % (t.name,)
+
+ # Test info
+ """
+ <h3>Test Info</h3>
+ <table border=1 cellborder=1>
+ <tr>
+ <th>Key</th>
+ <th>Value</th>
+ </tr>
+ </thead>
+ """
+ for item in t.info.values():
+ """
+ <tr>
+ <td>%s</td>
+ <td>%s</td>
+ </tr>""" % (item.key, item.value)
+ """
+ </table>
+ """
+
+ # List samples
+ """
+ <h3>Associated Samples</h3>
+ <table class="sortable" border=1 cellborder=1>
+ <thead>
+ <tr>
+ <th>Run ID</th>
+ <th>Value</th>
+ </tr>
+ </thead>
+ """
+ for s in db.samples(test=t):
+ """
+ <tr>
+ <td><a href="../../runs/%d/">%d</a></td>
+ <td>%s</td>
+ </tr>
+ """ % (s.run_id, s.run_id, s.value)
+ """
+ </table>
+ """
+
+ self.root.getFooter()
+
+class TestsDirectory(Directory):
+ _q_exports = [""]
+
+ def __init__(self, root):
+ Directory.__init__(self)
+ self.root = root
+
+ def _q_index [plain] (self):
+ """
+ test access
+ """
+
+ def _q_lookup(self, component):
+ return TestUI(self.root, component)
Added: zorg/trunk/lnt/viewer/wsgi_restart.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/wsgi_restart.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/wsgi_restart.py (added)
+++ zorg/trunk/lnt/viewer/wsgi_restart.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1,115 @@
+# This code lifted from the mod_wsgi docs.
+
+import os
+import sys
+import time
+import signal
+import threading
+import atexit
+import Queue
+
+_interval = 1.0
+_times = {}
+_files = []
+
+_running = False
+_queue = Queue.Queue()
+_lock = threading.Lock()
+
+def _restart(path):
+ _queue.put(True)
+ prefix = 'monitor (pid=%d):' % os.getpid()
+ print >> sys.stderr, '%s Change detected to \'%s\'.' % (prefix, path)
+ print >> sys.stderr, '%s Triggering process restart.' % prefix
+ os.kill(os.getpid(), signal.SIGINT)
+
+def _modified(path):
+ try:
+ # If path doesn't denote a file and were previously
+ # tracking it, then it has been removed or the file type
+ # has changed so force a restart. If not previously
+ # tracking the file then we can ignore it as probably
+ # pseudo reference such as when file extracted from a
+ # collection of modules contained in a zip file.
+
+ if not os.path.isfile(path):
+ return path in _times
+
+ # Check for when file last modified.
+
+ mtime = os.stat(path).st_mtime
+ if path not in _times:
+ _times[path] = mtime
+
+ # Force restart when modification time has changed, even
+ # if time now older, as that could indicate older file
+ # has been restored.
+
+ if mtime != _times[path]:
+ return True
+ except:
+ # If any exception occured, likely that file has been
+ # been removed just before stat(), so force a restart.
+
+ return True
+
+ return False
+
+def _monitor():
+ while 1:
+ # Check modification times on all files in sys.modules.
+
+ for module in sys.modules.values():
+ if not hasattr(module, '__file__'):
+ continue
+ path = getattr(module, '__file__')
+ if not path:
+ continue
+ if os.path.splitext(path)[1] in ['.pyc', '.pyo', '.pyd']:
+ path = path[:-1]
+
+ if _modified(path):
+ return _restart(path)
+
+ # Check modification times on files which have
+ # specifically been registered for monitoring.
+
+ for path in _files:
+ if _modified(path):
+ return _restart(path)
+
+ # Go to sleep for specified interval.
+
+ try:
+ return _queue.get(timeout=_interval)
+ except:
+ pass
+
+_thread = threading.Thread(target=_monitor)
+_thread.setDaemon(True)
+
+def _exiting():
+ try:
+ _queue.put(True)
+ except:
+ pass
+ _thread.join()
+
+atexit.register(_exiting)
+
+def track(path):
+ if not path in _files:
+ _files.append(path)
+
+def start(interval=1.0):
+ global _interval
+ if interval < _interval:
+ _interval = interval
+
+ global _running
+ _lock.acquire()
+ if not _running:
+ prefix = 'monitor (pid=%d):' % os.getpid()
+ print >> sys.stderr, '%s Starting change monitor.' % prefix
+ _running = True
+ _thread.start()
Added: zorg/trunk/lnt/viewer/zorg.cfg.sample
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/zorg.cfg.sample?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/zorg.cfg.sample (added)
+++ zorg/trunk/lnt/viewer/zorg.cfg.sample Sat Mar 20 20:00:06 2010
@@ -0,0 +1,45 @@
+# -*- Python -*-
+
+# LNT (aka Zorg) configuration file
+#
+# Paths are resolved relative to this file.
+
+# Path to the LNT root.
+zorg = ".."
+
+# Path to the LNT server.
+zorgURL = "http://llvm.org/perf/"
+
+# The list of available databases, and their properties.
+databases = {
+ 'default' : { 'path' : '../data/default.db',
+ 'showNightlytest' : 1 },
+ 'test' : { 'path' : '../data/test.db',
+ 'showNightlytest' : 1,
+ 'showGeneral' : 1 },
+ 'nt' : { 'path' : '../data/nt_internal.db',
+ 'showNightlytest' : 1 },
+ 'nt_mysql' : { 'path' : 'mysql://root:admin@localhost/nt_internal',
+ 'showNightlytest' : 1 },
+ }
+
+# The LNT email configuration.
+#
+# The 'to' field can be either a single email address, or a list of
+# (regular-expression, address) pairs. In the latter form, the machine name of
+# the submitted results is matched against the regular expressions to determine
+# which email address to use for the results.
+nt_emailer = {
+ 'enabled' : True,
+ 'host' : "llvm.org",
+ 'from' : "lnt at llvm.org",
+
+ # This is a list of (filter-regexp, address) pairs -- it is evaluated in
+ # order based on the machine name. This can be used to dispatch different
+ # reports to different email address.
+ 'to' : [(".*", "llvm-testresults at cs.uiuc.edu")]),
+ }
+
+# Enable automatic restart using the wsgi_restart module; this should be off in
+# a production environment.
+wsgi_restart = False
Added: zorg/trunk/lnt/viewer/zorg.cgi
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/zorg.cgi?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/zorg.cgi (added)
+++ zorg/trunk/lnt/viewer/zorg.cgi Sat Mar 20 20:00:06 2010
@@ -0,0 +1,45 @@
+#!/usr/bin/env python
+# -*- Python -*-
+
+import sys
+import os
+
+# These were just some local hacks I used at some point to enable testing with
+# MySQL. We were running afoul of cimport issues, I think. Revisit when we care
+# about MySQL.
+if 0:
+ os.environ['PATH'] += ':/usr/local/mysql/bin'
+
+ os.environ['PYTHON_EGG_CACHE'] = '/tmp'
+ import MySQLdb
+
+ import PerfDB
+ db = PerfDB.PerfDB("mysql://root:admin@localhost/nt_internal")
+ from PerfDB import Machine
+ q = db.session.query(Machine.name).distinct().order_by(Machine.name)
+ for i in q[:1]:
+ break
+
+def create_publisher():
+ import warnings
+ warnings.simplefilter("ignore", category=DeprecationWarning)
+
+ # We expect the config file to be adjacent to the absolute path of
+ # the cgi script.
+ configPath = os.path.join(os.path.dirname(os.path.realpath(__file__)),
+ "zorg.cfg")
+ configData = {}
+ exec open(configPath) in configData
+
+ # Find the zorg installation dir.
+ zorgDir = os.path.join(os.path.dirname(configPath),
+ configData.get('zorg', ''))
+ if zorgDir and zorgDir not in sys.path:
+ sys.path.append(zorgDir)
+
+ from viewer import publisher
+ return publisher.create_publisher(configPath, configData)
+
+if __name__ == '__main__':
+ from quixote.server import cgi_server
+ cgi_server.run(create_publisher)
Propchange: zorg/trunk/lnt/viewer/zorg.cgi
------------------------------------------------------------------------------
svn:executable = *
Added: zorg/trunk/lnt/viewer/zorg.wsgi
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/zorg.wsgi?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/zorg.wsgi (added)
+++ zorg/trunk/lnt/viewer/zorg.wsgi Sat Mar 20 20:00:06 2010
@@ -0,0 +1,41 @@
+#!/usr/bin/env python2.6
+# -*- Python -*-
+
+import sys
+import os
+
+def create_publisher():
+ import warnings
+ warnings.simplefilter("ignore", category=DeprecationWarning)
+
+ # We expect the config file to be adjacent to the absolute path of
+ # the cgi script.
+ configPath = os.path.join(os.path.dirname(os.path.realpath(__file__)),
+ "zorg.cfg")
+ configData = {}
+ exec open(configPath) in configData
+
+ # Find the zorg installation dir.
+ zorgDir = os.path.join(os.path.dirname(configPath),
+ configData.get('zorg', ''))
+ if zorgDir and zorgDir not in sys.path:
+ sys.path.append(zorgDir)
+
+ # Optionally enable auto-restart.
+ if configData.get('wsgiAutoRestart', 'True'):
+ from viewer import wsgi_restart
+ wsgi_restart.track(configPath)
+ wsgi_restart.start()
+
+ from viewer import publisher
+ return publisher.create_publisher(configPath, configData, threaded=True)
+
+from quixote.wsgi import QWIP
+application = QWIP(create_publisher())
+
+if __name__ == '__main__':
+ from wsgiref.simple_server import make_server
+ print "Running test application."
+ print " open http://localhost:8000/"
+ httpd = make_server('', 8000, application)
+ httpd.serve_forever()
Propchange: zorg/trunk/lnt/viewer/zorg.wsgi
------------------------------------------------------------------------------
svn:executable = *
Added: zorg/trunk/lnt/viewer/zview/__init__.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/zview/__init__.py?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/zview/__init__.py (added)
+++ zorg/trunk/lnt/viewer/zview/__init__.py Sat Mar 20 20:00:06 2010
@@ -0,0 +1 @@
+__all__ = []
Added: zorg/trunk/lnt/viewer/zview/zviewui.ptl
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/viewer/zview/zviewui.ptl?rev=99107&view=auto
==============================================================================
--- zorg/trunk/lnt/viewer/zview/zviewui.ptl (added)
+++ zorg/trunk/lnt/viewer/zview/zviewui.ptl Sat Mar 20 20:00:06 2010
@@ -0,0 +1,211 @@
+from quixote.directory import Directory
+from quixote.html import htmltext
+
+from viewer.PerfDB import Machine, Run, Sample, Test
+from viewer import NTUtil
+
+from sqlalchemy import func
+
+import json
+
+class ZViewUI(Directory):
+ _q_exports = ["", "get_machines", "get_tests", "get_test_data",
+ "get_test_names"]
+
+ def __init__(self, root):
+ self.root = root
+
+ def get_machines(self):
+ db = self.root.getDB()
+ q = db.session.query(Machine.name.distinct())
+ q = q.order_by(Machine.name)
+ return json.dumps(q.all())
+
+ def get_tests(self):
+ db = self.root.getDB()
+ q = db.session.query(Test.id, Test.name)
+ q = q.order_by([Test.name])
+ return json.dumps(q.all())
+
+ def get_test_data(self):
+ import quixote, time
+ from sqlalchemy import orm
+
+ request = quixote.get_request()
+ machine_name = str(request.form.get('machine_name'))
+ test_name = str(request.form.get('test_name'))
+ component = str(request.form.get('component'))
+
+ full_test_name = 'nightlytest.' + test_name + '.' + component
+
+ # FIXME: Return data about machine crossings.
+ db = self.root.getDB()
+ q = db.session.query(Test.id).filter(Test.name == full_test_name)
+ q = db.session.query(Run.start_time, Sample.value)
+ q = q.join(Sample).join(Test)
+ q = q.filter(Test.name == full_test_name)
+ q = q.join(Machine)
+ q = q.filter(Machine.name == machine_name)
+ q = q.order_by(Run.start_time.desc())
+ return json.dumps([(time.mktime(run_time.timetuple()), value)
+ for run_time,value in q])
+
+ def get_test_names(self):
+ # FIXME: We should fix the DB to be able to do this directly.
+ left = NTUtil.kPrefix + '.'
+ right = '.' + NTUtil.kSentinelKeyName
+ f = func.substr(Test.name, len(left) + 1,
+ func.length(Test.name) - len(left) - len(right))
+
+ db = self.root.getDB()
+ q = db.session.query(f)
+ q = q.filter(Test.name.startswith(left))
+ q = q.filter(Test.name.endswith(right))
+ q = q.order_by(Test.name.desc())
+ return json.dumps(q.all())
+
+ def _q_index [html] (self):
+ db = self.root.getDB()
+
+ script = """
+machines = null;
+tests = null;
+graph = null;
+active_test_data = null;
+
+function update_machine_list(data, text) {
+ machines = data;
+
+ var elt = $('test_select_form_machine');
+ elt.length = data.length;
+ for (var i = 0; i != data.length; ++i) {
+ elt[i].value = data[i];
+ elt[i].text = data[i];
+ }
+
+ handle_test_change();
+}
+
+function update_test_list(data, text) {
+ tests = data;
+
+ var elt = $('test_select_form_test');
+ elt.length = data.length;
+ for (var i = 0; i != data.length; ++i) {
+ elt[i].value = data[i];
+ elt[i].text = data[i];
+ }
+
+ handle_test_change();
+}
+
+function update_graph() {
+ update_selected_status();
+
+ graph.clearPlots();
+ if (active_test_data && active_test_data.length) {
+ graph.clearColor = [1, 1, 1];
+ graph.addPlot(active_test_data, new Graph2D_LinePlotStyle(1, [0,0,0]));
+ } else {
+ graph.clearColor = [1, .8, .8];
+ }
+ graph.draw();
+}
+
+function update_selected_status() {
+ var machine_elt = $('test_select_form_machine');
+ var test_elt = $('test_select_form_test');
+ var machine = machines && machines[machine_elt.selectedIndex];
+ var test = tests && tests[test_elt.selectedIndex];
+ var numPts = active_test_data && active_test_data.length;
+ $('log').innerHTML = "<b>Machine:</b> " + machine + "<br>" +
+ "<b>Test:</b> " + test + "<br>" +
+ "<b>Num Points:</b> " + numPts;
+}
+
+function handle_test_change() {
+ if (machines === null || tests === null)
+ return;
+
+ var machine_elt = $('test_select_form_machine');
+ var test_elt = $('test_select_form_test');
+ var machine = machines[machine_elt.selectedIndex];
+ var test = tests[test_elt.selectedIndex];
+ var component = $('test_select_form_component').value;
+
+ new Request.JSON({
+ url: 'get_test_data',
+ method: 'get',
+ onSuccess: function(data, text) {
+ active_test_data = data;
+ update_graph();
+ },
+ data: "machine_name=" + encodeURIComponent(machine) + "&" +
+ "test_name=" + encodeURIComponent(test) + "&" +
+ "component=" + component,
+ }).send();
+}
+
+function init() {
+ // Initialize the graph object.
+ graph = new Graph2D("graph");
+ graph.xAxis.format = graph.xAxis.formats.day;
+ update_graph();
+
+ // Load the machine lists.
+ new Request.JSON({
+ url: 'get_machines',
+ onSuccess: update_machine_list,
+ }).send();
+
+ // Load the test list.
+ new Request.JSON({
+ url: 'get_test_names',
+ onSuccess: update_test_list,
+ }).send();
+}
+""" % locals()
+
+ self.root.getHeader("zorg", "..", addGraphJS=True, addJSScript=script,
+ onload='init()')
+
+ """
+ <h2>ZView</h2>
+ """
+
+ """
+ <h3>Test Selection</h3>
+ <form id="test_select_form">
+ <p>Machine: <select id="test_select_form_machine"
+ onChange="handle_test_change();">
+ <option value="">Loading...</option>
+ </select></p>
+
+ <p>Test: <select id="test_select_form_test"
+ onChange="handle_test_change();">
+ <option value="">Loading...</option>
+ </select></p>
+
+ <p>Component: <select id="test_select_form_component"
+ onChange="handle_test_change();">
+ """
+ for name,key in NTUtil.kComparisonKinds:
+ if key is None:
+ continue
+ """<option value="%s">%s</option>""" % (key, name)
+ """
+ </select></p>
+ </form>
+
+ <h3>Selected Test</h3>
+ <div id="log">
+ <p>Waiting...</p>
+ </div>
+
+ <h3>Graph</h3>
+ <div id="log">
+ <canvas id="graph" width="600" height="400"></canvas>
+ </div>
+ """
+
+ self.root.getFooter()
More information about the llvm-commits
mailing list