[LNT] r307222 - Load TestSuite descriptions from json files
Matthias Braun via llvm-commits
llvm-commits at lists.llvm.org
Wed Jul 5 16:34:37 PDT 2017
Author: matze
Date: Wed Jul 5 16:34:37 2017
New Revision: 307222
URL: http://llvm.org/viewvc/llvm-project?rev=307222&view=rev
Log:
Load TestSuite descriptions from json files
The possibility to create custom test suite shemas was already present
in LNT; however it was inconvenient to use as it required the
adminitrator to directly insert descriptions into some (meta) tables in
the database. This adds a more accessible solution:
This creates a new $INSTANCEPATH/schemas directory in which administrators
can place .yaml files describing a testsuite schema. The idea is that
those can be distributed with the testsuites and administrators can copy
or symlink the schema into their server instances.
See the added example in the documentation for an example of the format.
Added:
lnt/trunk/docs/schema-example.yaml
lnt/trunk/tests/server/db/Inputs/customschema-report.json
lnt/trunk/tests/server/db/yamlschema.shtest
Modified:
lnt/trunk/docs/importing_data.rst
lnt/trunk/lnt/lnttool/create.py
lnt/trunk/lnt/lnttool/viewcomparison.py
lnt/trunk/lnt/server/config.py
lnt/trunk/lnt/server/db/testsuite.py
lnt/trunk/lnt/server/db/testsuitedb.py
lnt/trunk/lnt/server/db/v4db.py
lnt/trunk/requirements.client.txt
Modified: lnt/trunk/docs/importing_data.rst
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/docs/importing_data.rst?rev=307222&r1=307221&r2=307222&view=diff
==============================================================================
--- lnt/trunk/docs/importing_data.rst (original)
+++ lnt/trunk/docs/importing_data.rst Wed Jul 5 16:34:37 2017
@@ -101,32 +101,20 @@ Simply put, suites are a collections of
You can define your own test-suites if the schema in a different suite does not
already meet your needs.
-Creating a suite requires database access, and shell access to the machine.
-First create the metadata tables, then tell LNT to build the suites tables from
-the new metadata you have added.
+To create a schema place a yaml file into the schemas directory of your lnt
+instance. Example:
- * Open the database you want to add the suite to.
- * Add a new row to the TestSite table, note the ID.
- * Add machine and run fields to TestSuiteMachineFields and TestSuiteRunFields.
- * Add an Order to TestSuiteOrderFields. The only order name that is regularly
- tested is llvm_project_revision, so you may want to use that name.
- * Add new entries to the TestSuiteSampleFields for each metric you want to
- collect.
- * Now create the new LNT tables via the shell interface. In this example
- we make a tables for the size testsuite in the ecc database::
+.. literalinclude:: schema-example.yaml
+ :language: yaml
- $ lnt runserver --shell ./foo
- Started file logging.
- Logging to : lnt.log
- Python 2.7.5 (default, Mar 9 2014, 22:15:05)
- [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
- Type "help", "copyright", "credits" or "license" for more information.
- (InteractiveConsole)
- >>> g.db_name = "ecc"
- >>> db = ctx.request.get_db()
- >>> db
- <lnt.server.db.v4db.V4DB object at 0x10ac4afd0>
- >>> import lnt.server.db.migrations.new_suite as ns
- >>> ns.init_new_testsuite(db.engine, db.session, "size")
- >>> db.session.commit()
+* LNT currently supports the following metric types:
+ - ``Integer``: Integer values; postgres limits this to 4 bytes,
+ sqlite support up to 8 bytes.
+ - ``Real``: 8-byte IEEE floating point values.
+ - ``Hash``: String values; limited to 256, sqlite is not enforcing the limit.
+ - ``Status``: StatusKind enum values (limited to 'PASS', 'FAIL', 'XFAIL' right
+ now).
+
+* You need to mark at least 1 of the run fields as ``order: true`` so LNT knows
+ how to sort runs.
Added: lnt/trunk/docs/schema-example.yaml
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/docs/schema-example.yaml?rev=307222&view=auto
==============================================================================
--- lnt/trunk/docs/schema-example.yaml (added)
+++ lnt/trunk/docs/schema-example.yaml Wed Jul 5 16:34:37 2017
@@ -0,0 +1,20 @@
+format_version: '2'
+name: size
+metrics:
+ - name: text_size
+ bigger_is_better: false
+ type: Integer
+ - name: data_size
+ bigger_is_better: false
+ type: Integer
+ - name: score
+ bigger_is_better: true
+ type: Real
+ - name: hash
+ type: Hash
+run_fields:
+ - name: llvm_project_revision
+ order: true
+machine_fields:
+ - name: hardware
+ - name: os
Modified: lnt/trunk/lnt/lnttool/create.py
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/lnttool/create.py?rev=307222&r1=307221&r2=307222&view=diff
==============================================================================
--- lnt/trunk/lnt/lnttool/create.py (original)
+++ lnt/trunk/lnt/lnttool/create.py Wed Jul 5 16:34:37 2017
@@ -145,11 +145,13 @@ LNT configuration.
cfg_path = os.path.join(basepath, config)
tmp_path = os.path.join(basepath, tmp_dir)
wsgi_path = os.path.join(basepath, wsgi)
+ schemas_path = os.path.join(basepath, "schemas")
secret_key = (secret_key or
hashlib.sha1(str(random.getrandbits(256))).hexdigest())
os.mkdir(instance_path)
os.mkdir(tmp_path)
+ os.mkdir(schemas_path)
# If the path does not contain database type, assume relative path.
if "://" not in db_dir:
Modified: lnt/trunk/lnt/lnttool/viewcomparison.py
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/lnttool/viewcomparison.py?rev=307222&r1=307221&r2=307222&view=diff
==============================================================================
--- lnt/trunk/lnt/lnttool/viewcomparison.py (original)
+++ lnt/trunk/lnt/lnttool/viewcomparison.py Wed Jul 5 16:34:37 2017
@@ -83,7 +83,7 @@ def action_view_comparison(report_a, rep
# profileDir, secretKey, databases, blacklist):
config = lnt.server.config.Config('LNT', url, db_path, tmpdir,
None, "Not secret key.",
- {'default': db_info},
+ {'default': db_info}, None,
None)
instance = lnt.server.instance.Instance(None, config)
Modified: lnt/trunk/lnt/server/config.py
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/server/config.py?rev=307222&r1=307221&r2=307222&view=diff
==============================================================================
--- lnt/trunk/lnt/server/config.py (original)
+++ lnt/trunk/lnt/server/config.py Wed Jul 5 16:34:37 2017
@@ -104,7 +104,7 @@ class Config:
dbDir = data.get('db_dir', '.')
profileDir = data.get('profile_dir', 'data/profiles')
-
+ schemasDir = os.path.join(baseDir, 'schemas')
# If the path does not contain database type, assume relative path.
dbDirPath = dbDir if "://" in dbDir else os.path.join(baseDir, dbDir)
@@ -125,7 +125,7 @@ class Config:
default_email_config,
0))
for k, v in data['databases'].items()]),
- blacklist, api_auth_token)
+ blacklist, schemasDir, api_auth_token)
@staticmethod
def dummy_instance():
@@ -133,6 +133,7 @@ class Config:
dbDir = '.'
profileDirPath = os.path.join(baseDir, 'profiles')
tempDir = os.path.join(baseDir, 'tmp')
+ schemasDir = os.path.join(baseDir, 'schemas')
secretKey = None
dbInfo = {'dummy': DBInfo.dummy_instance()}
blacklist = None
@@ -145,6 +146,7 @@ class Config:
secretKey,
dbInfo,
blacklist,
+ schemasDir,
"test_key")
def __init__(self,
@@ -156,6 +158,7 @@ class Config:
secretKey,
databases,
blacklist,
+ schemasDir,
api_auth_token=None):
self.name = name
self.zorgURL = zorgURL
@@ -164,6 +167,7 @@ class Config:
self.secretKey = secretKey
self.blacklist = blacklist
self.profileDir = profileDir
+ self.schemasDir = schemasDir
while self.zorgURL.endswith('/'):
self.zorgURL = zorgURL[:-1]
self.databases = databases
Modified: lnt/trunk/lnt/server/db/testsuite.py
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/server/db/testsuite.py?rev=307222&r1=307221&r2=307222&view=diff
==============================================================================
--- lnt/trunk/lnt/server/db/testsuite.py (original)
+++ lnt/trunk/lnt/server/db/testsuite.py Wed Jul 5 16:34:37 2017
@@ -84,6 +84,64 @@ class TestSuite(Base):
return '%s%r' % (self.__class__.__name__, (self.name, self.db_key_name,
self.version))
+ @staticmethod
+ def from_json(data):
+ if data.get('format_version') != '2':
+ raise ValueError("Expected \"format_version\": \"2\" in schema")
+ name = data['name']
+ ts = TestSuite(data['name'], data['name'])
+
+ machine_fields = []
+ for field_desc in data.get('machine_fields', []):
+ name = field_desc['name']
+ field = MachineField(name, info_key=None)
+ machine_fields.append(field)
+ ts.machine_fields = machine_fields
+
+ run_fields = []
+ order_fields = []
+ for field_desc in data.get('run_fields', []):
+ name = field_desc['name']
+ is_order = field_desc.get('order', False)
+ if is_order:
+ field = OrderField(name, info_key=None, ordinal=0)
+ order_fields.append(field)
+ else:
+ field = RunField(name, info_key=None)
+ run_fields.append(field)
+ ts.run_fields = run_fields
+ ts.order_fields = order_fields
+ assert(len(order_fields) > 0)
+
+ # Hardcode some sample types. I wonder whether we should rather query
+ # them from the core database?
+ # This needs to be kept in sync with testsuitedb.py
+ metric_types = {
+ 'Real': SampleType('Real'),
+ 'Integer': SampleType('Integer'),
+ 'Status': SampleType('Status'),
+ 'Hash': SampleType('Hash')
+ }
+
+ sample_fields = []
+ for metric_desc in data['metrics']:
+ name = metric_desc['name']
+ bigger_is_better = metric_desc.get('bigger_is_better', False)
+ metric_type_name = metric_desc.get('type', 'Real')
+ metric_type = metric_types.get(metric_type_name)
+ if metric_type is None:
+ raise ValueError("Unknown metric type '%s' (not in %s)" %
+ (metric_type_name,
+ metric_types.keys().join(",")))
+
+ bigger_is_better_int = 1 if bigger_is_better else 0
+ field = SampleField(name, metric_type, info_key=None,
+ status_field = None,
+ bigger_is_better=bigger_is_better_int)
+ sample_fields.append(field)
+ ts.sample_fields = sample_fields
+ return ts
+
class FieldMixin(object):
@property
Modified: lnt/trunk/lnt/server/db/testsuitedb.py
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/server/db/testsuitedb.py?rev=307222&r1=307221&r2=307222&view=diff
==============================================================================
--- lnt/trunk/lnt/server/db/testsuitedb.py (original)
+++ lnt/trunk/lnt/server/db/testsuitedb.py Wed Jul 5 16:34:37 2017
@@ -38,7 +38,7 @@ class TestSuiteDB(object):
through the model classes constructed by this wrapper object.
"""
- def __init__(self, v4db, name, test_suite):
+ def __init__(self, v4db, name, test_suite, create_tables=False):
testsuitedb = self
self.v4db = v4db
self.name = name
@@ -416,7 +416,7 @@ class TestSuiteDB(object):
worse than other potential values for this field.
"""
for field in self.Sample.fields:
- if field.type.name == 'Real':
+ if field.type.name in ['Real', 'Integer']:
yield field
@staticmethod
@@ -451,6 +451,8 @@ class TestSuiteDB(object):
if item.type.name == 'Real':
item.column = Column(item.name, Float)
+ elif item.type.name == 'Integer':
+ item.column = Column(item.name, Integer)
elif item.type.name == 'Status':
item.column = Column(item.name, Integer, ForeignKey(
testsuite.StatusKind.id))
@@ -666,6 +668,9 @@ class TestSuiteDB(object):
self.query = self.v4db.query
self.rollback = self.v4db.rollback
+ if create_tables:
+ self.base.metadata.create_all(v4db.engine)
+
def get_baselines(self):
return self.query(self.Baseline).all()
Modified: lnt/trunk/lnt/server/db/v4db.py
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/server/db/v4db.py?rev=307222&r1=307221&r2=307222&view=diff
==============================================================================
--- lnt/trunk/lnt/server/db/v4db.py (original)
+++ lnt/trunk/lnt/server/db/v4db.py Wed Jul 5 16:34:37 2017
@@ -1,4 +1,7 @@
from lnt.testing.util.commands import fatal
+import glob
+import yaml
+import logging
try:
import threading
@@ -28,11 +31,18 @@ class V4DB(object):
class TestSuiteAccessor(object):
def __init__(self, v4db):
self.v4db = v4db
+ self._extra_suites = {}
self._cache = {}
def __iter__(self):
for name, in self.v4db.query(testsuite.TestSuite.name):
yield name
+ for name in self._extra_suites.keys():
+ yield name
+
+ def add_suite(self, suite):
+ name = suite.name
+ self._extra_suites[name] = suite
def get(self, name, default = None):
# Check the test suite cache, to avoid gratuitous reinstantiation.
@@ -41,15 +51,20 @@ class V4DB(object):
if name in self._cache:
return self._cache[name]
- # Get the test suite object.
- ts = self.v4db.query(testsuite.TestSuite).\
- filter(testsuite.TestSuite.name == name).first()
- if ts is None:
- return default
+ create_tables = False
+ ts = self._extra_suites.get(name)
+ if ts:
+ create_tables = True
+ else:
+ # Get the test suite object.
+ ts = self.v4db.query(testsuite.TestSuite).\
+ filter(testsuite.TestSuite.name == name).first()
+ if ts is None:
+ return default
# Instantiate the per-test suite wrapper object for this test suite.
self._cache[name] = ts = lnt.server.db.testsuitedb.TestSuiteDB(
- self.v4db, name, ts)
+ self.v4db, name, ts, create_tables=create_tables)
return ts
def __getitem__(self, name):
@@ -69,6 +84,23 @@ class V4DB(object):
for name in self:
yield name,self[name]
+ def _load_schema_file(self, schema_file):
+ with open(schema_file) as schema_fd:
+ data = yaml.load(schema_fd)
+ suite = testsuite.TestSuite.from_json(data)
+ self.testsuite.add_suite(suite)
+ logging.info("External TestSuite '%s' loaded from '%s'" %
+ (suite.name, schema_file))
+
+ def _load_shemas(self):
+ schemasDir = self.config.schemasDir
+ for schema_file in glob.glob('%s/*.yaml' % schemasDir):
+ try:
+ self._load_schema_file(schema_file)
+ except:
+ logging.error("Could not load schema '%s'" % schema_file,
+ exc_info=True)
+
def __init__(self, path, config, baseline_revision=0, echo=False):
# If the path includes no database type, assume sqlite.
if lnt.server.db.util.path_has_no_database_type(path):
@@ -126,6 +158,8 @@ class V4DB(object):
except KeyError:
fatal("sample types not initialized!")
+ self._load_shemas()
+
def close(self):
if self.session is not None:
self.session.close()
Modified: lnt/trunk/requirements.client.txt
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/requirements.client.txt?rev=307222&r1=307221&r2=307222&view=diff
==============================================================================
--- lnt/trunk/requirements.client.txt (original)
+++ lnt/trunk/requirements.client.txt Wed Jul 5 16:34:37 2017
@@ -17,3 +17,4 @@ WTForms==2.0.2
Flask-WTF==0.12
typing
click==6.7
+pyyaml==3.12
Added: lnt/trunk/tests/server/db/Inputs/customschema-report.json
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/tests/server/db/Inputs/customschema-report.json?rev=307222&view=auto
==============================================================================
--- lnt/trunk/tests/server/db/Inputs/customschema-report.json (added)
+++ lnt/trunk/tests/server/db/Inputs/customschema-report.json Wed Jul 5 16:34:37 2017
@@ -0,0 +1,20 @@
+{
+ "format_version": "2",
+ "machine": {
+ "name": "sizetester0",
+ "hostname": "mymachine.local"
+ },
+ "run": {
+ "llvm_project_revision": "642040",
+ "end_time": "2017-04-18 23:31:18",
+ "start_time": "2017-04-18 23:01:34"
+ },
+ "tests": [
+ {
+ "name": "/obj/Debug/src/lib/embUnit~armv7_cortex-r5_Oz_floatabisoft/embUnit/AssertImpl_c~armv7_cortex-r5_Oz_floatabisoft_o",
+ "text_size": 402,
+ "data_size": 568,
+ "score": 42.42
+ }
+ ]
+}
Added: lnt/trunk/tests/server/db/yamlschema.shtest
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/tests/server/db/yamlschema.shtest?rev=307222&view=auto
==============================================================================
--- lnt/trunk/tests/server/db/yamlschema.shtest (added)
+++ lnt/trunk/tests/server/db/yamlschema.shtest Wed Jul 5 16:34:37 2017
@@ -0,0 +1,15 @@
+# RUN: rm -rf "%t.install"
+# RUN: lnt create "%t.install"
+# RUN: ln -sf %{src_root}/docs/schema-example.yaml "%t.install/schemas/size.yaml"
+# RUN: lnt import "%t.install" -s size %S/Inputs/customschema-report.json --commit=1 | FileCheck %s
+
+# CHECK: Import succeeded.
+# CHECK: Imported Data
+# CHECK: -------------
+# CHECK: Added Machines: 1
+# CHECK: Added Runs : 1
+# CHECK: Added Tests : 1
+#
+# CHECK: Results
+# CHECK: ----------------
+# CHECK: PASS : 3
More information about the llvm-commits
mailing list