[llvm-commits] [zorg] r146776 - in /zorg/trunk/lnt: docs/todo.rst lnt/server/db/testsuite.py lnt/server/db/testsuitedb.py tests/server/db/CreateV4TestSuiteInstance.py
Daniel Dunbar
daniel at zuster.org
Fri Dec 16 15:14:57 PST 2011
Author: ddunbar
Date: Fri Dec 16 17:14:57 2011
New Revision: 146776
URL: http://llvm.org/viewvc/llvm-project?rev=146776&view=rev
Log:
[v0.4]: Move to a model where the test suite is also responsible for defining
the set of sample values that can be reported by a run.
- Based on substantial discussion with Jakob, see the todo.rst changes for
longer explanation.
Main notable implications:
* Should result in substantially better performance.
* Eliminates need for mangling metadata into test names.
* Makes some of my future goals more easy to realize (reporting binary hashes,
for example).
* Eliminates my reservations w.r.t. moving the status field (by itself) into
the Sample.
I'm a bit worried about what this will mean for the UI code, but we will cross
that bridge when we come to it.
Modified:
zorg/trunk/lnt/docs/todo.rst
zorg/trunk/lnt/lnt/server/db/testsuite.py
zorg/trunk/lnt/lnt/server/db/testsuitedb.py
zorg/trunk/lnt/tests/server/db/CreateV4TestSuiteInstance.py
Modified: zorg/trunk/lnt/docs/todo.rst
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/docs/todo.rst?rev=146776&r1=146775&r2=146776&view=diff
==============================================================================
--- zorg/trunk/lnt/docs/todo.rst (original)
+++ zorg/trunk/lnt/docs/todo.rst Fri Dec 16 17:14:57 2011
@@ -121,24 +121,35 @@
but makes the UI code very painful. Especially, some things like making a graph
of the test values for all tests which passed become incredibly complex.
-It is obvious that we need to move the "test status" indication into the sample
-field. However, this raises the question of what *else* should move into the
-status field. Is there a more general problem here?
-
-Also, just moving the test status in introduces other modeling problem, for
-example what about tests for which there is no meaningful notion of status? For
-example, an aggregate test that may produce a large number of results on
-success. Does it make sense to think about all those tests as succeeding? It
-seems ok, but not quite accurate.
-
-The only other option I can think of at the moment would be to allow multiple
-kinds of samples. This might be worth considering, because it also solves other
-problems like associating types with samples (instead of just requiring every
-sample to be a float).
-
-In particular, one thing I would very much like would be to be able to report
-the hash of the generated exutable along whenever we compile something (and
-eventually dispatch them to an archival server).
+The plan is to handle this problem by also constructing the Sample tables on the
+fly, and allowing the test suite to define the keys that go into a sample. Thus,
+any one sample will reflect *all* of the statistics that were reported for that
+test.
+
+This has many advantages:
+ * We can start using types for the columns (e.g., easy to start reporting hash
+ of produced binaries, for example).
+ * The performance impact of adding new sample values should be much lower than
+ in the old schema.
+ * The database explicitly models the fact that sample values were produced from
+ a single run, whereas before sample values and status could not technically
+ be correlated correctly.
+ * We eliminate the need to mangle subtest/test result key information into the
+ test name, which is a big win.
+
+The has some disadvantages, however:
+ * Poorly models suites where different tests reported different test results.
+ * Poorly models situations where we want to support a large number of test
+ results and that set might change frequently or dynamically (e.g., suppose we
+ reported one test for each LLVM Statistic counter).
+
+However, at least for our current usage this scheme should work well enough and
+be **substantially** faster than the old scheme.
+
+This will probably mean that we have to do a bit of work (similar to what we had
+to do for parameter sets) to handle what the UI for this should look
+like. However, we should have better infrastructure for defining how the UI
+should handle things in the metadata.
Other Antipatterns In Use
+++++++++++++++++++++++++
@@ -186,6 +197,13 @@
- ID
- Name
+SampleType
+ - ID
+ - Name
+
+ Initially, we will only have Real and Status. Integer and Hash are obvious
+ extensions.
+
TestSuite
- ID
- Name
@@ -201,7 +219,7 @@
TestSuiteMachineKeys
- ID INTEGER PRIMARY KEY
- TestSuite FOREIGN KEY TestSuite(ID)
- - Name VARCHAR(512)
+ - Name VARCHAR(256)
- Type?
- Default?
- InfoKey
@@ -212,7 +230,7 @@
TestSuiteOrderKeys
- ID INTEGER PRIMARY KEY
- TestSuite FOREIGN KEY TestSuite(ID)
- - Name VARCHAR(512)
+ - Name VARCHAR(256)
- Index INTEGER
The ordinal index this order key should appear in.
@@ -223,7 +241,7 @@
TestSuiteRunKeys
- ID INTEGER PRIMARY KEY
- TestSuite FOREIGN KEY TestSuite(ID)
- - Name VARCHAR(512)
+ - Name VARCHAR(256)
- Type?
- Default?
- InfoKey
@@ -231,6 +249,17 @@
This is used for migration purposes (possibly permanent), it is the key name
to expect this field to be present as in a submitted info dictionary.
+TestSuiteSampleKeys
+ - ID INTEGER PRIMARY KEY
+ - TestSuite FOREIGN KEY TestSuite(ID)
+ - Name VARCHAR(256)
+ - Type FOREIGN KEY SampleType(ID)
+ - Default?
+ - InfoKey
+
+ This is used for migration purposes (possibly permanent), it is the key name
+ to expect this field to be present as in a submitted info dictionary.
+
Per Test Suite
++++++++++++++
@@ -293,8 +322,7 @@
- ID INTEGER PRIMARY KEY
- Run FOREIGN Key <TS>_Run(ID)
- Test FOREIGN KEY <TS>_Test(ID)
- - Value REAL
- - Status FOREIGN KEY StatusKind(ID)
+ - ... keys here are defined by TestSuite(SampleKeys) relation ...
- Indices::
CREATE INDEX [<TS>_Samples_RunID_IDX] ON <TS>_Sample(RunID);
@@ -347,35 +375,16 @@
This is the one area where I really don't want to change the test data
serialization format, so maybe this is even the right long term approach.
+ In the new model of collapsing samples into a single row, this is going to
+ mean that we will need to assume that tests mangled into subtest names are
+ specified in the same order.
+
Unaddressed Issues
~~~~~~~~~~~~~~~~~~
There are couple design problems in the current system which I *am not*
intending to address as part of the v0.4 changes.
-Test / Subtest Relationships
-++++++++++++++++++++++++++++
-
-We currently impose a certain amount of structure on the tests and mangle it
-into the name (e.g., as "<test_name>.compile" or "<test_name>.exec", not to
-mention the status indicators). The status indicators are going to go away in
-the redesign, but we will still have the distinction between ".compile" and
-".exec".
-
-This works ok up to the point where we want to do things in the UI based on the
-test structure (for example, separate out compile time results from execution
-results). At the moment we have gross code in the UI which just happens to
-"know" these manglings, but it would be much better to have this be explicit in
-the schema.
-
-While I don't have a plan to solve this in the current iteration, I can imagine
-one way to solve this is to allow the test suite to define additional metadata
-that is present in the Test table. We would allow that metadata to specify how
-tests should be displayed, their units, etc.
-
-This could be more important problem going forward if we wanted to start
-reporting large numbers of additional statistics (like number of spills, etc.).
-
Machine Naming
++++++++++++++
Modified: zorg/trunk/lnt/lnt/server/db/testsuite.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/server/db/testsuite.py?rev=146776&r1=146775&r2=146776&view=diff
==============================================================================
--- zorg/trunk/lnt/lnt/server/db/testsuite.py (original)
+++ zorg/trunk/lnt/lnt/server/db/testsuite.py Fri Dec 16 17:14:57 2011
@@ -10,6 +10,15 @@
from sqlalchemy.orm import relation
Base = sqlalchemy.ext.declarative.declarative_base()
+class SampleType(Base):
+ __tablename__ = 'SampleType'
+
+ id = Column("ID", Integer, primary_key=True)
+ name = Column("Name", String(256))
+
+ def __init__(self, name):
+ self.name = name
+
class StatusKind(Base):
__tablename__ = 'StatusKind'
@@ -30,6 +39,7 @@
machine_fields = relation('MachineField', backref='test_suite')
order_fields = relation('OrderField', backref='test_suite')
run_fields = relation('RunField', backref='test_suite')
+ sample_fields = relation('SampleField', backref='test_suite')
def __init__(self, name, db_key_name, version):
self.name = name
@@ -79,3 +89,16 @@
def __init__(self, name, info_key):
self.name = name
self.info_key = info_key
+
+class SampleField(Base):
+ __tablename__ = 'TestSuiteSampleFields'
+
+ id = Column("ID", Integer, primary_key=True)
+ test_suite_id = Column("TestSuiteID", Integer, ForeignKey('TestSuite.ID'))
+ name = Column("Name", String(256))
+ type = Column("Type", Integer, ForeignKey('SampleType.ID'))
+ info_key = Column("InfoKey", String(256))
+
+ def __init__(self, name, info_key):
+ self.name = name
+ self.info_key = info_key
Modified: zorg/trunk/lnt/lnt/server/db/testsuitedb.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/server/db/testsuitedb.py?rev=146776&r1=146775&r2=146776&view=diff
==============================================================================
--- zorg/trunk/lnt/lnt/server/db/testsuitedb.py (original)
+++ zorg/trunk/lnt/lnt/server/db/testsuitedb.py Fri Dec 16 17:14:57 2011
@@ -98,17 +98,15 @@
id = Column("ID", Integer, primary_key=True)
run_id = Column("RunID", Integer, ForeignKey(Run.id))
test_id = Column("TestID", Integer, ForeignKey(Test.id))
- value = Column("Value", Float)
- status_id = Column("StatusID", Integer, ForeignKey(
- testsuite.StatusKind.id))
run = sqlalchemy.orm.relation(Run)
test = sqlalchemy.orm.relation(Test)
- def __init__(self, run, test, value):
+ # ... FIXME: Add test suite sample keys ...
+
+ def __init__(self, run, test):
self.run = run
self.test = test
- self.value = value
def __repr__(self):
return '%s_%s%r' % (db_key_name, self.__class__.__name__,
Modified: zorg/trunk/lnt/tests/server/db/CreateV4TestSuiteInstance.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/tests/server/db/CreateV4TestSuiteInstance.py?rev=146776&r1=146775&r2=146776&view=diff
==============================================================================
--- zorg/trunk/lnt/tests/server/db/CreateV4TestSuiteInstance.py (original)
+++ zorg/trunk/lnt/tests/server/db/CreateV4TestSuiteInstance.py Fri Dec 16 17:14:57 2011
@@ -31,7 +31,7 @@
order = ts_db.Order()
run = ts_db.Run(machine, order, start_time, end_time)
test = ts_db.Test("test-a")
-sample = ts_db.Sample(run, test, 1.0)
+sample = ts_db.Sample(run, test)
# Add and commit.
ts_db.add(machine)
@@ -76,4 +76,3 @@
assert sample.run is run
assert sample.test is test
-assert sample.value == 1.0
More information about the llvm-commits
mailing list