[llvm-commits] [zorg] r146776 - in /zorg/trunk/lnt: docs/todo.rst lnt/server/db/testsuite.py lnt/server/db/testsuitedb.py tests/server/db/CreateV4TestSuiteInstance.py

Daniel Dunbar daniel at zuster.org
Fri Dec 16 15:14:57 PST 2011

Author: ddunbar
Date: Fri Dec 16 17:14:57 2011
New Revision: 146776

URL: http://llvm.org/viewvc/llvm-project?rev=146776&view=rev
[v0.4]: Move to a model where the test suite is also responsible for defining
the set of sample values that can be reported by a run.
 - Based on substantial discussion with Jakob, see the todo.rst changes for
   longer explanation.

Main notable implications:
 * Should result in substantially better performance.
 * Eliminates need for mangling metadata into test names.
 * Makes some of my future goals more easy to realize (reporting binary hashes,
   for example).
 * Eliminates my reservations w.r.t. moving the status field (by itself) into
   the Sample.

I'm a bit worried about what this will mean for the UI code, but we will cross
that bridge when we come to it.


Modified: zorg/trunk/lnt/docs/todo.rst
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/docs/todo.rst?rev=146776&r1=146775&r2=146776&view=diff
--- zorg/trunk/lnt/docs/todo.rst (original)
+++ zorg/trunk/lnt/docs/todo.rst Fri Dec 16 17:14:57 2011
@@ -121,24 +121,35 @@
 but makes the UI code very painful. Especially, some things like making a graph
 of the test values for all tests which passed become incredibly complex.
-It is obvious that we need to move the "test status" indication into the sample
-field. However, this raises the question of what *else* should move into the
-status field. Is there a more general problem here?
-Also, just moving the test status in introduces other modeling problem, for
-example what about tests for which there is no meaningful notion of status? For
-example, an aggregate test that may produce a large number of results on
-success. Does it make sense to think about all those tests as succeeding? It
-seems ok, but not quite accurate.
-The only other option I can think of at the moment would be to allow multiple
-kinds of samples. This might be worth considering, because it also solves other
-problems like associating types with samples (instead of just requiring every
-sample to be a float).
-In particular, one thing I would very much like would be to be able to report
-the hash of the generated exutable along whenever we compile something (and
-eventually dispatch them to an archival server).
+The plan is to handle this problem by also constructing the Sample tables on the
+fly, and allowing the test suite to define the keys that go into a sample. Thus,
+any one sample will reflect *all* of the statistics that were reported for that
+This has many advantages:
+ * We can start using types for the columns (e.g., easy to start reporting hash
+   of produced binaries, for example).
+ * The performance impact of adding new sample values should be much lower than
+   in the old schema.
+ * The database explicitly models the fact that sample values were produced from
+   a single run, whereas before sample values and status could not technically
+   be correlated correctly.
+ * We eliminate the need to mangle subtest/test result key information into the
+   test name, which is a big win.
+The has some disadvantages, however:
+ * Poorly models suites where different tests reported different test results.
+ * Poorly models situations where we want to support a large number of test
+   results and that set might change frequently or dynamically (e.g., suppose we
+   reported one test for each LLVM Statistic counter).
+However, at least for our current usage this scheme should work well enough and
+be **substantially** faster than the old scheme.
+This will probably mean that we have to do a bit of work (similar to what we had
+to do for parameter sets) to handle what the UI for this should look
+like. However, we should have better infrastructure for defining how the UI
+should handle things in the metadata.
 Other Antipatterns In Use
@@ -186,6 +197,13 @@
  - ID
  - Name
+ - ID
+ - Name
+   Initially, we will only have Real and Status. Integer and Hash are obvious
+   extensions.
  - ID
  - Name
@@ -201,7 +219,7 @@
  - TestSuite FOREIGN KEY TestSuite(ID)
- - Name VARCHAR(512)
+ - Name VARCHAR(256)
  - Type?
  - Default?
  - InfoKey
@@ -212,7 +230,7 @@
  - TestSuite FOREIGN KEY TestSuite(ID)
- - Name VARCHAR(512)
+ - Name VARCHAR(256)
  - Index INTEGER
    The ordinal index this order key should appear in.
@@ -223,7 +241,7 @@
  - TestSuite FOREIGN KEY TestSuite(ID)
- - Name VARCHAR(512)
+ - Name VARCHAR(256)
  - Type?
  - Default?
  - InfoKey
@@ -231,6 +249,17 @@
    This is used for migration purposes (possibly permanent), it is the key name
    to expect this field to be present as in a submitted info dictionary.
+ - TestSuite FOREIGN KEY TestSuite(ID)
+ - Name VARCHAR(256)
+ - Type FOREIGN KEY SampleType(ID)
+ - Default?
+ - InfoKey
+   This is used for migration purposes (possibly permanent), it is the key name
+   to expect this field to be present as in a submitted info dictionary.
 Per Test Suite
@@ -293,8 +322,7 @@
  - Run FOREIGN Key <TS>_Run(ID)
  - Test FOREIGN KEY <TS>_Test(ID)
- - Value REAL
- - Status FOREIGN KEY StatusKind(ID)
+ - ... keys here are defined by TestSuite(SampleKeys) relation ...
  - Indices::
      CREATE INDEX [<TS>_Samples_RunID_IDX] ON <TS>_Sample(RunID);
@@ -347,35 +375,16 @@
    This is the one area where I really don't want to change the test data
    serialization format, so maybe this is even the right long term approach.
+   In the new model of collapsing samples into a single row, this is going to
+   mean that we will need to assume that tests mangled into subtest names are
+   specified in the same order.
 Unaddressed Issues
 There are couple design problems in the current system which I *am not*
 intending to address as part of the v0.4 changes.
-Test / Subtest Relationships
-We currently impose a certain amount of structure on the tests and mangle it
-into the name (e.g., as "<test_name>.compile" or "<test_name>.exec", not to
-mention the status indicators). The status indicators are going to go away in
-the redesign, but we will still have the distinction between ".compile" and
-This works ok up to the point where we want to do things in the UI based on the
-test structure (for example, separate out compile time results from execution
-results). At the moment we have gross code in the UI which just happens to
-"know" these manglings, but it would be much better to have this be explicit in
-the schema.
-While I don't have a plan to solve this in the current iteration, I can imagine
-one way to solve this is to allow the test suite to define additional metadata
-that is present in the Test table. We would allow that metadata to specify how
-tests should be displayed, their units, etc.
-This could be more important problem going forward if we wanted to start
-reporting large numbers of additional statistics (like number of spills, etc.).
 Machine Naming

Modified: zorg/trunk/lnt/lnt/server/db/testsuite.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/server/db/testsuite.py?rev=146776&r1=146775&r2=146776&view=diff
--- zorg/trunk/lnt/lnt/server/db/testsuite.py (original)
+++ zorg/trunk/lnt/lnt/server/db/testsuite.py Fri Dec 16 17:14:57 2011
@@ -10,6 +10,15 @@
 from sqlalchemy.orm import relation
 Base = sqlalchemy.ext.declarative.declarative_base()
+class SampleType(Base):
+    __tablename__ = 'SampleType'
+    id = Column("ID", Integer, primary_key=True)
+    name = Column("Name", String(256))
+    def __init__(self, name):
+        self.name = name
 class StatusKind(Base):
     __tablename__ = 'StatusKind'
@@ -30,6 +39,7 @@
     machine_fields = relation('MachineField', backref='test_suite')
     order_fields = relation('OrderField', backref='test_suite')
     run_fields = relation('RunField', backref='test_suite')
+    sample_fields = relation('SampleField', backref='test_suite')
     def __init__(self, name, db_key_name, version):
         self.name = name
@@ -79,3 +89,16 @@
     def __init__(self, name, info_key):
         self.name = name
         self.info_key = info_key
+class SampleField(Base):
+    __tablename__ = 'TestSuiteSampleFields'
+    id = Column("ID", Integer, primary_key=True)
+    test_suite_id = Column("TestSuiteID", Integer, ForeignKey('TestSuite.ID'))
+    name = Column("Name", String(256))
+    type = Column("Type", Integer, ForeignKey('SampleType.ID'))
+    info_key = Column("InfoKey", String(256))
+    def __init__(self, name, info_key):
+        self.name = name
+        self.info_key = info_key

Modified: zorg/trunk/lnt/lnt/server/db/testsuitedb.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/lnt/server/db/testsuitedb.py?rev=146776&r1=146775&r2=146776&view=diff
--- zorg/trunk/lnt/lnt/server/db/testsuitedb.py (original)
+++ zorg/trunk/lnt/lnt/server/db/testsuitedb.py Fri Dec 16 17:14:57 2011
@@ -98,17 +98,15 @@
             id = Column("ID", Integer, primary_key=True)
             run_id = Column("RunID", Integer, ForeignKey(Run.id))
             test_id = Column("TestID", Integer, ForeignKey(Test.id))
-            value = Column("Value", Float)
-            status_id = Column("StatusID", Integer, ForeignKey(
-                    testsuite.StatusKind.id))
             run = sqlalchemy.orm.relation(Run)
             test = sqlalchemy.orm.relation(Test)
-            def __init__(self, run, test, value):
+            # ... FIXME: Add test suite sample keys ...
+            def __init__(self, run, test):
                 self.run = run
                 self.test = test
-                self.value = value
             def __repr__(self):
                 return '%s_%s%r' % (db_key_name, self.__class__.__name__,

Modified: zorg/trunk/lnt/tests/server/db/CreateV4TestSuiteInstance.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/lnt/tests/server/db/CreateV4TestSuiteInstance.py?rev=146776&r1=146775&r2=146776&view=diff
--- zorg/trunk/lnt/tests/server/db/CreateV4TestSuiteInstance.py (original)
+++ zorg/trunk/lnt/tests/server/db/CreateV4TestSuiteInstance.py Fri Dec 16 17:14:57 2011
@@ -31,7 +31,7 @@
 order = ts_db.Order()
 run = ts_db.Run(machine, order, start_time, end_time)
 test = ts_db.Test("test-a")
-sample = ts_db.Sample(run, test, 1.0)
+sample = ts_db.Sample(run, test)
 # Add and commit.
@@ -76,4 +76,3 @@
 assert sample.run is run
 assert sample.test is test
-assert sample.value == 1.0

More information about the llvm-commits mailing list