[LLVMbugs] [Bug 379] NEW: Separate NIghtlyTest Computation And Presentation

[bugzilla-daemon at cs.uiuc.edu]

http://llvm.cs.uiuc.edu/bugs/show_bug.cgi?id=379

           Summary: Separate NIghtlyTest Computation And Presentation
           Product: Test Suite
           Version: 1.2
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: Nightly Tester
        AssignedTo: unassignedbugs at nondot.org
        ReportedBy: rspencer at x10sys.com
                CC: tonic at nondot.org


Currently, the NightlyTest.pl script has numerous dependencies (qmtest, perl 5,
python 2.2, libpng, etc. etc.) which discourages developers from running the
tests. What we want to do is to have all testing done on any platform by anyone
reported (optionally) back to a server (zion? llvm.x10sys.com?) that presents
the information.  This reduces the complexity of the testing and would allow us
to alter the presentation of the nightly test data without affecting the
collection of the test data. This should have the following goals:

1. Make collection of test data have as few tool dependencies as possible. 
   Ideally you'd just need gmake and a compiler. Practically, you'll need some
   kind of test system (QMTest or dejagnu) to assist with the process.

2. After a successful run of the test suite, the results are packaged in a 
   common text-based database format (CSV or some such) and sent to a central
   server (probably via HTTP upload).

3. The server, when it receives a test results file, unpacks it, and adds it
   to a database for keeping track of the test results.  

4. The server's web system provides an interface for querying the test data by
   time, test, platform, and probably other criteria. Plots are dynamically
   generated based on the queries.  Additionally, "canned" daily reports are
   generated to keep track of the status of testing over time.

5. We probably want to use PHP for web site generation since it has good 
   database/web integration built in (Pear is now in PHP 5) and its as easy to
   write as Perl.

6. There should be some criteria/security for test data submission 
   because the data could become corrupted if we accepted submissions 
   from anywhere. As a backup measure, it should be possible to easily delete
   an entire submission from an administrative (password protected) page.

7. Before venturing off to build this, we might want to see what's already
   out there. I'm sure someone has at least attempted to crack this nut 
   before.

Ideas? Comments? Did I forget anything?



------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.