[LLVMbugs] [Bug 1319] NEW: Improve failure detection in llvm/test

[bugzilla-daemon at cs.uiuc.edu]

http://llvm.org/bugs/show_bug.cgi?id=1319

           Summary: Improve failure detection in llvm/test
           Product: Build scripts
           Version: 1.6
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: Makefiles
        AssignedTo: unassignedbugs at nondot.org
        ReportedBy: rspencer at x10sys.com


Currently we are doubling up our test scripts with things like:

; RUN: llvm-as < %s | opt -loop-rotate -disable-output &&
; RUN: llvm-as < %s | opt -loop-rotate | llvm-dis | not grep "\[ .tmp224" 

in order to ensure that any failures early in the pipeline are not causing false
positives simply because no output was generated. There are two issues here:

1. Correctness. If we had a mechanism that checked each member of a pipeline
   for 0 return code, that would be preferable as the test case writer doesn't
   have to think about that sort of thing happening. By having a mechanism that
   does this automatically, additional test cases, not covered by the above
   duplication would be made more robust. 

2. Speed. This solution to the problem is coming at the expense of invoking the
   same pipelines multiple times. While this isn't an issue now while the test
   suite runs quickly, it will in due time cause issues. The test suite has 
   grown by over 1000 tests in the last year or so. Its likely to accellerate
   that growth rate for some time to come. 

This isn't a pressing need, just filing this so we don't forget about it.
Reid.



------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.