[lldb-dev] [Bug 25562] New: Some failing tests are being reported as passes

[via lldb-dev]

https://llvm.org/bugs/show_bug.cgi?id=25562

            Bug ID: 25562
           Summary: Some failing tests are being reported as passes
           Product: lldb
           Version: unspecified
          Hardware: PC
                OS: Windows NT
            Status: NEW
          Severity: normal
          Priority: P
         Component: All Bugs
          Assignee: lldb-dev at lists.llvm.org
          Reporter: zturner at google.com
                CC: llvm-bugs at lists.llvm.org
    Classification: Unclassified

If I run this test manually, two of the tests fails.  Example:

d:\src\llvmbuild\ninja\lldb-test-traces>c:\python27_lldb\x86\python_d.exe
D:\src\llvm\tools\lldb\test\dotest.py -q --arch=i686 --executable
D:/src/llvmbuild/ninja/bin/lldb.exe -s D:/src/llvmbuild/ninja/lldb-test-traces
-u CXXFLAGS -u CFLAGS --enable-crash-dialog -C
d:\src\llvmbuild\ninja_release\bin\clang.exe
D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\functionalities\process_launch
--no-multiprocess
The 'lldb-mi' executable cannot be located.  The lldb-mi tests can not be run
as a result.
UNSUPPORTED: LLDB (d:\src\llvmbuild\ninja_release\bin\clang.exe-i686) ::
test_environment_with_special_char_dsym
(TestProcessLaunch.ProcessLaunchTestCase) (dsym tests)
FAIL: LLDB (d:\src\llvmbuild\ninja_release\bin\clang.exe-i686) ::
test_environment_with_special_char_dwarf
(TestProcessLaunch.ProcessLaunchTestCase)
FAIL: LLDB (d:\src\llvmbuild\ninja_release\bin\clang.exe-i686) ::
test_environment_with_special_char_dwo
(TestProcessLaunch.ProcessLaunchTestCase)
UNSUPPORTED: LLDB (d:\src\llvmbuild\ninja_release\bin\clang.exe-i686) ::
test_io_dsym (TestProcessLaunch.ProcessLaunchTestCase) (dsym tests)
PASS: LLDB (d:\src\llvmbuild\ninja_release\bin\clang.exe-i686) :: test_io_dwarf
(TestProcessLaunch.ProcessLaunchTestCase)
PASS: LLDB (d:\src\llvmbuild\ninja_release\bin\clang.exe-i686) :: test_io_dwo
(TestProcessLaunch.ProcessLaunchTestCase)
UNSUPPORTED: LLDB (d:\src\llvmbuild\ninja_release\bin\clang.exe-i686) ::
test_set_working_dir_dsym (TestProcessLaunch.ProcessLaunchTestCase) (dsym
tests)

When I run the multiprocess test runner via dosep, it's being reported as a
success.

410 out of 410 test suites processed - TestAddDsymCommand.py
Ran 410 test suites (1 failed) (0.243902%)
Ran 318 test cases (4 failed) (1.257862%)
Failing Tests (1)
FAIL: LLDB (suite) :: TestBreakpointLanguage.py (Windows zturner-win81 8
6.2.9200 AMD64 Intel64 Family 6 Model 45 Stepping 7, GenuineIntel)
[79964 refs]
ninja: build stopped: subcommand failed.

Stepping through the code, in dosep.py when it goes to print the summary list
of failed tests, 'TestProcessLaunch.py' is in the list of passed tests.  So
something is wrong here, and a potentially unknown number of tests are
currently being misreported.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151118/cf216ca3/attachment.html>