[lldb-dev] Digging into Linux unexpected successes

Tue Sep 15 09:08:04 PDT 2015

Just to make sure I'm reading these right:

========== Compiler: totclang Architecture: x86_64 ==========

UnexpectedSuccess
TestMiInterpreterExec.MiInterpreterExecTestCase.test_lldbmi_settings_set_target_run_args_before
(250/250 100.000000%)
TestRaise.RaiseTestCase.test_restart_bug_with_dwarf (119/250 47.600000%)
TestMiSyntax.MiSyntaxTestCase.test_lldbmi_process_output (250/250
100.000000%)
TestInferiorAssert.AssertingInferiorTestCase.test_inferior_asserting_expr_dwarf
(195/250 78.000000%)

This is saying that running the tests with a top of tree clang, on x86_64,
we see (for example):
* test_lldbmi_settings_set_target_run_args_before() is always passing,
* test_inferior_asserting_expr_dwarf() is always passing
* test_restart_bug_with_dwarf() is failing more often than passing.

This is incredibly useful for figuring out the true disposition of a test
on different configurations.  What method did you use to gather that data?

On Tue, Sep 15, 2015 at 9:03 AM, Todd Fiala <todd.fiala at gmail.com> wrote:

> Wow Tamas, this is perfect.  Thanks for pulling that together!
>
> Don't worry about the bigger file.
>
> Thanks much.
>
> -Todd
>
> On Tue, Sep 15, 2015 at 8:56 AM, Tamas Berghammer <tberghammer at google.com>
> wrote:
>
>> I created a new statistic what separates the data based on compiler and
>> architecture and I also extended it to the last 250 builds on the Linux
>> build bot. If you would like to see the build IDs for the different
>> outcomes then let me know, because I have them collected out, but it is a
>> quite big file.
>>
>> Tamas
>>
>> On Tue, Sep 15, 2015 at 3:37 PM Todd Fiala <todd.fiala at gmail.com> wrote:
>>
>>> On Tue, Sep 15, 2015 at 2:57 AM, Tamas Berghammer <
>>> tberghammer at google.com> wrote:
>>>
>>>> Hi Todd,
>>>>
>>>> I attached the statistic of the last 100 test run on the Linux x86_64
>>>> builder (
>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake). The
>>>> data might be a little bit noisy because of the actual test failures
>>>> happening because of a temporary regression, but they should give you a
>>>> general idea about what is happening.
>>>>
>>>>
>>> Thanks, Tamas!  I'll have a look.
>>>
>>>
>>>> I will try to create a statistic where the results are displayed
>>>> separately for each compiler and architecture to get a bit more detailed
>>>> view, but it will take some time. If you want I can include the list of
>>>> build numbers for all outcome, but it will be a very log list (currently
>>>> only included for Timeout and Failure)
>>>>
>>>>
>>> I'll know better when I have a look at what you provided.  The hole I
>>> see right now is we're not adequately dealing with unexpected successes for
>>> different configurations.  Any reporting around that is helpful.
>>>
>>> Thanks!
>>>
>>>
>>>> Tamas
>>>>
>>>> On Mon, Sep 14, 2015 at 11:24 PM Todd Fiala via lldb-dev <
>>>> lldb-dev at lists.llvm.org> wrote:
>>>>
>>>>> On an Ubuntu 14.04 x86_64 system, I'm seeing the following results:
>>>>>
>>>>> *cmake/ninja/clang-3.6:*
>>>>>
>>>>> Testing: 395 test suites, 24 threads
>>>>> 395 out of 395 test suites processed - TestGdbRemoteKill.py
>>>>> Ran 395 test suites (0 failed) (0.000000%)
>>>>> Ran 478 test cases (0 failed) (0.000000%)
>>>>>
>>>>> Unexpected Successes (6)
>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestConstVariables.py
>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestEvents.py
>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiBreak.py
>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiGdbSetShow.py
>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiInterpreterExec.py
>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiSyntax.py
>>>>>
>>>>>
>>>>> *cmake/ninja/gcc-4.9.2:*
>>>>>
>>>>> 395 out of 395 test suites processed - TestMultithreaded.py
>>>>> Ran 395 test suites (1 failed) (0.253165%)
>>>>> Ran 457 test cases (1 failed) (0.218818%)
>>>>> Failing Tests (1)
>>>>> FAIL: LLDB (suite) :: TestRegisterVariables.py
>>>>>
>>>>> Unexpected Successes (6)
>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestDataFormatterSynth.py
>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiBreak.py
>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiGdbSetShow.py
>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiInterpreterExec.py
>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiSyntax.py
>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestRaise.py
>>>>>
>>>>>
>>>>> I will look into those.  I suspect some of them are compiler-version
>>>>> specific, much like some of the OS X ones I dug into earlier.
>>>>> --
>>>>> -Todd
>>>>> _______________________________________________
>>>>> lldb-dev mailing list
>>>>> lldb-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>
>>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>
>
> --
> -Todd
>

-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150915/5eaeb2ba/attachment-0001.html>