[lldb-dev] Digging into Linux unexpected successes

[Todd Fiala via lldb-dev]

Wow Tamas, this is perfect.  Thanks for pulling that together!

Don't worry about the bigger file.

Thanks much.

-Todd

On Tue, Sep 15, 2015 at 8:56 AM, Tamas Berghammer <tberghammer at google.com>
wrote:

> I created a new statistic what separates the data based on compiler and
> architecture and I also extended it to the last 250 builds on the Linux
> build bot. If you would like to see the build IDs for the different
> outcomes then let me know, because I have them collected out, but it is a
> quite big file.
>
> Tamas
>
> On Tue, Sep 15, 2015 at 3:37 PM Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> On Tue, Sep 15, 2015 at 2:57 AM, Tamas Berghammer <tberghammer at google.com
>> > wrote:
>>
>>> Hi Todd,
>>>
>>> I attached the statistic of the last 100 test run on the Linux x86_64
>>> builder (
>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake). The
>>> data might be a little bit noisy because of the actual test failures
>>> happening because of a temporary regression, but they should give you a
>>> general idea about what is happening.
>>>
>>>
>> Thanks, Tamas!  I'll have a look.
>>
>>
>>> I will try to create a statistic where the results are displayed
>>> separately for each compiler and architecture to get a bit more detailed
>>> view, but it will take some time. If you want I can include the list of
>>> build numbers for all outcome, but it will be a very log list (currently
>>> only included for Timeout and Failure)
>>>
>>>
>> I'll know better when I have a look at what you provided.  The hole I see
>> right now is we're not adequately dealing with unexpected successes for
>> different configurations.  Any reporting around that is helpful.
>>
>> Thanks!
>>
>>
>>> Tamas
>>>
>>> On Mon, Sep 14, 2015 at 11:24 PM Todd Fiala via lldb-dev <
>>> lldb-dev at lists.llvm.org> wrote:
>>>
>>>> On an Ubuntu 14.04 x86_64 system, I'm seeing the following results:
>>>>
>>>> *cmake/ninja/clang-3.6:*
>>>>
>>>> Testing: 395 test suites, 24 threads
>>>> 395 out of 395 test suites processed - TestGdbRemoteKill.py
>>>> Ran 395 test suites (0 failed) (0.000000%)
>>>> Ran 478 test cases (0 failed) (0.000000%)
>>>>
>>>> Unexpected Successes (6)
>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestConstVariables.py
>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestEvents.py
>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiBreak.py
>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiGdbSetShow.py
>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiInterpreterExec.py
>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiSyntax.py
>>>>
>>>>
>>>> *cmake/ninja/gcc-4.9.2:*
>>>>
>>>> 395 out of 395 test suites processed - TestMultithreaded.py
>>>> Ran 395 test suites (1 failed) (0.253165%)
>>>> Ran 457 test cases (1 failed) (0.218818%)
>>>> Failing Tests (1)
>>>> FAIL: LLDB (suite) :: TestRegisterVariables.py
>>>>
>>>> Unexpected Successes (6)
>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestDataFormatterSynth.py
>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiBreak.py
>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiGdbSetShow.py
>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiInterpreterExec.py
>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiSyntax.py
>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestRaise.py
>>>>
>>>>
>>>> I will look into those.  I suspect some of them are compiler-version
>>>> specific, much like some of the OS X ones I dug into earlier.
>>>> --
>>>> -Todd
>>>> _______________________________________________
>>>> lldb-dev mailing list
>>>> lldb-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>
>>>
>>
>>
>> --
>> -Todd
>>
>


-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150915/dd4ab63d/attachment.html>