[lldb-dev] Digging into Linux unexpected successes

Tue Sep 15 09:25:38 PDT 2015

> The cmake builder runs in GCE and it uploads all test logs to Google
Cloud Storage (including full host logs and server logs). I used a python
script (running also in GCE) to download this data and to parse the test
output from the test traces.

Are the GCE logs public?  If not, do you know if our buildbot protocol
supports polling this info via another method straight from the build bot?
 (The latter is ultimately preferable so we can pull from multiple
builders, e.g. macosx, freebsd, etc.)  I suspect worst case the web
interface could be botted and the data collected and scraped, but hopefully
that isn't necessary.

Thanks again for sharing the info!

On Tue, Sep 15, 2015 at 9:16 AM, Tamas Berghammer <tberghammer at google.com>
wrote:

> Yes, you are reading it correctly (for totclang we mean the totclang at
> the time when the test suit was run).
>
> The cmake builder runs in GCE and it uploads all test logs to Google Cloud
> Storage (including full host logs and server logs). I used a python script
> (running also in GCE) to download this data and to parse the test output
> from the test traces.
>
> On Tue, Sep 15, 2015 at 5:08 PM Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> Just to make sure I'm reading these right:
>>
>> ========== Compiler: totclang Architecture: x86_64 ==========
>>
>> UnexpectedSuccess
>> TestMiInterpreterExec.MiInterpreterExecTestCase.test_lldbmi_settings_set_target_run_args_before
>> (250/250 100.000000%)
>> TestRaise.RaiseTestCase.test_restart_bug_with_dwarf (119/250 47.600000%)
>> TestMiSyntax.MiSyntaxTestCase.test_lldbmi_process_output (250/250
>> 100.000000%)
>> TestInferiorAssert.AssertingInferiorTestCase.test_inferior_asserting_expr_dwarf
>> (195/250 78.000000%)
>>
>>
>> This is saying that running the tests with a top of tree clang, on
>> x86_64, we see (for example):
>> * test_lldbmi_settings_set_target_run_args_before() is always passing,
>> * test_inferior_asserting_expr_dwarf() is always passing
>> * test_restart_bug_with_dwarf() is failing more often than passing.
>>
>> This is incredibly useful for figuring out the true disposition of a test
>> on different configurations.  What method did you use to gather that data?
>>
>> On Tue, Sep 15, 2015 at 9:03 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>>
>>> Wow Tamas, this is perfect.  Thanks for pulling that together!
>>>
>>> Don't worry about the bigger file.
>>>
>>> Thanks much.
>>>
>>> -Todd
>>>
>>> On Tue, Sep 15, 2015 at 8:56 AM, Tamas Berghammer <
>>> tberghammer at google.com> wrote:
>>>
>>>> I created a new statistic what separates the data based on compiler and
>>>> architecture and I also extended it to the last 250 builds on the Linux
>>>> build bot. If you would like to see the build IDs for the different
>>>> outcomes then let me know, because I have them collected out, but it is a
>>>> quite big file.
>>>>
>>>> Tamas
>>>>
>>>> On Tue, Sep 15, 2015 at 3:37 PM Todd Fiala <todd.fiala at gmail.com>
>>>> wrote:
>>>>
>>>>> On Tue, Sep 15, 2015 at 2:57 AM, Tamas Berghammer <
>>>>> tberghammer at google.com> wrote:
>>>>>
>>>>>> Hi Todd,
>>>>>>
>>>>>> I attached the statistic of the last 100 test run on the Linux x86_64
>>>>>> builder (
>>>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake).
>>>>>> The data might be a little bit noisy because of the actual test failures
>>>>>> happening because of a temporary regression, but they should give you a
>>>>>> general idea about what is happening.
>>>>>>
>>>>>>
>>>>> Thanks, Tamas!  I'll have a look.
>>>>>
>>>>>
>>>>>> I will try to create a statistic where the results are displayed
>>>>>> separately for each compiler and architecture to get a bit more detailed
>>>>>> view, but it will take some time. If you want I can include the list of
>>>>>> build numbers for all outcome, but it will be a very log list (currently
>>>>>> only included for Timeout and Failure)
>>>>>>
>>>>>>
>>>>> I'll know better when I have a look at what you provided.  The hole I
>>>>> see right now is we're not adequately dealing with unexpected successes for
>>>>> different configurations.  Any reporting around that is helpful.
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>>> Tamas
>>>>>>
>>>>>> On Mon, Sep 14, 2015 at 11:24 PM Todd Fiala via lldb-dev <
>>>>>> lldb-dev at lists.llvm.org> wrote:
>>>>>>
>>>>>>> On an Ubuntu 14.04 x86_64 system, I'm seeing the following results:
>>>>>>>
>>>>>>> *cmake/ninja/clang-3.6:*
>>>>>>>
>>>>>>> Testing: 395 test suites, 24 threads
>>>>>>> 395 out of 395 test suites processed - TestGdbRemoteKill.py
>>>>>>> Ran 395 test suites (0 failed) (0.000000%)
>>>>>>> Ran 478 test cases (0 failed) (0.000000%)
>>>>>>>
>>>>>>> Unexpected Successes (6)
>>>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestConstVariables.py
>>>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestEvents.py
>>>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiBreak.py
>>>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiGdbSetShow.py
>>>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiInterpreterExec.py
>>>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiSyntax.py
>>>>>>>
>>>>>>>
>>>>>>> *cmake/ninja/gcc-4.9.2:*
>>>>>>>
>>>>>>> 395 out of 395 test suites processed - TestMultithreaded.py
>>>>>>> Ran 395 test suites (1 failed) (0.253165%)
>>>>>>> Ran 457 test cases (1 failed) (0.218818%)
>>>>>>> Failing Tests (1)
>>>>>>> FAIL: LLDB (suite) :: TestRegisterVariables.py
>>>>>>>
>>>>>>> Unexpected Successes (6)
>>>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestDataFormatterSynth.py
>>>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiBreak.py
>>>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiGdbSetShow.py
>>>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiInterpreterExec.py
>>>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiSyntax.py
>>>>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestRaise.py
>>>>>>>
>>>>>>>
>>>>>>> I will look into those.  I suspect some of them are compiler-version
>>>>>>> specific, much like some of the OS X ones I dug into earlier.
>>>>>>> --
>>>>>>> -Todd
>>>>>>> _______________________________________________
>>>>>>> lldb-dev mailing list
>>>>>>> lldb-dev at lists.llvm.org
>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Todd
>>>>>
>>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>>
>>
>> --
>> -Todd
>>
>

-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150915/6d436ede/attachment-0001.html>