[lldb-dev] [llvm-dev] [lit] check-all hanging

Frédéric Riss via lldb-dev lldb-dev at lists.llvm.org
Thu Jan 3 08:30:00 PST 2019


-llvm-dev + lldb-dev for the lldv test failures.

> On Jan 3, 2019, at 7:33 AM, Joel E. Denny <jdenny.ornl at gmail.com> wrote:
> 
> All,
> 
> Thanks for the replies.  Kuba: For LLDB, when were things expected to have improved?  It's possible things improved for me at some point, but this isn't something I've found time to track carefully, and I still see problems. 
> 
> I ran check-all a couple of times last night at r350238, which I pulled yesterday.  Here are the results:
> 
> ```
> ********************
> Testing Time: 5043.24s
> ********************
> Unexpected Passing Tests (2):
>     lldb-Suite :: functionalities/asan/TestMemoryHistory.py
>     lldb-Suite :: functionalities/asan/TestReportData.py
> 
> ********************
> Failing Tests (54):
>     Clang :: CXX/modules-ts/basic/basic.link/p2/module.cpp
>     Clang :: Modules/ExtDebugInfo.cpp
>     Clang :: Modules/using-directive-redecl.cpp
>     Clang :: Modules/using-directive.cpp
>     Clang :: PCH/chain-late-anonymous-namespace.cpp
>     Clang :: PCH/cxx-namespaces.cpp
>     Clang :: PCH/namespaces.cpp
>     LLDB :: ExecControl/StopHook/stop-hook-threads.test
>     LeakSanitizer-AddressSanitizer-x86_64 :: TestCases/Linux/use_tls_dynamic.cc
>     LeakSanitizer-Standalone-x86_64 :: TestCases/Linux/use_tls_dynamic.cc
>     MemorySanitizer-X86_64 :: dtls_test.c
>     MemorySanitizer-lld-X86_64 :: dtls_test.c
>     lldb-Suite :: functionalities/register/register_command/TestRegisters.py
>     lldb-Suite :: tools/lldb-server/TestGdbRemoteRegisterState.py

It’s hard to diagnose dotest failures without the log. 

>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestBorrowedReferences
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestDictionaryResolutionWithDot
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestExtractingUInt64ThroughStructuredData
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestGlobalNameResolutionNoDot
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestGlobalNameResolutionWithDot
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestInstanceNameResolutionNoDot
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestModuleNameResolutionNoDot
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestObjectAttributes
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestOwnedReferences
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonByteArray
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonBytes
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonCallableCheck
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonCallableInvoke
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryManipulation
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryToStructuredDictionary
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryValueEquality
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonFile
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonInteger
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonIntegerToStr
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonIntegerToStructuredInteger
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListManipulation
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListToStructuredList
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListValueEquality
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonString
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonStringToStr
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonStringToStructuredString
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleInitializerList
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleInitializerList2
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleSize
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleToStructuredList
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleValues
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestResetting
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestTypeNameResolutionNoDot
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAcquisitionSemantics
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAutoRestoreChanged
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAutoRestoreSemantics
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestDiscardSemantics
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestExceptionStateChecking
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestManualRestoreSemantics
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestResetSemantics

Those unit test failures don’t ring a bell. Anyone’s seen this? Here too, knowing exactly the error message would help.

>   Expected Passes    : 57489
>   Expected Failures  : 276
>   Unsupported Tests  : 1883
>   Unexpected Passes  : 2
>   Unexpected Failures: 54
> 
> 14 warning(s) in tests.
> FAILED: CMakeFiles/check-all
> ```
> 
> I immediately ran it again and saw one new unexpected fail:
> 
> ```
>     lldb-Suite :: tools/lldb-mi/syntax/TestMiSyntax.py
> ```

Adrian, do we have remaining flakiness in the MI tests? Is this one of the GSoc tests?

> and one new unresolved test:
> 
> ```
>     lldb-Suite :: tools/lldb-vscode/breakpoint/TestVSCode_setBreakpoints.py
> ```
> 
> On the second run but not the first, it hung all night long waiting for TestVSCode_setBreakpoints.py to terminate.  I killed dotest.py to get the final results.

We have disabled the VScode tests on Darwin because they very flaky on the bots. +Greg who added those.

Fred

> I currently clone <https://github.com/llvm-project/llvm-project-20170507 <https://github.com/llvm-project/llvm-project-20170507>>.  I configure with `BUILD_SHARED_LIBS=true` and `-DLLVM_ENABLE_PROJECTS='clang;openmp;libcxx;libcxxabi;lldb;compiler-rt;lld;polly'`, among other options.  I have to run check-all with LD_LIBRARY_PATH pointing at my build's lib directory, or there are many more LLDB failures.  I believe that's not true for most test suites.  I'm building and testing under Ubuntu 18.04.1.
> 
> Hope that helps.  I'm happy to provide more details.  Just tell me where you'd like to start.
> 
> Thanks.
> 
> Joel
> 
> On Wed, Jan 2, 2019 at 5:51 PM Kuba Mracek <mracek at apple.com <mailto:mracek at apple.com>> wrote:
> +Fred, +me
> 
> For LLDB tests: I believe this got much much better recently. Are you still seeing flaky LLDB tests? Any details you can share?
> For sanitizer tests: I'm very much interesting in removing flakiness as well. Any specific tests you see as flaky?
> 
> Kuba
> 
>> On Jan 2, 2019, at 2:05 PM, Joel E. Denny via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> 
>> Hi David, Chandler,
>> 
>> I see lldb tests hang often, and then I kill the dotest process.
>> 
>> I'd like to stop running check-all too, but I feel it's important when I modify FileCheck.  The flakiness that Chandler mentioned makes it time-consuming to verify test results.
>> 
>> Joel
>> 
>> On Wed, Jan 2, 2019 at 4:41 PM Chandler Carruth via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> What you're seeing is just the fact that lit is waiting on subprocesses (select is waiting on the pipes i suspect).
>> 
>> Anyways, you'll need to dig into what it is waiting on, and what *that* process is doing that is stuck to make progress.
>> 
>> I've not seen anything like this, but I basically never run `check-all` these days because LLDB and sanitizer tests are too flaky. =[ I've not been able to interest anyone in fixing this either sadly.
>> 
>> On Wed, Jan 2, 2019 at 10:09 AM David Greene via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> Hi,
>> 
>> From time to time, I see check-all hang during running of lit tests.
>> The hang always happens at the > 90% completion stage and I'm pretty
>> sure all tests have been run and check-all is just waiting for
>> lit/python to exit.  I see a single python processing running, taking
>> very little CPU time.  An strace of that process shows this:
>> 
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 32168}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 1000})  = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 2000})  = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 4000})  = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 8000})  = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 16000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 32000}) = 0 (Timeout)
>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, ffffffff) = 0
>> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, ffffffff) = -1 EAGAIN (Resourc
>> e temporarily unavailable)                                                                             
>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, ffffffff) = -1 EAGAIN (Resourc
>> e temporarily unavailable)                                                                             
>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> 
>> It appears that python is waiting for some I/O or something which never
>> appears.
>> 
>> Has anyone else seen this before?  Any ideas of what is going on or how
>> to fix it?
>> 
>>                               -David
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20190103/cb29cdcf/attachment-0001.html>


More information about the lldb-dev mailing list