[llvm-dev] [lit] check-all hanging

Joel E. Denny via llvm-dev llvm-dev at lists.llvm.org
Thu Jan 3 07:33:04 PST 2019 

All,

Thanks for the replies.  Kuba: For LLDB, when were things expected to have
improved?  It's possible things improved for me at some point, but this
isn't something I've found time to track carefully, and I still see
problems.

I ran check-all a couple of times last night at r350238, which I pulled
yesterday.  Here are the results:

```
********************
Testing Time: 5043.24s
********************
Unexpected Passing Tests (2):
    lldb-Suite :: functionalities/asan/TestMemoryHistory.py
    lldb-Suite :: functionalities/asan/TestReportData.py

********************
Failing Tests (54):
    Clang :: CXX/modules-ts/basic/basic.link/p2/module.cpp
    Clang :: Modules/ExtDebugInfo.cpp
    Clang :: Modules/using-directive-redecl.cpp
    Clang :: Modules/using-directive.cpp
    Clang :: PCH/chain-late-anonymous-namespace.cpp
    Clang :: PCH/cxx-namespaces.cpp
    Clang :: PCH/namespaces.cpp
    LLDB :: ExecControl/StopHook/stop-hook-threads.test
    LeakSanitizer-AddressSanitizer-x86_64 ::
TestCases/Linux/use_tls_dynamic.cc
    LeakSanitizer-Standalone-x86_64 :: TestCases/Linux/use_tls_dynamic.cc
    MemorySanitizer-X86_64 :: dtls_test.c
    MemorySanitizer-lld-X86_64 :: dtls_test.c
    lldb-Suite :: functionalities/register/register_command/TestRegisters.py
    lldb-Suite :: tools/lldb-server/TestGdbRemoteRegisterState.py
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestBorrowedReferences
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestDictionaryResolutionWithDot
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestExtractingUInt64ThroughStructuredData
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestGlobalNameResolutionNoDot
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestGlobalNameResolutionWithDot
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestInstanceNameResolutionNoDot
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestModuleNameResolutionNoDot
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestObjectAttributes
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestOwnedReferences
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonByteArray
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonBytes
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonCallableCheck
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonCallableInvoke
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryManipulation
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryToStructuredDictionary
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryValueEquality
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonFile
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonInteger
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonIntegerToStr
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonIntegerToStructuredInteger
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListManipulation
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListToStructuredList
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListValueEquality
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonString
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonStringToStr
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonStringToStructuredString
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleInitializerList
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleInitializerList2
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleSize
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleToStructuredList
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleValues
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestResetting
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestTypeNameResolutionNoDot
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAcquisitionSemantics
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAutoRestoreChanged
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAutoRestoreSemantics
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestDiscardSemantics
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestExceptionStateChecking
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestManualRestoreSemantics
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestResetSemantics

  Expected Passes    : 57489
  Expected Failures  : 276
  Unsupported Tests  : 1883
  Unexpected Passes  : 2
  Unexpected Failures: 54

14 warning(s) in tests.
FAILED: CMakeFiles/check-all
```

I immediately ran it again and saw one new unexpected fail:

```
    lldb-Suite :: tools/lldb-mi/syntax/TestMiSyntax.py
```

and one new unresolved test:

```
    lldb-Suite :: tools/lldb-vscode/breakpoint/TestVSCode_setBreakpoints.py
```

On the second run but not the first, it hung all night long waiting for
TestVSCode_setBreakpoints.py to terminate.  I killed dotest.py to get the
final results.

I currently clone <https://github.com/llvm-project/llvm-project-20170507>.
I configure with `BUILD_SHARED_LIBS=true` and
`-DLLVM_ENABLE_PROJECTS='clang;openmp;libcxx;libcxxabi;lldb;compiler-rt;lld;polly'`,
among other options.  I have to run check-all with LD_LIBRARY_PATH pointing
at my build's lib directory, or there are many more LLDB failures.  I
believe that's not true for most test suites.  I'm building and testing
under Ubuntu 18.04.1.

Hope that helps.  I'm happy to provide more details.  Just tell me where
you'd like to start.

Thanks.

Joel

On Wed, Jan 2, 2019 at 5:51 PM Kuba Mracek <mracek at apple.com> wrote:

> +Fred, +me
>
> For LLDB tests: I believe this got much much better recently. Are you
> still seeing flaky LLDB tests? Any details you can share?
> For sanitizer tests: I'm very much interesting in removing flakiness as
> well. Any specific tests you see as flaky?
>
> Kuba
>
> On Jan 2, 2019, at 2:05 PM, Joel E. Denny via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Hi David, Chandler,
>
> I see lldb tests hang often, and then I kill the dotest process.
>
> I'd like to stop running check-all too, but I feel it's important when I
> modify FileCheck.  The flakiness that Chandler mentioned makes it
> time-consuming to verify test results.
>
> Joel
>
> On Wed, Jan 2, 2019 at 4:41 PM Chandler Carruth via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> What you're seeing is just the fact that lit is waiting on subprocesses
>> (select is waiting on the pipes i suspect).
>>
>> Anyways, you'll need to dig into what it is waiting on, and what *that*
>> process is doing that is stuck to make progress.
>>
>> I've not seen anything like this, but I basically never run `check-all`
>> these days because LLDB and sanitizer tests are too flaky. =[ I've not been
>> able to interest anyone in fixing this either sadly.
>>
>> On Wed, Jan 2, 2019 at 10:09 AM David Greene via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Hi,
>>>
>>> From time to time, I see check-all hang during running of lit tests.
>>> The hang always happens at the > 90% completion stage and I'm pretty
>>> sure all tests have been run and check-all is just waiting for
>>> lit/python to exit.  I see a single python processing running, taking
>>> very little CPU time.  An strace of that process shows this:
>>>
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 32168}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 1000})  = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 2000})  = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 4000})  = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 8000})  = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 16000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 32000}) = 0 (Timeout)
>>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>>> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0,
>>> NULL, ffffffff) = 0
>>> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0,
>>> NULL, ffffffff) = -1 EAGAIN (Resourc
>>> e temporarily unavailable)
>>>
>>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0,
>>> NULL, ffffffff) = -1 EAGAIN (Resourc
>>> e temporarily unavailable)
>>>
>>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>>
>>> It appears that python is waiting for some I/O or something which never
>>> appears.
>>>
>>> Has anyone else seen this before?  Any ideas of what is going on or how
>>> to fix it?
>>>
>>>                               -David
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190103/e5c5e7d8/attachment.html>

More information about the llvm-dev mailing list