[Release-testers] [llvm-dev] [3.8 Release] RC3 has been tagged

[Hans Wennborg via Release-testers]

On Thu, Feb 25, 2016 at 2:00 PM, Elias Pipping <pipping.elias at gmail.com> wrote:
>
>> On 23 Feb 2016, at 22:51, Hans Wennborg via llvm-dev <llvm-dev at lists.llvm.org> wrote:
>>
>> Dear testers,
>>
>> Release Candidate 3 has just been tagged [1]. Please build, test, and
>> upload to the sftp.
>>
>> If there are no regressions from previous release candidates, this
>> will be the last release candidate before the final release.
>>
>> Release notes can still go into the branch.
>>
>> Thanks again for all your work!
>> Hans
>
> Hi,
>
> I’ve run into some unexpected problems with the rc3 tarballs for OSX and would like to ask for advice.
>
> I’ve repeatedly started from scratch (that’s what’s been taking so long) only to always end up at the same point: When Phase3 is built and the tests are run, I get a hang. Immediately, before a single test is run. Every time. There’s a few lines of the type
>
> lit.py: util.py:254: note: using SDKROOT: '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk'
> lit.py: lit.cfg:195: note: using clang: '/Users/pipping/llvm-testing/rc3/Phase3/Release/llvmCore-3.8.0-rc3.obj/./bin/clang'
> lit.py: util.py:254: note: using SDKROOT: '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk'
>
> and then silence. I can also manually reproduce this by `cd`ing to
>
>   rc3/Phase3/Release/llvmCore-3.8.0-rc3.obj/
>
> and running `make check-all` from there. The line that I then get (that I otherwise don’t see) is
>
>                           -- Testing: 32833 tests, 4 threads --
>   0% [-----------------------------------------------------------------------------------]
>
> beyond which there is no progress. A Ctrl-C tells me
>
> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
>     send(obj)
>
> here. The actual command that is run can be obtained via
>
>   % make -n check-all | grep -w lit
>
> It is
>
>   % /usr/bin/python2.7 /Users/pipping/llvm-testing/rc3/llvm.src/utils/lit/lit.py -sv
>
> followed by a few config files and then large amounts of test directories. If I put them into corresponding arrays, that’s
>
>   % /usr/bin/python2.7 /Users/pipping/llvm-testing/rc3/llvm.src/utils/lit/lit.py -sv ${args[@]} ${tests[@]}
>
> Now, it’s clear that not a single test is to blame: There are 28 test directories (the last one being by far the largest, containing about half of all tests) and both
>
>   % /usr/bin/python2.7 /Users/pipping/llvm-testing/rc3/llvm.src/utils/lit/lit.py -sv ${args[@]} ${tests[1,27]} # 17231 tests
>   % /usr/bin/python2.7 /Users/pipping/llvm-testing/rc3/llvm.src/utils/lit/lit.py -sv ${args[@]} ${tests[28]} # 15602 tests
>
> immediately start to run tests. So it seems to be related to the number of tests or the number of directories. Moreover, it’s related to parallelisation: With
>
>   % /usr/bin/python2.7 /Users/pipping/llvm-testing/rc3/llvm.src/utils/lit/lit.py -j1 -sv ${args[@]} ${tests[@]}
>
> the tests do run but with
>
>   % /usr/bin/python2.7 /Users/pipping/llvm-testing/rc3/llvm.src/utils/lit/lit.py -j2 -sv ${args[@]} ${tests[@]}
>
> they don’t. Finally, I tried this, suspecting that the command might simply be too long
>
>   % /usr/bin/python2.7 /Users/pipping/llvm-testing/rc3/llvm.src/utils/lit/lit.py -sv ${args[@]} \
>     $(echo ${tests[@]} | sed s:/Users/pipping/llvm-testing/rc3/Phase3/Release/llvmCore-3.8.0-rc3.obj/:./:g)
>
> but unfortunately that didn’t help.
>
> Now the part that makes me a bit uncomfortable is that I tried again with rc2, which previously worked, but no longer does. I threw my /usr/local out of the window and looked for potential sources. Maybe it’s a system update that I’ve since had. I really have no idea what’s causing this and wonder if anyone has seen something similar.

It really does sound like a system problem rather than a problem with
any specific tests. It sounds like when trying to run in parallel, the
test framework hangs. Are you still on a beta version of OS X?

FWIV, the tests pass fine on my Mac with -j32.