[Release-testers] [llvm-dev] [3.8 Release] RC3 has been tagged

[Elias Pipping via Release-testers]

> On 23 Feb 2016, at 22:51, Hans Wennborg via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> Dear testers,
> 
> Release Candidate 3 has just been tagged [1]. Please build, test, and
> upload to the sftp.
> 
> If there are no regressions from previous release candidates, this
> will be the last release candidate before the final release.
> 
> Release notes can still go into the branch.
> 
> Thanks again for all your work!
> Hans

Hi,

I’ve run into some unexpected problems with the rc3 tarballs for OSX and would like to ask for advice.

I’ve repeatedly started from scratch (that’s what’s been taking so long) only to always end up at the same point: When Phase3 is built and the tests are run, I get a hang. Immediately, before a single test is run. Every time. There’s a few lines of the type

lit.py: util.py:254: note: using SDKROOT: '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk'
lit.py: lit.cfg:195: note: using clang: '/Users/pipping/llvm-testing/rc3/Phase3/Release/llvmCore-3.8.0-rc3.obj/./bin/clang'
lit.py: util.py:254: note: using SDKROOT: '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk'

and then silence. I can also manually reproduce this by `cd`ing to

  rc3/Phase3/Release/llvmCore-3.8.0-rc3.obj/

and running `make check-all` from there. The line that I then get (that I otherwise don’t see) is

                          -- Testing: 32833 tests, 4 threads --                           
  0% [-----------------------------------------------------------------------------------]

beyond which there is no progress. A Ctrl-C tells me

File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)

here. The actual command that is run can be obtained via

  % make -n check-all | grep -w lit

It is

  % /usr/bin/python2.7 /Users/pipping/llvm-testing/rc3/llvm.src/utils/lit/lit.py -sv

followed by a few config files and then large amounts of test directories. If I put them into corresponding arrays, that’s

  % /usr/bin/python2.7 /Users/pipping/llvm-testing/rc3/llvm.src/utils/lit/lit.py -sv ${args[@]} ${tests[@]}

Now, it’s clear that not a single test is to blame: There are 28 test directories (the last one being by far the largest, containing about half of all tests) and both

  % /usr/bin/python2.7 /Users/pipping/llvm-testing/rc3/llvm.src/utils/lit/lit.py -sv ${args[@]} ${tests[1,27]} # 17231 tests
  % /usr/bin/python2.7 /Users/pipping/llvm-testing/rc3/llvm.src/utils/lit/lit.py -sv ${args[@]} ${tests[28]} # 15602 tests

immediately start to run tests. So it seems to be related to the number of tests or the number of directories. Moreover, it’s related to parallelisation: With

  % /usr/bin/python2.7 /Users/pipping/llvm-testing/rc3/llvm.src/utils/lit/lit.py -j1 -sv ${args[@]} ${tests[@]}
 
the tests do run but with

  % /usr/bin/python2.7 /Users/pipping/llvm-testing/rc3/llvm.src/utils/lit/lit.py -j2 -sv ${args[@]} ${tests[@]}

they don’t. Finally, I tried this, suspecting that the command might simply be too long

  % /usr/bin/python2.7 /Users/pipping/llvm-testing/rc3/llvm.src/utils/lit/lit.py -sv ${args[@]} \
    $(echo ${tests[@]} | sed s:/Users/pipping/llvm-testing/rc3/Phase3/Release/llvmCore-3.8.0-rc3.obj/:./:g)

but unfortunately that didn’t help.

Now the part that makes me a bit uncomfortable is that I tried again with rc2, which previously worked, but no longer does. I threw my /usr/local out of the window and looked for potential sources. Maybe it’s a system update that I’ve since had. I really have no idea what’s causing this and wonder if anyone has seen something similar.


Elias