<div dir="ltr">On OS X, I'm also not seeing growth in the --test-runner-name threading-pool (the one you were using on Windows).<div><br></div><div>Perhaps you can dig into if you're experiencing some kind of file leak on Windows. It's possible you're hitting a platform-specific leak? I recall Ed Maste hitting a FreeBSD-only leak in one or more of the python 2.7.x releases.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 5, 2015 at 11:26 AM, Todd Fiala <span dir="ltr"><<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hmm, on OS X the file handles seem to be well behaved on the --test-runner-name threading. I'm not seeing any file handle growth beyond the file handles I expect to be open.<div><br></div><div>I'll see if the threading-pool behaves differently. (That is similar to threading but uses the multiprocessing.pool mechanism, at the expense of me not being able to catch Ctrl-C at all).</div><div><br></div><div>It's possible the pool is introducing some leakage at the file level.</div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Mon, Oct 5, 2015 at 11:20 AM, Todd Fiala <span dir="ltr"><<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Interesting, okay..<div><br></div><div>This does appear to be an accumulation issue. You made it most of the way through before the issue hit. I suspect we're leaking file handles. It probably doesn't hit the per-process limit on multiprocessing because the leaked files get spread across more processes.</div><div><br></div><div>(All speculation but does fit the results).</div><div><br></div><div>I'll see if I can look into what's there - if we've got an obvious leak, I'll take care of it.</div></div><div class="gmail_extra"><div><div><br><div class="gmail_quote">On Mon, Oct 5, 2015 at 9:58 AM, Adrian McCarthy <span dir="ltr"><<a href="mailto:amccarth@google.com" target="_blank">amccarth@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks for the ideas.<div><br></div><div><span style="font-size:16px">With `--test-runner-name threading-pool`, I get too many open files.</span><br></div><div><span style="font-size:16px"><br></span></div><div><span style="font-size:16px">With `</span><span style="font-size:16px">--test-runner-name multiprocessing-pool`, the suite runs fine.</span></div><div><span style="font-size:16px"><br></span></div><div><span style="font-size:16px">My machine has 40 logical cores.</span></div><div><span style="font-size:16px"><br></span></div><div><span style="font-size:16px">With `--threads=20`: SUCCESS (and perhaps _faster_).</span></div><div><span style="font-size:16px"><br></span></div><div><span style="font-size:16px">With </span><span style="font-size:16px">`--threads=30`: SUCCESS.</span></div><div><span style="font-size:16px"><br></span></div><div><span style="font-size:16px">With </span><span style="font-size:16px">`--threads=36`: SUCCESS.</span><span style="font-size:16px"><br></span></div><div><span style="font-size:16px"><br></span></div><div><span style="font-size:16px">With </span><span style="font-size:16px">`--threads=38`: TOO MANY OPEN FILES.</span><span style="font-size:16px"><br></span></div><div><span style="font-size:16px"><br></span></div><div><span style="font-size:16px">So we're right at the edge. I'll keep investigating.</span></div><div><br></div><div>So it seems we're on the bleeding edge.</div><div><br></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Oct 2, 2015 at 5:38 PM, Todd Fiala <span dir="ltr"><<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">(swapped out the lldb list for the newer one)<div class="gmail_extra"><div><div><br><div class="gmail_quote">On Fri, Oct 2, 2015 at 5:37 PM, Todd Fiala <span dir="ltr"><<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hmm, sounds suspicious.<div><br></div><div>Can you try running the tests with two options and see if you get different results?</div><div><br></div><div># should be equivalent for the default on Windows, thus should match your above results. This one uses a thread per worker queue.</div><div>--test-runner-name threading-pool</div><div><br></div><div># should use a different test runner. This one uses a process per worker queue.</div><div>--test-runner-name multiprocessing-pool</div><div><br></div><div>Aside from that, it seems like the total number of open files is exceeding some process/system maximum, which sounds like (maybe) we're leaking files somewhere. Not enough info yet to guess where that might be coming in from, but maybe a part of the test runner isn't closing files somewhere.</div><div><br></div><div>The other thing you can try is reducing the total number of threads, with:</div><div>--threads {some-number-lower-than-your-total-number-of-logical-cores}</div><div><br></div><div>in the event that your machine has a mongo number of logical cores, and perhaps it is trying to do too much. (In that case, the multiprocessing-pool runner might actually help).</div><div><br></div><div>Thanks!</div><div><br></div><div>-Todd</div></div><div class="gmail_extra"><div><div><br><div class="gmail_quote">On Fri, Oct 2, 2015 at 5:31 PM, Adrian McCarthy <span dir="ltr"><<a href="mailto:amccarth@google.com" target="_blank">amccarth@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">When running LLDB tests on Windows, I started getting a "too many open files" error from Python. I used git bisect to narrow it down to this revision:<div><br></div><div><a href="http://llvm.org/viewvc/llvm-project?view=revision&revision=249182" target="_blank">http://llvm.org/viewvc/llvm-project?view=revision&revision=249182</a><br></div><div><br></div><div>The error output is:</div><div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><font face="monospace, monospace" size="1">Command invoked: D:\src\Python-2.7.9\PCbuild\python_d.exe D:\src\llvm\llvm\tools\lldb\test\dotest.py -q --arch=i686 --executable D:/src/llvm/build/ninja/bin/lldb.exe -s D:/src/llvm/build/ninja/lldb-test-traces -u CXXFLAGS -u CFLAGS --enable-crash-dialog -C D:\src\llvm\build\ninja_release\bin\clang.exe --inferior -p TestRecursiveTypes.py D:\src\llvm\llvm\tools\lldb\test --event-add-entries worker_index=7:int</font></div></div><div><div><font face="monospace, monospace" size="1">384 out of 400 test suites processed - TestRecursiveTypes.py Traceback (most recent call last):</font></div></div><div><div><font face="monospace, monospace" size="1"> File "D:/src/llvm/llvm/tools/lldb/test/dotest.py", line 1457, in <module></font></div></div><div><div><font face="monospace, monospace" size="1"> File "D:\src\llvm\llvm\tools\lldb\test\dosep.py", line 1355, in main</font></div></div><div><div><font face="monospace, monospace" size="1"> File "D:\src\llvm\llvm\tools\lldb\test\dosep.py", line 968, in walk_and_invoke</font></div></div><div><div><font face="monospace, monospace" size="1"> File "D:\src\llvm\llvm\tools\lldb\test\dosep.py", line 1095, in <lambda></font></div></div><div><div><font face="monospace, monospace" size="1"> File "D:\src\llvm\llvm\tools\lldb\test\dosep.py", line 889, in threading_test_runner_pool</font></div></div><div><div><font face="monospace, monospace" size="1"> File "D:\src\llvm\llvm\tools\lldb\test\dosep.py", line 774, in map_async_run_loop</font></div></div><div><div><font face="monospace, monospace" size="1"> File "D:\src\Python-2.7.9\Lib\multiprocessing\pool.py", line 558, in get</font></div></div><div><div><font face="monospace, monospace" size="1">OSError: [Errno 24] Too many open files</font></div></div><div><div><font face="monospace, monospace" size="1">[77809 refs]</font></div></div><div><div><font face="monospace, monospace" size="1">ninja: build stopped: subcommand failed.</font></div></div></blockquote><div><br></div><div>Any clue what might have caused this or what can be done to fix it?</div><div><br></div><div>It's Friday afternoon, so there's no urgency from my perspective. I'll probably get back to this on Monday morning.</div><div><br></div><div>Thanks,</div><div>Adrian McCarthy</div></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div><div dir="ltr">-Todd</div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div><div dir="ltr">-Todd</div></div>
</font></span></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div><div dir="ltr">-Todd</div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br><div><div dir="ltr">-Todd</div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">-Todd</div></div>
</div>