[lldb-dev] The lit test driver gets killed because of SIGHUP

[Jonas Devlieghere via lldb-dev]

Hi everyone,

Since we switched to lit as the test driver we've been seeing it getting killed as the result of a SIGHUP signal. The problem doesn't reproduce on every machine and there seems to be a correlation between number of occurrences and thread count. 

Davide and Raphael spent some time narrowing down what particular test is causing this and it seems that TestChangeProcessGroup.py is always involved. However it never reproduces when running just this test. I was able to reproduce pretty consistently with the following filter:

./bin/llvm-lit ../llvm/tools/lldb/lit/Suite/ --filter="process"

Bisecting the test itself didn't help much, the problem reproduces as soon as we attach to the inferior. 

At this point it is still not clear who is sending the SIGHUP and why it's reaching the lit test driver. Fred suggested that it might have something to do with process groups (which would be an interesting coincidence given the previously mentioned test) and he suggested having the test run in different process groups. Indeed, adding a call to os.setpgrp() in lit's executeCommand and having a different process group per test prevent us from seeing this. Regardless of this issue I think it's reasonable to have tests run in their process group, so if nobody objects I propose adding this to lit in llvm. 

Still, I'd like to understand where the signal is coming from and fix the root cause in addition to the symptom. Maybe someone here has an idea of what might be going on? 

Thanks,
Jonas

PS
 
1. There's two places where we send a SIGHUP ourself, with that code removed we still receive the signal, which suggests that it might be coming from Python or the OS.  
2. If you're able to reproduce you'll see that adding an early return before the attach in TestChangeProcessGroup.py hides/prevents the problem. Moving the return down one line and it pops up again.