[llvm] [lit] Truncate process output to 10 kiB (PR #206355)

[Alexis Engelke via llvm-commits]

aengelke wrote:

> 10KB isn't all that much

Happy to change the limit.

> esp. if this applies for data piped between processes in the internal shell.

This should only truncate output that is printed; data piped between processes uses file descriptors, this is handled earlier (see default_stdin = procs[-1].stdout) and this is not truncated (otherwise, a great deal more of the tests would fail).

(libcxx tests fail because they do strange things with lit, but this should be fixable.)

> Why does lit send all the output back to the main process?

For printing? Note that we can also print the output of successful tests.

>From what I understand, this is what happens:

- All non-piped/redirected output of processes is collected (communicate() for the last process of the pipe, read() for all previous.)
  - It's not good that we *collect* the entire output at all (frequent realloc+memcpy for large buffers) and I'd rather not have a possibly large output not stored in Python at all.
  - As a stop gap measure, this PR immediately truncates the output before it is processed further.
- The output is converted into strings (memcpy) and stored in the results list of executeScriptInternal.
- executeScriptInternal builds the debug output combining all these stdout/stderr.
  - It performs a lot of `out += ...`, which most allocates (malloc+memcpy) a new string on every change. There are many of these concatenations.
- The combined debug output is returned (together with other things) to _runShTest, which determines whether the test passed, executing the test multiple times if necessary. It also string-formats the output.
- ShTest returns the result, which is pickled, sent to the main process (write/read), unpickled, and processed further e.g. for writing the test result to stdout.

I hope this clarifies what I think is happening here.

https://github.com/llvm/llvm-project/pull/206355