[llvm] [CMake][LIT] Add option to run lit testsuites in parallel (PR #82899)

[Patrick Dougherty via llvm-commits]

patrickdoc wrote:

Hey @joker-eph, the pictures in my example weren't my actual builds, just a demonstration of the intended change. I'm aware that lit also tries to parallelize on top of ninja's parallelism so have reduced lit's parallelism with `-j X` to compensate.

My use-case is CI builds, where we are running the whole build + all the tests, with the goal of being as fast as possible.

Using a single mega lit target has some downsides:

1. It artificially splits the build into first a compile phase, then a test phase. This means that tests which could be running against early compilation targets get delayed until the end of the build
2. It doesn't allow flexibility for different sizes of tests. i.e. N small unit tests would have poor utilization of available cpu/mem/etc. while N larger end-to-end integration tests could run into the oversubscribing issue you've described
3. It doesn't play well with other testing targets in the build. As you mentioned, lit assumes it owns the machine and so runs one test per core. But Ninja doesn't know that so can schedule other targets alongside lit.

The wins I am seeing come from moving parallelism logic out of lit and into Ninja. So we end up with a much smoother build graph because Ninja knows more about the build than lit does.

But to do that, we need to allow Ninja to flexibly schedule the lit targets. The upside of the `USES_TERMINAL` setting is that you can get live progress bar output, but the downside is that Ninja can only run one target as a time.

Truthfully even just removing this flag altogether wouldn't affect the "single lit target" use-case (aside from losing the progress bar) because it is already delayed until after every other target anyway. But I've added it as an option here to allow folks to opt-in.

<hr>

For clarification in the docs, would something like this be better:

```
**LLVM_PARALLEL_LIT**:BOOL
  Defaults to ``OFF``. If using more than one lit CMake target and the Ninja
  backend, setting this to ``ON`` allows Ninja to run the targets in parallel. This
  could lead to too many tests running at once, so it is recommended to also
  reduce lit's internal parallelism with something like`-DLLVM_LIT_ARGS="-j 1"`.
```

https://github.com/llvm/llvm-project/pull/82899