[flang-commits] [flang] [flang] Add FLANG_PARALLEL_COMPILE_JOBS option (PR #127364)

[Michael Kruse via flang-commits]

Meinersbur wrote:

To elaborate on https://github.com/llvm/llvm-project/issues/112789#issuecomment-2429553410:

I think this approach is a bigger footgun than the problem it wants to solve. If someone sets `LLVM_RAM_PER_COMPILE_JOB=2000` (2GB), which seems to be the documented solution for running out of memory during compilation, of all the jobs, this PR will make that setting to be ignored for flang compile jobs.

Using multiple limited jobs pools is problematic by itself. Say we limit the number of flang jobs to 4 and other jobs to 8. That still means that up to 12 jobs can run concurrently, stealing each other's memory. Unfortunately its seems that it is not poosible for a job to belong to multiple pools, being limited by the most restricted of them.

What I have been doing:
1. Run ninja with Flang's dependencies: `ninja clang FIRTransforms HLFIRTransforms MLIRDebug MLIROptLib opt llc mlir-translate mlir-opt -j12`
2. Run ninja for flang with limited parallelism: `ninja bbc flang -j3`
3. Build everything else with normal parallelism: `ninja -j12`

https://github.com/llvm/llvm-project/pull/127364