[clang] [Clang] add option --offload-jobs=N (PR #135229)

[Yaxun Liu via cfe-commits]

================
@@ -1233,6 +1233,10 @@ def offload_compression_level_EQ : Joined<["--"], "offload-compression-level=">,
   Flags<[HelpHidden]>,
   HelpText<"Compression level for offload device binaries (HIP only)">;
 
+def offload_jobs_EQ : Joined<["--"], "offload-jobs=">,
+  HelpText<"Set the number of threads for the clang-linker-wrapper. Defaults to"
+           " half of the available hardware threads if not specified.">;
----------------
yxsamliu wrote:

Theoretically, we might be able to create cmake functions that can create a custom command with proper inter-dependency for each internal command created by clang-linker-wrapper, then we could let cmake parallelize these commands together with other compilation/linking commands. We may consider adding some cmake module for this purpose.

That said, I think allowing clang driver to parallelize its own jobs is still useful to interactive users. Even for cmake users, the concern about overloading system can be alleviated by using the `-l` options of gmake or ninja, which automatically limits new jobs based on current system load. In practice, we have used an out-of-tree parallel jobs option for HIP that is similar to clang-linker-wrapper for many years. It is a necessity for HIP projects containing large source files since the bottleneck is a time-consuming job that needs to be done for multiple GPU arch's. 

https://github.com/llvm/llvm-project/pull/135229