[lld] [lld][MachO]Multi-threaded i/o. Twice as fast linking a large project. (PR #147134)

[Fangrui Song via llvm-commits]

MaskRay wrote:

Thank you for enhancing the performance! Although I don’t maintain the Mach-O port, I can see why maintainers might be concerned about this change. Implementing fully parallel input file scanning is challenging, but I’ve applied it successfully to the ELF port. Could this approach reduce the need for the Mach-O workaround?

---

I have some slides about the ELF work https://gist.github.com/MaskRay/a44a7eb24ba4da029d6bca87b3ae306f#file-optimize-lld-elf-md

## Optimize "Parse input files"

* initialization of sections (embarrassingly parallel)
* COMDAT group resolution
* initialization of local symbols (embarrassingly parallel)
* initialization of non-local symbols
* symbol resolution, possibly archive member extraction

mold has parallelized all the above.

* Initialization of non-local symbols and symbol resolution may affect semantics
* difficult to do with the current lld architecture
* some features make full parallelization difficult: `SHT_*_ATTRIBUTES`, `SHT_LLVM_DEPENDENT_LIBRARIES`

That said, I managed to parallelize all the "initialization" passes as of lld 16.


---

```cpp
for (size_t i = 0; i < files.size(); ++i)
  parseFile(files[i]);
```

Prior to lld 14, parsing a relocatable object file makes section initialization, symbol initialization, and symbol resolution tangled.
```cpp
template <class ELFT> void ObjFile<ELFT>::parse(bool ignoreComdats) {
  // Initialize sections and handle COMDAT groups.
  initializeSections(ignoreComdats);
  // Initialize local and non-local symbols, and perform symbol resolution.
  // Lazy object file extraction may cause parsing of other files.
  initializeSymbols();
}
```

---

```mermaid {scale: 0.8}
flowchart
  comdat["Handle section groups and resolve COMDAT"]
  nonLocalSymbols["Initialize non-local symbols (possiblly interleaved with archive member extraction)"]
  section["(parallel) Initialize non-group sections and local symbols"]
  postParse["(parallel) postParse: report symbol errors, set symbol's section, and
  change symbols defined in non-prevailing sections to Undefined"]
  comdat --> nonLocalSymbols
  nonLocalSymbols --> section
  section --> postParse

  style comdat fill:#eee
  style nonLocalSymbols fill:#eee
```

* Parallelize local symbol initialization: [D119909](https://reviews.llvm.org/D119909)
* Move section assignment from `initializeSymbols` to `postParse`: [D120626](https://reviews.llvm.org/D120626)
* Parallelize input section initialization: [D130810](https://reviews.llvm.org/D130810)

https://github.com/llvm/llvm-project/pull/147134