[lld] [lld][MachO]Multi-threaded i/o. Twice as fast linking a large project. (PR #147134)

[Daniel Rodríguez Troitiño via llvm-commits]

================
@@ -282,11 +284,83 @@ static void saveThinArchiveToRepro(ArchiveFile const *file) {
           ": Archive::children failed: " + toString(std::move(e)));
 }
 
-static InputFile *addFile(StringRef path, LoadType loadType,
-                          bool isLazy = false, bool isExplicit = true,
-                          bool isBundleLoader = false,
-                          bool isForceHidden = false) {
-  std::optional<MemoryBufferRef> buffer = readFile(path);
+class DeferredFile {
+public:
+  DeferredFile(StringRef path, bool isLazy, MemoryBufferRef buffer)
+      : path(path), isLazy(isLazy), buffer(buffer) {}
+  StringRef path;
+  bool isLazy;
+  MemoryBufferRef buffer;
+};
+using DeferredFiles = std::vector<DeferredFile>;
+
+// Most input files have been mapped but not yet paged in.
+// This code forces the page-ins on multiple threads so
+// the process is not stalled waiting on disk buffer i/o.
+void multiThreadedPageInBackground(const DeferredFiles &deferred) {
+  static const size_t pageSize = Process::getPageSizeEstimate();
+  static size_t totalBytes = 0;
+  std::atomic_int index = 0;
+
+  parallelFor(0, config->readThreads, [&](size_t I) {
----------------
drodriguez wrote:

Because of how you are using `parallelFor`, `config->readThreads` is only the maximum number of threads that might be spawn for this, not the exact number of threads, which can be lower, governed by the `-threads` parameter. `parallelFor` uses `llvm::parallel::strategy` internally to decide the actual number of threads, which is setup globally when the driver finds the `-threads` argument.

Using `parallelFor` as its authors intended will also avoid the need of the `index` variable and keeping track of it. I provided a snippet before of how I would switch the strategy to one that fits your idea of having a different number of threads for reading files. It should be in some old comment.

https://github.com/llvm/llvm-project/pull/147134