[llvm-bugs] [Bug 48193] New: nproc threads spawn by default regardless of input

[via llvm-bugs]

https://bugs.llvm.org/show_bug.cgi?id=48193

            Bug ID: 48193
           Summary: nproc threads spawn by default regardless of input
           Product: lld
           Version: unspecified
          Hardware: All
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P
         Component: All Bugs
          Assignee: unassignedbugs at nondot.org
          Reporter: mjguzik at gmail.com
                CC: llvm-bugs at lists.llvm.org, smithp352 at googlemail.com

I happen to be running this on FreeBSD with:
FreeBSD clang version 11.0.0 (git at github.com:llvm/llvm-project.git
llvmorg-11.0.0-0-g176249bd673)

But the same problem exists on Linux.

My test box has 104 hardware threads and lld spawns this many threads even when
linking helloworld. This becomes a problem when there are many binaries built
in parallel (for example when building the entire system or worse yet, building
all 3rd party packages). It cannot possibly benefit from this many threads for
such a small binary.

There is Linux-specific code to limit the thread count based on the result of
sched_getaffinity (as found in computeHostNumHardwareThreads), but using custom
affinity is not a solution at least for 2 reasons:
1. it restricts not only how many hw threads can be used, but which ones, which
requires hand-scheduling. in contrast, say if the machine has 16 threads, one
lld spawns 4 and another spawns 8, the system can schedule them as it sees fit.
2. it requires guessing for the compiler how many threads can be effectively
used for given binary

Neither thing is feasible to do with thousands of binaries to build (which is
the case for the 3rd party package build mentioned above).

I don't know anything about the internals so I don't have a solution. However,
the least which can be done is have a heuristic which estimates how many
threads are useful to spawn in the first place and it is fine if it
overestimates to some extent.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20201116/64b28a54/attachment.html>