[lld] [LLD][ELF] add bp-* options in ELF (PR #120514)

[via llvm-commits]

Andarwinux wrote:

If I understand correctly, a typical use case for BP would look like this:
```
clang -flto=thin -O3 -fprofile-generate=profile -mllvm -pgo-temporal-instrumentation foo.c -o a.out

./a.out

llvm-profdata merge profile/*.profraw -o foo.profdata

clang -flto=thin -O3 -fprofile-use=foo.profdata -Wl,--irpgo-profile=foo.profdata,--bp-compression-sort=both,--bp-startup-sort=function foo.c -o b.out

# linker call emitted by clang
"ld.lld" "-plugin-opt=cs-profile-path=foo.profdata" "--irpgo-profile=foo.profdata"  "--bp-compression-sort=both" "bp-startup-sort=function" ...
```
I'm not sure if the CSIR+BP use case is supported, it might look something like this:

```
clang -flto=thin -O3 -fprofile-generate=profile -mllvm -pgo-temporal-instrumentation foo.c -o a.out

./a.out

llvm-profdata merge profile/*.profraw -o foo.profdata

clang -flto=thin -O3 -fcs-profile-generate=csprofile -fprofile-use=foo.profdata -mllvm -pgo-temporal-instrumentation -Wl,--irpgo-profile=foo.profdata,--bp-compression-sort=both,--bp-startup-sort=function foo.c -o b.out

# linker call emitted by clang
# foo.profdata is for BP only
"ld.lld" "-plugin-opt=cs-profile-generate" "-plugin-opt=cs-profile-path=csprofile/default_%m.profraw" "--irpgo-profile=foo.profdata"  "--bp-compression-sort=both" "bp-startup-sort=function"

./b.out

llvm-profdata merge csprofile/*.profraw foo.profdata -o foo_cs.profdata

clang -flto=thin -O3 -fprofile-use=foo_cs.profdata -Wl,--irpgo-profile=foo_cs.profdata,--bp-compression-sort=both,--bp-startup-sort=function foo.c -o c.out

# linker call emitted by clang
# foo_cs.profdata for both BP and CSIR PGO
"ld.lld" "-plugin-opt=cs-profile-path=foo_cs.profdata" "--irpgo-profile=foo_cs.profdata" "--bp-compression-sort=both" "bp-startup-sort=function" ...
```
>From the end-user's point of view, it is very strange to need to pass the same profdata path to clang and lld separately.

If it were possible to use irpgo-profile as a generic way for the linker to read profdata, and cs-profile-path only as an option to specify the path where the CSIR profraw is to be generated, it would look like this:
```
clang -flto=thin -O3 -fprofile-generate=profile -mllvm -pgo-temporal-instrumentation foo.c -o a.out

./a.out

llvm-profdata merge profile/*.profraw -o foo.profdata

clang -flto=thin -O3 -fprofile-use=foo.profdata -Wl,--bp-compression-sort=both,--bp-startup-sort=function foo.c -o b.out

# linker call emitted by clang
"ld.lld" "-plugin-opt=irpgo-profile=foo.profdata" "--bp-compression-sort=both" "bp-startup-sort=function" ...
```
or CSIR+BP
```
clang -flto=thin -O3 -fprofile-generate=profile -mllvm -pgo-temporal-instrumentation foo.c -o a.out

./a.out

llvm-profdata merge profile/*.profraw -o foo.profdata

clang -flto=thin -O3 -fcs-profile-generate=csprofile -fprofile-use=foo.profdata -mllvm -pgo-temporal-instrumentation -Wl,--bp-compression-sort=both,--bp-startup-sort=function foo.c -o b.out

# linker call emitted by clang
# foo.profdata is for BP only
"ld.lld" "-plugin-opt=cs-profile-generate" "-plugin-opt=cs-profile-path=csprofile/default_%m.profraw" "-plugin-opt=irpgo-profile=foo.profdata" "--bp-compression-sort=both" "bp-startup-sort=function"

./b.out

llvm-profdata merge csprofile/*.profraw foo.profdata -o foo_cs.profdata

clang -flto=thin -O3 -fprofile-use=foo_cs.profdata -Wl,--bp-compression-sort=both,--bp-startup-sort=function foo.c -o c.out

# linker call emitted by clang
# foo_cs.profdata for both BP and CSIR PGO
"ld.lld" "-plugin-opt=irpgo-profile=foo_cs.profdata" "--bp-compression-sort=both" "bp-startup-sort=function" ...
```

Personally, I think the second design is cleaner? This may not be entirely relevant to this PR, but since both BP and CSIR have cases where profdata will be passed to the linker, I think it still makes sense to consider interactions in such cases.

https://github.com/llvm/llvm-project/pull/120514