[PATCH] D65430: Add `--dependency-files` option, which is equivalent to compiler option -MD.

Mon Aug 5 22:43:47 PDT 2019

MaskRay added a subscriber: bd1976llvm.
MaskRay added a comment.

In D65430#1615429 <https://reviews.llvm.org/D65430#1615429>, @phosek wrote:

> One use case for `ld.lld --dependency-file=` is auto-linking which is supported by lld for both ELF and COFF. In case of auto-linking, you don't know which additional dependencies are going to be included until you process all input files. Another case that's specific to ELF are linker scripts which allows specifying additional inputs using the `INPUT` statement e.g. libc++ installs the following linker script `INPUT(libc++.so.1 -lunwind -lc++abi)` as `libc++.so`.
>
> Regarding the depfile format, the simplified Makefile format is the de-facto standard for depfiles and is emitted and supported by build systems like Make or Ninja (which is generated by number of other high-level build systems like CMake, Meson, GN, etc.). Using this format would make this functionality immediately usable.

I carefully think about this. deplibs is probably the only reasonable use case. Let me give an example to ensure I understand it correctly:

  touch d.h

  cat > a.c <<e
  #include "d.h"
  #pragma comment(lib, "b.a")
  #pragma comment(lib, "c.a")
  extern int b, c;
  int main() {}
  e

  cat > b.c <<e
  int b;
  e

  cat > c.c <<e
  int c;
  e

build.ninja:

  rule cc
    depfile = $out.d
    command = clang -MD -MT $out -MF $out.d -c $in -o $out

  rule link
    depfile = $out.link.d
    command = clang $in -fuse-ld=lld -Wl,--dependency-file=$out.link.d -o $out

  build a: link a.o
  build a.o: cc a.c

Alternatively, Makefile:

  CFLAGS = -MD -MP -MF $@.d  # note -MP is not required in build.ninja
  CC = clang
  LDFLAGS = -fuse-ld=lld -Wl,--dependency-file=$@.link.d
  a_deps = a.o  # explicit deps (.deplibs are excluded)

  # implicit rule of a cannot be used because it would link all files specified in a.link.d together
  a: $(a_deps)
  	$(LINK.c) $(a_deps) $(LDLIBS) -o $@

  -include a.o.d
  -include a.link.d

This dependency file approach has an assumption: `a.o.d` is not older than `a.o`, `a.link.d` is not older than `a`.

1. if `a.o.d` does not exist, according to the assumption, `a.o` does not exist as well. `ninja` or `make` generates `a.o` and `a.o.d`. `a` and `a.link.d` are similar.
2. if `a.c` or `d.h` is updated, `a` get generated due to the dependency described in `a.o.d`.
3. if `d.h` is deleted and `#include "d.h"` is deleted from `a.c`. Note there is no rule to generate `d.h`. ninja seems to ignore its nonexistence, while Makefile needs a force target. That is why in the Makefile version we use `-MP`.
4. if `b.a` is deleted and `#pragma comment(lib, "b.a")` is deleted from `a.c`. ninja ignores b.a and generates `a`. However, GNU make will error: `make: *** No rule to make target 'b.a', needed by 'a'.  Stop.`

I think `ld.lld --write-dependencies=` can be plugged into a ninja based build system (including `cmake -G Ninja`) without change, but not into a Makefile based build system.
To use this feature in meson/bazel/buck/..., a post-processing tool is definitely needed.
However, if a post-processing tool is ever needed, `ld.lld -t` will be just as simple (or difficult) to use.

Let's revisit this ninja fragment:

  rule link
    depfile = $out.link.d
    command = clang $in -fuse-ld=lld -Wl,--dependency-file=$out.link.d -o $out

`#pragma comment(lib, "b.a")` is not common. If the build system goes through the trouble to use deplibs,
there may likely be other post processing steps. I wonder if it can just point `-fuse-ld=` to a custom wrapper that processes `ld.lld -t` output and generates `$out.link.d`.

So the question is whether we consider this feature too application-specific.
For simple dependency information, we can already use `-t`.
For richer information, there is a proposal for dependency analysis (https://lists.llvm.org/pipermail/llvm-dev/2019-February/130565.html)

I'd also like to hear from @bd1976llvm, who implemented the deplibs feature in the first place.

================
Comment at: lld/ELF/Config.h:87
   llvm::CachePruningPolicy thinLTOCachePolicy;
+  llvm::SetVector<StringRef> dependencyFiles; // for --dependency-file
   llvm::StringMap<uint64_t> sectionStartMap;
----------------
This probably should be

  llvm::SetVector<std::string, std::vector<std::string>,
                  llvm::StringSet<llvm::MallocAllocator>>
      dependencyFiles; // for --dependency-file

Some `StringRef` elements are not internalized.

================
Comment at: lld/ELF/Driver.cpp:1377

+// Handle --write-dependencies=<path>. If that option is given, lld creates a
+// file at a given path with the following contents:
----------------
`--dependency-file`

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D65430/new/

https://reviews.llvm.org/D65430