[llvm] [DTLTO][LLVM] Integrated Distributed ThinLTO (DTLTO) (PR #127749)

[Fangrui Song via llvm-commits]

================
@@ -0,0 +1,101 @@
+; Check that the JSON output from DTLTO is as expected. Note that validate.py
+; checks the JSON structure so we just check the field contents in this test.
+
+RUN: rm -rf %t && split-file %s %t && cd %t
+
+; Generate bitcode files with summary.
+RUN: opt -thinlto-bc t1.ll -o t1.bc
+RUN: opt -thinlto-bc t2.ll -o t2.bc
+
+; Perform DTLTO.
+RUN: not llvm-lto2 run t1.bc t2.bc -o my.output \
+RUN:     -r=t1.bc,t1,px -r=t2.bc,t2,px \
+RUN:     -dtlto-distributor=%python \
+RUN:     -thinlto-distributor-arg=%llvm_src_root/utils/dtlto/validate.py \
+RUN:     -thinlto-remote-compiler=my_clang.exe \
+RUN:     -thinlto-remote-compiler-arg=--rota1=10 \
+RUN:     -thinlto-remote-compiler-arg=--rota2=20 \
+RUN:     -thinlto-distributor-arg=--da1=10 \
+RUN:     -thinlto-distributor-arg=--da2=10 \
+RUN:   2>&1 | FileCheck %s
+
+CHECK: distributor_args=['--da1=10', '--da2=10']
+
+; Check the common object.
+CHECK:      "linker_output": "my.output"
+CHECK:      "args":
+CHECK:      "my_clang.exe"
+CHECK:      "-o"
+CHECK-NEXT: [
+CHECK-NEXT: "primary_output"
+CHECK-NEXT: 0
+CHECK-NEXT: ]
+CHECK:      "-c"
+CHECK:      "-x"
+CHECK:      "ir"
+CHECK-NEXT: [
+CHECK-NEXT: "primary_input"
+CHECK-NEXT:  0
+CHECK-NEXT: ]
+CHECK:      "summary_index"
+CHECK-NEXT: "-fthinlto-index="
+CHECK-NEXT: 0
+CHECK-NEXT: ]
+CHECK:      "--target=x86_64-unknown-linux-gnu"
+CHECK:      "-O2",
----------------
MaskRay wrote:

This looks like "arguments" (https://clang.llvm.org/docs/JSONCompilationDatabase.html). Every argument on a separate line? This might be too wasteful...

The "command" form, while it has some shell-escape complexity, is probably better in both compactness and debuggability.
I sometimes extract a command from CMake-generated `compile_commands.json` (which uses "command") and invoke it in the shell. The "arguments" form would be less convenient to invoke.

https://github.com/llvm/llvm-project/pull/127749