[libcxx-commits] [PATCH] D77657: [WIP][libc++] Add an executor that bundles tests for deferred execution

[Julian Lettner via Phabricator via libcxx-commits]

yln added a comment.

This is very exciting!  I would definitely want to adopt this for compiler-rt tests.  I am sure the Swift folks would be interested in this as well.

My unorganized thoughts:
I agree that we should teach lit the notion of "deferred execution" in some form or another to make this work.  We essentially need to be able to break up tests into separate steps: setup (compile, generate inputs on host), execute (run on device), assert (inspect outputs on host) and then change execution to do thing stage-by-stage: setup, copy to device, run on device, copy back results, finish tests; instead of test-by-test.  I think most of compiler-rt tests would be amendable to this without changes.

> Lit could have a notion of HOST-RUN: and TARGET-RUN: lines.

Did you consider inferring this via the presence of `%{exec}`?  In compiler-rt, we have `%run`, which is empty if host==target, but does `simctl spawn` for the iOS simulator.

> In addition to the above, we could solve (2) by running Lit on the target in a mode where only the TARGET-RUN: lines are executed. Unfortunately, this requires the target to support running Lit, which is not always reasonable (imagine small embedded devices).

I think we should keep the execute step on the target as "dumb" as possible, i.e., no running lit on the target.  I would restrict the target to do what's specified via `%{exec}`: execute the copied binary and produce outputs.

> Lit could support multi-line RUN: lines, where it would basically create a small script and consider it like a single RUN line. We could then use the %{exec} substitution on such a multi-line RUN instead of having multiple calls to %{exec}. This would solve (3). Alternatively, Lit could have builtin support for running HOST-RUN and TARGET-RUN lines specially.

Did you consider to generate one bundle per test including all of the `%{exec}` lines, instead one bundle per `%{exec}` line?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D77657/new/

https://reviews.llvm.org/D77657