[libcxx-dev] Support for deferred execution in lit

[Petr Hosek via libcxx-dev]

This is a topic that came out in our recent discussions about remote test
execution in libc++ and other runtimes.

libc++ lit test suite has support for running tests remotely using a custom
executor. compiler-rt has a similar support. The problem is that this is
done in a very ad-hoc way on a per-command basis.

The most basic example looks as follows:

    RUN: %{exec} %t.exe

When executing tests locally, %{exec} would be empty (or it could be a
binary like env). When executing tests remotely, for example over SSH which
is the most common case, %{exec} is expanded into a script
<https://github.com/llvm/llvm-project/blob/b8b23aa/libcxx/utils/ssh.py> that
uses SCP to copy the binary to the remote target and SSH to execute it.
While we're waiting for the command to finish, the test execution is
blocked.

When you only have a handful of tests, it's a reasonable approach, but it
becomes a problem with a large number of tests (as in the case of libc++)
because the overhead of copying and executing tests one-by-one can be
significant. It gets worse if setting up the target test environment is
expensive, which can be the case for some embedded environments.

It would be more efficient to bundle up all binaries (with their
dependencies), copy them over to the target and run them all but that
pattern is difficult to express in lit right now.

https://reviews.llvm.org/D77657 is one possible implementation but there
are some unresolved issues described in the details of that change.

While we could try and workaround some of these issues, we think that a
better solution would be to introduce a notion of "deferred execution" into
lit, so any RUN lines marked as deferred wouldn't be run immediately and
the test would be reported as having a new status DEFERRED. We would then
ideally have some way of collecting all deferred commands and providing a
custom handler (for example via TestingConfig) that could do things like
packaging up all binaries and executing them on the target device.

We think that such a feature would be generally useful but I'd like to
collect more feedback before we go ahead with the implementation. Do you
think such a feature would be useful? Is there another way of supporting
batched/deferred execution of test binaries with lit?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/libcxx-dev/attachments/20210715/395b327f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3996 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.llvm.org/pipermail/libcxx-dev/attachments/20210715/395b327f/attachment.bin>