[libcxx-commits] [PATCH] D77657: [WIP][libc++] Add an executor that bundles tests for deferred execution

[Louis Dionne via Phabricator via libcxx-commits]

ldionne created this revision.
Herald added subscribers: libcxx-commits, broadwaylamb, dexonsmith, jkorous.
Herald added a project: libc++.
Herald added a reviewer: libc++.
ldionne added subscribers: yln, danalbert, thakis.
ldionne added a comment.

This review is a WIP for solving remote execution of libc++ tests with `lit` better than we currently do. See the review summary for the initial discussion.

Adding a bunch of folks that might be interested by the discussion -- feel free to unsubscribe if you have no interest!


ldionne added a comment.

Also, just to explain how I'm currently running the tests with this patch applied:

  rsync -am --include='*.bundle' --include='*/' --exclude='*' $PWD/build $PWD/all-tests
  
  for bundle in $(find all-tests -name '*.bundle'); do
      if ! sh ${bundle}; then
          echo ${bundle} failed
      fi
  done

I'm currently running them locally, but if the tests were compiled for some device, I could just as easily copy the bundles over and execute them with a single `ssh` command.


Instead of actually running tests, this executor instead bundles the
executable and its dependencies into a tarball, and then creates a
self-contained script that can be used to run that test later.

The script contains the tarball of dependencies embedded inside itself,
so one can simply copy the script anywhere and execute it, provided the
machine and the executable have compatible architectures. This is an
alternative to SSH executors, which run tests remotely as `lit` executes
instead of deferring that execution. Deferring execution allows doing
batch copies and executes, which can be much faster than SSHing to a
remote machine for each test.

However, there are still some problems with this approach as of right now:

1. Lit doesn't have a notion of "deferred execution", so tests that should actually run a command will be marked as PASS even though they might fail when the test is actually run. Simmilarly, if the test is expected to XFAIL at runtime, lit will think that the test XPASSed if the bundling step succeeds.
2. Similarly, when running the tests on the target machine, any XFAIL test that (correctly) fails will kind-of be reported as a failure, since the executable will return a non-zero code.
3. When running a ShTest that has multiple RUN commands (all prefixed by %{exec}), the bundle will be re-created for each line instead of having a single bundle for all RUN lines. This basically doesn't work at all.

I believe there are solutions to all of the above, but solving them
properly will require some thinking and some changes to lit. Here are
some ideas:

- Lit could have a notion of HOST-RUN: and TARGET-RUN: lines. The idea is that Lit wouldn't even try running TARGET-RUN: lines on the host, and if any such line exists, the test would be reported as having a new status HOST-PASS or TARGET-DEFERRED or whatever. That new test status would represent that Lit can't know for sure yet whether the test will pass on the target. That would solve (1).

- In addition to the above, we could solve (2) by running Lit on the target in a mode where only the TARGET-RUN: lines are executed. Unfortunately, this requires the target to support running Lit, which is not always reasonable (imagine small embedded devices).

- Lit could support multi-line RUN: lines, where it would basically create a small script and consider it like a single RUN line. We could then use the %{exec} substitution on such a multi-line RUN instead of having multiple calls to %{exec}. This would solve (3). Alternatively, Lit could have builtin support for running HOST-RUN and TARGET-RUN lines specially.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D77657

Files:
  libcxx/utils/bundle.py
  libcxx/utils/libcxx/test/config.py

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D77657.255695.patch
Type: text/x-patch
Size: 6422 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/libcxx-commits/attachments/20200407/68b47453/attachment-0001.bin>