[llvm-dev] Support for deferred execution in lit

Thu Jul 15 22:54:46 PDT 2021

We are executing these library tests on a baremetal target, so for us it 
is not possible to copy over batches of files (we can't even use SCP and 
SSH). At the moment we use a simple python script as our %{exec} command 
that loads the cross-compiled binary into the targets memory, where it 
is then executed.

Having the tests compiled in batch and then executed one by one would 
work for us however. In general, as long as our use case continues to 
work without any major changes to our test infrastructure, I'm all for 
trying to improve the testing performance of the libraries.

Cheers,

Dominik

On 15.07.21 22:04, Stephen Neuendorffer via llvm-dev wrote:
> We've been looking at some related questions and have some other 
> options that you might be interested in.  First of all a comment: The 
> idea of 'deferred execution' seems problematic since it fundamentally 
> changes the semantics of the report from lit.  I'm curious how you are 
> able to use this in practice to, for instance, check that tests were 
> eventually run somewhere?
>
> We have an embedded environment where building LLVM itself is barely 
> feasible due to resource constraints. As a result we cross-compile 
> LLVM and generate an installed tarball copied to the embedded system.  
> However, this means that we can't test the cross-compiled LLVM 
> executables until we get to the embedded system.
> Our approach has been to factor the test directory into a hierarchical 
> cmake project.  This project then uses the standard LLVM cmake export 
> mechanisms (i.e. find_project) to find LLVM.  This refactoring has no 
> effect on a regular in-tree toplevel build.  However, we can checkout 
> the LLVM tree on the embedded system and build *just the test area* 
> using the installed tarball of LLVM.  I think this refactoring of 
> cmake is something that would be relatively easy to carry out on the 
> LLVM tree.  Relative to your current approach, this moves the problem 
> of tarballing and remote code execution out of lit's responsibility 
> and into a more devops/release responsibility, which makes more sense 
> to me.
>
> Perhaps you also have other goals, such as partitioning tests to run 
> on multiple target nodes?  I haven't thought too much about how this 
> would interact.
>
> Separately, we also have the problem of tests that need to behave 
> differently in different contexts.  e.g.:
> RUN: clang --target=my_cross_target ... -o test.elf
> RUN: %run% test.elf
>
> In this case, we'd like to be able to test the compilation part 
> outside of the target, but when we run the same test on the target 
> machine, we can compile and run.  In this case we do something similar 
> (as you see above) using a lit subsitution that varies depending on 
> the cmake environment. Doing this is somewhat clumsy and I've thought 
> it would be nicer to move this into lit, allowing the test to be:
>
> RUN: clang --target=my_cross_target ... -o test.elf
> RUN_ON_TARGET: %run% test.elf
>
> In this case the behavior of RUN*: lines would be configurable in the 
> lit.cfg.py <http://lit.cfg.py>.  This could implement part of your 
> current use case (although maybe there would be impacts on how the 
> reporting is done?)
>
> Steve
>
>
>
> On Thu, Jul 15, 2021 at 11:54 AM Petr Hosek via llvm-dev 
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>
>     This is a topic that came out in our recent discussions about
>     remote test execution in libc++ and other runtimes.
>
>     libc++ lit test suite has support for running tests remotely using
>     a custom executor. compiler-rt has a similar support. The problem
>     is that this is done in a very ad-hoc way on a per-command basis.
>
>     The most basic example looks as follows:
>
>         RUN: %{exec} %t.exe
>
>     When executing tests locally, %{exec} would be empty (or it could
>     be a binary like env). When executing tests remotely, for example
>     over SSH which is the most common case, %{exec} is expanded into a
>     script
>     <https://github.com/llvm/llvm-project/blob/b8b23aa/libcxx/utils/ssh.py> that
>     uses SCP to copy the binary to the remote target and SSH to
>     execute it. While we're waiting for the command to finish, the
>     test execution is blocked.
>
>     When you only have a handful of tests, it's a reasonable approach,
>     but it becomes a problem with a large number of tests (as in the
>     case of libc++) because the overhead of copying and executing
>     tests one-by-one can be significant. It gets worse if setting up
>     the target test environment is expensive, which can be the case
>     for some embedded environments.
>
>     It would be more efficient to bundle up all binaries (with their
>     dependencies), copy them over to the target and run them all but
>     that pattern is difficult to express in lit right now.
>
>     https://reviews.llvm.org/D77657 <https://reviews.llvm.org/D77657>
>     is one possible implementation but there are some unresolved
>     issues described in the details of that change.
>
>     While we could try and workaround some of these issues, we think
>     that a better solution would be to introduce a notion of "deferred
>     execution" into lit, so any RUN lines marked as deferred wouldn't
>     be run immediately and the test would be reported as having a new
>     status DEFERRED. We would then ideally have some way of collecting
>     all deferred commands and providing a custom handler (for example
>     via TestingConfig) that could do things like packaging up all
>     binaries and executing them on the target device.
>
>     We think that such a feature would be generally useful but I'd
>     like to collect more feedback before we go ahead with the
>     implementation. Do you think such a feature would be useful? Is
>     there another way of supporting batched/deferred execution of test
>     binaries with lit?
>     _______________________________________________
>     LLVM Developers mailing list
>     llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>     https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>     <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210716/7e1ce8ab/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 6822 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210716/7e1ce8ab/attachment.bin>