[cfe-dev] Wrapping lit commands for device testing

[Pavel Labath via cfe-dev]

Hello George,

I would be very much interested in a (generic) solution that would allow
running tests on a remote host/device. My intended use case for this is
lldb+android, and I am mostly interested in googletest-style tests. In
fact, I proposed something vaguely similar to this (but limited to gtest)
last year <http://lists.llvm.org/pipermail/llvm-dev/2017-May/113370.html>.
However, due to some internal reshuffles, this idea never got past the
prototype stage. Nonetheless, I would still like do something like this
eventually (even better if someone else does it for me :) ).

I would love to hear more about your proposal, as my head is currently
filled with questions about how this would work. (I admit I don't know
anything about how libFuzzer is tested.) It's not clear to me whether you
intend to implement a general purpose remote test execution framework, or a
solution tailored to the needs of libFuzzer. Do you automatically handle
copying the inputs to the remote target? How do you identify what the
inputs are? Do you copy over any llvm tools executed in the run lines? What
about their dependent libraries? etc.

Maybe if you could give an example of a typical test you intend to support
and how it would work, a lot of these questions would be answered. If this
turns out to be something that could be reused for our use case, I would be
willing to chip in to help make it happen.

regards,
pavel

On Mon, 23 Apr 2018 at 15:09, George Karpenkov via cfe-dev <
cfe-dev at lists.llvm.org> wrote:

> Hi,

> Tests in compiler-rt run under both iOS simulators and Android devices,
which requires “wrapping” the run line with the “adb” command for Android,
and adding a prefix / setting some environment variables for simulators/
> In case of Android this is done by changing the compilation command to
produce a wrapper which already calls “adb”, and in case of iOS simulators
by explicitly adding the “%run” command in front of run lines.
> The solution is far from perfect though: many tests (e.g. libFuzzer)
extensively rely on using arbitrary shell commands to create and manipulate
temporary text files.
> Performing those commands in a different environment is done in a very
ad-hoc manner: from what I see, for sanitizer tests those temporary files
are then usually references in the environment variables present in the run
command, which are then parsed to carry those files into the executable
environment.
> Doing this for libFuzzer would be very difficult and would require a lot
of modifications to test.

> I would prefer a more transparent approach: if LIT configuration was to
provide a hook to perform arbitrary replacements/wrapping on each of the
executed lines, the wrapping could be performed without modifying tests at
all.

> My implementation preference would be adding a method  `def wrap(self,
s)` (str -> str) to the format class (lit/formats/*), and then using it (if
non-empty) on each line being run.
> I think that is preferable to a method on a `config` object, since line
transformation is a very heavyweight operation, akin to other methods
available on the `format` class (test discovery semantics and test
execution semantics).
> Would there be any objections against such an implementation plan?

> Regards,
> George
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev