[llvm-dev] [cfe-dev] Wrapping lit commands for device testing

[Pavel Labath via llvm-dev]

Thank you for the explanation George. I have too came to believe that a
general all-encompassing solution would be incredibly complicated. Our
(lldb) use case is in many ways similar to compiler-rt (we want to test
code which was not compiled for the host), but then it also has it's own
"thing" (we have *two* processes/binaries, one debugging another,
theoretically they don't even have to have the same bitness).

I am glad you were able to make your use case work. I'll probably try
creating something a bit custom for lldb, when I actually get around to it.

regards,
pavel
On Sat, 5 May 2018 at 03:38, George Karpenkov <ekarpenkov at apple.com> wrote:

> Hi Pavel,

> I was thinking about a generic proposal for a while,
> but then I’ve realized that a much simpler heuristic suffices for all of
my needs:

>    - Prepend all commands which should execute on the device with %run,
which runs a wrapper python script
>    - Inside that script use a simple regexp to find all paths
>    - sync all paths to the device
>    - run the executable on the device
>    - sync all paths form the device back to host

> > On Apr 23, 2018, at 8:09 AM, Pavel Labath <labath at google.com> wrote:
> >
> > Hello George,
> >
> > I would be very much interested in a (generic) solution that would allow
> > running tests on a remote host/device. My intended use case for this is
> > lldb+android, and I am mostly interested in googletest-style tests.

> I’m not sure about those, all my problems were in LIT tests.

> > In
> > fact, I proposed something vaguely similar to this (but limited to
gtest)
> > last year <http://lists.llvm.org/pipermail/llvm-dev/2017-May/113370.html
> .
> > However, due to some internal reshuffles, this idea never got past the
> > prototype stage. Nonetheless, I would still like do something like this
> > eventually (even better if someone else does it for me :) ).
> >
> > I would love to hear more about your proposal, as my head is currently
> > filled with questions about how this would work. (I admit I don't know
> > anything about how libFuzzer is tested.) It's not clear to me whether
you
> > intend to implement a general purpose remote test execution framework,
or a
> > solution tailored to the needs of libFuzzer.

> Seemed generic enough for most of compiler-rt.

> > Do you automatically handle
> > copying the inputs to the remote target? How do you identify what the
> > inputs are?

> Yes, the simple heuristic above seems to work here.

> > Do you copy over any llvm tools executed in the run lines?

> I copy the command directly after %run wrapper, but not FileCheck/etc.
> %run syncs it over to the device.

> > What
> > about their dependent libraries? etc.

> Dynamically linked? Mostly we don’t have any, and for those tests where
we do,
> some rpath substitution magic is required.


> > Maybe if you could give an example of a typical test you intend to
support
> > and how it would work, a lot of these questions would be answered. If
this
> > turns out to be something that could be reused for our use case, I
would be
> > willing to chip in to help make it happen.

> Basically, any of libFuzzer tests with %run prepended, e.g.
https://reviews.llvm.org/D45235

> George

> >
> > regards,
> > pavel
> >
> > On Mon, 23 Apr 2018 at 15:09, George Karpenkov via cfe-dev <
> > cfe-dev at lists.llvm.org> wrote:
> >
> >> Hi,
> >
> >> Tests in compiler-rt run under both iOS simulators and Android devices,
> > which requires “wrapping” the run line with the “adb” command for
Android,
> > and adding a prefix / setting some environment variables for simulators/
> >> In case of Android this is done by changing the compilation command to
> > produce a wrapper which already calls “adb”, and in case of iOS
simulators
> > by explicitly adding the “%run” command in front of run lines.
> >> The solution is far from perfect though: many tests (e.g. libFuzzer)
> > extensively rely on using arbitrary shell commands to create and
manipulate
> > temporary text files.
> >> Performing those commands in a different environment is done in a very
> > ad-hoc manner: from what I see, for sanitizer tests those temporary
files
> > are then usually references in the environment variables present in the
run
> > command, which are then parsed to carry those files into the executable
> > environment.
> >> Doing this for libFuzzer would be very difficult and would require a
lot
> > of modifications to test.
> >
> >> I would prefer a more transparent approach: if LIT configuration was to
> > provide a hook to perform arbitrary replacements/wrapping on each of the
> > executed lines, the wrapping could be performed without modifying tests
at
> > all.
> >
> >> My implementation preference would be adding a method  `def wrap(self,
> > s)` (str -> str) to the format class (lit/formats/*), and then using it
(if
> > non-empty) on each line being run.
> >> I think that is preferable to a method on a `config` object, since line
> > transformation is a very heavyweight operation, akin to other methods
> > available on the `format` class (test discovery semantics and test
> > execution semantics).
> >> Would there be any objections against such an implementation plan?
> >
> >> Regards,
> >> George
> >> _______________________________________________
> >> cfe-dev mailing list
> >> cfe-dev at lists.llvm.org
> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev