[llvm-dev] [cfe-dev] Wrapping lit commands for device testing

[George Karpenkov via llvm-dev]

Hi Pavel,

I was thinking about a generic proposal for a while,
but then I’ve realized that a much simpler heuristic suffices for all of my needs:

 - Prepend all commands which should execute on the device with %run, which runs a wrapper python script
 - Inside that script use a simple regexp to find all paths
 - sync all paths to the device
 - run the executable on the device
 - sync all paths form the device back to host

> On Apr 23, 2018, at 8:09 AM, Pavel Labath <labath at google.com> wrote:
> 
> Hello George,
> 
> I would be very much interested in a (generic) solution that would allow
> running tests on a remote host/device. My intended use case for this is
> lldb+android, and I am mostly interested in googletest-style tests.

I’m not sure about those, all my problems were in LIT tests.

> In
> fact, I proposed something vaguely similar to this (but limited to gtest)
> last year <http://lists.llvm.org/pipermail/llvm-dev/2017-May/113370.html>.
> However, due to some internal reshuffles, this idea never got past the
> prototype stage. Nonetheless, I would still like do something like this
> eventually (even better if someone else does it for me :) ).
> 
> I would love to hear more about your proposal, as my head is currently
> filled with questions about how this would work. (I admit I don't know
> anything about how libFuzzer is tested.) It's not clear to me whether you
> intend to implement a general purpose remote test execution framework, or a
> solution tailored to the needs of libFuzzer.

Seemed generic enough for most of compiler-rt.

> Do you automatically handle
> copying the inputs to the remote target? How do you identify what the
> inputs are?

Yes, the simple heuristic above seems to work here.

> Do you copy over any llvm tools executed in the run lines?

I copy the command directly after %run wrapper, but not FileCheck/etc.
%run syncs it over to the device.

> What
> about their dependent libraries? etc.

Dynamically linked? Mostly we don’t have any, and for those tests where we do,
some rpath substitution magic is required.


> Maybe if you could give an example of a typical test you intend to support
> and how it would work, a lot of these questions would be answered. If this
> turns out to be something that could be reused for our use case, I would be
> willing to chip in to help make it happen.

Basically, any of libFuzzer tests with %run prepended, e.g. https://reviews.llvm.org/D45235

George

> 
> regards,
> pavel
> 
> On Mon, 23 Apr 2018 at 15:09, George Karpenkov via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
> 
>> Hi,
> 
>> Tests in compiler-rt run under both iOS simulators and Android devices,
> which requires “wrapping” the run line with the “adb” command for Android,
> and adding a prefix / setting some environment variables for simulators/
>> In case of Android this is done by changing the compilation command to
> produce a wrapper which already calls “adb”, and in case of iOS simulators
> by explicitly adding the “%run” command in front of run lines.
>> The solution is far from perfect though: many tests (e.g. libFuzzer)
> extensively rely on using arbitrary shell commands to create and manipulate
> temporary text files.
>> Performing those commands in a different environment is done in a very
> ad-hoc manner: from what I see, for sanitizer tests those temporary files
> are then usually references in the environment variables present in the run
> command, which are then parsed to carry those files into the executable
> environment.
>> Doing this for libFuzzer would be very difficult and would require a lot
> of modifications to test.
> 
>> I would prefer a more transparent approach: if LIT configuration was to
> provide a hook to perform arbitrary replacements/wrapping on each of the
> executed lines, the wrapping could be performed without modifying tests at
> all.
> 
>> My implementation preference would be adding a method  `def wrap(self,
> s)` (str -> str) to the format class (lit/formats/*), and then using it (if
> non-empty) on each line being run.
>> I think that is preferable to a method on a `config` object, since line
> transformation is a very heavyweight operation, akin to other methods
> available on the `format` class (test discovery semantics and test
> execution semantics).
>> Would there be any objections against such an implementation plan?
> 
>> Regards,
>> George
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev