[LLVMdev] QEMU testing for LIT execution tests

[Jonathan Roelofs]

On 9/8/14, 3:58 PM, Jonathan Roelofs wrote:
>
>
> On 9/8/14, 3:31 PM, Greg Fitzgerald wrote:
>> Jon,
>>
>>> One issue to start with is that none of the libcxx/libcxxabi tests have RUN
>>> lines.
>>
>> Sorry, I didn't mean to imply that 'RUN:' lines should be added to
>> libcxx tests.  It's fine that every one is implicitly:
>>
>>      // RUN: %clang %s -o %t && %run %t
>>
> ... and then when an explicit 'RUN' line is provided, use that instead? That
> sounds reasonable.
>>
>>> I think that this necessitates inventing a new LIT syntax, something like '//
>>> DEPENDS-ON: filename.dat'
>>
>> If you didn't want to change lit, this would be a good place to add a
>> 'RUN:' line.  Something like:
>>
>>      // RUN: %clang %s -o %t && %run %t filename.dat
>>
>> and then your %run implementation would see the dependency on the
>> local file and ship it over to TARGET.
> This also sounds reasonable.
>>
>>
>>> A third issue is that I don't think the current '%run' solution works for
>>> full Canadian cross testing (i.e. run lit on BUILD, clang/llvm/whatever
>>> on HOST, and the tests themselves on TARGET, where {BUILD,
>>> HOST,TARGET} are all different machines).
>>
>> In compiler-rt, I'd configure such a build with the following CMake variables:
> I'm not sure we're on the same page... Your example, and the way that lit is
> currently set up, assumes BUILD and HOST are the same machine.
or, at least, it assumes that lit is running on HOST.
>
>> LLVM_LIT=$BUILD/lit
>> CMAKE_C_COMPILER=$HOST/clang
>> COMPILER_RT_EMULATOR=$TARGET/qemu
> Let me give a concrete example:
>
> BUILD=i686-pc-linux-gnu
> HOST=i386-pc-win32
> TARGET=arm-elf (on say, real hardware, or i686-pc-linux/bin/qemu-system-arm, or
> i386-pc-win32/bin/qemu-system-arm)
>
> Here I'd like to run lit on the linux machine such that test compilation happens
> on the windows machine and test execution happens on arm hardware or qemu (where
> qemu is running on either the windows machine, or the linux machine, depending
> on how the lit.site.cfg is set up).
>
>
> Jon
>>
>> For libcxx, how about adding LIBCXX_EMULATOR, and then changing
>> lit.cfg to acknowledge it when invoking executables?  You'd then put
>> your ssh functionality in a standalone script.  If a command-line
>> interface is too inflexible (or painful), perhaps add a
>> LIBCXX_PY_EMULATOR variable that points to a Python module.
>>
>> -Greg
>>
>>
>> On Mon, Sep 8, 2014 at 12:49 PM, Jonathan Roelofs
>> <jonathan at codesourcery.com> wrote:
>>>
>>>
>>> On 9/8/14, 12:39 PM, Greg Fitzgerald wrote:
>>>>
>>>> Hi Jon,
>>>>
>>>> Compiler-rt, libcxx, libcxxabi should all use the same
>>>> cross-compilation testing strategy.  Compiler-rt already has a working
>>>> solution.  If it is inadequate for libcxx or libcxxabi, can we start
>>>> by addressing those deficiencies?
>>>
>>> One issue to start with is that none of the libcxx/libcxxabi tests have RUN
>>> lines. All of the tests are either: *.fail.cpp ones that fail to compile, or
>>> *.pass.cpp ones that are expected to exit with a return status of 0 without
>>> failing an asserts. Granted, this is more of a test semantic issue, but
>>> there are a *lot* of tests to update in order to converge on compiler_rt's
>>> 'RUN: ' format.
>>>
>>> [Sidenote: I don't see any 'RUN:' lines on the builtins unit tests. How does
>>> it work for those ones?]
>>>
>>> Another issue, which Dan Albert recently pointed out to me, is that there
>>> are several libcxx tests which would require copying in of test data (in
>>> addition to test executables).  I think that this necessitates inventing a
>>> new LIT syntax, something like '// DEPENDS-ON: filename.dat' in order to
>>> inform the lit.cfg that the dependent file also needs to be copied to the
>>> remote target.
>>>
>>> A third issue is that I don't think the current '%run' solution works for
>>> full Canadian cross testing (i.e. run lit on BUILD, clang/llvm/whatever on
>>> HOST, and the tests themselves on TARGET, where {BUILD,HOST,TARGET} are all
>>> different machines).  This use case is of particular interest to me. I
>>> suppose %run could be renamed %run_target, add %run_host, and then have an
>>> implied no-op %run_build on commands that don't have %run_target or
>>> %run_host. Perhaps someone might want to have BUILD be a different machine
>>> than where LIT runs, in which case %run_build would have to be explicit (but
>>> I'm not really interested in that use case).
>>>
>>>
>>> Cheers,
>>> Jon
>>>>
>>>>
>>>> Thanks,
>>>> Greg
>>>>
>>>>
>>>
>>> --
>>> Jon Roelofs
>>> jonathan at codesourcery.com
>>> CodeSourcery / Mentor Embedded
>