[libcxx-dev] How to execute cross-compiled libcxx tests on a remote target?

[Louis Dionne via libcxx-dev]

> On Jun 12, 2020, at 05:05, Dominik Montada <dominik.montada at hightec-rt.com> wrote:
> 
> Hi Louis,
> 
> thanks for the response and the commit to support this use case!
> 
> In the meantime I have implemented a feature to load an additional configuration file from the auto generated one, which solves the problem for me so far. Especially since I also need to set some additional flags (depending on the configuration) which get passed to my execution script, which seems like something which is not supported by your last commit?

Yes, you can pass e.g. -DLIBCXX_EXECUTOR="<your-remote-execution-script> --host <ip-address> --other-args...".

> 
> By the way, if you're interested I'd be happy to create patch from my extension.

Yes, I'd like you to send a Phab review with it so I can see what you've got. I also have a local patch that allows bypassing the default config file, and I was waiting to land other patches before I landed that one. Looking at yours might be useful.

> 
> While we're at it, is there a way to declare some tests as unsupported? We have a couple which we do not support on our target and would like to exclude them so they don't show up as failures at all.

You'd have to add `// UNSUPPORTED: <some-Lit-feature-defined-on-your-target>` to each unsupported test. Is that not an option? We'll be happy to take such a patch if your system is part of our test matrix, otherwise another option would be to add it to your downstream fork if you have one.

Cheers,
Louis

> 
> Cheers,
> 
> Dominik
> 
> Am 11.06.20 um 22:30 schrieb Louis Dionne:
>> BCC: LLVM Dev
>> 
>> Hi,
>> 
>>> On May 28, 2020, at 06:19, Dominik Montada <dominik.montada at hightec-rt.com> wrote:
>>> 
>>> Hi Louis,
>>> 
>>> I saw that you were the one who implemented the new testing system for libcxx, so I figured that you might be able to help me out with my problem.
>>> 
>>> I'm working on a downstream target and we're now in the process of adding support for libcxx. We would like to leverage the existing libcxx test suite to make sure that the implementation is working correctly. The problem is that our target is a bare-metal target and we need to cross-compile the tests and execute them remotely. There isn't really any documentation for how to do it, but I came across an old email thread from 2015 which said to implement an executor and use that. However, it seems like those executors are not used with the new test format any longer (except for a check whether to use `run.py` or `ssh.py`).
>>> 
>>> Also, in the original Phabricator review where you first introduced the new format, you said:
>>> 
>>>> As a side effect of this design, configuration files for the test suite can be as simple as:
>>>> 
>>>> config.substitutions.append(('%{cxx}', '<path-to-compiler>'))
>>>> config.substitutions.append(('%{compile_flags}', '<flags>'))
>>>> config.substitutions.append(('%{link_flags}', '<flags>'))
>>>> config.substitutions.append(('%{exec}', '<script-to-execute>'))
>>>> 
>>>> This should allow storing lit.cfg files for various configurations directly in the repository
>>> However I'm not sure what exactly I need to do here. Do I need to write my own `lit.site.cfg` and point CMake to it? If possible I'd like to reuse the default- and auto-generated config as much as possible.
>>> 
>>> We have a script which can execute a cross-compiled binary on our remote target. I just want the test suite to call this script. Could you give me some help on what I need to do to make this work?
>> As of 96e6cbbf941d0f937b7e823433d4c222967a1817 (committed today), you should be able to use:
>> 
>>     cmake [...] -DLIBCXX_EXECUTOR="<path-to-script>"
>> 
>> and that script will be used to run executables in the test suite. Look at how ssh.py and run.py handle their arguments for the arguments that should be accepted by your script, however it doesn't really differ much from old style executors. I'll work on documenting those in the libcxx docs.
>> 
>> Please reach out again if you run into issues with the test suite, I'll help you resolve them. I'd like everyone to move to the new format, which is simpler and more flexible.
>> 
>> Cheers,
>> Louis
>> 
>> 
> -- 
> ----------------------------------------------------------------------
> Dominik Montada                   Email: dominik.montada at hightec-rt.com
> HighTec EDV-Systeme GmbH          Phone: +49 681 92613 19
> Europaallee 19                    Fax:   +49-681-92613-26
> D-66113 Saarbrücken               WWW: http://www.hightec-rt.com
> 
> Managing Director: Vera Strothmann
> Register Court: Saarbrücken, HRB 10445, VAT ID: DE 138344222
> 
> This e-mail may contain confidential and/or privileged information. If
> you are not the intended recipient please notify the sender immediately
> and destroy this e-mail. Any unauthorised copying, disclosure or
> distribution of the material in this e-mail is strictly forbidden.
> ---
>