[llvm-dev] Help with RISCV and QEMU in llvm testsuite lit testing

[Sam Elliott via llvm-dev]

Hi!

At lowRISC, we are running the llvm test suite for RISC-V using qemu.

I note two major differences between how we run our test-suite builds, and how you are running yours.

The first is that instead of running cmake and then lit, we use lnt, which will run cmake and then lit for us. The advantage of LNT is that it reports when compilation fails as well, in a unified format. 

The other difference is that we don’t use `TEST_SUITE_USE_PERF`. I don’t know how this interacts with the use of qemu. I would suggest starting by turning this off, and seeing if that fixes your problem. At that point you’ll know whether the problem lies in the timeit.sh script, or somewhere else.

Micheal Kruse has pointed to where the “.out” files come from. To dig a little deeper: the build system creates a “.test" file for lit. This is then modified by some of the modules in litsupport/modules (the list of modules that modify the tests are in the `LIT_MODULES` cmake variable). The important one is `timeit.py`, which wraps the executable invocation (excluding cd) in an invocation of `timeit` which records how long the script takes etc. Timeit is responsible for redirecting output and recording exit status. 

You are correct that the value of `config.traditional_output` is set in a `lit.local.cfg` in `SingleSource`.	 This should be being enabled, but maybe this is being skipped? It might be worth trying to print from this cfg file (it’s just python) and seeing if anything gets printed or not.

I don’t believe that `TEST_SUITE_USER_MODE_EMULATION` changes what is compared with what. The output is always saved to a file and that file is compared with a reference output file. The difference is just that we need to build `fpcmp` so that it can be executed on the host, rather than from inside qemu (the target).

I hope this helps you to debug your issue. As I said, a good first step is to turn off using PERF, and go from there.

Sam

> On 27 Sep 2019, at 7:41 pm, Mohamed Shafi via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> On Thu, 26 Sep 2019 at 23:02, Michael Kruse <llvmdev at meinersbur.de> wrote:
>> 
>> Am Mi., 25. Sept. 2019 um 17:04 Uhr schrieb Mohamed Shafi via llvm-dev
>> <llvm-dev at lists.llvm.org>:
>>> I am not sure who is responsible for the *.test.out file. I am assuming that this should be generated by QEMU. But I see that there is "exit 0" in the reference files that are not produced in the QEMU output. Should qemu be run via a wrapper that captures the output and "exit 0" in a *.test.out file ?
>> 
>> The "exit 0" should be produced by the tools/timeit.sh (since you are
>> using  -DTEST_SUITE_USE_PERF=ON) and has a cmdline switch
>> --append-exitstatus. The command line for it is produced by
>> litsupport/modules/timeit.py. No idea why it does not work in your
>> case, but I hope this help for looking into the issue.
>> 
>> Michael
> 
> Thanks for the response. What you have pointed out is correct. Since i
> have enabled TEST_SUITE_USER_MODE_EMULATION the comparison with the
> reference file is done with a *.out file rather than with the
> simulator output directly. The appropriate out file with the simulator
> output and the exit code will get generated if timeit.sh is invoked
> with --append-exitstatus and --redirect-output options. timeit.sh gets
> invoked by the timeit.py file with these options when
> config.traditional_output is true. Interestingly lit.local.cfg in
> test-suite\SingleSource has this enabled. But for some reason which i
> still haven't found out, timeit.sh is invoked as though
> config.traditional_output is false. From what i see this issue should
> crop up when RISCV target (or a QEMU target for that matter) is tested
> with llvm test suite. It would be of great help if some one could
> point out what i am doing wrong.
> 
> Regards,
> Shafi
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

--
Sam Elliott
Software Developer - LLVM
lowRISC CIC
--