[llvm-dev] Running lit tests with pre-built binaries

[Kaylor, Andrew via llvm-dev]

> Do you want to try to recreate this package or do you want to try to use the Fedora package directly. 

Neither, I think. I'm just trying to understand how it works to see if that will be helpful in setting up the test environment I need.

To be specific, I'm trying to figure out how to run the SYCL lit tests with different offload hardware without doing a build. The SYCL runtime library is not part of the main LLVM repo yet, but I think it's conceptually similar enough to the libomptarget library that the same methodology should apply, but I don't see a way to do what I want with libomptarget either. If you want to see more details, the tests I want to run are here: https://github.com/intel/llvm/tree/sycl/sycl/test

I think the question I'm trying to ask is how I get the lit configuration (including the base LLVM lit configuration, which is used here) to be run without building the whole project.

Do I need to extract the commands that CMake generates for the lit configuration process and put them into a separate script?

Thanks,
Andy

-----Original Message-----
From: Tom Stellard <tstellar at redhat.com> 
Sent: Thursday, December 17, 2020 1:43 PM
To: Kaylor, Andrew <andrew.kaylor at intel.com>; llvm-dev at lists.llvm.org
Subject: Re: [llvm-dev] Running lit tests with pre-built binaries

On 12/17/20 1:28 PM, Kaylor, Andrew wrote:
> Hi Tom,
> 
> Thanks for the tip. It sounds very promising. It looks like that does essentially what I need to do. Now I just need to figure out the mechanics of how it works.
> 
> How is the package created? Do you make it directly from the test directories in the source tree or is there an install target for the tests? How is the lit configuration handled?
> 

The process is not all that straight-forward.  You can look at the Fedora spec file[1] to see exactly how it works.  At a high-level what it does is manually install (no install target)l the test directory along with a fedora specific lit config file[2].  The LLVM lit config file is patched[3] to load the fedora config.  The fedora lit config has some parameters that can be overridden at runtime, which is what the run-lit-tests script does.

Do you want to try to recreate this package or do you want to try to use the Fedora package directly.  I have never tried this, but it may work to point the tests at a non-system install just by updating the config.llvm_tools_dir in lit.fedora.cfg.py.

-Tom

[1] https://src.fedoraproject.org/rpms/llvm/blob/master/f/llvm.spec
[2] https://src.fedoraproject.org/rpms/llvm/blob/master/f/lit.fedora.cfg.py
[3]  https://src.fedoraproject.org/rpms/llvm/blob/master/f/llvm.spec#_335
> Thanks,
> Andy
> 
> -----Original Message-----
> From: Tom Stellard <tstellar at redhat.com>
> Sent: Wednesday, December 16, 2020 8:12 PM
> To: Kaylor, Andrew <andrew.kaylor at intel.com>; llvm-dev at lists.llvm.org
> Subject: Re: [llvm-dev] Running lit tests with pre-built binaries
> 
> On 12/16/20 4:44 PM, Kaylor, Andrew via llvm-dev wrote:
>> Hi everyone,
>>
>> Can anyone tell me if there is a way to run in-tree lit tests using 
>> pre-built binaries rather than building everything in the current workspace?
>>
> 
> On Fedora, we package the lit tests into an RPM that can be installed and run against the system LLVM:
> 
> dnf install llvm-test
> /usr/libexec/tests/llvm/run-lit-tests
> 
> I'm not sure if it will work for your use case, but it might be a good starting point.
> 
> -Tom
> 
>> The specific problem I’m trying to solve is testing a heterogeneous 
>> runtime library with multiple offload configurations. The host system 
>> architecture is the same for all the cases I want to test, so I’d 
>> like to be able to build the compiler once and then deploy it to 
>> different systems and run the runtime library’s LIT tests on each 
>> of those systems. Some of the systems I need to test don’t have a 
>> lot of host compute power, so I’d like to avoid rebuilding the project if possible.
>>
>> It seems like this should be possible, but I’m not sure of the best 
>> way to do it.
>>
>> I suppose I could copy the entire build directory from my build 
>> system into an appropriate location on the test system to make it 
>> look like it was built there, but that feels like a very fragile solution.
>>
>> I could also duplicate the tests into a configuration like the 
>> llvm-test-suite that is designed to be run with a pre-built compiler, 
>> but that would introduce problems with keeping the tests in synch 
>> with the source.
>>
>> Would it be reasonable to modify the runtime library’s lit config 
>> files to provide an in-tree option for them to use an external 
>> compiler and library path?
>>
>> Thanks in advance for any suggestions.
>>
>> -Andy
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>