[cfe-dev] Adding SYCL tests in test-suite

[Kaylor, Andrew via cfe-dev]

Thanks, Johannes!

It would be nice to have some additional infrastructure to control execution of tests that have special resource requirements like this. We've seen some problems in our internal testing with parallel test execution causing system gridlock. Having a common way to address that would be great.

One reason I thought separate SYCL folders (either at the top level or elsewhere in the tree) would be useful is that I think we'll probably want a single option to turn these tests on or off as a group. A "parallel" folder may make sense for the same reason. I see your point about mixed languages, but perhaps we could add a "parallel/mixed" subfolder if/when such a test is added.

Any other opinions?

-Andy


-----Original Message-----
From: Johannes Doerfert <johannesdoerfert at gmail.com> 
Sent: Wednesday, April 15, 2020 11:45 PM
To: Kaylor, Andrew <andrew.kaylor at intel.com>
Cc: LLVM Developers <llvm-dev at lists.llvm.org>; cfe-dev at lists.llvm.org
Subject: Re: [cfe-dev] Adding SYCL tests in test-suite


Hi Andy,

On 4/15/20 7:17 PM, Kaylor, Andrew via cfe-dev wrote:
 > We'd like to some SYCL tests to LLVM's test suite. The SYCL support in  > the LLVM repo is still very much a work-in-progress, but since the  > test-suite is supposed to be able to support compilers other than  > clang, I thought it would be reasonable to start adding the tests  > there now, disabled by default, rather than maintaining a fork of the  > test-suite repo until SYCL support is fully in place in the main repo.

I think adding such tests is generally a good idea and I do think we could have them in the test suite while SYCL support in clang is still maturing.


 > These tests would involve compiling one or more source files  > containing SYCL kernels and then executing the resulting binaries  > using either host, CPU, or accelerator devices, based on the user's  > configuration. Running these tests in some configurations would  > require an OpenCL runtime and, of course, the chosen hardware.

Now this is where the dicey part begins. The test suite, as of now, is not well equipped for tests "competing for shared resources". I mean, parallel tests or tests running on accelerators. To be fair, I am assuming we don't want the test suite to be run completely sequential if the SYCL parts are run, potentially with the non-SYCL parts. I'm saying this because OpenMP is in a similar position right now.

Our goal is to run OpenMP tests as part of the test suite but that requires infrastructure we only started to build (downstream so far). I think the goals of basically all parallel (and offloading) extensions are pretty much the same so we should work together on them.

 > Am I over-reaching?

No.


 > If not, I'd like some feedback on where the tests should go. For the  > most part, I think these will be correctness tests, though I expect  > we'll want to add benchmarks at some point. For the correctness tests,  > I thought it would make sense to either (a) create a top-level SYCL  > folder with SingleSource and MultiSource folders under it, or (b)  > create SYCL folders in appropriate locations under the existing  > SingleSource and MultiSource folders (e.g.
 > llvm-test-suite/SingleSource/UnitTests/SYCL). I would favor (b) but I  > wasn't sure if SYCL is enough of a departure from the normal C/C++  > tests to push it into its own location.

Option (c) would be an parallel or accelerator subfolder in which we nest things. Howe we nest things is again the question you asked. While I can see the appeal of a (almost) top-level SYCL folder it also has drawbacks, mostly when it comes to these really cool applications that come in 2+ different languages. Keeping these sources together as they are seems pretty important. In that spirit, I am unsure why we should do any of the options (a-c) at all. We could just put a test, let's say XYZ, into the existing structure SingleSource/Benchmarks/XYZ regardless of the language XYZ is written in. We mixed C and C++, why stop there.

To be honest, I don't really care too much and the above is mostly to start a discussion on the pros and cons of the different options.

 > I haven't added anything to test-suite before, so if I'm approaching  > this in completely the wrong way don't be shy about telling me so.

I think writing an email like this is exactly the right way :)

Cheers,
   Johannes


 > Thanks,
 > Andy
 >
 >
 >
 > _______________________________________________
 > cfe-dev mailing list
 > cfe-dev at lists.llvm.org
 > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev