[cfe-dev] Adding SYCL tests in test-suite

[Johannes Doerfert via cfe-dev]

Hi Andy,

On 4/15/20 7:17 PM, Kaylor, Andrew via cfe-dev wrote:
 > We'd like to some SYCL tests to LLVM's test suite. The SYCL support in
 > the LLVM repo is still very much a work-in-progress, but since the
 > test-suite is supposed to be able to support compilers other than
 > clang, I thought it would be reasonable to start adding the tests
 > there now, disabled by default, rather than maintaining a fork of the
 > test-suite repo until SYCL support is fully in place in the main repo.

I think adding such tests is generally a good idea and I do think we
could have them in the test suite while SYCL support in clang is still
maturing.


 > These tests would involve compiling one or more source files
 > containing SYCL kernels and then executing the resulting binaries
 > using either host, CPU, or accelerator devices, based on the user's
 > configuration. Running these tests in some configurations would
 > require an OpenCL runtime and, of course, the chosen hardware.

Now this is where the dicey part begins. The test suite, as of now, is
not well equipped for tests "competing for shared resources". I mean,
parallel tests or tests running on accelerators. To be fair, I am
assuming we don't want the test suite to be run completely sequential if
the SYCL parts are run, potentially with the non-SYCL parts. I'm saying
this because OpenMP is in a similar position right now.

Our goal is to run OpenMP tests as part of the test suite but that
requires infrastructure we only started to build (downstream so far). I
think the goals of basically all parallel (and offloading) extensions
are pretty much the same so we should work together on them.


 > Am I over-reaching?

No.


 > If not, I'd like some feedback on where the tests should go. For the
 > most part, I think these will be correctness tests, though I expect
 > we'll want to add benchmarks at some point. For the correctness tests,
 > I thought it would make sense to either (a) create a top-level SYCL
 > folder with SingleSource and MultiSource folders under it, or (b)
 > create SYCL folders in appropriate locations under the existing
 > SingleSource and MultiSource folders (e.g.
 > llvm-test-suite/SingleSource/UnitTests/SYCL). I would favor (b) but I
 > wasn't sure if SYCL is enough of a departure from the normal C/C++
 > tests to push it into its own location.

Option (c) would be an parallel or accelerator subfolder in which we
nest things. Howe we nest things is again the question you asked. While
I can see the appeal of a (almost) top-level SYCL folder it also has
drawbacks, mostly when it comes to these really cool applications that
come in 2+ different languages. Keeping these sources together as they
are seems pretty important. In that spirit, I am unsure why we should do
any of the options (a-c) at all. We could just put a test, let's say
XYZ, into the existing structure SingleSource/Benchmarks/XYZ regardless
of the language XYZ is written in. We mixed C and C++, why stop there.

To be honest, I don't really care too much and the above is mostly to
start a discussion on the pros and cons of the different options.

 > I haven't added anything to test-suite before, so if I'm approaching
 > this in completely the wrong way don't be shy about telling me so.

I think writing an email like this is exactly the right way :)

Cheers,
   Johannes


 > Thanks,
 > Andy
 >
 >
 >
 > _______________________________________________
 > cfe-dev mailing list
 > cfe-dev at lists.llvm.org
 > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev