[cfe-dev] [RFC][OpenCL] Testing all OpenCL builtin function declarations in clang

Tue Apr 6 10:23:37 PDT 2021

We would like to ask for feedback regarding testing strategies for the large
amount of OpenCL builtin function declarations.

==========
Background:
==========

The OpenCL standard has many builtin functions [1] implicitly declared during
the kernel compilation. The regular opencl-c.h header in clang containing all
declarations is nearly 17K lines and it is therefore very expensive in terms of
compilation time to include. This has caused issues with its testing because
clang-check testing time is affected. This is one of the main reasons it has
been tested very sparsely (~150 lines of tests) so far in llvm-project. The full
functionality is being tested outside of llvm-project in various vendor and
open-source tooling projects. However, it doesn’t help to detect issues during
development cycles in llvm-project.

In the meantime we have developed a new mechanism to implicitly include the
function declarations leveraging Tablegen[2][3][4] that addressed parsing speed
and other limitations. However, the testing has still remained sparse due to the
time considerations. This proposal is addressing testing limitations of OpenCL
builtin functions declared via Tablegen approach and it is based on the detailed
discussions in [5]. Below we present a summary of some options that we have
found promising and invite for feedback to help us define the directions.

================
Testing Alternatives:
================

(1) Standard testing via clang invocation and use of -verify.

This is a regular straightforward format of testing that aims at checking that
headers can be parsed for various configurations. Similar tests are already part
of clang test suite, e.g. Headers/arm-neon-header.c or
Headers/x86-intrinsics-headers.c.

In contrast for our case we would like to call each overload of builtin functions to
verify the right functionality is exposed through the header. The tests could be
written manually from scratch or auto-generated initially and then manually
evolved/maintained. It is expected that the test file will be quite large - the
auto-generated version currently has about 60K lines (see details in [5]). For
comparison, the largest test file in clang is currently around 20K lines. It is
possible to partition into multiple test files but it doesn’t change the amount of
testing that has to be performed.

Advantages

- Straightforward and easy to understand.
- Fully covers clang functionality including interplay between frontend and the
headers.

Disadvantages

- A lot of extra testing to maintain.
- Clang testing time increase (see detailed measurements in [7]).

(2) Combined approach: (a) auto-generate and run tests (b) diff summaries.
This is similar to (1) but instead of committing the large test file into the
repository it can be auto-generated at each run of check-clang target. After
test generation from Tablegen invocation, the test will be run regularly
through clang.

Since both the function declaration and tests are generated from the same
OpenCLBuiltins.td definition and therefore sensitive to errors in the original
Tablegen description file, we suggest to complement this testing with checking
in some statistics about each function that can be compared to the output
summary also produced from the Tablegen definitions. This brings the additional
benefit that not every configuration has to be tested with the full test:
instead, different configurations can be tested using statistics, which would
bring down the overall testing time.

Advantages

- No large checked-in files.
- Easy to modify and maintain.
- Should cover most scenarios where interplay between the frontend and builtin
headers causes issues for OpenCL users.

Disadvantages

- Not as straightforward as just checking in a large test file.
- Not giving perfect coverage as there is no way of checking that the Tablegen
definitions are correct, i.e. subtle errors might slip through, such as for
example changing the argument type of a builtin.
- Clang testing time increases, but less than in (1), because we can rely on the
generated statistics to reduce the amount of tested configurations.

(3) An alternative approach to avoid parsing large test files and therefore
negatively impacting the testing time is to use the existing opencl-c.h header
to diff against an auto-generated header that can be produced from the Tablegen
definition. In this approach the contents of OpenCLBuiltins.td is compared to
the contents of opencl-c.h by diffing the headers in a pretty-printed format.
Proof of concept is available in [6].

When this test is deployed, both opencl-c.h and OpenCLBuiltins.td will be
incrementally modified keeping the functionality in alignment.

Advantages

- No large test files to be checked in or run.
- Diffing is very fast compared to parsing. See details in [8].

Disadvantages

- No coverage through clang invocation. Note that we already test elsewhere
that the basic parsing of the headers succeeds.
- Requires maintaining builtin declarations in two places: the Tablegen definitions
and opencl-c.h.

=======
Actions:
=======

We would like to get feedback from the community regarding whether the
exhaustive testing of the builtins is something that we should try to avoid
(see item (1)). And if so what would be a preferable approach to the testing:
(2) or (3).

Overall, it would also be possible to look into more hybrid approaches
combining the above alternatives. But it would add extra complexity into
testing that we would like to avoid unless it’s necessary.

If there are similar situations in other areas it would be good to understand
whether we could look into more generalized solutions to such testing in clang
or even wider in llvm-project.

=========
References:
=========

[1] www.khronos.org/registry/OpenCL/specs/3.0-unified/html/OpenCL_C.html#built-in-functions
[2] llvm.org/devmtg/2019-10/talk-abstracts.html#lit5
[3] clang.llvm.org/docs/OpenCLSupport.html#opencl-builtins
[4] lists.llvm.org/pipermail/cfe-dev/2021-February/067610.html
[5] reviews.llvm.org/D97869
[6] reviews.llvm.org/D99577
[7] reviews.llvm.org/D97869#2648821
[8] reviews.llvm.org/D99577#2663201
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20210406/f5430c83/attachment-0001.html>