[PATCH] D97869: [OpenCL][Draft] Add OpenCL builtin test generator

[Sven van Haastregt via Phabricator via cfe-commits]

svenvh added a comment.

To be honest, I don't have a concrete picture yet of how to deploy this emitter for testing.  So here are a few thoughts about the various options that come to mind:

1. Regenerate the test file every time we run check-clang and use it to validate the `-fdeclare-opencl-builtins` option.
------------------------------------------------------------------------------------------------------------------------

Advantage:

- This gives confidence that each builtin described in `OpenCLBuiltins.td` is usable in OpenCL code.

Disadvantages:

- This doesn't protect against unintended removals of builtins/overloads, as the same source of truth is used for implementation and testing.
- It is quick to generate (<1 second), but it takes some time to parse the entire test with a debug build of Clang (35 seconds).



2. As 1, but also keep maintaining `opencl-c.h` and test with both `opencl-c.h` and `-fdeclare-opencl-builtins`.
----------------------------------------------------------------------------------------------------------------

This addresses the disadvantage of option 1 at the expense of maintaining `opencl-c.h` and increased testing time.

3. Generate the test file once, manually verify that it is correct, check it in to git, and maintain it from there.
-------------------------------------------------------------------------------------------------------------------

Advantage:

- Different sources of truth for implementation (`OpenCLBuiltins.td`) and testing (the generated and maintained test).

Disadvantages:

- The output is big, currently already 56k lines, and I expect it will still get a bit bigger when we add conditionals.  I don't feel comfortable about putting such big autogenerated files under git (regardless of whether we add it as a single file, or as a collection of smaller files).  Ideally the repo should only contain the source from which we generate things.
- Updates to `OpenCLBuiltins.td` will give large diffs when regenerating the tests (regardless of whether the file is split or not).
- Manually verifying that the generated test is actually correct is not trivial.



4. Generate a condensed summary of all builtins, check that in to git.
----------------------------------------------------------------------

Then regenerate the summary during testing and diff it against the checked in reference. This builds upon @azabaznov's suggestion.  We could for example use a count for each builtin and maintain that.  We could maintain multiple counts to cover different configurations (e.g. CL versions).
Advantages:

- There are only about 1000 different builtin function names, so this reduces the size of the reference data considerably, making it easier to verify the initial checkin and making maintainance easier.
- Generating the summary and diffing it against a reference is quick (<1 second).
- It would catch the case when builtins are accidentally removed from a configuration: if the count in the summary is not updated, the test will fail.  I believe this is an important scenario to test in light of the OpenCL 3 work?

Disadvantage:

- Not giving perfect coverage, so subtle errors might slip through, such as for example changing the argument type of a builtin.

Other ideas / feedback welcome!


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D97869/new/

https://reviews.llvm.org/D97869