[libcxx-dev] RFC: freestanding libc++ lit tests

[Ben Craig via libcxx-dev]

I'd like to start annotating the libc++ lit tests to indicate which can run in freestanding vs. hosted, while also allowing for vendor extensions to freestanding.  I'm going to need to add a new feature to lit though.

Recommendation:
Mark all the hosted tests as "REQUIRES: hosted".  Most platforms would have the "hosted" feature set automatically.  Freestanding tests would not have any new REQUIRES attribute added.

To handle vendor extensions, things get a little more involved.  I would like to add a "REQUIRES_ONE" tag, so that we can put expressions like "REQUIRES_ONE: hosted, darwinKernel" and "REQUIRES_ONE: hosted, winKernel".  The expression would be satisfied so long as at least one of the listed features is present.

Pros:

  *   Clearly expresses separation between tests required for wg21 compliance and tests of vendor extensions

Cons:

  *   Requires new lit features
  *   Requires work on hosted tests to support freestanding (which is backwards)
  *   Only alludes to freestanding rather than mentioning it directly

Alternative 1:
Use "UNSUPPORTED" instead of "REQUIRES".  Hosted tests would all get "UNSUPPORTED:freestanding".  Vendor extensions would be handled by deny-listing all the other freestanding platforms with libc++ support.  For example, an extension in darwinKernel may say "UNSUPPORTED:winKernel,wellSupportedFreestandingPlatform2,wellSupportedFreestandingPlatform3".

Pros:

  *   Doesn't require a new lit feature
  *   Mention freestanding by name, rather than alluding to it via hosted

Cons:

  *   Scales poorly with the number of freestanding platforms available
  *   Requires a lot of work from new platform authors

Alternative 2:
Add "negative features" to lit.  If a lit configuration has a certain negative feature, then it will only run tests that mention that feature.  For example, freestanding configurations would have the "freestanding" feature, and would only run tests that have the "TO_BE_DETERMINED: freestanding" annotation.  lit configurations without the "freestanding" feature (i.e. hosted implementations) would still be able to run the test.  I don't see a good way to handle vendor extensions.

Pros:

  *    Allow-lists just the freestanding tests

Cons:

  *   Requires a new lit feature
  *   Doesn't handle vendor extensions

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/libcxx-dev/attachments/20201115/bc38e008/attachment.html>