[libcxx-dev] RFC: freestanding libc++ lit tests

Ben Craig via libcxx-dev libcxx-dev at lists.llvm.org
Sun Nov 15 09:19:52 PST 2020

I'd like to start annotating the libc++ lit tests to indicate which can run in freestanding vs. hosted, while also allowing for vendor extensions to freestanding.  I'm going to need to add a new feature to lit though.

Mark all the hosted tests as "REQUIRES: hosted".  Most platforms would have the "hosted" feature set automatically.  Freestanding tests would not have any new REQUIRES attribute added.

To handle vendor extensions, things get a little more involved.  I would like to add a "REQUIRES_ONE" tag, so that we can put expressions like "REQUIRES_ONE: hosted, darwinKernel" and "REQUIRES_ONE: hosted, winKernel".  The expression would be satisfied so long as at least one of the listed features is present.


  *   Clearly expresses separation between tests required for wg21 compliance and tests of vendor extensions


  *   Requires new lit features
  *   Requires work on hosted tests to support freestanding (which is backwards)
  *   Only alludes to freestanding rather than mentioning it directly

Alternative 1:
Use "UNSUPPORTED" instead of "REQUIRES".  Hosted tests would all get "UNSUPPORTED:freestanding".  Vendor extensions would be handled by deny-listing all the other freestanding platforms with libc++ support.  For example, an extension in darwinKernel may say "UNSUPPORTED:winKernel,wellSupportedFreestandingPlatform2,wellSupportedFreestandingPlatform3".


  *   Doesn't require a new lit feature
  *   Mention freestanding by name, rather than alluding to it via hosted


  *   Scales poorly with the number of freestanding platforms available
  *   Requires a lot of work from new platform authors

Alternative 2:
Add "negative features" to lit.  If a lit configuration has a certain negative feature, then it will only run tests that mention that feature.  For example, freestanding configurations would have the "freestanding" feature, and would only run tests that have the "TO_BE_DETERMINED: freestanding" annotation.  lit configurations without the "freestanding" feature (i.e. hosted implementations) would still be able to run the test.  I don't see a good way to handle vendor extensions.


  *    Allow-lists just the freestanding tests


  *   Requires a new lit feature
  *   Doesn't handle vendor extensions

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/libcxx-dev/attachments/20201115/bc38e008/attachment.html>

More information about the libcxx-dev mailing list