[PATCH] D124434: [Clang][Test] Run tests in C++14 mode explicitly.

[Aaron Ballman via Phabricator via cfe-commits]

aaron.ballman added a comment.

In D124434#3475006 <https://reviews.llvm.org/D124434#3475006>, @dblaikie wrote:

> @rsmith @aaron.ballman - might be especially interesting to know your thoughts on the C++ chapter-based testing and what the intent there is as clang changes default versions/new versions are added. (& also whether there's an approach (like moving the default language into the driver) that'd be worthwhile to lower the maintenance cost of these sort of migrations without compromising on test coverage/ending up with all the test coverage testing previous versions of the language and not what everyone's using in the wild)

In general, my concern with the this patch is that it loses test coverage by specifying an explicit language mode. We typically prefer to fix the tests so that they can work in any language mode (and perhaps add additional RUN lines in the process to do so).

However, one thought I've had a few times in recent history is that I'd like to see an adjustment to the way lit tests are run. Ultimately, what I'd like to see is that lit (perhaps as an opt-in option) have the ability to test a single test file under every language mode the test file is valid for. Then we can modify the RUN lines of tests with a lit %magic_variable to specify what those language modes are. My thinking is that lit would take note of the use of the magic variable (or absence of any -std mode) and automatically generate a RUN line for each standard mode based on the template using the magic variable. This would give us the flexibility to write tests that work for any given language mode without losing test coverage.

e.g.,

  // This gets run under C++14 and up, but not earlier versions
  // RUN: %clang_cc1 -fsyntax-only -std=%cxx14+ foo.cpp
  
  // This gets run under C++14 through C++20, but not earlier or later versions
  // RUN: %clang_cc1 -fsyntax-only -std=%cxx14-%cxx20 foo.cpp
  
  // This gets run under all C++ language modes (based on the file extension) because no -std was specified
  // RUN: %clang_cc1 -fsyntax-only foo.cpp
  
  // This gets run only in C++11 mode, as it is today
  // RUN: %clang_cc1 -fsyntax-only -std=c++11 foo.cpp

If the lit behavior is opt-in and the user doesn't opt into it, then we can replace any lit magic variable with the latest standards mode in the given range or the current default language mode as appropriate.

I think this sort of flexibility would be really helpful for many of our tests and I think it would help shake out bugs in Clang that result from specifying an explicit language mode. However, I have no idea if others think this is a good idea, whether it's possible/easy to do in lit, etc.

As for the chapter-based testing... I am not convinced that the experiment has panned out well. 1) it only works for C++ where there are stable names, but it's not usable for C where everything is based on clause numbers that change frequently, 2) even in C++ it falls down somewhat regularly because we name the test files after a paragraph number, and p3 in C++N may be p2 or p4 in C++N+1, 3) we don't actively encourage people to add tests to the chapters (maybe that's my fault as a reviewer and I should encourage that more strongly?). So I'm not certain I'd be sad if we distributed the chapter tests out amongst the other test directories or froze the chapter tests in amber for a specific language version.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D124434/new/

https://reviews.llvm.org/D124434