[cfe-dev] Paragraph numbers in C++ tests

[Bruno Ricci via cfe-dev]

On 02/05/2019 08:51, Aaron Ballman wrote:
> On Wed, May 1, 2019 at 4:32 PM Richard Smith <richard at metafoo.co.uk> wrote:
>>
>> The problems here are not restricted to changes in paragraph numbers.
>> We lay out test/CXX by section structure, and that too has changed a
>> lot between versions.
>>
>> Here are some options:
>>
>> # Exploded-with-symlinks
>>
>> Approach: rearrange test/CXX to have a subdirectory for each standard
>> version, each structured as test/CXX is today. When the same test can
>> be reused across standard versions, symlink it from the old location
>> to the new one. Teach lit to pass the appropriate -std= flag to
>> %clang_cc1 invocations in test/CXX/<std version>.
>> Pro: test layout matches the standard and we can more easily see that
>> we have coverage of all sections of all supported standards.
>> Pro: when a new standard is added, can easily check which sections
>> have been tested and easily add a copy of all current tests that will
>> be run with the new -std= mode.
>> Con: when editing a test, it may not be obvious that it's also used by
>> other language modes.
>> Con: huge cost, both up-front and when each new standard is added.
>>
>> # Exploded-with-copies
>>
>> Approach: as exploded-with-symlinks, but make copies rather than symlinks.
>> Pro: same as for exploded-with-symlinks.
>> Pro: easy to remove the C++98 test suite when the day comes to remove
>> C++98 support.
>> Con: duplication of tests; more work when adding or updating tests, or
>> changing diagnostic text; more "real" files in test suite.
>> Con: huge cost, both up-front and when each new standard is added.
>>
>> # Flat-by-section
>>
>> Approach: under test/CXX, have only one level of directories named
>> after standard section names. Name all new paragraph-numbered tests
>> with the standard version corresponding to the paragraph.
>> (Alternatively: concatenate section label and paragraph number and
>> remove all hierarchy.)
>> Pro: easier to find the test for a particular paragraph, can avoid duplication
>> Con: hard to tell which paragraphs of which standard versions are
>> covered, and by what tests, based on filename
>>
>> # Flat-unumbered-by-section
>>
>> Approach: as flat-by-section, but name tests after the functionality
>> they're testing rather than paragraph numbers (alternatively: just
>> number the tests arbitrarily rather than trying to pick meaningful
>> names).
>> Pro: no paragraph number confusion
>> Con: very hard to tell which paragraphs of which standard versions are
>> covered, and by what tests, based on filenames
>>
>> # Status quo
>>
>> Approach: sections arranged in directory hierarchy roughly by the
>> earliest standard version to contain them, file names numbered with
>> paragraph numbers of roughly the earliest standard version to contain
>> those paragraphs. Use standard version suffixes only to avoid file
>> name collisions.
>> Pro: zero up-front cost
>> Con: organization is outdated by ongoing standard changes
>>
>> # Match the latest standard
>>
>> Approach: sections arranged by directory hierarchy attempting to match
>> the current working draft, file names numbered with paragraph numbers
>> attempting to match the latest working draft. Use version suffixes for
>> tests that don't correspond to anything in the latest standard.
>> Pro: matching tests against the current specification is simple; can
>> more easily determine coverage for latest specification
>> Con: ongoing maintenance cost keeping up with standardization
>> Con: unclear how to organize and find tests for sections of the
>> standard that have been removed
>>
>>
>> Largely orthogonally to the above, we have Aaron's suggestion to
>> enable coverage analysis: include comments in each test file in a
>> parseable format indicating each standard version, section label, and
>> paragraph covered by that test.
>>
>>
>> Goals:
>>  * obvious organizational structure: no time spent thinking about
>> where to put a new test
>>  * easy to find existing tests
>>  * feasible to build coverage checking tool to determine how much of
>> which standards we test for
>>  * low up-front and ongoing maintenance cost
>>
>> Thoughts? Perhaps flat-by-section + Aaron's suggestion is the best
>> tradeoff here? (That is: remove all hierarchy from the test tree other
>> than the leaf directories, add a -c++XY suffix to all new test names,
>> and include machine-readable comments in test files indicating which
>> paragraphs of which standards are covered.)
> 
> +1, I think this is a reasonable approach, though I do wonder if we
> will want some extra hierarchy for really large sections like [expr]
> or [dcl.dcl] or not.
> 
> ~Aaron
> 

Exploded-with-symlinks and Exploded-with-copies are probably way too costly
to maintain. Would the Flat-by-section model imply that if a section was
renamed from X to Y, then some tests would be in X/ and some tests in Y/ ?
Additionally what should be done if a test can be written against multiple
versions ? (Pick the oldest version/the latest released version/the latest
draft)

Bruno