[cfe-dev] Paragraph numbers in C++ tests

Richard Smith via cfe-dev cfe-dev at lists.llvm.org
Wed May 1 13:31:50 PDT 2019 

The problems here are not restricted to changes in paragraph numbers.
We lay out test/CXX by section structure, and that too has changed a
lot between versions.

Here are some options:

# Exploded-with-symlinks

Approach: rearrange test/CXX to have a subdirectory for each standard
version, each structured as test/CXX is today. When the same test can
be reused across standard versions, symlink it from the old location
to the new one. Teach lit to pass the appropriate -std= flag to
%clang_cc1 invocations in test/CXX/<std version>.
Pro: test layout matches the standard and we can more easily see that
we have coverage of all sections of all supported standards.
Pro: when a new standard is added, can easily check which sections
have been tested and easily add a copy of all current tests that will
be run with the new -std= mode.
Con: when editing a test, it may not be obvious that it's also used by
other language modes.
Con: huge cost, both up-front and when each new standard is added.

# Exploded-with-copies

Approach: as exploded-with-symlinks, but make copies rather than symlinks.
Pro: same as for exploded-with-symlinks.
Pro: easy to remove the C++98 test suite when the day comes to remove
C++98 support.
Con: duplication of tests; more work when adding or updating tests, or
changing diagnostic text; more "real" files in test suite.
Con: huge cost, both up-front and when each new standard is added.

# Flat-by-section

Approach: under test/CXX, have only one level of directories named
after standard section names. Name all new paragraph-numbered tests
with the standard version corresponding to the paragraph.
(Alternatively: concatenate section label and paragraph number and
remove all hierarchy.)
Pro: easier to find the test for a particular paragraph, can avoid duplication
Con: hard to tell which paragraphs of which standard versions are
covered, and by what tests, based on filename

# Flat-unumbered-by-section

Approach: as flat-by-section, but name tests after the functionality
they're testing rather than paragraph numbers (alternatively: just
number the tests arbitrarily rather than trying to pick meaningful
names).
Pro: no paragraph number confusion
Con: very hard to tell which paragraphs of which standard versions are
covered, and by what tests, based on filenames

# Status quo

Approach: sections arranged in directory hierarchy roughly by the
earliest standard version to contain them, file names numbered with
paragraph numbers of roughly the earliest standard version to contain
those paragraphs. Use standard version suffixes only to avoid file
name collisions.
Pro: zero up-front cost
Con: organization is outdated by ongoing standard changes

# Match the latest standard

Approach: sections arranged by directory hierarchy attempting to match
the current working draft, file names numbered with paragraph numbers
attempting to match the latest working draft. Use version suffixes for
tests that don't correspond to anything in the latest standard.
Pro: matching tests against the current specification is simple; can
more easily determine coverage for latest specification
Con: ongoing maintenance cost keeping up with standardization
Con: unclear how to organize and find tests for sections of the
standard that have been removed


Largely orthogonally to the above, we have Aaron's suggestion to
enable coverage analysis: include comments in each test file in a
parseable format indicating each standard version, section label, and
paragraph covered by that test.


Goals:
 * obvious organizational structure: no time spent thinking about
where to put a new test
 * easy to find existing tests
 * feasible to build coverage checking tool to determine how much of
which standards we test for
 * low up-front and ongoing maintenance cost

Thoughts? Perhaps flat-by-section + Aaron's suggestion is the best
tradeoff here? (That is: remove all hierarchy from the test tree other
than the leaf directories, add a -c++XY suffix to all new test names,
and include machine-readable comments in test files indicating which
paragraphs of which standards are covered.)

On Wed, 1 May 2019 at 12:40, Aaron Ballman via cfe-dev
<cfe-dev at lists.llvm.org> wrote:
>
> On Wed, May 1, 2019 at 11:58 AM Bruno Ricci via cfe-dev
> <cfe-dev at lists.llvm.org> wrote:
> >
> >
> >
> > On 30/04/2019 17:53, David Blaikie wrote:
> > > On Tue, Apr 30, 2019 at 8:12 AM via cfe-dev <cfe-dev at lists.llvm.org> wrote:
> > >>
> > >>
> > >>
> > >>> -----Original Message-----
> > >>> From: cfe-dev [mailto:cfe-dev-bounces at lists.llvm.org] On Behalf Of Bruno
> > >>> Ricci via cfe-dev
> > >>> Sent: Tuesday, April 30, 2019 10:34 AM
> > >>> To: cfe-dev at lists.llvm.org
> > >>> Subject: Re: [cfe-dev] Paragraph numbers in C++ tests
> > >>>
> > >>>
> > >>>
> > >>> On 29/04/2019 19:33, David Blaikie wrote:
> > >>>> Even if they're updated to reflect current paragraph numbers, they'd
> > >>>> still fall into disrepair in future standards & be hard to know which
> > >>>> version they applied to. So I'd support adding C++ version numbers to
> > >>>> them. At least on principle - but the reality/situations they're used
> > >>>> in might make that awkward/problematic, I don't know.
> > >>>
> > >>> I was thinking about the naming of tests for sections of the C++ standard.
> > >>> For example the tests for [namespace.udecl] are:
> > >>>   p1.cpp
> > >>>   p3.cpp
> > >>>   p4.cpp
> > >>>   p5-cxx0x.cpp
> > >>>   p6-cxx11.cpp
> > >>>   p7.cpp
> > >>>   p8.cpp
> > >>>   p8-cxx0x.cpp
> > >>>   p10.cpp
> > >>>   p11.cpp
> > >>>   p12.cpp
> > >>>   p13.cpp
> > >>>   p15.cpp
> > >>>   p18.cpp
> > >>>
> > >>> I think that naming the tests pn-cxxmm.cpp to mean, "paragraph pn of
> > >>> cxxmm"
> > >>> is reasonable, with the understanding that naming a test p3-cxx17.cpp does
> > >>> not
> > >>> mean that the corresponding rule is specific to C++17, just that it was
> > >>> written
> > >>> against C++17.
> > >>>
> > >>> Bruno
> > >>
> > >> The real issue is, "p3" in C++11 might be renumbered "p4" in C++14
> > >> and "p5" in C++17 and ...  so what does "p3.cpp" actually test?
> > >> You can't tell just from the name of the file because the name's
> > >> meaning is context-sensitive.
> > >>
> > >> Intuitively I'd think "p3-cxx11.cpp" means "tests p3 as it existed in
> > >> the C++11 standard" and whether that paragraph is still p3 in later
> > >> editions is an exercise for the reader.
> > >
> > > I think that^ is what Bruno was saying.
> >
> > Yes
> >
> > >
> > >> I recommend, give up on naming tests after paragraph numbers, because
> > >> they aren't actually meaningful.
> > >
> > > Eh, I think there's merit to having tests by feature in the standard
> > > and being able to inspect their coverage (yeah, that gets wonky when
> > > paragraphs are renumbered - so you might have to update the numbering
> > > - but the benefit of a correspondence between some part/version of the
> > > standard even if you have to do some work remapping to a consistent
> > > version.
> > >
> >
> > I think I agree with the coverage argument. In any case the person writting
> > a new test has to consider the previous versions of the standard to see if
> > the rule was changed.
>
> Perhaps a better approach is to use a stable name + a counter for the
> test (stable_name_1.cpp, stable_name_2.cpp) and the file can have a
> list of comments after the RUN lines:
>
> // C++11 [foo.bar]p2
> // C++14 [foo.bar]p3
> // whatever
>
> This also helps with coverage if we use a consistent convention across
> files, I believe. It also means the file names don't have to be as
> meaningful, and we can keep interesting historical information rather
> than losing it when changing the name of the file.
>
> ~Aaron
>
> >
> > Bruno
> >
> > >> --paulr
> > >>
> > >>>
> > >>>>
> > >>>> On Mon, Apr 29, 2019 at 7:28 AM Bruno Ricci via cfe-dev
> > >>>> <cfe-dev at lists.llvm.org> wrote:
> > >>>>>
> > >>>>> Hi,
> > >>>>>
> > >>>>> An annoying part of writing tests for C++ is that the paragraph numbers
> > >>> are not
> > >>>>> stable between versions of the standard. Is there an accepted way to
> > >>> deal with
> > >>>>> this issue ? I can imagine several ways to deal with this:
> > >>>>>   - Rename the tests so that they match the latest standard/draft.
> > >>>>>   - Add a suffix/prefix with the version for the paragraph number.
> > >>>>>   - ... ?
> > >>>>>
> > >>>>> Bruno
> > >>>>> _______________________________________________
> > >>>>> cfe-dev mailing list
> > >>>>> cfe-dev at lists.llvm.org
> > >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> > >>> _______________________________________________
> > >>> cfe-dev mailing list
> > >>> cfe-dev at lists.llvm.org
> > >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> > >> _______________________________________________
> > >> cfe-dev mailing list
> > >> cfe-dev at lists.llvm.org
> > >> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> > _______________________________________________
> > cfe-dev mailing list
> > cfe-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

More information about the cfe-dev mailing list