[cfe-dev] Paragraph numbers in C++ tests

Aaron Ballman via cfe-dev cfe-dev at lists.llvm.org
Thu May 2 00:51:08 PDT 2019 

On Wed, May 1, 2019 at 4:32 PM Richard Smith <richard at metafoo.co.uk> wrote:
>
> The problems here are not restricted to changes in paragraph numbers.
> We lay out test/CXX by section structure, and that too has changed a
> lot between versions.
>
> Here are some options:
>
> # Exploded-with-symlinks
>
> Approach: rearrange test/CXX to have a subdirectory for each standard
> version, each structured as test/CXX is today. When the same test can
> be reused across standard versions, symlink it from the old location
> to the new one. Teach lit to pass the appropriate -std= flag to
> %clang_cc1 invocations in test/CXX/<std version>.
> Pro: test layout matches the standard and we can more easily see that
> we have coverage of all sections of all supported standards.
> Pro: when a new standard is added, can easily check which sections
> have been tested and easily add a copy of all current tests that will
> be run with the new -std= mode.
> Con: when editing a test, it may not be obvious that it's also used by
> other language modes.
> Con: huge cost, both up-front and when each new standard is added.
>
> # Exploded-with-copies
>
> Approach: as exploded-with-symlinks, but make copies rather than symlinks.
> Pro: same as for exploded-with-symlinks.
> Pro: easy to remove the C++98 test suite when the day comes to remove
> C++98 support.
> Con: duplication of tests; more work when adding or updating tests, or
> changing diagnostic text; more "real" files in test suite.
> Con: huge cost, both up-front and when each new standard is added.
>
> # Flat-by-section
>
> Approach: under test/CXX, have only one level of directories named
> after standard section names. Name all new paragraph-numbered tests
> with the standard version corresponding to the paragraph.
> (Alternatively: concatenate section label and paragraph number and
> remove all hierarchy.)
> Pro: easier to find the test for a particular paragraph, can avoid duplication
> Con: hard to tell which paragraphs of which standard versions are
> covered, and by what tests, based on filename
>
> # Flat-unumbered-by-section
>
> Approach: as flat-by-section, but name tests after the functionality
> they're testing rather than paragraph numbers (alternatively: just
> number the tests arbitrarily rather than trying to pick meaningful
> names).
> Pro: no paragraph number confusion
> Con: very hard to tell which paragraphs of which standard versions are
> covered, and by what tests, based on filenames
>
> # Status quo
>
> Approach: sections arranged in directory hierarchy roughly by the
> earliest standard version to contain them, file names numbered with
> paragraph numbers of roughly the earliest standard version to contain
> those paragraphs. Use standard version suffixes only to avoid file
> name collisions.
> Pro: zero up-front cost
> Con: organization is outdated by ongoing standard changes
>
> # Match the latest standard
>
> Approach: sections arranged by directory hierarchy attempting to match
> the current working draft, file names numbered with paragraph numbers
> attempting to match the latest working draft. Use version suffixes for
> tests that don't correspond to anything in the latest standard.
> Pro: matching tests against the current specification is simple; can
> more easily determine coverage for latest specification
> Con: ongoing maintenance cost keeping up with standardization
> Con: unclear how to organize and find tests for sections of the
> standard that have been removed
>
>
> Largely orthogonally to the above, we have Aaron's suggestion to
> enable coverage analysis: include comments in each test file in a
> parseable format indicating each standard version, section label, and
> paragraph covered by that test.
>
>
> Goals:
>  * obvious organizational structure: no time spent thinking about
> where to put a new test
>  * easy to find existing tests
>  * feasible to build coverage checking tool to determine how much of
> which standards we test for
>  * low up-front and ongoing maintenance cost
>
> Thoughts? Perhaps flat-by-section + Aaron's suggestion is the best
> tradeoff here? (That is: remove all hierarchy from the test tree other
> than the leaf directories, add a -c++XY suffix to all new test names,
> and include machine-readable comments in test files indicating which
> paragraphs of which standards are covered.)

+1, I think this is a reasonable approach, though I do wonder if we
will want some extra hierarchy for really large sections like [expr]
or [dcl.dcl] or not.

~Aaron



>
> On Wed, 1 May 2019 at 12:40, Aaron Ballman via cfe-dev
> <cfe-dev at lists.llvm.org> wrote:
> >
> > On Wed, May 1, 2019 at 11:58 AM Bruno Ricci via cfe-dev
> > <cfe-dev at lists.llvm.org> wrote:
> > >
> > >
> > >
> > > On 30/04/2019 17:53, David Blaikie wrote:
> > > > On Tue, Apr 30, 2019 at 8:12 AM via cfe-dev <cfe-dev at lists.llvm.org> wrote:
> > > >>
> > > >>
> > > >>
> > > >>> -----Original Message-----
> > > >>> From: cfe-dev [mailto:cfe-dev-bounces at lists.llvm.org] On Behalf Of Bruno
> > > >>> Ricci via cfe-dev
> > > >>> Sent: Tuesday, April 30, 2019 10:34 AM
> > > >>> To: cfe-dev at lists.llvm.org
> > > >>> Subject: Re: [cfe-dev] Paragraph numbers in C++ tests
> > > >>>
> > > >>>
> > > >>>
> > > >>> On 29/04/2019 19:33, David Blaikie wrote:
> > > >>>> Even if they're updated to reflect current paragraph numbers, they'd
> > > >>>> still fall into disrepair in future standards & be hard to know which
> > > >>>> version they applied to. So I'd support adding C++ version numbers to
> > > >>>> them. At least on principle - but the reality/situations they're used
> > > >>>> in might make that awkward/problematic, I don't know.
> > > >>>
> > > >>> I was thinking about the naming of tests for sections of the C++ standard.
> > > >>> For example the tests for [namespace.udecl] are:
> > > >>>   p1.cpp
> > > >>>   p3.cpp
> > > >>>   p4.cpp
> > > >>>   p5-cxx0x.cpp
> > > >>>   p6-cxx11.cpp
> > > >>>   p7.cpp
> > > >>>   p8.cpp
> > > >>>   p8-cxx0x.cpp
> > > >>>   p10.cpp
> > > >>>   p11.cpp
> > > >>>   p12.cpp
> > > >>>   p13.cpp
> > > >>>   p15.cpp
> > > >>>   p18.cpp
> > > >>>
> > > >>> I think that naming the tests pn-cxxmm.cpp to mean, "paragraph pn of
> > > >>> cxxmm"
> > > >>> is reasonable, with the understanding that naming a test p3-cxx17.cpp does
> > > >>> not
> > > >>> mean that the corresponding rule is specific to C++17, just that it was
> > > >>> written
> > > >>> against C++17.
> > > >>>
> > > >>> Bruno
> > > >>
> > > >> The real issue is, "p3" in C++11 might be renumbered "p4" in C++14
> > > >> and "p5" in C++17 and ...  so what does "p3.cpp" actually test?
> > > >> You can't tell just from the name of the file because the name's
> > > >> meaning is context-sensitive.
> > > >>
> > > >> Intuitively I'd think "p3-cxx11.cpp" means "tests p3 as it existed in
> > > >> the C++11 standard" and whether that paragraph is still p3 in later
> > > >> editions is an exercise for the reader.
> > > >
> > > > I think that^ is what Bruno was saying.
> > >
> > > Yes
> > >
> > > >
> > > >> I recommend, give up on naming tests after paragraph numbers, because
> > > >> they aren't actually meaningful.
> > > >
> > > > Eh, I think there's merit to having tests by feature in the standard
> > > > and being able to inspect their coverage (yeah, that gets wonky when
> > > > paragraphs are renumbered - so you might have to update the numbering
> > > > - but the benefit of a correspondence between some part/version of the
> > > > standard even if you have to do some work remapping to a consistent
> > > > version.
> > > >
> > >
> > > I think I agree with the coverage argument. In any case the person writting
> > > a new test has to consider the previous versions of the standard to see if
> > > the rule was changed.
> >
> > Perhaps a better approach is to use a stable name + a counter for the
> > test (stable_name_1.cpp, stable_name_2.cpp) and the file can have a
> > list of comments after the RUN lines:
> >
> > // C++11 [foo.bar]p2
> > // C++14 [foo.bar]p3
> > // whatever
> >
> > This also helps with coverage if we use a consistent convention across
> > files, I believe. It also means the file names don't have to be as
> > meaningful, and we can keep interesting historical information rather
> > than losing it when changing the name of the file.
> >
> > ~Aaron
> >
> > >
> > > Bruno
> > >
> > > >> --paulr
> > > >>
> > > >>>
> > > >>>>
> > > >>>> On Mon, Apr 29, 2019 at 7:28 AM Bruno Ricci via cfe-dev
> > > >>>> <cfe-dev at lists.llvm.org> wrote:
> > > >>>>>
> > > >>>>> Hi,
> > > >>>>>
> > > >>>>> An annoying part of writing tests for C++ is that the paragraph numbers
> > > >>> are not
> > > >>>>> stable between versions of the standard. Is there an accepted way to
> > > >>> deal with
> > > >>>>> this issue ? I can imagine several ways to deal with this:
> > > >>>>>   - Rename the tests so that they match the latest standard/draft.
> > > >>>>>   - Add a suffix/prefix with the version for the paragraph number.
> > > >>>>>   - ... ?
> > > >>>>>
> > > >>>>> Bruno
> > > >>>>> _______________________________________________
> > > >>>>> cfe-dev mailing list
> > > >>>>> cfe-dev at lists.llvm.org
> > > >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> > > >>> _______________________________________________
> > > >>> cfe-dev mailing list
> > > >>> cfe-dev at lists.llvm.org
> > > >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> > > >> _______________________________________________
> > > >> cfe-dev mailing list
> > > >> cfe-dev at lists.llvm.org
> > > >> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> > > _______________________________________________
> > > cfe-dev mailing list
> > > cfe-dev at lists.llvm.org
> > > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> > _______________________________________________
> > cfe-dev mailing list
> > cfe-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

More information about the cfe-dev mailing list