[llvm-dev] Multiple documents in one test file

[Robinson, Paul via llvm-dev]

> -----Original Message-----
> From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Pavel Labath
> via llvm-dev
> Sent: Tuesday, July 14, 2020 2:58 AM
> To: David Blaikie <dblaikie at gmail.com>; Fangrui Song <maskray at google.com>;
> Richard Smith <richard at metafoo.co.uk>
> Cc: llvm-dev <llvm-dev at lists.llvm.org>
> Subject: Re: [llvm-dev] Multiple documents in one test file
> 
> On 14/07/2020 03:27, David Blaikie via llvm-dev wrote:
> > (+Richard - it's handy to include folks from previous discussions
> > explicitly so everyone can more easily keep track of the conversation)
> >
> > On Mon, Jul 13, 2020 at 6:17 PM Fangrui Song via llvm-dev
> > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> >
> >     Sometimes it is convenient if we can specify multiple independent
> tests
> >     in one file. To give an example, let's discuss
> >     test/MC/ELF/debug-md5.s and
> >     test/MC/ELF/debug-md5-err.s (.file directive in the assembler).
> >
> >     a) An invalid .file makes the whole file invalid. Because errors
> >     lead to a
> >     non-zero exit code, We have to use `RUN: not llvm-mc %s` for the
> >     whole file.
> >     This often lead to users placing good (`RUN: llvm-mc %s`) and bad
> >     tests (`RUN:
> >     not llvm-mc %s`) separately. For some features, having both good and
> >     bad tests
> >     in one file may improve readability.
> >     b) .debug_line is a global resource. Whenever we add a (valid)
> .file, we
> >     contribute an entry to the global resource. If we want to test some
> >     characteristics when include_directories[0] is A, and other
> >     characteristics
> >     when include_directories[0] is B, we have to use another test file.
> >
> >     The arguments apply to many other types of tests (opt on .ll, llc on
> >     .ll and .mir, clang on .c, yaml2obj on .yaml, etc).
> >
> >     I have a patch teaching llvm-mc about an option to split input:
> >     https://reviews.llvm.org/D83725
> >     (30+ lines)
> >
> >     In a comment, Richard Smith mentioned whether we can add a separate
> >     extractor utility:
> >
> >     ```
> >     # RUN: extract bb %s | llvm-mc - 2>&1 | FileCheck %s --check-
> prefix=BB
> >
> >     or
> >
> >     # RUN: extract bb %s -o %t.bb <http://t.bb>
> >     # RUN: llvm-mc %t.bb <http://t.bb> 2>&1 | FileCheck %t.bb
> <http://t.bb>
> >     ```
> >
> >
> > Could make "extract" work a bit like "tee" so it can still be one line:
> >
> > # RUN: extract bb %s -o %t.bb <http://t.bb> | llvm-mc - 2>&1 | FileCheck
> > %t.bb <http://t.bb>
> >
> > (could even make it a bit shorter for convenience - 'ex' or something)
> >
> >
> >     The advantage is its versatility. The downside is somewhat verbose
> >     syntax.
> >
> >
> >     Some questoms:
> >
> >     1. What do people think of the two approaches? An in-utility option
> >     vs a standalone utility.
> >     2. For llvm-mc, if we go with an option, is there a better name than
> >     --doc-id? David Blaikie proposed --asm-id
> >        (This is my personal preference, trading 30+ lines in a utility
> >     for simpler syntax)
> 
> FWIW, the way I've done this in llvm-mc so far is via a combination of
> "--defsym CASE<N>" command line argument and ".ifdef" asm directives.
> This has the advantage that individual "documents" don't need to be
> fully standalone (though they can be), so you can put the common parts
> of the tests into an unconditionally compiled block.
> 
> That said, I was using this technique for constructing test cases for
> other tools via llvm-mc. Things might get a bit awkward if you try to
> test .ifdef processing itself this way...
> 
> pl

+1 for --defsym.  I'm sure I've written pairs of tests that could have been
done more simply this way.

Regarding maskray's original post, which is about valid/invalid .file
directives in the same .s file, --defsym and .ifdef work perfectly:

.ifdef CASE1
.file 1 "./case1.c"
.endif
.ifdef CASE2
.file 1 "./case2.c"
.endif
nop

llvm-mc --defsym=CASE1=1 will emit a .debug_line with "case1.c"
and --defsym=CASE2=1 will emit it with "case2.c".  So this would be
ideal for verifying the MD5 cases (good and bad) in one test file.
--paulr