[llvm-dev] Multiple documents in one test file

[Fangrui Song via llvm-dev]

On 2020-07-14, Robinson, Paul wrote:
>
>
>> -----Original Message-----
>> From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Pavel Labath
>> via llvm-dev
>> Sent: Tuesday, July 14, 2020 2:58 AM
>> To: David Blaikie <dblaikie at gmail.com>; Fangrui Song <maskray at google.com>;
>> Richard Smith <richard at metafoo.co.uk>
>> Cc: llvm-dev <llvm-dev at lists.llvm.org>
>> Subject: Re: [llvm-dev] Multiple documents in one test file
>>
>> On 14/07/2020 03:27, David Blaikie via llvm-dev wrote:
>> > (+Richard - it's handy to include folks from previous discussions
>> > explicitly so everyone can more easily keep track of the conversation)
>> >
>> > On Mon, Jul 13, 2020 at 6:17 PM Fangrui Song via llvm-dev
>> > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> >
>> >     Sometimes it is convenient if we can specify multiple independent
>> tests
>> >     in one file. To give an example, let's discuss
>> >     test/MC/ELF/debug-md5.s and
>> >     test/MC/ELF/debug-md5-err.s (.file directive in the assembler).
>> >
>> >     a) An invalid .file makes the whole file invalid. Because errors
>> >     lead to a
>> >     non-zero exit code, We have to use `RUN: not llvm-mc %s` for the
>> >     whole file.
>> >     This often lead to users placing good (`RUN: llvm-mc %s`) and bad
>> >     tests (`RUN:
>> >     not llvm-mc %s`) separately. For some features, having both good and
>> >     bad tests
>> >     in one file may improve readability.
>> >     b) .debug_line is a global resource. Whenever we add a (valid)
>> .file, we
>> >     contribute an entry to the global resource. If we want to test some
>> >     characteristics when include_directories[0] is A, and other
>> >     characteristics
>> >     when include_directories[0] is B, we have to use another test file.
>> >
>> >     The arguments apply to many other types of tests (opt on .ll, llc on
>> >     .ll and .mir, clang on .c, yaml2obj on .yaml, etc).
>> >
>> >     I have a patch teaching llvm-mc about an option to split input:
>> >     https://reviews.llvm.org/D83725
>> >     (30+ lines)
>> >
>> >     In a comment, Richard Smith mentioned whether we can add a separate
>> >     extractor utility:
>> >
>> >     ```
>> >     # RUN: extract bb %s | llvm-mc - 2>&1 | FileCheck %s --check-
>> prefix=BB
>> >
>> >     or
>> >
>> >     # RUN: extract bb %s -o %t.bb <http://t.bb>
>> >     # RUN: llvm-mc %t.bb <http://t.bb> 2>&1 | FileCheck %t.bb
>> <http://t.bb>
>> >     ```
>> >
>> >
>> > Could make "extract" work a bit like "tee" so it can still be one line:
>> >
>> > # RUN: extract bb %s -o %t.bb <http://t.bb> | llvm-mc - 2>&1 | FileCheck
>> > %t.bb <http://t.bb>
>> >
>> > (could even make it a bit shorter for convenience - 'ex' or something)
>> >
>> >
>> >     The advantage is its versatility. The downside is somewhat verbose
>> >     syntax.
>> >
>> >
>> >     Some questoms:
>> >
>> >     1. What do people think of the two approaches? An in-utility option
>> >     vs a standalone utility.
>> >     2. For llvm-mc, if we go with an option, is there a better name than
>> >     --doc-id? David Blaikie proposed --asm-id
>> >        (This is my personal preference, trading 30+ lines in a utility
>> >     for simpler syntax)
>>
>> FWIW, the way I've done this in llvm-mc so far is via a combination of
>> "--defsym CASE<N>" command line argument and ".ifdef" asm directives.
>> This has the advantage that individual "documents" don't need to be
>> fully standalone (though they can be), so you can put the common parts
>> of the tests into an unconditionally compiled block.
>>
>> That said, I was using this technique for constructing test cases for
>> other tools via llvm-mc. Things might get a bit awkward if you try to
>> test .ifdef processing itself this way...
>>
>> pl
>
>+1 for --defsym.  I'm sure I've written pairs of tests that could have been
>done more simply this way.
>
>Regarding maskray's original post, which is about valid/invalid .file
>directives in the same .s file, --defsym and .ifdef work perfectly:
>
>.ifdef CASE1
>.file 1 "./case1.c"
>.endif
>.ifdef CASE2
>.file 1 "./case2.c"
>.endif
>nop
>
>llvm-mc --defsym=CASE1=1 will emit a .debug_line with "case1.c"
>and --defsym=CASE2=1 will emit it with "case2.c".  So this would be
>ideal for verifying the MD5 cases (good and bad) in one test file.
>--paulr
>

My bad memory that I should have considered .ifdef (I just saw it last
week https://sourceware.org/pipermail/binutils/2020-July/112193.html ).
The syntax appears to be good enoguh. An advantage other than being a
built-in feature is that line numbers are retained.



A note about the proposed external tool 'extract': we probably should
insert '\n' to retain the original line numbers, so that the following
can work:

```
#--- aa
[[#@LINE+1]]: error: .....
.....
```