[llvm-dev] Multiple documents in one test file

[David Blaikie via llvm-dev]

(+Richard - it's handy to include folks from previous discussions
explicitly so everyone can more easily keep track of the conversation)

On Mon, Jul 13, 2020 at 6:17 PM Fangrui Song via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> Sometimes it is convenient if we can specify multiple independent tests
> in one file. To give an example, let's discuss test/MC/ELF/debug-md5.s and
> test/MC/ELF/debug-md5-err.s (.file directive in the assembler).
>
> a) An invalid .file makes the whole file invalid. Because errors lead to a
> non-zero exit code, We have to use `RUN: not llvm-mc %s` for the whole
> file.
> This often lead to users placing good (`RUN: llvm-mc %s`) and bad tests
> (`RUN:
> not llvm-mc %s`) separately. For some features, having both good and bad
> tests
> in one file may improve readability.
> b) .debug_line is a global resource. Whenever we add a (valid) .file, we
> contribute an entry to the global resource. If we want to test some
> characteristics when include_directories[0] is A, and other characteristics
> when include_directories[0] is B, we have to use another test file.
>
> The arguments apply to many other types of tests (opt on .ll, llc on .ll
> and .mir, clang on .c, yaml2obj on .yaml, etc).
>
> I have a patch teaching llvm-mc about an option to split input:
> https://reviews.llvm.org/D83725
> (30+ lines)
>
> In a comment, Richard Smith mentioned whether we can add a separate
> extractor utility:
>
> ```
> # RUN: extract bb %s | llvm-mc - 2>&1 | FileCheck %s --check-prefix=BB
>
> or
>
> # RUN: extract bb %s -o %t.bb
> # RUN: llvm-mc %t.bb 2>&1 | FileCheck %t.bb
> ```
>

Could make "extract" work a bit like "tee" so it can still be one line:

# RUN: extract bb %s -o %t.bb | llvm-mc - 2>&1 | FileCheck %t.bb

(could even make it a bit shorter for convenience - 'ex' or something)


> The advantage is its versatility. The downside is somewhat verbose syntax.
>
>
> Some questoms:
>
> 1. What do people think of the two approaches? An in-utility option vs a
> standalone utility.
> 2. For llvm-mc, if we go with an option, is there a better name than
> --doc-id? David Blaikie proposed --asm-id
>    (This is my personal preference, trading 30+ lines in a utility for
> simpler syntax)
> 3. If we add a standalone utility, how shall we name it? (Note that
> llvm-extract exists, but people can probably distinguish 'extract' from
> llvm-extract
>

I think some of the truly internal utilities are named without the llvm
prefix - isn't stuff like "not" actually implemented as a local tool? hmm,
guess not, maybe that's a built-in inside lit.
Only risk I can think of with the name is the auto-name expansion of lit
replacing any token 'ex' with the full path to the tool, so you might have
to be careful about not using that character sequence as a standalone
argument on a RUN line - but that seems OK.


> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200713/f6e7dc12/attachment.html>