[llvm-dev] Multiple documents in one test file

Fangrui Song via llvm-dev llvm-dev at lists.llvm.org
Mon Jul 13 18:17:03 PDT 2020

Sometimes it is convenient if we can specify multiple independent tests
in one file. To give an example, let's discuss test/MC/ELF/debug-md5.s and
test/MC/ELF/debug-md5-err.s (.file directive in the assembler).

a) An invalid .file makes the whole file invalid. Because errors lead to a
non-zero exit code, We have to use `RUN: not llvm-mc %s` for the whole file.
This often lead to users placing good (`RUN: llvm-mc %s`) and bad tests (`RUN:
not llvm-mc %s`) separately. For some features, having both good and bad tests
in one file may improve readability.
b) .debug_line is a global resource. Whenever we add a (valid) .file, we
contribute an entry to the global resource. If we want to test some
characteristics when include_directories[0] is A, and other characteristics
when include_directories[0] is B, we have to use another test file.

The arguments apply to many other types of tests (opt on .ll, llc on .ll and .mir, clang on .c, yaml2obj on .yaml, etc).

I have a patch teaching llvm-mc about an option to split input: https://reviews.llvm.org/D83725
(30+ lines)

In a comment, Richard Smith mentioned whether we can add a separate extractor utility:

# RUN: extract bb %s | llvm-mc - 2>&1 | FileCheck %s --check-prefix=BB


# RUN: extract bb %s -o %t.bb
# RUN: llvm-mc %t.bb 2>&1 | FileCheck %t.bb

The advantage is its versatility. The downside is somewhat verbose syntax.

Some questoms:

1. What do people think of the two approaches? An in-utility option vs a standalone utility.
2. For llvm-mc, if we go with an option, is there a better name than --doc-id? David Blaikie proposed --asm-id
   (This is my personal preference, trading 30+ lines in a utility for simpler syntax)
3. If we add a standalone utility, how shall we name it? (Note that llvm-extract exists, but people can probably distinguish 'extract' from llvm-extract

More information about the llvm-dev mailing list