[llvm-dev] Multiple documents in one test file

[Fangrui Song via llvm-dev]

On 2020-07-14, Chen, Yuanfang wrote:
>Interesting. `-split-input-file` is indeed terse and performant for tests with similar run lines. To generalize the functionality a bit, I'm not sure if people have had the idea of let LIT to the splitting using some common separator lines like `# SEP: #`, `// SEP: //` etc.
>
>________________________________________
>From: Mehdi AMINI <joker.eph at gmail.com>
>Sent: Monday, July 13, 2020 8:06 PM
>To: Chen, Yuanfang
>Cc: llvm-dev; Fangrui Song
>Subject: Re: [llvm-dev] Multiple documents in one test file
>
>We have a similar option (-split-input-file) in `mlir-opt`: https://github.com/llvm/llvm-project/blob/master/mlir/test/Dialect/Affine/invalid.mlir
>
>With a single `RUN:` lit invocation the tool itself will loop over all the split sections in the file. This is convenient to test error cases where the tool would abort at the first error otherwise. I don't think we can easily achieve this with a single pipe and a separate `extract` command though?

Some metaprogramming in lit will be helpful.

For example, the below is adapted from a real test case. The test wants to
checks several variants of Mips. The duplicated code is acceptable
becuase the commands are short:

   # RUN: llvm-mc -triple=mips-unknown-linux-gnu %s \
   # RUN:   | FileCheck --check-prefix=ASM %s
   # RUN: llvm-mc -triple=mips64-unknown-linux-gnu %s \
   # RUN:   | FileCheck --check-prefix=ASM %s
   # RUN: llvm-mc -triple=mipsel-unknown-linux-gnu %s --target-abi=o32 \
   # RUN:   | FileCheck --check-prefix=ASM %s

   (Similar examples can be found in clang: -std=c++11 -std=c++14 -std=c++17)

In lld/test/ELF/ there are some bad cases, e.g lld/test/ELF/ppc64-tls-le.s
It needs to test both little-endian and big-endian variants. Since one
test consists of multiple commands, it looks awful:

   # RUN: llvm-mc -filetype=obj -triple=powerpc64le %s -o %t.o
   # RUN: llvm-readobj -r %t.o | FileCheck --check-prefix=INPUT-REL %s
   ## IE
   # RUN: ld.lld -shared %t.o -o %t.so
   # RUN: llvm-readobj -r %t.so | FileCheck --check-prefix=IE-REL %s
   # RUN: llvm-objdump -d --no-show-raw-insn %t.so | FileCheck --check-prefix=IE %s
   ## IE -> LE
   # RUN: ld.lld %t.o -o %t
   # RUN: llvm-readelf -r %t | FileCheck --check-prefix=NOREL %s
   # RUN: llvm-objdump -d --no-show-raw-insn %t | FileCheck --check-prefix=LE %s
   
   # RUN: llvm-mc -filetype=obj -triple=powerpc64 %s -o %t.o
   # RUN: llvm-readobj -r %t.o | FileCheck --check-prefix=INPUT-REL %s
   ## IE
   # RUN: ld.lld -shared %t.o -o %t.so
   # RUN: llvm-readobj -r %t.so | FileCheck --check-prefix=IE-REL %s
   # RUN: llvm-objdump -d --no-show-raw-insn %t.so | FileCheck --check-prefix=IE %s
   ## IE -> LE
   # RUN: ld.lld %t.o -o %t
   # RUN: llvm-readelf -r %t | FileCheck --check-prefix=NOREL %s
   # RUN: llvm-objdump -d --no-show-raw-insn %t | FileCheck --check-prefix=LE %s

RISC-V tests can be unpleasant as well (32-bit and 64-bit variants)

   # RUN: llvm-mc -filetype=obj -triple=riscv32 %s -o %t.32.o
   # RUN: ld.lld -pie %t.32.o -o %t.32
   # RUN: llvm-readelf -s %t.32 | FileCheck --check-prefix=SYM32 %s
   # RUN: llvm-readelf -S %t.32 | FileCheck --check-prefix=SEC32 %s
   # RUN: not ld.lld -shared %t.32.o -o /dev/null 2>&1 | FileCheck --check-prefix=ERR %s
   
   # RUN: llvm-mc -filetype=obj -triple=riscv64 %s -o %t.64.o
   # RUN: ld.lld -pie %t.64.o -o %t.64
   # RUN: llvm-readelf -s %t.64 | FileCheck --check-prefix=SYM64 %s
   # RUN: llvm-readelf -S %t.64 | FileCheck --check-prefix=SEC64 %s
   # RUN: not ld.lld -shared %t.64.o -o /dev/null 2>&1 | FileCheck --check-prefix=ERR %s

This can be simplified if lit supports for loop constructs:

   # RUN: %for i in 32 64
   # RUN:   llvm-mc -filetype=obj -triple=riscv%i %s -o %t.%i.o
   # RUN:   ld.lld -pie %t.%i.o -o %t.%i
   # RUN:   llvm-readelf -s %t.%i | FileCheck --check-prefix=SYM%i %s
   # RUN:   llvm-readelf -S %t.%i | FileCheck --check-prefix=SEC%i %s
   # RUN:   not ld.lld -shared %t.%i.o -o /dev/null 2>&1 | FileCheck --check-prefix=ERR %s
   # RUN: %}