[llvm-dev] RFC: Add a preprocessor to yaml2obj (and other YAML tools)

[Fangrui Song via llvm-dev]

I am adding -D k=v to yaml2obj, similar to clang -D. This makes it easy
to generate {32-bit,64-bit} x {big-endian,little-endian} tests.

   --- !ELF
   FileHeader:
     Class:   ELFCLASS[[BITS]]
     Data:    ELFDATA2[[ENCODE]]
     Type:    ET_DYN
     Machine: EM_X86_64

# RUN: yaml2obj -D BITS=32 -D ENCODE=LSB %s -o %t.32le
# RUN: yaml2obj -D BITS=32 -D ENCODE=MSB %s -o %t.32le
# RUN: yaml2obj -D BITS=64 -D ENCODE=LSB %s -o %t.64le
# RUN: yaml2obj -D BITS=64 -D ENCODE=MSB %s -o %t.64be

See https://reviews.llvm.org/D73828 for examples how -D simplifies tests.

Do people think it may be useful in other YAML tools? If yes, I'll move
the yaml2obj implementation (https://reviews.llvm.org/D73821 ) to
include/llvm/Support/YAMLTraits.h llvm::yaml::Input so that other YAML
tools can use the feature.

Do people prefer a different syntax? I think [[PATTERN]] is nice because
it is what FileCheck -DFILE=... uses:

   # CHECK: ... [[FILE]]

   FileCheck only preprocesses patterns in CHECK lines.
   D73821 preprocesses both comment lines (which include CHECK lines) and non-comment lines (which include YAML).
   It is not a problem that the YAML preprocessor also processes CHECK lines, because tokens on a comment line will be ignored.

If -D UNDEF= is not specified, should [[UNDEF]] in the source be considered an error?
I think it is fine not to treat it as an error because there can be
legitimate use cases of unterminated [[, for example, [[ in a string literal.
YAML parsing is complex. I don't expect the preprocessor to be smart
enough to recognize string literals. (llvm/lib/Support/YAMLParser.cpp does not seem to provide raw strings of
spaces and comments. Hooking a preprocessor into the scanner does not seem to be simple.)

Do people know other preprocessing features which may be useful?