[flang-dev] RFC: Port test_modfile.sh tests in Flang to use FileCheck

[Richard Barton via flang-dev]

Hi flang-dev

I'm looking at porting the Flang regression tests over to using standard LLVM utilities like FileCheck.

The modfile tests are a bunch of tests in Semantics that define modules and have check lines saying which module files are expected to be created and what they are expected to contain. The test_modfile.sh script runs Flang on the sources to generate said module files then checks that all expected module files have the correct contents and that no additional, unexpected module files are created.

The ported tests will exec Flang in the same way to create modfiles. We then use FileCheck on each module file in turn by name to ensure the correct name is generated and check the contents in the same way using CHECK-NEXT to ensure no additional information is added. The ported tests use the -module option to put the module files in a temporary location to make sure the tests cannot spuriously pass due to a dirty build directory. To check that no extra modfiles are generated, the tests use ls on the temporary location and ensure the right number of files are present.

Here is an example port of modfile01: https://reviews.llvm.org/D83320

The ported tests are more verbose than the previous approach using test_modfile.sh, at the expense of making more clear exactly what the test is doing without needing to also understand how test_modfile.sh works. I think the ported tests are preferable to the original on this principal of least surprise.

If folks are happy with this porting approach then I will do the rest of the tests and re-submit as a single patch.

Note: at present, lit has a bug when printing out UTF characters to a non-UTF shell if run with python 2. Flang's modfiles contain a short UTF BOM at the start of the file so the lit issue would be hit were these tests to fail in such an environment with -v or -a enabled. I have submitted a fix for this issue here: https://reviews.llvm.org/D82754 and I will not push the ports until that issue is fixed.

Thanks
Rich

Richard Barton
Principal Engineer - Arm Compiler for Linux
Arm Manchester