[PATCH] D54769: [FileCheck] New option -warn

Mon Dec 3 11:18:42 PST 2018

dblaikie added a comment.

In D54769#1317107 <https://reviews.llvm.org/D54769#1317107>, @jdenny wrote:

> For simple tests, an external script could parse (static) test files as I described.  That script could run as a separate test case.

Not exactly possible, unfortunately - tests can/should only read files in the "Inputs" directory relative to the test file location - they can't walk the entire test tree/read everything (this keeps tests a bit isolated, so you can move them around without worrying about some other random test depending on your test - and also it's used internally at Google to avoid shipping all the tests/inputs to every machine that runs any of the tests)

> I'm not sure which tests would be "simple", but any test that generates the FileCheck check file dynamically would certainly not work with such a script.

I don't think there are any of those - at least I rather hope not.

> For a (hopefully) complete solution, here's a possible approach.  Each FileCheck call builds (1) a list of all prefixes defined (by `--check-prefix[es]`) for that one call and (2) a list of all prefixes used in directives but not defined in the first list, and then it merges those lists into a database.  At the end of a test suite run, lit calls a special FileCheck incantation that compiles that database into a list of unused directives.

"merges those lists into a database" is a bit vague/concerning in terms of how that'd run in the build system, etc.

>> Each FileCheck run would itself look for other FileCheck commands in RUN lines to compute the total set of prefixes used & validate them all? (that'd mean redundant work (O(N^2) in the number of FileCheck commands/prefixes - but probably scale OK) & redundant error messages (since each FileCheck command would report all the unused check names itself))
> 
> That seems possible too.  We could experiment with different approaches.

Yeah, I'd suggest scratching something simple up with grep/sed/python/whatever to get a sense of the magnitude of the problem before sinking time into trying to design/build a long-term solution. My feeling is still that this isn't a problem worth solving given the complexity of the solutions thus far (the self-checking FileCheck version (despite its duplicate error messages, etc) feels like the cheapest I can think of & still isn't terribly nice, I think) - but data will help.

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D54769/new/

https://reviews.llvm.org/D54769