[PATCH] D72703: Add a warning, flags and pragmas to limit the number of pre-processor tokens in a translation unit

Thu Jan 23 22:43:21 PST 2020

kimgr added a comment.

In D72703#1834177 <https://reviews.llvm.org/D72703#1834177>, @hans wrote:

> I assume the same correlation could also be found with lines of code, but I think tokens is a better dimension to measure since it's less likely to be gamed, and it also is also kind of the basic work unit that the compiler deals with.

Yeah, and code with lots of documentatinon comments isn't penalized.

>> Could you expose a flag for printing token count so users can run their own analysis? Or does that already exist in baseline clang? It's easier to set a maximum for a codebase if the distribution is known.
> 
> I used this patch with -fmax-tokens 1 and scraped the output for my measurements. I would like to avoid adding a separate flag if we can avoid it.

Ah, it didn't occur to me that `-fmax-tokens` itself could be used like this. Thanks!

FWIW, I'm not ecstatic about `max_tokens_here`, I thought `max_tokens_lexed` had a nicer ring to it. /peanut.

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D72703/new/

https://reviews.llvm.org/D72703