[PATCH] D108742: [WIP] Reclassify form-feed and vertical tab as vertical WS for the purposes of lexing.

[Corentin Jabot via Phabricator via cfe-commits]

cor3ntin added a comment.

In D108742#2970218 <https://reviews.llvm.org/D108742#2970218>, @hubert.reinterpretcast wrote:

> I find the lack of changes to tests other than API unittests to be somewhat concerning.
> Indeed, I applied the patch and did not notice behaviour changes in how `\f` or `\v` were handled in various contexts sensitive to line-termination.
>
> For example:
>
>   clang -fsyntax-only -xc -<<<$'#define X\f int\n#define Y\v x;\ncha\\\v\nr x;\n// \fextern int x;\n'
>   <stdin>:3:5: warning: backslash and newline separated by space [-Wbackslash-newline-escape]
>   cha\<U+000B>
>       ^
>   1 warning generated.
>
> It appears phase 3 whitespace conversion is applied to `\f` and `\v` and the macro definitions extend to the `\n`s.
> The `\v\n` sequence is treated as only one newline (backslash escaped) and not two (with only the `\v` escaped).
> It also appears that the `//` comment is not terminated by the `\f`.

Yep, I noticed that today, \n is hardcoded in many places in the lexer, as it turns out, Hence why I marked the change WIP until I reassess the situation.
I should have been more suspicious than no test broke...

I also observed that gcc behavior seems somewhat consistent with clang.

I'm starting to fear that we might not be able to align compilers to consistency with Unicode.
Also as far as comments are concerned clang is strictly non-conforming (but neither seems to be gcc ?) - or rather as per the standard it produces ill-formed NDR programs, which is interesting.

It might be that the best solution is to abandon this route and instead treat \v and \f as whitespace rather than new-line in the standard and be inconsistent with Unicode.
The is probably not a huge motivation for having \v and \f being line breaking anyway (beyond Unicode saying so).


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D108742/new/

https://reviews.llvm.org/D108742