r289308 - [clang-format] Another attempt at python 3 compatibility

[Vedant Kumar via cfe-commits]

Thanks for double-checking this. I realized that there's a problem with my
patch: skipping the encoding step when using Python 3 could cause problems when
we use difflib.SequenceMatcher.

I don't know how to fix this, since the strings in vim.current.buffer appear to
be 'raw'. They are not encoded in any valid way. This happens with both the
python 2/3 vim interfaces. You can test this by creating a buffer with an emoji
in it ('😀') and then running:

  :set fileencoding=utf8
  :python import vim
  :python print(len(vim.current.buffer[0]))
  4

That makes us responsible for picking an encoding and using it consistently [*].
However, I don't know how to properly encode an invalid string in Python 3. Is
there some way to 'reinterpret_cast' it to a bytes object?

vedant

[*] I'd rather get rid of the encoding/decoding steps entirely, but I don't think
this is possible, because that would prevent us from using the JSON module.

> On Dec 10, 2016, at 6:16 AM, Nico Weber <thakis at chromium.org> wrote:
> 
> Shouldn't this at least use >=? (But it feels like there is probably a way that doesn't have to do UA sniffing, so to speak)
> 
> On Dec 9, 2016 8:04 PM, "Vedant Kumar via cfe-commits" <cfe-commits at lists.llvm.org> wrote:
> Author: vedantk
> Date: Fri Dec  9 18:54:13 2016
> New Revision: 289308
> 
> URL: http://llvm.org/viewvc/llvm-project?rev=289308&view=rev
> Log:
> [clang-format] Another attempt at python 3 compatibility
> 
> The entries in vim.current.buffer appear to be decoded strings, which
> means that python3 won't allow invoking 'decode' on them. Keep the old
> behavior when running under python2, but skip the error-inducing decode
> step with python3..
> 
> Modified:
>     cfe/trunk/tools/clang-format/clang-format.py
> 
> Modified: cfe/trunk/tools/clang-format/clang-format.py
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/clang-format/clang-format.py?rev=289308&r1=289307&r2=289308&view=diff
> ==============================================================================
> --- cfe/trunk/tools/clang-format/clang-format.py (original)
> +++ cfe/trunk/tools/clang-format/clang-format.py Fri Dec  9 18:54:13 2016
> @@ -29,6 +29,7 @@ from __future__ import print_function
> 
>  import difflib
>  import json
> +import platform
>  import subprocess
>  import sys
>  import vim
> @@ -48,10 +49,15 @@ fallback_style = None
>  if vim.eval('exists("g:clang_format_fallback_style")') == "1":
>    fallback_style = vim.eval('g:clang_format_fallback_style')
> 
> +def get_buffer(encoding):
> +  if platform.python_version_tuple()[0] == '3':
> +    return vim.current.buffer
> +  return [ line.decode(encoding) for line in vim.current.buffer ]
> +
>  def main():
>    # Get the current text.
>    encoding = vim.eval("&encoding")
> -  buf = [ line.decode(encoding) for line in vim.current.buffer ]
> +  buf = get_buffer(encoding)
>    text = '\n'.join(buf)
> 
>    # Determine range to format.
> 
> 
> _______________________________________________
> cfe-commits mailing list
> cfe-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits