[PATCH] D38289: [opt-viewer] Don't Decode HTML bytes for Python 2

[Roman Lebedev via Phabricator via llvm-commits]

lebedev.ri created this revision.
Herald added a subscriber: fhahn.

https://reviews.llvm.org/D36624 added some python3 compatibility. But that fix has a problem..

With python2 (which is specified by `#!/usr/bin/env python2.7`), if the env variables do not specify the UTF8,
and the source file is UTF8 (contains non-ASCII symbols), then the `.decode('utf-8')` causes the following exception:

  Reading YAML files...
  Rendering HTML files...
          8 of 41Traceback (most recent call last):
    File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 277, in <module>
      print_progress)
    File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 213, in generate_report
      should_print_progress)
    File "/build/llvm/tools/opt-viewer/optpmap.py", line 45, in pmap
      result = map(_wrapped_func, func_and_args, *args, **kwargs)
    File "/build/llvm/tools/opt-viewer/optpmap.py", line 25, in _wrapped_func
      return func(argument)
    File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 174, in _render_file
      SourceFileRenderer(source_dir, output_dir, filename).render(remarks)
    File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 125, in render
      self.render_source_lines(self.source_stream, line_remarks)
    File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 79, in render_source_lines
      </tr>'''.format(**locals()), file=self.stream)
  UnicodeEncodeError: 'ascii' codec can't encode character u'\xf4' in position 47: ordinal not in range(128)

This is similar to https://bugs.llvm.org/show_bug.cgi?id=33548, which was fixed by https://reviews.llvm.org/D37661

Unlike that fix, here, *removing* `.decode('utf-8')` actually fixes it.

Since i assume that the original fix is needed, i simply made
that fix conditional, since for python2 it actually breaks things.


Repository:
  rL LLVM

https://reviews.llvm.org/D38289

Files:
  tools/opt-viewer/opt-viewer.py


Index: tools/opt-viewer/opt-viewer.py
===================================================================
--- tools/opt-viewer/opt-viewer.py
+++ tools/opt-viewer/opt-viewer.py
@@ -10,6 +10,7 @@
 import os.path
 import re
 import shutil
+import sys
 
 from pygments import highlight
 from pygments.lexers.c_cpp import CppLexer
@@ -62,7 +63,10 @@
         html_highlighted = highlight(
             file_text,
             self.cpp_lexer,
-            self.html_formatter).decode('utf-8')
+            self.html_formatter)
+
+        if sys.version_info >= (3, 0):
+          html_highlighted = html_highlighted.decode('utf-8')
 
         # Take off the header and footer, these must be
         #   reapplied line-wise, within the page structure


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D38289.116696.patch
Type: text/x-patch
Size: 745 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20170926/0e40ec41/attachment-0001.bin>