[PATCH] D37946: [lit] Fix some Python 3 compatibility issues.

[Zachary Turner via Phabricator via llvm-commits]

zturner created this revision.
Herald added a reviewer: modocache.

It seems we already have bots running lit under Python 3, so as I'm making more changes, I need to test under Python 3.  Unfortunately, there are some Windows differences that this bot doesn't exercise, and so I can't verify that my changes work without getting the suite Python 3 clean //on Windows//.

There were two main problems, both having to do with str / bytes compatibility.

1. On Windows we sometimes re-open stdout as binary instead of text.  But this means we have to write `bytes` instead of `str` in Py3.

2. We were using `str.decode('string_escape')`, but a) `string_escape` doesn't even exist in Py3 (you have to use `unicode_escape`), and b) you can't decode a `str` in Py3, you can only decode a `bytes`.

The patch here solves this by:

a) providing a wrapper that takes a `str` and encodes as utf8 `bytes` if and only if we re-opened stdout in binary mode.
b) Making the unescaper normalize on `bytes` up front, then decoding as either `string_escape` or `unicode_escape` depending on the version.  The result is always a `str`, which can then be passed to the function in a).

After this patch there's only 1 test still failing for me under Python 3, but it's non critical to verifying that my future patches don't break anything.


https://reviews.llvm.org/D37946

Files:
  llvm/utils/lit/lit/TestRunner.py


Index: llvm/utils/lit/lit/TestRunner.py
===================================================================
--- llvm/utils/lit/lit/TestRunner.py
+++ llvm/utils/lit/lit/TestRunner.py
@@ -251,13 +251,17 @@
     # Some tests have un-redirected echo commands to help debug test failures.
     # Buffer our output and return it to the caller.
     is_redirected = True
+    encode = lambda x : x
     if stdout == subprocess.PIPE:
         is_redirected = False
         stdout = StringIO()
     elif kIsWindows:
         # Reopen stdout in binary mode to avoid CRLF translation. The versions
         # of echo we are replacing on Windows all emit plain LF, and the LLVM
         # tests now depend on this.
+        # When we open as binary, however, this also means that we have to write
+        # 'bytes' objects to stdout instead of 'str' objects.
+        encode = lit.util.to_bytes
         stdout = open(stdout.name, stdout.mode + 'b')
         opened_files.append((None, None, stdout, None))
 
@@ -278,17 +282,18 @@
     def maybeUnescape(arg):
         if not interpret_escapes:
             return arg
-        # Python string escapes and "echo" escapes are obviously different, but
-        # this should be enough for the LLVM test suite.
-        return arg.decode('string_escape')
+
+        arg = lit.util.to_bytes(arg)
+        codec = 'string_escape' if sys.version_info < (3,0) else 'unicode_escape'
+        return arg.decode(codec)
 
     if args:
         for arg in args[:-1]:
-            stdout.write(maybeUnescape(arg))
-            stdout.write(' ')
-        stdout.write(maybeUnescape(args[-1]))
+            stdout.write(encode(maybeUnescape(arg)))
+            stdout.write(encode(' '))
+        stdout.write(encode(maybeUnescape(args[-1])))
     if write_newline:
-        stdout.write('\n')
+        stdout.write(encode('\n'))
 
     for (name, mode, f, path) in opened_files:
         f.close()


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D37946.115541.patch
Type: text/x-patch
Size: 1924 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20170916/be7bef4d/attachment.bin>