[PATCH] D37946: [lit] Fix some Python 3 compatibility issues.
Zachary Turner via Phabricator via llvm-commits
llvm-commits at lists.llvm.org
Sat Sep 16 10:29:23 PDT 2017
zturner created this revision.
Herald added a reviewer: modocache.
It seems we already have bots running lit under Python 3, so as I'm making more changes, I need to test under Python 3. Unfortunately, there are some Windows differences that this bot doesn't exercise, and so I can't verify that my changes work without getting the suite Python 3 clean //on Windows//.
There were two main problems, both having to do with str / bytes compatibility.
1. On Windows we sometimes re-open stdout as binary instead of text. But this means we have to write `bytes` instead of `str` in Py3.
2. We were using `str.decode('string_escape')`, but a) `string_escape` doesn't even exist in Py3 (you have to use `unicode_escape`), and b) you can't decode a `str` in Py3, you can only decode a `bytes`.
The patch here solves this by:
a) providing a wrapper that takes a `str` and encodes as utf8 `bytes` if and only if we re-opened stdout in binary mode.
b) Making the unescaper normalize on `bytes` up front, then decoding as either `string_escape` or `unicode_escape` depending on the version. The result is always a `str`, which can then be passed to the function in a).
After this patch there's only 1 test still failing for me under Python 3, but it's non critical to verifying that my future patches don't break anything.
https://reviews.llvm.org/D37946
Files:
llvm/utils/lit/lit/TestRunner.py
Index: llvm/utils/lit/lit/TestRunner.py
===================================================================
--- llvm/utils/lit/lit/TestRunner.py
+++ llvm/utils/lit/lit/TestRunner.py
@@ -251,13 +251,17 @@
# Some tests have un-redirected echo commands to help debug test failures.
# Buffer our output and return it to the caller.
is_redirected = True
+ encode = lambda x : x
if stdout == subprocess.PIPE:
is_redirected = False
stdout = StringIO()
elif kIsWindows:
# Reopen stdout in binary mode to avoid CRLF translation. The versions
# of echo we are replacing on Windows all emit plain LF, and the LLVM
# tests now depend on this.
+ # When we open as binary, however, this also means that we have to write
+ # 'bytes' objects to stdout instead of 'str' objects.
+ encode = lit.util.to_bytes
stdout = open(stdout.name, stdout.mode + 'b')
opened_files.append((None, None, stdout, None))
@@ -278,17 +282,18 @@
def maybeUnescape(arg):
if not interpret_escapes:
return arg
- # Python string escapes and "echo" escapes are obviously different, but
- # this should be enough for the LLVM test suite.
- return arg.decode('string_escape')
+
+ arg = lit.util.to_bytes(arg)
+ codec = 'string_escape' if sys.version_info < (3,0) else 'unicode_escape'
+ return arg.decode(codec)
if args:
for arg in args[:-1]:
- stdout.write(maybeUnescape(arg))
- stdout.write(' ')
- stdout.write(maybeUnescape(args[-1]))
+ stdout.write(encode(maybeUnescape(arg)))
+ stdout.write(encode(' '))
+ stdout.write(encode(maybeUnescape(args[-1])))
if write_newline:
- stdout.write('\n')
+ stdout.write(encode('\n'))
for (name, mode, f, path) in opened_files:
f.close()
-------------- next part --------------
A non-text attachment was scrubbed...
Name: D37946.115541.patch
Type: text/x-patch
Size: 1924 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20170916/be7bef4d/attachment.bin>
More information about the llvm-commits
mailing list