[libcxx-commits] [libcxx] r364170 - [libcxx] [test] Read files as bytestrings to fix py3 encoding issues

[Michal Gorny via libcxx-commits]

Author: mgorny
Date: Mon Jun 24 02:40:33 2019
New Revision: 364170

URL: http://llvm.org/viewvc/llvm-project?rev=364170&view=rev
Log:
[libcxx] [test] Read files as bytestrings to fix py3 encoding issues

Use binary mode to read test files in libcxx LibcxxTestFormat class.
This ensures that tests are read correctly independently of encoding,
and therefore fixes UnicodeDecodeError when file is opened in Python 3
that defaults to pure ASCII encoding.

Technically this could be also fixed via conditionally appending
encoding argument when opening the file in Python 3.  However, since
the code in question only searches for fixed ASCII substrings reading
it in binary mode is simpler and more universal.

Differential Revision: https://reviews.llvm.org/D63346

Modified:
    libcxx/trunk/utils/libcxx/test/format.py

Modified: libcxx/trunk/utils/libcxx/test/format.py
URL: http://llvm.org/viewvc/llvm-project/libcxx/trunk/utils/libcxx/test/format.py?rev=364170&r1=364169&r2=364170&view=diff
==============================================================================
--- libcxx/trunk/utils/libcxx/test/format.py (original)
+++ libcxx/trunk/utils/libcxx/test/format.py Mon Jun 24 02:40:33 2019
@@ -135,9 +135,9 @@ class LibcxxTestFormat(object):
             # If we see this we need to build the test against uniquely built
             # modules.
             if is_libcxx_test:
-                with open(test.getSourcePath(), 'r') as f:
+                with open(test.getSourcePath(), 'rb') as f:
                     contents = f.read()
-                if '#define _LIBCPP_ASSERT' in contents:
+                if b'#define _LIBCPP_ASSERT' in contents:
                     test_cxx.useModules(False)
 
         if is_objcxx_test:
@@ -224,10 +224,11 @@ class LibcxxTestFormat(object):
     def _evaluate_fail_test(self, test, test_cxx, parsers):
         source_path = test.getSourcePath()
         # FIXME: lift this detection into LLVM/LIT.
-        with open(source_path, 'r') as f:
+        with open(source_path, 'rb') as f:
             contents = f.read()
-        verify_tags = ['expected-note', 'expected-remark', 'expected-warning',
-                       'expected-error', 'expected-no-diagnostics']
+        verify_tags = [b'expected-note', b'expected-remark',
+                       b'expected-warning', b'expected-error',
+                       b'expected-no-diagnostics']
         use_verify = self.use_verify_for_fail and \
                      any([tag in contents for tag in verify_tags])
         # FIXME(EricWF): GCC 5 does not evaluate static assertions that
@@ -249,7 +250,8 @@ class LibcxxTestFormat(object):
             #
             # Therefore, we check if the test was expected to fail because of
             # nodiscard before enabling it
-            test_str_list = ['ignoring return value', 'nodiscard', 'NODISCARD']
+            test_str_list = [b'ignoring return value', b'nodiscard',
+                             b'NODISCARD']
             if any(test_str in contents for test_str in test_str_list):
                 test_cxx.flags += ['-Werror=unused-result']
         cmd, out, err, rc = test_cxx.compile(source_path, out=os.devnull)