[libcxx-commits] [PATCH] D63346: [libcxx] [test] Read files as bytestrings to fix py3 encoding issues

[Michał Górny via Phabricator via libcxx-commits]

mgorny created this revision.
mgorny added a reviewer: EricWF.
Herald added a subscriber: christof.

Use binary mode to read test files in libcxx LibcxxTestFormat class.
This ensures that tests are read correctly independently of encoding,
and therefore fixes UnicodeDecodeError when file is opened in Python 3
that defaults to pure ASCII encoding.

Technically this could be also fixed via conditionally appending
encoding argument when opening the file in Python 3.  However, since
the code in question only searches for fixed ASCII substrings reading
it in binary mode is simpler and more universal.


https://reviews.llvm.org/D63346

Files:
  libcxx/utils/libcxx/test/format.py


Index: libcxx/utils/libcxx/test/format.py
===================================================================
--- libcxx/utils/libcxx/test/format.py
+++ libcxx/utils/libcxx/test/format.py
@@ -135,9 +135,9 @@
             # If we see this we need to build the test against uniquely built
             # modules.
             if is_libcxx_test:
-                with open(test.getSourcePath(), 'r') as f:
+                with open(test.getSourcePath(), 'rb') as f:
                     contents = f.read()
-                if '#define _LIBCPP_ASSERT' in contents:
+                if b'#define _LIBCPP_ASSERT' in contents:
                     test_cxx.useModules(False)
 
         if is_objcxx_test:
@@ -224,10 +224,11 @@
     def _evaluate_fail_test(self, test, test_cxx, parsers):
         source_path = test.getSourcePath()
         # FIXME: lift this detection into LLVM/LIT.
-        with open(source_path, 'r') as f:
+        with open(source_path, 'rb') as f:
             contents = f.read()
-        verify_tags = ['expected-note', 'expected-remark', 'expected-warning',
-                       'expected-error', 'expected-no-diagnostics']
+        verify_tags = [b'expected-note', b'expected-remark',
+                       b'expected-warning', b'expected-error',
+                       b'expected-no-diagnostics']
         use_verify = self.use_verify_for_fail and \
                      any([tag in contents for tag in verify_tags])
         # FIXME(EricWF): GCC 5 does not evaluate static assertions that
@@ -249,7 +250,8 @@
             #
             # Therefore, we check if the test was expected to fail because of
             # nodiscard before enabling it
-            test_str_list = ['ignoring return value', 'nodiscard', 'NODISCARD']
+            test_str_list = [b'ignoring return value', b'nodiscard',
+                             b'NODISCARD']
             if any(test_str in contents for test_str in test_str_list):
                 test_cxx.flags += ['-Werror=unused-result']
         cmd, out, err, rc = test_cxx.compile(source_path, out=os.devnull)


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D63346.204804.patch
Type: text/x-patch
Size: 2083 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/libcxx-commits/attachments/20190614/86230ce7/attachment.bin>