[PATCH] D63254: [lit] Fix UnicodeEncodeError when test commands contain non-ASCII chars

[Michał Górny via Phabricator via llvm-commits]

mgorny created this revision.
mgorny added reviewers: rnk, ddunbar, ruiu, serge-sans-paille, jmittert.
Herald added a subscriber: delcypher.
Herald added a project: LLVM.

Ensure that the bash script written by lit TestRunner is open with UTF-8
encoding when using Python 3.  Otherwise, attempt to write non-ASCII
characters causes UnicodeEncodeError.  This happened e.g. with
the following LLD test:

  UNRESOLVED: lld :: ELF/format-binary-non-ascii.s (657 of 2119)
  ******************** TEST 'lld :: ELF/format-binary-non-ascii.s' FAILED ********************
  Exception during script execution:
  Traceback (most recent call last):
    File "/home/mgorny/llvm-project/llvm/utils/lit/lit/worker.py", line 63, in _execute_test
      result = test.config.test_format.execute(test, lit_config)
    File "/home/mgorny/llvm-project/llvm/utils/lit/lit/formats/shtest.py", line 25, in execute
      self.execute_external)
    File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1644, in executeShTest
      res = _runShTest(test, litConfig, useExternalSh, script, tmpBase)
    File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1590, in _runShTest
      res = executeScript(test, litConfig, tmpBase, script, execdir)
    File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1157, in executeScript
      f.write('{ ' + '; } &&\n{ '.join(commands) + '; }')
  UnicodeEncodeError: 'ascii' codec can't encode character '\xa3' in position 274: ordinal not in range(128)


https://reviews.llvm.org/D63254

Files:
  llvm/utils/lit/lit/TestRunner.py


Index: llvm/utils/lit/lit/TestRunner.py
===================================================================
--- llvm/utils/lit/lit/TestRunner.py
+++ llvm/utils/lit/lit/TestRunner.py
@@ -1133,9 +1133,12 @@
 
     # Write script file
     mode = 'w'
+    open_kwargs = {}
     if litConfig.isWindows and not isWin32CMDEXE:
-      mode += 'b'  # Avoid CRLFs when writing bash scripts.
-    f = open(script, mode)
+        mode += 'b'  # Avoid CRLFs when writing bash scripts.
+    elif sys.version_info > (3,0):
+        open_kwargs['encoding'] = 'utf-8'
+    f = open(script, mode, **open_kwargs)
     if isWin32CMDEXE:
         for i, ln in enumerate(commands):
             commands[i] = re.sub(kPdbgRegex, "echo '\\1' > nul && ", ln)


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D63254.204496.patch
Type: text/x-patch
Size: 735 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20190613/e50dacd3/attachment.bin>