[PATCH] D43165: [lit] Fix problem in how Python versions open files with different encodings

[Reid Kleckner via Phabricator via llvm-commits]

rnk added a comment.

In https://reviews.llvm.org/D43165#1054919, @asmith wrote:

> I committed a one line fix before noticing you reverted the change.
>  Did you happen to check if the fix works for you?


I had do do a few more fixes, and I still have these test failures:

  diff --git a/llvm/utils/lit/lit/TestRunner.py b/llvm/utils/lit/lit/TestRunner.py
  index 02745e62dcf..d0c7417c248 100644
  --- a/llvm/utils/lit/lit/TestRunner.py
  +++ b/llvm/utils/lit/lit/TestRunner.py
  @@ -12,6 +12,7 @@ import shutil
   import tempfile
   import threading
  
  +import io
   try:
       from StringIO import StringIO
   except ImportError:
  @@ -388,6 +389,8 @@ def executeBuiltinDiff(cmd, cmd_shenv):
       def compareTwoFiles(filepaths):
           filelines = []
           for file in filepaths:
  +            compare_bytes = False
  +            encoding = None
               try:
                   with open(file, 'r') as f:
                       filelines.append(f.readlines())
  ...
  
  Unresolved Tests (5):
      LLVM :: ThinLTO/X86/distributed_import.ll
      LLVM :: Transforms/ThinLTOBitcodeWriter/no-type-md.ll
      LLVM :: Transforms/ThinLTOBitcodeWriter/split.ll
      LLVM :: Transforms/ThinLTOBitcodeWriter/unsplittable.ll
      LLVM :: tools/llvm-objcopy/drawf-fission.test

The failures look like:

  $ ./bin/llvm-lit.py  -v ../llvm/test/ThinLTO/X86/distributed_import.ll
  -- Testing: 1 tests, 1 threads --
  UNRESOLVED: LLVM :: ThinLTO/X86/distributed_import.ll (1 of 1)
  ******************** TEST 'LLVM :: ThinLTO/X86/distributed_import.ll' FAILED ********************
  Exception during script execution:
  Traceback (most recent call last):
    File "C:/src/llvm-project/llvm\utils\lit\lit\run.py", line 202, in _execute_test_impl
      result = test.config.test_format.execute(test, lit_config)
    File "C:/src/llvm-project/llvm\utils\lit\lit\formats\shtest.py", line 25, in execute
      self.execute_external)
    File "C:/src/llvm-project/llvm\utils\lit\lit\TestRunner.py", line 1534, in executeShTest
      res = _runShTest(test, litConfig, useExternalSh, script, tmpBase)
    File "C:/src/llvm-project/llvm\utils\lit\lit\TestRunner.py", line 1482, in _runShTest
      res = executeScriptInternal(test, litConfig, tmpBase, script, execdir)
    File "C:/src/llvm-project/llvm\utils\lit\lit\TestRunner.py", line 994, in executeScriptInternal
      exitCode, timeoutInfo = executeShCmd(cmd, shenv, results, timeout=litConfig.maxIndividualTestTime)
    File "C:/src/llvm-project/llvm\utils\lit\lit\TestRunner.py", line 151, in executeShCmd
      finalExitCode = _executeShCmd(cmd, shenv, results, timeoutHelper)
    File "C:/src/llvm-project/llvm\utils\lit\lit\TestRunner.py", line 717, in _executeShCmd
      res = _executeShCmd(cmd.lhs, shenv, results, timeoutHelper)
    File "C:/src/llvm-project/llvm\utils\lit\lit\TestRunner.py", line 717, in _executeShCmd
      res = _executeShCmd(cmd.lhs, shenv, results, timeoutHelper)
    File "C:/src/llvm-project/llvm\utils\lit\lit\TestRunner.py", line 717, in _executeShCmd
      res = _executeShCmd(cmd.lhs, shenv, results, timeoutHelper)
    File "C:/src/llvm-project/llvm\utils\lit\lit\TestRunner.py", line 717, in _executeShCmd
      res = _executeShCmd(cmd.lhs, shenv, results, timeoutHelper)
    File "C:/src/llvm-project/llvm\utils\lit\lit\TestRunner.py", line 717, in _executeShCmd
      res = _executeShCmd(cmd.lhs, shenv, results, timeoutHelper)
    File "C:/src/llvm-project/llvm\utils\lit\lit\TestRunner.py", line 717, in _executeShCmd
      res = _executeShCmd(cmd.lhs, shenv, results, timeoutHelper)
    File "C:/src/llvm-project/llvm\utils\lit\lit\TestRunner.py", line 717, in _executeShCmd
      res = _executeShCmd(cmd.lhs, shenv, results, timeoutHelper)
    File "C:/src/llvm-project/llvm\utils\lit\lit\TestRunner.py", line 722, in _executeShCmd
      res = _executeShCmd(cmd.rhs, shenv, results, timeoutHelper)
    File "C:/src/llvm-project/llvm\utils\lit\lit\TestRunner.py", line 775, in _executeShCmd
      cmdResult = executeBuiltinDiff(cmd.commands[0], shenv)
    File "C:/src/llvm-project/llvm\utils\lit\lit\TestRunner.py", line 545, in executeBuiltinDiff
      exitCode = compareTwoFiles(filepaths)
    File "C:/src/llvm-project/llvm\utils\lit\lit\TestRunner.py", line 408, in compareTwoFiles
      return compareTwoTextFiles(filepaths, encoding)
    File "C:/src/llvm-project/llvm\utils\lit\lit\TestRunner.py", line 435, in compareTwoTextFiles
      filelines.append(f.readlines())
    File "C:\Python27\lib\encodings\cp1252.py", line 23, in decode
      return codecs.charmap_decode(input,self.errors,decoding_table)[0]
  UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 42: character maps to <undefined>

Some kind of binary file comparison?


Repository:
  rL LLVM

https://reviews.llvm.org/D43165