[llvm] r329022 - [lit] Fix problem in how Python versions open files with different encodings
via llvm-commits
llvm-commits at lists.llvm.org
Mon Apr 2 18:33:07 PDT 2018
Yup, thanks for the quick fix!
Douglas Yung
> -----Original Message-----
> From: Aaron Smith [mailto:aaron.smith at microsoft.com]
> Sent: Monday, April 02, 2018 18:21
> To: Yung, Douglas
> Cc: llvm-commits at lists.llvm.org
> Subject: Re: [llvm] r329022 - [lit] Fix problem in how Python versions open
> files with different encodings
>
> Your Windows bot is good now.
>
> http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-
> fast/builds/16216
>
> On 4/2/18, 5:19 PM, "douglas.yung at sony.com" <douglas.yung at sony.com> wrote:
>
> Hi Aaron,
>
> I don't know if you are still working on this, but the PS4 Windows bot
> still seems to be failing.
>
>
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flab.llvm.org%3
> A8011%2Fbuilders%2Fllvm-clang-lld-x86_64-scei-ps4-windows10pro-
> fast%2Fbuilds%2F16215&data=02%7C01%7Caaron.smith%40microsoft.com%7C8410e12b0f3
> f49cf7d1308d598f89f6d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63658311595
> 9611850&sdata=F1k%2FZszjahxmMLOvWADr0xultK%2FSUChzl1IWjv6Waew%3D&reserved=0
>
> UNRESOLVED: Clang :: Modules/Werror-Wsystem-headers.m (8206 of 39290)
> ******************** TEST 'Clang :: Modules/Werror-Wsystem-headers.m'
> FAILED ********************
> Exception during script execution:
> Traceback (most recent call last):
> File "C:/ps4-buildslave2/llvm-clang-lld-x86_64-scei-ps4-windows10pro-
> fast/llvm.src\utils\lit\lit\run.py", line 202, in _execute_test_impl
> result = test.config.test_format.execute(test, lit_config)
> File "C:/ps4-buildslave2/llvm-clang-lld-x86_64-scei-ps4-windows10pro-
> fast/llvm.src\utils\lit\lit\formats\shtest.py", line 25, in execute
> self.execute_external)
> File "C:/ps4-buildslave2/llvm-clang-lld-x86_64-scei-ps4-windows10pro-
> fast/llvm.src\utils\lit\lit\TestRunner.py", line 1537, in executeShTest
> res = _runShTest(test, litConfig, useExternalSh, script, tmpBase)
> File "C:/ps4-buildslave2/llvm-clang-lld-x86_64-scei-ps4-windows10pro-
> fast/llvm.src\utils\lit\lit\TestRunner.py", line 1485, in _runShTest
> res = executeScriptInternal(test, litConfig, tmpBase, script, execdir)
> File "C:/ps4-buildslave2/llvm-clang-lld-x86_64-scei-ps4-windows10pro-
> fast/llvm.src\utils\lit\lit\TestRunner.py", line 997, in executeScriptInternal
> exitCode, timeoutInfo = executeShCmd(cmd, shenv, results,
> timeout=litConfig.maxIndividualTestTime)
> File "C:/ps4-buildslave2/llvm-clang-lld-x86_64-scei-ps4-windows10pro-
> fast/llvm.src\utils\lit\lit\TestRunner.py", line 152, in executeShCmd
> finalExitCode = _executeShCmd(cmd, shenv, results, timeoutHelper)
> File "C:/ps4-buildslave2/llvm-clang-lld-x86_64-scei-ps4-windows10pro-
> fast/llvm.src\utils\lit\lit\TestRunner.py", line 720, in _executeShCmd
> res = _executeShCmd(cmd.lhs, shenv, results, timeoutHelper)
> File "C:/ps4-buildslave2/llvm-clang-lld-x86_64-scei-ps4-windows10pro-
> fast/llvm.src\utils\lit\lit\TestRunner.py", line 720, in _executeShCmd
> res = _executeShCmd(cmd.lhs, shenv, results, timeoutHelper)
> File "C:/ps4-buildslave2/llvm-clang-lld-x86_64-scei-ps4-windows10pro-
> fast/llvm.src\utils\lit\lit\TestRunner.py", line 725, in _executeShCmd
> res = _executeShCmd(cmd.rhs, shenv, results, timeoutHelper)
> File "C:/ps4-buildslave2/llvm-clang-lld-x86_64-scei-ps4-windows10pro-
> fast/llvm.src\utils\lit\lit\TestRunner.py", line 778, in _executeShCmd
> cmdResult = executeBuiltinDiff(cmd.commands[0], shenv)
> File "C:/ps4-buildslave2/llvm-clang-lld-x86_64-scei-ps4-windows10pro-
> fast/llvm.src\utils\lit\lit\TestRunner.py", line 548, in executeBuiltinDiff
> exitCode = compareTwoFiles(filepaths)
> File "C:/ps4-buildslave2/llvm-clang-lld-x86_64-scei-ps4-windows10pro-
> fast/llvm.src\utils\lit\lit\TestRunner.py", line 411, in compareTwoFiles
> return compareTwoTextFiles(filepaths, encoding)
> File "C:/ps4-buildslave2/llvm-clang-lld-x86_64-scei-ps4-windows10pro-
> fast/llvm.src\utils\lit\lit\TestRunner.py", line 438, in compareTwoTextFiles
> filelines.append(f.readlines())
> File "C:\Python27\lib\encodings\cp1252.py", line 23, in decode
> return codecs.charmap_decode(input,self.errors,decoding_table)[0]
> UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 44:
> character maps to <undefined>
>
>
> Can you take a look?
>
> Douglas Yung
>
>
> > -----Original Message-----
> > From: llvm-commits [mailto:llvm-commits-bounces at lists.llvm.org] On
> Behalf Of
> > Aaron Smith via llvm-commits
> > Sent: Monday, April 02, 2018 15:09
> > To: llvm-commits at lists.llvm.org
> > Subject: [llvm] r329022 - [lit] Fix problem in how Python versions open
> files
> > with different encodings
> >
> > Author: asmith
> > Date: Mon Apr 2 15:08:56 2018
> > New Revision: 329022
> >
> > URL:
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fllvm.org%2Fvie
> wvc%2Fllvm-
> project%3Frev%3D329022%26view%3Drev&data=02%7C01%7Caaron.smith%40microsoft.com
> %7C8410e12b0f3f49cf7d1308d598f89f6d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0
> %7C636583115959611850&sdata=48Ap866YXiWKlO0Ytc4jgZYVh9kN3V633J375vymhUc%3D&res
> erved=0
> > Log:
> > [lit] Fix problem in how Python versions open files with different
> encodings
> >
> > Reapply D43165 which was reverted because of different versions of
> python
> > failing.
> > The one line fix for the different python versions was commited at the
> same
> > time that D43165 was reverted. If this change is giving you issues then
> get in
> > touch with your python version and we will fix it.
> >
> > Modified:
> > llvm/trunk/utils/lit/lit/TestRunner.py
> >
> > Modified: llvm/trunk/utils/lit/lit/TestRunner.py
> > URL:
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fllvm.org%2Fvie
> wvc%2Fllvm-
> &data=02%7C01%7Caaron.smith%40microsoft.com%7C8410e12b0f3f49cf7d1308d598f89f6d
> %7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636583115959611850&sdata=uFndL6L
> ctMk4SGrgR3qllhb8%2BShSgC3syYfxpn60yNQ%3D&reserved=0
> >
> project/llvm/trunk/utils/lit/lit/TestRunner.py?rev=329022&r1=329021&r2=329022&
> > view=diff
> >
> ==============================================================================
> > --- llvm/trunk/utils/lit/lit/TestRunner.py (original)
> > +++ llvm/trunk/utils/lit/lit/TestRunner.py Mon Apr 2 15:08:56 2018
> > @@ -388,7 +388,47 @@ def executeBuiltinDiff(cmd, cmd_shenv):
> > def compareTwoFiles(filepaths):
> > filelines = []
> > for file in filepaths:
> > - with io.open(file, 'r') as f:
> > + try:
> > + with open(file, 'r') as f:
> > + filelines.append(f.readlines())
> > + except UnicodeDecodeError:
> > + try:
> > + with open(file, 'r', encoding="utf-8") as f:
> > + filelines.append(f.readlines())
> > + encoding = "utf-8"
> > + except:
> > + compare_bytes = True
> > +
> > + if compare_bytes:
> > + return compareTwoBinaryFiles(filepaths)
> > + else:
> > + return compareTwoTextFiles(filepaths, encoding)
> > +
> > + def compareTwoBinaryFiles(filepaths):
> > + filelines = []
> > + for file in filepaths:
> > + with open(file, 'rb') as f:
> > + filelines.append(f.readlines())
> > +
> > + exitCode = 0
> > + if hasattr(difflib, 'diff_bytes'):
> > + # python 3.5 or newer
> > + diffs = difflib.diff_bytes(difflib.unified_diff,
> filelines[0],
> > filelines[1], filepaths[0].encode(), filepaths[1].encode())
> > + diffs = [diff.decode() for diff in diffs]
> > + else:
> > + # python 2.7
> > + func = difflib.unified_diff if unified_diff else
> > difflib.context_diff
> > + diffs = func(filelines[0], filelines[1], filepaths[0],
> > + filepaths[1])
> > +
> > + for diff in diffs:
> > + stdout.write(diff)
> > + exitCode = 1
> > + return exitCode
> > +
> > + def compareTwoTextFiles(filepaths, encoding):
> > + filelines = []
> > + for file in filepaths:
> > + with io.open(file, 'r', encoding=encoding) as f:
> > filelines.append(f.readlines())
> >
> > exitCode = 0
> >
> >
> > _______________________________________________
> > llvm-commits mailing list
> > llvm-commits at lists.llvm.org
> >
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.llvm.org
> %2Fcgi-bin%2Fmailman%2Flistinfo%2Fllvm-
> commits&data=02%7C01%7Caaron.smith%40microsoft.com%7C8410e12b0f3f49cf7d1308d59
> 8f89f6d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636583115959611850&sdata=
> SZQhyk5vxFV8EmVqWTdamdZKGKYww7RyrNL8mpjSHXA%3D&reserved=0
>
More information about the llvm-commits
mailing list