[PATCH] D54071: [Bug 39548][Clang] PGO bootstrap fails with python3: errors in perf-helper.py

[Romain Geissler via Phabricator via cfe-commits]

Romain-Geissler-1A created this revision.
Romain-Geissler-1A added reviewers: rsmith, vsk, beanz.
Herald added a subscriber: cfe-commits.
Romain-Geissler-1A edited the summary of this revision.

Hi,

Current clang fail to bootstrap in PGO mode when only python3 is available, because perf-helper.py is not compatible with python 3. We can see two errors:

    File "/workdir/src/llvm-7.0.0.src/tools/clang/utils/perf-training/perf-helper.py", line 299
      by_count.sort(key = lambda (_,n): -n)
                                 ^
  SyntaxError: invalid syntax

which can be fixed by removing the parameter parenthesis.

And

  $ "/usr/bin/python3.6" "/workdir/src/llvm-7.0.0.src/tools/clang/utils/perf-training/perf-helper.py" "cc1" "/workdir/build/final-system/llvm-build/tools/clang/stage2-instrumented-bins/bin/clang" "-Wall" "-pedantic" "-c" "/home/jenkins/workspace/OTF_Toolchain_release_2.0-HLXHYRKCVDYQJLF23VGZ3MVAU6VGURX537LUE3KFVM2SSPMZ6IOA/output/src/llvm-7.0.0.src/tools/clang/utils/perf-training/cxx/hello_world.cpp"
  # command stderr:
  Traceback (most recent call last):
    File "/workdir/src/llvm-7.0.0.src/tools/clang/utils/perf-training/perf-helper.py", line 408, in <module>
      main()
    File "/workdir/src/llvm-7.0.0.src/tools/clang/utils/perf-training/perf-helper.py", line 405, in main
      sys.exit(f(sys.argv[2:]))
    File "/workdir/src/llvm-7.0.0.src/tools/clang/utils/perf-training/perf-helper.py", line 159, in cc1
      cc1_cmd = get_cc1_command_for_args(cmd, cc1_env)
    File "/workdir/src/llvm-7.0.0.src/tools/clang/utils/perf-training/perf-helper.py", line 119, in get_cc1_command_for_args
      for ln in cc_output.split('\n'):
  TypeError: a bytes-like object is required, not 'str'
  
  error: command failed with exit status: 1

which can be made both 2 and 3 compatible by using the attribute universal_newlines=True

Cheers,
Romain


Repository:
  rC Clang

https://reviews.llvm.org/D54071

Files:
  utils/perf-training/perf-helper.py


Index: utils/perf-training/perf-helper.py
===================================================================
--- utils/perf-training/perf-helper.py
+++ utils/perf-training/perf-helper.py
@@ -114,7 +114,7 @@
   # Find the cc1 command used by the compiler. To do this we execute the
   # compiler with '-###' to figure out what it wants to do.
   cmd = cmd + ['-###']
-  cc_output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, env=env).strip()
+  cc_output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, env=env, universal_newlines=True).strip()
   cc_commands = []
   for ln in cc_output.split('\n'):
       # Filter out known garbage.
@@ -296,7 +296,7 @@
       counts[a] = counts.get(a,0) + 1
 
   by_count = counts.items()
-  by_count.sort(key = lambda (_,n): -n)
+  by_count.sort(key = lambda _,n: -n)
   return [s for s,n in by_count]
  
 def form_by_random(symbol_lists):
@@ -340,7 +340,7 @@
   # If the user gave us a binary, get all the symbols in the binary by
   # snarfing 'nm' output.
   if opts.binary_path is not None:
-     output = subprocess.check_output(['nm', '-P', opts.binary_path])
+     output = subprocess.check_output(['nm', '-P', opts.binary_path], universal_newlines=True)
      lines = output.split("\n")
      all_symbols = [ln.split(' ',1)[0]
                     for ln in lines


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D54071.172491.patch
Type: text/x-patch
Size: 1331 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20181103/91357100/attachment.bin>