<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Zinovy,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I have reverted this change in r356630 in order to get the build bots back to green.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I was able to reproduce the issue locally on my machine, it appears that your use of “import yaml” is not part of the standard python distribution and so is not
found.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Douglas Yung<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> cfe-commits <cfe-commits-bounces@lists.llvm.org>
<b>On Behalf Of </b>Galina Kistanova via cfe-commits<br>
<b>Sent:</b> Wednesday, March 20, 2019 14:03<br>
<b>To:</b> Zinovy Nis <zinovy.nis@gmail.com><br>
<b>Cc:</b> cfe-commits <cfe-commits@lists.llvm.org><br>
<b>Subject:</b> Re: [clang-tools-extra] r356565 - Reland r356547 after fixing the tests for Linux.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Hello Zinovy,<br>
<br>
This commit broke test on the next builder:<br>
<a href="http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-ubuntu-fast/builds/45557">http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-ubuntu-fast/builds/45557</a><br>
. . .<br>
Failing Tests (1):<br>
Clang Tools :: clang-tidy/clang-tidy-diff.cpp<br>
<br>
Please have a look?<br>
<br>
Thanks<br>
<br>
Galina<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, Mar 20, 2019 at 8:49 AM Zinovy Nis via cfe-commits <<a href="mailto:cfe-commits@lists.llvm.org">cfe-commits@lists.llvm.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Author: zinovy.nis<br>
Date: Wed Mar 20 08:50:26 2019<br>
New Revision: 356565<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=356565&view=rev" target="_blank">
http://llvm.org/viewvc/llvm-project?rev=356565&view=rev</a><br>
Log:<br>
Reland r356547 after fixing the tests for Linux.<br>
<br>
[clang-tidy] Parallelize clang-tidy-diff.py<br>
<br>
This patch has 2 rationales:<br>
<br>
- large patches lead to long command lines and often cause max command line length restrictions imposed by OS;<br>
- clang-tidy runs on modified files are independent and can be done in parallel, the same as done for run-clang-tidy.<br>
<br>
Differential Revision: <a href="https://reviews.llvm.org/D57662" target="_blank">
https://reviews.llvm.org/D57662</a><br>
<br>
<br>
Modified:<br>
clang-tools-extra/trunk/clang-tidy/tool/clang-tidy-diff.py<br>
<br>
Modified: clang-tools-extra/trunk/clang-tidy/tool/clang-tidy-diff.py<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clang-tidy/tool/clang-tidy-diff.py?rev=356565&r1=356564&r2=356565&view=diff" target="_blank">
http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/clang-tidy/tool/clang-tidy-diff.py?rev=356565&r1=356564&r2=356565&view=diff</a><br>
==============================================================================<br>
--- clang-tools-extra/trunk/clang-tidy/tool/clang-tidy-diff.py (original)<br>
+++ clang-tools-extra/trunk/clang-tidy/tool/clang-tidy-diff.py Wed Mar 20 08:50:26 2019<br>
@@ -24,10 +24,90 @@ Example usage for git/svn users:<br>
"""<br>
<br>
import argparse<br>
+import glob<br>
import json<br>
+import multiprocessing<br>
+import os<br>
import re<br>
+import shutil<br>
import subprocess<br>
import sys<br>
+import tempfile<br>
+import threading<br>
+import traceback<br>
+import yaml<br>
+<br>
+is_py2 = sys.version[0] == '2'<br>
+<br>
+if is_py2:<br>
+ import Queue as queue<br>
+else:<br>
+ import queue as queue<br>
+<br>
+<br>
+def run_tidy(task_queue, lock, timeout):<br>
+ watchdog = None<br>
+ while True:<br>
+ command = task_queue.get()<br>
+ try:<br>
+ proc = subprocess.Popen(command,<br>
+ stdout=subprocess.PIPE,<br>
+ stderr=subprocess.PIPE)<br>
+<br>
+ if timeout is not None:<br>
+ watchdog = threading.Timer(timeout, proc.kill)<br>
+ watchdog.start()<br>
+<br>
+ stdout, stderr = proc.communicate()<br>
+<br>
+ with lock:<br>
+ sys.stdout.write(stdout.decode('utf-8') + '\n')<br>
+ sys.stdout.flush()<br>
+ if stderr:<br>
+ sys.stderr.write(stderr.decode('utf-8') + '\n')<br>
+ sys.stderr.flush()<br>
+ except Exception as e:<br>
+ with lock:<br>
+ sys.stderr.write('Failed: ' + str(e) + ': '.join(command) + '\n')<br>
+ finally:<br>
+ with lock:<br>
+ if (not timeout is None) and (not watchdog is None):<br>
+ if not watchdog.is_alive():<br>
+ sys.stderr.write('Terminated by timeout: ' +<br>
+ ' '.join(command) + '\n')<br>
+ watchdog.cancel()<br>
+ task_queue.task_done()<br>
+<br>
+<br>
+def start_workers(max_tasks, tidy_caller, task_queue, lock, timeout):<br>
+ for _ in range(max_tasks):<br>
+ t = threading.Thread(target=tidy_caller, args=(task_queue, lock, timeout))<br>
+ t.daemon = True<br>
+ t.start()<br>
+<br>
+def merge_replacement_files(tmpdir, mergefile):<br>
+ """Merge all replacement files in a directory into a single file"""<br>
+ # The fixes suggested by clang-tidy >= 4.0.0 are given under<br>
+ # the top level key 'Diagnostics' in the output yaml files<br>
+ mergekey = "Diagnostics"<br>
+ merged = []<br>
+ for replacefile in glob.iglob(os.path.join(tmpdir, '*.yaml')):<br>
+ content = yaml.safe_load(open(replacefile, 'r'))<br>
+ if not content:<br>
+ continue # Skip empty files.<br>
+ merged.extend(content.get(mergekey, []))<br>
+<br>
+ if merged:<br>
+ # MainSourceFile: The key is required by the definition inside<br>
+ # include/clang/Tooling/ReplacementsYaml.h, but the value<br>
+ # is actually never used inside clang-apply-replacements,<br>
+ # so we set it to '' here.<br>
+ output = { 'MainSourceFile': '', mergekey: merged }<br>
+ with open(mergefile, 'w') as out:<br>
+ yaml.safe_dump(output, out)<br>
+ else:<br>
+ # Empty the file:<br>
+ open(mergefile, 'w').close()<br>
<br>
<br>
def main():<br>
@@ -47,7 +127,10 @@ def main():<br>
r'.*\.(cpp|cc|c\+\+|cxx|c|cl|h|hpp|m|mm|inc)',<br>
help='custom pattern selecting file paths to check '<br>
'(case insensitive, overridden by -regex)')<br>
-<br>
+ parser.add_argument('-j', type=int, default=1,<br>
+ help='number of tidy instances to be run in parallel.')<br>
+ parser.add_argument('-timeout', type=int, default=None,<br>
+ help='timeout per each file in seconds.')<br>
parser.add_argument('-fix', action='store_true', default=False,<br>
help='apply suggested fixes')<br>
parser.add_argument('-checks',<br>
@@ -84,7 +167,7 @@ def main():<br>
match = re.search('^\+\+\+\ \"?(.*?/){%s}([^ \t\n\"]*)' % args.p, line)<br>
if match:<br>
filename = match.group(2)<br>
- if filename == None:<br>
+ if filename is None:<br>
continue<br>
<br>
if args.regex is not None:<br>
@@ -102,44 +185,79 @@ def main():<br>
line_count = int(match.group(3))<br>
if line_count == 0:<br>
continue<br>
- end_line = start_line + line_count - 1;<br>
+ end_line = start_line + line_count - 1<br>
lines_by_file.setdefault(filename, []).append([start_line, end_line])<br>
<br>
- if len(lines_by_file) == 0:<br>
+ if not any(lines_by_file):<br>
print("No relevant changes found.")<br>
sys.exit(0)<br>
<br>
- line_filter_json = json.dumps(<br>
- [{"name" : name, "lines" : lines_by_file[name]} for name in lines_by_file],<br>
- separators = (',', ':'))<br>
-<br>
- quote = "";<br>
- if sys.platform == 'win32':<br>
- line_filter_json=re.sub(r'"', r'"""', line_filter_json)<br>
- else:<br>
- quote = "'";<br>
+ max_task_count = args.j<br>
+ if max_task_count == 0:<br>
+ max_task_count = multiprocessing.cpu_count()<br>
+ max_task_count = min(len(lines_by_file), max_task_count)<br>
<br>
- # Run clang-tidy on files containing changes.<br>
- command = [args.clang_tidy_binary]<br>
- command.append('-line-filter=' + quote + line_filter_json + quote)<br>
- if args.fix:<br>
- command.append('-fix')<br>
+ tmpdir = None<br>
if args.export_fixes:<br>
- command.append('-export-fixes=' + args.export_fixes)<br>
+ tmpdir = tempfile.mkdtemp()<br>
+<br>
+ # Tasks for clang-tidy.<br>
+ task_queue = queue.Queue(max_task_count)<br>
+ # A lock for console output.<br>
+ lock = threading.Lock()<br>
+<br>
+ # Run a pool of clang-tidy workers.<br>
+ start_workers(max_task_count, run_tidy, task_queue, lock, args.timeout)<br>
+<br>
+ # Form the common args list.<br>
+ common_clang_tidy_args = []<br>
+ if args.fix:<br>
+ common_clang_tidy_args.append('-fix')<br>
if args.checks != '':<br>
- command.append('-checks=' + quote + args.checks + quote)<br>
+ common_clang_tidy_args.append('-checks=' + args.checks)<br>
if args.quiet:<br>
- command.append('-quiet')<br>
+ common_clang_tidy_args.append('-quiet')<br>
if args.build_path is not None:<br>
- command.append('-p=%s' % args.build_path)<br>
- command.extend(lines_by_file.keys())<br>
+ common_clang_tidy_args.append('-p=%s' % args.build_path)<br>
for arg in args.extra_arg:<br>
- command.append('-extra-arg=%s' % arg)<br>
+ common_clang_tidy_args.append('-extra-arg=%s' % arg)<br>
for arg in args.extra_arg_before:<br>
- command.append('-extra-arg-before=%s' % arg)<br>
- command.extend(clang_tidy_args)<br>
+ common_clang_tidy_args.append('-extra-arg-before=%s' % arg)<br>
+<br>
+ for name in lines_by_file:<br>
+ line_filter_json = json.dumps(<br>
+ [{"name": name, "lines": lines_by_file[name]}],<br>
+ separators=(',', ':'))<br>
+<br>
+ # Run clang-tidy on files containing changes.<br>
+ command = [args.clang_tidy_binary]<br>
+ command.append('-line-filter=' + line_filter_json)<br>
+ if args.export_fixes:<br>
+ # Get a temporary file. We immediately close the handle so clang-tidy can<br>
+ # overwrite it.<br>
+ (handle, tmp_name) = tempfile.mkstemp(suffix='.yaml', dir=tmpdir)<br>
+ os.close(handle)<br>
+ command.append('-export-fixes=' + tmp_name)<br>
+ command.extend(common_clang_tidy_args)<br>
+ command.append(name)<br>
+ command.extend(clang_tidy_args)<br>
+<br>
+ task_queue.put(command)<br>
+<br>
+ # Wait for all threads to be done.<br>
+ task_queue.join()<br>
+<br>
+ if args.export_fixes:<br>
+ print('Writing fixes to ' + args.export_fixes + ' ...')<br>
+ try:<br>
+ merge_replacement_files(tmpdir, args.export_fixes)<br>
+ except:<br>
+ sys.stderr.write('Error exporting fixes.\n')<br>
+ traceback.print_exc()<br>
+<br>
+ if tmpdir:<br>
+ shutil.rmtree(tmpdir)<br>
<br>
- sys.exit(subprocess.call(' '.join(command), shell=True))<br>
<br>
if __name__ == '__main__':<br>
main()<br>
<br>
<br>
_______________________________________________<br>
cfe-commits mailing list<br>
<a href="mailto:cfe-commits@lists.llvm.org" target="_blank">cfe-commits@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</body>
</html>