[Lldb-commits] [PATCH] D125107: [lldb] Parallelize fetching symbol files in crashlog.py

[Jonas Devlieghere via Phabricator via lldb-commits]

JDevlieghere created this revision.
JDevlieghere added reviewers: mib, clayborg.
Herald added a subscriber: kristof.beyls.
Herald added a project: All.
JDevlieghere requested review of this revision.

When using dsymForUUID, the majority of time symbolication a crashlog with crashlog.py is spent waiting for it to complete. Currently, we're calling dsymForUUID sequentially when iterating over the modules. We can drastically cut down this time by calling dsymForUUID in parallel. This patch uses Python's ThreadPoolExecutor (introduced in Python 3.2) to parallelize this IO-bound operation.

The performance improvement is hard to benchmark, because even with an empty local cache, consecutive calls to dsymForUUID for the same UUID complete faster. With warm caches, I'm seeing a ~30% performance improvement (~90s -> ~30s). I suspect the gains will be much bigger for a cold cache.


https://reviews.llvm.org/D125107

Files:
  lldb/examples/python/crashlog.py


Index: lldb/examples/python/crashlog.py
===================================================================
--- lldb/examples/python/crashlog.py
+++ lldb/examples/python/crashlog.py
@@ -26,7 +26,6 @@
 #   PYTHONPATH=/path/to/LLDB.framework/Resources/Python ./crashlog.py ~/Library/Logs/DiagnosticReports/a.crash
 #----------------------------------------------------------------------
 
-from __future__ import print_function
 import cmd
 import contextlib
 import datetime
@@ -43,6 +42,7 @@
 import sys
 import time
 import uuid
+import concurrent.futures
 
 try:
     # First try for LLDB in case PYTHONPATH is already correctly setup.
@@ -269,7 +269,7 @@
             self.resolved = True
             uuid_str = self.get_normalized_uuid_string()
             if self.show_symbol_progress():
-                print('Getting symbols for %s %s...\n' % (uuid_str, self.path), end=' ')
+                print('Getting symbols for %s %s...' % (uuid_str, self.path))
             if os.path.exists(self.dsymForUUIDBinary):
                 dsym_for_uuid_command = '%s %s' % (
                     self.dsymForUUIDBinary, uuid_str)
@@ -319,7 +319,7 @@
                     pass
             if (self.resolved_path and os.path.exists(self.resolved_path)) or (
                     self.path and os.path.exists(self.path)):
-                print('Resolved symbols for %s %s...\n' % (uuid_str, self.path), end=' ')
+                print('Resolved symbols for %s %s...' % (uuid_str, self.path))
                 return True
             else:
                 self.unavailable = True
@@ -914,7 +914,6 @@
         option_parser = CrashLogOptionParser()
         return option_parser.format_help()
 
-
 def SymbolicateCrashLog(crash_log, options):
     if options.debug:
         crash_log.dump()
@@ -961,9 +960,16 @@
                 else:
                     print('error: can\'t find image for identifier "%s"' % ident)
 
-    for image in images_to_load:
-        if image not in loaded_images:
-            err = image.add_module(target)
+    futures = []
+    with concurrent.futures.ThreadPoolExecutor() as executor:
+        def add_module(image, target):
+            return image, image.add_module(target)
+
+        for image in images_to_load:
+            futures.append(executor.submit(add_module, image=image, target=target))
+
+        for future in concurrent.futures.as_completed(futures):
+            image, err = future.result()
             if err:
                 print(err)
             else:


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D125107.427678.patch
Type: text/x-patch
Size: 2511 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/lldb-commits/attachments/20220506/9560bb6d/attachment.bin>