[libcxx-commits] [libcxx] r373402 - Determine endianness at a time when it doesn't inadvertantly clear gdb's wrap_buffer via gdb.execute.

[Sterling Augustine via libcxx-commits]

Author: saugustine
Date: Tue Oct  1 15:42:37 2019
New Revision: 373402

URL: http://llvm.org/viewvc/llvm-project?rev=373402&view=rev
Log:
Determine endianness at a time when it doesn't inadvertantly clear gdb's wrap_buffer via gdb.execute.

Summary:
I haven't managed a small reproduction for this bug, it involves
complicated and deeply nested data structures with a wide variety
of pretty printers. But in general, we shouldn't be combining
gdb's command line interface (via gdb.execute) with pretty-printers.

Subscribers: christof, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D68306

Modified:
    libcxx/trunk/utils/gdb/libcxx/printers.py

Modified: libcxx/trunk/utils/gdb/libcxx/printers.py
URL: http://llvm.org/viewvc/llvm-project/libcxx/trunk/utils/gdb/libcxx/printers.py?rev=373402&r1=373401&r2=373402&view=diff
==============================================================================
--- libcxx/trunk/utils/gdb/libcxx/printers.py (original)
+++ libcxx/trunk/utils/gdb/libcxx/printers.py Tue Oct  1 15:42:37 2019
@@ -27,6 +27,7 @@ _void_pointer_type = gdb.lookup_type("vo
 
 _long_int_type = gdb.lookup_type("unsigned long long")
 
+_libcpp_big_endian = False
 
 def addr_as_long(addr):
     return int(addr.cast(_long_int_type))
@@ -195,19 +196,14 @@ class StdStringPrinter(object):
         field = short_field.type.fields()[1].type.fields()[0]
         libcpp_abi_alternate_string_layout = field.name and "__padding" in field.name
 
-        # Strictly, this only tells us the current mode, not how libcxx was
-        # compiled.
-        libcpp_big_endian = "big endian" in gdb.execute("show endian",
-                                                        to_string=True)
-
         # This logical structure closely follows the original code (which is clearer
         # in C++).  Keep them parallel to make them easier to compare.
         if libcpp_abi_alternate_string_layout:
-            if libcpp_big_endian:
+            if _libcpp_big_endian:
                 return short_size >> 1
             else:
                 return short_size
-        elif libcpp_big_endian:
+        elif _libcpp_big_endian:
             return short_size
         else:
             return short_size >> 1
@@ -969,6 +965,14 @@ _libcxx_printer_name = "libcxx_pretty_pr
 # certain pathological cases. Limit our pretty printers to the progspace.
 def _register_libcxx_printers(event):
     progspace = event.new_objfile.progspace
+    # It would be ideal to get the endianness at print time, but
+    # gdb.execute clears gdb's internal wrap buffer, removing any values
+    # already generated as part of a larger data structure, and there is
+    # no python api to get the endianness. Mixed-endianness debugging
+    # rare enough that this workaround should be adequate.
+    _libcpp_big_endian = "big endian" in gdb.execute("show endian",
+                                                     to_string=True)
+
     if not getattr(progspace, _libcxx_printer_name, False):
         print("Loading libc++ pretty-printers.")
         gdb.printing.register_pretty_printer(