[llvm-bugs] [Bug 37447] New: encodeULEB128() buffer overflow cause tblgen "stack smashing detected" crash

[via llvm-bugs]

https://bugs.llvm.org/show_bug.cgi?id=37447

            Bug ID: 37447
           Summary: encodeULEB128() buffer overflow cause tblgen "stack
                    smashing detected" crash
           Product: new-bugs
           Version: trunk
          Hardware: PC
                OS: Windows NT
            Status: NEW
          Severity: enhancement
          Priority: P
         Component: new bugs
          Assignee: unassignedbugs at nondot.org
          Reporter: kaz-marukawa at xr.jp.nec.com
                CC: llvm-bugs at lists.llvm.org

Created attachment 20301
  --> https://bugs.llvm.org/attachment.cgi?id=20301&action=edit
patch to fix described problem

When I tried to implement a backend for a CPU using 64 bit wide instructions,
tblgen crashes very often with "stack smashing detected" error.

I drilled down it and found tblgen calls encodeULEB128() with 64 bit value
for input and 8 bytes buffer for output sometimes.  This encodeULEB128()
encodes a given value into buffer using SDNV (Self-Delimiting Numeric Value)
format.  This format requires 10 bytes buffer at maximum (abs(64/7) = 10).
For example, 0x8000000000000000 input is converted into "80 80 80 80 80 80
80 80 80 01".

Once I applied attached patch, tblgen works fine with my modified backend.
Please review this patch and fix the bug.  Thanks.

In this patch, modification of TableGen/FixedLenDecoderEmitter.cpp is what
above crash causes.  Modification of ProfileData/InstrProf.cpp is what I
found by grep and this may not be needed.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20180514/11a4e1e9/attachment.html>