[llvm] [yaml2obj][MachO] Fix crash from integer underflow with invalid cmdsize (PR #165924)

[Ryan Mansfield via llvm-commits]

https://github.com/rjmansfield created https://github.com/llvm/llvm-project/pull/165924

yaml2obj would crash when processing Mach-O load commands with cmdsize smaller than the actual structure size e.g. LC_SEGMENT_64 with cmdsize=56 instead of 72. The crash occurred due to integer underflow when calculating padding: cmdsize - BytesWritten wraps to a large value when negative, causing a massive allocation attempt.

>From 1857805719a0c32263f4356c3902c63d32db15c8 Mon Sep 17 00:00:00 2001
From: Ryan Mansfield <ryan_mansfield at apple.com>
Date: Fri, 31 Oct 2025 16:21:11 -0400
Subject: [PATCH] [yaml2obj][MachO] Fix crash from integer underflow with
 invalid cmdsize

yaml2obj would crash when processing Mach-O load commands with cmdsize
smaller than the actual structure size e.g. LC_SEGMENT_64 with
cmdsize=56 instead of 72. The crash occurred due to integer underflow
when calculating padding: cmdsize - BytesWritten wraps to a large value
when negative, causing a massive allocation attempt.
---
 llvm/lib/ObjectYAML/MachOEmitter.cpp          | 10 +++++-
 .../MachO/segment-cmdsize-too-small.yaml      | 33 +++++++++++++++++++
 2 files changed, 42 insertions(+), 1 deletion(-)
 create mode 100644 llvm/test/ObjectYAML/MachO/segment-cmdsize-too-small.yaml

diff --git a/llvm/lib/ObjectYAML/MachOEmitter.cpp b/llvm/lib/ObjectYAML/MachOEmitter.cpp
index 35d442e8e3437..a3555b211ab53 100644
--- a/llvm/lib/ObjectYAML/MachOEmitter.cpp
+++ b/llvm/lib/ObjectYAML/MachOEmitter.cpp
@@ -285,7 +285,15 @@ void MachOWriter::writeLoadCommands(raw_ostream &OS) {
 
     // Fill remaining bytes with 0. This will only get hit in partially
     // specified test cases.
-    auto BytesRemaining = LC.Data.load_command_data.cmdsize - BytesWritten;
+    // Prevent integer underflow if BytesWritten exceeds cmdsize.
+    if (BytesWritten > LC.Data.load_command_data.cmdsize) {
+      errs() << "warning: load command " << LC.Data.load_command_data.cmd
+             << " cmdsize too small (" << LC.Data.load_command_data.cmdsize
+             << " bytes) for actual size (" << BytesWritten << " bytes)\n";
+    }
+    auto BytesRemaining = (BytesWritten < LC.Data.load_command_data.cmdsize)
+                              ? LC.Data.load_command_data.cmdsize - BytesWritten
+                              : 0;
     if (BytesRemaining > 0) {
       ZeroFillBytes(OS, BytesRemaining);
     }
diff --git a/llvm/test/ObjectYAML/MachO/segment-cmdsize-too-small.yaml b/llvm/test/ObjectYAML/MachO/segment-cmdsize-too-small.yaml
new file mode 100644
index 0000000000000..250a0631a5a9c
--- /dev/null
+++ b/llvm/test/ObjectYAML/MachO/segment-cmdsize-too-small.yaml
@@ -0,0 +1,33 @@
+## Test that yaml2obj handles cmdsize that is too small for LC_SEGMENT_64
+## without crashing (due to integer underflow).
+
+# RUN: yaml2obj %s -o %t 2>&1 | FileCheck %s --check-prefix=WARNING
+# RUN: not llvm-readobj --file-headers %t 2>&1 | FileCheck %s --check-prefix=MALFORMED
+
+# WARNING: warning: load command 25 cmdsize too small (56 bytes) for actual size (72 bytes)
+
+# MALFORMED: error: {{.*}}: truncated or malformed object (load command 0 LC_SEGMENT_64 cmdsize too small)
+
+--- !mach-o
+FileHeader:
+  magic:           0xFEEDFACF
+  cputype:         0x01000007
+  cpusubtype:      0x00000003
+  filetype:        0x00000001
+  ncmds:           1
+  sizeofcmds:      56
+  flags:           0x00002000
+  reserved:        0x00000000
+LoadCommands:
+  - cmd:             LC_SEGMENT_64
+    cmdsize:         56          ## Should be 72 for LC_SEGMENT_64
+    segname:         '__TEXT'
+    vmaddr:          0x1000
+    vmsize:          0x10
+    fileoff:         0
+    filesize:        0
+    maxprot:         7
+    initprot:        5
+    nsects:          0
+    flags:           0
+...