[llvm-bugs] [Bug 44399] New: Memory Consumption Reduction for Large Array Initialization?

[via llvm-bugs]

https://bugs.llvm.org/show_bug.cgi?id=44399

            Bug ID: 44399
           Summary: Memory Consumption Reduction for Large Array
                    Initialization?
           Product: clang
           Version: trunk
          Hardware: PC
                OS: Linux
            Status: NEW
          Severity: enhancement
          Priority: P
         Component: Frontend
          Assignee: unassignedclangbugs at nondot.org
          Reporter: phdofthehouse at gmail.com
                CC: llvm-bugs at lists.llvm.org, neeilans at live.com,
                    richard-llvm at metafoo.co.uk

I have a large number of files that I want to turn into data initializers for
(static/global/const) arrays and then compile into my program. I generally rely
on these arrays to be optimizer-transparent. Occasionally, they go up to 50~60
Megabytes.

The files are often dumped by xxd. A C++ header file representing 40 MB using
XXD goes to about a 250 MB or so file.

When I want to do work (constexpr processing in some cases) on some of these
arrays of (unsigned) char, I spend a lot of time just compiling and memory
usage shoots up. When I spent a lot of time tracking down the parts of what I
end up paying for, for just using a single entry out of an e.g. 20 MB array I
would use ~2.1 GB of memory. For example, for a 20 MB file named "image.bin", I
would use 
xxd -i image.bin image.bin.h" and then attempt to return a single item from
that array through main() to just test the cost of parsing and holding onto
that data in the compiler:

#include"<image_bin.h"

int main () {
      return image_bin[2];
}

This program took ~2.1 GB of memory and around 39~40 seconds of compilation
time, using 3 CPUs to nearly max. When I actually do real C++ with it, it gets
much worse.

I am not interested in using the linker to create an object file and directly
link it in, because that inhibits my ability to do constexpr/template
metaprogramming and create more compact, interesting data structures from
industry-standard file formats.

I would like to know if there are places in the Clang codebase I could
contribute to, that would allow me to improve the throughput of such commonly
deployed constructs. While I personally am the author of proposals heading to
the C and C++ groups to make working with files less painful, maybe there is
something I or others can do without new proposed functionality to help speed
this kind of code up and reduce the memory footprint?

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20191229/4b440b52/attachment.html>