<html>
<head>
<base href="https://bugs.llvm.org/">
</head>
<body><table border="1" cellspacing="0" cellpadding="8">
<tr>
<th>Bug ID</th>
<td><a class="bz_bug_link
bz_status_NEW "
title="NEW - Memory Consumption Reduction for Large Array Initialization?"
href="https://bugs.llvm.org/show_bug.cgi?id=44399">44399</a>
</td>
</tr>
<tr>
<th>Summary</th>
<td>Memory Consumption Reduction for Large Array Initialization?
</td>
</tr>
<tr>
<th>Product</th>
<td>clang
</td>
</tr>
<tr>
<th>Version</th>
<td>trunk
</td>
</tr>
<tr>
<th>Hardware</th>
<td>PC
</td>
</tr>
<tr>
<th>OS</th>
<td>Linux
</td>
</tr>
<tr>
<th>Status</th>
<td>NEW
</td>
</tr>
<tr>
<th>Severity</th>
<td>enhancement
</td>
</tr>
<tr>
<th>Priority</th>
<td>P
</td>
</tr>
<tr>
<th>Component</th>
<td>Frontend
</td>
</tr>
<tr>
<th>Assignee</th>
<td>unassignedclangbugs@nondot.org
</td>
</tr>
<tr>
<th>Reporter</th>
<td>phdofthehouse@gmail.com
</td>
</tr>
<tr>
<th>CC</th>
<td>llvm-bugs@lists.llvm.org, neeilans@live.com, richard-llvm@metafoo.co.uk
</td>
</tr></table>
<p>
<div>
<pre>I have a large number of files that I want to turn into data initializers for
(static/global/const) arrays and then compile into my program. I generally rely
on these arrays to be optimizer-transparent. Occasionally, they go up to 50~60
Megabytes.
The files are often dumped by xxd. A C++ header file representing 40 MB using
XXD goes to about a 250 MB or so file.
When I want to do work (constexpr processing in some cases) on some of these
arrays of (unsigned) char, I spend a lot of time just compiling and memory
usage shoots up. When I spent a lot of time tracking down the parts of what I
end up paying for, for just using a single entry out of an e.g. 20 MB array I
would use ~2.1 GB of memory. For example, for a 20 MB file named "image.bin", I
would use
xxd -i image.bin image.bin.h" and then attempt to return a single item from
that array through main() to just test the cost of parsing and holding onto
that data in the compiler:
#include"<image_bin.h"
int main () {
return image_bin[2];
}
This program took ~2.1 GB of memory and around 39~40 seconds of compilation
time, using 3 CPUs to nearly max. When I actually do real C++ with it, it gets
much worse.
I am not interested in using the linker to create an object file and directly
link it in, because that inhibits my ability to do constexpr/template
metaprogramming and create more compact, interesting data structures from
industry-standard file formats.
I would like to know if there are places in the Clang codebase I could
contribute to, that would allow me to improve the throughput of such commonly
deployed constructs. While I personally am the author of proposals heading to
the C and C++ groups to make working with files less painful, maybe there is
something I or others can do without new proposed functionality to help speed
this kind of code up and reduce the memory footprint?</pre>
</div>
</p>
<hr>
<span>You are receiving this mail because:</span>
<ul>
<li>You are on the CC list for the bug.</li>
</ul>
</body>
</html>