[cfe-dev] [RFC] Compressing AST files by default?

[Alexey Sidorin via cfe-dev]

Hi Gabor!

Yes, your guess is right. This work is XTU-related.
We didn't see any performance improvements, however. But Android's ASTs 
have the size reduction to 38Gb from 70Gb. I think it's a nice improvement.


21.10.2016 22:13, Gábor Horváth via cfe-dev пишет:
> Hi!
>
> On 20 October 2016 at 18:12, Mehdi Amini via cfe-dev 
> <cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>> wrote:
>
>
>     > On Oct 20, 2016, at 2:23 AM, Ilya Palachev via cfe-dev
>     <cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>> wrote:
>     >
>     > Hi,
>     >
>     > It seems that compressing AST files with simple "gzip --fast"
>     makes them 30-40% smaller.
>     > So the questions are:
>     > 1. Is current AST serialization format really non-compressed
>     (only abbreviations in bit stream format)?
>     > 2. Is it worthwhile to compress AST by default (with -emit-ast)?
>     > 3. Will this break things like PCH?
>     > 4. What's the current trade-off between PCH compile time and
>     disk usage? If AST compression makes compilation a bit slower, but
>     reduces the disk usage significantly, will this be appropriate for
>     users or not?
>
>     Is there a need for this disk usage? If the main use of AST files
>     is C++ modules / PCH, what is a typical size for a module cache
>     directory?
>     (Compression is expensive)
>
>
>
> In some cases compression can actually improve the peformance, because 
> in some cases the bottleneck is the I/O, and less data read from the 
> disk and a fast decompression can be beneficial to the overall 
> performance.
>
> In case someone wants to do a whole project analysis on merged ASTs, 
> this compression can be a very significant saving. Dumping all of LLVM 
> and Clang TUs to the disk occupies about 45 GB of disk space at the 
> moment.
>
>
>     —
>     Mehdi
>
>
>     >
>     > LLVM already has a support for compression (functions
>     compress/uncompress in include/llvm/Support/Compression.h).
>     >
>     > Best regards,
>     > Ilya Palachev
>     > _______________________________________________
>     > cfe-dev mailing list
>     > cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>
>     > http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>     <http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev>
>
>     _______________________________________________
>     cfe-dev mailing list
>     cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>
>     http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>     <http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev>
>
>
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20161021/399b0468/attachment.html>