[llvm] [LLVM] add LZMA for compression/decompression (PR #83297)

[Artem Belevich via llvm-commits]

Artem-B wrote:

>  I do not understand the assertion around its decompression speed. 

For small apps (let's say < 100MB), it probably does not matter. Most of compression algorithms will be fast enough.
It's the large apps where it matters. It's not uncommon for a large machine learning app to carry literally gigabytes of GPU binaries. Then 100MB/s vs 500MB/s decompression rate will make quite a bit of a difference.
There are typically way more users, than builders, so the aggregate cost balance w/ slow decompression is not very good, IMO. 

> Compression is certainly slower, but it is not a lot slower than the competition.

An order of magnitude difference on @yxsamliu sample would qualify as "a lot", IMO.
That said, comparison with zstd's -9/-6 is probably not fair as -9  would be roughly in the middle of zstd's compression levels. LZMA compression does appear to be better at both speed and compression ratio than zstd's high compression levels (11+, according to the chart below).
![image](https://github.com/llvm/llvm-project/assets/526795/9bc3ec76-a126-4e03-b2cd-89708e6f29ae)

@yxsamliu zstd's compression levels don't seem to match those of lzma (e.g 9 is the highest compression for lzma, but only about a middle of the range for zstd). Could you also measure with `zstd -20` and `zstd -15` ?

On a side note, given that there's a huge jump in compression ratio between lzma -6 and -9, it suggests that it may have something to do with the compression window size. It may need to be large enough to cover multiple similar chunks of the binaries. I suspect we may be able to tweak zstd parameters to improve its compression ratio, too. 

Did anybody try training zstd on binaries and check how much it would help us in this case?


https://github.com/llvm/llvm-project/pull/83297