[SOLVED] problem with lrzip

Toutatis · 03-08-2021, 12:40 AM

It seems that lrzip does not compress a lot:

Using the last kernel source release : kernel-source-5.10.21-noarch-1.txz

I obtain

Quote:

--> ls -l kernel-source-5.10.21-noarch-1.txz
117327296 kernel-source-5.10.21-noarch-1.txz
--> xz -d kernel-source-5.10.21-noarch-1.txz
--> ls -l kernel-source-5.10.21-noarch-1.tar
1033308160 kernel-source-5.10.21-noarch-1.tar
--> lrzip kernel-source-5.10.21-noarch-1.tar
Output filename is: kernel-source-5.10.21-noarch-1.tar.lrz
kernel-source-5.10.21-noarch-1.tar - Compression Ratio: 1.359. Average Compression Speed: 98.500MB/s.
Total time: 00:00:10.25
--> ls -l kernel-source-5.10.21-noarch-1.tar.lrz
760074999 kernel-source-5.10.21-noarch-1.tar.lrz

chrisretusn · 03-08-2021, 03:52 AM

What are you expecting to see?

I first did this.

Code:

mkdir kernel-source-5.10.21/
cd kernel-source-5.10.21/
explodepkg ../kernel-source-5.10.21-noarch-1.txz
# snip the output of that.
tar -cf kernel-source-5.10.21.tar kernel-source-5.10.21
lrzip kernel-source-5.10.21.tar
Output filename is: kernel-source-5.10.21.tar.lrz
kernel-source-5.10.21.tar - Compression Ratio: 1.341. Average Compression Speed: 34.138MB/s.
Total time: 00:00:29.43

I then did this:

Code:

xz -d kernel-source-5.10.21-noarch-1.txz
lrzip kernel-source-5.10.21-noarch-1.tar 
Output filename is: kernel-source-5.10.21-noarch-1.tar.lrz
kernel-source-5.10.21-noarch-1.tar - Compression Ratio: 1.402. Average Compression Speed: 24.625MB/s.
Total time: 00:00:39.16

File sizes:

Code:

-rw-r--r-- 1 root root 1033308160 Mar  8 04:14 kernel-source-5.10.21-noarch-1.tar
-rw-r--r-- 1 root root  736789046 Mar  8 04:14 kernel-source-5.10.21-noarch-1.tar.lrz
-rw-r--r-- 1 root root  117327296 Mar  8 04:14 kernel-source-5.10.21-noarch-1.txz
-rw-r--r-- 1 root root 1038632960 Mar  8 17:26 kernel-source-5.10.21.tar
-rw-r--r-- 1 root root  774727995 Mar  8 17:26 kernel-source-5.10.21.tar.lrz

-rw-r--r-- 1 root root 986M Mar  8 04:14 kernel-source-5.10.21-noarch-1.tar
-rw-r--r-- 1 root root 703M Mar  8 04:14 kernel-source-5.10.21-noarch-1.tar.lrz
-rw-r--r-- 1 root root 112M Mar  8 04:14 kernel-source-5.10.21-noarch-1.txz
-rw-r--r-- 1 root root 991M Mar  8 17:26 kernel-source-5.10.21.tar
-rw-r--r-- 1 root root 739M Mar  8 17:26 kernel-source-5.10.21.tar.lrz

Toutatis · 03-08-2021, 04:02 AM

The compression ratio 1.341 is ridiculous. lrzip is supposed tu use LZMA compression.
For very large files, lrzip should be slighly better than xz
It does not even change with the '-z' option (with zpac, maximum compression).

chrisretusn · 03-08-2021, 04:09 AM

Quote:

Originally Posted by Toutatis

The compression ratio 1.341 is ridiculous. lrzip is supposed tu use LZMA compression.
For very large files, lrzip should be slighly better than xz
It does not even change with the '-z' option (with zpac, maximum compression).

Based on what?

I found this article just now.
https://www.royfrancis.com/file-compression-use-linux/

Read 'man lzma'

Toutatis · 03-08-2021, 04:19 AM

xz and lrzip are supposed to use the same compression algorithm (LZMA). It is why I don't understand why lrzip compressed files are 6 to 7 larger than xz ones.

Martinus2u · 03-08-2021, 06:56 AM

for starters you can choose different compression levels. But the main point is, lrzip is designed for use cases where you have to compress 100 kernel source trees. LR stands for long range.

https://github.com/ckolivas/lrzip

Toutatis · 03-08-2021, 07:58 AM

So I try with that

Quote:

tar cvf lib64.tar /usr/lib64

ls -l lib64.tar

-rw-r--r-- 1 root root 6540462080 mars 8 14:53 lib64.tar

lrzip lib64.tar

Output filename is: lib64.tar.lrz

lib64.tar - Compression Ratio: 1.299. Average Compression Speed: 60.553MB/s.

Total time: 00:01:43.55

ls -l lib64.tar.lrz

-rw-r--r-- 1 root root 5035625925 mars 8 14:53 lib64.tar.lrz

1.299 : not very much

Toutatis · 03-08-2021, 08:53 AM

There is a problem with new versions (or compilations) of lrzip.

I tried with a file compressed a long time ago with lrzip, uncompress it and compress it with new lrzip. And compare

Quote:

ls -l FreeMat-4.2-Source.tar.lrz
58128707 FreeMat-4.2-Source.tar.lrz

lrzip -d FreeMat-4.2-Source.tar.lrz
Output filename is: FreeMat-4.2-Source.tar
Decompressing...
100% 261.58 / 261.58 MB
Average DeCompression Speed: 87.000MB/s
Output filename is: FreeMat-4.2-Source.tar: [OK] - 274287616 bytes

rm FreeMat-4.2-Source.tar.lrz

lrzip FreeMat-4.2-Source.tar
Output filename is: FreeMat-4.2-Source.tar.lrz
FreeMat-4.2-Source.tar - Compression Ratio: 3.047. Average Compression Speed: 43.500MB/s.

ls -l FreeMat-4.2-Source.tar.lrz
90004750 mars 8 15:44 FreeMat-4.2-Source.tar.lrz

not the same. Compress ratio of the old file : 4.719
Compress ratio of the new file : 3.047

Martinus2u · 03-08-2021, 02:48 PM

what if you choose the highest compression level? if you found a regression, let the author know.

Petri Kaukasoina · 03-08-2021, 03:02 PM

Code:

Mon Mar  8 19:19:03 UTC 2021
a/lrzip-0.641-x86_64-1.txz:  Upgraded.
  This update fixes the poor compression ratio reported by Toutatis.

Martinus2u · 03-08-2021, 03:12 PM

chrisretusn · 03-09-2021, 02:32 AM

Code:

lrzip-0.641

Critical bugfix for broken lz4 testing which would prevent secondary
compression from being enabled.

Before:

Code:

xz -d kernel-source-5.10.21-noarch-1.txz
lrzip kernel-source-5.10.21-noarch-1.tar 
Output filename is: kernel-source-5.10.21-noarch-1.tar.lrz
kernel-source-5.10.21-noarch-1.tar - Compression Ratio: 1.402. Average Compression Speed: 24.625MB/s.
Total time: 00:00:39.16

After:

Code:

xz -d kernel-source-5.10.21-noarch-1.txz
lrzip kernel-source-5.10.21-noarch-1.tar 
Output filename is: kernel-source-5.10.21-noarch-1.tar.lrz
kernel-source-5.10.21-noarch-1.tar - Compression Ratio: 8.215. Average Compression Speed:  5.629MB/s.
Total time: 00:02:54.36