What is the best compression method for huge folders?

swalker23 · 04-14-2009, 12:57 PM

I need to format my backup storage HDD, it contains my library of anime, family movies, and other lil stuff I keep on it. The HDD is 1TB in size and I'm using roughly 700 gb. I need to compress some folders so I can store it on my linux HDD(640gb) while it is being formated. With linux I can use many of the compression methods but I don't know which one will give me the best results for huge folders. So which method would be the best for me?

rweaver · 04-14-2009, 01:13 PM

Quote:

Originally Posted by swalker23

I need to format my backup storage HDD, it contains my library of anime, family movies, and other lil stuff I keep on it. The HDD is 1TB in size and I'm using roughly 700 gb. I need to compress some folders so I can store it on my linux HDD(640gb) while it is being formated. With linux I can use many of the compression methods but I don't know which one will give me the best results for huge folders. So which method would be the best for me?

It really depends, but if you're talking raw compression ratio bzip2 offers a high level of compression and works relatively fast. Are you going to be taring the directories up? Do you need to access individual files in the tarballs? If both answered yes then I would change my answer from bzip2 to perhaps zip running at highest compression ratio.

But for raw compression ratio, bzip2 is pretty good.

linus72 · 04-14-2009, 01:15 PM

Don't know if it helps but you could also compress it like a KNOPPIX image-my DSL goes from 200+MB to 48MB after compressing-is that comparable to bz2?

swalker23 · 04-14-2009, 01:18 PM

I'm not going to be accessing the files through the compression. Just gonna compress them, move them to my linux HDD, format backup HDD, and move/uncompress them back to the backup HDD. Thanks for the reply I'll use the bzip2 method and will tar up the folder since I have directories in them.

eco · 04-14-2009, 01:18 PM

well, if it's holding movies and pictures, you might get very minimal compression ratio as these are already compressed formats.

Still, have a look at bz2 or rar

Code:

$ tar jcvf myLargeZipFile.tar.bz2 /mnt/mydisk

or something like

Code:

$ rar a -m5 -v2048000b myLargeZipFile.rar /mnt/mydisk

-v will split the rar files to avoid one large file.
Check the syntax though

taxtropel · 04-14-2009, 01:28 PM

Unless you have a lot of text files (very few binaries) then you are not going to fit 640GB + 700GB (compressed) on to a 1TB drive. Your best bet is to back up your files (/home) and your configuration files (/etc) and anything else you cannot replace on your linux drive. Then reinstall linux after the format.

That being said. Don't take my word for it; try some archiving programs out.

Basic archiving programs commonly used are:

Tar
Cpio

The 4 main compression types are:

Zip (plain'ole Zip files)
GZip
XZ-Utils (lzma)
BZip2

If you want to compress a folder with tar and bzip2 then use:

Quote:

tar -cfj my_archive_name.tar.bz2 /folder

swalker23 · 04-14-2009, 01:34 PM

Quote:

Originally Posted by taxtropel

Unless you have a lot of text files (very few binaries) then you are not going to fit 640GB + 700GB (compressed) on to a 1TB drive. Your best bet is to back up your files (/home) and your configuration files (/etc) and anything else you cannot replace on your linux drive. Then reinstall linux after the format.

That being said. Don't take my word for it; try some archiving programs out.

Basic archiving programs commonly used are:

Tar
Cpio

The 4 main compression types are:

Zip (plain'ole Zip files)
GZip
XZ-Utils (lzma)
BZip2

If you want to compress a folder with tar and bzip2 then use:

You must have miss understood me. My backup HDD is 1tb and I'm using 700 gb of that. I want to compress the 700gb so it will fit on my 640gb HDD which linux is installed on so I can format the 1tb HDD.