I'm looking for suggestions -- maybe not doable.

I have a 118G bz2 compressed tarfile on an external USB 2.0 drive. The uncompressed files use 450G. This took 4 hours to restore from the USB drive.

Does that seem reasonable?

That's a fair amt of stuff.
Did you run the untar/uncompress cmd direct against the usb drive?

Depending on the internals of how tar works, it might(?) be quicker overall to just stream the file onto the local hard disk first, and then run the uncompress/untar from there (especially if it's SSD) .
It's got to be worth a try, just to see...

1 members found this post helpful.

It amounts to 8M/sec compressed - which is well below the USB 2.0 limit, and 32M/sec uncompressed which by modern standards is pathetic. Of course it depends on what's being restored and where - it could be ok if you are restoring a gazillion of small files or writing to SMR spinner, and there are many reasons for poor SSD write performance, usually not happy ones.


tar stands for Tape ARchive, it is specifically designed to work with sequential streams

there can be other factors too, like free memory or fragmentation, type of filesystem, ... but in general you can measure it by:
to check the read speed. Additionally you can try the following:
to check the decompression speed.
lbzip2 and pbzip2 are both faster than the original bzip2.
And finally you can also check the write speed of your disk.

Yes, directly against the USB drive.
I'm going to try this idea of first copying the backup to the drive first ... as soon as I set up a machine with a similar RAID-1.
Not a gazillion small files. There are half-a-dozen small files and one really large 144G (uncompressed) virtual machine .vdi image.
These tarfiles are intended for long term backup and seem like the right choice for such backups. Better/faster "boutique" programs might not work years from now depending on what flavors of Linux things might evolve to. tar ships with all distros and likely will be forever. But, I'm open to suggestions on alternatives.
Excellent suggestions. I might give those a try after trying chrism01's suggestion.bzip2 is specified in tar by -j. I don't see options (in my distro's tar) for lbzip2 or pbzip2. Is there such an option? I have lbzip2, but not pbzip2. Without a tar option for lbzip2 I'd have to compress through lbzip2 as an extra step which might confuse future sysadmins not knowing how the tarfile was compressed. See "boutique" comment above. 'tar -x' figures out the bzip2 compression (and other tar-native compressions) and you don't have to specify 'tar -xj'. [lp]zip2 would have to offer significant speed improvements to make the effort worthwhile, and more "sophisticated" compression algorithms might actually take longer to uncompress.

you need to use tar -I <program> instead of tar -j. Check the man page.

1 members found this post helpful.

Some of the extreme compression levels might do that. Some compression types need support in hardware for best performance.

Seems a bit long. In theory tar is a file by file stacked one after another so the read ought to be fairly fast.

If the host needs to both read and write to the usb you are then looking at the 4 hours.

Yes, I know ... That wasn't my point ...

I just finished restoring these same files to a new rig with a newly created/assembled RAID-1. the .bz2 is 118G and the uncompressed files use 450G. I first copied the .bz2 tarfile from the USB drive to the target hard drive, then untarred from there. It took 2 hours and 51 minutes to restore, much better than the 4+ hours described in post #1 (which was not a RAID). And, this restore to a slower computer with less memory, though I don't really know how the SATA transfer rate compares.

So the conclusion is to copy the tarfile from the USB to the hard drive first, then untar.

I'm going to try these restores again, possibly this evening, to the original machine.

you don't really need to copy, it is unnecessary. You can immediately untar/uncompress from the usb drive too. At least you can try it and compare the speeds. If you wish.

Does not make sense.
Untaring from another physical drive is faster - less stress on the target drive.
And a RAID1 speeds up reading but not writing.

If you notice from my original post, that exactly what I did and it too 4+ hours to restore the same tarfile. chrism01 suggested copying to the hard-drive first in post #2, and that did prove faster. Probably because the USB is slower than the hard-drive.

you would need to sum up all the times (copy, uncompress, untar, whatever). From the start (you have a backup on an usb drive) to the end (restore is done). And you will see which one is faster.

if i had a case like this and the spare time to explore where performance might be limited:

1. i would check to see what file system type 118G backup.tar.bz2 is on. is it formatted as ext4? some other file system type? none at all (reading a raw partition)?

2. i would try to see just how fast a process can read the backup.tar.bz2 stream, not decompressing it while watching the drive activity light(s) to see if it stays solidly busy or if not, just how busy it is. of course i would time this activity.

3. i would also be running the top command to see how CPU busy this is.

4. i would do #2 and #3 again but this time with decompression active to see how much this slows it down.

5. i would do a regular restore into a freshly formatted file system as a base reference. i would be sure not to have both the source backup.tar.bz2 and the target partition on the same device if it is an older spinning platter device to avoid head movement contention. i have not studied how things like this might work on SSD but worrying about how they spend time to clear space to make way for new bits, i'd still use different devices just to be sure. i'd still prefer SSD over rotators for the speed. if i was really limited to one device (such as because only one connection), i'd go with SSD.

Those are all good ideas, but I don't know if cost effective time-wise since, at best I think we're talking a matter of 1/2 hour-ish among more optimal techniques and this is not something I need to do routinely.
Right, In my 2 hours and 51 minutes time to restore I did not include the time to copy from the USB to the local hard drive before untarring. I timed that with a USB 2.0 port, but not to the same target machine (not available). That took a little over 16 minutes both for rsync and cp. So the total time for copy-from-USB/restore-from-disk would be 3 hours, 7 minutes, still much better than uncompressing/restore from USB.

Skaperen's proposed set of test would probably shed more light on this, but I think I'm satisfied that copying from USB to hard-drive first, then uncompressing/restoring is about 25% faster than uncompressing/restoring the USB resident tarfile.