LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 07-18-2008, 05:08 PM   #1
DeepSeaNautilus
Member
 
Registered: Jul 2008
Posts: 65

Rep: Reputation: 15
Estimating the size of a tar file


Hello, I am making a script to backup all my information on the server and send it to my email. I tried recently, but it didnīt let me because the resulting targz file was bigger than the maximum file size allowed. In order to solve that problem I figured out on making several targz files of a predefined size.
How can I find out or estimate the size of the tar file of that directory before taring it?
 
Old 07-18-2008, 06:01 PM   #2
stress_junkie
Senior Member
 
Registered: Dec 2005
Location: Massachusetts, USA
Distribution: Ubuntu 10.04 and CentOS 5.5
Posts: 3,873

Rep: Reputation: 335Reputation: 335Reputation: 335Reputation: 335
Each type of file will compress differently. That is to say that ASCII files will compress very well while JPEG file will not compress at all. So if you know how much space is occupied by each type of file then you can make a formula. For example: (space of ASCII files)/12 + (space of JPEG files) + (space of binary executables)/2.

Or you could use the multi-volume feature of tar which will split the tar file up into parts of a size that you specify. So you could just tell tar that the maximum file size of a partial tar file is 12 MB and it will split the tar output into 12 MB files.

Or you could make one large backup file and use the split utility to split it into segments of a size that you specify. It makes more sense to force tar to split up the archive file, though, so that you don't have to recreate the single archive file in order to restore files from the archive.

Of course it's a little bit suspicious that you want to use email to transport the files to their destination. I hope that you aren't copying things from work. Use your power for good and not for evil.

Last edited by stress_junkie; 07-18-2008 at 06:06 PM.
 
Old 07-18-2008, 06:02 PM   #3
matthewg42
Senior Member
 
Registered: Oct 2003
Location: UK
Distribution: Kubuntu 12.10 (using awesome wm though)
Posts: 3,530

Rep: Reputation: 65
Not easily, at least not with any accuracy. Gzip compression ratios vary very widely depending on what is to be compressed. English text can be compressed a lot - the compressed size might be 10% of the original. However, mp3 or jpeg files won't get compressed very much, if at all.

Why do you want to know. Why not just make the file, and then use split to chop it into chunks, and then cat the together at the other end?
 
Old 07-18-2008, 06:02 PM   #4
matthewg42
Senior Member
 
Registered: Oct 2003
Location: UK
Distribution: Kubuntu 12.10 (using awesome wm though)
Posts: 3,530

Rep: Reputation: 65
stress_junkie you beat me to it.
 
Old 07-18-2008, 06:04 PM   #5
stress_junkie
Senior Member
 
Registered: Dec 2005
Location: Massachusetts, USA
Distribution: Ubuntu 10.04 and CentOS 5.5
Posts: 3,873

Rep: Reputation: 335Reputation: 335Reputation: 335Reputation: 335
Quote:
Originally Posted by matthewg42 View Post
stress_junkie you beat me to it.
I hate when I'm the one to post second. Oh well. I guess it happens to us all once in a while.
 
Old 07-18-2008, 06:33 PM   #6
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 63
Its even more difficult than this. Attachments will need to be MIME-encoded, so will be inflated by a certain percentage during this process. Be sure to take this into consideration.
 
Old 07-19-2008, 06:03 AM   #7
eddiebaby1023
Member
 
Registered: May 2005
Posts: 378

Rep: Reputation: 33
Generate the file anyway, then you'll know how big it is and if it's too big you can use split(1) to send it in pieces.
 
Old 07-19-2008, 06:31 AM   #8
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 244Reputation: 244Reputation: 244
Note the maximum file size, use tar to create multivolume, each volume less than maximum file size. Send.
 
Old 07-19-2008, 09:29 PM   #9
DeepSeaNautilus
Member
 
Registered: Jul 2008
Posts: 65

Original Poster
Rep: Reputation: 15
Thanks to all of you for your information. I didnīt know tar command has a multi volume feature and it would certainly be very useful to me, would you tell me how to use it?
I prefer this rather than using the split command because I donīt have to restore small files to get the original one. I am also intrested in this because I want to make a zip utility in the future just to find out how things work.
I know my question might sound suspiciuous, but Iīm a trainee in IT and I have a lot of hard work and information on the server of all my subjects and I have wondered that it is possible that my information could be unavailable or accidentaly deleted. Anyway, due to permision restrictions I can only backup my own information.
Thanks
 
Old 07-19-2008, 09:36 PM   #10
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 244Reputation: 244Reputation: 244
see tar's man page. Also google for "tar multi volume". you should be able to find something useful
 
Old 07-20-2008, 08:44 AM   #11
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,369

Rep: Reputation: 2753Reputation: 2753Reputation: 2753Reputation: 2753Reputation: 2753Reputation: 2753Reputation: 2753Reputation: 2753Reputation: 2753Reputation: 2753Reputation: 2753
You might want to look at rsync, which will stream your data across, then on subsequent uses, only sends the changes ie a much smaller amt.
 
Old 07-20-2008, 09:24 AM   #12
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 244Reputation: 244Reputation: 244
if i am not wrong, OP needs to send to his email.
 
Old 07-20-2008, 08:44 PM   #13
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,369

Rep: Reputation: 2753Reputation: 2753Reputation: 2753Reputation: 2753Reputation: 2753Reputation: 2753Reputation: 2753Reputation: 2753Reputation: 2753Reputation: 2753Reputation: 2753
I was wondering whether he NEEDS to use email or just did that way because it seems the obvious/easy way if he hasn't messed much with this level of usage before.
Depends where the target system is...
 
Old 07-21-2008, 09:14 AM   #14
rubadub
Member
 
Registered: Jun 2004
Posts: 236

Rep: Reputation: 33
Quote:
Originally Posted by ghostdog74
see tar's man page
use the info page instead for tar, you'll find out why...


Something to consider, tar will increase the initial file size. Using the standard block size (512), will produce a 512b header for each file or directory and will round each filesize upto 512. Then it will also want to be a minimum of a multiple of blocks, which if I remember correctly is 20 as default...

For example if you have a single file of 3 bytes:
header: 512
file: 512
= 1024
yet it must be a multiple of 20 blocks, so fill with zeros until 10240 bytes.

result = 10240 bytes
 
Old 07-21-2008, 12:21 PM   #15
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 244Reputation: 244Reputation: 244
Quote:
Originally Posted by rubadub View Post
use the info page instead for tar, you'll find out why...
OP should have read the man page by now and know how to go further
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
cannot tar a big size file shipon_97 Linux - Newbie 3 11-18-2007 10:26 PM
tar file size limit sidra Red Hat 4 04-03-2007 12:08 AM
is there a file size limit to tar backups? seanarthur Linux - Server 1 08-22-2006 09:01 PM
gave wrong syntax for tar as tar -cvzf file file.tgz how to recover the file gautham Linux - General 4 04-13-2005 03:15 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 09:14 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration