Incremental backups?

arashi256 · 07-01-2009, 09:36 AM

I've got our shiny new Samba server (well, not really shiny or new - I built it out of bits from our storeroom and threw in a 500GB drive) fired up. Anyway....I need to backup the Samba share directory (and all sub-directories) to an external drive - which is mounted - once a month. This bit is easy. The hard bit is that the boss wants incremental daily backups of files that have changed in this directory tree to be copied to a daily sub-directory named with todays date on the external drive.

Can I use rdiff-backup for this? Or something else? Or some sort of scripting? The directory struture is rather complex. If there is no other way, can I use Bash or Perl or something to cycle through directories and compare the file timestamp on the Samba share to the one on the same file on the backup drive and copy it that way?

What is the easiest way to achieve this?

Thanks for any advice.

JulianTosh · 07-01-2009, 09:42 AM

BackupPC is the most awesome backup software (linux of course) I've used to date. Very easy install. Very easy maintenance and configuration. Restore process couldnt be easier.

Love it!

arashi256 · 07-01-2009, 09:48 AM

Thanks for the reply - I'll look into BackupPC but ideally (sorry, forgot to mention this) I'd like a command-line solution.

xeleema · 07-01-2009, 01:14 PM

Greetingz!
If you're looking for something you can dump into a script and then slip into a crontab, you may want to consider a cp/rsync/ssh trifecta;

First, you want to use the "find" command to display and create a list of all files modified within the past 24hrs;

$ find /directory/to/backup -type f -mtime -1 -print |\
> tee ~/filelist.wri

Create a directory to store the daily backup;

$ mkdir /path/to/backup_directory/backup-`date +%a`

(NOTE: the date command is surrounded by "backticks", not single-quotes. The end-result is a directory named similarly to "backup-Wed". See the "date" manpage for more tricks.)

Now copy the files listed in our filelist.wri file into the backup directory;

$ for filename in $(cat ~/filelist.tmp)
> do
> cp "$filename" /path/to/backup/directory/backup-`date +%a`
> done

(NOTE: if you're doing this in a script you may want to set a Variable for "backup-`date +%a`". Otherwise, if the above for-loop is started before midnight, then right after the stroke of midnight, all of the remaining copies will fail)

Now, you'll have to wipe out the previous "Wednesday" backup before you do this, else you're just going to add files into that directory.

So all in all, you might have a script that looks something like this

Code:

#!/bin/sh
## Incremental Backup - Script that copies whole files to a separate
##    directory for safe keeping.  Only keeps seven days of files.
######################################################
## Set the Day (Mon, Tues, Wed, Thurs, Fri, etc, etc)
##   NOTE: if you *really* want all lower-case letters
##   in the date, comment out the short ${DAY} and uncomment
##   the second version.
DAY=`date +%a`
#DAY=`date +%a | tr '[:upper:]' '[:lower:]'`
## Set the full path to the directory that we'll store backups in.
TARGET=/path/to/backup/directory
## Find and wipe out the previous $DAY's backup.
if [ -r ${TARGET}/backup-${DAY} ]; then
   rm -rf ${TARGET}/backup-${DAY}
fi
## Find all files that aren't older than 24 hours
find /directory/to/backup -type f -mtime -1 -print | \
tee ~/filelist.tmp
## Create our backup directory and copy our files over to it
##    Note that we use double-quotes around our $filename because
##    Samba users like to use spaces in their filenames, which
##    freaks out our shells (Thanks MSFT!)
##    Sub-Note - check the manpage for 'cp', "-a" should be
##    defined. If not, use "-dpR".
mkdir ${TARGET}/backup-`date +%a`
for filename in $(cat ~/filelist.wri)
do
cp -a "$filename" ${TARGET}/backup-`date +%a`
done
## Maybe get paranoid, maybe do a little post-backup file
##    integrety checking just in case the boss thinks our
##    backup didn't backup the file correctly (it helps
##    to be able to do a quick 'md5sum -c filelist.md5'
##    just to be sure our data is intact).
for filename in $(cat ~/filelist.wri)
do
md5sum "$filename" | tee ${TARGET}/backup-${DAY}/filelist.md5
done

## Clean up after ourselves
mv ~/filelist.wri ${TARGET}/backup-${DAY}/filelist.md5
rm ~/filelist

## End of File.

arashi256 · 07-01-2009, 01:48 PM

That is fantastic, thank you! A very detailed reply.

xeleema · 07-01-2009, 03:57 PM

You're welcome! Let me know if that works!
(Most of my freehand scripting does, however YMMV).

arashi256 · 07-02-2009, 06:53 AM

I modified it a little bit, but essentially that working brilliantly. I had to tweak it so that it created the destination directory structure as well, as some files that might be modified were in different directories but had the same filename, but aside from that - awesome.

xeleema · 07-02-2009, 08:51 PM

Glad to hear it!

arashi256 · 07-06-2009, 09:27 AM

This version is even better - it only creates the directory structure for newly modified files if there are any files that have been modified (i.e. any files listed in FILE-LIST.tmp), so I don't end up with a bunch of empty directories (had to figure out sed to make this work - nightmare!)

Code:

#!/bin/sh
DAY=`date +%F`
BACKUP_TARGET=/media/LaCie/BACKUP
find /home/sambashare/ -type f -mtime -1 -print > ~/FILE-LIST.tmp
LINECOUNT=`wc -l ~/FILE-LIST.tmp | sed -e 's/[A-Za-z/./-]*//g'`
if [ "$LINECOUNT" -gt "0" ]; then
	mkdir ${BACKUP_TARGET}/backup-${DAY}
	for filename in $(cat ~/FILE-LIST.tmp)
	do
		DIRECTORY=${filename%/*}
		mkdir -p ${BACKUP_TARGET}/backup-${DAY}${DIRECTORY}
		cp -a "$filename" ${BACKUP_TARGET}/backup-${DAY}${DIRECTORY}/
	done
	mv ~/FILE-LIST.tmp ${BACKUP_TARGET}/backup-$backup${DAY}/FILE-LIST.tmp
fi

I'm liking Bash scripting...

xeleema · 07-06-2009, 08:46 PM

"I'm liking Bash scripting..."

The great part is, notice the "#!/bin/sh" at the top of the script? If you keep a close eye on your syntax, you should be able to move that script around to various UNIX distributions. The "sh" in there implies the Bourne Shell (not the Bourne Again Shell, a.k.a Bash), which has been a long running shell for several decades.

One trick to remember, specify the full path to your binaries, you never know what the $PATH environment variable is gonna have. (Also, that's just good practice for scripts that run as root...)

(And I do realize that I didn't do it in my version *sheepish_grin*)

Happy scripting!