snapshots are snapshots, like a photograph ... there is no "initial" snapshot per se.

an initial "send" of the system (which can be from any snapshot) is restored as a whole file system it knows nothing about the past, only what it looks like at the time ... but everything that was there at the time of the snapshot is available in it.

an incremental "send" takes the name of the previous snapshot and the snapshot you want to get too and sends the equivalent of the block changes on the disk ... if only part of a file has changed then only the bit is sent.

Just to note at this point, if you make a "snapshot" on the file system it sits there until you want to "send" it or delete it ... months or years could pass before you do anything with it ... you can access all the existing snapshots at any time you want ... unlike live file systems which change after you've backed them up.

If a client wants the file back at the 14th, you need to get to the 14th ... if the file hasn't changed up till April 1st, then an initial "send" to get that snapshot will get you the file back.

we only do one initial "send" ... and thereafter increment our way through the days ... we don't need another initial "send" if we're up to date.

We looked at DLT, but that scared us ... you needed to push data so fast at it, and it liked concurrent streams ... we stuck with what we knew, but it just couldn't cope with what we needed.

[QUOTE=Skaperen;4348390]For bulk backup, I think hard drives are still better than flash drives. But for bootable rescue systems and installation media, flash drives are now the way to go (more reliable than CDs and DVDs).

If it were to get the capability to do reverse increments, it could be the win for me.

just tried:
jadrevenge@laptop:~$ zfs send -i rpool/export/home/jadrevenge@zfs-auto-snap_daily-2011-05-09-09h08 rpool/export/home/jadrevenge@zfs-auto-snap_daily-2011-05-03-07h50 > /tmp/zfs-send.txt
WARNING: could not send rpool/export/home/jadrevenge@zfs-auto-snap_daily-2011-05-03-07h50:
incremental source (rpool/export/home/jadrevenge@zfs-auto-snap_daily-2011-05-09-09h08) is not earlier than it

so no you can't reverse the order ...


we've had issues ... almost exclusively with the power supplies ...

and yes, USB was not our first choice ...

I wanted ESATA, but that's just not stable enough either.

as I said, this isn't a problem for us because we _always_ restore so we have no issues with increments.

Jon

But that is a "view" within the filesystem itself. If we were not talking of making a backup, then all we are considering is the view in and of itself, within the filesystem. At least that's how I understand it.

This would be the whole filesystem as would normally be seen at the time of the snapshot.

But this is a forward increment, right?

Is this the only thing you can do with a snapshot? Or is it possible to "view" (mount) the specific snapshot in a read-only mode? By comparison, I can do an export from SVN from a specific revision number.

If it is possible to view that snapshot that way, then it could be possible to make a reverse increment from an application that compares both trees. Restoration might still be tricky.

But if I deleted all data up to April 10, then I'm out of luck because if the file had not changed between the oldest "send" still present (April 10) and the desired date (April 14) then the incremental sends would not have any of this file's data. Even if there were changes, only the blocks that were changed would be in these sends.

That's where a reverse increment has the advantage since it is based on the current full filesystem state. To restore a file to an older state, you work backwards from the current state.

These reverse increments are NOT a "restore the whole filesystem when the disk dies" type of backup. They are just historical increments, only. But this is done on the first backup server itself rather than on the working servers directly. So this is the primary backup of the working server, plus a generated archive based on what gets backed up. Then I make a backup of the primary backup server. This is the secondary backup/archive (the archive on the secondary backup server is not generated here, but instead is just a replica of the archive generated by the primary backup server). I can make multiple secondary backup/archive servers.

Right now I have one primary backup server on site and one secondary backup server offsite. Plans include multiple primary backup servers to split up the load (each handling just a configured set of working servers), and 2 or 3 offsite secondary servers for each (near offsite and far offsite).

The file level granularity is working OK for now. Block level granularity might be a plus in the future.

But sliding window deletion of the sent files isn't really possible with this. A delete of an initial send invalidates all the increments after it, up to the next time you do another initial send. If you only did one send, the whole backup depends on keeping it and everything after it.

If I do an initial send on the 1st of each month, then at least I could delete all the March sends when I no longer need any data prior to April 1. But that's a one month granularity of deletions.

What I mean by sliding window deletion, in case it's not understood, is that I decide to keep for example 180 days of increments. The window size is thus 180 days. Each day the window slides forward by one day and the increment that falls off at the far end is deleted. So there is one delete per day. In actuality, it would be more based on space available, so the window can also grow or shrink. This part is not implemented in an automated way, yet. It's down manually for now.

I doubt it could have been that simple to have a reverse increment.

For home (and desktop at work) I am trying out SATA connected hotswap bays that use internal 3.5 SATA drives in a tray. There are 3 drive slots in a frame that fits in 2 external drive bays. So far these are working fine. They use internal drives which are lower in price than external USB drives, are faster than USB, and don't require a bunch of electrical outlets to power them through a tangle of cords. They are a little awkward if you don't have extra trays. But there are now being made similar bays that don't use trays at all.