ZFS Storage System Design Advice Needed

kayasaman · 09-02-2012, 03:50 PM

Hi,

currently I am running FreeBSD 8.2 x64 as a mainframe server/storage solution with 12TB on it. I actually needed to run FreeBSD due to Mini-ITX system board with only 4GB RAM and also Ports collection as I run too main FreeBSD based Jailed services (rev proxy, dns, ntp, ftp, tftp, nfs...etc).

The root drive, a 40GB SSD runs UFS2 while I have 2 ZFS - RAID 0 storage pools on the machine split across 2x 2TB and 2x 4TB drives.

Of course this doesn't give me the latest version of ZFS and I haven't used the ZFS file system at all just storing data on raw ZFS pool; also not good for resilliancy as currently one of my drives has semi-exploded coming up with 'DMA READ errors'....

FreeBSD aside...

I have recently bought a 3U NAS Storage enclosure with internal 2.5" drive support and 16x 3.5" hotswap drive capability.

My aim is to use a SuperMicro system board inside it and run Solaris 11 as storage solution. Due to latest production ready ZFS version, encryption and overall features.

I have a few questions however relating to the design of the system:

1. is it better to run a ZPOOL spanning all drives in RAIDZ 5/6 format then create individual file systems or is it better to create a bunch of smaller pools, combination of ZFS RAID1/RAIDZ 5/6 depending on category eg. Documents, Music, Pictures, Movies, TV Recordings etc....?

2. If run smaller ZPOOL's with ZFS filesystem on top how much space is needed for snapshots - actually where do they get stored on ZPOOL or on root fs?

3. Recovery and Backup - my answer to backup is always mirror the system or use a suitable alternative (box with a bunch of DAS/JBODs attached and rsync across - automate using cronjobs

- However recovery is where things get a bit blurred... a conventional filesystem like UFS1/2 or EXT2/3/4 can easily be dd'd or mounted on a seperate machine then using various tools like fsck can be recovered (providing error isn't bad enough). Or even remote mounted in read-only format and data extracted from the drive using rsync or similar. How does one manage to do this with a ZPOOL as to my knowledge they can't be remotely mounted unless imported but for that the pool needs to be exported?
- in the above situation if something happens to the root drive how does one recover the whole pool?

Thanks for any responses, the above are simple questions and concerns as I am a huge ZFS fan though never really had the hardware to be able to use it properly uptil now. - I also tried introducing a Sun Storage array at work but it got blown off in favor of NetApp though we are currently using some clunky Hp SAN system which is reliable at least but again totally corporate and not really dynamic enough for me to sit infront of and actually 'learn' about how things work to get a better understanding of different situations and techniques.....

vermaden · 09-03-2012, 12:17 AM

Quote:

Originally Posted by kayasaman

My aim is to use a SuperMicro system board inside it and run Solaris 11 as storage solution. Due to latest production ready ZFS version, encryption and overall features.

You can have encryption with ZFS on FreeBSD with GELI.

Quote:

1. is it better to run a ZPOOL spanning all drives in RAIDZ 5/6 format then create individual file systems or is it better to create a bunch of smaller pools, combination of ZFS RAID1/RAIDZ 5/6 depending on category eg. Documents, Music, Pictures, Movies, TV Recordings etc....?

Do not put more then 7-8 disks into a ZRAID1/ZRAID2. Stripe the ZRAIDs together, for exameple, for 13 disks, do 2 * RAIDZ2 and put one drive into hot spare, that way You have RAID0 of 2 * ZRAID2 with 1 hot spare.

Quote:

2. If run smaller ZPOOL's with ZFS filesystem on top how much space is needed for snapshots - actually where do they get stored on ZPOOL or on root fs?

Snapshot takes that much space as You changed the data there, for example You created snapshot, deleted 10MB file so snapshot now holds 10MB data. Snapshots are kept on the same dataset as they were created.

Quote:

3. Recovery and Backup - my answer to backup is always mirror the system or use a suitable alternative (box with a bunch of DAS/JBODs attached and rsync across - automate using cronjobs

Use ZFS snapshots and ZFS SEND and ZFS RECV.

Quote:

- However recovery is where things get a bit blurred... a conventional filesystem like UFS1/2 or EXT2/3/4 can easily be dd'd or mounted on a seperate machine then using various tools like fsck can be recovered (providing error isn't bad enough). Or even remote mounted in read-only format and data extracted from the drive using rsync or similar. How does one manage to do this with a ZPOOL as to my knowledge they can't be remotely mounted unless imported but for that the pool needs to be exported?

ZFS does not need and old archaism like FSCK. About DD, it works the same under ZFS.

Quote:

- in the above situation if something happens to the root drive how does one recover the whole pool?

In FreeBSD You would just use live CD, but I havent checked if Solaris 11 also hase live version.

jlliagre · 09-03-2012, 01:28 AM

Some comments in addition to vermaden's answer.

- I would create a single pool with 2 or 3 RAIDZ plus SSDs for ZIL and L2ARC and a spare disk.

- As already stated, ZFS usually need no recovery or fsck as the filesystem either consistent or unavailable. Should for some reason, the pool has been corrupted anyway, Solaris allows importing a pool in recovery mode.

- It is recommended to have the root pool mirrored for easy recovery should a disk fail. In any case, you can boot on the Solaris 11 live media should your root pool doesn't boot.

kayasaman · 09-11-2012, 02:26 PM

Thank you very much for your responses!

Sorry for the delay in mine I have been quite ill recently so wasn't able to get back sooner

Quote:

- I would create a single pool with 2 or 3 RAIDZ plus SSDs for ZIL and L2ARC and a spare disk.

This is what I was thinking too, using SSD's for 'logging' and 'caching'.

With single pool just to be clear:

For a 16 drive based system for example sake:

1 x pool with RAIDZ2 meaning

disk1 + disk2 ... disk8

Mirror that with:

disk9 + disk10 ... disk16

then add different file systems on that, eg:

Documents
Multimedia
OS
TFTP
etc...

and add 1x SSD for logging and another for cache.

I guess for command reference would be something like:

Code:

zpool create POOL_1 raidz2 disk1 disk2 disk3 disk4 disk5 disk6 disk7 spare disk8 cache ssd_disk1

zpool create POOL_2 raidz2 disk9 disk10 disk11 disk12 disk13 disk14 disk15 spare disk16 cache ssd_disk2

Create file system:

Code:

zfs create ZPOOL_1/Documents
zfs create ZPOOL_1/Multimedia
zfs create ZPOOL_1/other

Then using something like rsync in a cronjob:

Code:

rsync -avr --inplace --progress /ZPOOL_1/* /ZPOOL_2/

jlliagre · 09-11-2012, 03:36 PM

Depending on the number of files to synchronize, "zfs snapshot; zfs send | zfs receive" might be much faster than rsync.