Quote:
Originally Posted by pbwtortilla
Important Note: 6 of the drives are connected to two Sil3114 SATA controller cards whilst 2 of the drives are connected to the on-board SATA controller (I don't know which model it is).
|
That's quite a way to destroy the redundancy of a RAID. RAID6 tolerates failure of two drives, but you've got three drives on at least two of the controllers. So one controller failure = destruction of data.
Fortunately, it was your on-board controller that had the problem, though, and that only had two devices on it to fail.
Quote:
Originally Posted by pbwtortilla
At the time, not knowing the cause of the sudden RAID failure, I attempted to force mdadm to start the arrays anyways (the RAID 1 arrays with 8 members each were no causes for concern, of course, but I wanted to back up my data on the degraded md3 array as soon as possible)
|
I guess now that the stupidity of such a decision (to force past error checks when you "need" to back up the data on the other side of them) has occurred to you.
The first thing anyone should do if they suspect data is at risk is STOP, THINK and possibly turn stuff off until they've read up on everything they need to know to attempt recovery.
Quote:
Originally Posted by pbwtortilla
Sure enough, after moving all 8 drives to the Silicon Image controllers, the drives were all recognized without any problems.
|
I hope this is just a temporary arrangement to recover the data.
Quote:
Originally Posted by pbwtortilla
I know that as soon as I re-add the two missing drives back into the md3 (RAID 6) array, the system will attempt to rebuild the array, using the data from the 6 drives.
|
Correct. Fortunately you chose RAID6 and only had two drives on the failed controller or you could have just lost the entire array before you got to do anything silly anyway.
Quote:
Originally Posted by pbwtortilla
Given the size of the array and the type of the disk drives being used (off-the-shelf SATA drives with bit error rate of 1 out of 10^14 bits), I think it is highly likely that the system will encounter one or more bit errors during the rebuild.
|
Why? Do you buy SATA drives that are notoriously unreliable? A rebuild affects only the degraded drives - the other drives are used mainly for read operations to generate the missing parity data. By this logic, anything you do will be prone to bit errors. Yes, you have 2.9 x 10^12 (roughly) bits of data but it's spread across six drives, thus the failure rate is relatively low. Just *building* the array initially was, by these numbers, quite likely to generate a bit error in it. The drives are built to cope with this, with ECC, spare sectors etc.
Your only alternative now is to image all the RAID6 data to another set of drives/image files as a backup and then perform exactly what you intend to do now - an array rebuild.
I would HIGHLY suggest that you do this. You can even make the RAID rebuild from a file image of a drive partition if necessary (the beauty of the "everything is a file" idea in Unix). This way, you can store an image of those RAID6 partitions on a computer somewhere and see WHAT WOULD HAPPEN if you were to rebuild the RAID with that data/parity before you actually mess about putting those disks/controllers into a machine.
Quote:
Originally Posted by pbwtortilla
1. If mdadm encounters a bit error during a RAID 6 rebuild, will it just give up on that particular file and move on to recover other data on the array? Or will it trash the entire array?
|
It doesn't work on files, it knows nothing about them. It works on the bit-level.
It's quite likely that it will abandon an array rebuild as soon as it encounters a problems. It's also quite likely that the force option (which you are only SUPPOSED to use in such circumstances where you have no choice to get working data back) will let you ignore those errors and continue the rebuild, which could potentially leave you with either a corrupt RAID (if you've removed all the locations of the parity data, etc.), corrupt filesystem (if the error hits in the file system itself), or a corrupt file or two (if the error hits inside a file's data).
Quote:
Originally Posted by pbwtortilla
Is it possible to cheat mdadm by somehow replacing the new "raid metadata" on the 6 drives with the old data on the 2 drives? Will it make mdadm think the array is clean, consistent and nothing ever happened?
|
ARGH. No, no, really don't play any more with the bits on these disks. "Let's just write some unrelated data to the drives and hope the RAID copes..." - this is just the same as accidental corruption and it will do the same thing - error, or be forced to attempt recovery. If you DO fool it into thinking everything is okay, the chances are that everything WOULDN'T be okay, and then you have an inconsistent metadata/disk problem which is a million times worse than the odd corrupt bit.
Read (DO NOT MOUNT WITH THE WRITE OPTION) the data off those RAID6 partitions onto a large harddrive or existing filesystem (e.g. dd if=/dev/sda3 of=/home/user/data/sda3-image ), power down all those drives and see what happens when you try to rebuild the RAID from those file images (or make sure the file images are 100% safe and then try to rebuild the RAID from the drives themselves).
And in future... BACKUP. To a non-disk medium. RAID is USELESS against file/disk/controller corruption. It is USELESS against unreliable hardware. RAID *cannot* compensate for deliberate missing with its metadata. It is USELESS against hardware which degrades past its stated tolerances (e.g. three drives failing in a RAID6 etc.). RAID6 is USELESS against failures of more than a single disk while it's rebuilding.
Personally, I'd go out, buy a couple of the largest hard drives I can find and put images of the RAID6 partitions on BOTH of them. Then I'd stick one drive back in the box and put it somewhere safe, and make the other drive the ONLY drive in a machine. Then I'd attempt a RAID6 recovery on those image files and see what happens. If it all goes well, I'd power on the original machine, wipe out it's RAID6 and build a new empty one, then copy the data (NOT THE REPAIRED FILE IMAGES) from the recovered array (making sure that the other surviving copy of the original disks, that disk I put back in it's box, was kept very safe).