[SOLVED] Raid Repair Now wont boot - Other mounting problems

Ser Olmy · 05-20-2012, 06:55 PM

I would forget about /dev/md3 for now. You can revisit that problem if you get the server up and running. Otherwise, you'll need on-site support anyway.

bigstack · 05-20-2012, 07:10 PM

md3 is where the data i need is

.. crap Wonder what happens if I try to boot now

Ser Olmy · 05-20-2012, 07:15 PM

Quote:

Originally Posted by bigstack

md3 is where the data i need is

.. crap Wonder what happens if I try to boot now

You should try that now. If it works, the "mkfs (- dump2fs) - fsck -b" procedure may actually fix your /dev/md3 file system.

(Assuming you don't actually run mkfs on /dev/md3, but use the -n option.)

bigstack · 05-20-2012, 08:34 PM

OK because I am fearless at this point and seeing the light at the end of the tunnel I did you fsck.ext3 -b 7962624 -p -y /dev/md1 on the md3 (changing the ###### to one that was found by running mke2fs -n /dev/md1 and getting the list of super block locations

I added the -C to the line so i could see progress was happening. the end line was fsck -C -b 4096000 /dev/md3 i dont mind holding down the enter key i just feel safer not telling it to yes its way through anything. here is the current status

Code:

root@rescue:~# fsck -C -b 4096000 /dev/md3
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
/var was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
/var: |=====================                                   \ 38.0%

Ser Olmy · 05-20-2012, 08:51 PM

Quote:

Originally Posted by bigstack

OK because I am fearless at this point and seeing the light at the end of the tunnel I did you fsck.ext3 -b 7962624 -p -y /dev/md1 on the md3 (changing the ###### to one that was found by running mke2fs -n /dev/md1 and getting the list of super block locations

I hope you mean the superblock location you found by running "mke2fs -n /dev/md3", but that's probably just a typo.

bigstack · 05-20-2012, 08:53 PM

Yes that was a type

Anyway it finished this as end result

Code:

/var: ***** FILE SYSTEM WAS MODIFIED *****
/var: 46848/117620736 files (15.3% non-contiguous), 174370132/470457584 blocks

AND IT MOUNTS ... Starting Backup (need to find a place to send it)

Once i backup this 700GB of stuff should i try to reboot ? Now that we have disabled the bad drive it should boot? Also check ur PM.

Ser Olmy · 05-20-2012, 09:02 PM

It should boot, but you never know; you've (ab)used the partition as swap space, albeit for a very short time and under extremely light load.

The system may have written data to the partition, but then again it may not. The kernel file and essential system binaries may be OK, or perhaps one or more of them are corrupt. GRUB may be able to find the config file, or it may find garbage and throw a fit.

There's only one way to find out, really.

bigstack · 05-20-2012, 09:57 PM

OK I will update this thread tomorrow well like 12 hrs once the backup is done and I attempt to boot it from the normal HD instead of the netboot.

The waiting game

bigstack · 05-21-2012, 04:09 AM

Update:

Booted Server and it came up just fine with the 1 drive. Now I just need to get that replaced? Will the Raid AUTO config / sync the data or will I need to do something? I set the last drive to fail do i need to enable?

Ser Olmy · 05-21-2012, 10:37 AM

Once the drive has been replaced, you need to partition the new drive and add the new /dev/sdb1 and /dev/sdb3 partitions to md1 and md3 respectively. Then you should initiate a rebuild.

May I also suggest you create another md device out of /dev/sda2 and /dev/sdb2 for use as a swap partition? Your current setup with separate swap partitions on both drives is very vulnerable to disk failures. Should one of the drives fail, you're likely to get a server crash (kernel oops). I suspect that's exactly what happened in this case.

bigstack · 05-21-2012, 11:56 AM

UPDATE about info below: I googled some more and found this guide: http://www.howtoforge.com/replacing_...#comment-31477 it tells me to use --Linux in the line i guess it foreces it? I believe this is the solution but want to bounce it off someone first.

OK im on the rebuild part.. however that guide seems to have a problem? it wants me to remove the working disk from the array?

OVH Directed me toward a guide which is here:
http://help.ovh.co.uk/RaidSoft
and then I found an even better guide HERE:
http://www.kernelhardware.org/replac...ed-raid-drive/

However when i run the copy partition line over i get

Code:

root@rescue:~# sfdisk -d /dev/sda | sfdisk /dev/sdb
Checking that no-one is using this disk right now ...
OK

Disk /dev/sdb: 243201 cylinders, 255 heads, 63 sectors/track

sfdisk: ERROR: sector 0 does not have an msdos signature
 /dev/sdb: unrecognized partition table type
Old situation:
No partitions found
New situation:
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sdb1   *      4096  81922048   81917953  fd  Linux raid autodetect
/dev/sdb2      81922049 143360000   61437952  82  Linux swap / Solaris
/dev/sdb3     143360001 3907020800 3763660800  fd  Linux raid autodetect
/dev/sdb4             0         -          0   0  Empty
Warning: partition 1 does not end at a cylinder boundary

sfdisk: I don't like these partitions - nothing changed.
(If you really want this, use the --force option.)

B got replaced so were trying to clone A to B... Now In that readout there is sdb1 - 4 A only has 1-3 is something wrong? Should I use the force option? Or do i need to low lever format this disk before I can do that guide.

bigstack · 05-21-2012, 07:21 PM

Ok so the raid has completed the rebuild. And Rebooted the Server in NORMAL mode and It is NOW WORKING !!!

Thanks for all the help Ser Olmy