Recover Data From Striped Logical Volume Group With Failing Drive
Linux - HardwareThis forum is for Hardware issues.
Having trouble installing a piece of hardware? Want to know if that peripheral is compatible with Linux?
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Back in #11, I gave the command for copying the image back to the new drive. Then you can assemble the array, unlock the encryption, and run lvscan to find the LVM logical volumes.
Now when I run lvscan I get this:
Code:
WARNING: Device for PV 2PT2Kv-403c-iO2B-kKAY-B4uV-aMaI-1mlSQI not found or rejected by a filter.
inactive '/dev/VolGroup00/LogVol00' [5.45 TiB] inherit
inactive '/dev/VolGroup00/LogVol01' [5.41 GiB] inherit
Is this saying I'm missing a drive? I have all 7 drives, connected back the way it was.
vgscan gives this:
Code:
Reading all physical volumes. This may take a while...
WARNING: Device for PV 2PT2Kv-403c-iO2B-kKAY-B4uV-aMaI-1mlSQI not found or rejected by a filter.
WARNING: Device for PV 2PT2Kv-403c-iO2B-kKAY-B4uV-aMaI-1mlSQI not found or rejected by a filter.
Found volume group "VolGroup00" using metadata type lvm2
fdisk -l gives this:
Code:
Disk /dev/sda: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0002db59
Device Boot Start End Blocks Id System
/dev/sda1 * 63 208844 104391 83 Linux
/dev/sda2 208845 1953520064 976655610 8e Linux LVM
Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00016492
Device Boot Start End Blocks Id System
/dev/sdb1 * 63 1953520064 976760001 8e Linux LVM
Disk /dev/sdf: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000baca0
Device Boot Start End Blocks Id System
/dev/sdf1 * 63 1953520064 976760001 8e Linux LVM
Disk /dev/sde: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000088fa
Device Boot Start End Blocks Id System
/dev/sde1 * 63 1953520064 976760001 8e Linux LVM
Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0003f54f
Device Boot Start End Blocks Id System
/dev/sdc1 * 63 1953520064 976760001 8e Linux LVM
Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: dos
Disk identifier: 0x0003b67a
Device Boot Start End Blocks Id System
/dev/sdd1 * 63 1953520064 976760001 8e Linux LVM
Partition 1 does not start on physical sector boundary.
Disk /dev/sdg: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/live-rw: 8589 MB, 8589934592 bytes, 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/live-base: 8589 MB, 8589934592 bytes, 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/live-osimg-min: 8589 MB, 8589934592 bytes, 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/luks-a6382836-5a73-42c6-ad78-ee5d4d701c7c: 1000.2 GB, 1000201712640 bytes, 1953518970 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/luks-61681fae-c55b-4d79-ada1-ce7ca497cfb5: 1000.2 GB, 1000201712640 bytes, 1953518970 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/luks-25db43f5-fa88-4ab8-8568-0e439e1b62df: 1000.2 GB, 1000201712640 bytes, 1953518970 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Alignment offset: 512 bytes
Disk /dev/mapper/luks-6eada242-6252-4b48-8e0e-cb3f9d737062: 1000.2 GB, 1000201712640 bytes, 1953518970 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/luks-bce272a9-0ad7-4297-afb0-fa7482cacd91: 1000.2 GB, 1000201712640 bytes, 1953518970 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
So what is going on here? I don't want to move forward without understanding what the problem may be,
Thanks
You never posted the final status from ddrescue. Are there still unrecovered sectors (blocks with a status flag other than "+" in the log file)?
Did the array assemble OK? What is the content of /proc/mdstat?
I can't figure out what is where from what you have posted. With things in the same state as they were for the above posting, please post the output from lsblk.
So ddrescue was able to recover the entire content of the drive. That is good.
I now see 5 partitions identified as LUKS containers, not just the single LUKS partition in your post back in #10. That pokes holes in my theory at that time that your partitions were all elements of a RAID-0 array that was encrypted as a single unit.
To see what you might have, I set up a VM with 7 disks and did a CentOS 6 minimal install, telling the installer to use all space and encrypt the system. It appears that sda2 should have been shown as a LUKS partition, and there should also be a partition sdg1 that is LUKS. You can run "file -s /dev/sda2" to verify that lsblk just didn't show that for some reason, but where is partition sdg1? I'm guessing that is your missing LVM PV.
Try running testdisk on /dev/sdg and see if it identifies a LUKS partition. It won't get the size right (LUKS containers don't indicate their size), but should get the starting location right. Assuming that testdisk identifies the LUKS container, you'll need to use fdisk to delete and re-create the partition with the correct starting location and extending to the end of the disk. Hopefully, that will clean things up, but do NOT try to write out a partition table until you verify that the LUKS container does not start at the beginning of the unpartitioned disk. That would damage the LUKS header and make that drive forever unrecoverable. In fact, save the first 5 megabytes of that drive somewhere before messing with it.
There is one other problem you'll run into. The new sdd drive has a 4K physical sector size, and its partition is not properly aligned for that. It should work that way, but with very bad write performance. Lets get things working first, and cross that bridge when we come to it.
Since LVM seems to be happy with the volume group, you can try mounting /dev/VolGroup00/LogVol00 and /dev/VolGroup00/LogVol01 read-only and see what's there. An alternative would be first to try running "fsck -f -n /dev/VolGroup00/LogVol00", but if that is not an ext4 filesystem it's going to take a huge amount of time to check even if the filesystem is mostly empty. An fsck of /dev/VolGroup00/LogVol01 would not take nearly as long. Of course if LogVol01 is your swap space there is nothing to check there.
You should take a look at sdg and see what is in there. Since it looks like you can only account for about 6TiB of space, it might be totally unused. Running "hexedit -s /dev/sdg" and looking at the first few sectors, a few sectors starting with sector 64, and a few starting with sector 2048 might be informative.
Once you have everything working and figure out what sdg is (or isn't), the next issue will be the partition alignment on sdd.
Last edited by rknichols; 05-04-2016 at 08:04 PM.
Reason: Add comment about LogVol01
I was able to mount /dev/VolGroup00/LogVol00 in read only and it appears all the files I needed from /var/www/html are there, they have just been moved into the /lost+found directory.
I have no idea why sdg is even in there, but if I can get the /var/www/html files copied to an external drive than that was all I needed to accomplish.
If you don't see any problems with copying these files to the external drive, I will do that and report back.
You should be good to go, then. I get the feeling you're not very interested in the rest of that system. If you do have any further questions, check back in.
First and foremost, I want to thank you and syg00 for your expert advice. The problem was definitely out of my area of expertise. I am glad to report that all of the files are recovered as far as I can tell, I will be going through them but it looks like a 100% success rate.
As far as the OS goes I really don't mind wiping it out and finding a better strategy than the previous one. I hope this thread will help someone out in the future if they ever run into a similar problem.
Don't thank me - I stayed out of it so as to not confuse things.
Make sure you click the "did you find this helpful" on a couple of @rknichols posts to boost reputation.
As far as the OS goes I really don't mind wiping it out and finding a better strategy than the previous one.
Note that spreading a single filesystem across multiple disk drives with no redundancy is a recipie for the sort of problem you experienced. If any drive fails, you lose access to everything.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
