Please advise how to troubleshoot/fix. I have backups of the /home partition just in case.


Problem: Over the weekend, one or both of these partitions could have run out of space. /home and /scratch are no longer mounted.
# mount -a
mount: special device /dev/datavg01/datalv does not exist
mount: special device /dev/datavg01/scratch does not exist

Notes:
OS: CentOS 4.5 (no packages have been updated since July 2008!)
sda, sdb is a (software raid) and appears Ok,
sdc is a spare (unused) 80 gb disk. sdd is the external raid.

See entries from /var/log/messages. System already rebooted itself:

Jan 10 08:55:08 mycluster kernel: sdd: Unit Not Ready, sense:
Jan 10 08:55:08 mycluster kernel: sdd : READ CAPACITY failed.
Jan 10 08:55:08 mycluster kernel: sdd : status=1, message=00, host=0, driver=08
Jan 10 08:55:08 mycluster kernel: sdd: asking for cache data failed
Jan 10 08:55:08 mycluster kernel: sdd: assuming drive cache: write through
Jan 10 08:55:08 mycluster kernel: sdd: Unit Not Ready, sense:
Jan 10 08:55:08 mycluster kernel: sdd : READ CAPACITY failed.
Jan 10 08:55:08 mycluster kernel: sdd : status=1, message=00, host=0, driver=08
Jan 10 08:55:08 mycluster kernel: sdd: asking for cache data failed
Jan 10 08:55:08 mycluster kernel: sdd: assuming drive cache: write through
Jan 10 08:55:08 mycluster kernel: sdd:Current sdd: sense key No Sense
Jan 10 08:55:08 mycluster kernel: Attached scsi disk sdd at scsi4, channel 0, id 0, lun 0
Jan 10 08:52:29 mycluster lvm.static: WARNING: Physical Volume /dev/sdd is too large for underlying device
Jan 10 08:52:29 mycluster lvm.static: WARNING: Physical Volume /dev/sdd is too large for underlying device
Jan 10 08:52:30 mycluster lvm.static: WARNING: Physical Volume /dev/sdd is too large for underlying device
Jan 10 08:52:30 mycluster lvm.static: WARNING: Physical Volume /dev/sdd is too large for underlying device
Jan 10 08:52:30 mycluster lvm.static: WARNING: Physical Volume /dev/sdd is too large for underlying device
Jan 10 08:52:30 mycluster lvm.static: WARNING: Physical Volume /dev/sdd is too large for underlying device
Jan 10 08:53:32 mycluster vgchange: WARNING: Physical Volume /dev/sdd is too large for underlying device

# lvdisplay -a
WARNING: Physical Volume /dev/sdd is too large for underlying device
--- Logical volume ---
LV Name /dev/datavg01/datalv
VG Name datavg01
LV UUID JjtG47-hmVr-OIr1-cff4-H1F0-WQNV-TqsMMj
LV Write Access read/write
LV Status available
# open 0
LV Size 1.50 TB
Current LE 1536
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:0

--- Logical volume ---
LV Name /dev/datavg01/scratch
VG Name datavg01
LV UUID 9jvnCx-DkcY-7CW2-dsX4-I4SB-RgD9-VU2XaH
LV Write Access read/write
LV Status available
# open 0
LV Size 3.58 TB
Current LE 3666
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:1

Host: scsi4 Channel: 00 Id: 00 Lun: 00
Vendor: YI-16SAE Model: U4 Rev: R0.0
Type: Direct-Access ANSI SCSI revision: 03

# cat /etc/fstab
# This file is edited by fstab-sync - see 'man fstab-sync' for details
/dev/md1 / ext3 defaults 1 1
/dev/md0 /boot ext3 defaults 1 2
none /dev/pts devpts gid=5,mode=620 0 0
none /dev/shm tmpfs defaults 0 0
/dev/md3 /opt ext3 defaults 1 2
/dev/md2 /work1 ext3 defaults 1 2
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
/dev/sdb5 swap swap defaults 0 0
/dev/sda5 swap swap defaults 0 0
/dev/datavg01/datalv /home xfs defaults 0 0
/dev/datavg01/scratch /scratch xfs defaults 0 0[/SIZE]

Thanks in advance

More bits:
# vgdisplay
WARNING: Physical Volume /dev/sdd is too large for underlying device
--- Volume group ---
VG Name datavg01
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 4
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 5.08 TB
PE Size 1.00 GB
Total PE 5207
Alloc PE / Size 5202 / 5.08 TB
Free PE / Size 5 / 5.00 GB
VG UUID xZ0RXd-176y-v4ig-1hNW-C5hI-eWGa-jDL0JP

# pvdisplay
WARNING: Physical Volume /dev/sdd is too large for underlying device
WARNING: Physical Volume /dev/sdd is too large for underlying device
--- Physical volume ---
PV Name /dev/sdd
VG Name datavg01
PV Size 1.00 GB / not usable 8192.00 EB
Allocatable yes
PE Size (KByte) 1048576
Total PE 5207
Free PE 5
Allocated PE 5202
PV UUID UG9IUz-lJqq-wbCs-awxH-1uhA-y7ZV-rWqYwd

It appears that your logical volumes are in "inactive" state.


To verify that, try this

For example:-

# /sbin/pvscan
PV /dev/sdb5 VG Fedora_volume lvm2 [20.03 GB / 0 free]
PV /dev/sda5 VG Linux_volume lvm2 [44.81 GB / 32.00 MB free]
Total: 2 [64.84 GB] / in use: 2 [64.84 GB] / in no VG: 0 [0 ]


# /usr/sbin/lvscan
inactive '/dev/Fedora_volume/LogVol00' [14.03 GB] inherit
inactive '/dev/Fedora_volume/LogVol02' [5.00 GB] inherit
inactive '/dev/Fedora_volume/LogVol01' [1.00 GB] inherit
ACTIVE '/dev/Linux_volume/LogVol00' [33.78 GB] inherit
ACTIVE '/dev/Linux_volume/LogVol02' [10.00 GB] inherit
ACTIVE '/dev/Linux_volume/LogVol01' [1.00 GB] inherit

See from the above results, some volumes were marked as inactive.

To activate the volume group is very simple.

# /usr/sbin/lvchange -a y /dev/Fedora_volume

Then reboot your system and observe the results.

To mark any volume group "inactive" explicity, then try this

# /usr/sbin/lvchange -a n /dev/Fedora_volume

Thanks for the prompt feedback. There had been a power outage at the site. A normal shutdown/reboot fixed the problem, but now I will certainly read up on this subject.