Hi folks,
Because of Ganglia reporting high iowait, I re-check with iostat, sar, ...:
Code:
# iostat 1 3
Linux 2.6.18-128.1.10.el5.xs5.5.0.51xen (svr150-4107.localdomain) 11/30/2010
avg-cpu: %user %nice %system %iowait %steal %idle
0.02 0.04 0.03 2.05 0.00 97.86
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvdd 0.00 0.00 0.00 632 0
xvdb 0.04 0.00 1.74 7401 20043032
xvdb1 0.00 0.00 0.00 1471 1280
xvdb2 0.04 0.00 1.74 5274 20041752
xvda 1.63 2.80 26.83 32122608 308266520
xvda1 0.00 0.02 0.00 182654 15696
xvda2 1.63 2.78 26.83 31939394 308250824
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.00 100.00 0.00 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvdd 0.00 0.00 0.00 0 0
xvdb 0.00 0.00 0.00 0 0
xvdb1 0.00 0.00 0.00 0 0
xvdb2 0.00 0.00 0.00 0 0
xvda 0.00 0.00 0.00 0 0
xvda1 0.00 0.00 0.00 0 0
xvda2 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.00 100.00 0.00 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvdd 0.00 0.00 0.00 0 0
xvdb 0.00 0.00 0.00 0 0
xvdb1 0.00 0.00 0.00 0 0
xvdb2 0.00 0.00 0.00 0 0
xvda 3.00 0.00 120.00 0 120
xvda1 0.00 0.00 0.00 0 0
xvda2 3.00 0.00 120.00 0 120
Code:
12:22:30 PM CPU %user %nice %system %iowait %steal %idle
12:22:31 PM all 0.00 0.00 0.00 100.00 0.00 0.00
12:22:32 PM all 0.00 0.00 0.00 100.00 0.00 0.00
`fdisk -l` and other commands related to disk hangs in D state:
Code:
# ps -eo pid,user,state,cmd | awk '$3 ~ /D/ { print $0 }'
1142 root D lvm.static lvs --ignorelockingfailure --noheadings -o vg_name xvda2
7742 root D lvm.static lvs --ignorelockingfailure --noheadings -o vg_name xvda2
8061 root D lvm.static lvs --ignorelockingfailure --noheadings -o vg_name xvda2
8948 root D lvm.static lvs --ignorelockingfailure --noheadings -o vg_name xvda2
9848 root D lvm.static lvs --ignorelockingfailure --noheadings -o vg_name xvda2
9974 root D lvm.static lvs --ignorelockingfailure --noheadings -o vg_name xvda2
10143 root D lvm.static lvs --ignorelockingfailure --noheadings -o vg_name xvda2
19531 root D fdisk -l
26075 root D fdisk -l
26536 root D fdisk -l
26788 root D fdisk -l
30780 root D lvm.static lvs --ignorelockingfailure --noheadings -o vg_name xvda2
30906 root D lvm.static lvs --ignorelockingfailure --noheadings -o vg_name xvda2
therefore, the first thing come to my mind is I got the problem with virtual disks but I have no ideas to check.
I tried to update my whole system - CentOS 5.5 and prepare for the reboot (backuped with
fallback option in grub.conf), but I got stuck in mkinitrd:
Code:
# mkinitrd -v -f /boot/initrd-2.6.18-194.26.1.el5xen.img 2.6.18-194.26.1.el5xen
Creating initramfs
Looking for deps of module ehci-hcd
Looking for deps of module ohci-hcd
Looking for deps of module uhci-hcd
Looking for deps of module ext3: jbd
Looking for deps of module jbd
Found root device xvda2 for LABEL=/
... hangs in here
Any suggestions will be appreciated.
PS: my /etc/mtab:
Code:
/dev/xvda2 / ext3 rw 0 0
proc /proc proc rw 0 0
sysfs /sys sysfs rw 0 0
devpts /dev/pts devpts rw,gid=5,mode=620 0 0
/dev/xvdb2 /data ext3 rw 0 0
/dev/xvda1 /boot ext3 rw 0 0
tmpfs /dev/shm tmpfs rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
