Hi folks!

I am pretty new to this forum and linux server administration.

Recently I have added a new LUN to a Oracle RAC. The new LUN has been mapped to the devices sdaj and sdak by the server. After I reboot the server, redhat automatically re-align all devices and LUNs. This causes all the disk mount problem.

So, any one know how to identify which device map to which LUN; or which device is equivalent to which (both devices point to the same LUN).

The OS of the subject server are Redhat 4.5

Please guide me to the right direction to fix the problem.

Thank you,
Wei

I'm not really conversant in RAC, or ocfs2, but I think what you are looking at is at a lower level that that.

On Red Hat 4.5 you have DM-Multipath and LVM2, both of which I'm familiar with and we use heavily at my company.

This is a little wordy, so bear with me.

If you do an 'lsscsi', you should see all of your scsi block devices, i.e. /dev/sdaj and /dev/sdak. It won't tell you very much about them though, just that they have been recognized and are there.

If you are running DM-multipath (we do on our RAC systems) the underlying SAN scsi devices are "matched up" (by reading the serial number from each device) and are represented by an 'mpath' device, e.g. /dev/mapper/mpath2.

If you do a 'multipath -ll', you should see those underlying devices mapped to 'mpath' devices.

(There is also the dmsetup command, 'dmsetup ls', that shows the multipath devices (lvm, dm-crypt, dm-multipath, ...) whatever you are using.)

The ext2/3/4 LABEL or the UUID of a device can be used to mount it and eleviates the problem where you mount an explicit device name and the names have "shuffled" on you.

With LVM you have metadata written to the various "volumes" of LVM. When you do a 'pvcreate' on a partition (or full volume), metadata is written to that partition (or full volume); then when you do a 'vgcreate' or 'vgextend', more metatdata is written there; same with 'lvcreate' or 'lvextend'. When LVM is initialized as the system comes up, it scans for that metatdata, again alleviating a dependence on explicit device names.

On a "normal" non-RAC system, the mpath devices can be formatted (preferably with an ext2/3/4 LABEL so you don't have to worry about device names changing) and mounted (I think). A better practice is to use LVM.

With ocfs2 you don't use LVM or mount the devices directly. It does (at least in my shop) use the multipath device (e.g. /dev/mapper/mpath2) but mounts what look like partitions (e.g. /dev/mapper/mpath2p8 on /u08 and /dev/mapper/mpath2p9 on /u19).

Do a 'mount' command to see what you have mounted now.

Sorry I can't tell you much more than that. If you are using ocsf2, you can post another question that is ocfs2 specific and you should find someone that is more authoritative than me.

btw, RHEL4.5 is very old and was pretty buggy. We tried it and then went to RHEL4.6 + a higher level kernel. When Oracle finally supported RHEL5, we went to RHEL5.2 and then RHEL5.4.

Thank you Tommy;

Thank you for your quick reply.

Because RHEL 4.5 is old, we don't have multipath installed. Therefore, your suggestion doesn't work for me.
Do you have any otherway to work around?

I can't see (from googling) that ocfs2 does multipathing itself.

So, first question is "are you doing any kind or multipathing, using md for multipathing, or are you doing no multipathing at all?"
Second question is "are you using ocfs2?"

If you supply the output of a 'mount' command (so I can see what and how you have your filesystems mounted) it will help.

btw, DM-multipath was introduced in RHEL4.2. Probably not a good idea to implement it now. Before that md was used to support multipathing but did not perform very well and was not popular.

Hi Tommy:

I don't believe we use any multipath.

We do use ocfs

Here is the mount result.

/dev/sda7 on / type ext3 (rw)
none on /proc type proc (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbdevfs on /proc/bus/usb type usbdevfs (rw)
/dev/sda3 on /boot type ext3 (rw)
/dev/sda8 on /home type ext3 (rw)
/dev/sda5 on /opt/oracle type ext3 (rw)
none on /dev/shm type tmpfs (rw)
/dev/sda12 on /tmp type ext3 (rw)
/dev/sda6 on /usr type ext3 (rw)
/dev/sda13 on /data type ext3 (rw)
/dev/sdb8 on /u01 type ocfs (rw,_netdev)
/dev/sdc1 on /u03 type ocfs (rw,_netdev)
rac0:/data/oracle/local on /opt/oracle/local type nfs (rw,addr=167.165.###.###)


Thank you for your help.

I'm seeing posts for ocfs2 similar to yours where people are mounting SAN LUNs by device name and from boot to boot the device name is changing. The advice given is "you need to mount by UUID", and "you should not mount by UUID, you should mount by LABEL." (I'm hoping this advice is applicable to ocfs...)

I've done both LABEL= and UUID= mounts with ext3 and ext4 file systems. (And you can LABEL an ext2/3/4 filesystem after it's been formatted. It's not clear if you can add a label to an ocfs filesystem after it's been formatted, but it should have a UUID.)

With ext2/3/4, you can UUID mount with an /etc/fstab entry like:
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /<mountpoint> ext3 defaults 1 2

or LABEL mount with:
LABEL=label /<mountpoint> ext3 defaults 1 2

You can try using the vol_id command to determine the UUID of the filesystem that you want to mount.

As an example, you should be able to do '/lib/udev/vol_id -u /dev/sdb8' to find the UUID of that FS.
I'm not sure you can use UUID=<uuid> the way you can with ext2/3/4, but I've seen a recommendation that it can by mounted with an fstab entry similar to this,
/dev/disk/by-uuid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /<mountpoint> ocfs etc...

(An 'ls -l /dev/disk/by-uuid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' will tell you if this works on RHEL4.5.)

Hope this is going in the right direction. I have no RHEL4 systems to try any of this on.

Hi Tommy:

Thank you for your help, my problem has been solved by 3rd party vendor.