prerequisite: Oracle VM Server configuration: multiple networks, multipathed SAN storage

Configure Multipathed SAN Storage

The diagram above demonstrates the sharing of SAN-provided block storage via multipath fiber channel protocol (FCP) interfaces.

This is can also be accomplised by using:

1. SAN-provided block storage via multipath iSCSI ethernet interfaces
2. SAN-provided network storage, via network file system (NFS) ethernet interfaces

If using ethernet interfaces, it's strongly recommended that a separate set of interfaces be used to those that are providing network (non-storage) connectivity.

Enable ALUA on the SAN

ALUA enablement, if possible, is very vendor dependent. For the backstory on why this is a good thing, see this page.

1. Enable ALUA on a Data ONTAP 7.2.x or later device

Configure the Linux Multipath Device Mapper

Multipath block access is controlled by a combination of

1. the multipath device mapper (dm) device, and
2. the multipathd monitoring daemon, which watches and reacts to changes in path status

Install device mapper multipath packages

• RHEL and derivatives using yum:
  yum install device-mapper-multipath
• Oracle EL and derivatives using up2date:up2date device-mapper-multipath

Check availability of kernel module

The device mapper multipath module is named "dm_multipath", it links to the device mapper (dm_mod) infrastructure

[root@virtualserver ~]# lsmod | grep dm
dm_multipath 52433 0
dm_snapshot 50057 0
dm_zero 35265 0
dm_mirror 60993 0
dm_mod 93713 10 dm_multipath,dm_snapshot,dm_zero,dm_mirror

Ensure that Fiber Channel Host Bus Adapter (HBA) Driver is installed

This example uses Emulex cards, via the lpfc driver.
Other prominent vendors are Qlogic, via the qla2xxx driver, and JNI (now AMCC).

[root@virtualserver network-scripts]# lsmod | grep lpfc
lpfc 219061 0
scsi_transport_fc 73161 1 lpfc
scsi_mod 184057 6 sg,lpfc,scsi_transport_fc,libata,aacraid,sd_mod

Get the port capabilities, an World Wide Port Name (WWPN)

A WWPN is also known as a Port World Wide Name (PWWN), depending on the vendor.

This documentation applies to systems using udev, such as Oracle VM Server, Oracle Enterprise Linux 5.x, RHEL 5.x, Debian 4.0.

[root@virtualserver /]# cd /sys/class/fc_host/
[root@virtualserver fc_host]# ls
host2 host3 <<< provides the host adapter numbers

[root@virtualserver fc_host]# cat host2/port_name
0x10000000c0ffee7e
[root@virtualserver fc_host]# cat host3/port_name
0x10000000c0ffee7f

Dynamically rescanning FCP ports if a fabric change occured

A handy thing to do to update connected servers when something changes.

• based on the ports above, host2 and host3, force a rescan:

echo "- - -" > /sys/class/scsi_host/host2/scan
echo "- - -" > /sys/class/scsi_host/host3/scan

• update UDEV (BUG WARNING: udevtrigger -v will break 802.3ad bonded interfaces in Oracle VM server 2.1!)
• if multipathing is already configured, "multipath -ll" will now show updated fabric information

SAN Fabric Configuration - set Zoning

This is a task for the SAN administrator.

They must ensure that the VM server host HBA's are properly zoned, enabling all LUNs for the VM servers to be made visible to the entire group of VM servers.

Configure multipathd, via /etc/multipath.conf

This configuration is very vendor specific. Best practices change frequently, consult your vendor support group and operating sytem / virtualization platform vendor for current recommendations.

Remember to stipulate an ALUA-compatible configuration if the SAN disk supports it.

1. multipathd.conf configuration for Data ONTAP 7.2.x with ALUA

Enable the multipath daemon

This starts the multipath daemon, and ensure that it restarts on system reboot.
[root@virtualserver etc]# chkconfig --add multipathd
[root@virtualserver etc]# chkconfig multipathd on
[root@virtualserver etc]# /etc/init.d/multipathd start
Starting multipathd daemon: [ OK ]

Rebuild the Linux initrd

May be necessary, depending on vendor.

mkinitrd -v -f initrd-$(uname -r).img $(uname -r)

Reboot the system

really.

On first configuration, a lot changes in the system. The config may work fine, but only after a reboot is completed. This should only be needed once to confirm that all is good.

Terminology: HBA number, channel ID, SCSI ID, Lun Number

HBA number, channel ID, SCSI ID, LUN number
Troubleshooting and dynamic (no rescan) update of multipath disk needs

• A the host bus adapter (HBA) number
• B the channel id on the HBA
• C the SCSI ID of the new device
• D the LUN of the new device

"multipath -ll" shows this, eg below A=3, B=0, C=0, D=0

Query the Status of multipath system

Here we have two LUNs, mpath0 and mpath1 managed by the multipath system. Path priority is reported for each round-robin group if the ALUA data and multiple paths are available.

These LUNs are accessed via /dev/mapper/mpath0 and /dev/mapper/mpath1

[root@virtualserver ~]# multipath -ll
mpath1 (360a9800043346466614a447271456c45) dm-2 NETAPP,LUN
[size=16G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=100][enabled] <<< PREFERRED PATHS
\_ 3:0:0:1 sdg 8:96 [active][ready]
\_ 3:0:1:1 sdi 8:128 [active][ready]
\_ round-robin 0 [prio=20][enabled] <<< SECONDARY PATHS
\_ 2:0:0:1 sdc 8:32 [active][ready]
\_ 2:0:1:1 sde 8:64 [active][ready]
mpath0 (360a9800043346466614a44613656744e) dm-1 NETAPP,LUN
[size=16G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=100][enabled] <<< PREFERRED PATHS
\_ 3:0:0:0 sdf 8:80 [active][ready]
\_ 3:0:1:0 sdh 8:112 [active][ready]
\_ round-robin 0 [prio=20][enabled] <<< SECONDARY PATHS
\_ 2:0:0:0 sdb 8:16 [active][ready]
\_ 2:0:1:0 sdd 8:48 [active][ready]

Other Investigative and Troubleshooting Commands

• "multipath -v3 -ll" to get full debugging information.
• "dmsetup ls --target=multipath" will show short device name, and (major,minor) device numbers