What's going on here? See:

1. Oracle VM Server Configuration- using OCFS2 in a group of VM hosts to share block storage
2. Oracle VM Server use case: multiple networks, multipathed SAN storage

Other potential ways of addressing this are to:

1. Use RedHat's Cluster Logical Volume Manager (CLVM), coupled with RedHat's Global File System (GFS)
   

We will take a SAN LUN, which is accessible to a group of four VM server hosts, and mount it to the "/resource" directory of these servers. OCFS2 will control access and lock management.

The SAN LUN on the Enterprise Disk System is shared to a series of VM servers as per the diagram below:


Four node configuration, all steps below must be completed on each server!!!

1. virtualserver1, 10.1.1.101
2. virtualserver2, 10.1.1.102
3. virtualserver3, 10.1.1.103
4. virtualserver4, 10.1.1.104

Allow OCFS2 communication to pass through the host firewall

The "system-config-firewall" tool could also be used to for this, opening port 7777/tcp.

• edit /etc/sysconfig/iptables, adding line:

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 7777 -j ACCEPT

• full /etc/sysconfig/iptables:
  

# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -p tcp -m state --state NEW -m tcp --dport 21 -j DROP
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j DROP
-A INPUT -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 21 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 2049 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 5900:5950 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 7777 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8002 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8899 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT

• Restart the firewall

[root@virtualserver network-scripts]# service iptables restart
Flushing firewall rules: [ OK ]
Setting chains to policy ACCEPT: filter [ OK ]
Unloading iptables modules: [ OK ]
Applying iptables firewall rules: [ OK ]
Loading additional iptables modules: ip_conntrack_netbios_n[ OK ]

OCFS2 Configuration

• create configuration directory

mkdir /etc/ocfs2

• create cluster.conf file

[root@virtual6 ~]# cat /etc/ocfs2/cluster.conf
node:
ip_port = 7777
ip_address = 10.1.1.101
number = 1
name = virtualserver1.mydomain
cluster = ocfs2

node:
ip_port = 7777
ip_address = 10.1.1.102
number = 2
name = virtualserver2.mydomain
cluster = ocfs2

node:
ip_port = 7777
ip_address = 10.1.1.103
number = 3
name = virtualserver3.mydomain
cluster = ocfs2

node:
ip_port = 7777
ip_address = 10.1.1.104
number = 4
name = virtualserver4.mydomain
cluster = ocfs2

cluster:
node_count = 4
name = ocfs2

• o2cb: check status

[root@virtualserver3 etc]# service o2cb status
Module "configfs": Not loaded
Filesystem "configfs": Not mounted
Module "ocfs2_nodemanager": Not loaded
Module "ocfs2_dlm": Not loaded
Module "ocfs2_dlmfs": Not loaded
Filesystem "ocfs2_dlmfs": Not mounted

• o2cb: load modules, configuration files

[root@virtualserver3 etc]# service o2cb load
Loading module "configfs": OK
Mounting configfs filesystem at /sys/kernel/config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Creating directory '/dlm': OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK

• o2cb: bring service online

[root@virtualserver3 etc]# service o2cb online
Starting O2CB cluster ocfs2: OK

• check service startup parameters

[root@virtualserver3 ~]# chkconfig --list ocfs2
ocfs2 0:off 1:off 2:on 3:on 4:on 5:on 6:off

• allow o2cb service to start on boot, this is a non-standard process!

[root@virtualserver3 etc]# /etc/init.d/o2cb configure
Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot. The current values will be shown in brackets ('[]'). Hitting
<ENTER> without typing an answer will keep that current value. Ctrl-C
will abort.

Load O2CB driver on boot (y/n) [n]: y
Cluster to start on boot (Enter "none" to clear) [ocfs2]:
Specify heartbeat dead threshold (>=7) [31]:
Specify network idle timeout in ms (>=5000) [30000]:
Specify network keepalive delay in ms (>=1000) [2000]:
Specify network reconnect delay in ms (>=2000) [2000]:
Writing O2CB configuration: OK
Loading module "configfs": OK
Mounting configfs filesystem at /sys/kernel/config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Creating directory '/dlm': OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting O2CB cluster ocfs2: OK

• select shared disk, using lun0 exposed from SAN multipath /dev/mapper/mpath0

[root@virtualserver3 etc]# multipath -ll
mpath0 (360a9800043346466614a456f38374478) dm-2 NETAPP,LUN
[size=36G][features=1 queue_if_no_path][hwhandler=0]

[root@virtualserver3 etc]# ls -l /dev/mapper/mpath0
brw-rw---- 1 root disk 253, 2 Nov 20 02:10 /dev/mapper/mpath0

• format devices as ocfs2

[root@virtualserver3 etc]# mkfs.ocfs2 /dev/mapper/mpath0
mkfs.ocfs2 1.2.7
Filesystem label=
Block size=4096 (bits=12)
Cluster size=4096 (bits=12)
Volume size=38654705664 (9437184 clusters) (9437184 blocks)
293 cluster groups (tail covers 18432 clusters, rest cover 32256 clusters)
Journal size=241590272
Initial number of node slots: 4
Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Writing backup superblock: 3 block(s)
Formatting Journals: done
Writing lost+found: done
mkfs.ocfs2 successful

• mount OCFS2 volume on each node

[root@virtualserver3 ~]# mount /dev/mapper/mpath0 /resource -t ocfs2

[root@virtualserver3 ~]# df -h /resource
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/mpath0 36G 36G 724M 99% /resource

• make mount persist via /etc/fstab, adding lines:

# Cluster mounts, ocfs2
/dev/mapper/mpath0 /resource ocfs2 defaults 1 0

OCFS2 Troubleshooting

There is much more to OCFS2 than these quick commands

• get OCFS2 volume UUIDs for a device

OCFS tracks potentially disparate devices names via a UUID, which can be read with a simpleish command. Here the shared block device is /dev/mapper/mpath0

[root@virtualserver3 ovs]# echo "stats" | debugfs.ocfs2 -n /dev/mapper/mpath0 | grep UUID | while read a uuid; do echo $uuid; done

B4821C0EE92843FEB6104F0C774255FF

• DID YOU OPEN UP THE FIREWALL? If you get the message below, read this page again from the top!

root@virtualserver3 /]# mount /dev/mapper/mpath0 /mnt -t ocfs2
mount.ocfs2: Transport endpoint is not connected while mounting /dev/mapper/mpath0 on /mnt. Check 'dmesg' for more information on this error.

• dmesg is your friend

OCFS2 Node Manager 1.2.7 Fri Nov 9 18:53:36 EST 2007 (build 1c4508be0c999c7bec8d997b52240db8)
OCFS2 DLM 1.2.7 Fri Nov 9 18:53:36 EST 2007 (build 71e582227caabd47a4b94bc0f322b660)
OCFS2 DLMFS 1.2.7 Fri Nov 9 18:53:36 EST 2007 (build 71e582227caabd47a4b94bc0f322b660)
OCFS2 User DLM kernel interface loaded
OCFS2 Node Manager 1.2.7 Fri Nov 9 18:53:36 EST 2007 (build 1c4508be0c999c7bec8d997b52240db8)
OCFS2 DLM 1.2.7 Fri Nov 9 18:53:36 EST 2007 (build 71e582227caabd47a4b94bc0f322b660)
OCFS2 DLMFS 1.2.7 Fri Nov 9 18:53:36 EST 2007 (build 71e582227caabd47a4b94bc0f322b660)
OCFS2 User DLM kernel interface loaded
OCFS2 1.2.7 Fri Nov 9 18:53:30 EST 2007 (build d443ce77532cea8d1e167ab2de51b8c8)
ocfs2_dlm: Nodes in domain ("F461248D74CB430CA345E4F8CEFFCD0B"): 0
JBD: Ignoring recovery information on journal
kjournald starting. Commit interval 5 seconds
ocfs2: Mounting device (253,2) on (node 0, slot 0)
(31256,0):o2net_connect_expired:1571 ERROR: no connection established with node 1 after 30.0 seconds, giving up and returning errors.

Test the filesystem!

• on virtualserver1

[root@virtualserver1 etc]# echo "This is a write test to an OCFS2 volume" > /resource/testfile

• on virtualserver2
  

[root@virtualserver2 /]# cat /resource/testfile
This is a write test to an OCFS2 volume