ORACLE RAC LAB BUILD

The purpose of the RAC Lab is to:

a) build a similar system to the work RAC systems and to go through the build installation to identify configuration and installation issues before attempting the same work on the systems at work.

b) become familiar with RAC configuration and installation

c) create a sandbox platform to experiment with different configurations and functional testing

LAB Equipment:

Servers:
2 Dual AMD Opteron 64bit w/4Gb RAM and 2 gigabit ethernet ports,
1 120Gb internal HD

Operating system:
CentOS 5.2 (Red Hat 5.2 binary equiv)

Oracle Software:
Cluster file system OCFS2 1.4.1
ASMLib 2.0
11g Oracle Clusterware
11g RAC ASM
11g Database using 11g RAC ASM
10g Database using 11g RAC ASM

Shared Storage:
Openfiler 2.3
Installed on AMD PC with 4Gb RAM, single 640Gb HD and single gigabit
ethernet port

Network Equipment:
1 8 port gigabit switch (Private Interconnect and Storage Connection)
1 4 port 100mb/s hub (Public)

Staging server:
Apple Macbook

SIGNIFICANT DIFFERENCES BETWEEN LAB SETUP AND WORK SERVERS
=======================================================================================
ITEM LAB WORK
-----------------------------------------------------------------------------------------------------------------------------
Hardware 2 Rackable Systems 2 HP DL585 G5's
Dual AMD Opteron 4 Quad Core AMD Opterons
single core CPU's w/128Gb RAM
w/4Gb RAM & Dual 6 Gigabit Ethernet w/4
Gigabit Ethernet bonded in pairs for private
and public networks 3 HBA's for SAN connections

Disk iSCSI disks from EMC SAN Disks Multipathed
dedicated PC with & connected through HBA's
openfiler 2.3
connected to the
RAC nodes via the
private network

OS CentOS 5.2 Red Hat 5.2

Security No special add ons Keon ESM

Monitoring No special add ons Tivoli, netsnmp/snmp-DC,
BMC Perform&Predict

Backup No special add ons Veritas Netbackup

DB Monitoring Local DB Console Oracle GridControl (Centralized)

INSTALLATION ACTIONS
=======================================================================================
1)INSTALL OS ON EACH NODE

choose CentOS 5.2 x86_64
select base install
when it reboots, for the first time login:
disable firewall and SELinux
skip creating a user
log in as root
Do NOT run automatic updates when complete!!!

2)SETUP NETWORK

Open network admin utility and set up Network

Set hostname
disable DHCP

port eth0 = 10.0.1.6X
port eth1 = 10.0.2.6X

restart network services

Open terminal and edit /etc/hosts

Remove hostname from 127.0.0.1

add the following:

10.0.1.69 racbaby1
10.0.1.68 racbaby2
10.0.2.69 racbaby1-priv
10.0.2.68 racbaby2-priv
10.0.1.169 racbaby1-vip
10.0.1.168 racbaby2-vip
10.0.2.66 openfiler1

3)MODIFY KERNEL AND SETTINGS

use GUI "services" utility to disable updater, card reader, bluetooth, etc

use GUI "Add Software" tool to install these items listed under "development"
development tools
development libraries
java development
legacy systems development

In a terminal window install the following additional packages:

yum install libaio-devel
yum install sysstat
yum install elfutils-libelf-devel
yum install iscsi-initiator-utils.x86_64
yum install lsscsi
yum install xorg-x11-deprecated-libs
yum install libXp.so.6

Set parameters using these commands:

sysctl -w net.core.rmem_default=4194304
sysctl -w net.core.wmem_default=262144
sysctl -w net.core.rmem_max=4194304
sysctl -w net.core.wmem_max=262144
echo "# Default setting in bytes of the socket receive buffer" >>/etc/sysctl.conf
echo "net.core.rmem_default= 4194304" >>/etc/sysctl.conf
echo "# Default setting in bytes of the socket send buffer" >>/etc/sysctl.conf
echo "net.core.wmem_default=262144" >>/etc/sysctl.conf
echo "# Maximum socket receive buffer size which may be set by using" >>/etc/sysctl.conf
echo "# the SO_RCVBUF socket option" >>/etc/sysctl.conf
echo "net.core.rmem_max= 4194304" >>/etc/sysctl.conf
echo "# Maximum socket send buffer size which may be set by using" >>/etc/sysctl.conf
echo "# the SO_SNDBUF socket option" >>/etc/sysctl.conf
echo "net.core.wmem_max=262144" >>/etc/sysctl.conf

sysctl -w kernel.shmmax=2147483648
echo "kernel.shmmax=2147483648" >> /etc/sysctl.conf
sysctl -w kernel.sem="250 32000 100 128"
echo "kernel.sem=250 32000 100 128" >> /etc/sysctl.conf
sysctl -w fs.file-max=65536
echo "fs.file-max=65536" >> /etc/sysctl.conf
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
echo "net.ipv4.ip_local_port_range = 1024 65000" >> /etc/sysctl.conf

cat >> /etc/security/limits.conf <<EOF
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
EOF

cat >> /etc/pam.d/login <<EOF
session required /lib/security/pam_limits.so
EOF

cat >> /etc/profile <<EOF
if [ \$USER = "oracle" ]; then
if [ \$SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
EOF

4)CREATE ORACLE USER

groupadd -g 115 oinstall
groupadd -g 116 dba
useradd -m -u 175 -g oinstall -G dba -d /home/oracle -s /bin/bash -c "Oracle Software Owner" oracle
passwd oracle

5)CREATE SPACE FOR HOMES

mkdir -p /u01/app/oracle
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/app/oracle
mkdir -p /u01/app/11.1.0
chown -R oracle:oinstall /u01/app/11.1.0
chmod -R 775 /u01/app/11.1.0

6)CREATE ISCSI DISKS

Use openfiler to set up disks and map luns to the two nodes. I'm not going to
listout how I setup Openfiler as it doesn't pertain to the purpose of the LAB
build.

For simplicity I named the luns according to function, so that when they appear on the nodes you'll see something like the following in /var/log/messages:

[root@racbaby2 log]# cat messages.1|grep openf|sort
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm10,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm1,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm2,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm3,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm4,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm5,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm6,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm7,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm8,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm9,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:ocfs,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:ocr1,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:ocr2,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:spfile,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:vote1,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:vote2,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:vote3,10.0.2.66,3260]

7)COPY OVER SOFTWARE FILES FROM STAGING SERVER

Hangcheck Timer
hangcheck-0.8.0.tar.gz

Oracle CLUSTER and DB Software packages:
linux.x64_11gR1_clusterware.zip
linux.x64_11gR1_database_1013.zip
10201_database_linux_x86_64.cpio.gz

Oracle OCFS RPM's
ocfs2-2.6.18-92.el5-1.4.1-1.el5.x86_64.rpm
ocfs2-tools-1.4.1-1.el5.x86_64.rpm
ocfs2-tools-debuginfo-1.4.1-1.el5.x86_64.rpm
ocfs2-tools-devel-1.4.1-1.el5.x86_64.rpm
ocfs2console-1.4.1-1.el5.x86_64.rpm

ASMLib RPM's
oracleasm-2.6.18-92.1.22.el5-2.0.5-1.el5.x86_64.rpm
oracleasm-2.6.18-92.el5-2.0.5-1.el5.x86_64.rpm
oracleasm-support-2.1.2-1.el5.x86_64.rpm
oracleasmlib-2.0.3-1.el5.x86_64.rpm

homemade env variable switcher script
oradb

Oracle patchsets:
p6810189_10204_Linux-x86-64.zip
p6890831_111070_Linux-x86-64.zip

8)SET UP ISCSI ON RAC NODES

as root on each node:

service iscsi restart
chkconfig --level 345 iscsi on
iscsiadm -m discovery -t sendtargets -p openfiler1
service iscsi restart

verify disks are there

fdisk -l

run fdisk on each device to create a primary partition

9)SET UP CONSISTENT UDEV DEVICE NAMES AND PERMISSIONS ON EACH NODE

As root on each node:

look at /proc/scsi/scsi for this:
Host: scsi20 Channel: 00 Id: 00 Lun: 00
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04

then edit /etc/iscsi/iscsid.conf and add at the bottom:

vendor=OPNFILER, model="VIRTUAL-DISK". options=-g

now add the following line at the end of /etc/udev/rules.d/50-udev.rules

KERNEL=="sd?", BUS=="scsi", ENV{ID_MODEL}=="VIRTUAL-DISK", ENV{ID_PATH}=="*iscsi*", OWNER="oracle", \
GROUP="dba", MODE="640", RUN+="/etc/udev/scripts/iscsi $env{ID_PATH} %k"
KERNEL=="sd?1", BUS=="scsi", ENV{ID_MODEL}=="VIRTUAL-DISK", ENV{ID_PATH}=="*iscsi*", OWNER="oracle", \
GROUP="dba", MODE="640"

create a file called /etc/udev/scripts/iscsi and add the following

#!/bin/bash
LINK_NAME=`/bin/echo $1 | awk -F":" '{ print $3 }'`
/bin/ln -s /dev/${2}1 /dev/${LINK_NAME}

Then make it executable using:
chmod 700 /etc/udev/scripts/iscsi

reboot both nodes

verify you see good named links to the partitioned disks (may have to reboot again for disks to show up correctly)

$ ls -l /dev |grep ">" |grep sd |egrep -v "core|MAKE|floppy|proc"
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm1 -> /dev/sdg1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm10 -> /dev/sdd1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm2 -> /dev/sdk1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm3 -> /dev/sdc1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm4 -> /dev/sdn1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm5 -> /dev/sdf1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm6 -> /dev/sdb1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm7 -> /dev/sdo1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm8 -> /dev/sdj1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm9 -> /dev/sdm1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 ocfs -> /dev/sde1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 ocr1 -> /dev/sdp1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 ocr2 -> /dev/sdh1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 spfile -> /dev/sdi1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 vote1 -> /dev/sdl1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 vote2 -> /dev/sdq1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 vote3 -> /dev/sdr1

verify disks are reachable by oracle and not owned by root:

$ ls -l /dev/sd*1|grep -v ">"|grep -v sda1
brw-r----- 1 oracle dba 8, 17 Mar 7 10:32 /dev/sdb1
brw-r----- 1 oracle dba 8, 33 Mar 7 10:32 /dev/sdc1
brw-r----- 1 oracle dba 8, 49 Mar 7 10:32 /dev/sdd1
brw-r----- 1 oracle dba 8, 65 Mar 7 10:32 /dev/sde1
brw-r----- 1 oracle dba 8, 81 Mar 7 10:32 /dev/sdf1
brw-r----- 1 oracle dba 8, 97 Mar 7 10:32 /dev/sdg1
brw-r----- 1 oracle dba 8, 113 Mar 7 10:48 /dev/sdh1
brw-r----- 1 oracle dba 8, 129 Mar 7 10:32 /dev/sdi1
brw-r----- 1 oracle dba 8, 145 Mar 7 10:32 /dev/sdj1
brw-r----- 1 oracle dba 8, 161 Mar 7 10:32 /dev/sdk1
brw-r----- 1 oracle dba 8, 177 Mar 7 13:23 /dev/sdl1
brw-r----- 1 oracle dba 8, 193 Mar 7 10:32 /dev/sdm1
brw-r----- 1 oracle dba 8, 209 Mar 7 10:32 /dev/sdn1
brw-r----- 1 oracle dba 8, 225 Mar 7 10:32 /dev/sdo1
brw-r----- 1 oracle dba 8, 241 Mar 7 10:48 /dev/sdp1
brw-r----- 1 oracle dba 65, 1 Mar 7 13:23 /dev/sdq1
brw-r----- 1 oracle dba 65, 17 Mar 7 13:23 /dev/sdr1

10)CREATE OCFS2 DIRECTORIES

mkdir -p /u02/app/oracle/exports
chown -R oracle:dba /u02/app/oracle/exports
chmod -R 775 /u02/app/oracle/exports

mkdir -p /u02/app/oracle/ASM/spfile
chown -R oracle:dba /u02/app/oracle/ASM/spfile
chmod -R 775 /u02/app/oracle/ASM/spfile

11)EDIT ORACLE BASH PROFILE

edit .bash_profile for oracle to match as follows:

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

alias oradb=". /home/oracle/oradb 2>/dev/null"

12)EDIT ORATAB FILE

as root

touch /etc/oratab
chown oracle:dba /etc/oratab

edit and add some starting settings for installation purposes:

# Oracle oratab file
#
ASMINS:/u01/app/oracle/product/11.1.0/asm_1:N
CRS:/u01/app/11.1.0/crs
MT11107:/u01/app/oracle/product/11.1.0/db_1:N
MT10204:/u01/app/oracle/product/10.2.0/db_1:N

test oradb as oracle

oradb ASMINS
echo $ORACLE_HOME
echo $PATH
echo $LD_LIBRARY_PATH
echo $ORACLE_SID

oradb
should get a menu to select env.
echo $ORACLE_HOME
echo $PATH
echo $LD_LIBRARY_PATH
echo $ORACLE_SID

The paths should not be cumulative after switching env's.

13)VERIFY KERNEL PARAMETERS

sysctl -p # should return:
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.core.rmem_default = 4194304
net.core.wmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_max = 262144
kernel.shmmax = 2147483648
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000

14)CONFIG HANGCHECK TIMER

find /lib/modules -name "hangcheck-timer.ko"
echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> /etc/modprobe.conf
echo "/sbin/modprobe hangcheck-timer" >> /etc/rc.local
modprobe hangcheck-timer
grep Hangcheck /var/log/messages | tail -2

15)INSTALL OCFS2

install from root media dir as root using

rpm -Uvh ocfs2*

validate packages were installed:

[root@racbaby2 media]# rpm -qa |grep ocfs |sort
ocfs2-2.6.18-92.el5-1.4.1-1.el5
ocfs2console-1.4.1-1.el5
ocfs2-tools-1.4.1-1.el5
ocfs2-tools-debuginfo-1.4.1-1.el5
ocfs2-tools-devel-1.4.1-1.el5

16)INSTALL ASMLIB

install from root media dir using

rpm -Uvh oracleasm*

validate packages were installed:

[root@racbaby2 media]# rpm -qa |grep oracleasm|sort
oracleasm-2.6.18-92.1.22.el5-2.0.5-1.el5
oracleasm-2.6.18-92.el5-2.0.5-1.el5
oracleasmlib-2.0.3-1.el5
oracleasm-support-2.1.2-1.el5

17)SET DATE ON BOTH NODES (IF NEEDED)

date -s "6/25/2007 23:00:00"
date -s "6/25/2007 23:00:20"

18)SETUP SSH INTRA NODE CONNECTIVITY FOR ORACLE

as oracle

ssh-keygen -t rsa -P ""
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
ssh racbaby2
ssh-keygen -t rsa -P ""
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
exit
scp .ssh/id_rsa.pub racbaby2:/home/oracle/.ssh/racbaby1.pub
scp racbaby2:/home/oracle/.ssh/id_rsa.pub /home/oracle/.ssh/racbaby2.pub
cat .ssh/racbaby2.pub >> .ssh/authorized_keys
ssh racbaby2
cat .ssh/racbaby1.pub >> .ssh/authorized_keys
exit

TEST

Each oracle account should be able to ssh to itself and to the other nodes without password prompts or banner.

19)CONFIGURE OCFS2

configure as root using:

ocfs2console &

Using the ocfs2console GUI tool, perform the following steps:
This needs to be performed on one of the Oracle RAC nodes in the cluster.

1. Select [Cluster] -> [Configure Nodes...]. This will start the OCFS2 Cluster Stack and bring up the "Node Configuration" dialog.
2. On the "Node Configuration" dialog, click the [Add] button.
* This will bring up the "Add Node" dialog.
* In the "Add Node" dialog, enter both nodes using
racbaby1 / 10.0.2.69
racbaby2 / 10.0.2.68
* Click [Apply] on the "Node Configuration" dialog - All nodes should now be "Active"
* Click [Close] on the "Node Configuration" dialog.
3. Select cluster -> 'propagate'
as soon as you see the Finished! close the window that opened
4. After verifying all values are correct, exit the application using [File] -> [Quit].

On the other node, if o2cb is not running, run the console to start the cluster. Then Check status & config

/etc/init.d/o2cb status
/etc/init.d/o2cb offline ocfs2
/etc/init.d/o2cb unload
/etc/init.d/o2cb configure
/etc/init.d/o2cb load
/etc/init.d/o2cb online ocfs2

20)FORMAT OCFS2 FILE SYSTEM

ON ONE NODE AS ROOT

mkfs.ocfs2 -b 4K -C 32K -N 4 -L oracleexports /dev/ocfs
mkfs.ocfs2 -b 4K -C 32K -N 4 -L asmspfile /dev/spfile

ON BOTH NODES AS ROOT

mount -t ocfs2 -o datavolume,nointr -L "oracleexports" /u02/app/oracle/exports
mount -t ocfs2 -o datavolume,nointr -L "asmspfile" /u02/app/oracle/ASM/spfile

make reboot proof:

echo "LABEL=oracleexports /u02/app/oracle/exports ocfs2 _netdev,datavolume,nointr 0 0" >> /etc/fstab
echo "LABEL=asmspfile /u02/app/oracle/ASM/spfile ocfs2 _netdev,datavolume,nointr 0 0" >> /etc/fstab

verify:

chkconfig --list o2cb
o2cb 0:off 1:off 2:on 3:on 4:on 5:on 6:off

21)INSTALL CVUQDISK

as root
cd /home/oracle/CLUSTER/clusterware/rpm
rpm -iv cvuqdisk-1.0.1-1.rpm

22)CONFIGURE ASMLIB

as root
/etc/init.d/oracleasm configure

/etc/init.d/oracleasm createdisk asmdisk1 /dev/asm1
/etc/init.d/oracleasm createdisk asmdisk2 /dev/asm2
/etc/init.d/oracleasm createdisk asmdisk3 /dev/asm3
/etc/init.d/oracleasm createdisk asmdisk4 /dev/asm4
/etc/init.d/oracleasm createdisk asmdisk5 /dev/asm5
/etc/init.d/oracleasm createdisk asmdisk6 /dev/asm6
/etc/init.d/oracleasm createdisk asmdisk7 /dev/asm7
/etc/init.d/oracleasm createdisk asmdisk8 /dev/asm8
/etc/init.d/oracleasm createdisk asmdisk9 /dev/asm9
/etc/init.d/oracleasm createdisk asmdisk10 /dev/asm10

/etc/init.d/oracleasm scandisks
/etc/init.d/oracleasm listdisks

[root@racbaby2 ~]# /etc/init.d/oracleasm listdisks
ASMDISK1
ASMDISK10
ASMDISK2
ASMDISK3
ASMDISK4
ASMDISK5
ASMDISK6
ASMDISK7
ASMDISK8
ASMDISK9

23)PREPARE ORAINVENTORY DIR ON BOTH NODES

as root
cd /u01/app
mkdir oraInventory
chown oracle:oinstall oraInventory/

24)INSTALL ORACLE CLUSTERWARE

as root:
copy cluster linux.x64_11gR1_clusterware.zip to a CLUSTER dir as root
chown oracle:dba CLUSTER
mv CLUSTER /home/oracle

as oracle:
set environment variables
oradb CRS

cd CLUSTER
unzip linux.x64_11gR1_clusterware.zip
cd clusterware

./runInstaller

name: OraCrs11g_home
path: /u01/app/11.1.0/crs

<checks for parameters - fix any that fail and retry (none did)>

Add node, give hostname

Set private and public network interface to use

OCR location

Normal redundancy
enter /dev/ocr1, /dev/ocr2

Voting Disk Location

Normal redundancy
enter /dev/vote1, /dev/vote2, /dev/vote3

Install

When prompted run on both nodes, one at a time:

/u01/app/oraInventory/orainstRoot.sh
/u01/app/11.1.0/crs/root.sh

If CentOS is used, when complete and it runs the cluster verification utility it will fail if you don't do the
following before going back to the installer and hitting next.

vi /u01/app/11.1.0/crs/cv/admin/cvu_config
and change line
#CV_ASSUME_DISTID=RHEL4
to
CV_ASSUME_DISTID=RHEL5

Finish

25)UPGRADE CRS_HOME to 11.1.0.7

On one node only

As root
mv p6890831_111070_Linux-x86-64.zip /home/oracle/CLUSTER/

As Oracle
set environment variables
oradb CRS
cd CLUSTER
unzip p6890831_111070_Linux-x86-64.zip
cd Disk1
./runInstaller

name: OraCrs11g_home
path: /u01/app/11.1.0/crs

Cluster installation default

Checks for prerequisites - fix any that fail (none did)

Install

When prompted:

1.Log in as the root user.
2.As the root user, perform the following tasks:
a.Shutdown the CRS daemons by issuing the following command:
/u01/app/11.1.0/crs/bin/crsctl stop crs
b.Run the shell script located at:
/u01/app/11.1.0/crs/install/root111.sh
3.After completing this procedure, proceed to the next node and repeat.


26)INSTALL ORACLE ASM HOME

as root,
cd media
mkdir ASM+DB
mv linux.x64_11gR1_database_1013.zip ASM+DB
chown oracle:dba ASM+DB
mv ASM+DB /home/oracle

as oracle
set environment variables
oradb ASMINS

cd ASM+DB
unzip linux.x64_11gR1_database_1013.zip

cd database

./runInstaller

choose Enterprise Edition

Set Name to OraASM11g_home1
Set Path to /u01/app/oracle/product/11.1.0/asm_1

Prerequisite test - ignored swap warning

Select Software Only

Set dba for OSDBA, OSOPER and OSASM

Install

When prompted run
/u01/app/oracle/product/11.1.0/asm_1/root.sh

27)UPGRADE ASM HOME to 11.0.1.7

Use the same p6890831_111070_Linux-x86-64.zip as you did with the CRS HOME

As Oracle
set environment variables
oradb ASMINS

cd CLUSTER
cd Disk1

./runInstaller

name: OraASM11g_home1
path: /u01/app/oracle/product/11.1.0/asm_1

ignore the email for updates request

cluster install selected

Checks for prerequisites - fix any that fail (none did)

Install

When prompted run as root:

/u01/app/oracle/product/11.1.0/asm_1/root.sh
say 'yes' to replace files

28)INSTALL ORACLE 11G DB HOME

Use the same linux.x64_11gR1_database_1013.zip you used for ASM Home install

as oracle
set environment variables
oradb MT11107_1

cd ASM+DB
cd database

./runInstaller

choose Enterprise Edition

Set Name to OraDB11g_home1
Set Path to /u01/app/oracle/product/11.1.0/db_1

Select all nodes in cluster

Prerequisite test - ignored swap warning

Select Software Only

Set dba for OSDBA, OSOPER and OSASM

Install

When prompted run as root:

/u01/app/oracle/product/11.1.0/db_1/root.sh
say 'no' to replace files

29)UPGRADE 11G DB HOME to 11.0.1.7

Use the same p6890831_111070_Linux-x86-64.zip as you did with the CRS and ASM HOME

As Oracle
set environment variables
oradb MT11107
cd CLUSTER
cd Disk1
./runInstaller

name: OraDB11g_home
path: /u01/app/oracle/product/11.1.0/db_1

Checks for prerequisites - fix any that fail (none did)

Install


When prompted run as root:

/u01/app/oracle/product/11.1.0/db_1/root.sh
say 'no' to replace files

30)INSTALL ORACLE 10G HOME

as root
cd media
mkdir 10g
cp 10201_database_linux_x86_64.cpio.gz 10g
chown oracle:dba 10g
mv 10g /home/oracle

as oracle
set environment variables
oradb MT10204

set a different location for TEMP to avoid java.lang.UnsatisfiedLinkError
(Metalink Doc ID: 370120.1)

mkdir /home/oracle/temp
export TEMP=/home/oracle/temp

cd 10g
Unzip the file: gunzip 10201_database_linux_x86_64.cpio.gz
cpio -idmv < 10201_database_linux_x86_64.cpio

cd database

./runInstaller -ignoreSysPrereqs

name: OraDB10g_home
path: /u01/app/oracle/product/10.2.0/db_1

Select all rac nodes

!! Verifications and validations will fail, click on checkboxes
!! and retry until all are greyed out

Select install software only

Install

When prompted run as root on both nodes:

/u01/app/oracle/product/10.2.0/db_1/root.sh
(respond 'n' to replace files)

31)UPGRADE ORACLE 10G HOME TO 10.2.0.4

as root:

cd media
mkdir 10204_patch
mv p6810189_10204_Linux-x86-64.zip 10204_patch/
chown oracle:dba 10204_patch/
mv 10204_patch/ /home/oracle/10g/

as oracle:
set environment variables
oradb MT10204

cd 10g/10204_patch
unzip p6810189_10204_Linux-x86-64.zip

cd Disk1

./runInstaller -ignoreSysPrereqs

name: OraDB10g_home
path: /u01/app/oracle/product/10.2.0/db_1

Select all rac nodes

Verifications did not fail

skip the Oracle Configuration Manager setup

Install

When prompted run as root on both nodes:

/u01/app/oracle/product/10.2.0/db_1/root.sh
(respond 'n' to replace files)

32)CONFIGURE ASM INSTANCE AND DISK GROUPS

As Oracle

set environment variables
oradb ASMINS

Configure the cluster ASM listener

cd $ORACLE_HOME/bin

./netca

Select Cluster configuration
Select all nodes
Select listener configuration
Select Add
Use default LISTENER name
Use TCP as the Selected Protocols
Use standard port number
Select 'No' to configure another listener
Will get a complete! message
Select finish

Configure ASM Instance

./dbca
Select 'Oracle Real Application Clusters Database'
Select 'Configure Automatic Storage management'
Select all the nodes in the cluster
Specify password for ASM instance
Specify the location for spfile in
/u02/app/oracle/ASM/spfile/spfile+ASM.ora
Click Next
A popup will say DBCA will create and start the ASM instance
click OK

Skip the ASM diskgroup creation - click Finish
A popup will ask if you want to perform another operation - Click No

Remove ASMINS oratab entry (no longer needed)

vi /etc/oratab
remove the ASMINS line

Move +ASM oratab entries to the top of the list

vi /etc/oratab
move the +ASMx line to the top of the list of instances

reset environment variables
oradb +ASM1

Login to one ASM instance as SYSASM and create the DiskGroups

sqlplus "/ as SYSASM"
Create DB_DATA and DB_FLASH diskgroups manually:

SQL> create diskgroup MT11107_DATA normal redundancy
disk 'ORCL:ASMDISK1','ORCL:ASMDISK2'
attribute 'compatible.asm'='11.1','compatible.rdbms'='10.2','au_size'='4M';
SQL> create diskgroup MT11107_FLASH normal redundancy
disk 'ORCL:ASMDISK3','ORCL:ASMDISK4','ORCL:ASMDISK5'
attribute 'compatible.asm'='11.1','compatible.rdbms'='10.2','au_size'='4M';
SQL> create diskgroup MT10204_DATA normal redundancy
disk 'ORCL:ASMDISK6','ORCL:ASMDISK7'
attribute 'compatible.asm'='11.1','compatible.rdbms'='10.2','au_size'='4M';
SQL> create diskgroup MT10204_FLASH normal redundancy
disk 'ORCL:ASMDISK8','ORCL:ASMDISK9','ORCL:ASMDISK10'
attribute 'compatible.asm'='11.1','compatible.rdbms'='10.2','au_size'='4M';

Login to ASM instances ON BOTH NODES as SYSASM and create the DiskGroups

reset environment variables
oradb +ASMx (where x is the instance number on the node)

sqlplus "/ as SYSASM" on BOTH NODES

Mount the disk groups
SQL> alter diskgroup MT11107_DATA mount;
SQL> alter diskgroup MT10204_FLASH mount;
SQL> alter diskgroup MT11107_FLASH mount;
SQL> alter diskgroup MT10204_DATA mount;

Add disk groups to spfile on BOTH NODES
SQL> alter system set asm_diskgroups='MT11107_DATA','MT11107_FLASH','MT10204_DATA','MT10204_FLASH' scope=both;

Verify the diskgroups, attributes and automount on instance startup in BOTH NODES

SQL> shutdown immediate;
startup
SQL> select name, total_mb, free_mb from v$asm_disk;

NAME TOTAL_MB FREE_MB
------------------------------ ---------- ----------
ASMDISK1 50012 49936
ASMDISK10 50012 49936
ASMDISK2 50012 49936
ASMDISK3 50012 49936
ASMDISK4 50012 49936
ASMDISK5 50012 49936
ASMDISK6 50012 49936
ASMDISK7 50012 49936
ASMDISK8 50012 49936
ASMDISK9 50012 49936

10 rows selected.

SQL> select name, state, total_mb, free_mb from v$asm_diskgroup;

NAME STATE TOTAL_MB FREE_MB
------------------------------ ----------- ---------- ----------
MT10204_DATA MOUNTED 100024 99872
MT10204_FLASH MOUNTED 150036 149808
MT11107_DATA MOUNTED 100024 99872
MT11107_FLASH MOUNTED 150036 149808

33)CREATE CENTRAL LOCATION FOR ORATAB AND LISTENER FILES

as root on both nodes:

mkdir /var/opt/oracle
chown oracle:dba /var/opt/oracle
mv /etc/oratab /var/opt/oracle
cd /etc
ln -s /var/opt/oracle/oratab oratab

as oracle on both nodes:

set environment variables
oradb +ASMx (where x is the instance number on the node)

shutdown listener
lsnrctl stop

move the listener.ora from ASM home
cd /var/opt/oracle
mv $ORACLE_HOME/network/admin/listener.ora .

move the tnsnames.ora from ASM home (if it exists - otherwise create an empty file in
/var/opt/oracle)

cd /var/opt/oracle
mv $ORACLE_HOME/network/admin/tnsnames.ora .
(or touch /var/opt/oracle/tnsnames.ora if it doesn't exist)

setup links from ASM home, 10G home, CRS home and 11G home
cd $ORACLE_HOME/network/admin
ln -s /var/opt/oracle/listener.ora listener.ora
ln -s /var/opt/oracle/tnsnames.ora tnsnames.ora

oradb MT10204
cd $ORACLE_HOME/network/admin
ln -s /var/opt/oracle/listener.ora listener.ora
ln -s /var/opt/oracle/tnsnames.ora tnsnames.ora

oradb CRS
cd $ORACLE_HOME/network/admin
ln -s /var/opt/oracle/listener.ora listener.ora
ln -s /var/opt/oracle/tnsnames.ora tnsnames.ora

oradb MT11107
cd $ORACLE_HOME/network/admin
ln -s /var/opt/oracle/listener.ora listener.ora
ln -s /var/opt/oracle/tnsnames.ora tnsnames.ora

Verify links are setup
ls -l /u01/app/oracle/product/*/*/network/admin/*.ora
ls -l /u01/app/11.1.0/crs/network/admin/*.ora

edit custom oradb script and replace 2 entries of TNS_ADMIN:

vi /home/oracle/oradb
replace:
export TNS_ADMIN=$ORACLE_HOME/network/admin
with:
export TNS_ADMIN=/var/opt/oracle

copy new oradb to other node
scp oradb racbaby1:/home/oracle

reset environment variables with new oradb
oradb +ASMx (where x is the instance number on the node)

start listener
lsnrctl start

Verify the listener parameter file is listed as the one in /var/opt/oracle

34)REBOOT BOTH NODES AND VERIFY ALL SERVICES REQUIRED COME UP AUTOMATICALLY

as root:
init 6

After servers come back up log in as root (open a window for each node)
ssh -X root@10.0.1.68
ssh -X root@10.0.1.69

On both nodes as root:
check udev and iscsi disk privs (should be oracle:dba)
ls -l /dev/sd* (ignore the ones for local HD)
ls -l /dev/asm*
ls -l /dev/vote*
ls -l /dev/ocfs
ls -l /dev/ocr*

check CRS status on both nodes:
[root@racbaby2 ~]# /etc/init.d/o2cb status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold = 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Checking O2CB heartbeat: Active

[root@racbaby1 ~]# /etc/init.d/o2cb status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold = 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Checking O2CB heartbeat: Active

check ocfs2 mounts by creating a dummy file on one node and verifying
it shows up on the other:

[root@racbaby2 spfile]# cd /u02/app/oracle/ASM/spfile
[root@racbaby2 spfile]# touch bob
[root@racbaby2 spfile]# ls -l
total 64
-rw-r--r-- 1 root root 0 Mar 8 11:44 bob
drwxr-xr-x 2 root root 4096 Mar 7 17:13 lost+found
-rw-r----- 1 oracle oinstall 2560 Mar 7 17:47 spfile+ASM.ora
[root@racbaby2 spfile]# ssh racbaby1 ls -l /u02/app/oracle/ASM/spfile
root@racbaby1's password:
total 64
-rw-r--r-- 1 root root 0 Mar 8 11:44 bob
drwxr-xr-x 2 root root 4096 Mar 7 17:13 lost+found
-rw-r----- 1 oracle oinstall 2560 Mar 7 17:47 spfile+ASM.ora
[root@racbaby2 spfile]# rm bob
rm: remove regular empty file `bob'? y

[root@racbaby2 exports]# cd /u02/app/oracle/exports
[root@racbaby2 exports]# touch bob
[root@racbaby2 exports]# ssh racbaby1 ls -l /u02/app/oracle/exports
root@racbaby1's password:
total 32
-rw-r--r-- 1 root root 0 Mar 8 2009 bob
drwxr-xr-x 2 root root 4096 Mar 7 17:12 lost+found
[root@racbaby2 exports]# rm bob
rm: remove regular empty file `bob'? y

Check hangcheck timer on both nodes:

[root@racbaby2 exports]# grep Hangcheck /var/log/messages | tail -2
Mar 8 11:40:19 racbaby2 kernel: Hangcheck: starting hangcheck timer 0.9.0 (tick is 30 seconds, margin is 180 seconds).
Mar 8 11:40:19 racbaby2 kernel: Hangcheck: Using monotonic_clock().

[root@racbaby1 ~]# grep Hangcheck /var/log/messages | tail -2
Mar 8 11:39:55 racbaby1 kernel: Hangcheck: starting hangcheck timer 0.9.0 (tick is 30 seconds, margin is 180 seconds).
Mar 8 11:39:55 racbaby1 kernel: Hangcheck: Using monotonic_clock().

On both nodes as oracle:

[root@racbaby2 exports]# su - oracle

check ASM listener is up from ASM home

[oracle@racbaby2 ~]$ ps -ef|grep tns
oracle 4437 1 0 12:32 ? 00:00:00 /u01/app/oracle/product/11.1.0/asm_1/bin/tnslsnr LISTENER -inherit
[oracle@racbaby1 ~]$ ps -ef|grep tns
oracle 5140 1 0 12:32 ? 00:00:00 /u01/app/oracle/product/11.1.0/asm_1/bin/tnslsnr LISTENER -inherit

check ASM instance and diskgroups on both nodes

[oracle@racbaby2 ~]$ oradb

Oracle Env Selection Menu
----------------------------------------------------------
## ENV ORACLE_HOME
----------------------------------------------------------
1. (+ASM2 ) /u01/app/oracle/product/11.1.0/asm_1
2. (CRS ) /u01/app/11.1.0/crs
3. (MT11107 ) /u01/app/oracle/product/11.1.0/db_1
4. (MT10204 ) /u01/app/oracle/product/10.2.0/db_1

Q. Quit w/o setting variables

Enter environment Choice (1-4) [q]: 1

oracle@racbaby2 [/home/oracle] +ASM2
$ sqlplus "/ as sysasm"

SQL*Plus: Release 11.1.0.7.0 - Production on Sun Mar 8 11:55:20 2009

Copyright (c) 1982, 2008, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

SQL> select name, state, total_mb, free_mb from v$asm_diskgroup;

NAME STATE TOTAL_MB FREE_MB
------------------------------ ----------- ---------- ----------
MT10204_DATA MOUNTED 100024 99872
MT10204_FLASH MOUNTED 150036 149808
MT11107_DATA MOUNTED 100024 99872
MT11107_FLASH MOUNTED 150036 149808

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--

35)CREATE 11G database MT11107

Log in as oracle, run cluvfy before starting

ssh -X 10.0.1.69

set environment variables
oradb CRS

cd $ORACLE_HOME/bin
./cluvfy stage -pre dbinst -n racbaby1,racbaby2 -verbose

Make sure it says successful

Create the database
set environment variables
oradb MT11107

cd $ORACLE_HOME/bin
./dbca
Select Real Application Clusters Database
Select create a database
Select all nodes
Select Custom Database
set Global database name and SID to MT11107
Uncheck Configure Enterprise Manager
select use the Same Administrative Passwords for all accounts
set password
Select ASM for the storage
Select MT11107_DATA and MT11107_FLASH ASM Disk groups
Select Use Oracle-Managed Files
+MT11107_DATA should be filled in for the Database Area
Select Specify Flash Recovery Area
click browse and select MT11107_FLASH
Leave Size at 2048 (will make it bigger later as the max size is 49826)
Select Enable Archiving
Deselect all DB Components except "Oracle Text" - ignore custom scripts tab
Next Panel:
Memory Tab
Set SGA Memory Size 1024
Sizing Tab
Leave block size at 8192 and processes at 150
Character Sets Tab
Leave default selected
Connection Mode tab
Leave Dedicated Server Mode selected
Leave "Keep the enhanced 11g default security settings" selected
Leave "Enable automatic maintenance tasks" selected
Accept Storage Defaults
Select "Create Database" and "Generate Database Creation scripts"
Click Ok
A second or two later the database creation scripts dialog will start,
a popup will say "scripts created" - click ok
Next the DBCA will create the database
Wait until the "Creating and starting Oracle instance" step completes, then get a beer
When complete the password management dialog will appear. Click Exit. Wait.
Another dialog will appear stating cluster db MT11107 and it's instances MT111071, MT111072 are starting.
Then it will close all windows without ceremony.

Validate

As oracle check for pmon's and listener running on each node:

oracle@racbaby1 [/u01/app/oracle/product/11.1.0/db_1/bin] MT11107
$ ps -ef|grep pmon|grep -v grep
oracle 5085 1 0 14:05 ? 00:00:00 ora_pmon_MT111071
oracle 9318 1 0 11:40 ? 00:00:00 asm_pmon_+ASM1

oracle@racbaby1 [/u01/app/oracle/product/11.1.0/db_1/bin] MT11107
$ ps -ef|grep tns|grep -v grep
oracle 5140 1 0 12:32 ? 00:00:00 /u01/app/oracle/product/11.1.0/asm_1/bin/tnslsnr LISTENER -inherit

oracle@racbaby2 [/home/oracle] MT11107
$ ps -ef|grep pmon|grep -v grep
oracle 2820 1 0 14:06 ? 00:00:00 ora_pmon_MT111072
oracle 9249 1 0 11:41 ? 00:00:00 asm_pmon_+ASM2

oracle@racbaby2 [/home/oracle] MT11107
$ ps -ef|grep tns|grep -v grep
oracle 4437 1 0 12:32 ? 00:00:00 /u01/app/oracle/product/11.1.0/asm_1/bin/tnslsnr LISTENER -inherit


Check tnsping for the new node. Ping the database and each instance

$ tnsping MT11107
$ tnsping MT111071
$ tnsping MT111072

You should get something like:

Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racbaby1-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = racbaby2-vip)(PORT = 1521)) (LOAD_BALANCE = yes)
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = MT11107)))
OK (0 msec)

Test connect to the database and each instance

$ sqlplus sys@MT11107 as sysdba

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
MT111071

SQL> exit

$ sqlplus sys@MT111071 as sysdba

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
MT111071

SQL> exit

$ sqlplus sys@MT111072 as sysdba

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
MT111072

SQL> exit

shutdown the database

srvctl stop database -d MT11107 -o immediate

Verify shutdown:

oracle@racbaby1 [/u01/app/oracle/product/11.1.0/db_1/bin] MT11107
$ ps -ef|grep pmon|grep -v grep
oracle 9318 1 0 11:40 ? 00:00:00 asm_pmon_+ASM1

oracle@racbaby1 [/u01/app/oracle/product/11.1.0/db_1/bin] MT11107
$ ssh racbaby2 ps -ef|grep pmon|grep -v grep
oracle 9249 1 0 11:41 ? 00:00:00 asm_pmon_+ASM2

36)CREATE 10G database MT10204

set environment variables
oradb MT10204

cd $ORACLE_HOME/bin

./dbca

Select Real Application Clusters Database
Select create a database
Select all nodes
Select Custom Database
Global database name and SID to MT10204
Uncheck Configure Enterprise Manager
select use the Same Administrative Passwords for all accounts
set password
Select ASM for the storage
Select MT10204_DATA and MT10204_FLASH ASM Disk groups
Select Use Oracle-Managed Files
+MT10204_DATA should be filled in for the Database Area
Select Specify Flash Recovery Area
click browse and select MT10204_FLASH
Leave Size at 2048 (will make it bigger later as the max size is 49826)
Select Enable Archiving
Deselect all DB Components except "Oracle Data Mining" - ignore custom scripts tab
Accept the single MT10204 Database Service
Next Panel:
Memory Tab
select custom
Set SGA Memory Size 600mb
Sizing Tab
Leave block size at 8192 and processes at 150
Character Sets Tab
Leave default selected
Connection Mode tab
Leave Dedicated Server Mode selected
Accept Storage Defaults
Select "Create Database" and "Generate Database Creation scripts"
Click Finish
Click Ok
A second or two later the database creation scripts dialog will start,
a popup will say "scripts created" - click ok
Next the DBCA will create the database
Wait until the "Creating and starting Oracle instance" step completes, then get another beer
When complete the password management dialog will appear. Click Exit. Wait.
Another dialog will appear stating cluster db MT10204 and it's instances MT102041, MT102042 are starting.
Then it will close all windows without ceremony.


Validate

As oracle check for pmon's running on each node:

$ ps -ef|grep pmon|grep -v grep


Check tnsping for the new node. Ping the database and each instance

$ tnsping MT10204
$ tnsping MT102041
$ tnsping MT102042

You should get something like:

Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racbaby1-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = racbaby2-vip)(PORT = 1521)) (LOAD_BALANCE = yes)
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = MT10204)))
OK (0 msec)

Test connect to the database and each instance

$ sqlplus sys@MT10204 as sysdba

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
MT102042

SQL> exit

$ sqlplus sys@MT102041 as sysdba

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
MT102041

SQL> exit

$ sqlplus sys@MT102042 as sysdba

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
MT102042

SQL> exit

shutdown the database

srvctl stop database -d MT10204 -o immediate

Verify shutdown:

$ ps -ef|grep pmon|grep -v grep
$ ssh racbaby2 ps -ef|grep pmon|grep -v grep