Sign in or 
| Version | User | Scope of changes |
|---|---|---|
| Mar 27 2009, 1:09 PM EDT (current) | corpunk | 5002 words added |
| Mar 27 2009, 1:02 PM EDT | corpunk |
Changes
Key: Additions Deletions
ORACLE RAC LAB BUILD
The purpose of the RAC Lab is to:
a) build a similar system to the work RAC systems and to go through the build installation to identify configuration and installation issues before attempting the same work on the systems at work.
b) become familiar with RAC configuration and installation
c) create a sandbox platform to experiment with different configurations and functional testing
LAB Equipment:
Servers:
2 Dual AMD Opteron 64bit w/4Gb RAM and 2 gigabit ethernet ports,
1 120Gb internal HD
Operating system:
CentOS 5.2 (Red Hat 5.2 binary equiv)
Oracle Software:
Cluster file system OCFS2 1.4.1
ASMLib 2.0
11g Oracle Clusterware
11g RAC ASM
11g Database using 11g RAC ASM
10g Database using 11g RAC ASM
Shared Storage:
Openfiler 2.3
Installed on AMD PC with 4Gb RAM, single 640Gb HD and single gigabit
ethernet port
Network Equipment:
1 8 port gigabit switch (Private Interconnect and Storage Connection)
1 4 port 100mb/s hub (Public)
Staging server:
Apple Macbook
SIGNIFICANT DIFFERENCES BETWEEN LAB SETUP AND WORK SERVERS
=======================================================================================
ITEM LAB WORK
-----------------------------------------------------------------------------------------------------------------------------
Hardware 2 Rackable Systems 2 HP DL585 G5's
Dual AMD Opteron 4 Quad Core AMD Opterons
single core CPU's w/128Gb RAM
w/4Gb RAM & Dual 6 Gigabit Ethernet w/4
Gigabit Ethernet bonded in pairs for private
and public networks 3 HBA's for SAN connections
Disk iSCSI disks from EMC SAN Disks Multipathed
dedicated PC with & connected through HBA's
openfiler 2.3
connected to the
RAC nodes via the
private network
OS CentOS 5.2 Red Hat 5.2
Security No special add ons Keon ESM
Monitoring No special add ons Tivoli, netsnmp/snmp-DC,
BMC Perform&Predict
Backup No special add ons Veritas Netbackup
DB Monitoring Local DB Console Oracle GridControl (Centralized)
INSTALLATION ACTIONS
=======================================================================================
1)INSTALL OS ON EACH NODE
choose CentOS 5.2 x86_64
select base install
when it reboots, for the first time login:
disable firewall and SELinux
skip creating a user
log in as root
Do NOT run automatic updates when complete!!!
2)SETUP NETWORK
Open network admin utility and set up Network
Set hostname
disable DHCP
port eth0 = 10.0.1.6X
port eth1 = 10.0.2.6X
restart network services
Open terminal and edit /etc/hosts
Remove hostname from 127.0.0.1
add the following:
10.0.1.69 racbaby1
10.0.1.68 racbaby2
10.0.2.69 racbaby1-priv
10.0.2.68 racbaby2-priv
10.0.1.169 racbaby1-vip
10.0.1.168 racbaby2-vip
10.0.2.66 openfiler1
3)MODIFY KERNEL AND SETTINGS
use GUI "services" utility to disable updater, card reader, bluetooth, etc
use GUI "Add Software" tool to install these items listed under "development"
development tools
development libraries
java development
legacy systems development
In a terminal window install the following additional packages:
yum install libaio-devel
yum install sysstat
yum install elfutils-libelf-devel
yum install iscsi-initiator-utils.x86_64
yum install lsscsi
yum install xorg-x11-deprecated-libs
yum install libXp.so.6
Set parameters using these commands:
sysctl -w net.core.rmem_default=4194304
sysctl -w net.core.wmem_default=262144
sysctl -w net.core.rmem_max=4194304
sysctl -w net.core.wmem_max=262144
echo "# Default setting in bytes of the socket receive buffer" >>/etc/sysctl.conf
echo "net.core.rmem_default= 4194304" >>/etc/sysctl.conf
echo "# Default setting in bytes of the socket send buffer" >>/etc/sysctl.conf
echo "net.core.wmem_default=262144" >>/etc/sysctl.conf
echo "# Maximum socket receive buffer size which may be set by using" >>/etc/sysctl.conf
echo "# the SO_RCVBUF socket option" >>/etc/sysctl.conf
echo "net.core.rmem_max= 4194304" >>/etc/sysctl.conf
echo "# Maximum socket send buffer size which may be set by using" >>/etc/sysctl.conf
echo "# the SO_SNDBUF socket option" >>/etc/sysctl.conf
echo "net.core.wmem_max=262144" >>/etc/sysctl.conf
sysctl -w kernel.shmmax=2147483648
echo "kernel.shmmax=2147483648" >> /etc/sysctl.conf
sysctl -w kernel.sem="250 32000 100 128"
echo "kernel.sem=250 32000 100 128" >> /etc/sysctl.conf
sysctl -w fs.file-max=65536
echo "fs.file-max=65536" >> /etc/sysctl.conf
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
echo "net.ipv4.ip_local_port_range = 1024 65000" >> /etc/sysctl.conf
cat >> /etc/security/limits.conf <<EOF
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
EOF
cat >> /etc/pam.d/login <<EOF
session required /lib/security/pam_limits.so
EOF
cat >> /etc/profile <<EOF
if [ \$USER = "oracle" ]; then
if [ \$SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
EOF
4)CREATE ORACLE USER
groupadd -g 115 oinstall
groupadd -g 116 dba
useradd -m -u 175 -g oinstall -G dba -d /home/oracle -s /bin/bash -c "Oracle Software Owner" oracle
passwd oracle
5)CREATE SPACE FOR HOMES
mkdir -p /u01/app/oracle
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/app/oracle
mkdir -p /u01/app/11.1.0
chown -R oracle:oinstall /u01/app/11.1.0
chmod -R 775 /u01/app/11.1.0
6)CREATE ISCSI DISKS
Use openfiler to set up disks and map luns to the two nodes. I'm not going to
listout how I setup Openfiler as it doesn't pertain to the purpose of the LAB
build.
For simplicity I named the luns according to function, so that when they appear on the nodes you'll see something like the following in /var/log/messages:
[root@racbaby2 log]# cat messages.1|grep openf|sort
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm10,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm1,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm2,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm3,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm4,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm5,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm6,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm7,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm8,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm9,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:ocfs,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:ocr1,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:ocr2,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:spfile,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:vote1,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:vote2,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:vote3,10.0.2.66,3260]
7)COPY OVER SOFTWARE FILES FROM STAGING SERVER
Hangcheck Timer
hangcheck-0.8.0.tar.gz
Oracle CLUSTER and DB Software packages:
linux.x64_11gR1_clusterware.zip
linux.x64_11gR1_database_1013.zip
10201_database_linux_x86_64.cpio.gz
Oracle OCFS RPM's
ocfs2-2.6.18-92.el5-1.4.1-1.el5.x86_64.rpm
ocfs2-tools-1.4.1-1.el5.x86_64.rpm
ocfs2-tools-debuginfo-1.4.1-1.el5.x86_64.rpm
ocfs2-tools-devel-1.4.1-1.el5.x86_64.rpm
ocfs2console-1.4.1-1.el5.x86_64.rpm
ASMLib RPM's
oracleasm-2.6.18-92.1.22.el5-2.0.5-1.el5.x86_64.rpm
oracleasm-2.6.18-92.el5-2.0.5-1.el5.x86_64.rpm
oracleasm-support-2.1.2-1.el5.x86_64.rpm
oracleasmlib-2.0.3-1.el5.x86_64.rpm
homemade env variable switcher script
oradb
Oracle patchsets:
p6810189_10204_Linux-x86-64.zip
p6890831_111070_Linux-x86-64.zip
8)SET UP ISCSI ON RAC NODES
as root on each node:
service iscsi restart
chkconfig --level 345 iscsi on
iscsiadm -m discovery -t sendtargets -p openfiler1
service iscsi restart
verify disks are there
fdisk -l
run fdisk on each device to create a primary partition
9)SET UP CONSISTENT UDEV DEVICE NAMES AND PERMISSIONS ON EACH NODE
As root on each node:
look at /proc/scsi/scsi for this:
Host: scsi20 Channel: 00 Id: 00 Lun: 00
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
then edit /etc/iscsi/iscsid.conf and add at the bottom:
vendor=OPNFILER, model="VIRTUAL-DISK". options=-g
now add the following line at the end of /etc/udev/rules.d/50-udev.rules
KERNEL=="sd?", BUS=="scsi", ENV{ID_MODEL}=="VIRTUAL-DISK", ENV{ID_PATH}=="*iscsi*", OWNER="oracle", \
GROUP="dba", MODE="640", RUN+="/etc/udev/scripts/iscsi $env{ID_PATH} %k"
KERNEL=="sd?1", BUS=="scsi", ENV{ID_MODEL}=="VIRTUAL-DISK", ENV{ID_PATH}=="*iscsi*", OWNER="oracle", \
GROUP="dba", MODE="640"
create a file called /etc/udev/scripts/iscsi and add the following
#!/bin/bash
LINK_NAME=`/bin/echo $1 | awk -F":" '{ print $3 }'`
/bin/ln -s /dev/${2}1 /dev/${LINK_NAME}
Then make it executable using:
chmod 700 /etc/udev/scripts/iscsi
reboot both nodes
verify you see good named links to the partitioned disks (may have to reboot again for disks to show up correctly)
$ ls -l /dev |grep ">" |grep sd |egrep -v "core|MAKE|floppy|proc"
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm1 -> /dev/sdg1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm10 -> /dev/sdd1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm2 -> /dev/sdk1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm3 -> /dev/sdc1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm4 -> /dev/sdn1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm5 -> /dev/sdf1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm6 -> /dev/sdb1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm7 -> /dev/sdo1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm8 -> /dev/sdj1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm9 -> /dev/sdm1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 ocfs -> /dev/sde1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 ocr1 -> /dev/sdp1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 ocr2 -> /dev/sdh1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 spfile -> /dev/sdi1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 vote1 -> /dev/sdl1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 vote2 -> /dev/sdq1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 vote3 -> /dev/sdr1
verify disks are reachable by oracle and not owned by root:
$ ls -l /dev/sd*1|grep -v ">"|grep -v sda1
brw-r----- 1 oracle dba 8, 17 Mar 7 10:32 /dev/sdb1
brw-r----- 1 oracle dba 8, 33 Mar 7 10:32 /dev/sdc1
brw-r----- 1 oracle dba 8, 49 Mar 7 10:32 /dev/sdd1
brw-r----- 1 oracle dba 8, 65 Mar 7 10:32 /dev/sde1
brw-r----- 1 oracle dba 8, 81 Mar 7 10:32 /dev/sdf1
brw-r----- 1 oracle dba 8, 97 Mar 7 10:32 /dev/sdg1
brw-r----- 1 oracle dba 8, 113 Mar 7 10:48 /dev/sdh1
brw-r----- 1 oracle dba 8, 129 Mar 7 10:32 /dev/sdi1
brw-r----- 1 oracle dba 8, 145 Mar 7 10:32 /dev/sdj1
brw-r----- 1 oracle dba 8, 161 Mar 7 10:32 /dev/sdk1
brw-r----- 1 oracle dba 8, 177 Mar 7 13:23 /dev/sdl1
brw-r----- 1 oracle dba 8, 193 Mar 7 10:32 /dev/sdm1
brw-r----- 1 oracle dba 8, 209 Mar 7 10:32 /dev/sdn1
brw-r----- 1 oracle dba 8, 225 Mar 7 10:32 /dev/sdo1
brw-r----- 1 oracle dba 8, 241 Mar 7 10:48 /dev/sdp1
brw-r----- 1 oracle dba 65, 1 Mar 7 13:23 /dev/sdq1
brw-r----- 1 oracle dba 65, 17 Mar 7 13:23 /dev/sdr1
10)CREATE OCFS2 DIRECTORIES
mkdir -p /u02/app/oracle/exports
chown -R oracle:dba /u02/app/oracle/exports
chmod -R 775 /u02/app/oracle/exports
mkdir -p /u02/app/oracle/ASM/spfile
chown -R oracle:dba /u02/app/oracle/ASM/spfile
chmod -R 775 /u02/app/oracle/ASM/spfile
11)EDIT ORACLE BASH PROFILE
edit .bash_profile for oracle to match as follows:
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
alias oradb=". /home/oracle/oradb 2>/dev/null"
12)EDIT ORATAB FILE
as root
touch /etc/oratab
chown oracle:dba /etc/oratab
edit and add some starting settings for installation purposes:
# Oracle oratab file
#
ASMINS:/u01/app/oracle/product/11.1.0/asm_1:N
CRS:/u01/app/11.1.0/crs
MT11107:/u01/app/oracle/product/11.1.0/db_1:N
MT10204:/u01/app/oracle/product/10.2.0/db_1:N
test oradb as oracle
oradb ASMINS
echo $ORACLE_HOME
echo $PATH
echo $LD_LIBRARY_PATH
echo $ORACLE_SID
oradb
should get a menu to select env.
echo $ORACLE_HOME
echo $PATH
echo $LD_LIBRARY_PATH
echo $ORACLE_SID
The paths should not be cumulative after switching env's.
13)VERIFY KERNEL PARAMETERS
sysctl -p # should return:
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.core.rmem_default = 4194304
net.core.wmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_max = 262144
kernel.shmmax = 2147483648
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
14)CONFIG HANGCHECK TIMER
find /lib/modules -name "hangcheck-timer.ko"
echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> /etc/modprobe.conf
echo "/sbin/modprobe hangcheck-timer" >> /etc/rc.local
modprobe hangcheck-timer
grep Hangcheck /var/log/messages | tail -2
15)INSTALL OCFS2
install from root media dir as root using
rpm -Uvh ocfs2*
validate packages were installed:
[root@racbaby2 media]# rpm -qa |grep ocfs |sort
ocfs2-2.6.18-92.el5-1.4.1-1.el5
ocfs2console-1.4.1-1.el5
ocfs2-tools-1.4.1-1.el5
ocfs2-tools-debuginfo-1.4.1-1.el5
ocfs2-tools-devel-1.4.1-1.el5
16)INSTALL ASMLIB
install from root media dir using
rpm -Uvh oracleasm*
validate packages were installed:
[root@racbaby2 media]# rpm -qa |grep oracleasm|sort
oracleasm-2.6.18-92.1.22.el5-2.0.5-1.el5
oracleasm-2.6.18-92.el5-2.0.5-1.el5
oracleasmlib-2.0.3-1.el5
oracleasm-support-2.1.2-1.el5
17)SET DATE ON BOTH NODES (IF NEEDED)
date -s "6/25/2007 23:00:00"
date -s "6/25/2007 23:00:20"
18)SETUP SSH INTRA NODE CONNECTIVITY FOR ORACLE
as oracle
ssh-keygen -t rsa -P ""
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
ssh racbaby2
ssh-keygen -t rsa -P ""
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
exit
scp .ssh/id_rsa.pub racbaby2:/home/oracle/.ssh/racbaby1.pub
scp racbaby2:/home/oracle/.ssh/id_rsa.pub /home/oracle/.ssh/racbaby2.pub
cat .ssh/racbaby2.pub >> .ssh/authorized_keys
ssh racbaby2
cat .ssh/racbaby1.pub >> .ssh/authorized_keys
exit
TEST
Each oracle account should be able to ssh to itself and to the other nodes without password prompts or banner.
19)CONFIGURE OCFS2
configure as root using:
ocfs2console &
Using the ocfs2console GUI tool, perform the following steps:
This needs to be performed on one of the Oracle RAC nodes in the cluster.
1. Select [Cluster] -> [Configure Nodes...]. This will start the OCFS2 Cluster Stack and bring up the "Node Configuration" dialog.
2. On the "Node Configuration" dialog, click the [Add] button.
* This will bring up the "Add Node" dialog.
* In the "Add Node" dialog, enter both nodes using
racbaby1 / 10.0.2.69
racbaby2 / 10.0.2.68
* Click [Apply] on the "Node Configuration" dialog - All nodes should now be "Active"
* Click [Close] on the "Node Configuration" dialog.
3. Select cluster -> 'propagate'
as soon as you see the Finished! close the window that opened
4. After verifying all values are correct, exit the application using [File] -> [Quit].
On the other node, if o2cb is not running, run the console to start the cluster. Then Check status & config
/etc/init.d/o2cb status
/etc/init.d/o2cb offline ocfs2
/etc/init.d/o2cb unload
/etc/init.d/o2cb configure
/etc/init.d/o2cb load
/etc/init.d/o2cb online ocfs2
20)FORMAT OCFS2 FILE SYSTEM
ON ONE NODE AS ROOT
mkfs.ocfs2 -b 4K -C 32K -N 4 -L oracleexports /dev/ocfs
mkfs.ocfs2 -b 4K -C 32K -N 4 -L asmspfile /dev/spfile
ON BOTH NODES AS ROOT
mount -t ocfs2 -o datavolume,nointr -L "oracleexports" /u02/app/oracle/exports
mount -t ocfs2 -o datavolume,nointr -L "asmspfile" /u02/app/oracle/ASM/spfile
make reboot proof:
echo "LABEL=oracleexports /u02/app/oracle/exports ocfs2 _netdev,datavolume,nointr 0 0" >> /etc/fstab
echo "LABEL=asmspfile /u02/app/oracle/ASM/spfile ocfs2 _netdev,datavolume,nointr 0 0" >> /etc/fstab
verify:
chkconfig --list o2cb
o2cb 0:off 1:off 2:on 3:on 4:on 5:on 6:off
21)INSTALL CVUQDISK
as root
cd /home/oracle/CLUSTER/clusterware/rpm
rpm -iv cvuqdisk-1.0.1-1.rpm
22)CONFIGURE ASMLIB
as root
/etc/init.d/oracleasm configure
/etc/init.d/oracleasm createdisk asmdisk1 /dev/asm1
/etc/init.d/oracleasm createdisk asmdisk2 /dev/asm2
/etc/init.d/oracleasm createdisk asmdisk3 /dev/asm3
/etc/init.d/oracleasm createdisk asmdisk4 /dev/asm4
/etc/init.d/oracleasm createdisk asmdisk5 /dev/asm5
/etc/init.d/oracleasm createdisk asmdisk6 /dev/asm6
/etc/init.d/oracleasm createdisk asmdisk7 /dev/asm7
/etc/init.d/oracleasm createdisk asmdisk8 /dev/asm8
/etc/init.d/oracleasm createdisk asmdisk9 /dev/asm9
/etc/init.d/oracleasm createdisk asmdisk10 /dev/asm10
/etc/init.d/oracleasm scandisks
/etc/init.d/oracleasm listdisks
[root@racbaby2 ~]# /etc/init.d/oracleasm listdisks
ASMDISK1
ASMDISK10
ASMDISK2
ASMDISK3
ASMDISK4
ASMDISK5
ASMDISK6
ASMDISK7
ASMDISK8
ASMDISK9
23)PREPARE ORAINVENTORY DIR ON BOTH NODES
as root
cd /u01/app
mkdir oraInventory
chown oracle:oinstall oraInventory/
24)INSTALL ORACLE CLUSTERWARE
as root:
copy cluster linux.x64_11gR1_clusterware.zip to a CLUSTER dir as root
chown oracle:dba CLUSTER
mv CLUSTER /home/oracle
as oracle:
set environment variables
oradb CRS
cd CLUSTER
unzip linux.x64_11gR1_clusterware.zip
cd clusterware
./runInstaller
name: OraCrs11g_home
path: /u01/app/11.1.0/crs
<checks for parameters - fix any that fail and retry (none did)>
Add node, give hostname
Set private and public network interface to use
OCR location
Normal redundancy
enter /dev/ocr1, /dev/ocr2
Voting Disk Location
Normal redundancy
enter /dev/vote1, /dev/vote2, /dev/vote3
Install
When prompted run on both nodes, one at a time:
/u01/app/oraInventory/orainstRoot.sh
/u01/app/11.1.0/crs/root.sh
If CentOS is used, when complete and it runs the cluster verification utility it will fail if you don't do the
following before going back to the installer and hitting next.
vi /u01/app/11.1.0/crs/cv/admin/cvu_config
and change line
#CV_ASSUME_DISTID=RHEL4
to
CV_ASSUME_DISTID=RHEL5
Finish
25)UPGRADE CRS_HOME to 11.1.0.7
On one node only
As root
mv p6890831_111070_Linux-x86-64.zip /home/oracle/CLUSTER/
As Oracle
set environment variables
oradb CRS
cd CLUSTER
unzip p6890831_111070_Linux-x86-64.zip
cd Disk1
./runInstaller
name: OraCrs11g_home
path: /u01/app/11.1.0/crs
Cluster installation default
Checks for prerequisites - fix any that fail (none did)
Install
When prompted:
1.Log in as the root user.
2.As the root user, perform the following tasks:
a.Shutdown the CRS daemons by issuing the following command:
/u01/app/11.1.0/crs/bin/crsctl stop crs
b.Run the shell script located at:
/u01/app/11.1.0/crs/install/root111.sh
3.After completing this procedure, proceed to the next node and repeat.
26)INSTALL ORACLE ASM HOME
as root,
cd media
mkdir ASM+DB
mv linux.x64_11gR1_database_1013.zip ASM+DB
chown oracle:dba ASM+DB
mv ASM+DB /home/oracle
as oracle
set environment variables
oradb ASMINS
cd ASM+DB
unzip linux.x64_11gR1_database_1013.zip
cd database
./runInstaller
choose Enterprise Edition
Set Name to OraASM11g_home1
Set Path to /u01/app/oracle/product/11.1.0/asm_1
Prerequisite test - ignored swap warning
Select Software Only
Set dba for OSDBA, OSOPER and OSASM
Install
When prompted run
/u01/app/oracle/product/11.1.0/asm_1/root.sh
27)UPGRADE ASM HOME to 11.0.1.7
Use the same p6890831_111070_Linux-x86-64.zip as you did with the CRS HOME
As Oracle
set environment variables
oradb ASMINS
cd CLUSTER
cd Disk1
./runInstaller
name: OraASM11g_home1
path: /u01/app/oracle/product/11.1.0/asm_1
ignore the email for updates request
cluster install selected
Checks for prerequisites - fix any that fail (none did)
Install
When prompted run as root:
/u01/app/oracle/product/11.1.0/asm_1/root.sh
say 'yes' to replace files
28)INSTALL ORACLE 11G DB HOME
Use the same linux.x64_11gR1_database_1013.zip you used for ASM Home install
as oracle
set environment variables
oradb MT11107_1
cd ASM+DB
cd database
./runInstaller
choose Enterprise Edition
Set Name to OraDB11g_home1
Set Path to /u01/app/oracle/product/11.1.0/db_1
Select all nodes in cluster
Prerequisite test - ignored swap warning
Select Software Only
Set dba for OSDBA, OSOPER and OSASM
Install
When prompted run as root:
/u01/app/oracle/product/11.1.0/db_1/root.sh
say 'no' to replace files
29)UPGRADE 11G DB HOME to 11.0.1.7
Use the same p6890831_111070_Linux-x86-64.zip as you did with the CRS and ASM HOME
As Oracle
set environment variables
oradb MT11107
cd CLUSTER
cd Disk1
./runInstaller
name: OraDB11g_home
path: /u01/app/oracle/product/11.1.0/db_1
Checks for prerequisites - fix any that fail (none did)
Install
When prompted run as root:
/u01/app/oracle/product/11.1.0/db_1/root.sh
say 'no' to replace files
30)INSTALL ORACLE 10G HOME
as root
cd media
mkdir 10g
cp 10201_database_linux_x86_64.cpio.gz 10g
chown oracle:dba 10g
mv 10g /home/oracle
as oracle
set environment variables
oradb MT10204
set a different location for TEMP to avoid java.lang.UnsatisfiedLinkError
(Metalink Doc ID: 370120.1)
mkdir /home/oracle/temp
export TEMP=/home/oracle/temp
cd 10g
Unzip the file: gunzip 10201_database_linux_x86_64.cpio.gz
cpio -idmv < 10201_database_linux_x86_64.cpio
cd database
./runInstaller -ignoreSysPrereqs
name: OraDB10g_home
path: /u01/app/oracle/product/10.2.0/db_1
Select all rac nodes
!! Verifications and validations will fail, click on checkboxes
!! and retry until all are greyed out
Select install software only
Install
When prompted run as root on both nodes:
/u01/app/oracle/product/10.2.0/db_1/root.sh
(respond 'n' to replace files)
31)UPGRADE ORACLE 10G HOME TO 10.2.0.4
as root:
cd media
mkdir 10204_patch
mv p6810189_10204_Linux-x86-64.zip 10204_patch/
chown oracle:dba 10204_patch/
mv 10204_patch/ /home/oracle/10g/
as oracle:
set environment variables
oradb MT10204
cd 10g/10204_patch
unzip p6810189_10204_Linux-x86-64.zip
cd Disk1
./runInstaller -ignoreSysPrereqs
name: OraDB10g_home
path: /u01/app/oracle/product/10.2.0/db_1
Select all rac nodes
Verifications did not fail
skip the Oracle Configuration Manager setup
Install
When prompted run as root on both nodes:
/u01/app/oracle/product/10.2.0/db_1/root.sh
(respond 'n' to replace files)
32)CONFIGURE ASM INSTANCE AND DISK GROUPS
As Oracle
set environment variables
oradb ASMINS
Configure the cluster ASM listener
cd $ORACLE_HOME/bin
./netca
Select Cluster configuration
Select all nodes
Select listener configuration
Select Add
Use default LISTENER name
Use TCP as the Selected Protocols
Use standard port number
Select 'No' to configure another listener
Will get a complete! message
Select finish
Configure ASM Instance
./dbca
Select 'Oracle Real Application Clusters Database'
Select 'Configure Automatic Storage management'
Select all the nodes in the cluster
Specify password for ASM instance
Specify the location for spfile in
/u02/app/oracle/ASM/spfile/spfile+ASM.ora
Click Next
A popup will say DBCA will create and start the ASM instance
click OK
Skip the ASM diskgroup creation - click Finish
A popup will ask if you want to perform another operation - Click No
Remove ASMINS oratab entry (no longer needed)
vi /etc/oratab
remove the ASMINS line
Move +ASM oratab entries to the top of the list
vi /etc/oratab
move the +ASMx line to the top of the list of instances
reset environment variables
oradb +ASM1
Login to one ASM instance as SYSASM and create the DiskGroups
sqlplus "/ as SYSASM"
Create DB_DATA and DB_FLASH diskgroups manually:
SQL> create diskgroup MT11107_DATA normal redundancy
disk 'ORCL:ASMDISK1','ORCL:ASMDISK2'
attribute 'compatible.asm'='11.1','compatible.rdbms'='10.2','au_size'='4M';
SQL> create diskgroup MT11107_FLASH normal redundancy
disk 'ORCL:ASMDISK3','ORCL:ASMDISK4','ORCL:ASMDISK5'
attribute 'compatible.asm'='11.1','compatible.rdbms'='10.2','au_size'='4M';
SQL> create diskgroup MT10204_DATA normal redundancy
disk 'ORCL:ASMDISK6','ORCL:ASMDISK7'
attribute 'compatible.asm'='11.1','compatible.rdbms'='10.2','au_size'='4M';
SQL> create diskgroup MT10204_FLASH normal redundancy
disk 'ORCL:ASMDISK8','ORCL:ASMDISK9','ORCL:ASMDISK10'
attribute 'compatible.asm'='11.1','compatible.rdbms'='10.2','au_size'='4M';
Login to ASM instances ON BOTH NODES as SYSASM and create the DiskGroups
reset environment variables
oradb +ASMx (where x is the instance number on the node)
sqlplus "/ as SYSASM" on BOTH NODES
Mount the disk groups
SQL> alter diskgroup MT11107_DATA mount;
SQL> alter diskgroup MT10204_FLASH mount;
SQL> alter diskgroup MT11107_FLASH mount;
SQL> alter diskgroup MT10204_DATA mount;
Add disk groups to spfile on BOTH NODES
SQL> alter system set asm_diskgroups='MT11107_DATA','MT11107_FLASH','MT10204_DATA','MT10204_FLASH' scope=both;
Verify the diskgroups, attributes and automount on instance startup in BOTH NODES
SQL> shutdown immediate;
startup
SQL> select name, total_mb, free_mb from v$asm_disk;
NAME TOTAL_MB FREE_MB
------------------------------ ---------- ----------
ASMDISK1 50012 49936
ASMDISK10 50012 49936
ASMDISK2 50012 49936
ASMDISK3 50012 49936
ASMDISK4 50012 49936
ASMDISK5 50012 49936
ASMDISK6 50012 49936
ASMDISK7 50012 49936
ASMDISK8 50012 49936
ASMDISK9 50012 49936
10 rows selected.
SQL> select name, state, total_mb, free_mb from v$asm_diskgroup;
NAME STATE TOTAL_MB FREE_MB
------------------------------ ----------- ---------- ----------
MT10204_DATA MOUNTED 100024 99872
MT10204_FLASH MOUNTED 150036 149808
MT11107_DATA MOUNTED 100024 99872
MT11107_FLASH MOUNTED 150036 149808
33)CREATE CENTRAL LOCATION FOR ORATAB AND LISTENER FILES
as root on both nodes:
mkdir /var/opt/oracle
chown oracle:dba /var/opt/oracle
mv /etc/oratab /var/opt/oracle
cd /etc
ln -s /var/opt/oracle/oratab oratab
as oracle on both nodes:
set environment variables
oradb +ASMx (where x is the instance number on the node)
shutdown listener
lsnrctl stop
move the listener.ora from ASM home
cd /var/opt/oracle
mv $ORACLE_HOME/network/admin/listener.ora .
move the tnsnames.ora from ASM home (if it exists - otherwise create an empty file in
/var/opt/oracle)
cd /var/opt/oracle
mv $ORACLE_HOME/network/admin/tnsnames.ora .
(or touch /var/opt/oracle/tnsnames.ora if it doesn't exist)
setup links from ASM home, 10G home, CRS home and 11G home
cd $ORACLE_HOME/network/admin
ln -s /var/opt/oracle/listener.ora listener.ora
ln -s /var/opt/oracle/tnsnames.ora tnsnames.ora
oradb MT10204
cd $ORACLE_HOME/network/admin
ln -s /var/opt/oracle/listener.ora listener.ora
ln -s /var/opt/oracle/tnsnames.ora tnsnames.ora
oradb CRS
cd $ORACLE_HOME/network/admin
ln -s /var/opt/oracle/listener.ora listener.ora
ln -s /var/opt/oracle/tnsnames.ora tnsnames.ora
oradb MT11107
cd $ORACLE_HOME/network/admin
ln -s /var/opt/oracle/listener.ora listener.ora
ln -s /var/opt/oracle/tnsnames.ora tnsnames.ora
Verify links are setup
ls -l /u01/app/oracle/product/*/*/network/admin/*.ora
ls -l /u01/app/11.1.0/crs/network/admin/*.ora
edit custom oradb script and replace 2 entries of TNS_ADMIN:
vi /home/oracle/oradb
replace:
export TNS_ADMIN=$ORACLE_HOME/network/admin
with:
export TNS_ADMIN=/var/opt/oracle
copy new oradb to other node
scp oradb racbaby1:/home/oracle
reset environment variables with new oradb
oradb +ASMx (where x is the instance number on the node)
start listener
lsnrctl start
Verify the listener parameter file is listed as the one in /var/opt/oracle
34)REBOOT BOTH NODES AND VERIFY ALL SERVICES REQUIRED COME UP AUTOMATICALLY
as root:
init 6
After servers come back up log in as root (open a window for each node)
ssh -X root@10.0.1.68
ssh -X root@10.0.1.69
On both nodes as root:
check udev and iscsi disk privs (should be oracle:dba)
ls -l /dev/sd* (ignore the ones for local HD)
ls -l /dev/asm*
ls -l /dev/vote*
ls -l /dev/ocfs
ls -l /dev/ocr*
check CRS status on both nodes:
[root@racbaby2 ~]# /etc/init.d/o2cb status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold = 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Checking O2CB heartbeat: Active
[root@racbaby1 ~]# /etc/init.d/o2cb status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold = 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Checking O2CB heartbeat: Active
check ocfs2 mounts by creating a dummy file on one node and verifying
it shows up on the other:
[root@racbaby2 spfile]# cd /u02/app/oracle/ASM/spfile
[root@racbaby2 spfile]# touch bob
[root@racbaby2 spfile]# ls -l
total 64
-rw-r--r-- 1 root root 0 Mar 8 11:44 bob
drwxr-xr-x 2 root root 4096 Mar 7 17:13 lost+found
-rw-r----- 1 oracle oinstall 2560 Mar 7 17:47 spfile+ASM.ora
[root@racbaby2 spfile]# ssh racbaby1 ls -l /u02/app/oracle/ASM/spfile
root@racbaby1's password:
total 64
-rw-r--r-- 1 root root 0 Mar 8 11:44 bob
drwxr-xr-x 2 root root 4096 Mar 7 17:13 lost+found
-rw-r----- 1 oracle oinstall 2560 Mar 7 17:47 spfile+ASM.ora
[root@racbaby2 spfile]# rm bob
rm: remove regular empty file `bob'? y
[root@racbaby2 exports]# cd /u02/app/oracle/exports
[root@racbaby2 exports]# touch bob
[root@racbaby2 exports]# ssh racbaby1 ls -l /u02/app/oracle/exports
root@racbaby1's password:
total 32
-rw-r--r-- 1 root root 0 Mar 8 2009 bob
drwxr-xr-x 2 root root 4096 Mar 7 17:12 lost+found
[root@racbaby2 exports]# rm bob
rm: remove regular empty file `bob'? y
Check hangcheck timer on both nodes:
[root@racbaby2 exports]# grep Hangcheck /var/log/messages | tail -2
Mar 8 11:40:19 racbaby2 kernel: Hangcheck: starting hangcheck timer 0.9.0 (tick is 30 seconds, margin is 180 seconds).
Mar 8 11:40:19 racbaby2 kernel: Hangcheck: Using monotonic_clock().
[root@racbaby1 ~]# grep Hangcheck /var/log/messages | tail -2
Mar 8 11:39:55 racbaby1 kernel: Hangcheck: starting hangcheck timer 0.9.0 (tick is 30 seconds, margin is 180 seconds).
Mar 8 11:39:55 racbaby1 kernel: Hangcheck: Using monotonic_clock().
On both nodes as oracle:
[root@racbaby2 exports]# su - oracle
check ASM listener is up from ASM home
[oracle@racbaby2 ~]$ ps -ef|grep tns
oracle 4437 1 0 12:32 ? 00:00:00 /u01/app/oracle/product/11.1.0/asm_1/bin/tnslsnr LISTENER -inherit
[oracle@racbaby1 ~]$ ps -ef|grep tns
oracle 5140 1 0 12:32 ? 00:00:00 /u01/app/oracle/product/11.1.0/asm_1/bin/tnslsnr LISTENER -inherit
check ASM instance and diskgroups on both nodes
[oracle@racbaby2 ~]$ oradb
Oracle Env Selection Menu
----------------------------------------------------------
## ENV ORACLE_HOME
----------------------------------------------------------
1. (+ASM2 ) /u01/app/oracle/product/11.1.0/asm_1
2. (CRS ) /u01/app/11.1.0/crs
3. (MT11107 ) /u01/app/oracle/product/11.1.0/db_1
4. (MT10204 ) /u01/app/oracle/product/10.2.0/db_1
Q. Quit w/o setting variables
Enter environment Choice (1-4) [q]: 1
oracle@racbaby2 [/home/oracle] +ASM2
$ sqlplus "/ as sysasm"
SQL*Plus: Release 11.1.0.7.0 - Production on Sun Mar 8 11:55:20 2009
Copyright (c) 1982, 2008, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
SQL> select name, state, total_mb, free_mb from v$asm_diskgroup;
NAME STATE TOTAL_MB FREE_MB
------------------------------ ----------- ---------- ----------
MT10204_DATA MOUNTED 100024 99872
MT10204_FLASH MOUNTED 150036 149808
MT11107_DATA MOUNTED 100024 99872
MT11107_FLASH MOUNTED 150036 149808
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
35)CREATE 11G database MT11107
Log in as oracle, run cluvfy before starting
ssh -X 10.0.1.69
set environment variables
oradb CRS
cd $ORACLE_HOME/bin
./cluvfy stage -pre dbinst -n racbaby1,racbaby2 -verbose
Make sure it says successful
Create the database
set environment variables
oradb MT11107
cd $ORACLE_HOME/bin
./dbca
Select Real Application Clusters Database
Select create a database
Select all nodes
Select Custom Database
set Global database name and SID to MT11107
Uncheck Configure Enterprise Manager
select use the Same Administrative Passwords for all accounts
set password
Select ASM for the storage
Select MT11107_DATA and MT11107_FLASH ASM Disk groups
Select Use Oracle-Managed Files
+MT11107_DATA should be filled in for the Database Area
Select Specify Flash Recovery Area
click browse and select MT11107_FLASH
Leave Size at 2048 (will make it bigger later as the max size is 49826)
Select Enable Archiving
Deselect all DB Components except "Oracle Text" - ignore custom scripts tab
Next Panel:
Memory Tab
Set SGA Memory Size 1024
Sizing Tab
Leave block size at 8192 and processes at 150
Character Sets Tab
Leave default selected
Connection Mode tab
Leave Dedicated Server Mode selected
Leave "Keep the enhanced 11g default security settings" selected
Leave "Enable automatic maintenance tasks" selected
Accept Storage Defaults
Select "Create Database" and "Generate Database Creation scripts"
Click Ok
A second or two later the database creation scripts dialog will start,
a popup will say "scripts created" - click ok
Next the DBCA will create the database
Wait until the "Creating and starting Oracle instance" step completes, then get a beer
When complete the password management dialog will appear. Click Exit. Wait.
Another dialog will appear stating cluster db MT11107 and it's instances MT111071, MT111072 are starting.
Then it will close all windows without ceremony.
Validate
As oracle check for pmon's and listener running on each node:
oracle@racbaby1 [/u01/app/oracle/product/11.1.0/db_1/bin] MT11107
$ ps -ef|grep pmon|grep -v grep
oracle 5085 1 0 14:05 ? 00:00:00 ora_pmon_MT111071
oracle 9318 1 0 11:40 ? 00:00:00 asm_pmon_+ASM1
oracle@racbaby1 [/u01/app/oracle/product/11.1.0/db_1/bin] MT11107
$ ps -ef|grep tns|grep -v grep
oracle 5140 1 0 12:32 ? 00:00:00 /u01/app/oracle/product/11.1.0/asm_1/bin/tnslsnr LISTENER -inherit
oracle@racbaby2 [/home/oracle] MT11107
$ ps -ef|grep pmon|grep -v grep
oracle 2820 1 0 14:06 ? 00:00:00 ora_pmon_MT111072
oracle 9249 1 0 11:41 ? 00:00:00 asm_pmon_+ASM2
oracle@racbaby2 [/home/oracle] MT11107
$ ps -ef|grep tns|grep -v grep
oracle 4437 1 0 12:32 ? 00:00:00 /u01/app/oracle/product/11.1.0/asm_1/bin/tnslsnr LISTENER -inherit
Check tnsping for the new node. Ping the database and each instance
$ tnsping MT11107
$ tnsping MT111071
$ tnsping MT111072
You should get something like:
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racbaby1-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = racbaby2-vip)(PORT = 1521)) (LOAD_BALANCE = yes)
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = MT11107)))
OK (0 msec)
Test connect to the database and each instance
$ sqlplus sys@MT11107 as sysdba
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
MT111071
SQL> exit
$ sqlplus sys@MT111071 as sysdba
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
MT111071
SQL> exit
$ sqlplus sys@MT111072 as sysdba
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
MT111072
SQL> exit
shutdown the database
srvctl stop database -d MT11107 -o immediate
Verify shutdown:
oracle@racbaby1 [/u01/app/oracle/product/11.1.0/db_1/bin] MT11107
$ ps -ef|grep pmon|grep -v grep
oracle 9318 1 0 11:40 ? 00:00:00 asm_pmon_+ASM1
oracle@racbaby1 [/u01/app/oracle/product/11.1.0/db_1/bin] MT11107
$ ssh racbaby2 ps -ef|grep pmon|grep -v grep
oracle 9249 1 0 11:41 ? 00:00:00 asm_pmon_+ASM2
36)CREATE 10G database MT10204
set environment variables
oradb MT10204
cd $ORACLE_HOME/bin
./dbca
Select Real Application Clusters Database
Select create a database
Select all nodes
Select Custom Database
Global database name and SID to MT10204
Uncheck Configure Enterprise Manager
select use the Same Administrative Passwords for all accounts
set password
Select ASM for the storage
Select MT10204_DATA and MT10204_FLASH ASM Disk groups
Select Use Oracle-Managed Files
+MT10204_DATA should be filled in for the Database Area
Select Specify Flash Recovery Area
click browse and select MT10204_FLASH
Leave Size at 2048 (will make it bigger later as the max size is 49826)
Select Enable Archiving
Deselect all DB Components except "Oracle Data Mining" - ignore custom scripts tab
Accept the single MT10204 Database Service
Next Panel:
Memory Tab
select custom
Set SGA Memory Size 600mb
Sizing Tab
Leave block size at 8192 and processes at 150
Character Sets Tab
Leave default selected
Connection Mode tab
Leave Dedicated Server Mode selected
Accept Storage Defaults
Select "Create Database" and "Generate Database Creation scripts"
Click Finish
Click Ok
A second or two later the database creation scripts dialog will start,
a popup will say "scripts created" - click ok
Next the DBCA will create the database
Wait until the "Creating and starting Oracle instance" step completes, then get another beer
When complete the password management dialog will appear. Click Exit. Wait.
Another dialog will appear stating cluster db MT10204 and it's instances MT102041, MT102042 are starting.
Then it will close all windows without ceremony.
Validate
As oracle check for pmon's running on each node:
$ ps -ef|grep pmon|grep -v grep
Check tnsping for the new node. Ping the database and each instance
$ tnsping MT10204
$ tnsping MT102041
$ tnsping MT102042
You should get something like:
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racbaby1-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = racbaby2-vip)(PORT = 1521)) (LOAD_BALANCE = yes)
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = MT10204)))
OK (0 msec)
Test connect to the database and each instance
$ sqlplus sys@MT10204 as sysdba
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
MT102042
SQL> exit
$ sqlplus sys@MT102041 as sysdba
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
MT102041
SQL> exit
$ sqlplus sys@MT102042 as sysdba
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
MT102042
SQL> exit
shutdown the database
srvctl stop database -d MT10204 -o immediate
Verify shutdown:
$ ps -ef|grep pmon|grep -v grep
$ ssh racbaby2 ps -ef|grep pmon|grep -v grep
The purpose of the RAC Lab is to:
a) build a similar system to the work RAC systems and to go through the build installation to identify configuration and installation issues before attempting the same work on the systems at work.
b) become familiar with RAC configuration and installation
c) create a sandbox platform to experiment with different configurations and functional testing
LAB Equipment:
Servers:
2 Dual AMD Opteron 64bit w/4Gb RAM and 2 gigabit ethernet ports,
1 120Gb internal HD
Operating system:
CentOS 5.2 (Red Hat 5.2 binary equiv)
Oracle Software:
Cluster file system OCFS2 1.4.1
ASMLib 2.0
11g Oracle Clusterware
11g RAC ASM
11g Database using 11g RAC ASM
10g Database using 11g RAC ASM
Shared Storage:
Openfiler 2.3
Installed on AMD PC with 4Gb RAM, single 640Gb HD and single gigabit
ethernet port
Network Equipment:
1 8 port gigabit switch (Private Interconnect and Storage Connection)
1 4 port 100mb/s hub (Public)
Staging server:
Apple Macbook
SIGNIFICANT DIFFERENCES BETWEEN LAB SETUP AND WORK SERVERS
=======================================================================================
ITEM LAB WORK
-----------------------------------------------------------------------------------------------------------------------------
Hardware 2 Rackable Systems 2 HP DL585 G5's
Dual AMD Opteron 4 Quad Core AMD Opterons
single core CPU's w/128Gb RAM
w/4Gb RAM & Dual 6 Gigabit Ethernet w/4
Gigabit Ethernet bonded in pairs for private
and public networks 3 HBA's for SAN connections
Disk iSCSI disks from EMC SAN Disks Multipathed
dedicated PC with & connected through HBA's
openfiler 2.3
connected to the
RAC nodes via the
private network
OS CentOS 5.2 Red Hat 5.2
Security No special add ons Keon ESM
Monitoring No special add ons Tivoli, netsnmp/snmp-DC,
BMC Perform&Predict
Backup No special add ons Veritas Netbackup
DB Monitoring Local DB Console Oracle GridControl (Centralized)
INSTALLATION ACTIONS
=======================================================================================
1)INSTALL OS ON EACH NODE
choose CentOS 5.2 x86_64
select base install
when it reboots, for the first time login:
disable firewall and SELinux
skip creating a user
log in as root
Do NOT run automatic updates when complete!!!
2)SETUP NETWORK
Open network admin utility and set up Network
Set hostname
disable DHCP
port eth0 = 10.0.1.6X
port eth1 = 10.0.2.6X
restart network services
Open terminal and edit /etc/hosts
Remove hostname from 127.0.0.1
add the following:
10.0.1.69 racbaby1
10.0.1.68 racbaby2
10.0.2.69 racbaby1-priv
10.0.2.68 racbaby2-priv
10.0.1.169 racbaby1-vip
10.0.1.168 racbaby2-vip
10.0.2.66 openfiler1
3)MODIFY KERNEL AND SETTINGS
use GUI "services" utility to disable updater, card reader, bluetooth, etc
use GUI "Add Software" tool to install these items listed under "development"
development tools
development libraries
java development
legacy systems development
In a terminal window install the following additional packages:
yum install libaio-devel
yum install sysstat
yum install elfutils-libelf-devel
yum install iscsi-initiator-utils.x86_64
yum install lsscsi
yum install xorg-x11-deprecated-libs
yum install libXp.so.6
Set parameters using these commands:
sysctl -w net.core.rmem_default=4194304
sysctl -w net.core.wmem_default=262144
sysctl -w net.core.rmem_max=4194304
sysctl -w net.core.wmem_max=262144
echo "# Default setting in bytes of the socket receive buffer" >>/etc/sysctl.conf
echo "net.core.rmem_default= 4194304" >>/etc/sysctl.conf
echo "# Default setting in bytes of the socket send buffer" >>/etc/sysctl.conf
echo "net.core.wmem_default=262144" >>/etc/sysctl.conf
echo "# Maximum socket receive buffer size which may be set by using" >>/etc/sysctl.conf
echo "# the SO_RCVBUF socket option" >>/etc/sysctl.conf
echo "net.core.rmem_max= 4194304" >>/etc/sysctl.conf
echo "# Maximum socket send buffer size which may be set by using" >>/etc/sysctl.conf
echo "# the SO_SNDBUF socket option" >>/etc/sysctl.conf
echo "net.core.wmem_max=262144" >>/etc/sysctl.conf
sysctl -w kernel.shmmax=2147483648
echo "kernel.shmmax=2147483648" >> /etc/sysctl.conf
sysctl -w kernel.sem="250 32000 100 128"
echo "kernel.sem=250 32000 100 128" >> /etc/sysctl.conf
sysctl -w fs.file-max=65536
echo "fs.file-max=65536" >> /etc/sysctl.conf
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
echo "net.ipv4.ip_local_port_range = 1024 65000" >> /etc/sysctl.conf
cat >> /etc/security/limits.conf <<EOF
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
EOF
cat >> /etc/pam.d/login <<EOF
session required /lib/security/pam_limits.so
EOF
cat >> /etc/profile <<EOF
if [ \$USER = "oracle" ]; then
if [ \$SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
EOF
4)CREATE ORACLE USER
groupadd -g 115 oinstall
groupadd -g 116 dba
useradd -m -u 175 -g oinstall -G dba -d /home/oracle -s /bin/bash -c "Oracle Software Owner" oracle
passwd oracle
5)CREATE SPACE FOR HOMES
mkdir -p /u01/app/oracle
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/app/oracle
mkdir -p /u01/app/11.1.0
chown -R oracle:oinstall /u01/app/11.1.0
chmod -R 775 /u01/app/11.1.0
6)CREATE ISCSI DISKS
Use openfiler to set up disks and map luns to the two nodes. I'm not going to
listout how I setup Openfiler as it doesn't pertain to the purpose of the LAB
build.
For simplicity I named the luns according to function, so that when they appear on the nodes you'll see something like the following in /var/log/messages:
[root@racbaby2 log]# cat messages.1|grep openf|sort
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm10,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm1,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm2,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm3,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm4,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm5,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm6,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm7,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm8,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm9,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:ocfs,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:ocr1,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:ocr2,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:spfile,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:vote1,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:vote2,10.0.2.66,3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:vote3,10.0.2.66,3260]
7)COPY OVER SOFTWARE FILES FROM STAGING SERVER
Hangcheck Timer
hangcheck-0.8.0.tar.gz
Oracle CLUSTER and DB Software packages:
linux.x64_11gR1_clusterware.zip
linux.x64_11gR1_database_1013.zip
10201_database_linux_x86_64.cpio.gz
Oracle OCFS RPM's
ocfs2-2.6.18-92.el5-1.4.1-1.el5.x86_64.rpm
ocfs2-tools-1.4.1-1.el5.x86_64.rpm
ocfs2-tools-debuginfo-1.4.1-1.el5.x86_64.rpm
ocfs2-tools-devel-1.4.1-1.el5.x86_64.rpm
ocfs2console-1.4.1-1.el5.x86_64.rpm
ASMLib RPM's
oracleasm-2.6.18-92.1.22.el5-2.0.5-1.el5.x86_64.rpm
oracleasm-2.6.18-92.el5-2.0.5-1.el5.x86_64.rpm
oracleasm-support-2.1.2-1.el5.x86_64.rpm
oracleasmlib-2.0.3-1.el5.x86_64.rpm
homemade env variable switcher script
oradb
Oracle patchsets:
p6810189_10204_Linux-x86-64.zip
p6890831_111070_Linux-x86-64.zip
8)SET UP ISCSI ON RAC NODES
as root on each node:
service iscsi restart
chkconfig --level 345 iscsi on
iscsiadm -m discovery -t sendtargets -p openfiler1
service iscsi restart
verify disks are there
fdisk -l
run fdisk on each device to create a primary partition
9)SET UP CONSISTENT UDEV DEVICE NAMES AND PERMISSIONS ON EACH NODE
As root on each node:
look at /proc/scsi/scsi for this:
Host: scsi20 Channel: 00 Id: 00 Lun: 00
Vendor: OPNFILER Model: VIRTUAL-DISK Rev: 0
Type: Direct-Access ANSI SCSI revision: 04
then edit /etc/iscsi/iscsid.conf and add at the bottom:
vendor=OPNFILER, model="VIRTUAL-DISK". options=-g
now add the following line at the end of /etc/udev/rules.d/50-udev.rules
KERNEL=="sd?", BUS=="scsi", ENV{ID_MODEL}=="VIRTUAL-DISK", ENV{ID_PATH}=="*iscsi*", OWNER="oracle", \
GROUP="dba", MODE="640", RUN+="/etc/udev/scripts/iscsi $env{ID_PATH} %k"
KERNEL=="sd?1", BUS=="scsi", ENV{ID_MODEL}=="VIRTUAL-DISK", ENV{ID_PATH}=="*iscsi*", OWNER="oracle", \
GROUP="dba", MODE="640"
create a file called /etc/udev/scripts/iscsi and add the following
#!/bin/bash
LINK_NAME=`/bin/echo $1 | awk -F":" '{ print $3 }'`
/bin/ln -s /dev/${2}1 /dev/${LINK_NAME}
Then make it executable using:
chmod 700 /etc/udev/scripts/iscsi
reboot both nodes
verify you see good named links to the partitioned disks (may have to reboot again for disks to show up correctly)
$ ls -l /dev |grep ">" |grep sd |egrep -v "core|MAKE|floppy|proc"
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm1 -> /dev/sdg1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm10 -> /dev/sdd1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm2 -> /dev/sdk1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm3 -> /dev/sdc1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm4 -> /dev/sdn1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm5 -> /dev/sdf1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm6 -> /dev/sdb1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm7 -> /dev/sdo1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm8 -> /dev/sdj1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 asm9 -> /dev/sdm1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 ocfs -> /dev/sde1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 ocr1 -> /dev/sdp1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 ocr2 -> /dev/sdh1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 spfile -> /dev/sdi1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 vote1 -> /dev/sdl1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 vote2 -> /dev/sdq1
lrwxrwxrwx 1 root root 9 Mar 7 10:32 vote3 -> /dev/sdr1
verify disks are reachable by oracle and not owned by root:
$ ls -l /dev/sd*1|grep -v ">"|grep -v sda1
brw-r----- 1 oracle dba 8, 17 Mar 7 10:32 /dev/sdb1
brw-r----- 1 oracle dba 8, 33 Mar 7 10:32 /dev/sdc1
brw-r----- 1 oracle dba 8, 49 Mar 7 10:32 /dev/sdd1
brw-r----- 1 oracle dba 8, 65 Mar 7 10:32 /dev/sde1
brw-r----- 1 oracle dba 8, 81 Mar 7 10:32 /dev/sdf1
brw-r----- 1 oracle dba 8, 97 Mar 7 10:32 /dev/sdg1
brw-r----- 1 oracle dba 8, 113 Mar 7 10:48 /dev/sdh1
brw-r----- 1 oracle dba 8, 129 Mar 7 10:32 /dev/sdi1
brw-r----- 1 oracle dba 8, 145 Mar 7 10:32 /dev/sdj1
brw-r----- 1 oracle dba 8, 161 Mar 7 10:32 /dev/sdk1
brw-r----- 1 oracle dba 8, 177 Mar 7 13:23 /dev/sdl1
brw-r----- 1 oracle dba 8, 193 Mar 7 10:32 /dev/sdm1
brw-r----- 1 oracle dba 8, 209 Mar 7 10:32 /dev/sdn1
brw-r----- 1 oracle dba 8, 225 Mar 7 10:32 /dev/sdo1
brw-r----- 1 oracle dba 8, 241 Mar 7 10:48 /dev/sdp1
brw-r----- 1 oracle dba 65, 1 Mar 7 13:23 /dev/sdq1
brw-r----- 1 oracle dba 65, 17 Mar 7 13:23 /dev/sdr1
10)CREATE OCFS2 DIRECTORIES
mkdir -p /u02/app/oracle/exports
chown -R oracle:dba /u02/app/oracle/exports
chmod -R 775 /u02/app/oracle/exports
mkdir -p /u02/app/oracle/ASM/spfile
chown -R oracle:dba /u02/app/oracle/ASM/spfile
chmod -R 775 /u02/app/oracle/ASM/spfile
11)EDIT ORACLE BASH PROFILE
edit .bash_profile for oracle to match as follows:
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
alias oradb=". /home/oracle/oradb 2>/dev/null"
12)EDIT ORATAB FILE
as root
touch /etc/oratab
chown oracle:dba /etc/oratab
edit and add some starting settings for installation purposes:
# Oracle oratab file
#
ASMINS:/u01/app/oracle/product/11.1.0/asm_1:N
CRS:/u01/app/11.1.0/crs
MT11107:/u01/app/oracle/product/11.1.0/db_1:N
MT10204:/u01/app/oracle/product/10.2.0/db_1:N
test oradb as oracle
oradb ASMINS
echo $ORACLE_HOME
echo $PATH
echo $LD_LIBRARY_PATH
echo $ORACLE_SID
oradb
should get a menu to select env.
echo $ORACLE_HOME
echo $PATH
echo $LD_LIBRARY_PATH
echo $ORACLE_SID
The paths should not be cumulative after switching env's.
13)VERIFY KERNEL PARAMETERS
sysctl -p # should return:
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.core.rmem_default = 4194304
net.core.wmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_max = 262144
kernel.shmmax = 2147483648
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
14)CONFIG HANGCHECK TIMER
find /lib/modules -name "hangcheck-timer.ko"
echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> /etc/modprobe.conf
echo "/sbin/modprobe hangcheck-timer" >> /etc/rc.local
modprobe hangcheck-timer
grep Hangcheck /var/log/messages | tail -2
15)INSTALL OCFS2
install from root media dir as root using
rpm -Uvh ocfs2*
validate packages were installed:
[root@racbaby2 media]# rpm -qa |grep ocfs |sort
ocfs2-2.6.18-92.el5-1.4.1-1.el5
ocfs2console-1.4.1-1.el5
ocfs2-tools-1.4.1-1.el5
ocfs2-tools-debuginfo-1.4.1-1.el5
ocfs2-tools-devel-1.4.1-1.el5
16)INSTALL ASMLIB
install from root media dir using
rpm -Uvh oracleasm*
validate packages were installed:
[root@racbaby2 media]# rpm -qa |grep oracleasm|sort
oracleasm-2.6.18-92.1.22.el5-2.0.5-1.el5
oracleasm-2.6.18-92.el5-2.0.5-1.el5
oracleasmlib-2.0.3-1.el5
oracleasm-support-2.1.2-1.el5
17)SET DATE ON BOTH NODES (IF NEEDED)
date -s "6/25/2007 23:00:00"
date -s "6/25/2007 23:00:20"
18)SETUP SSH INTRA NODE CONNECTIVITY FOR ORACLE
as oracle
ssh-keygen -t rsa -P ""
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
ssh racbaby2
ssh-keygen -t rsa -P ""
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
exit
scp .ssh/id_rsa.pub racbaby2:/home/oracle/.ssh/racbaby1.pub
scp racbaby2:/home/oracle/.ssh/id_rsa.pub /home/oracle/.ssh/racbaby2.pub
cat .ssh/racbaby2.pub >> .ssh/authorized_keys
ssh racbaby2
cat .ssh/racbaby1.pub >> .ssh/authorized_keys
exit
TEST
Each oracle account should be able to ssh to itself and to the other nodes without password prompts or banner.
19)CONFIGURE OCFS2
configure as root using:
ocfs2console &
Using the ocfs2console GUI tool, perform the following steps:
This needs to be performed on one of the Oracle RAC nodes in the cluster.
1. Select [Cluster] -> [Configure Nodes...]. This will start the OCFS2 Cluster Stack and bring up the "Node Configuration" dialog.
2. On the "Node Configuration" dialog, click the [Add] button.
* This will bring up the "Add Node" dialog.
* In the "Add Node" dialog, enter both nodes using
racbaby1 / 10.0.2.69
racbaby2 / 10.0.2.68
* Click [Apply] on the "Node Configuration" dialog - All nodes should now be "Active"
* Click [Close] on the "Node Configuration" dialog.
3. Select cluster -> 'propagate'
as soon as you see the Finished! close the window that opened
4. After verifying all values are correct, exit the application using [File] -> [Quit].
On the other node, if o2cb is not running, run the console to start the cluster. Then Check status & config
/etc/init.d/o2cb status
/etc/init.d/o2cb offline ocfs2
/etc/init.d/o2cb unload
/etc/init.d/o2cb configure
/etc/init.d/o2cb load
/etc/init.d/o2cb online ocfs2
20)FORMAT OCFS2 FILE SYSTEM
ON ONE NODE AS ROOT
mkfs.ocfs2 -b 4K -C 32K -N 4 -L oracleexports /dev/ocfs
mkfs.ocfs2 -b 4K -C 32K -N 4 -L asmspfile /dev/spfile
ON BOTH NODES AS ROOT
mount -t ocfs2 -o datavolume,nointr -L "oracleexports" /u02/app/oracle/exports
mount -t ocfs2 -o datavolume,nointr -L "asmspfile" /u02/app/oracle/ASM/spfile
make reboot proof:
echo "LABEL=oracleexports /u02/app/oracle/exports ocfs2 _netdev,datavolume,nointr 0 0" >> /etc/fstab
echo "LABEL=asmspfile /u02/app/oracle/ASM/spfile ocfs2 _netdev,datavolume,nointr 0 0" >> /etc/fstab
verify:
chkconfig --list o2cb
o2cb 0:off 1:off 2:on 3:on 4:on 5:on 6:off
21)INSTALL CVUQDISK
as root
cd /home/oracle/CLUSTER/clusterware/rpm
rpm -iv cvuqdisk-1.0.1-1.rpm
22)CONFIGURE ASMLIB
as root
/etc/init.d/oracleasm configure
/etc/init.d/oracleasm createdisk asmdisk1 /dev/asm1
/etc/init.d/oracleasm createdisk asmdisk2 /dev/asm2
/etc/init.d/oracleasm createdisk asmdisk3 /dev/asm3
/etc/init.d/oracleasm createdisk asmdisk4 /dev/asm4
/etc/init.d/oracleasm createdisk asmdisk5 /dev/asm5
/etc/init.d/oracleasm createdisk asmdisk6 /dev/asm6
/etc/init.d/oracleasm createdisk asmdisk7 /dev/asm7
/etc/init.d/oracleasm createdisk asmdisk8 /dev/asm8
/etc/init.d/oracleasm createdisk asmdisk9 /dev/asm9
/etc/init.d/oracleasm createdisk asmdisk10 /dev/asm10
/etc/init.d/oracleasm scandisks
/etc/init.d/oracleasm listdisks
[root@racbaby2 ~]# /etc/init.d/oracleasm listdisks
ASMDISK1
ASMDISK10
ASMDISK2
ASMDISK3
ASMDISK4
ASMDISK5
ASMDISK6
ASMDISK7
ASMDISK8
ASMDISK9
23)PREPARE ORAINVENTORY DIR ON BOTH NODES
as root
cd /u01/app
mkdir oraInventory
chown oracle:oinstall oraInventory/
24)INSTALL ORACLE CLUSTERWARE
as root:
copy cluster linux.x64_11gR1_clusterware.zip to a CLUSTER dir as root
chown oracle:dba CLUSTER
mv CLUSTER /home/oracle
as oracle:
set environment variables
oradb CRS
cd CLUSTER
unzip linux.x64_11gR1_clusterware.zip
cd clusterware
./runInstaller
name: OraCrs11g_home
path: /u01/app/11.1.0/crs
<checks for parameters - fix any that fail and retry (none did)>
Add node, give hostname
Set private and public network interface to use
OCR location
Normal redundancy
enter /dev/ocr1, /dev/ocr2
Voting Disk Location
Normal redundancy
enter /dev/vote1, /dev/vote2, /dev/vote3
Install
When prompted run on both nodes, one at a time:
/u01/app/oraInventory/orainstRoot.sh
/u01/app/11.1.0/crs/root.sh
If CentOS is used, when complete and it runs the cluster verification utility it will fail if you don't do the
following before going back to the installer and hitting next.
vi /u01/app/11.1.0/crs/cv/admin/cvu_config
and change line
#CV_ASSUME_DISTID=RHEL4
to
CV_ASSUME_DISTID=RHEL5
Finish
25)UPGRADE CRS_HOME to 11.1.0.7
On one node only
As root
mv p6890831_111070_Linux-x86-64.zip /home/oracle/CLUSTER/
As Oracle
set environment variables
oradb CRS
cd CLUSTER
unzip p6890831_111070_Linux-x86-64.zip
cd Disk1
./runInstaller
name: OraCrs11g_home
path: /u01/app/11.1.0/crs
Cluster installation default
Checks for prerequisites - fix any that fail (none did)
Install
When prompted:
1.Log in as the root user.
2.As the root user, perform the following tasks:
a.Shutdown the CRS daemons by issuing the following command:
/u01/app/11.1.0/crs/bin/crsctl stop crs
b.Run the shell script located at:
/u01/app/11.1.0/crs/install/root111.sh
3.After completing this procedure, proceed to the next node and repeat.
26)INSTALL ORACLE ASM HOME
as root,
cd media
mkdir ASM+DB
mv linux.x64_11gR1_database_1013.zip ASM+DB
chown oracle:dba ASM+DB
mv ASM+DB /home/oracle
as oracle
set environment variables
oradb ASMINS
cd ASM+DB
unzip linux.x64_11gR1_database_1013.zip
cd database
./runInstaller
choose Enterprise Edition
Set Name to OraASM11g_home1
Set Path to /u01/app/oracle/product/11.1.0/asm_1
Prerequisite test - ignored swap warning
Select Software Only
Set dba for OSDBA, OSOPER and OSASM
Install
When prompted run
/u01/app/oracle/product/11.1.0/asm_1/root.sh
27)UPGRADE ASM HOME to 11.0.1.7
Use the same p6890831_111070_Linux-x86-64.zip as you did with the CRS HOME
As Oracle
set environment variables
oradb ASMINS
cd CLUSTER
cd Disk1
./runInstaller
name: OraASM11g_home1
path: /u01/app/oracle/product/11.1.0/asm_1
ignore the email for updates request
cluster install selected
Checks for prerequisites - fix any that fail (none did)
Install
When prompted run as root:
/u01/app/oracle/product/11.1.0/asm_1/root.sh
say 'yes' to replace files
28)INSTALL ORACLE 11G DB HOME
Use the same linux.x64_11gR1_database_1013.zip you used for ASM Home install
as oracle
set environment variables
oradb MT11107_1
cd ASM+DB
cd database
./runInstaller
choose Enterprise Edition
Set Name to OraDB11g_home1
Set Path to /u01/app/oracle/product/11.1.0/db_1
Select all nodes in cluster
Prerequisite test - ignored swap warning
Select Software Only
Set dba for OSDBA, OSOPER and OSASM
Install
When prompted run as root:
/u01/app/oracle/product/11.1.0/db_1/root.sh
say 'no' to replace files
29)UPGRADE 11G DB HOME to 11.0.1.7
Use the same p6890831_111070_Linux-x86-64.zip as you did with the CRS and ASM HOME
As Oracle
set environment variables
oradb MT11107
cd CLUSTER
cd Disk1
./runInstaller
name: OraDB11g_home
path: /u01/app/oracle/product/11.1.0/db_1
Checks for prerequisites - fix any that fail (none did)
Install
When prompted run as root:
/u01/app/oracle/product/11.1.0/db_1/root.sh
say 'no' to replace files
30)INSTALL ORACLE 10G HOME
as root
cd media
mkdir 10g
cp 10201_database_linux_x86_64.cpio.gz 10g
chown oracle:dba 10g
mv 10g /home/oracle
as oracle
set environment variables
oradb MT10204
set a different location for TEMP to avoid java.lang.UnsatisfiedLinkError
(Metalink Doc ID: 370120.1)
mkdir /home/oracle/temp
export TEMP=/home/oracle/temp
cd 10g
Unzip the file: gunzip 10201_database_linux_x86_64.cpio.gz
cpio -idmv < 10201_database_linux_x86_64.cpio
cd database
./runInstaller -ignoreSysPrereqs
name: OraDB10g_home
path: /u01/app/oracle/product/10.2.0/db_1
Select all rac nodes
!! Verifications and validations will fail, click on checkboxes
!! and retry until all are greyed out
Select install software only
Install
When prompted run as root on both nodes:
/u01/app/oracle/product/10.2.0/db_1/root.sh
(respond 'n' to replace files)
31)UPGRADE ORACLE 10G HOME TO 10.2.0.4
as root:
cd media
mkdir 10204_patch
mv p6810189_10204_Linux-x86-64.zip 10204_patch/
chown oracle:dba 10204_patch/
mv 10204_patch/ /home/oracle/10g/
as oracle:
set environment variables
oradb MT10204
cd 10g/10204_patch
unzip p6810189_10204_Linux-x86-64.zip
cd Disk1
./runInstaller -ignoreSysPrereqs
name: OraDB10g_home
path: /u01/app/oracle/product/10.2.0/db_1
Select all rac nodes
Verifications did not fail
skip the Oracle Configuration Manager setup
Install
When prompted run as root on both nodes:
/u01/app/oracle/product/10.2.0/db_1/root.sh
(respond 'n' to replace files)
32)CONFIGURE ASM INSTANCE AND DISK GROUPS
As Oracle
set environment variables
oradb ASMINS
Configure the cluster ASM listener
cd $ORACLE_HOME/bin
./netca
Select Cluster configuration
Select all nodes
Select listener configuration
Select Add
Use default LISTENER name
Use TCP as the Selected Protocols
Use standard port number
Select 'No' to configure another listener
Will get a complete! message
Select finish
Configure ASM Instance
./dbca
Select 'Oracle Real Application Clusters Database'
Select 'Configure Automatic Storage management'
Select all the nodes in the cluster
Specify password for ASM instance
Specify the location for spfile in
/u02/app/oracle/ASM/spfile/spfile+ASM.ora
Click Next
A popup will say DBCA will create and start the ASM instance
click OK
Skip the ASM diskgroup creation - click Finish
A popup will ask if you want to perform another operation - Click No
Remove ASMINS oratab entry (no longer needed)
vi /etc/oratab
remove the ASMINS line
Move +ASM oratab entries to the top of the list
vi /etc/oratab
move the +ASMx line to the top of the list of instances
reset environment variables
oradb +ASM1
Login to one ASM instance as SYSASM and create the DiskGroups
sqlplus "/ as SYSASM"
Create DB_DATA and DB_FLASH diskgroups manually:
SQL> create diskgroup MT11107_DATA normal redundancy
disk 'ORCL:ASMDISK1','ORCL:ASMDISK2'
attribute 'compatible.asm'='11.1','compatible.rdbms'='10.2','au_size'='4M';
SQL> create diskgroup MT11107_FLASH normal redundancy
disk 'ORCL:ASMDISK3','ORCL:ASMDISK4','ORCL:ASMDISK5'
attribute 'compatible.asm'='11.1','compatible.rdbms'='10.2','au_size'='4M';
SQL> create diskgroup MT10204_DATA normal redundancy
disk 'ORCL:ASMDISK6','ORCL:ASMDISK7'
attribute 'compatible.asm'='11.1','compatible.rdbms'='10.2','au_size'='4M';
SQL> create diskgroup MT10204_FLASH normal redundancy
disk 'ORCL:ASMDISK8','ORCL:ASMDISK9','ORCL:ASMDISK10'
attribute 'compatible.asm'='11.1','compatible.rdbms'='10.2','au_size'='4M';
Login to ASM instances ON BOTH NODES as SYSASM and create the DiskGroups
reset environment variables
oradb +ASMx (where x is the instance number on the node)
sqlplus "/ as SYSASM" on BOTH NODES
Mount the disk groups
SQL> alter diskgroup MT11107_DATA mount;
SQL> alter diskgroup MT10204_FLASH mount;
SQL> alter diskgroup MT11107_FLASH mount;
SQL> alter diskgroup MT10204_DATA mount;
Add disk groups to spfile on BOTH NODES
SQL> alter system set asm_diskgroups='MT11107_DATA','MT11107_FLASH','MT10204_DATA','MT10204_FLASH' scope=both;
Verify the diskgroups, attributes and automount on instance startup in BOTH NODES
SQL> shutdown immediate;
startup
SQL> select name, total_mb, free_mb from v$asm_disk;
NAME TOTAL_MB FREE_MB
------------------------------ ---------- ----------
ASMDISK1 50012 49936
ASMDISK10 50012 49936
ASMDISK2 50012 49936
ASMDISK3 50012 49936
ASMDISK4 50012 49936
ASMDISK5 50012 49936
ASMDISK6 50012 49936
ASMDISK7 50012 49936
ASMDISK8 50012 49936
ASMDISK9 50012 49936
10 rows selected.
SQL> select name, state, total_mb, free_mb from v$asm_diskgroup;
NAME STATE TOTAL_MB FREE_MB
------------------------------ ----------- ---------- ----------
MT10204_DATA MOUNTED 100024 99872
MT10204_FLASH MOUNTED 150036 149808
MT11107_DATA MOUNTED 100024 99872
MT11107_FLASH MOUNTED 150036 149808
33)CREATE CENTRAL LOCATION FOR ORATAB AND LISTENER FILES
as root on both nodes:
mkdir /var/opt/oracle
chown oracle:dba /var/opt/oracle
mv /etc/oratab /var/opt/oracle
cd /etc
ln -s /var/opt/oracle/oratab oratab
as oracle on both nodes:
set environment variables
oradb +ASMx (where x is the instance number on the node)
shutdown listener
lsnrctl stop
move the listener.ora from ASM home
cd /var/opt/oracle
mv $ORACLE_HOME/network/admin/listener.ora .
move the tnsnames.ora from ASM home (if it exists - otherwise create an empty file in
/var/opt/oracle)
cd /var/opt/oracle
mv $ORACLE_HOME/network/admin/tnsnames.ora .
(or touch /var/opt/oracle/tnsnames.ora if it doesn't exist)
setup links from ASM home, 10G home, CRS home and 11G home
cd $ORACLE_HOME/network/admin
ln -s /var/opt/oracle/listener.ora listener.ora
ln -s /var/opt/oracle/tnsnames.ora tnsnames.ora
oradb MT10204
cd $ORACLE_HOME/network/admin
ln -s /var/opt/oracle/listener.ora listener.ora
ln -s /var/opt/oracle/tnsnames.ora tnsnames.ora
oradb CRS
cd $ORACLE_HOME/network/admin
ln -s /var/opt/oracle/listener.ora listener.ora
ln -s /var/opt/oracle/tnsnames.ora tnsnames.ora
oradb MT11107
cd $ORACLE_HOME/network/admin
ln -s /var/opt/oracle/listener.ora listener.ora
ln -s /var/opt/oracle/tnsnames.ora tnsnames.ora
Verify links are setup
ls -l /u01/app/oracle/product/*/*/network/admin/*.ora
ls -l /u01/app/11.1.0/crs/network/admin/*.ora
edit custom oradb script and replace 2 entries of TNS_ADMIN:
vi /home/oracle/oradb
replace:
export TNS_ADMIN=$ORACLE_HOME/network/admin
with:
export TNS_ADMIN=/var/opt/oracle
copy new oradb to other node
scp oradb racbaby1:/home/oracle
reset environment variables with new oradb
oradb +ASMx (where x is the instance number on the node)
start listener
lsnrctl start
Verify the listener parameter file is listed as the one in /var/opt/oracle
34)REBOOT BOTH NODES AND VERIFY ALL SERVICES REQUIRED COME UP AUTOMATICALLY
as root:
init 6
After servers come back up log in as root (open a window for each node)
ssh -X root@10.0.1.68
ssh -X root@10.0.1.69
On both nodes as root:
check udev and iscsi disk privs (should be oracle:dba)
ls -l /dev/sd* (ignore the ones for local HD)
ls -l /dev/asm*
ls -l /dev/vote*
ls -l /dev/ocfs
ls -l /dev/ocr*
check CRS status on both nodes:
[root@racbaby2 ~]# /etc/init.d/o2cb status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold = 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Checking O2CB heartbeat: Active
[root@racbaby1 ~]# /etc/init.d/o2cb status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold = 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Checking O2CB heartbeat: Active
check ocfs2 mounts by creating a dummy file on one node and verifying
it shows up on the other:
[root@racbaby2 spfile]# cd /u02/app/oracle/ASM/spfile
[root@racbaby2 spfile]# touch bob
[root@racbaby2 spfile]# ls -l
total 64
-rw-r--r-- 1 root root 0 Mar 8 11:44 bob
drwxr-xr-x 2 root root 4096 Mar 7 17:13 lost+found
-rw-r----- 1 oracle oinstall 2560 Mar 7 17:47 spfile+ASM.ora
[root@racbaby2 spfile]# ssh racbaby1 ls -l /u02/app/oracle/ASM/spfile
root@racbaby1's password:
total 64
-rw-r--r-- 1 root root 0 Mar 8 11:44 bob
drwxr-xr-x 2 root root 4096 Mar 7 17:13 lost+found
-rw-r----- 1 oracle oinstall 2560 Mar 7 17:47 spfile+ASM.ora
[root@racbaby2 spfile]# rm bob
rm: remove regular empty file `bob'? y
[root@racbaby2 exports]# cd /u02/app/oracle/exports
[root@racbaby2 exports]# touch bob
[root@racbaby2 exports]# ssh racbaby1 ls -l /u02/app/oracle/exports
root@racbaby1's password:
total 32
-rw-r--r-- 1 root root 0 Mar 8 2009 bob
drwxr-xr-x 2 root root 4096 Mar 7 17:12 lost+found
[root@racbaby2 exports]# rm bob
rm: remove regular empty file `bob'? y
Check hangcheck timer on both nodes:
[root@racbaby2 exports]# grep Hangcheck /var/log/messages | tail -2
Mar 8 11:40:19 racbaby2 kernel: Hangcheck: starting hangcheck timer 0.9.0 (tick is 30 seconds, margin is 180 seconds).
Mar 8 11:40:19 racbaby2 kernel: Hangcheck: Using monotonic_clock().
[root@racbaby1 ~]# grep Hangcheck /var/log/messages | tail -2
Mar 8 11:39:55 racbaby1 kernel: Hangcheck: starting hangcheck timer 0.9.0 (tick is 30 seconds, margin is 180 seconds).
Mar 8 11:39:55 racbaby1 kernel: Hangcheck: Using monotonic_clock().
On both nodes as oracle:
[root@racbaby2 exports]# su - oracle
check ASM listener is up from ASM home
[oracle@racbaby2 ~]$ ps -ef|grep tns
oracle 4437 1 0 12:32 ? 00:00:00 /u01/app/oracle/product/11.1.0/asm_1/bin/tnslsnr LISTENER -inherit
[oracle@racbaby1 ~]$ ps -ef|grep tns
oracle 5140 1 0 12:32 ? 00:00:00 /u01/app/oracle/product/11.1.0/asm_1/bin/tnslsnr LISTENER -inherit
check ASM instance and diskgroups on both nodes
[oracle@racbaby2 ~]$ oradb
Oracle Env Selection Menu
----------------------------------------------------------
## ENV ORACLE_HOME
----------------------------------------------------------
1. (+ASM2 ) /u01/app/oracle/product/11.1.0/asm_1
2. (CRS ) /u01/app/11.1.0/crs
3. (MT11107 ) /u01/app/oracle/product/11.1.0/db_1
4. (MT10204 ) /u01/app/oracle/product/10.2.0/db_1
Q. Quit w/o setting variables
Enter environment Choice (1-4) [q]: 1
oracle@racbaby2 [/home/oracle] +ASM2
$ sqlplus "/ as sysasm"
SQL*Plus: Release 11.1.0.7.0 - Production on Sun Mar 8 11:55:20 2009
Copyright (c) 1982, 2008, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
SQL> select name, state, total_mb, free_mb from v$asm_diskgroup;
NAME STATE TOTAL_MB FREE_MB
------------------------------ ----------- ---------- ----------
MT10204_DATA MOUNTED 100024 99872
MT10204_FLASH MOUNTED 150036 149808
MT11107_DATA MOUNTED 100024 99872
MT11107_FLASH MOUNTED 150036 149808
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
35)CREATE 11G database MT11107
Log in as oracle, run cluvfy before starting
ssh -X 10.0.1.69
set environment variables
oradb CRS
cd $ORACLE_HOME/bin
./cluvfy stage -pre dbinst -n racbaby1,racbaby2 -verbose
Make sure it says successful
Create the database
set environment variables
oradb MT11107
cd $ORACLE_HOME/bin
./dbca
Select Real Application Clusters Database
Select create a database
Select all nodes
Select Custom Database
set Global database name and SID to MT11107
Uncheck Configure Enterprise Manager
select use the Same Administrative Passwords for all accounts
set password
Select ASM for the storage
Select MT11107_DATA and MT11107_FLASH ASM Disk groups
Select Use Oracle-Managed Files
+MT11107_DATA should be filled in for the Database Area
Select Specify Flash Recovery Area
click browse and select MT11107_FLASH
Leave Size at 2048 (will make it bigger later as the max size is 49826)
Select Enable Archiving
Deselect all DB Components except "Oracle Text" - ignore custom scripts tab
Next Panel:
Memory Tab
Set SGA Memory Size 1024
Sizing Tab
Leave block size at 8192 and processes at 150
Character Sets Tab
Leave default selected
Connection Mode tab
Leave Dedicated Server Mode selected
Leave "Keep the enhanced 11g default security settings" selected
Leave "Enable automatic maintenance tasks" selected
Accept Storage Defaults
Select "Create Database" and "Generate Database Creation scripts"
Click Ok
A second or two later the database creation scripts dialog will start,
a popup will say "scripts created" - click ok
Next the DBCA will create the database
Wait until the "Creating and starting Oracle instance" step completes, then get a beer
When complete the password management dialog will appear. Click Exit. Wait.
Another dialog will appear stating cluster db MT11107 and it's instances MT111071, MT111072 are starting.
Then it will close all windows without ceremony.
Validate
As oracle check for pmon's and listener running on each node:
oracle@racbaby1 [/u01/app/oracle/product/11.1.0/db_1/bin] MT11107
$ ps -ef|grep pmon|grep -v grep
oracle 5085 1 0 14:05 ? 00:00:00 ora_pmon_MT111071
oracle 9318 1 0 11:40 ? 00:00:00 asm_pmon_+ASM1
oracle@racbaby1 [/u01/app/oracle/product/11.1.0/db_1/bin] MT11107
$ ps -ef|grep tns|grep -v grep
oracle 5140 1 0 12:32 ? 00:00:00 /u01/app/oracle/product/11.1.0/asm_1/bin/tnslsnr LISTENER -inherit
oracle@racbaby2 [/home/oracle] MT11107
$ ps -ef|grep pmon|grep -v grep
oracle 2820 1 0 14:06 ? 00:00:00 ora_pmon_MT111072
oracle 9249 1 0 11:41 ? 00:00:00 asm_pmon_+ASM2
oracle@racbaby2 [/home/oracle] MT11107
$ ps -ef|grep tns|grep -v grep
oracle 4437 1 0 12:32 ? 00:00:00 /u01/app/oracle/product/11.1.0/asm_1/bin/tnslsnr LISTENER -inherit
Check tnsping for the new node. Ping the database and each instance
$ tnsping MT11107
$ tnsping MT111071
$ tnsping MT111072
You should get something like:
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racbaby1-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = racbaby2-vip)(PORT = 1521)) (LOAD_BALANCE = yes)
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = MT11107)))
OK (0 msec)
Test connect to the database and each instance
$ sqlplus sys@MT11107 as sysdba
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
MT111071
SQL> exit
$ sqlplus sys@MT111071 as sysdba
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
MT111071
SQL> exit
$ sqlplus sys@MT111072 as sysdba
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
MT111072
SQL> exit
shutdown the database
srvctl stop database -d MT11107 -o immediate
Verify shutdown:
oracle@racbaby1 [/u01/app/oracle/product/11.1.0/db_1/bin] MT11107
$ ps -ef|grep pmon|grep -v grep
oracle 9318 1 0 11:40 ? 00:00:00 asm_pmon_+ASM1
oracle@racbaby1 [/u01/app/oracle/product/11.1.0/db_1/bin] MT11107
$ ssh racbaby2 ps -ef|grep pmon|grep -v grep
oracle 9249 1 0 11:41 ? 00:00:00 asm_pmon_+ASM2
36)CREATE 10G database MT10204
set environment variables
oradb MT10204
cd $ORACLE_HOME/bin
./dbca
Select Real Application Clusters Database
Select create a database
Select all nodes
Select Custom Database
Global database name and SID to MT10204
Uncheck Configure Enterprise Manager
select use the Same Administrative Passwords for all accounts
set password
Select ASM for the storage
Select MT10204_DATA and MT10204_FLASH ASM Disk groups
Select Use Oracle-Managed Files
+MT10204_DATA should be filled in for the Database Area
Select Specify Flash Recovery Area
click browse and select MT10204_FLASH
Leave Size at 2048 (will make it bigger later as the max size is 49826)
Select Enable Archiving
Deselect all DB Components except "Oracle Data Mining" - ignore custom scripts tab
Accept the single MT10204 Database Service
Next Panel:
Memory Tab
select custom
Set SGA Memory Size 600mb
Sizing Tab
Leave block size at 8192 and processes at 150
Character Sets Tab
Leave default selected
Connection Mode tab
Leave Dedicated Server Mode selected
Accept Storage Defaults
Select "Create Database" and "Generate Database Creation scripts"
Click Finish
Click Ok
A second or two later the database creation scripts dialog will start,
a popup will say "scripts created" - click ok
Next the DBCA will create the database
Wait until the "Creating and starting Oracle instance" step completes, then get another beer
When complete the password management dialog will appear. Click Exit. Wait.
Another dialog will appear stating cluster db MT10204 and it's instances MT102041, MT102042 are starting.
Then it will close all windows without ceremony.
Validate
As oracle check for pmon's running on each node:
$ ps -ef|grep pmon|grep -v grep
Check tnsping for the new node. Ping the database and each instance
$ tnsping MT10204
$ tnsping MT102041
$ tnsping MT102042
You should get something like:
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racbaby1-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = racbaby2-vip)(PORT = 1521)) (LOAD_BALANCE = yes)
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = MT10204)))
OK (0 msec)
Test connect to the database and each instance
$ sqlplus sys@MT10204 as sysdba
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
MT102042
SQL> exit
$ sqlplus sys@MT102041 as sysdba
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
MT102041
SQL> exit
$ sqlplus sys@MT102042 as sysdba
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
MT102042
SQL> exit
shutdown the database
srvctl stop database -d MT10204 -o immediate
Verify shutdown:
$ ps -ef|grep pmon|grep -v grep
$ ssh racbaby2 ps -ef|grep pmon|grep -v grep
