Oracle RAC Installation with Raw device Vmware server 와 oracle 에서제공하는 RHEL4 기반 Linux x86 (32-bit) 대상정보기술 DB 지원팀여현승
설정된 OS환경 node1, node2(hostname) 해상도 : 1024X768 호스트머신과시간동기화 Network setting Eth0 () Eth1 Disk (localdisk & shared storage) SCSI 0:0 : localdisk(20gb) SCSI 1:0 : disk1(20gb) SCSI 1:1 : disk2(20gb) SCSI 1:2 : disk3(20gb) SCSI 1:3 : disk4(20gb) 오라클사용자생성 Home directory : /home/oracle Group : dba Password : oracle Shell : bash 전체구성도
Oracle software설치에필요한패키지설치 binutils-2.15.92.0.2-10.el4 binutils-2.15.92.0.2-13.0.0.0.2.x86_64 compat-db-4.1.25-9 control-center-2.8.0-12 gcc-3.4.3-9.el4 gcc-c++-3.4.3-9.el4 glibc-2.3.4-2 glibc-common-2.3.4-2 gnome-libs-1.4.1.2.90-44.1 libstdc++-3.4.3-9.el4 libstdc++-devel-3.4.3-9.el4 make-3.80-5 libaio-0.3.105-2.i386.rpm openmotif21-2.1.30-11.rhel4.6.i386.rpm Shell Limits for the oracle User Linux 에설치된 software성능향상을위해반드시 oracle user의 shell limit설정 /etc/security/limits.conf oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536 /etc/pam.d/login session required /lib/security/pam_limits.so
/etc/profile if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi Fi Configuring Kernel Parameters /etc/sysctl.conf kernel.shmall = 2097152 kernel.shmmax = 2147483648 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 65536 net.ipv4.ip_local_port_range = 1024 65000 net.core.rmem_default = 1048576 net.core.rmem_max = 1048576 net.core.wmem_default = 262144 net.core.wmem_max = 262144
IP Address Requirements Node Interface Name Type IP Address node1 node1 Public 211.239.128.137 node1 node1-vip Virtual 192.168.2.37 node1 node1-priv Private 10.10.10.45 node2 node2 Public 211.239.128.147 node2 node2-vip Virtual 192.168.2.47 node2 node2-priv Private 10.10.10.46 /etc/hosts 127.0.0.1 localhost 211.239.128.137 node1.mycorpdomain.com node1 192.168.2.37 node1-vip.mycorpdomain.com node1-vip 10.10.10.45 node1-priv.mycorpdomain.com node1-priv 211.239.128.147 node2.mycorpdomain.com node2 192.168.2.47 node2-vip.mycorpdomain.com node2-vip 10.10.10.46 node2-priv.mycorpdomain.com node2-priv Hangcheck-timer Module on Kernel /etc/modprobe.conf options hangcheck-timer hangcheck_tick=30 hangcheck-timer
Raw device 구성 [root@node1 ~]# fdisk -l Disk /dev/sda: 21.4 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 910 7309543+ 83 Linux /dev/sda2 911 1170 2088450 82 Linux swap /dev/sda3 1171 2610 11566800 83 Linux Disk /dev/sdb: 21.4 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System Disk /dev/sdc: 21.4 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System Disk /dev/sdd: 21.4 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System Disk /dev/sde: 21.4 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System
Raw Partitions or Logical Volumes Required for Database Files on Linux Number Partition Size (MB) Purpose 1 500 SYSTEM tablespace 1 300 + (Number of instances * 250) SYSAUX tablespace Number of instances 500 UNDOTBSn tablespace (One tablespace for each instance) 1 250 TEMP tablespace 1 160 EXAMPLE tablespace 1 120 USERS tablespace 2 * number of instances 120 Two online redo log files for each instance 2 110 First and second control files 1 5 Server parameter file (SPFILE) 1 5 Password file
파티셔닝계획 모든파티셔닝은 extended 로구성 Purpose Minimum Size (in MB) Standardized Size (in MB) Disk Device Raw Device Oracle Cluster Registry 100 200 /dev/sdb5 /dev/raw/raw1 Oracle CRS Voting 20 50 /dev/sdb6 /dev/raw/raw2 SYSTEM Tablespace 500 600 /dev/sdc5 /dev/raw/raw3 SYSAUX Tablespace 800 1200 /dev/sdd5 /dev/raw/raw4 UNDOTBS1 Tablespace 500 600 /dev/sdc6 /dev/raw/raw5 UNDOTBS2 Tablespace 500 600 /dev/sdc7 /dev/raw/raw6 EXAMPLE Tablespace 160 200 /dev/sdd6 /dev/raw/raw7 USERS Tablespace 120 200 /dev/sdc8 /dev/raw/raw8 TEMP Tablespace 250 600 /dev/sdd7 /dev/raw/raw9 SPFILE 5 50 /dev/sde5 /dev/raw/raw10 Password File 5 50 /dev/sde6 /dev/raw/raw11 Control File 1 110 200 /dev/sde7 /dev/raw/raw12 Control File 2 110 200 /dev/sde8 /dev/raw/raw13 Redo Log 1_1 120 200 /dev/sdd8 /dev/raw/raw14 Redo Log 1_2 120 200 /dev/sde9 /dev/raw/raw15 Redo Log 2_1 120 200 /dev/sdd9 /dev/raw/raw16 Redo Log 2_2 120 200 /dev/sde10 /dev/raw/raw17
Fdisk 를이용한파티셔닝 (/dev/sdb) -- Oracle Cluster Registry, Oracle CRS Voting [root@node1 ~]# fdisk /dev/sdb The number of cylinders for this disk is set to 2610. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Command (m for help): n Command action e extended p primary partition (1-4) e Partition number (1-4): 1 First cylinder (1-2610, default 1): Last cylinder or +size or +sizem or +sizek (1-2610, default 2610): Using default value 2610 Command (m for help): n Command action l logical (5 or over) p primary partition (1-4) l First cylinder (1-2610, default 1): Using default value 1 Last cylinder or +size or +sizem or +sizek (1-2610, default 2610): +200M Command (m for help): n Command action l logical (5 or over) p primary partition (1-4) l First cylinder (26-2610, default 26): Using default value 26 Last cylinder or +size or +sizem or +sizek (26-2610, default 2610): +50M Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.
/dev/sdc 파티셔닝 /dev/sdb 와동일한방법으로진행 SYSTEM Tablespace UNDOTBS1 Tablespace UNDOTBS2 Tablespace USERS Tablespace /dev/sdd /dev/sdb 와동일한방법으로진행 EXAMPLE Tablespace TEMP Tablespace SYSAUX Tablespace Redo Log 1_1 Redo Log 2_1 /dev/sde /dev/sdb 와동일한방법으로진행 SPFILE Password File Control File 1 Control File 2 Redo Log 1_2 Redo Log 2_2
Raw device 와의 mapping Purpose Disk device Raw device Oracle Cluster Registry /dev/sdb5 /dev/raw/raw1 Oracle CRS Voting /dev/sdb6 /dev/raw/raw2 SYSTEM Tablespace /dev/sdc5 /dev/raw/raw3 SYSAUX Tablespace /dev/sdd5 /dev/raw/raw4 UNDOTBS1 Tablespace /dev/sdc6 /dev/raw/raw5 UNDOTBS2 Tablespace /dev/sdc7 /dev/raw/raw6 EXAMPLE Tablespace /dev/sdd6 /dev/raw/raw7 USERS Tablespace /dev/sdc8 /dev/raw/raw8 TEMP Tablespace /dev/sdd7 /dev/raw/raw9 SPFILE /dev/sde5 /dev/raw/raw10 Password File /dev/sde6 /dev/raw/raw11 Control File 1 /dev/sde7 /dev/raw/raw12 Control File 2 /dev/sde8 /dev/raw/raw13 Redo Log 1_1 /dev/sdd8 /dev/raw/raw14 Redo Log 1_2 /dev/sde9 /dev/raw/raw15 Redo Log 2_1 /dev/sdd9 /dev/raw/raw16 Redo Log 2_2 /dev/sde10 /dev/raw/raw17
최종파티션구성 Disk /dev/sda: 21.4 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 910 7309543+ 83 Linux /dev/sda2 911 1170 2088450 82 Linux swap /dev/sda3 1171 2610 11566800 83 Linux Disk /dev/sdb: 21.4 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 1 2610 20964793+ 5 Extended /dev/sdb5 1 25 200749+ 83 Linux /dev/sdb6 26 32 56196 83 Linux Disk /dev/sdd: 21.4 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System Disk /dev/sde: 21.4 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sde1 1 2610 20964793+ 5 Extended /dev/sde5 1 7 56164+ 83 Linux /dev/sde6 8 14 56196 83 Linux /dev/sde7 15 39 200781 83 Linux /dev/sde8 40 64 200781 83 Linux /dev/sde9 65 89 200781 83 Linux /dev/sde10 90 114 200781 83 Linux Disk /dev/sdc: 21.4 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdc1 1 2610 20964793+ 5 Extended /dev/sdc5 1 74 594342 83 Linux /dev/sdc6 75 148 594373+ 83 Linux /dev/sdc7 149 222 594373+ 83 Linux /dev/sdc8 223 247 200781 83 Linux
Raw device mapping /etc/sysconfig/rawdevices /dev/raw/raw1 /dev/sdb5 /dev/raw/raw2 /dev/sdb6 /dev/raw/raw3 /dev/sdc5 /dev/raw/raw4 /dev/sdd5 /dev/raw/raw5 /dev/sdc6 /dev/raw/raw6 /dev/sdc7 /dev/raw/raw7 /dev/sdd6 /dev/raw/raw8 /dev/sdc8 /dev/raw/raw9 /dev/sdd7 /dev/raw/raw10 /dev/sde5 /dev/raw/raw11 /dev/sde6 /dev/raw/raw12 /dev/sde7 /dev/raw/raw13 /dev/sde8 /dev/raw/raw14 /dev/sdd8 /dev/raw/raw15 /dev/sde9 /dev/raw/raw16 /dev/sdd9 /dev/raw/raw17 /dev/sde10 Mapping 설정즉시적용 Service rawdevices restart 즉시적용이불가능할때에는재부팅시에적용 Raw device 권한설정 chown oracle:dba /dev/raw/raw[1-9] chown oracle:dba /dev/raw/raw1[0-7] /etc/udev/permissions.d/50-udev.permissions 수정 Rebooting 시에적용 # raw devices ram*:root:disk:0660 #raw/*:root:disk:0660 raw/*:oracle:dba:0660 OS 및 raw device setting 끝 Node1 과 node2 에현재까지세팅한내용을똑같이세팅 (ip 정보는 5 번슬라이드참고 )
각노드의 oracle user에대해.bash_profile수정 node1 export PS1="`/bin/hostname -s`@`pwd` ] " export EDITOR=vi export ORACLE_SID=devdb1 export ORACLE_BASE=/oracle export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1 export ORA_CRS_HOME=$ORACLE_BASE/product/10.2.0/crs_1 export LD_LIBRARY_PATH=$ORACLE_HOME/lib export PATH=$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:/bin:\ /usr/bin:/usr/sbin:/usr/local/bin:/usr/x11r6/bin umask 022 Node2 export PS1="`/bin/hostname -s`@`pwd` ] " export EDITOR=vi export ORACLE_SID=devdb2 export ORACLE_BASE=/oracle export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1 export ORA_CRS_HOME=$ORACLE_BASE/product/10.2.0/crs_1 export LD_LIBRARY_PATH=$ORACLE_HOME/lib export PATH=$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:/bin:\ /usr/bin:/usr/sbin:/usr/local/bin:/usr/x11r6/bin umask 022
Configuring SSH on All Cluster Nodes Oracle Universal Installer 는설치중에노드간에서 ssh 와 scp command 를 file copy 를위해사용한다. 따라서명령프롬프트가실행되지않도록설정해주어야한다. Create RSA and DSA keys on each node Oracle user 로로그인 node1 에서 node1@/home/oracle ] mkdir ~/.ssh node1@/home/oracle ] chmod 700 ~/.ssh node1@/home/oracle ] ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/oracle/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/oracle/.ssh/id_rsa. Your public key has been saved in /home/oracle/.ssh/id_rsa.pub. The key fingerprint is: a4:3e:cd:d2:bc:e3:b7:b6:83:df:cc:13:a1:74:12:ab oracle@node1.mycorpdomain.com node1@/home/oracle ] ssh-keygen -t dsa Generating public/private dsa key pair. Enter file in which to save the key (/home/oracle/.ssh/id_dsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/oracle/.ssh/id_dsa. Your public key has been saved in /home/oracle/.ssh/id_dsa.pub. The key fingerprint is: 72:c5:57:36:87:44:90:a7:4c:7b:f6:99:a1:86:1f:ea oracle@node1.mycorpdomain.com Node2 에서도똑같이실행
Add keys to an authorized key file ~/.ssh에들어가서 id_dsa.pub 와 id_rsa.pub 가있는지확인 node1@/home/oracle ] cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys node1@/home/oracle ] cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys node1@/home/oracle ] ssh node2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys The authenticity of host 'node2 (192.168.2.132)' can't be established. RSA key fingerprint is 63:d3:52:d4:4d:e2:cb:ac:8d:4a:66:9f:f1:ab:28:1f. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'node2,192.168.2.132' (RSA) to the list of known hosts. oracle@node2's password: node1@/home/oracle ] ssh node2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys oracle@node2's password: node1@/home/oracle ] scp ~/.ssh/authorized_keys node2:~/.ssh/authorized_keys oracle@node2's password: authorized_keys 100% 1716 1.7KB/s 00:00 연결테스트 node1@/home/oracle ] ssh node1 date node1@/home/oracle ] ssh node2 date node1@/home/oracle ] ssh node1-priv date node1@/home/oracle ] ssh node2-priv date node1@/home/oracle ] ssh node1.mycorpdomain.com date node1@/home/oracle ] ssh node2.mycorpdomain.com date node1@/home/oracle ] ssh node1-priv.mycorpdomain.com date node1@/home/oracle ] ssh node2-priv.mycorpdomain.com date
Install Oracle Clusterware 파일을다운로드받은후압축을푼다. node1@/home/oracle ] unzip 10201_clusterware_linux32.zip 생성된 clusterware폴더안에 runinstaller를실행시킨다 node1@/home/oracle ]../runinstaller GUI화면이뜨면Next클릭
Inventory 경로와그룹이름을확인하고 Next 클릭
Clusterware 의홈경로설정 Path 의 db_1 을 crs_1 로바꾸고 Next 클릭
물리적메모리에관련된 warning 은무시하고 Next 클릭 Warning 메세지박스가뜨면 yes 클릭
Add 를클릭하여아래그림과같이 cluster node 를추가시킨다.
이더넷카드와서브넷 ip 를확인하고 Next 클릭
The Oracle Clusterware Voting Disk and Oracle Cluster Registry The Oracle Clusterware requires the following two critical files: Voting Disk Manages cluster membership by way of a health check and arbitrates cluster ownership among the instances in case of network failures. RAC uses the voting disk to determine which instances are members of a cluster. The voting disk must reside on shared disk. For high availability, Oracle recommends that you have multiple voting disks. The Oracle Clusterware enables multiple voting disks but you must have an odd number of voting disks, such as three, five, and so on. If you define a single voting disk, then you should use external mirroring to provide redundancy. Oracle Cluster Registry (OCR) Maintains cluster configuration information as well as configuration information about any cluster database within the cluster. The OCR also manages information about processes that the Oracle Clusterware controls. The OCR stores configuration information in a series of key-value pairs within a directory tree structure. The OCR must reside on shared disk that is accessible by all of the nodes in your cluster. The Oracle Clusterware can multiplex the OCR and Oracle recommends that you use this feature to ensure cluster high availability. You can replace a failed OCR online, and you can update the OCR through supported APIs such as Enterprise Manager, the Server Control Utility (SRVCTL), or the Database Configuration Assistant (DBCA). Note: Both the voting disks and the OCRs must reside on either cluster file system files or on shared raw devices that you configure before you install the Oracle Clusterware and RAC.
OCR 구성 그림과같이 External Redundancy 에체크하고 raw device 명을입력한다. Test 가아닌실제운영서버에서는 OCR 을미러링구성해야한다.
Voting disk 구성 그림과같이 External Redundancy 에체크하고 raw device 명을입력한다. Test 가아닌실제운영서버에서는 voting disk 를미러링구성해야한다
Summary 내용을확인하고 Install 클릭
설치화면
스크립트실행 진행바가 100% 까지차게되면양쪽노드에서 root사용자로 2개의스크립트를실행시킨다. orainstroot.sh root.sh 실행이끝나면 OK클릭
설치끝 Oracle 유저로정상적으로설치완료되었는지양쪽노드에서확인한다. node1@/home/oracle ] olsnodes node1 node2
Install Oracle Database Software runinstaller 를실행시킨다 Next 클릭
Select Install Type Custom 을선택하고 Next 클릭
Specify Home Details $ORACLE_HOME 경로를확인하고 Next 클릭
Specify Hardware Cluster Installation Mode Cluster 로설치할건지 local 로설치할건지를결정한다. Cluster 로선택하고 Select All 클릭 -> Next
Available Product Components 설치될 components 들을확인하고 Next 클릭
Product-Specific Prerequisite Checks Memory 관련 warning 을무시하고 Next 클릭
Privileged Operating System Groups dba 그룹확인후 Next 클릭
Create Database Install database Software only 를선택하고 Next
Summary 설치될 product 들을확인하고 Install 클릭
Install
Script 실행 양쪽노드에서동일하게실행
End of Installation Exit 클릭
CRS, Oracle engine 10.2.0.3 patch CRS patch 다운로드받은파일의압축을풀고 Disk1 폴더의 runinstaller 를실행시킨다.
Specify Home Details Name 과 Path 를그림과같이 CRS 구성과맞게바꿔준다.
Specify Hardware Cluster Installation Mode Next 클릭
Summary 설치내용을확인하고 Install 클릭
Install
End of Installation Exit 클릭
Oracle Engine patch runinstaller 실행
Specify Home Details Name 과 Path 가 $ORACLE_HOME 과맞는지확인하고 Next 클릭
Specify Hardware Cluster Installation Mode Next 클릭
Summary 설치내용을확인하고 Install 클릭
Install
Configuration Assistants
Script 실행 양쪽노드에서 root user 로실행한다.
End of Installation
Network구성 Node1에서 Oracle user로 NETCA실행 node1@/home/oracle ] netca Cluster configuration을선택하고 Next 클릭
Select all nodes 를선택하고 Next 클릭
Listener configuration 을선택하고 Next 클릭
Add 를선택하고 Next 클릭
Listener 이름구성 Next 클릭
Protocol 선택
Setting port number
More listener No 를선택하고 Next 클릭
Done Next 클릭 다음화면에서 Finish 클릭
Create Database Oracle user로접속하여다음명령실행 node1@/home/oracle ] dbca Oracle Real Application Cluster database 를선택하고 Next 클릭
Create a Database 를선택하고 Next 클릭
Node Selection Select All 을선택하고 Next 클릭
Database Templates Custom Database 를선택하고 Next 클릭
Database Identification devdb 를입력하고 Next 클릭
Management Options 체크를해제하면 EM 을구성할수없다. Next 클릭
SYS, SYSTEM 의암호입력
Storage Options Raw Devices 를선택한후 Next 클릭
Recovery Configuration Recovery area 는지정하지않음 -> Next
Database Contents Enterprise Manager Repository 만남기고나머지는해제 Standard Database Components 를클릭하여 JVM 만남기고해제
Database Services Next 클릭
SPFILE 위치지정 8 번슬라이드를참조하여구성 Memory, Sizing, Character Sets 는환경에맞게수정
Database Storage Control file, data file, redo log file 의경로를지정해준다. 8 번슬라이드를참고하여각각에맞는 raw device 를지정해준다.
Control file
Data file
Redo log file
Generate Database Creation Scripts 에체크하고 Finish 클릭
Summary Ok 를클릭하면 Script 를생성한다.
Database 생성
DB 생성완료