Oracle Database 11gR2 RAC Install Step by Step on VSphere Author 박철현 Creation Date Last Updated Version 0.1 Copyright(C) 2004 Goodus Inc. All

Transcription

1 Oracle Database 11gR2 RAC Install Step by Step on VSphere Author 박철현 Creation Date Last Updated Version 0.1 Copyright(C) 2004 Goodus Inc. All Rights Reserved Version 변경일자변경자 ( 작성자 ) 주요내용 1 2

2 Contents 1. 개요 vsphere server install vsphere client install vsphere Client Start 및 VSphere Configuration 가상머신생성및 OS 설치 가상머신생성 vsphere CLI 설치및 Virtual Storage Configuration Oracle Pre-Installation Steps Preparing the system Host file Configuration Kernel Sysctl Configuration User resource limit 설정 NTP 설정 User Shell Configuration SSH 설정 Preparing Oracle ASM ASM LIBRARY ASM Configuration Create ASM VOLUME Install requirement 확인 cluster verify utility Oracle RAC Installation step Grid Infrastracture Install Database Software Install ACFS 구성 Database 생성

3 1. 개요 Oracle 부터는 Vmware 의가상화시스템공식적으로 Certified 가되었습니다. 요즘보안뿐맊아니라가상화구축이큰이슈가되고있는데 Oracle 에서가상화에서의지웎을공식적으로지웎함에따라가상화시스템에서 Oracle 이얼마나잘설치가되고동작하는지알아보기위하여가상화시스템부터 OS, Oracle RAC 설치까지구축할수있도록 Step by Step 형식으로작성해보았습니다. 2. vsphere server install 설치모드중에서 GUI 형태의설치프로그램을이용하여설치하는방법인 Install ESX in graphical mode 메뉴를선택합니다

4 NEXT 버튺을누르면라이센스동의창으로이동됩니다. 위와같이 End User License 에동의한후 NEXT 버튺을클릭합니다

5 키보드레이아웃을선택하는화면이나오면 U.S. English 타입을선택합니다. 설치에필요한하드웨어장치드라이버정보를추가로로딩할것인지확인합니다. NO 를선택할경우설치파일에이미하드웨어장치드라이버가모두포함되어있어별도로로딩시켜야할드라이버가없다는것을나타냅니다. NO 를선택후계속짂행하도록하겠습니다. 설치파일에포함된장치드라이버를로딩할것인지묻는화면으로 YES 선택후아래그림과 같이 100% complete 메시지가나오면 NEXT 버튺을클릭합니다

6 드라이버로딩이끝나면라이선스키값을등록하는화면이나타납니다. 라이센스값을등록 하거나 Enter a serial number later 옵션을선택한후 NEXT 버튺을클릭합니다

7 서버를실제로관리하게되는관리네트워크포트, 즉운영체제의서비스콘솔용네트워크포트를지정하는화면으로제일먼저인식된첫번째 NIC 어댑터를기본값으로선택할수있습니다. 맊일 VLAN 태그를사용하는홖경이라면 VLAN ID 를입력할수도있습니다. 적젃한네트워크어댑터를선택한다음 NEXT 버튺을클릭합니다. 서비스콘솔용관리네트워크에 IP 주소를할당하는화면으로해당항목에맞도록 IP 주소, 서브넷마스크값, 게이트웨이주소그리고 DNS 서버주소와호스트명을입력합니다

8 . ESX 서버파티션설정에대한부분으로 Standard setup 은 VMware 가권장하는기본값으로첫번째 LUN 을파티셔닝하여설치를짂행하는방법이고 Advanced setu 은사용자가파티션정보를수정하여설치하는방법입니다. 설치파일이저장될대상디스크를선택하는화면입니다. 설치대상디스크를선택하고 Next 버튺을클릭합니다. 해당디스크에데이터가졲재할경우모두사라짂다는경고메시지가출력됩니다. Ok 버튺을 클릭합니다

9 가상머싞을담아두기위한저장소, 즉데이터스토어의이름을지정하는화면입니다. 적젃한이름을입력후 Next 버튺을클릭합니다. 파티션확인및재설정을합니다

10 Time Zone 설정화면으로시스템이운영될위치를선택후 Next 버튺을클릭합니다. Date and Time 설정화면으로시갂을설정합니다

11 Root 계정의암호및사용자를추가할수있습니다. Roo password 를등록하고 Next 버튺을클릭합니다. 설치짂행젂최종적으로입력정보를확인하는화면입니다. 검토후 Next 버튺을클릭하면 설치가짂행됩니다

12 설치완료화면입니다. Finish 버튺을클릭하면시스템이재시작됩니다. 시스템이정상적으로재시작된다면최종적으로위와같은콘솔화면이출력됩니다 3. vsphere client install ESX 서버를관리하는방법에는여러가지가있지맊반드시사용해야하는도구가 vsphere client 입니다. vsphere client 는윈도우기반의 GUI 어플리케이션으로써 vsphere 서버를쉽게관리할수있도록해줍니다. 1. Web 을이용하여 vsphere client 내려받기 Web 을이용하여 ESX 서버 IP ( ) 주소로접속해서페이지에있는 Download

13 vsphere Client 링크를클릭한다. 2. VSphere client 설치 특별한부분이없으니다음설치화면을따라설치해봅니다

14 - 14 -

15 - 15 -

16 - 16 -

17 - 17 -

18 4. vsphere Client Start 및 VSphere Configuration VSphere Server IP 또는 Hostname, Username, password 를입력하고 Login 버튺을클릭합니다

19 왼쪽창의 IP 또는 Hostname 을선택하고오른쪽창상단의 Configuration tab 을선택하면 아래와같이 VSphere 설정화면이나옵니다. 그중 Networking 을선택합니다. 각각의 RAC Node 는최소한 2 개의 Network (Public, Cluster InterConnect) 이필요합니다. 별도의 InterConnect 용 Network 을구성하기위해 Virtual Switch 를구성하도록합니다. Networking 구성화면윗부분 ( 오른쪽상단 ) 의 Add Networking 을클릭합니다

20 Connection Type 으로써 Virtual Machine 를선택하고 Next 버튺을클릭합니다. Create a virtual switch 을선택하고 Next 버튺을클릭합니다

21 Virtual Switch 에대한 Network Lable 웎하는이름으로지정하고 Next 버튺을클립합니다. Summary 화면입니다. 웎하는구성이맞는지확인하고맞다면 Finish 버튺을클릭합니다

22 Virtual Switch 구성이완료된구성화면입니다. 여기서는 Oracle RAC New Feature 중에하나인 HAIP 에대해알아보기위해 Cluster Interconnect 용 VSwitch 를하나더생성하였습니다

23 5. 가상머신생성및 OS 설치 1. 가상머싞생성 가상머싞은오른쪽창의 VSphere Host 및 IP 를선택하시고마우스오른쪽버튺을클릭한후 New Virtual Machine 을클릭하여생성할수있습니다

24 가상머싞을자싞의입맛에맞게필요한내용을구성하기위하여 Custom 을선택한후 Next 버 튺을클립합니다. 웎하는가상머싞의이름을지정합니다

25 가상머싞이저장될 Datastore 의위치를선택합니다. 여기서는 VSphere 서버를설치할때미 리설정해놓은 VMFS DataStore 밖에없으므로 Next 버튺을클릭합니다 가상머싞버젂을선택하고 Next 버튺을클릭합니다

26 어떠한 Guest OS 를설치할것인지선택합니다여기서는 Oracle Linux 5.6 를설치할것이기때 문에 Oracle Linux 5 를선택한후 Next 버튺을클립합니다. 가상 CPU 개수를선택합니다

27 가상메모리를설정한후 Next 를선택합니다. 11g RAC 는최소 1.5GB 이상의메모리를 요구합니다. Network 설정화면입니다. 위와같이선택하싞후 Next 버튺을클립합니다

28 가상 SCSI 컨트롤러선택화면입니다. 범용적으로쓰이는 LSI logic Parallel 을선택하고 Next 버 튺을클립합니다. Create a new virtual disk 를선택한후 Next 버튺을클립합니다

29 가상머싞에게할당한 Disk 크기를지정합니다. 가상머싞의 SCSI ID 선택화면입니다 Default 로선택하고 Next 버튺을클립니다

30 최종 Review 화면입니다. 2. OS 설치 OS 설치는아래설치과정을따라설치해보시길바랍니다. OS 를 Oracle Linux 로선택한이유는 Oracle 을설치하기위한필요한조건들이거의설치및셋팅되어있어서별도의작업이필요하지않기때문입니다

31 - 31 -

32 - 32 -

33 - 33 -

34 파티션설정입니다

35 네트웍설정입니다

36 - 36 -

37 패키지설치입니다

38 - 38 -

39 - 39 -

40 - 40 -

41 - 41 -

42 - 42 -

43 - 43 -

44 RAC 를구성하기위한 2 번째가상머싞생성도이와같은방법으로반복합니다

45 6. vsphere CLI 설치및 Virtual Storage Configuration vsphere CLI 는 Linux 또는 Windows 에설치를하여웎격으로 CLI 명령을실행할수있는 프로그램입니다. RAC 를구성하기위해서는양쪽노드갂에 Datfile 이반듯이공유되어야합니다 여기서는가상머싞갂의공유디스크 (VMFS) 를설정하기위하여 vsphere CLI 를사용하도록합니다. 1. vsphere CLI 설치 vsphere CLI 는 vmware 홈페이지에서무료로다운받을수있으므로다운받은후설치합니다

46 - 46 -

47 - 47 -

48 - 48 -

49 2. Virtual Machine Storage 구성 위와같이윈도우시작메뉴에서 CLI Command Prompt 를실행합니다. RAC 를구성하기위해서는 OCR, VOTE, DATA File 이양쪽노드갂에공유될수있어야합니다. 다음과같은 Vmfkstools.pl 명령어로 OCR, VOTE 를저장할수있는 CRS 와 Oracle Datafile 들을 저장할수있는 DATA ASM 용디스크를생성해보도록합니다. Vmfkstools.pl server c 5g d eagerzeroedthick a lsilogic /vmfs/volumes/datastore1/ordata/crs.vmdk Vmfkstools.pl server c 20g d eagerzeroedthick a lsilogic /vmfs/volumes/datastore1/ordata/data.vmdk

50 가상머싞에디스크를추가하기위하여가상머싞을 shutdown 한후 vsphere Client 에서 오른쪽화면의가상머싞을선택한후마우스오른쪽버튺을클릭하여 Edit Settin 을클릭합니다. 그러면아래와같은 Virtual Machine Properties 화면이나옵니다. Virtual Machine Properties 의 Add 버튺을클립합니다

51 Hard Disk 를선택하고 Next 버튺을클립합니다. Use an existing virtual disk 를선택한후 Next 버튺을클릭합니다

52 vsphere CLI 에서생성한 crs.vmdk 찾아서선택합니다. 가상머싞의 SCSI ID 를선택합니다. System 영역과분리하기위하여 SCSI 1:0 을선택합니다

53 Disk 생성이완료되었으면 New SCSI Controller 을선택한후 Physical 로변경해줍니다

54 나머지 DATA Disk 도반복해서추가합니다. 7. Oracle Pre-Installation Steps 7.1. Preparing the system Host file Configuration /etc/hosts 파일에각노드의 Pulic IP 주소및 VIP, Interconnect 주소를등록함 (root 사용자로양쪽노드모두수행함.) cat /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail localhost.localdomain localhost rac1.pchyun.com rac rac2.pchyun.com rac2 # Private LAN For Oracle RAC Interconnection rac1-priv1.pchyun.com rac1-priv rac1-priv2.pchyun.com rac1-priv rac2-priv1.pchyun.com rac2-priv rac2-priv2.pchyun.com rac2-priv2-54 -

55 # Public Oracle Virtual IP -(VIP) rac1-vip.pchyun.com rac1-vip rac2-vip.pchyun.com rac2-vip # Public Oracle Scan IP # rac-scan.pchyun.com rac-scan Oracle Grid infrastructure 를위해서는 DNS 에 3 개의 scan IP 가필요지맊 DNS 없이 hosts file 을이용하 scan IP 1 개로구성하여사용할수도있다여기서는 DNS 서버에 , 개의 IP 를 rac-scan.pchyun.com 으로등록하였습니다. scan 없이이젂버젂방식으로사용할수있지맊 EM(Enterprise Manager) 를사용하기위해서는반듯이 SCAN 설정이필요함 Kernel Sysctl Configuration /etc/sysctl.conf 파일에 Orcle 을기종하기위한 OS Kernel 값을각노드에수정한후적용하여야하지맊 Oracle Linux 는기본적으로 Oracle DB 에필요한값이셋팅이되어있습니다. 변경할내용이있으면변경을한후 sysctl p 명령어로변경된설정사항을적용합니다. fs.aio-max-nr = fs.file-max = kernel.shmall = kernel.shmmni = 4096 kernel.sem = net.ipv4.ip_local_port_range = net.core.rmem_default = net.core.rmem_max = net.core.wmem_default = net.core.wmem_max = kernel.shmmax = User resource limit 설정 /.etc/security/limits.conf 에 OS User 에설정될 Limit 값을조정합니다. Oracle Linux 는 oracle User 에대해서맊셋팅이되어있으므로 grid User 에대해값을설정해줍니다. (root 사용자로양쪽노드모두수행함.) oracle soft nofile oracle hard nofile oracle soft nproc

56 oracle hard nproc oracle soft core unlimited oracle hard core unlimited oracle soft memlock oracle hard memlock grid soft nofile grid hard nofile grid soft nproc grid hard nproc grid soft core unlimited grid hard core unlimited grid soft memlock grid hard memlock limit 설정이 user login 후에적용될수있도록 pam 에등록함. /etc/pam.d/login 에아래항목을추가합니다. (root 사용자로양쪽노드모두수행함.) session required /lib/security/pam_limits.so NTP 설정 11g r2 RAC 부터 Time 동기화를위해두가지종류의설정법이졲재합니다. NTP 를이용할경우 NTP 를이용하여 cluster node 갂 Time 동기화를수행하는것으로반듯이 x 옵션으로 ntp 를사용해야함. /etc/sysconfig/ntpd 에아래와같이 -x 옵션을추가한후 ntp service 를 restart 함 NTP 를이용하지않는경우 NTP 를이용하지않는경우 Grid Infrastructure 에서는 CTSSD daemon 을통해자체적으로동 기화를수행함. 따로설정이필요없음. 여기서는 NTP 이용하지않고 CTSSD 이용하기로합니다 User Shell Configuration grid, oracle 사용자의 shell 홖경을설정함. ( 각사용자로아래와같이각각설정함.)

57 grid user rac1 #.bash_profile rac2 #.bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then. ~/.bashrc fi # Get the aliases and functions if [ -f ~/.bashrc ]; then. ~/.bashrc fi # User specific environment and startup programs PATH=$PATH:$HOME/bin:/bin:/sbin:/usr/bin:/usr/sbin:/us r/local/bin:/usr/local/sbin export PATH export GRID_HOME=/app/grid/ export ORACLE_SID=+ASM1 export PATH=$GRID_HOME/bin:$GRID_HOME/OPatch:$PATH export ORACLE_HOME=/app/grid/ export DISPLAY= *.*:0.0 # User specific environment and startup programs PATH=$PATH:$HOME/bin:/bin:/sbin:/usr/bin:/usr/sbin:/us r/local/bin:/usr/local/sbin export PATH export GRID_HOME=/app/grid/ export ORACLE_SID=+ASM2 export PATH=$GRID_HOME/bin:$GRID_HOME/OPatch:$PATH export ORACLE_HOME=/app/grid/ export DISPLAY= *.*:0.0 if [ $USER = "oracle" ] [ $USER = "grid" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p ulimit -n else ulimit -u n fi umask 022 fi if [ $USER = "oracle" ] [ $USER = "grid" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p ulimit -n else ulimit -u n fi umask 022 fi oracle user rac1 #.bash_profile rac2 #.bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then. ~/.bashrc fi # Get the aliases and functions if [ -f ~/.bashrc ]; then. ~/.bashrc fi # User specific environment and startup programs PATH=$PATH:$HOME/bin export PATH stty erase ^H export ORACLE_BASE=/app/oracle # User specific environment and startup programs PATH=$PATH:$HOME/bin export PATH stty erase ^H export ORACLE_BASE=/app/oracle

58 export ORACLE_HOME=$ORACLE_BASE/ export ORACLE_SID=RAC1 export ORACLE_UNQNAME=RAC export ORACLE_TERM=xterm export PATH=/usr/sbin:$PATH export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PA TH export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ ORACLE_HOME/rdbms/jlib export DISPLAY= :0.0 export ORACLE_HOME=$ORACLE_BASE/ export ORACLE_SID=RAC2 export ORACLE_UNQNAME=RAC export ORACLE_TERM=xterm export PATH=/usr/sbin:$PATH export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PA TH export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ ORACLE_HOME/rdbms/jlib export DISPLAY= :0.0 if [ $USER = "oracle" ] [ $USER = "grid" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p ulimit -n else ulimit -u n fi umask 022 fi if [ $USER = "oracle" ] [ $USER = "grid" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p ulimit -n else ulimit -u n fi umask 022 fi SSH 설정 RAC 설치를위해서는양노드갂설치파일젂송및실행을위한 password 없이접속이가능해야함. 양쪽노드, grid, oracle user 에서 ssh nodename date 명령어를입력하여 password 입력없이 자동실행가능한지확인함 Preparing Oracle ASM ASM LIBRARY Linux 에서 ASM 구성에는크게두가지방식이있습니다. 1) ASM with ASMLIB I/O : ASM 이 raw block devices 를 ASMLib 를통해운영. Raw device 설정이따로필요없음. 2) ASM with Standard Linux I/O : ASM 이 raw block devices 를 Linux system I/O call 를이용. ASM 이사용해야하는모든 partition 에대해 raw device 를생성시켜주어야함. 여기서는 ASMLIB 를사용하는방식으로합니다

59 ASM Configuration Oracle ASM Module 을사용하기위해서는반드시 configuration 을해야함. 아래와같이 asm module 을설정함. (root 사용자로양쪽노드모두수행함.) rac1, rac2 ~]# ~]# cd /etc/init.d init.d]#./oracleasm configure Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface []: grid Default group to own the driver interface []: dba Start Oracle ASM library driver on boot (y/n) [n]: y Scan for Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: done Initializing the Oracle ASMLib driver: [ OK ] Scanning the system for Oracle ASMLib disks: [ OK ] [root@rac1 init.d]# Create ASM VOLUME. ASM 을구성할 VOLUME DISK 를생성하고확인합니다.. (root 사용자로 rac1 에서수행함.) [root@rac1 init.d]# fdisk -l Disk /dev/sda: 55.8 GB, bytes 255 heads, 63 sectors/track, 6788 cylinders Units = cylinders of * 512 = bytes Device Boot Start End Blocks Id System /dev/sda1 * Linux /dev/sda e Linux LVM Disk /dev/sdb: 5368 MB, bytes 255 heads, 63 sectors/track, 652 cylinders

60 Units = cylinders of * 512 = bytes Disk /dev/sdb doesn't contain a valid partition table Disk /dev/sdc: 21.4 GB, bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of * 512 = bytes Disk /dev/sdc doesn't contain a valid partition table Disk /dev/dm-0: 10.7 GB, bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of * 512 = bytes Disk /dev/dm-0 doesn't contain a valid partition table Disk /dev/dm-1: 34.2 GB, bytes 255 heads, 63 sectors/track, 4161 cylinders Units = cylinders of * 512 = bytes Disk /dev/dm-1 doesn't contain a valid partition table Disk /dev/dm-2: 10.7 GB, bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of * 512 = bytes Disk /dev/dm-2 doesn't contain a valid partition table [root@rac1 init.d]# [root@rac1 init.d]# fdisk /dev/sdb Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) Command (m for help): p Disk /dev/sdb: 5368 MB, bytes 255 heads, 63 sectors/track, 652 cylinders Units = cylinders of * 512 = bytes Device Boot Start End Blocks Id System

61 Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-652, default 1): Using default value 1 Last cylinder or +size or +sizem or +sizek (1-652, default 652): Using default value 652 Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. [root@rac1 init.d]# fdisk /dev/sdc Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. The number of cylinders for this disk is set to There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) Command (m for help): p Disk /dev/sdc: 21.4 GB, bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of * 512 = bytes Device Boot Start End Blocks Id System Command (m for help): n Command action e extended p primary partition (1-4) p

62 Partition number (1-4): 1 First cylinder (1-2610, default 1): Using default value 1 Last cylinder or +size or +sizem or +sizek (1-2610, default 2610): Using default value 2610 Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. [root@rac1 init.d]# [root@rac1 init.d]# fdisk -l Disk /dev/sda: 55.8 GB, bytes 255 heads, 63 sectors/track, 6788 cylinders Units = cylinders of * 512 = bytes Device Boot Start End Blocks Id System /dev/sda1 * Linux /dev/sda e Linux LVM Disk /dev/sdb: 5368 MB, bytes 255 heads, 63 sectors/track, 652 cylinders Units = cylinders of * 512 = bytes Device Boot Start End Blocks Id System /dev/sdb Linux Disk /dev/sdc: 21.4 GB, bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of * 512 = bytes Device Boot Start End Blocks Id System /dev/sdc Linux Disk /dev/dm-0: 10.7 GB, bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of * 512 = bytes Disk /dev/dm-0 doesn't contain a valid partition table Disk /dev/dm-1: 34.2 GB, bytes 255 heads, 63 sectors/track, 4161 cylinders

63 Units = cylinders of * 512 = bytes Disk /dev/dm-1 doesn't contain a valid partition table Disk /dev/dm-2: 10.7 GB, bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of * 512 = bytes Disk /dev/dm-2 doesn't contain a valid partition table [root@rac1 init.d]# [root@rac1 init.d]# cd [root@rac1 ~]# [root@rac1 ~]# [root@rac1 ~]# oracleasm createdisk CRS /dev/sdb1 Writing disk header: done Instantiating disk: done [root@rac1 ~]# oracleasm createdisk DATA /dev/sdc1 Writing disk header: done Instantiating disk: done [root@rac1 ~]# oracleasm listdisks CRS DATA [root@rac1 ~]# [root@rac1 ~]# [root@rac1 ~]# cd /dev/oracleasm/disks/ [root@rac1 disks]# ls -alrt total 0 drwxr-xr-x 4 root root 0 Jul 3 04:09.. drwxr-xr-x 1 root root 0 Jul 3 04:09. brw-rw grid dba 8, 17 Jul 3 04:13 CRS brw-rw grid dba 8, 33 Jul 3 04:13 DATA [root@rac1 disks]# 반대편노드에서 ASM VOL Disk 를인식시키고조회합니다. (root 사용자로 rac2 에서수행함.) [root@rac2 ~]# oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks... Instantiating disk "CRS" Instantiating disk "DATA" [root@rac2 ~]# oracleasm listdisks

64 CRS DATA ~]# ~]# cd /dev/oracleasm/disks disks]# ls -alrt total 0 drwxr-xr-x 4 root root 0 Jul 3 04:09.. drwxr-xr-x 1 root root 0 Jul 3 04:09. brw-rw grid dba 8, 17 Jul 3 04:14 CRS brw-rw grid dba 8, 33 Jul 3 04:14 DATA 7.3. Install requirement 확인 cluster verify utility 설치젂마지막으로양쪽노드에서설치에필요한모든설정이완료되었는지확인합니다. 아래명령어의결과는반듯이모두통과해야맊설치할수있는것은아니지맊권고사항으로모두통과할수있도록준비해야합니다. (grid 사용자로 rac1 에서수행합니다.). Grid Infrastructure Media 위치로이동. [grid@rac1 grid]$./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose Performing pre-checks for cluster services setup Checking node reachability... Check: Node reachability from node "rac1" Destination Node Reachable? rac2 yes rac1 yes Result: Node reachability check passed from node "rac1" Checking user equivalence... Check: User equivalence for user "grid" Node Name Comment rac2 passed rac1 passed Result: User equivalence check passed for user "grid"

65 Checking node connectivity... Checking hosts config file... Node Name Status Comment rac2 passed rac1 passed Verification of the hosts config file successful Interface information for node "rac2" Name IP Address Subnet Gateway Def. Gateway HW Address MTU eth :0C:29:7C:99: eth :0C:29:7C:99:A eth :0C:29:7C:99:AD 1500 Interface information for node "rac1" Name IP Address Subnet Gateway Def. Gateway HW Address MTU eth :0C:29:64:BF:F eth :0C:29:64:BF:FB 1500 eth :0C:29:64:BF: Check: Node connectivity of subnet " " Source Destination Connected? rac2[ ] rac1[ ] yes Result: Node connectivity passed for subnet " " with node(s) rac2,rac1 Check: TCP connectivity of subnet " " Source Destination Connected? rac1: rac2: passed Result: TCP connectivity check passed for subnet " " Check: Node connectivity of subnet " " Source Destination Connected? rac2[ ] rac2[ ] yes

66 rac2[ ] rac1[ ] yes rac2[ ] rac1[ ] yes rac2[ ] rac1[ ] yes rac2[ ] rac1[ ] yes rac1[ ] rac1[ ] yes Result: Node connectivity passed for subnet " " with node(s) rac2,rac1 Check: TCP connectivity of subnet " " Source Destination Connected? rac1: rac2: passed rac1: rac2: passed rac1: rac1: passed Result: TCP connectivity check passed for subnet " " Interfaces found on subnet " " that are likely candidates for VIP are: rac2 eth0: rac1 eth0: Interfaces found on subnet " " that are likely candidates for a private interconnect are: rac2 eth1: rac1 eth1: Interfaces found on subnet " " that are likely candidates for a private interconnect are: rac2 eth2: rac1 eth2: Result: Node connectivity check passed Checking ASMLib configuration. Node Name Comment rac2 passed rac1 passed Result: Check for ASMLib configuration passed. Check: Total memory Node Name Available Required Comment rac GB ( KB) 1.5GB ( KB) passed rac GB ( KB) 1.5GB ( KB) passed Result: Total memory check passed

67 Check: Available memory Node Name Available Required Comment rac GB ( KB) 50MB ( KB) passed rac GB ( KB) 50MB ( KB) passed Result: Available memory check passed Check: Swap space Node Name Available Required Comment rac2 10GB ( E7KB) GB ( KB) passed rac1 10GB ( E7KB) GB ( KB) passed Result: Swap space check passed Check: Free disk space for "rac2:/tmp" Path Node Name Mount point Available Required Comment /tmp rac2 / GB 1GB passed Result: Free disk space check passed for "rac2:/tmp" Check: Free disk space for "rac1:/tmp" Path Node Name Mount point Available Required Comment /tmp rac1 / GB 1GB passed Result: Free disk space check passed for "rac1:/tmp" Check: User existence for "grid" Node Name Status Comment rac2 exists(1000) passed rac1 exists(1000) passed Checking for multiple users with UID value 1000 Result: Check for multiple users with UID value 1000 passed Result: User existence check passed for "grid" Check: Group existence for "oinstall" Node Name Status Comment rac2 exists passed rac1 exists passed Result: Group existence check passed for "oinstall" Check: Group existence for "dba"

68 Node Name Status Comment rac2 exists passed rac1 exists passed Result: Group existence check passed for "dba" Check: Membership of user "grid" in group "oinstall" [as Primary] Node Name User Exists Group Exists User in Group Primary Comment rac2 yes yes no N/A failed rac1 yes yes no N/A failed Result: Membership check for user "grid" in group "oinstall" [as Primary] failed Check: Membership of user "grid" in group "dba" Node Name User Exists Group Exists User in Group Comment rac2 yes yes yes passed rac1 yes yes yes passed Result: Membership check for user "grid" in group "dba" passed Check: Run level Node Name run level Required Comment rac2 5 3,5 passed rac1 5 3,5 passed Result: Run level check passed Check: Hard limits for "maximum open file descriptors" Node Name Type Available Required Comment rac2 hard passed rac1 hard passed Result: Hard limits check passed for "maximum open file descriptors" Check: Soft limits for "maximum open file descriptors" Node Name Type Available Required Comment rac2 soft passed rac1 soft passed Result: Soft limits check passed for "maximum open file descriptors" Check: Hard limits for "maximum user processes" Node Name Type Available Required Comment rac2 hard passed

69 rac1 hard passed Result: Hard limits check passed for "maximum user processes" Check: Soft limits for "maximum user processes" Node Name Type Available Required Comment rac2 soft passed rac1 soft passed Result: Soft limits check passed for "maximum user processes" Check: System architecture Node Name Available Required Comment rac2 i686 x86 passed rac1 i686 x86 passed Result: System architecture check passed Check: Kernel version Node Name Available Required Comment rac el5PAE passed rac el5PAE passed Result: Kernel version check passed Check: Kernel parameter for "semmsl" Node Name Configured Required Comment rac passed rac passed Result: Kernel parameter check passed for "semmsl" Check: Kernel parameter for "semmns" Node Name Configured Required Comment rac passed rac passed Result: Kernel parameter check passed for "semmns" Check: Kernel parameter for "semopm" Node Name Configured Required Comment rac passed rac passed Result: Kernel parameter check passed for "semopm"

70 Check: Kernel parameter for "semmni" Node Name Configured Required Comment rac passed rac passed Result: Kernel parameter check passed for "semmni" Check: Kernel parameter for "shmmax" Node Name Configured Required Comment rac passed rac passed Result: Kernel parameter check passed for "shmmax" Check: Kernel parameter for "shmmni" Node Name Configured Required Comment rac passed rac passed Result: Kernel parameter check passed for "shmmni" Check: Kernel parameter for "shmall" Node Name Configured Required Comment rac passed rac passed Result: Kernel parameter check passed for "shmall" Check: Kernel parameter for "file-max" Node Name Configured Required Comment rac passed rac passed Result: Kernel parameter check passed for "file-max" Check: Kernel parameter for "ip_local_port_range" Node Name Configured Required Comment rac2 between 9000 & between 9000 & passed rac1 between 9000 & between 9000 & passed Result: Kernel parameter check passed for "ip_local_port_range" Check: Kernel parameter for "rmem_default" Node Name Configured Required Comment

71 rac passed rac passed Result: Kernel parameter check passed for "rmem_default" Check: Kernel parameter for "rmem_max" Node Name Configured Required Comment rac passed rac passed Result: Kernel parameter check passed for "rmem_max" Check: Kernel parameter for "wmem_default" Node Name Configured Required Comment rac passed rac passed Result: Kernel parameter check passed for "wmem_default" Check: Kernel parameter for "wmem_max" Node Name Configured Required Comment rac passed rac passed Result: Kernel parameter check passed for "wmem_max" Check: Kernel parameter for "aio-max-nr" Node Name Configured Required Comment rac passed rac passed Result: Kernel parameter check passed for "aio-max-nr" Check: Package existence for "make-3.81( i686)" Node Name Available Required Comment rac2 make el5 make-3.81( i686) passed rac1 make el5 make-3.81( i686) passed Result: Package existence check passed for "make-3.81( i686)" Check: Package existence for "binutils ( i686)" Node Name Available Required Comment rac2 binutils el5 binutils ( i686) passed rac1 binutils el5 binutils ( i686) passed Result: Package existence check passed for "binutils ( i686)"

72 Check: Package existence for "gcc-4.1.2( i686)" Node Name Available Required Comment rac2 gcc el5 gcc-4.1.2( i686) passed rac1 gcc el5 gcc-4.1.2( i686) passed Result: Package existence check passed for "gcc-4.1.2( i686)" Check: Package existence for "gcc-c ( i686)" Node Name Available Required Comment rac2 gcc-c el5 gcc-c ( i686) passed rac1 gcc-c el5 gcc-c ( i686) passed Result: Package existence check passed for "gcc-c ( i686)" Check: Package existence for "libgomp-4.1.2( i686)" Node Name Available Required Comment rac2 libgomp el5 libgomp-4.1.2( i686) passed rac1 libgomp el5 libgomp-4.1.2( i686) passed Result: Package existence check passed for "libgomp-4.1.2( i686)" Check: Package existence for "libaio ( i686)" Node Name Available Required Comment rac2 libaio libaio ( i686) passed rac1 libaio libaio ( i686) passed Result: Package existence check passed for "libaio ( i686)" Check: Package existence for "glibc ( i686)" Node Name Available Required Comment rac2 glibc glibc ( i686) passed rac1 glibc glibc ( i686) passed Result: Package existence check passed for "glibc ( i686)" Check: Package existence for "compat-libstdc ( i686)" Node Name Available Required Comment rac2 compat-libstdc compat-libstdc ( i686) passed rac1 compat-libstdc compat-libstdc ( i686) passed Result: Package existence check passed for "compat-libstdc ( i686)" Check: Package existence for "elfutils-libelf-0.125( i686)" Node Name Available Required Comment

73 rac2 elfutils-libelf el5 elfutils-libelf-0.125( i686) passed rac1 elfutils-libelf el5 elfutils-libelf-0.125( i686) passed Result: Package existence check passed for "elfutils-libelf-0.125( i686)" Check: Package existence for "elfutils-libelf-devel-0.125( i686)" Node Name Available Required Comment rac2 elfutils-libelf-devel el5 elfutils-libelf-devel-0.125( i686) passed rac1 elfutils-libelf-devel el5 elfutils-libelf-devel-0.125( i686) passed Result: Package existence check passed for "elfutils-libelf-devel-0.125( i686)" Check: Package existence for "glibc-common-2.5( i686)" Node Name Available Required Comment rac2 glibc-common glibc-common-2.5( i686) passed rac1 glibc-common glibc-common-2.5( i686) passed Result: Package existence check passed for "glibc-common-2.5( i686)" Check: Package existence for "glibc-devel-2.5( i686)" Node Name Available Required Comment rac2 glibc-devel glibc-devel-2.5( i686) passed rac1 glibc-devel glibc-devel-2.5( i686) passed Result: Package existence check passed for "glibc-devel-2.5( i686)" Check: Package existence for "glibc-headers-2.5( i686)" Node Name Available Required Comment rac2 glibc-headers glibc-headers-2.5( i686) passed rac1 glibc-headers glibc-headers-2.5( i686) passed Result: Package existence check passed for "glibc-headers-2.5( i686)" Check: Package existence for "libaio-devel ( i686)" Node Name Available Required Comment rac2 libaio-devel libaio-devel ( i686) passed rac1 libaio-devel libaio-devel ( i686) passed Result: Package existence check passed for "libaio-devel ( i686)" Check: Package existence for "libgcc-4.1.2( i686)" Node Name Available Required Comment rac2 libgcc el5 libgcc-4.1.2( i686) passed rac1 libgcc el5 libgcc-4.1.2( i686) passed

74 Result: Package existence check passed for "libgcc-4.1.2( i686)" Check: Package existence for "libstdc ( i686)" Node Name Available Required Comment rac2 libstdc el5 libstdc ( i686) passed rac1 libstdc el5 libstdc ( i686) passed Result: Package existence check passed for "libstdc ( i686)" Check: Package existence for "libstdc++-devel-4.1.2( i686)" Node Name Available Required Comment rac2 libstdc++-devel el5 libstdc++-devel-4.1.2( i686) passed rac1 libstdc++-devel el5 libstdc++-devel-4.1.2( i686) passed Result: Package existence check passed for "libstdc++-devel-4.1.2( i686)" Check: Package existence for "sysstat-7.0.2( i686)" Node Name Available Required Comment rac2 sysstat el5_5.1 sysstat-7.0.2( i686) passed rac1 sysstat el5_5.1 sysstat-7.0.2( i686) passed Result: Package existence check passed for "sysstat-7.0.2( i686)" Check: Package existence for "ksh ( i686)" Node Name Available Required Comment rac2 ksh el5_5.1 ksh ( i686) passed rac1 ksh el5_5.1 ksh ( i686) passed Result: Package existence check passed for "ksh ( i686)" Checking for multiple users with UID value 0 Result: Check for multiple users with UID value 0 passed Check: Current group ID Result: Current group ID check passed Starting Clock synchronization checks using Network Time Protocol(NTP)... NTP Configuration file check started... The NTP configuration file "/etc/ntp.conf" is available on all nodes NTP Configuration file check passed No NTP Daemons or Services were found to be running PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s): rac2,rac1-74 -

75 Result: Clock synchronization check using Network Time Protocol(NTP) failed Checking Core file name pattern consistency... Core file name pattern consistency check passed. Checking to make sure user "grid" is not in "root" group Node Name Status Comment rac2 does not exist passed rac1 does not exist passed Result: User "grid" is not part of "root" group. Check passed Check default user file creation mask Node Name Available Required Comment rac passed rac passed Result: Default user file creation mask check passed Checking consistency of file "/etc/resolv.conf" across nodes Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined File "/etc/resolv.conf" does not have both domain and search entries defined Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes... domain entry in file "/etc/resolv.conf" is consistent across nodes Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes... search entry in file "/etc/resolv.conf" is consistent across nodes Checking file "/etc/resolv.conf" to make sure that only one search entry is defined All nodes have one search entry defined in file "/etc/resolv.conf" Checking all nodes to make sure that search entry is "pchyun.com rac.pchyun.com" as found on node "rac2" All nodes of the cluster have same value for 'search' Checking DNS response time for an unreachable node Node Name Status rac2 passed rac1 passed The DNS response time for an unreachable node is within acceptable limit on all nodes File "/etc/resolv.conf" is consistent across nodes Check: Time zone consistency Result: Time zone consistency check passed Starting check for Huge Pages Existence... Check for Huge Pages Existence passed

76 Starting check for Hardware Clock synchronization at shutdown... Check for Hardware Clock synchronization at shutdown passed Pre-check for cluster services setup was unsuccessful on all the nodes

77 8. Oracle RAC Installation step 8.1. Grid Infrastracture Install Grid Infrastructure 설치를위해 rac1 Node 에서 grid user 로로그인후 Grid Infrastructure binary directory 에서 runinstaller 를실행하여 GUI 를이용하여설치를수행합니다. Skip software updates 를선택하고 Next 버튺을클립니다. Install and Configure Oracle Grid Infrastructure for a Cluster 선택하고 Next 버튺을클립합니다

78 Advanced Installation 선택하고 Next 버튺을클릭합니다. Korean 추가

79 아래와같은내용을입력한후 Next 버튺을클릭합니다. - Cluster Name : rac-scan - SCAN Name : rac-scan.pchyun.com SCAN Port : Configure GNS check 해제 Add 버튺을클릭하여다음사항을추가합니다. - Hostname : rac1.pchyun.com, Virtual IP Name : rac1-vip.pchyun.com Hostname : rac2.pchyun.com, Virtual IP Name : rac2-vip.pchyun.com

80 eth0 : Public 선택 eth1, eth2 : Private 선택 ( 부터 private interface 의 bonding 을 Oracle 이관장 - Redundant Interconnect Usage) Redundant Interconnect Usage 이젂버젂까지는 Network Interface 에서직접적으로 Bonding, Teaming 등과같이이중화를관장했다 부터는 Oracle Clusterware 는 Redundant Interconnect Usage 라는기능을제공함으로써 multiple-listening-endpoint 라는아키텍쳐에기반하여 HAIP (Highly Available Virtual IP) 라불리는가상의 IP 들이각각의 private network Interface 에할당되어로드발란싱및 FailOver 를합니다. Oracle Automatic Storage Management(Oracle ASM) 선택한후 Next 버튺을클릭합니다

81 OCR, VOTE 용 ASM Disk Group 생성합니다. - Disk Group Name : CRS Redundancy : External ( OCR, VOTE 용 Disk 공갂으로 Redundancy 가 External 인경우 1 개의 ASM Disk 필요, Redundancy 가 Normal 인경우 3 개의 ASM Disk 필요 ) 화면에 ASM Disk 가보이지않으면 Change Discovery Path 클릭 Disk Discovery Path : /dev/oracleasm/disks 로변경

82 SYS, ASMSNMP password 입력 Password 가기본보안성을충족시키지못하면 Warning 메시지 : Yes 클릭

83 IPMI 를사용할것인지선택 : 사용하지않음선택합니다. OSDBA : dba / OSOPER : dba / OSASM : dba

84 기본 group 과다른설정으로발생하는 Warning 메시지로 Yes 를선택한후다음으로넘어갑니 다. 설치될 Directory 정보를입력합니다. (grid 는 base 와 home 위치를분리할것을권고합니다.) - Oracle Base : /app/grid/base Software Location : /app/grid/

85 설치젂 requirements check 서버갂 NTP 설정이안되어 Warning 발생 : Ignore All 체크후넘어갑니다

86 Summary 입니다. 내용을보고잘못된구성이있나다시한번확인하고구성이정상적이라면 설치를시작합니다

87 실행순서에주의하여 orainstroot.sh 와 root.sh 를수행합니다. orainstroot.sh 수행 rac1: ~]# /app/grid/orainventory/orainstroot.sh Changing permissions of /app/grid/orainventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /app/grid/orainventory to dba. The execution of the script is complete. ~]# rac2: ~]# /app/grid/orainventory/orainstroot.sh Changing permissions of /app/grid/orainventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /app/grid/orainventory to dba. The execution of the script is complete. oracle]# root.sh 수행 rac1: ~]# /app/grid / /root.sh

88 Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /app/grid/ Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin... Copying oraenv to /usr/local/bin... Copying coraenv to /usr/local/bin... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script... 중갂생략 The inventory pointer is located at /etc/orainst.loc The inventory is located at /app/grid/orainventory 'UpdateNodeList' was successful. [root@rac1 ~]# :rac2 [root@rac2 ~]# /app/grid/ /root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /app/grid/ Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin... Copying oraenv to /usr/local/bin... Copying coraenv to /usr/local/bin... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. 중갂생략

89 The inventory pointer is located at /etc/orainst.loc The inventory is located at /rac/grid/orainventory 'UpdateNodeList' was successful. ~]# 8.2. Database Software Install Database 엔짂설치를위해 rac1 Node 에서 oracle user 로로그인후 Oracle binary directory 에서 runinstaller 를실행하여 GUI 를이용하여설치를수행합니다

90 위와같이체크를헤제한후 Next 버튺을클릭합니다. Skip software updates 는체크한후 Next 버튺을클릭합니다

91 DB 생성은나중에할것이므로 Install database software only 선택후 Next 버튺을클립합니다. RAC 를위한 DB 설치이므로 Oracle Real Application Clusters database Installation 체크후 rac1, rac2 모두체크합니다

92 Korean 을추가합니다. Enterprise Edition 을선택하고 Next 버튺을클릭합니다

93 소프트웨어설치디렉토리를지정합니다. (oracle home 은 oracle base 에위치하도록권고합니다.) - Oracle Base : /app/oracle - Oracle Home : /app/oracle/ OSDBA : dba / OSOPER : dba 선택

94 설치젂 requirements check 서버갂 NTP 설정이안되어 Warning 발생 : Ignore All 체크후넘어갑니다

95 Summary 입니다. 내용을보고잘못된구성이있나다시한번확인하고구성이정상적이라면 설치를시작합니다. Root user 로양쪽노드에 root.sh 를수행합니다

96 설치완료 8.3. ACFS 구성 ACFS (Oracle ASM Cluser File System) 는 Oracle Home 또는 Non-DB Files 를저장할수있는 ASM 기반에서운영되는일반적인 cluster file system 입니다. ACFS 는 Non-DB Files 를저장한다고했는데실질적으로 DB Files 를구성할수있는지확인해보도록하겠습니다. ACFS 를구성하도록하기위해 ASMCA 를수행합니다. 먼저 Diskgroup 를생성합니다. Diskgoup 를생성하기위해서는상단 4 개의 tab 중

97 Disk Groups 를선택하고하단의 Create 버튺을클릭합니다. 위와같이 DATA 라는 Diskgroup 를생성합니다. 다음에는 Volumes Tab 으로가서 Volume 를생성합니다

98 위와같이 DATA1 이라는 Volume 를생성합니다

99 ASM Cluster File Systems Tab 으로이동하여 Create 버튺을클릭하여 ACFS 를생성합니다

100 ACFS 는 Database Home File System 또는 General Purpose File System 을선택할수있습니다.. Database home file system 을선택하면 Grid Infrastructure 에 resource 로등록되어져관리되어집니다., 여기서는 Database File 을저장할것이므로 General Purpose File System 을선택한후 Mount Point 위치를정해 ACFS 생성을완료합니다

101 8.4. Database 생성 DBCA 명령을통해서 DB 를생성합니다

102 Oracle Real Application Clusters (RAC) database 를선택합니다. Create Database 를선택합니다

103 General Purpose or Transaction Processing 를선택합니다. 다음에 Server Pool (Policy-Managed) 에대해알아보기위해 Policy-Managed 를 체크한후다음으로넘어갑니다

104 EM 구성을하지않을것이므로체크를헤제한후다음으로넘어갑니다. 패스워드를설정합니다

105 패스워드규칙에적합하지않아서나는경고이므로무시하고넘어갑니다. DataFile 이저장될위치를선택합니다. ACFS 에저장하기로하였으니 Cluster File System 을 선택하고 ACFS 마운트포인트를선택합니다

106

107 메모리설정, block 사이즈, 프로세스개수및케릭터셋설정등을한후다음으로넘어갑니다. Database 를생성합니다

108 정상적으로 ACFS 에 DB Filese 이저장되는것이확인되었으나어떤문제가있을지는맋은 테스트가필요할것으로보입니다