Linux 환경에서 3 Node 10g RAC 구성및 Data 이관하기 일시 : 2006. 08. 10
Contents I. RAC 구성환경및전체순서 II. Volume 및 Server 환경 Setup III. CRS 및 DB S/W 설치 IV. CRS 및 DB S/W Patchset V. File-System DBF raw-device 이관 VI. Sinlge DB RAC 수동전환 VII. 기타내용및 QnA 2
I. RAC 구성환경및전체순서 RAC 구성환경 현재는아래총 6 Node 의각각 Single DB 가 10.1.0.3 환경으로각각 Internal Disk 에 Oracle DBMS 가운영중임 ( 백두, 도봉, 지리, 금강, 한라, 태백 ) 향후 HP Storage 를구매하여 6 Node 를하나의 Storage 의 NAS 환경을구성하여 1 번 RAC Group ( 백두, 도봉, 지리 ) :: vg_01 와 2 번 RAC Group ( 금강, 태백, 한라 ) :: vg_02 의각 3 Node 씩구성하며, vg_03 Volume 은 Backup 용도로사용한다. Oracle DBF 는 OCFS2, ASM, raw-device 등의 3 가지방식으로 RAC 구성이가능하나 OCFS2 는성능문제, ASM 은기타문제등으로 raw-device 로결정을함. 향후 Linux Kernel 2.6.x 이상에서는 raw-device 지원이 Deprecated 될예정이어서이렇게되면 ASM 으로전환이필요한상황임. ** 현재고객사의 System 은 Rebooting 시 /dev/raw Folder 자체가사라지므로 Booting 후에반드시 mkdir /dev/raw Folder 를만들어주어야한다. 3
I. RAC 구성환경및전체순서 RAC 구성전체순서 1. OS 의 PV, LV, raw-device 생성및 Owner-ship 설정 - ssh or rsh, scp or rcp 환경설정 - Linux kernel 2.6 이상에서 raw device 가없는경우에수동으로생성!! 2. CRS & DB S/W Install 및 10.2.0.2 Patchset - Public & Private Node 설정정확히!! 3. File-System DBF raw-device 로전환 - 하나의 LV 는 32G 를넘지않도록!! 4. raw-device 를이용하여 Single 로 DB 를 Open 함과동시에 10.2.0.1 (or 10.1.0.3) 10.2.0.2 로 DBMS Upgrade 실시 - startup upgrade or alter database open upgrade; 5. Single DB RAC 로수동으로전환 - 2,3 번 Thead 의 Redo, Undo 생성및파라미터수정 6. 기타 Resource 등록및 netca, emca 수동실행 - srvctl add command 로수동으로등록!! 7. 기타 TAF 및 CTF 는 Fail-Over 설정및 Test - LocalListener 4
II. Volume 및 Server 환경 Setup vg_01 vg_02 1 2 3 1 2 3 88 79 92 89 90 91 백두도봉지리금강태백한라 raw1 ~ raw50 raw51 ~ raw100 raw101 ~ raw150 raw151 ~ raw200 raw201 ~ raw250 raw251 ~ raw300 vgchange a y vg_01 vgchange a y vg_02 * Physical Volume /dev/sdb1 :: vg_01 /dev/sdc1 :: vg_02 /dev/sdd1 :: vg_03 * System Booting 시 /root/rac_on.sh * Service Down srvctl stop databse -d kumkang srvctl stop nodeapps n kumkang crsctl stop crs root 유저에서 vg_01 OCR1 :: raw96 OCR2 :: raw97 Vote1 :: raw98 Vote2 :: raw99 Vote3 :: raw100 vg_02 OCR1 :: raw296 OCR2 :: raw297 Vote1 :: raw298 Vote2 :: raw299 Vote3 :: raw300 vg_03 :: Backup 공간 * DBF Mapping 순서 1 :: pwdfile 2 :: spfile 3~5 :: controlfile1~3 6~20 :: redolog1~3 21 :: system 22~24 :: undo1~3 25 :: sysaux 26 :: users 27 :: temp 28 ~ :: 일반 DBF 5
II. Volume 및 Server 환경 Setup Linux LV & raw-device 구성 1. Physical LV & Volume Group pvcreate /dev/sdb1 ~ /dev/sdd1 vgcreate vg_01 /dev/sdb1 2. LV 생성 Linux 는하나의 LV 가 2TB 이상지원이되나 Oracle Block 8K 는 Max : 32GB lvcreate -L 202m -n OCR1_01 /dev/vg_01 lvcreate -L 102m -n VOTE1_01 /dev/vg_01 3. raw device 용 node 생성 raw-device 이름별로생성한다. mknod /dev/raw/raw101 c 162 101 mknod /dev/raw/raw102 c 162 102 4. LV 와 raw-device Mapping /etc/sysconfig/rawdevcies /dev/raw/raw96 /dev/vg_01/ocr1_01 /dev/raw/raw97 /dev/vg_01/ocr1_02 5. Hang-Check timer Enable modprobe hangcheck-timer 6. raw-device 의 Owner-ship & Permission 6
II. Volume 및 Server 환경 Setup Volume 정보 [root@halla 0731]# pvscan PV /dev/sdd1 VG vg_03 lvm2 [1.37 TB / 1.37 TB free] PV /dev/sdc1 VG vg_02 lvm2 [1.95 TB / 1.69 TB free] PV /dev/sdb1 VG vg_01 lvm2 [2.00 TB / 1.48 TB free] Total: 3 [5.32 TB] / in use: 3 [5.32 TB] / in no VG: 0 [0 ] [root@jiri ~]# lvscan ACTIVE '/dev/vg_02/lv_kumkang' [70.00 GB] inherit ACTIVE '/dev/vg_02/lv_taebaek' [90.00 GB] inherit ACTIVE '/dev/vg_02/lv_halla' [120.00 GB] inherit [root@jiri ~]# raw qa 로 raw-device 확인!! [root@baekdu ~]# raw -qa /dev/raw/raw1: bound to major 253, minor 30 /dev/raw/raw2: bound to major 253, minor 31 /dev/raw/raw3: bound to major 253, minor 32 /dev/raw/raw4: bound to major 253, minor 33 /dev/raw/raw5: bound to major 253, minor 34 /dev/raw/raw6: bound to major 253, minor 35 /dev/raw/raw7: bound to major 253, minor 36 7
III. CRS 및 DB S/W 설치 CRS S/W Install 8
III. CRS 및 DB S/W 설치 CRS S/W Install 9
III. CRS 및 DB S/W 설치 CRS S/W Install 10
III. CRS 및 DB S/W 설치 CRS S/W Install 11
III. CRS 및 DB S/W 설치 DB S/W Install 12
III. CRS 및 DB S/W 설치 DB S/W Install 13
III. CRS 및 DB S/W 설치 DB S/W Install 14
IV. CRS 및 DB S/W Patchset CRS Install - Patchset 15
IV. CRS 및 DB S/W Patchset CRS Install - Patchset 16
IV. CRS 및 DB S/W Patchset CRS Install - Patchset 17
IV. CRS 및 DB S/W Patchset CRS 구성 - Patchset 18
IV. CRS 및 DB S/W Patchset DB Install - Patchset 19
IV. CRS 및 DB S/W Patchset Cluster Net 환경구성 20
IV. CRS 및 DB S/W Patchset Cluster Net 환경구성 21
IV. CRS 및 DB S/W Patchset Cluster Net 환경구성 22
V. File-System DBF raw-device 이관 File-System DBF raw-device 이관및 10.2.0.1 10.2.0.2 Patchset 1. dd 로 File-Copy ( 실시간또는 On-Line 중에 ) dd if=/oracle/oradata/kumkang/control01 of=/dev/raw/raw253 bs=8192 dd if=/oracle/oradata/kumkang/redo01.log of=/dev/raw/raw256 bs=8192 dd if=/oracle/oradata/kumkang/system01.dbf of=/dev/raw/raw271 bs=8192 2. Controlfile 재생성또는 DBF rename 방식중택일!! Create controlfile reuse KUMKANG Noresetlogs archivelog maxinstances 8 3. DB Open 및 Temp DBF 추가!! SQL> alter database open upgrade; 4. DB Upgrade (Patchset) Compatible 등기타 Parameter 수정!! SQL> @?/rdbms/admin/catupgrd SQL> @?/rdbms/admin/utlrp 23
VI. Single RAC 수동전환 Single RAC 수동전환 1. Cluster View Install SQL> @?/rdbms/admin/catclust 2. 2,3 번 Thread 용 Redo 생성 SQL> alter database add logfile thread 2 group 6 '/dev/raw/raw161' size 200M; SQL> alter database add logfile thread 2 group 7 '/dev/raw/raw162' size 200M; SQL> alter database add logfile thread 2 group 8 '/dev/raw/raw163' size 200M; SQL> alter database add logfile thread 2 group 9 '/dev/raw/raw164' size 200M; SQL> alter database add logfile thread 2 group 10 '/dev/raw/raw165' size 200M; SQL> alter database add logfile thread 3 group 11 '/dev/raw/raw166' size 200M; SQL> alter database add logfile thread 3 group 12 '/dev/raw/raw167' size 200M; SQL> alter database add logfile thread 3 group 13 '/dev/raw/raw168' size 200M; SQL> alter database add logfile thread 3 group 14 '/dev/raw/raw169' size 200M; SQL> alter database add logfile thread 3 group 15 '/dev/raw/raw170' size 200M; 3. 2,3 번 Undo Tablespace 생성 SQL> create undo tablespace undotbs2 datafile '/dev/raw/raw173' size 2000M reuse AUTOEXTEND OFF; SQL> create undo tablespace undotbs3 datafile '/dev/raw/raw174' size 2000M reuse AUTOEXTEND OFF; 4. 2,3 번 Thread Enable SQL> alter database enable thread 2; SQL> alter database enable thread 3; 24
VI. Single RAC 수동전환 Single RAC 수동전환 5. RAC init<sid>.ora Setup ## For RAC *.cluster_database=true *.cluster_database_instances=3 #*.remote_listener=listeners_kumkang kumkang1.instance_name=kumkang1 kumkang2.instance_name=kumkang2 kumkang3.instance_name=kumkang3 kumkang1.instance_number=1 kumkang2.instance_number=2 kumkang3.instance_number=3 kumkang1.thread=1 kumkang2.thread=2 kumkang3.thread=3 kumkang1.undo_tablespace=undotbs1 kumkang2.undo_tablespace=undotbs2 kumkang3.undo_tablespace=undotbs3 6. SPFILE 생성및 2,3 번 Node 에진행 SQL> create spfile= /dev/raw/raw251 from pfile; $ vi $ORACLE_HOME/dbs/init<SID>.ora SPFILE= /dev/raw/raw251 25
VI. Single RAC 수동전환 Single RAC 수동전환 7. Check RAC Instances SQL> select instance_name from gv$instance; INSTANCE_NAME ---------------- kumkang2 kumkang1 kumkang3 8. Resource 등록 srvctl add database -d kumkang -o /oracle/product/10.1.0 srvctl add instance -d kumkang -i kumkang1 -n kumkang srvctl add instance -d kumkang -i kumkang2 -n taebaek srvctl add instance -d kumkang -i kumkang3 -n halla 9. Cluster Net 환경구성 netca 10. EM 환경구성 $ emca -deconfig dbconsole db -repos drop $ emca -config dbconsole db -repos -cluster $ emca c r a R1!! 26
VII. 기타내용및 QnA /etc/hosts # Node1 219.240.39.88 baekdu 219.240.39.79 dobong 219.240.39.92 jiri 192.168.0.88 baekdu_priv 192.168.0.79 dobong_priv 192.168.0.92 jiri_priv 219.240.40.88 baekdu-vip 219.240.40.79 dobong-vip 219.240.40.92 jiri-vip 27
VII. 기타내용및 QnA System Rebooting 시실행 Script!! -- /root/rac_on.sh 1 번 RAC Group 서버 ( 백두 + 도봉 + 지리 ) ## Enable hangcheck-timer Module by Goodus /sbin/modprobe hangcheck-timer # vgscan by sasair /sbin/vgscan /sbin/vgchange -ay vg_01 ## Start rawdevice /sbin/service rawdevices restart ## rawdevice Ownership /bin/chmod 600 /dev/raw/raw* /bin/chown oracle:dba /dev/raw/raw* /bin/chown root:dba /dev/raw/raw96 /bin/chown root:dba /dev/raw/raw97 /bin/chmod 640 /dev/raw/raw96 /bin/chmod 640 /dev/raw/raw97 #/bin/chown root:dba /dev/raw/raw296 #/bin/chown root:dba /dev/raw/raw297 #/bin/chmod 640 /dev/raw/raw296 #/bin/chmod 640 /dev/raw/raw297 28
VII. 기타내용및 QnA RAC 구성후 [oracle@kumkang network]$ crsstat HA Resource Target State ----------- ------ ----- ora.halla.listener_halla.lsnr ONLINE ONLINE on halla ora.halla.db ONLINE ONLINE on taebaek ora.halla.gsd ONLINE ONLINE on halla ora.halla.halla1.inst ONLINE ONLINE on kumkang ora.halla.halla2.inst ONLINE ONLINE on taebaek ora.halla.halla3.inst ONLINE ONLINE on halla ora.halla.ons ONLINE ONLINE on halla ora.halla.vip ONLINE ONLINE on halla ora.kumkang.listener_kumkang.lsnr ONLINE ONLINE on kumkang ora.kumkang.db ONLINE ONLINE on kumkang ora.kumkang.gsd ONLINE ONLINE on kumkang ora.kumkang.kumkang1.inst ONLINE ONLINE on kumkang ora.kumkang.kumkang2.inst ONLINE ONLINE on taebaek ora.kumkang.kumkang3.inst ONLINE ONLINE on halla ora.kumkang.ons ONLINE ONLINE on kumkang ora.kumkang.vip ONLINE ONLINE on kumkang ora.taebaek.listener_taebaek.lsnr ONLINE ONLINE on taebaek ora.taebaek.db ONLINE ONLINE on taebaek ora.taebaek.gsd ONLINE ONLINE on taebaek ora.taebaek.ons ONLINE ONLINE on taebaek ora.taebaek.taebaek1.inst ONLINE ONLINE on kumkang ora.taebaek.taebaek2.inst ONLINE ONLINE on taebaek ora.taebaek.taebaek3.inst ONLINE ONLINE on halla ora.taebaek.vip ONLINE ONLINE on taebaek 29
VII. 기타내용및 QnA System Down 및 Rebooting 순서 1. System Rebooting 시.. ( 예 : jiri 서버 kumkang3, dobong3, jiri3 Instance 가기동중..) srvctl stop instance d kumkang i kumkang3 srvctl stop instance d dobong i dobong3 srvctl stop instance d jiri i jiri3 srvctl stop nodeapps n jiri 이때 vip 는백두나도봉으로자동 Fail-Over!! [root] crsctl stop crs [root] sync; shutdown r now 2. 필요한 raw-device 생성순서 ( 예 : 1 번 RAC Group) srvctl stop database d kumkang, dobong, jiri srvctl stop nodeapps n kumkang, dobong, jiri [root] crsctl stop crs (kumkang, dobong, jiri 각서버별로!!) [root[ vgchange a n vg_01 (kumkang, dobong, jiri 각서버별로!!) [root] lvcreate 로 LV 생성 [root] mknod /dev/raw/raw101 c 162 로 raw device 생성 [root[ vgchange a y vg_01 (kumkang, dobong, jiri 각서버별로!!) [root] /etc/sysconfig/rawdevices 에신규 raw-device 등록 [root] service rawdevices restart [root] chown oracle:dba /dev/raw/raw*, chmod 600 /dev/raw/raw* [root] chown root:dba /dev/raw/raw96~97, chmod 640 /dev/raw/raw96~97 [root] crsctl start crs (kumkang, dobong, jiri 각서버별로!!) 30
VII. 기타내용및 QnA RAC 구성후 DB 작업 1. DIAG Daemon Disable alter system set trace_enabled=false sid='*'; 2. Shared Server Disable alter system set shared_servers=0 sid='*'; 3. Server 별 Service 분산 ** 한라서버때문에다른 Server 에영향을주는것으로판단이되어금강, 태백에는한라 Instance 제외, 한라에는한라 Instance 만 Start!! [oracle@kumkang cssd]$ srvctl stop instance -d halla -i halla1 [oracle@kumkang cssd]$ srvctl stop instance -d halla -i halla2 [oracle@kumkang cssd]$ srvctl stop instance -d kumkang -i kumkang3 [oracle@kumkang cssd]$ srvctl stop instance -d taebaek -i taebaek3 $ mv halla1.inst.cap ora.halla.halla1.inst.cap $ mv halla2.inst.cap ora.halla.halla2.inst.cap $ crs_register -u ora.halla.halla1.inst -dir /tmp $ crs_register -u ora.halla.halla2.inst -dir /tmp $ crs_stat -p ora.taebaek.taebaek3.inst > /tmp/ora.taebaek.taebaek3.inst.cap $ crs_stat -p ora.kumkang.kumkang3.inst > /tmp/ora.kumkang.kumkang3.inst.cap $ crs_register -u ora.taebaek.taebaek3.inst -dir /tmp $ crs_register -u ora.kumkang.kumkang3.inst -dir /tmp 31
VII. 기타내용및 QnA RAC 구성후 DB 작업 4. 자동통계기능 Disable SQL> alter system set statistics_level=basic sid='*'; SQL> exec dbms_stats.alter_schema_tab_monitoring(null,true); SQL> exec DBMS_SCHEDULER.DISABLE('GATHER_STATS_JOB'); SQL> alter system set statistics_level=basic scope=spfile sid='*'; 5. RMAN Upgrade Catalog DB 가아니면필요없음 (Controlfile 이용하므로 ) $ rman catalog username/password@alias RMAN> upgrade catalog; 6. DB Control 재구성 $ emca -deconfig dbconsole db -repos drop $ emca -config dbconsole db -repos -cluster 32
VII. 기타내용및 QnA RAC 환경의 RMAN 작업시 Single 환경에서수동으로 RAC 환경으로전환을하면 RMAN 에의해 flash_recovery_area 에자동으로 archived log 를비롯해서각종 Backup 화일들이생성이됨. No-Catalog 환경에서 RMAN 관련작업시에는반드시아래와같이 RAC 각 Node 에동시에접속하여 Backup 및기타작업을수행하도록한다.. $ rman target / CONFIGURE RETENTION POLICY TO REDUNDANCY 3; CONFIGURE BACKUP OPTIMIZATION OFF; CONFIGURE DEFAULT DEVICE TYPE TO DISK; CONFIGURE CONTROLFILE AUTOBACKUP ON; CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/backup/flash_recovery_area/halla/%f'; CONFIGURE DEVICE TYPE DISK PARALLELISM 3; CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; CONFIGURE MAXSETSIZE TO UNLIMITED; CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/backup/flash_recovery_aread/halla/snapcf_v92321.f'; CONFIGURE CHANNEL 1 DEVICE TYPE DISK connect 'SYS/PWD@halla1'; CONFIGURE CHANNEL 2 DEVICE TYPE DISK connect 'SYS/PWD@halla2'; CONFIGURE CHANNEL 3 DEVICE TYPE DISK connect 'SYS/PWD@halla3'; CROSSCHECK ARCHIVELOG ALL; DELETE ARCHIVELOG UNTIL TIME sysdate-7 ; 33
감사합니다