Getting the most out of MetaLink 남궁혁 한국오라클 ( 주 ) 제품지원실
목차 1. Cluster 개요 2. OPROCD 3. EVMD 4. CSSD 5. CRSD 6. CRS Install 7. 참고자료
Clusters Alternative to symmetric multiprocessing (SMP) Group of interconnected, whole computers working together as a unified computing resource Illusion is that it is one machine Each individual node can run on its own.
Cluster 의 Components 클러스터구성 component Nodes와 Managers Nodes 리소스를제공하는 system Cluster Manager 두개의 node를하나의 machine처럼작동하게하는로직을제공
The Oracle Cluster VIP1 Node1 Database instance 1 ASM Instance 1 CRS Operating System public network cluster interconnect cache to cache VIP2 Node2 Database instance 2 ASM Instance 2 CRS Operating System cluster VIP3 Node3 Database instance 3 interconnect... ASM Instance 3 CRS Operating System shared storage concurrent access from every node = scale out redo logs all instances Database files control files OCR and Voting Disk more nodes = higher availability
Architecture Important Concepts: Services Node Applications Cluster Ready Services Cluster Resources
Cluster Applications Architecture AP AP HR HR Services CRM CRM Instance 1 Instance 2 Instance 3 Database Node Applications Listener OEM Ag. VIP Listener OEM Ag. VIP Listener OEM Ag. Cluster Ready Services Node 1 Node 2 Node 3 VIP
Cluster Applications Architecture
CRS [Cluster Ready Services] CRS is a stack of portable clusterware components: Exposed 10i CRS Major internal components CSS EVM CRS (OCR)
CRS Structure - Runtime View Started by init(1m) or Windows service controller init oprocd evmd cssd crsd evmlogger racgevt action action callout action action action action action
OPROCD [Processor Monitor Daemon] init oprocd
OPROCD [Processor Monitor Daemon] Our solution to Cluster I/O Fencing in 10g. Processor monitor for Cluster Provides hang check timer for cluster integrity [Hang check timer on Linux] Locked in memory, real time. Sleep a fixed time If wake up time is too much later, reset processor & reboot. Different implementations on different machines Runs as root. Failure causes reboot.
EVMD and descendants Event Forwarding Daemon init evmd evmlogger racgevt action action callout
EVMD [Event Forwarding Daemon] Spawns a permanent child evmlogger Evmlogger, on demand, spawns RACGEVT children RACGEVT scans callout directory and invokes callouts. Runs as Oracle. Restarted automatically on failure
EVM Architecture
CSSD [Cluster Synchronization Service] init cssd
CSSD [Cluster Synchronization Service] Group Services Extension of services previously provided by skgxn Oracle database instances join groups Existing members notified when new members become active in cluster, e.g. database instances Lock Services Lock services used by CRSD Shared and exclusive locks
CSSD [Cluster Synchronization Service] Node Information Services Cluster configuration information Static information Node name Node number Etc. Nodes configured may change Add node Delete node
CSSD [Cluster Synchronization Service] Multi-threaded application Does not require privileged mode Two distinct parts Node Monitor Group Manager Uses Network Services, for communication, both internode and intra-node
CSS Initial Cluster formation Connect attempts for all configured nodes Send joining msg to all active nodes Wait for reconfig to start If no reconfig starts within certain amount of time, lowest numbered node starts reconfig All joining nodes included in reconfig
Group Manager Manages group and lock services Locks are special types of groups One node serves as master All nodes serialize requests through master node Master node broadcasts membership changes to all nodes Group membership synchronized at each cluster reconfiguration
Client Interaction Node/cluster configuration information only requires access to OCR Most services require connection with CSS daemon Client communication uses Net Services
CSS in 2 Node Cluster RDBMS1 RDBMS2 RDBMS1 RDBMS2 CSS Daemon GM NM CSS Daemon GM NM OCR Voting Disk
CRSD init crsd action action action action action
CRSD Engine for HA operation Manages application resources Starts, stops, checks and fails them over Generates events when things happen Spawns separate actions to start/stop/check application resources Maintains configuration profiles in OCR. Stores current known state in OCR. Provides process for OCR caching ( ocrd ) Runs as root Restarted automatically on failure
CRS Resources When an oracle entity is created: CRS application resources are created automatically by Oracle tools May have dependencies on other resources May comprise more than one CRS resource
CRS Resources [10gR2] Lifecycle of a CRS resource crs_profile: Create/Edit the resource s attributes crs_register: Add the resource crs_start: Start the resource crs_stat: Find out about the resource crs_relocate: Move the resource crs_stop: Stop the resource crs_unregister: Remove the resource
CRS Resources: Resource Security Security attributes on a resource: crs_setperm crs_getperm % crs_getperm ora.starwars1.vip Name: ora.starwars1.vip owner:root:rwx,pgrp:svrtech:rwx,other::r--, user:oracle:r-x,
CRS application resources Resource state can be determined from the dynamic attributes: STATE TARGET NAME=ora.myResource1 Example crs_stat output: TYPE=application TARGET=ONLINE STATE=ONLINE on alphab-2 NAME=ora.myResource2 TYPE=application TARGET=ONLINE STATE=ONLINE on alphab-2
CRS Action
Starting CRS
Oracle Cluster Registry A mechanism for storing information. Clients include SRVM and CRS Directory tree structure Key-value pairs
OCR Cache Architecture Default clients talk to their local OCR cache OCR caches talk to a cluster master cache Master cache does actual reads/writes to storage Operations atomic per-key value update Writes complete when in storage and all caches are invalidated.
OCR Record Structure Each OCR record contains 3 fields: The key, a value and permissions Example: Database Instance record
Voting disk Used to determine quorum in case of failure and provides a second heartbeat mechanism to validate cluster health. Disk heartbeat maintained in voting disk Voting disk used to ascertain cluster state Voting disk has structures for each node in cluster During evictions, the voting disk contains the eviction message Voting disk needs to be mirrored, should it become unavailable, cluster will come down.
Installation Oracle Installer places software/configuration information in ORA_CRS_HOME Root.sh scripts establish init integration. RAC install runs CRS as root EVM as oracle CSS as oracle, node fatal on unexpected CSSD failure
Installation init integration Use init(1m) to restart critical processes Run from entries in /etc/inittab with respawn Rc.d scripts establish state Set values in flags files read by programs started by init Rc.d scripts do NOT directly start programs. Supported method to start is booting the machine Supported method to stop is shutdown the machine. For maintenance, go single user
Installation Some parameters are in Logfiles are in /etc/init.d/init.crsd /etc/init.d/init.cssd /etc/init.d/init.evmd $ORA_CRS_HOME/crs/log $ORA_CRS_HOME/css/log $ORA_CRS_HOME/evm/log Ensure root.sh scripts are run on all nodes during install
Diagnostics For resources owned by non-root users, profiles and action scripts are in $ORA_CRS_HOME/public For resources owned by the root user, profiles are in $ORA_CRS_HOME/crs/profile, action scripts are in $ORA_CRS_HOME/script CRSD log file: $ORA_CRS_HOME/crs/log/<nodename>.log Log trace contains Timestamp and Thread-id. Most of the diagnostic information comes from this log. CRSD core: $ORA_CRS_HOME/crs/init/<nodename>/core Other system files in $ORA_CRS_HOME/crs/init: <nodename>.pid, <nodename>.flags
참고문서 Note. 274526.1 How to download and install opatch Note. 242993.1 OPATCH FAQ (internal only) Note. 244241.1 Rolling Patch - OPatch Support for RAC Note. 189489.1 Oracle9i Data Server Interim Patch Installation OPATCH USERS GUIDE (Opatch/docs directory) OPATCH FAQ (Opatch/docs directory)